{"id":20056,"date":"2020-11-05T14:19:16","date_gmt":"2020-11-05T13:19:16","guid":{"rendered":"https:\/\/www.inovex.de\/blog\/?p=20056"},"modified":"2022-12-02T08:34:44","modified_gmt":"2022-12-02T07:34:44","slug":"grafana-annotations-prometheus-deep-dive","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/","title":{"rendered":"Grafana Annotations with Prometheus (a Deep Dive)"},"content":{"rendered":"<p>Grafana is often used in conjunction with Prometheus to visualize time series and compose dashboards for monitoring purposes. A variety of data fits really well into time series and can be visualized as graphs \u2013 such as request count, request duration or response size. Other data is more event-driven and does not quite fit in a graph \u2013 for example process restarts, config reloads or the deployment of a new release. The latter tend to happen less frequently and most of the time you would be looking at a flat line if you dedicate a graph for them. However, the events are still interesting to monitor and to correlate with the graphs. For example, a config reload of a malfunctioned config could explain a drop in requests. A nice way to do that in Grafana is to use annotations.<\/p>\n<p>In this blog article, I will dive deep into the specifics of annotations and how to use them with Prometheus as a data source.<!--more--><\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\"><p class=\"ez-toc-title\" style=\"cursor:inherit\"><\/p>\n<\/div><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/#What-are-annotations\" >What are annotations?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/#Annotations-from-Prometheus\" >Annotations from Prometheus<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/#Useful-annotations\" >Useful annotations<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/#Prometheus-Alerts\" >Prometheus Alerts<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/#Pod-restarts\" >Pod restarts<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/#Config-reload\" >Config reload<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/#Creation-of-a-pod\" >Creation of a pod<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/#Conclusion\" >Conclusion<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"What-are-annotations\"><\/span>What are annotations?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Annotations are descriptions of points in time which can be added to dashboards. They show as vertical lines in the panels and, when hovering over them, reveal a descriptive text and optionally a set of tags. Annotations can be technical or non-technical. For example, an annotation can be created for the start of a new marketing campaign in order to monitor its effectiveness by looking at the incoming requests. Annotations can be created manually, either directly in a panel or using the API. They are stored in Grafana&#8217;s database and are associated with the dashboard in which they were created. By default, annotations are only shown in the dashboard in which they were created. However, annotations can also be queried and displayed using a tag selector. Annotations are shown on all graphs of a dashboard. Support for annotations on a single graph are discussed <a href=\"https:\/\/github.com\/grafana\/grafana\/issues\/717\">here<\/a>.<\/p>\n<figure id=\"attachment_20066\" aria-describedby=\"caption-attachment-20066\" style=\"width: 845px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-20066 size-full\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/example_simple.png\" alt=\"\" width=\"845\" height=\"639\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/example_simple.png 845w, https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/example_simple-300x227.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/example_simple-768x581.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/example_simple-400x302.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/example_simple-360x272.png 360w\" sizes=\"auto, (max-width: 845px) 100vw, 845px\" \/><figcaption id=\"caption-attachment-20066\" class=\"wp-caption-text\">Fig. 1: A manually created annotation.<\/figcaption><\/figure>\n<h2><\/h2>\n<p>Grafana also has support for region annotations. They have a start and end time (which is different from the former) and the region in between is shown in a slightly different color on the panels. They can, for example, be used to denote an outage or the lifetime of a job.<\/p>\n<figure id=\"attachment_20068\" aria-describedby=\"caption-attachment-20068\" style=\"width: 877px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-20068 size-full\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/region_annotation.png\" alt=\"\" width=\"877\" height=\"570\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/region_annotation.png 877w, https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/region_annotation-300x195.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/region_annotation-768x499.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/region_annotation-400x260.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/region_annotation-360x234.png 360w\" sizes=\"auto, (max-width: 877px) 100vw, 877px\" \/><figcaption id=\"caption-attachment-20068\" class=\"wp-caption-text\">Fig. 2: The red area is a range annotation.<\/figcaption><\/figure>\n<p>Manual annotations work well for non-technical events or to enrich graphs after a debugging session. However, it would be cumbersome for automated events. To that end, Grafana allows to query annotations from external data sources. Multiple data sources and queries can be defined per dashboard. They are visualized in different colors and can be individually hidden by a toggle switch at the top of the dashboard. We will have a closer look at querying Prometheus for annotations.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Annotations-from-Prometheus\"><\/span>Annotations from Prometheus<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Data returned by Prometheus can be used to show annotations. As a prerequisite, Prometheus has to be configured as a data source in Grafana. To show time series in Grafana you have to specify a PromQL query. Annotations use that same query language. But how does Grafana transform the time series into annotations? Prometheus has no concept of annotations. Unfortunately, this feature is not well documented. I had a browse through the code and share my findings here. If you are not interested in the technical details, skip to the next section to find some examples for inspiration.<\/p>\n<p>To get started, go into the dashboard settings of your dashboard and choose \u201cAnnotations\u201c in the menu. It will show you all existing configurations, however, we want to create a new one by clicking \u201cNew\u201c. The settings \u201cEnabled\u201c, \u201cHidden\u201c and \u201cColor\u201c give us some options to manage a multitude of annotations more easily. We leave those settings as they are in this example. When Prometheus is properly configured, we can choose it as a data source here. Afterwards, we can specify a search expression, i.e. a PromQL query. Grafana performs a range query for annotations. That is, our PromQL query returns a vector of time series where each element covers the time range of the dashboard. Of the response, each time series is processed independently. Grafana considers all non-zero and non-null values for annotations. The remaining data points are gaps. Every non-zero and non-null data point which follows a gap denotes the beginning of a new annotation. All subsequent data points belong to the same annotation (which would make it a region annotation), until the next gap is reached.<\/p>\n<p>Let us consider an example. Suppose we want to <a href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/configuration\/alerting_rules\/#inspecting-alerts-during-runtime\">monitor Prometheus alerts<\/a>. Fig. 3 shows the required configuration. Fig. 2 shows a sample annotation.<\/p>\n<figure id=\"attachment_20064\" aria-describedby=\"caption-attachment-20064\" style=\"width: 1017px\" class=\"wp-caption alignright\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-20064 size-full\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/annotations_settings.png\" alt=\"\" width=\"1017\" height=\"740\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/annotations_settings.png 1017w, https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/annotations_settings-300x218.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/annotations_settings-768x559.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/annotations_settings-400x291.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/10\/annotations_settings-360x262.png 360w\" sizes=\"auto, (max-width: 1017px) 100vw, 1017px\" \/><figcaption id=\"caption-attachment-20064\" class=\"wp-caption-text\">Fig. 3: Configuring Grafana annotations for Prometheus alerts.<\/figcaption><\/figure>\n<p>The \u201cstep\u201c option can be used to change the resolution of the query (it is passed to Prometheus\u2019 range query API). The default is 60s. The setting is independent of the selected time range of the dashboard, with one exception. If the range query reaches Prometheus\u2019 maximum allowed data points of 11000 per time series, then the resolution is reduced to stay in bound of this limit. The static step size is different from panels, which adapt their resolution to avoid unnecessary expensive queries against Prometheus.<\/p>\n<p>The appropriate setting for steps depends on the type of event you are trying to obtain from the time series. It is a tradeoff between performance and integrity. If the resolution is too low an event might occur between two data points or two independent events might be merged because the gap between them is not sampled.<\/p>\n<p>Another aspect to consider when choosing the right value for the \u201cstep\u201c option is the visualization. With too many annotations in a small window, all the horizontal lines take away the focus of the underlying graph. Grafana only shows one annotation per step. That is, the lower the step size, the fewer annotations can potentially clutter the panels. The annotation configurations can be reordered to give priority to certain annotations.<\/p>\n<p>To keep the number of annotations manageable but still avoid missing an annotation you can make use of Prometheus\u2019 range vectors (not to be confused with range queries) in conjunction with a high step size. Range vectors must be aggregated over time before querying. A common use case is to calculate the rate of some counter. If that rate is mostly zero, it might be well suited for an annotation. A good example is the restart of a process.<\/p>\n<pre class=\"lang:default decode:true\">rate(process_restarts[30m])<\/pre>\n<p>If you choose a step size and range of 30 minutes, there will be at most one of those annotations in an half hour interval. On the downside, the time of the annotation may be off by 30 minutes and you don\u2019t know how many restarts occurred.<\/p>\n<p>Each annotation has a title, a text and a set of tags (see Fig. 1). The title and text are templated strings. They can include labels from the time series. The tags are sourced from label values, given a list of label keys. As of now, tags of external annotations are only displayed. There is no query or filter ability. Limited string manipulation for all annotation fields can be achieved by levering Prometheus\u2019 <a href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/functions\/#label_replace\">label_replace()<\/a>.<\/p>\n<p>By default, the values of the time series are only regarded to distinguish gaps (i.e. zero values) from annotations. Their actual values are of no further interest to Grafana and are not available for templating. However, some components export metrics where the value is a timestamp. To that end, Grafana supports taking the timestamp from the value by ticking the \u201cSeries value as timestamp\u201c option.<\/p>\n<p>Let\u2019s look at an example. The <a href=\"https:\/\/github.com\/kubernetes\/kube-state-metrics\">kube-state-metrics exporter<\/a> exposes the creation time of a pod as a metric. We can use that to visualize the creation of pods as annotations.<\/p>\n<pre class=\"lang:default decode:true\">kube_pod_created{pod=~\"my-pod-.*\"} * 1000<\/pre>\n<p>There are a few things to consider here.<\/p>\n<ul>\n<li>Grafana expects the values in milliseconds. By convention, Prometheus uses seconds, hence, we multiply by a thousand.<\/li>\n<li>The time series <span class=\"lang:default decode:true crayon-inline\">kube_pod_created<\/span> only exists for the lifetime of the pod. That means, to appear as an annotation the time series must be returned by Prometheus with at least one data point (the time series is constant over its lifetime). Even if the timestamp is taken from the value, Grafana still performs a range query. Hence, as long as the lifetime is longer than the configured step size the annotation is sampled.<\/li>\n<li>If the time series \u201cappears\u201c when the pod is created, why not just use the timestamp of the time series? This might be sufficient in some cases and, in fact, is a good workaround if no such time series is available. However, it comes with some drawbacks. First, you have to adapt the query to detect the rising edge, otherwise the annotation will become a region annotation. Second, the exported timestamp gives you a much more accurate representation. Third, if the exporter is down, then the metrics will show much later in Prometheus or might contain gaps, which is not a problem if you take the timestamp from the value.<\/li>\n<li>The example metric we used here has a constant value, the creation date never changes. What about time series that do change, for example a predicted end time? The behavior of this scenario is not documented and a quick look at the code reveals that Grafana does not properly handle this scenario in many cases. Timestamps are processed in the order in which they appear in the time series. This messes up the calculation of region annotations if the timestamps are non-monotonic. Also, there is no way to only take the last value of a time series, because Grafana always performs a range query. TL;DR If you want to stay in safe waters, only use constant timestamp metrics.<\/li>\n<\/ul>\n<p>A commonly used feature in Grafana are variables. They can be used to abstract particular aspects of dashboards and make them more flexible and versatile. Variables are defined in the dashboard settings and can be referenced in panels to customize their queries. The same works for annotation queries. For example:<\/p>\n<pre class=\"lang:default decode:true\">kube_pod_created{namespace=\"$namespace\"} * 1000<\/pre>\n<h2><span class=\"ez-toc-section\" id=\"Useful-annotations\"><\/span>Useful annotations<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Which annotations you want to show depends on your needs. Here are a few examples to get you started.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Prometheus-Alerts\"><\/span>Prometheus Alerts<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>If you manage your alerts in Prometheus, it would be nice to see them in Grafana as well.<\/p>\n<pre class=\"lang:default decode:true\">ALERTS{alertstate=\"firing\", component: \u201cdatabase\u201c}<\/pre>\n<p>The labels from the alerts are available for filtering and templating, for example by component. Alerts will be displayed as region annotations for the period in which they are firing.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Pod-restarts\"><\/span>Pod restarts<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Process restarts might be the cause of a service degradation. If you operate in a Kubernetes environment, restarts metrics are exposed by <a href=\"https:\/\/github.com\/kubernetes\/kube-state-metrics\">kube-state-metrics<\/a>.<\/p>\n<pre class=\"lang:default decode:true \">increase(kube_pod_container_status_restarts_total{pod=~\u201cmy-pod-.*\u201c}[60s])<\/pre>\n<p>We have to detect an increase, therefore we need a range vector. The range should correspond to the configured step size to avoid duplicates or undersampling. Furthermore, each range must contain at least two data points \u2013 an increase cannot be calculated from a single data point.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Config-reload\"><\/span>Config reload<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>A misconfiguration can introduce problems. This query plots an annotation for every change in the configuration of Prometheus.<\/p>\n<pre class=\"lang:default decode:true \">changes(prometheus_config_last_reload_success_timestamp_seconds{}[10m])<\/pre>\n<p>The range should correspond to the configured step size to avoid duplicates or undersampling.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Creation-of-a-pod\"><\/span>Creation of a pod<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>This query shows an annotation for every pod that is created.<\/p>\n<pre class=\"lang:default decode:true\">kube_pod_created{pod=~\"my-pod-.*\"} * 1000<\/pre>\n<p>Make sure to enable the option \u201cSeries value as timestamp\u201c. Grafana requires the timestamp in milliseconds.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span>Conclusion<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Prometheus is not about exact measurements and as such is not primarily built for events. However, the time series database already carries a lot of data, which makes it an interesting source for annotations. Furthermore, no additional infrastructure or scripting is required in a typical Grafana + Prometheus setup in order to get started with annotations.<\/p>\n<p>PromQL is purposely designed to be lean and built for time series in mind. It hands a lot of responsibility to Grafana. Grafana has put a lot of flexibility in panels (check out <a href=\"https:\/\/grafana.com\/blog\/2020\/05\/18\/grafana-v7.0-released-new-plugin-architecture-visualizations-transformations-native-trace-support-and-more\/#transformations\">transformations introduced in Grafana 7.0<\/a>) but those are not available for annotation queries. As a result, querying annotations using PromQL is a bit hacky.<\/p>\n<p>There are a few interesting feature requests regarding annotations, for example better support to deal with many annotations. You can find all the discussions around annotations on <a href=\"https:\/\/github.com\/grafana\/grafana\/labels\/area%2Fannotations\">GitHub<\/a>. Grafana is developed in the open. Feel free to join the discussion and contribute.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Grafana is often used in conjunction with Prometheus to visualize time series and compose dashboards for monitoring purposes. A variety of data fits really well into time series and can be visualized as graphs \u2013 such as request count, request duration or response size. Other data is more event-driven and does not quite fit in [&hellip;]<\/p>\n","protected":false},"author":196,"featured_media":20171,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"ep_exclude_from_search":false,"footnotes":""},"tags":[66],"service":[432],"coauthors":[{"id":196,"display_name":"Bj\u00f6rn Fischer","user_nicename":"bfischer"}],"class_list":["post-20056","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","tag-devops","service-devops"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.8 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Grafana Annotations with Prometheus (a Deep Dive) - inovex GmbH<\/title>\n<meta name=\"description\" content=\"Grafana is often used in conjunction with Prometheus to visualize time series and compose dashboards for monitoring purposes. In this blog article, I will dive deep into the specifics of Grafana annotations for data that does not fit into time series graphs \u2013 and how to use them with Prometheus as a data source.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/\" \/>\n<meta property=\"og:locale\" content=\"de_DE\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Grafana Annotations with Prometheus (a Deep Dive) - inovex GmbH\" \/>\n<meta property=\"og:description\" content=\"Grafana is often used in conjunction with Prometheus to visualize time series and compose dashboards for monitoring purposes. In this blog article, I will dive deep into the specifics of Grafana annotations for data that does not fit into time series graphs \u2013 and how to use them with Prometheus as a data source.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/\" \/>\n<meta property=\"og:site_name\" content=\"inovex GmbH\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/inovexde\" \/>\n<meta property=\"article:published_time\" content=\"2020-11-05T13:19:16+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-12-02T07:34:44+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/11\/grafana-prometheus-annotations.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"1080\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Bj\u00f6rn Fischer\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/11\/grafana-prometheus-annotations-1024x576.png\" \/>\n<meta name=\"twitter:creator\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:site\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:label1\" content=\"Verfasst von\" \/>\n\t<meta name=\"twitter:data1\" content=\"Bj\u00f6rn Fischer\" \/>\n\t<meta name=\"twitter:label2\" content=\"Gesch\u00e4tzte Lesezeit\" \/>\n\t<meta name=\"twitter:data2\" content=\"10\u00a0Minuten\" \/>\n\t<meta name=\"twitter:label3\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data3\" content=\"Bj\u00f6rn Fischer\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/grafana-annotations-prometheus-deep-dive\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/grafana-annotations-prometheus-deep-dive\\\/\"},\"author\":{\"name\":\"Bj\u00f6rn Fischer\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/person\\\/787ee4bac1886f882a67cd821a5bbb88\"},\"headline\":\"Grafana Annotations with Prometheus (a Deep Dive)\",\"datePublished\":\"2020-11-05T13:19:16+00:00\",\"dateModified\":\"2022-12-02T07:34:44+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/grafana-annotations-prometheus-deep-dive\\\/\"},\"wordCount\":2037,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/grafana-annotations-prometheus-deep-dive\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2020\\\/11\\\/grafana-prometheus-annotations.png\",\"keywords\":[\"DevOps\"],\"articleSection\":[\"English Content\",\"General\",\"Infrastructure\"],\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/grafana-annotations-prometheus-deep-dive\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/grafana-annotations-prometheus-deep-dive\\\/\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/grafana-annotations-prometheus-deep-dive\\\/\",\"name\":\"Grafana Annotations with Prometheus (a Deep Dive) - inovex GmbH\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/grafana-annotations-prometheus-deep-dive\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/grafana-annotations-prometheus-deep-dive\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2020\\\/11\\\/grafana-prometheus-annotations.png\",\"datePublished\":\"2020-11-05T13:19:16+00:00\",\"dateModified\":\"2022-12-02T07:34:44+00:00\",\"description\":\"Grafana is often used in conjunction with Prometheus to visualize time series and compose dashboards for monitoring purposes. In this blog article, I will dive deep into the specifics of Grafana annotations for data that does not fit into time series graphs \u2013 and how to use them with Prometheus as a data source.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/grafana-annotations-prometheus-deep-dive\\\/#breadcrumb\"},\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/grafana-annotations-prometheus-deep-dive\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/grafana-annotations-prometheus-deep-dive\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2020\\\/11\\\/grafana-prometheus-annotations.png\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2020\\\/11\\\/grafana-prometheus-annotations.png\",\"width\":1920,\"height\":1080,\"caption\":\"Grafana annotations on a prometheus data graph\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/grafana-annotations-prometheus-deep-dive\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Grafana Annotations with Prometheus (a Deep Dive)\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#website\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\",\"name\":\"inovex GmbH\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"de\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\",\"name\":\"inovex GmbH\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/inovex-logo-16-9-1.png\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/inovex-logo-16-9-1.png\",\"width\":1921,\"height\":1081,\"caption\":\"inovex GmbH\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/inovexde\",\"https:\\\/\\\/x.com\\\/inovexgmbh\",\"https:\\\/\\\/www.instagram.com\\\/inovexlife\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/inovex\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UC7r66GT14hROB_RQsQBAQUQ\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/person\\\/787ee4bac1886f882a67cd821a5bbb88\",\"name\":\"Bj\u00f6rn Fischer\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e7500175ec2e4f01050a2dcd12dc3701344243cb9bb10eea220cc49aed21732f?s=96&d=retro&r=g0bbb20c32382ec889c04d197edd907cf\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e7500175ec2e4f01050a2dcd12dc3701344243cb9bb10eea220cc49aed21732f?s=96&d=retro&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e7500175ec2e4f01050a2dcd12dc3701344243cb9bb10eea220cc49aed21732f?s=96&d=retro&r=g\",\"caption\":\"Bj\u00f6rn Fischer\"},\"sameAs\":[\"http:\\\/\\\/github.com\\\/fischerman\"],\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/author\\\/bfischer\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Grafana Annotations with Prometheus (a Deep Dive) - inovex GmbH","description":"Grafana is often used in conjunction with Prometheus to visualize time series and compose dashboards for monitoring purposes. In this blog article, I will dive deep into the specifics of Grafana annotations for data that does not fit into time series graphs \u2013 and how to use them with Prometheus as a data source.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/","og_locale":"de_DE","og_type":"article","og_title":"Grafana Annotations with Prometheus (a Deep Dive) - inovex GmbH","og_description":"Grafana is often used in conjunction with Prometheus to visualize time series and compose dashboards for monitoring purposes. In this blog article, I will dive deep into the specifics of Grafana annotations for data that does not fit into time series graphs \u2013 and how to use them with Prometheus as a data source.","og_url":"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/","og_site_name":"inovex GmbH","article_publisher":"https:\/\/www.facebook.com\/inovexde","article_published_time":"2020-11-05T13:19:16+00:00","article_modified_time":"2022-12-02T07:34:44+00:00","og_image":[{"width":1920,"height":1080,"url":"https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/11\/grafana-prometheus-annotations.png","type":"image\/png"}],"author":"Bj\u00f6rn Fischer","twitter_card":"summary_large_image","twitter_image":"https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/11\/grafana-prometheus-annotations-1024x576.png","twitter_creator":"@inovexgmbh","twitter_site":"@inovexgmbh","twitter_misc":{"Verfasst von":"Bj\u00f6rn Fischer","Gesch\u00e4tzte Lesezeit":"10\u00a0Minuten","Written by":"Bj\u00f6rn Fischer"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/#article","isPartOf":{"@id":"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/"},"author":{"name":"Bj\u00f6rn Fischer","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/787ee4bac1886f882a67cd821a5bbb88"},"headline":"Grafana Annotations with Prometheus (a Deep Dive)","datePublished":"2020-11-05T13:19:16+00:00","dateModified":"2022-12-02T07:34:44+00:00","mainEntityOfPage":{"@id":"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/"},"wordCount":2037,"commentCount":0,"publisher":{"@id":"https:\/\/www.inovex.de\/de\/#organization"},"image":{"@id":"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/11\/grafana-prometheus-annotations.png","keywords":["DevOps"],"articleSection":["English Content","General","Infrastructure"],"inLanguage":"de","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/","url":"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/","name":"Grafana Annotations with Prometheus (a Deep Dive) - inovex GmbH","isPartOf":{"@id":"https:\/\/www.inovex.de\/de\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/#primaryimage"},"image":{"@id":"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/11\/grafana-prometheus-annotations.png","datePublished":"2020-11-05T13:19:16+00:00","dateModified":"2022-12-02T07:34:44+00:00","description":"Grafana is often used in conjunction with Prometheus to visualize time series and compose dashboards for monitoring purposes. In this blog article, I will dive deep into the specifics of Grafana annotations for data that does not fit into time series graphs \u2013 and how to use them with Prometheus as a data source.","breadcrumb":{"@id":"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/#breadcrumb"},"inLanguage":"de","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/"]}]},{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/#primaryimage","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/11\/grafana-prometheus-annotations.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2020\/11\/grafana-prometheus-annotations.png","width":1920,"height":1080,"caption":"Grafana annotations on a prometheus data graph"},{"@type":"BreadcrumbList","@id":"https:\/\/www.inovex.de\/de\/blog\/grafana-annotations-prometheus-deep-dive\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.inovex.de\/de\/"},{"@type":"ListItem","position":2,"name":"Grafana Annotations with Prometheus (a Deep Dive)"}]},{"@type":"WebSite","@id":"https:\/\/www.inovex.de\/de\/#website","url":"https:\/\/www.inovex.de\/de\/","name":"inovex GmbH","description":"","publisher":{"@id":"https:\/\/www.inovex.de\/de\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.inovex.de\/de\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"de"},{"@type":"Organization","@id":"https:\/\/www.inovex.de\/de\/#organization","name":"inovex GmbH","url":"https:\/\/www.inovex.de\/de\/","logo":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","width":1921,"height":1081,"caption":"inovex GmbH"},"image":{"@id":"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/inovexde","https:\/\/x.com\/inovexgmbh","https:\/\/www.instagram.com\/inovexlife\/","https:\/\/www.linkedin.com\/company\/inovex","https:\/\/www.youtube.com\/channel\/UC7r66GT14hROB_RQsQBAQUQ"]},{"@type":"Person","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/787ee4bac1886f882a67cd821a5bbb88","name":"Bj\u00f6rn Fischer","image":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/secure.gravatar.com\/avatar\/e7500175ec2e4f01050a2dcd12dc3701344243cb9bb10eea220cc49aed21732f?s=96&d=retro&r=g0bbb20c32382ec889c04d197edd907cf","url":"https:\/\/secure.gravatar.com\/avatar\/e7500175ec2e4f01050a2dcd12dc3701344243cb9bb10eea220cc49aed21732f?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e7500175ec2e4f01050a2dcd12dc3701344243cb9bb10eea220cc49aed21732f?s=96&d=retro&r=g","caption":"Bj\u00f6rn Fischer"},"sameAs":["http:\/\/github.com\/fischerman"],"url":"https:\/\/www.inovex.de\/de\/blog\/author\/bfischer\/"}]}},"_links":{"self":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/20056","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/users\/196"}],"replies":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/comments?post=20056"}],"version-history":[{"count":4,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/20056\/revisions"}],"predecessor-version":[{"id":29976,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/20056\/revisions\/29976"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/media\/20171"}],"wp:attachment":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/media?parent=20056"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/tags?post=20056"},{"taxonomy":"service","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/service?post=20056"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/coauthors?post=20056"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}