🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

📈 धडा 07 — Prometheus आणि Grafana: scrape करा, साठवा, विचारा, काढा

📍 तुम्ही इथे आहात: 12 पैकी धडा 07 · मागे: lesson-06-opentelemetry · पुढे: lesson-08-alerting

📦 या ब्रँचमध्ये काय आहे

धडे 01–06, आणि सर्वात प्रचलित open-source metrics stack: Prometheus (scrape करणे आणि साठवणे, PromQL ने विचारणे) आणि Grafana (चित्र काढणे). तुम्ही pull model, scrape interval, rate() हे तुम्ही type करता ते पहिले function का असते, recording rules, आणि Alertmanager ची पहिली ओळख शिकता. obs/signals.py मधील rate() counter reset Prometheus प्रमाणेच हाताळते; obs/demo.py मधील prometheus() ते आणि रोजच्या तीन queries दाखवते.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

कतरिना प्रत्येक वर्गात बसू शकत नाही. म्हणून दर 15 सेकंदांनी तिची मदतनीस clipboard 📋 घेऊन व्हरांड्यातून फिरते आणि प्रत्येक वर्गाच्या दारावरचा मोजणी कागद वाचते. ती आकडा आणि वेळ लिहून घेते. हे म्हणजे pulling: मदतनीस खोल्यांकडे जाते; खोल्यांना काहीही पाठवावे लागत नाही. एखाद्या दारावर कागदच नसेल, किंवा खोलीला कुलूप असेल, तर मदतनीस तेही नोंदवते — हरवलेली खोली ही सुद्धा बातमी असते.

एका सकाळी 3A चा मोजणी कागद 2,200 वरून 150 वर जातो. 2,050 विद्यार्थी निघून गेले का? नाही — कागद खाली पडला आणि नवा कागद शून्यापासून सुरू झाला. हुशार मदतनीसला माहीत असते की मोजणी कधीच कमी होत नाही, म्हणून ती त्या 150 खुणा नव्या म्हणून मोजते.

मग दीपिका एका छोट्या, नेमक्या भाषेत प्रश्न विचारते: "गेल्या 5 मिनिटांत, प्रत्येक वर्गासाठी, दर सेकंदाला किती खुणा?" आणि उत्तरे शिक्षक कक्षातल्या एका मोठ्या फलकावर रेषा म्हणून काढली जातात — तो फलक म्हणजे Grafana.

🗺️ आकृती

flowchart LR
    subgraph targets["🎯 targets (each serves /metrics)"]
      a1["results-api pod 1"]
      a2["results-api pod 2"]
    end
    p["🔥 Prometheus<br/>scrape every 15 s (pull)<br/>TSDB · rules"]
    p -->|"GET /metrics"| a1
    p -->|"GET /metrics"| a2
    g["📊 Grafana<br/>dashboards"] -->|"PromQL"| p
    p -->|"firing alerts"| am["🚨 Alertmanager<br/>group · route · silence"]
    r["counter: 1000, 1600, 2200, 150 (restart), 750<br/>rate() = 8.1/s — not negative"]

🗺️ काढलेली आवृत्ती + एक lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l07

❓ काय

Prometheus — एक open-source metrics system (एक CNCF project): ते targets scrape करते, samples स्वतःच्या time-series database (TSDB) मध्ये साठवते, rules evaluate करते, आणि PromQL queries ची उत्तरे देते.
Pull model — Prometheus ठरलेल्या वेळापत्रकानुसार प्रत्येक target कडून GET /metrics आणते. Targets static lists किंवा service discovery (Kubernetes, EC2, Consul…) मधून येतात. प्रत्येक target साठी Prometheus up सुद्धा नोंदवते (1 = scrape चालला, 0 = अयशस्वी), म्हणून बंद पडलेला target हाच एक signal असतो. दोन scrapes दरम्यान संपणारे छोटे batch jobs त्याऐवजी Pushgateway कडे push करतात.
Scrape interval — प्रत्येक target किती वेळा वाचला जातो; सहसा 15–60 s (built-in default 1 मिनिट आहे). दोन scrapes दरम्यान घडून संपणारी कोणतीही गोष्ट gauge ला दिसत नाही; counter मात्र ती मोजतोच.
Instant vector विरुद्ध range vector — http_requests_total म्हणजे प्रत्येक series ची सर्वात नवी किंमत; http_requests_total[5m] म्हणजे गेल्या 5 मिनिटांचा प्रत्येक sample.
rate(counter[5m]) — range मधली दर सेकंदाची सरासरी वाढ. ते counter resets हाताळते: किंमत कमी झाली, तर Prometheus मानतो की process restart झाली आणि नवी किंमत शून्यापासूनची वाढ म्हणून मोजतो. ते range च्या कडांपर्यंत extrapolate सुद्धा करते. Scrape interval च्या किमान 4× range वापरा. increase() हेच range मधल्या एकूण वाढीच्या रूपात देते; irate() फक्त शेवटचे दोन samples वापरते (उड्या मारणारे — zoom केलेल्या graphs साठी, alerts साठी नाही).
आधी rate, मग sum — sum(rate(x[5m])), कधीच rate(sum(x)[5m]) नाही: आधी बेरीज केली तर एका pod चा reset इतरांच्या एकूण आकड्यात लपून जातो.
दर आठवड्याला तुम्ही type कराल ते PromQL:
- sum by (status) (rate(http_requests_total[5m])) — status नुसार दर सेकंदाच्या requests;
- sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) — error ratio;
- histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) — p99, pods मधून एकत्र करून (धडा 04).
Recording rules — Prometheus ठरलेल्या वेळापत्रकानुसार चालवून नव्या series म्हणून साठवतो अशा queries, जसे job:http_errors:ratio_rate5m. मग dashboards आणि alerts महागडी query पुन्हा मोजण्याऐवजी एक स्वस्त series वाचतात. नावाची पद्धत: level:metric:operations.
Alerting rules आणि Alertmanager — Prometheus alert rules evaluate करतो; वाजणारे alerts Alertmanager कडे जातात, जो त्यांना group करतो, route करतो (page, chat, ticket), मोठा alert वाजत असताना लहान alerts inhibit करतो, आणि ठरवलेल्या कामादरम्यान silences ठेवतो (धडा 08).
Grafana — dashboard tool. ते data sources (Prometheus, Loki, Tempo, CloudWatch, Datadog…) वाचते आणि panels काढते. Variables (services चा drop-down) आणि annotations (प्रत्येक deploy वेळी प्रत्येक graph वर एक रेषा — धडा 11) यांमुळे एकच dashboard अनेक services साठी चालतो.
Long-term storage — एक Prometheus local disk वर काही आठवडे ठेवतो. जास्त काळासाठी, Thanos, Grafana Mimir, किंवा Amazon Managed Service for Prometheus सारख्या managed services कडे remote write वापरा.

🤔 का

कारण counter ची कच्ची किंमत graph वर निरुपयोगी असते — ती फक्त चढत जाते — आणि साधे "शेवटचे वजा पहिले" प्रत्येक restart ला मोडते (deploy प्रत्येक pod restart करतो). rate() counter चे तुम्हाला हव्या असलेल्या आकड्यात रूपांतर करते, "दर सेकंदाला, आत्ता", आणि restarts मधून टिकते. Pull model मुळे कोणते targets असायला हवेत हे monitoring system ला माहीत असते, म्हणून शांततेमागे बंद पडलेली service लपू शकत नाही.

🔧 कसे (या repo मध्ये)

obs/signals.py मधील rate(samples) [(time, value), …] घेते, शेजारच्या samples मधली वाढ बेरीज करते — आणि एखादी किंमत कमी झाली, तर ऋण आकड्याऐवजी नवी किंमत (restart नंतरची मोजणी) जोडते — मग ती झाकलेल्या वेळेने भागते. (खरा Prometheus range च्या कडांपर्यंत extrapolate सुद्धा करतो, म्हणून त्याचे आकडे थोडे जास्त असू शकतात.) prometheus() त्याला दर 60 s ला scrape केलेला आणि 180 s ला restart झालेला counter देते.

🧪 करून पाहा

शून्य, एक आणि दोन restarts मधून rate() ची साध्या "(last − first) / time" शी तुलना करा:

python3 obs/demo.py prometheus
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from signals import rate
calm = [(0, 1000), (60, 1600), (120, 2200), (180, 2800), (240, 3400)]
restart = [(0, 1000), (60, 1600), (120, 2200), (180, 150), (240, 750)]
two = [(0, 1000), (60, 1600), (120, 100), (180, 700), (240, 50)]
for name, s in (("no restart", calm), ("one restart", restart), ("two restarts", two)):
    naive = (s[-1][1] - s[0][1]) / (s[-1][0] - s[0][0])
    print(f"{name:<13} rate() {rate(s):5.1f}/s · naive (last - first) / time {naive:6.1f}/s")
EOF
python3 obs/test_obs.py

✅ तपासा — तुम्हाला काय दिसायला हवे

prometheus हे print करते:

── a counter scraped every 60 s: [(0, 1000), (60, 1600), (120, 2200), (180, 150), (240, 750)] — the app restarted at 180 s
   rate over 4 min = 8.1 requests/s (the reset is handled, not a negative spike)
── PromQL you will type every week:
   sum by (status) (rate(http_requests_total[5m]))
   sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
   histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
   Prometheus PULLS /metrics every 15–60 s and stores series · Grafana draws them · recording rules pre-compute

तुमचा snippet हे print करतो:

no restart    rate()  10.0/s · naive (last - first) / time   10.0/s
one restart   rate()   8.1/s · naive (last - first) / time   -1.0/s
two restarts  rate()   5.6/s · naive (last - first) / time   -4.0/s

Tests मध्ये ✅ L07 rate() survives a counter reset असते.

🏁 तुम्ही आत्ताच काय सिद्ध केले

Restart नसताना दोन्ही पद्धती जुळतात (दर सेकंदाला 10 requests). एका restart सह साधी पद्धत दर सेकंदाला −1 सांगते — counter साठी अशक्य — तर rate() 8.1 सांगते: 240 s मध्ये 600 + 600 + 150 + 600 = 1,950 requests. Reset नंतरचे 150 नव्या requests म्हणून मोजले गेले. दोन restarts: साधी पद्धत −4.0, rate() 5.6. प्रत्येक deploy म्हणजे restart, म्हणून हे दर आठवड्याला घडते.

⚠️ नेहमीच्या चुका

rate() शिवाय http_requests_total graph करणे
15 s scrape सह rate(x[30s]) — खूप कमी samples; interval च्या किमान 4× वापरा
sum(rate(...)) ऐवजी rate(sum(...))
summed buckets वरच्या histogram_quantile ऐवजी panel मध्ये p99s ची सरासरी काढणे
60 panels असलेला आणि एकही प्रश्न नसलेला dashboard — प्रत्येक service साठी RED (rate, errors, duration) ने सुरुवात करा
पुढच्या scrape आधीच संपणारे अल्पायुषी jobs scrape करणे (Pushgateway किंवा OTLP push वापरा)
महत्त्वाच्या targets साठी, किंवा Prometheus साठीच, up == 0 वर alert नाही

🏭 प्रत्यक्ष वापरात

On a real account — static target, Kubernetes pod discovery, rule files आणि Alertmanager सह prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files: ["rules/*.yml"]

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: results-api
    static_configs:
      - targets: ["results-api:8000"]
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"

Recording rules (rules/results-api.yml) — error ratio आणि p99 एकदाच आधी मोजून ठेवा:

groups:
  - name: results-api-recording
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_request_duration_seconds:p99_5m
        expr: histogram_quantile(0.99, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))

Files load करण्यापूर्वी तपासा, आणि थेट HTTP API ला विचारा:

promtool check config prometheus.yml
promtool check rules rules/results-api.yml
curl -s 'http://prometheus:9090/api/v1/query' --data-urlencode 'query=job:http_errors:ratio_rate5m'

Grafana — provisioning file ने Prometheus data source म्हणून जोडा (provisioning/datasources/prometheus.yml), मग $job variable (label_values(up, job)) आणि तीन recorded series वरच्या panels सह एक RED dashboard बनवा:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

🏭 Production मध्ये हे का महत्त्वाचे: कोणीही custom dashboard design करण्यापूर्वी, प्रत्येक service ला तोच पहिला dashboard द्या — दर सेकंदाच्या requests, error ratio, p99, recording rules मधून. Incident दरम्यान, ओळखीचा dashboard हाच जलद असतो.

⏭️ पुढे

Graphs काढले आहेत. पहाटे 3 वाजता कोणीही graphs पाहत नाही. Alarm ने कधी कोणाला उठवावे — आणि कधी शांत राहिलेच पाहिजे?

git checkout lesson-08-alerting

📈 Lesson 07 — Prometheus & Grafana: scrape, store, ask, draw

📍 You are here: Lesson 07 of 12 · Previous: lesson-06-opentelemetry · Next: lesson-08-alerting

📦 What's in this branch

Lessons 01–06, plus the most common open-source metrics stack: Prometheus (scrape and store, ask with PromQL) and Grafana (draw). You learn the pull model, the scrape interval, why rate() is the first function you type, recording rules, and a first look at Alertmanager. rate() in obs/signals.py handles a counter reset the way Prometheus does; prometheus() in obs/demo.py shows it and three everyday queries.

🧒 Explain like I'm 5

Katrina cannot sit in every classroom. So every 15 seconds her helper walks the corridor with a clipboard 📋 and reads the tally sheet on each classroom door. She writes down the number and the time. That is pulling: the helper goes to the rooms; the rooms do not have to post anything. If a door has no sheet, or the room is locked, the helper notes that too — a missing room is news.

One morning the 3A tally sheet goes from 2,200 to 150. Did 2,050 pupils leave? No — the sheet fell down and a new one was started from zero. A smart helper knows a tally never goes down, so she counts the 150 as new marks.

Then Dipika asks questions in a short, exact language: "marks per second, per class, over the last 5 minutes?" And the answers are drawn as lines on a big board in the staff room — that board is Grafana.

🗺️ Diagram

flowchart LR
    subgraph targets["🎯 targets (each serves /metrics)"]
      a1["results-api pod 1"]
      a2["results-api pod 2"]
    end
    p["🔥 Prometheus<br/>scrape every 15 s (pull)<br/>TSDB · rules"]
    p -->|"GET /metrics"| a1
    p -->|"GET /metrics"| a2
    g["📊 Grafana<br/>dashboards"] -->|"PromQL"| p
    p -->|"firing alerts"| am["🚨 Alertmanager<br/>group · route · silence"]
    r["counter: 1000, 1600, 2200, 150 (restart), 750<br/>rate() = 8.1/s — not negative"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l07

❓ What

Prometheus — an open-source metrics system (a CNCF project): it scrapes targets, stores samples in its own time-series database (TSDB), evaluates rules, and answers PromQL queries.
Pull model — Prometheus fetches GET /metrics from each target on a schedule. Targets come from static lists or service discovery (Kubernetes, EC2, Consul…). For every target Prometheus also records up (1 = the scrape worked, 0 = it failed), so a dead target is itself a signal. Short batch jobs that finish between scrapes push to a Pushgateway instead.
Scrape interval — how often each target is read; commonly 15–60 s (the built-in default is 1 minute). Anything that happens and ends between two scrapes is invisible to a gauge; a counter still counts it.
Instant vector vs range vector — http_requests_total is the latest value of each series; http_requests_total[5m] is every sample of the last 5 minutes.
rate(counter[5m]) — the per-second average increase over the range. It handles counter resets: if a value drops, Prometheus assumes the process restarted and counts the new value as increase from zero. It also extrapolates to the edges of the range. Use a range of at least 4× the scrape interval. increase() is the same thing as a total over the range; irate() uses only the last two samples (spiky — for zoomed-in graphs, not alerts).
Rate first, then sum — sum(rate(x[5m])), never rate(sum(x)[5m]): summing first hides one pod's reset inside the others' totals.
PromQL you will type every week:
- sum by (status) (rate(http_requests_total[5m])) — requests per second by status;
- sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) — the error ratio;
- histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) — the p99, merged across pods (lesson 04).
Recording rules — queries Prometheus runs on a schedule and stores as new series, such as job:http_errors:ratio_rate5m. Dashboards and alerts then read one cheap series instead of recomputing an expensive query. Naming pattern: level:metric:operations.
Alerting rules and Alertmanager — Prometheus evaluates alert rules; firing alerts go to Alertmanager, which groups them, routes them (page, chat, ticket), inhibits lower alerts while a bigger one fires, and holds silences during planned work (lesson 08).
Grafana — the dashboard tool. It reads data sources (Prometheus, Loki, Tempo, CloudWatch, Datadog…) and draws panels. Variables (a drop-down of services) and annotations (a line on every graph at each deploy — lesson 11) make one dashboard serve many services.
Long-term storage — one Prometheus keeps a few weeks on local disk. For more, use remote write to Thanos, Grafana Mimir, or managed services such as Amazon Managed Service for Prometheus.

🤔 Why

Because the counter's raw value is useless on a graph — it only climbs — and a naive "last minus first" breaks at every restart (a deploy restarts every pod). rate() turns a counter into the number you want, "per second, now", and survives restarts. The pull model makes the monitoring system the one that knows which targets should exist, so silence cannot hide a dead service.

🔧 How (in this repo)

rate(samples) in obs/signals.py takes [(time, value), …], adds up the increase between neighbouring samples — and when a value drops, adds the new value (the count since the restart) instead of a negative number — then divides by the time covered. (Real Prometheus also extrapolates to the edges of the range, so its numbers can be slightly higher.) prometheus() feeds it a counter scraped every 60 s that restarted at 180 s.

🧪 Try it

Compare rate() with a naive "(last − first) / time" through zero, one and two restarts:

python3 obs/demo.py prometheus
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from signals import rate
calm = [(0, 1000), (60, 1600), (120, 2200), (180, 2800), (240, 3400)]
restart = [(0, 1000), (60, 1600), (120, 2200), (180, 150), (240, 750)]
two = [(0, 1000), (60, 1600), (120, 100), (180, 700), (240, 50)]
for name, s in (("no restart", calm), ("one restart", restart), ("two restarts", two)):
    naive = (s[-1][1] - s[0][1]) / (s[-1][0] - s[0][0])
    print(f"{name:<13} rate() {rate(s):5.1f}/s · naive (last - first) / time {naive:6.1f}/s")
EOF
python3 obs/test_obs.py

✅ Verify — what you should see

prometheus prints:

── a counter scraped every 60 s: [(0, 1000), (60, 1600), (120, 2200), (180, 150), (240, 750)] — the app restarted at 180 s
   rate over 4 min = 8.1 requests/s (the reset is handled, not a negative spike)
── PromQL you will type every week:
   sum by (status) (rate(http_requests_total[5m]))
   sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
   histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
   Prometheus PULLS /metrics every 15–60 s and stores series · Grafana draws them · recording rules pre-compute

Your snippet prints:

no restart    rate()  10.0/s · naive (last - first) / time   10.0/s
one restart   rate()   8.1/s · naive (last - first) / time   -1.0/s
two restarts  rate()   5.6/s · naive (last - first) / time   -4.0/s

The tests include ✅ L07 rate() survives a counter reset.

🏁 What you just proved

With no restart both methods agree (10 requests a second). With one restart the naive method says −1 per second — impossible for a counter — while rate() says 8.1: 600 + 600 + 150 + 600 = 1,950 requests in 240 s. The 150 after the reset was counted as new requests. Two restarts: naive −4.0, rate() 5.6. Every deploy is a restart, so this happens every week.

⚠️ Common mistakes

graphing http_requests_total without rate()
rate(x[30s]) with a 15 s scrape — too few samples; use at least 4× the interval
rate(sum(...)) instead of sum(rate(...))
averaging p99s in a panel instead of histogram_quantile over summed buckets
a dashboard with 60 panels and no question — start with RED (rate, errors, duration) per service
scraping short-lived jobs that finish before the next scrape (use a Pushgateway or OTLP push)
no alert on up == 0 for important targets, or on Prometheus itself

🏭 In production

On a real account — prometheus.yml with a static target, Kubernetes pod discovery, rule files and Alertmanager:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files: ["rules/*.yml"]

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: results-api
    static_configs:
      - targets: ["results-api:8000"]
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"

Recording rules (rules/results-api.yml) — pre-compute the error ratio and p99 once:

groups:
  - name: results-api-recording
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_request_duration_seconds:p99_5m
        expr: histogram_quantile(0.99, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))

Check the files before you load them, and ask the HTTP API directly:

promtool check config prometheus.yml
promtool check rules rules/results-api.yml
curl -s 'http://prometheus:9090/api/v1/query' --data-urlencode 'query=job:http_errors:ratio_rate5m'

Grafana — add Prometheus as a data source by provisioning file (provisioning/datasources/prometheus.yml), then build a RED dashboard with a $job variable (label_values(up, job)) and panels on the three recorded series:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

🏭 Why this matters in production: give every service the same first dashboard — requests per second, error ratio, p99, from recording rules — before anyone designs a custom one. During an incident, the familiar dashboard is the fast one.

⏭️ Next

The graphs are drawn. Nobody watches graphs at 3 a.m. When should the alarm wake someone — and when must it stay quiet?

git checkout lesson-08-alerting

← Previousopentelemetry Next →alerting

This page is the lesson's README from the lesson-07-prometheus-grafana branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.