🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

📊 धडा 03 — Metrics: तपासणी तक्ता

📍 तुम्ही इथे आहात: 12 पैकी धडा 03 · मागे: lesson-02-logs · पुढे: lesson-04-percentiles

📦 या ब्रँचमध्ये काय आहे

धडे 01–02, आणि दुसरा signal: metrics. तुम्ही metrics चे तीन प्रकार शिकता — counter, gauge, histogram — आणि labels, Prometheus चा text format, आणि cardinality चा सापळा. Counter, Gauge, Histogram, exposition() आणि cardinality() obs/signals.py मध्ये आहेत; obs/demo.py मधील metrics() ते print करते.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

आरोग्य कक्षाच्या भिंतीवर कतरिनाचा तपासणी तक्ता 📊 लटकलेला आहे. तो दैनंदिनीसारख्या गोष्टी सांगत नाही. त्यात वेळेनुसार आकडे असतात. त्याचे तीन प्रकार आहेत:

एक मोजणी (tally) — "आज तपासलेले विद्यार्थी": कतरिना प्रत्येक वेळी एक खूण जोडते. तो फक्त वाढतो. मध्यरात्री तो पुन्हा शून्यापासून सुरू होतो. कक्ष किती व्यस्त आहे हे जाणण्यासाठी ती "गेल्या तासात किती खुणा?" असे विचारते — "एकूण किती खुणा?" असे नाही.
एक thermometer चे वाचन — "आत्ता वापरात असलेले बेड": 4, मग 2, मग 3. ते वर आणि खाली होते. फक्त आत्ताचे वाचन महत्त्वाचे.
एक वर्गीकरणाचे कपाट — प्रत्येक भेटीचा कालावधी एका खोक्यात जातो: "5 मिनिटे किंवा कमी", "10 किंवा कमी", "30 किंवा कमी". प्रत्येक खोका त्याच्या आकारापर्यंतचे सगळे मोजतो, म्हणून "10 किंवा कमी" खोका 5 मिनिटांच्या भेटीही मोजतो.

आणि एक इशारा. कतरिना मोजणी वर्गानुसार विभागू शकते: 3A, 3B, 4A. ते ठीक आहे — काही स्तंभ. पण जर ती ती विद्यार्थ्याच्या नावानुसार विभागली, तर तक्त्याला शाळेतल्या प्रत्येक विद्यार्थ्यासाठी एक स्तंभ लागेल. भिंत तेवढी मोठी नाही.

🗺️ आकृती

flowchart LR
    subgraph types["📊 three metric types"]
      c["➕ counter<br/>http_requests_total<br/>only up · resets to 0 on restart"]
      g["🌡️ gauge<br/>print_queue_length 137<br/>up and down"]
      h["🗄️ histogram<br/>_bucket{le=…} cumulative<br/>+ _sum + _count"]
    end
    ex["📄 /metrics text format"]
    c --> ex
    g --> ex
    h --> ex
    lab["🏷️ labels: route × status = 2 × 3 = 6 series<br/>+ user_id (50,000) = 300,000 series 💥"]

🗺️ काढलेली आवृत्ती + एक lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l03

❓ काय

Metric — वेळेनुसार मोजलेला आकडा. प्रत्येक किंमत (timestamp, value) म्हणून साठवली जाते. Time series म्हणजे एक metric name आणि labels च्या किमतींचा एक नेमका संच.
Counter — एक एकूण आकडा जो फक्त वाढतो (हाताळलेल्या requests, errors, पाठवलेले bytes). तो process restart झाल्यावर 0 वर reset होतो. तुम्ही जवळजवळ कधीच त्याची कच्ची किंमत graph करत नाही; तुम्ही त्याचा rate graph करता — एका window मधली दर सेकंदाची वाढ (धडा 07). नावे _total ने संपतात.
Gauge — वर आणि खाली होणारी किंमत: queue length, वापरातली memory, वापरातले बेड, तापमान. तुम्ही ती आत्ता जशी आहे तशी वाचता.
Histogram — निरीक्षणे (जसे request durations) buckets मध्ये मोजतो. Prometheus मध्ये buckets cumulative असतात: le म्हणजे "less than or equal", म्हणून _bucket{le="0.25"} प्रत्येक ≤ 0.25 s request मोजतो, ≤ 0.1 s च्या requests सह. नेहमी एक le="+Inf" bucket असतो (प्रत्येक निरीक्षण). Histogram _sum (सगळ्या किमतींची बेरीज) आणि _count (किती) सुद्धा देतो. _sum / _count म्हणजे सरासरी. धडा 04 buckets चे percentiles मध्ये रूपांतर करतो.
Summary — durations साठी Prometheus चा दुसरा प्रकार. तो quantiles (जसे p99) app च्या आतच (client-side) मोजतो आणि थेट देतो. ते quantiles aggregate करता येत नाहीत: दहा pods चे p99 एकत्र करून एक p99 बनवता येत नाही (का ते धडा 04 दाखवतो). तुम्ही एकापेक्षा जास्त copies चालवत असाल तर histograms वापरा.
Labels — metric ला विभागणाऱ्या key–value जोड्या: http_requests_total{route="/results", status="500"}.
Cardinality — एक metric किती time series बनवतो ती संख्या. ती प्रत्येक label च्या किमतींच्या संख्यांचा गुणाकार असते: 2 routes × 3 statuses = 6 series; 50,000 users सह user_id जोडला तर 300,000. प्रत्येक series ला memory, storage आणि query time लागतो. User ids, emails, पूर्ण URLs, trace ids किंवा timestamps कधीच label values म्हणून वापरू नका. त्या logs आणि traces मध्ये जातात.
Exposition format — /metrics endpoint देतो तो साधा मजकूर: # HELP, # TYPE, मग प्रत्येक series साठी एक ओळ.
दोन उपयोगी checklists: services साठी RED — Rate, Errors, Duration; आणि resources साठी USE — Utilisation, Saturation, Errors.

🤔 का

कारण metrics स्वस्त आणि जलद असतात. Traffic कितीही असो, counter म्हणजे प्रत्येक series साठी एक आकडा: 10 requests असोत किंवा 1 कोटी, साठवायला तेवढाच खर्च. त्यामुळे dashboards आणि alerts साठी metrics हाच योग्य signal आहे. त्याची किंमत म्हणजे तपशील: metric सांगतो 20 requests अयशस्वी झाल्या, पण कोणत्या ते नाही. आणि metrics महाग करण्याचा एकमेव मार्ग म्हणजे high cardinality.

🔧 कसे (या repo मध्ये)

Counter.inc(labels, by) प्रत्येक label संचाच्या एकूण आकड्यात भर घालतो; Gauge.set(v) किंमत बदलतो; Histogram(name, help, buckets).observe_all(values) cumulative bucket counts आणि sum व count भरतो. exposition(*metrics) ते Prometheus /metrics endpoint प्रमाणे print करतो. cardinality(**label_values) प्रत्येक label च्या किमतींच्या संख्यांचा गुणाकार करतो. metrics() status नुसार 9,500 requests मोजते, print-queue gauge 137 वर ठेवते आणि दिवसाच्या 20 नमुना latencies पाच buckets मध्ये टाकते.

🧪 करून पाहा

आरोग्य कक्षाचा स्वतःचा तक्ता — मग labels कसे गुणले जातात ते पाहा:

python3 obs/demo.py metrics
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from signals import Counter, Gauge, Histogram, exposition, cardinality
c = Counter("pupils_seen_total", "Pupils seen by the nurse")
c.inc((("reason", "fever"),), 3); c.inc((("reason", "scraped knee"),), 5); c.inc((("reason", "fever"),))
g = Gauge("beds_in_use", "Beds in use right now"); g.set(4); g.set(2)
h = Histogram("visit_minutes", "Minutes per visit", [5, 10, 30]).observe_all([3, 4, 8, 12, 25, 40])
print(exposition(c, g, h))
routes, status = ["/results", "/notices"], ["200", "404", "500"]
print("route × status                  →", cardinality(route=routes, status=status))
print("+ method (4)                    →", cardinality(route=routes, status=status, method=["GET", "POST", "PUT", "DELETE"]))
print("+ method (4) + pod (20)         →", f"{cardinality(route=routes, status=status, method=list('abcd'), pod=list(range(20))):,}")
print("+ pod (20) + user_id (50,000)   →", f"{cardinality(route=routes, status=status, pod=list(range(20)), user_id=list(range(50_000))):,}")
EOF

✅ तपासा — तुम्हाला काय दिसायला हवे

metrics हे print करते (काही भाग):

   http_requests_total{route="/results",status="200"} 9450
   http_requests_total{route="/results",status="404"} 30
   http_requests_total{route="/results",status="500"} 20
   # HELP print_queue_length Certificates waiting
   # TYPE print_queue_length gauge
   print_queue_length 137
   http_request_duration_seconds_bucket{le="0.1"} 10
   http_request_duration_seconds_bucket{le="0.25"} 18
   http_request_duration_seconds_bucket{le="0.5"} 19
   http_request_duration_seconds_bucket{le="1.0"} 19
   http_request_duration_seconds_bucket{le="2.5"} 20
   http_request_duration_seconds_bucket{le="+Inf"} 20
   http_request_duration_seconds_sum 3.657
   http_request_duration_seconds_count 20
── labels route, status → 6 time series
── labels route, status, user_id → 300,000 time series

तुमचा snippet हे print करतो:

# HELP pupils_seen_total Pupils seen by the nurse
# TYPE pupils_seen_total counter
pupils_seen_total{reason="fever"} 4
pupils_seen_total{reason="scraped knee"} 5
# HELP beds_in_use Beds in use right now
# TYPE beds_in_use gauge
beds_in_use 2
# HELP visit_minutes Minutes per visit
# TYPE visit_minutes histogram
visit_minutes_bucket{le="5"} 2
visit_minutes_bucket{le="10"} 3
visit_minutes_bucket{le="30"} 5
visit_minutes_bucket{le="+Inf"} 6
visit_minutes_sum 92.0
visit_minutes_count 6
route × status                  → 6
+ method (4)                    → 24
+ method (4) + pod (20)         → 480
+ pod (20) + user_id (50,000)   → 6,000,000

🏁 तुम्ही आत्ताच काय सिद्ध केले

Counter ने फक्त बेरीज केली (3 + 1 ताप = 4). Gauge ने फक्त त्याचे शेवटचे वाचन ठेवले (2, 4 नाही). Histogram चे buckets cumulative आहेत: le="10" bucket मध्ये 3 भेटी आहेत — 3 आणि 4 मिनिटांच्या भेटी तसेच 8 मिनिटांची भेट — आणि 40 मिनिटांची भेट फक्त +Inf मध्ये दिसते. सरासरी भेट _sum / _count = 92 / 6 ≈ 15.3 मिनिटे. आणि cardinality हा गुणाकार आहे: निरुपद्रवी labels (method, pod) नी 6 series 480 पर्यंत नेल्या; एका user_id label ने त्या 60 लाख (6 million) पर्यंत नेल्या.

⚠️ नेहमीच्या चुका

counter च्या rate() ऐवजी त्याची कच्ची किंमत graph करणे (फक्त चढत जाणारी रेषा)
मोजायला हवे अशा गोष्टीसाठी gauge — 15 सेकंदांचा scrape दोन scrapes मध्ये घडलेले चुकवतो
user ids, emails, पूर्ण URLs (/results/3A/pupil/1234) किंवा error messages label values म्हणून
अनेक pods चालवत असताना summary वापरणे आणि मग quantiles एकत्र करण्याचा प्रयत्न करणे
units नसलेली metric names — _seconds, _bytes, आणि counters साठी _total वापरा
दोन services मध्ये एकाच label ची वेगवेगळी नावे (status आणि code)

🏭 प्रत्यक्ष वापरात

On a real account — Python ची prometheus_client library, port 8000 वर /metrics देणारी:

from prometheus_client import Counter, Gauge, Histogram, start_http_server

REQUESTS = Counter("http_requests_total", "Requests served", ["route", "status"])
QUEUE = Gauge("print_queue_length", "Certificates waiting")
LATENCY = Histogram("http_request_duration_seconds", "Request time", ["route"],
                    buckets=[0.05, 0.1, 0.25, 0.5, 1.0, 2.5])

start_http_server(8000)                     # Prometheus scrapes http://host:8000/metrics

def handle(route):                          # route = the TEMPLATE "/results/{cls}", not the full URL
    with LATENCY.labels(route=route).time():
        status = serve(route)
    REQUESTS.labels(route=route, status=str(status)).inc()
    QUEUE.set(print_queue.size())

Prometheus मध्ये सर्वात जास्त series असलेले metrics शोधा (Prometheus UI मध्ये चालवा):

topk(10, count by (__name__) ({__name__=~".+"}))

CloudWatch — dimensions सह एक metric (प्रत्येक dimension combination हा स्वतंत्र metric असतो, आणि custom metrics चे bill प्रति metric येते):

aws cloudwatch put-metric-data --namespace School/ResultsApi \
    --metric-name Requests --dimensions Route=/results,Status=500 --value 1 --unit Count

Datadog मध्ये हीच कल्पना tags म्हणून: results.requests{route:/results,status:500}; प्रत्येक वेगळे tag combination bill वर एक custom metric असते.

🏭 Production मध्ये हे का महत्त्वाचे: label जोडण्यापूर्वी गुणाकार करा. किमतींची संख्या × इतर प्रत्येक label च्या किमती × pods ची संख्या. उत्तर काही हजार series पेक्षा जास्त आले, तर ती किंमत त्याऐवजी log ओळीत किंवा span attribute मध्ये जायला हवी.

⏭️ पुढे

Histogram मध्ये buckets आहेत. Buckets चे "99% पालकांनी X पेक्षा कमी वेळ वाट पाहिली" मध्ये रूपांतर कसे करायचे? आणि त्या आकड्याची सरासरी कधीच का काढता येत नाही?

git checkout lesson-04-percentiles

📊 Lesson 03 — Metrics: the vital-signs chart

📍 You are here: Lesson 03 of 12 · Previous: lesson-02-logs · Next: lesson-04-percentiles

📦 What's in this branch

Lessons 01–02, plus the second signal: metrics. You learn the three metric types — counter, gauge, histogram — plus labels, the Prometheus text format, and the cardinality trap. Counter, Gauge, Histogram, exposition() and cardinality() live in obs/signals.py; metrics() in obs/demo.py prints them.

🧒 Explain like I'm 5

On the wall of the health room hangs Katrina's vital-signs chart 📊. It does not tell stories like the diary. It holds numbers over time. There are three kinds:

A tally — "pupils seen today": Katrina adds a mark each time. It only goes up. At midnight it starts again from zero. To know how busy the room is, she asks "how many marks in the last hour?" — not "how many marks in total?".
A thermometer reading — "beds in use right now": 4, then 2, then 3. It goes up and down. Only the reading now matters.
A sorting shelf — every visit's length goes into a box: "5 minutes or less", "10 or less", "30 or less". Each box counts everything up to its size, so the "10 or less" box also counts the 5-minute visits.

And a warning. Katrina can split the tally by class: 3A, 3B, 4A. That is fine — a few columns. If she splits it by pupil name, the chart needs a column for every pupil in the school. The wall is not big enough.

🗺️ Diagram

flowchart LR
    subgraph types["📊 three metric types"]
      c["➕ counter<br/>http_requests_total<br/>only up · resets to 0 on restart"]
      g["🌡️ gauge<br/>print_queue_length 137<br/>up and down"]
      h["🗄️ histogram<br/>_bucket{le=…} cumulative<br/>+ _sum + _count"]
    end
    ex["📄 /metrics text format"]
    c --> ex
    g --> ex
    h --> ex
    lab["🏷️ labels: route × status = 2 × 3 = 6 series<br/>+ user_id (50,000) = 300,000 series 💥"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l03

❓ What

Metric — a number measured over time. Each value is stored as (timestamp, value). A time series is one metric name plus one exact set of label values.
Counter — a total that only increases (requests served, errors, bytes sent). It resets to 0 when the process restarts. You almost never graph the raw value; you graph its rate — the increase per second over a window (lesson 07). Names end in _total.
Gauge — a value that goes up and down: queue length, memory in use, beds in use, temperature. You read it as it is now.
Histogram — counts observations (such as request durations) into buckets. In Prometheus the buckets are cumulative: le means "less than or equal", so _bucket{le="0.25"} counts every request ≤ 0.25 s, including those ≤ 0.1 s. There is always a le="+Inf" bucket (every observation). A histogram also exposes _sum (the total of all values) and _count (how many). _sum / _count is the mean. Lesson 04 turns buckets into percentiles.
Summary — the other Prometheus type for durations. It computes quantiles (such as p99) inside the app (client-side) and exposes them directly. Those quantiles cannot be aggregated: you cannot combine the p99 of ten pods into one p99 (lesson 04 shows why). Prefer histograms when you run more than one copy.
Labels — key–value pairs that split a metric: http_requests_total{route="/results", status="500"}.
Cardinality — the number of time series a metric creates. It is the product of the number of values of each label: 2 routes × 3 statuses = 6 series; add user_id with 50,000 users and it is 300,000. Each series costs memory, storage and query time. Never use user ids, emails, full URLs, trace ids or timestamps as label values. Those belong in logs and traces.
Exposition format — the plain text a /metrics endpoint serves: # HELP, # TYPE, then one line per series.
Two useful checklists: RED for services — Rate, Errors, Duration; and USE for resources — Utilisation, Saturation, Errors.

🤔 Why

Because metrics are cheap and fast. A counter is one number per series, whatever the traffic: 10 requests or 10 million cost the same to store. That makes metrics the right signal for dashboards and alerts. The price is detail: a metric says 20 requests failed, not which ones. And the one way to make metrics expensive is high cardinality.

🔧 How (in this repo)

Counter.inc(labels, by) adds to a total per label set; Gauge.set(v) replaces the value; Histogram(name, help, buckets).observe_all(values) fills cumulative bucket counts plus sum and count. exposition(*metrics) prints them the way a Prometheus /metrics endpoint does. cardinality(**label_values) multiplies the number of values of each label. metrics() counts 9,500 requests by status, sets a print-queue gauge to 137 and puts the day's 20 sample latencies into five buckets.

🧪 Try it

The health room's own chart — then watch the labels multiply:

python3 obs/demo.py metrics
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from signals import Counter, Gauge, Histogram, exposition, cardinality
c = Counter("pupils_seen_total", "Pupils seen by the nurse")
c.inc((("reason", "fever"),), 3); c.inc((("reason", "scraped knee"),), 5); c.inc((("reason", "fever"),))
g = Gauge("beds_in_use", "Beds in use right now"); g.set(4); g.set(2)
h = Histogram("visit_minutes", "Minutes per visit", [5, 10, 30]).observe_all([3, 4, 8, 12, 25, 40])
print(exposition(c, g, h))
routes, status = ["/results", "/notices"], ["200", "404", "500"]
print("route × status                  →", cardinality(route=routes, status=status))
print("+ method (4)                    →", cardinality(route=routes, status=status, method=["GET", "POST", "PUT", "DELETE"]))
print("+ method (4) + pod (20)         →", f"{cardinality(route=routes, status=status, method=list('abcd'), pod=list(range(20))):,}")
print("+ pod (20) + user_id (50,000)   →", f"{cardinality(route=routes, status=status, pod=list(range(20)), user_id=list(range(50_000))):,}")
EOF

✅ Verify — what you should see

metrics prints (in part):

   http_requests_total{route="/results",status="200"} 9450
   http_requests_total{route="/results",status="404"} 30
   http_requests_total{route="/results",status="500"} 20
   # HELP print_queue_length Certificates waiting
   # TYPE print_queue_length gauge
   print_queue_length 137
   http_request_duration_seconds_bucket{le="0.1"} 10
   http_request_duration_seconds_bucket{le="0.25"} 18
   http_request_duration_seconds_bucket{le="0.5"} 19
   http_request_duration_seconds_bucket{le="1.0"} 19
   http_request_duration_seconds_bucket{le="2.5"} 20
   http_request_duration_seconds_bucket{le="+Inf"} 20
   http_request_duration_seconds_sum 3.657
   http_request_duration_seconds_count 20
── labels route, status → 6 time series
── labels route, status, user_id → 300,000 time series

Your snippet prints:

# HELP pupils_seen_total Pupils seen by the nurse
# TYPE pupils_seen_total counter
pupils_seen_total{reason="fever"} 4
pupils_seen_total{reason="scraped knee"} 5
# HELP beds_in_use Beds in use right now
# TYPE beds_in_use gauge
beds_in_use 2
# HELP visit_minutes Minutes per visit
# TYPE visit_minutes histogram
visit_minutes_bucket{le="5"} 2
visit_minutes_bucket{le="10"} 3
visit_minutes_bucket{le="30"} 5
visit_minutes_bucket{le="+Inf"} 6
visit_minutes_sum 92.0
visit_minutes_count 6
route × status                  → 6
+ method (4)                    → 24
+ method (4) + pod (20)         → 480
+ pod (20) + user_id (50,000)   → 6,000,000

🏁 What you just proved

The counter only added (3 + 1 fevers = 4). The gauge kept only its last reading (2, not 4). The histogram's buckets are cumulative: the le="10" bucket holds 3 visits — the 3- and 4-minute ones as well as the 8-minute one — and the 40-minute visit appears only in +Inf. The mean visit is _sum / _count = 92 / 6 ≈ 15.3 minutes. And cardinality is a product: harmless labels (method, pod) took 6 series to 480; one user_id label took them to 6 million.

⚠️ Common mistakes

graphing a counter's raw value (a line that only climbs) instead of its rate()
a gauge for something that should be counted — a 15-second scrape misses what happens between scrapes
user ids, emails, full URLs (/results/3A/pupil/1234) or error messages as label values
using a summary when you run many pods and then trying to combine the quantiles
metric names without units — use _seconds, _bytes, and _total for counters
the same label with different names in two services (status and code)

🏭 In production

On a real account — the Python prometheus_client library, serving /metrics on port 8000:

from prometheus_client import Counter, Gauge, Histogram, start_http_server

REQUESTS = Counter("http_requests_total", "Requests served", ["route", "status"])
QUEUE = Gauge("print_queue_length", "Certificates waiting")
LATENCY = Histogram("http_request_duration_seconds", "Request time", ["route"],
                    buckets=[0.05, 0.1, 0.25, 0.5, 1.0, 2.5])

start_http_server(8000)                     # Prometheus scrapes http://host:8000/metrics

def handle(route):                          # route = the TEMPLATE "/results/{cls}", not the full URL
    with LATENCY.labels(route=route).time():
        status = serve(route)
    REQUESTS.labels(route=route, status=str(status)).inc()
    QUEUE.set(print_queue.size())

Find the metrics with the most series in Prometheus (run it in the Prometheus UI):

topk(10, count by (__name__) ({__name__=~".+"}))

CloudWatch — one metric with dimensions (each dimension combination is its own metric, and custom metrics are billed per metric):

aws cloudwatch put-metric-data --namespace School/ResultsApi \
    --metric-name Requests --dimensions Route=/results,Status=500 --value 1 --unit Count

In Datadog, the same idea is tags: results.requests{route:/results,status:500}; each unique tag combination is a custom metric on the bill.

🏭 Why this matters in production: before you add a label, multiply. Number of values × every other label's values × number of pods. If the answer has more than a few thousand series, the value belongs in a log line or a span attribute instead.

⏭️ Next

The histogram has buckets. How do you turn buckets into "99% of parents waited less than X"? And why can you never average that number?

git checkout lesson-04-percentiles

← Previouslogs Next →percentiles

This page is the lesson's README from the lesson-03-metrics branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.