🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🩺 धडा 01 — Observability का: शाळेचा आरोग्य कक्ष

📍 तुम्ही इथे आहात: 12 पैकी धडा 01 · पुढे: lesson-02-logs

📦 या ब्रँचमध्ये काय आहे

हा संपूर्ण कोर्स एकाच प्रश्नाचे उत्तर देतो: "काहीतरी चुकले आहे — काय, कुठे, आणि का?" आज निकालाचा दिवस आहे. 11:40 ला पालक सांगतात की results page तुटले आहे. System आधीच जे सांगते त्यावरून team का ते सांगू शकेल का — नवा code न जोडता आणि ते पुन्हा बिघडण्याची वाट न पाहता? संपूर्ण कोर्समध्ये तुम्ही वापरणार असलेल्या खऱ्या files:

obs/signals.py — आरोग्य कक्षाची उपकरणे: logs, metrics, traces आणि OpenTelemetry सारखा Collector, शुद्ध Python मधील छोटी प्रामाणिक models म्हणून (129 ओळी, एकही dependency नाही)
obs/reliability.py — आरोग्य कक्षाची नियमपुस्तिका: alerts, burn rates, error budgets, incident ची घड्याळे, संशयित बदल, पाच का आणि postmortem (69 ओळी)
obs/demo.py — निकालाचा एक दिवस मिनिटा-मिनिटाने पुन्हा चालवलेला, प्रत्येक धड्यासाठी एक section (python3 obs/demo.py <section>; sections: why logs metrics percentiles tracing otel prometheus alerting slo incident rootcause postmortem)
obs/test_obs.py — आरोग्य कक्षाच्या नियमांवरच्या 12 तपासण्या
demo-output.txt — निरोगी run काय print करतो

🎒 सुरू करण्यापूर्वी: तुम्हाला फक्त Python 3 लागते, बाकी काही नाही — cloud account नाही, pip install नाही. या lab मध्ये निकालाच्या 8 तासांच्या एका दिवसाची शिकवण्यासाठीची models आहेत (मिनिटाला 1,000 requests, एक slow leak, एक वाईट deploy). खरी tools म्हणजे Prometheus, Grafana, OpenTelemetry, CloudWatch आणि Datadog — कल्पना आणि गणित तेच आहे. On a real account असे चिन्हांकित commands आणि files साठी ती tools install असावी लागतात, आणि त्यातल्या काहींना पैसे लागतात.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

प्रत्येक शाळेत एक आरोग्य कक्ष 🩺 असतो. शाळेची परिचारिका कतरिना तिथे काम करते.

एखाद्या विद्यार्थ्याला बरे वाटत नसेल, तेव्हा कतरिना अंदाज लावत नाही. तिच्याकडे तीन साधने आहेत:

एक दैनंदिनी 📓 — प्रत्येक भेटीसाठी एक ओळ: वेळ, विद्यार्थी, काय घडले;
एक तपासणी तक्ता 📊 — वेळेनुसार आकडे: दर तासाला ताप, आज किती विद्यार्थी आले;
एक मार्ग कार्ड 🗺️ — विद्यार्थी एका खोलीतून दुसऱ्या खोलीत नेतो ते कार्ड; प्रत्येक खोली आत येण्याची आणि बाहेर जाण्याची वेळ त्यावर शिक्का मारून नोंदवते.

आता जुनी पद्धत. मुख्याध्यापिका दीपिकाने एकदा boiler वर एक घंटा लावली. "Boiler गरम झाला की घंटा वाजव." घंटा सकाळभर वाजत राहिली. कोणीच आजारी नव्हते. आणि ज्या दिवशी एक विद्यार्थी खरोखर आजारी होता, त्या दिवशी घंटा शांत होती — boiler ठीक होता.

हाच फरक आहे. Monitoring म्हणजे boiler वरची घंटा: तुम्ही आधीच अंदाज केलेली एकच गोष्ट ती पाहते. Observability म्हणजे आरोग्य कक्ष: दैनंदिनी, तक्ता आणि मार्ग कार्डे मिळून कतरिनाला नवे प्रश्न सोडवू देतात — "आज 3A वर्ग का आजारी आहे?" — ज्यांचा कोणीच आधी विचार केला नव्हता.

🗺️ आकृती

flowchart LR
    q["❓ 11:40 — 'the results page is broken'"]
    subgraph room["🩺 the health room"]
      m["📊 metrics<br/>THAT something is wrong<br/>errors 0.1% → 6%"]
      t["🗺️ traces<br/>WHERE the time went<br/>grade service: 865 of 1040 ms"]
      l["📓 logs<br/>WHY, in detail<br/>'grade service timeout'"]
    end
    cpu["🔔 old bell: CPU > 80%<br/>64 noisy minutes"]
    q --> m --> t --> l
    q -.->|"quiet or wrong"| cpu

🗺️ काढलेली आवृत्ती + एक lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l01

❓ काय

Monitoring — तुम्ही आधीच निवडलेले signals पाहणे (CPU, disk, 5xx count) आणि आधीच निवडलेल्या मर्यादांवर alert करणे. आधी ओळखता येणाऱ्या failures साठी उपयोगी.
Observability — system च्या बाहेर येणाऱ्या माहितीवरून त्याचे आतले किती चांगले समजते ते. System तेव्हा observable असते जेव्हा तुम्ही एखाद्या नव्या प्रश्नाचे उत्तर ("फक्त 3A वर्गच का? फक्त 11:30 पासूनच का?") ते आधीच देत असलेल्या data मधून, नवा code ship न करता देऊ शकता.
Telemetry — system स्वतःबद्दल जो data बाहेर टाकते तो. त्याचे तीन मुख्य प्रकार, ज्यांना अनेकदा तीन खांब (three pillars) म्हणतात:
- Metrics (धडा 03) — वेळेनुसार आकडे: दर सेकंदाला requests, error ratio, latency buckets. ठेवायला स्वस्त आणि query साठी जलद. ते सांगतात की काहीतरी चुकले आहे.
- Traces (धडा 05) — अनेक services मधून जाणाऱ्या एका request चा मार्ग, प्रत्येक ठिकाणी लागलेल्या वेळेसह. ते सांगतात कुठे.
- Logs (धडा 02) — प्रत्येक घटनेसाठी एक नोंद, तपशीलासह. ते सांगतात का.
Signal विरुद्ध noise — signal तुम्हाला users ना जाणवणारी गोष्ट सांगतो. Noise म्हणजे असा alarm जो users ठीक असताना वाजतो (किंवा त्यांना त्रास होत असताना शांत राहतो).
एका ओळीत कोर्स: चांगले signals बाहेर टाका (धडे 02–07) → users ना त्रास होत असेल तेव्हाच alert करा (08) → "पुरेसे चांगले" म्हणजे काय यावर एकमत करा (09) → अग्निशमन सराव हाताळा (10) → खरे कारण शोधा (11) → ते लिहून काढा आणि system दुरुस्त करा (12).

🤔 का

कारण आधुनिक systems नव्या पद्धतींनी बिघडतात. निकालाच्या दिवशी एकच page load एक gateway, एक auth service, results API, एक database आणि एक grade service यांना स्पर्श करतो. मागच्या वर्षीच्या outage साठी बनवलेला dashboard या वर्षीचा outage दाखवत नाही. फक्त monitoring असेल तर team outage नंतर नवा graph जोडते आणि पुढच्या outage ची वाट पाहते. Observability असेल तर data आधीच तिथे असतो — तुम्हाला फक्त विचारायचे असते.

आणि जुने signals अनेकदा noise असतात. Lab मध्ये दिवसभरात CPU 64 मिनिटे 80% च्या पार जातो. त्यातले एकही मिनिट पालकांना जाणवणाऱ्या गोष्टीबद्दल नाही.

🔧 कसे (या repo मध्ये)

demo.py निकालाचा 480 मिनिटांचा एक दिवस बनवतो: REQ (मिनिटाला 1,000 requests), ERR (मिनिटाला errors: साधारणपणे 0.1%, मिनिटे 100–379 मधील slow leak दरम्यान 0.9%, मिनिटे 400–431 मधील वाईट deploy नंतर 6%) आणि CPU (व्यस्त, पण errors शी संबंध नसलेला). obs/reliability.py मधील threshold_alerts(cpu_series, limit=80) CPU मर्यादेच्या वर असलेले प्रत्येक मिनिट परत करते — म्हणजेच "boiler वरची घंटा". why() गोष्ट print करते आणि ती मिनिटे मोजते.

🧪 करून पाहा

python3 obs/demo.py why
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from demo import CPU, ERR, REQ; from reliability import threshold_alerts
for limit in (70, 80, 90):
    print(f"alert when CPU > {limit}% → {len(threshold_alerts(CPU, limit)):>3} minutes of alerts")
bad = [m for m in range(len(ERR)) if ERR[m] / REQ[m] > 0.05]
cpu = threshold_alerts(CPU)
print(f"minutes when parents saw >5% errors: {len(bad)} · CPU alerts in those minutes: {len(set(bad) & set(cpu))}")
print(f"CPU during the bad deploy: min {min(CPU[400:432])}% · max {max(CPU[400:432])}% · on a calm minute (10): {CPU[10]}%")
EOF
python3 obs/test_obs.py

✅ तपासा — तुम्हाला काय दिसायला हवे

why हे print करते:

── results day, 11:40: 'parents say the results page is broken' — can we answer WHY without adding code?
   monitoring: dashboards for failures you predicted (CPU, disk, 5xx)
   observability: ask NEW questions of the system from what it already emits — logs, metrics, traces
   this morning: CPU crossed 80% in 64 minutes — but CPU is not what parents feel
   the three signals: metrics say THAT something is wrong · traces say WHERE · logs say WHY

तुमचा snippet हे print करतो:

alert when CPU > 70% → 224 minutes of alerts
alert when CPU > 80% →  64 minutes of alerts
alert when CPU > 90% →   0 minutes of alerts
minutes when parents saw >5% errors: 32 · CPU alerts in those minutes: 4
CPU during the bad deploy: min 55% · max 84% · on a calm minute (10): 65%

Tests शेवटी 12/12 passed दाखवतात.

🏁 तुम्ही आत्ताच काय सिद्ध केले

CPU alarm एकतर खूप मोठ्याने वाजतो (70% ला 224 मिनिटे) किंवा बहिरा असतो (90% ला 0 मिनिटे). 80% ला, पालकांना खरोखर errors दिसले त्या 32 मिनिटांपैकी फक्त 4 मिनिटांत तो वाजला — आणि त्यांना errors दिसत नव्हते अशा 60 मिनिटांत वाजला. वाईट deploy दरम्यानचा CPU इतर कोणत्याही तासासारखाच दिसत होता. महत्त्वाचा signal म्हणजे जो users ना जाणवतो: errors आणि हळूपणा. उरलेला कोर्स तो signal टप्प्याटप्प्याने तयार करतो.

⚠️ नेहमीच्या चुका

"आपल्याला कोणत्या प्रश्नांची उत्तरे द्यावी लागतील?" हे विचारण्याआधीच dashboards बनवणे
symptoms (errors, latency) ऐवजी causes (CPU, memory, एका pod चा restart) वर लोकांना page करणे
logs, metrics आणि traces तीन वेगवेगळ्या tools मध्ये ठेवणे, त्यांना जोडणारा सामायिक id नसणे (धडा 02)
"आपल्याकडे भरपूर data आहे" — outage दरम्यान जलद query करता येत नाही असा data उपयोगाचा नाही
observability ला विकत घेण्याचे tool समजणे; ती तुम्ही बनवलेल्या system चा गुणधर्म आहे

🏭 प्रत्यक्ष वापरात

On a real account — नेहमीच्या stacks मधले तेच तीन signals:

Signal	Open source	AWS	Datadog
📊 Metrics	Prometheus + Grafana	CloudWatch Metrics	Datadog Metrics
📓 Logs	Loki, OpenSearch	CloudWatch Logs + Logs Insights	Datadog Logs
🗺️ Traces	Tempo, Jaeger	AWS X-Ray (CloudWatch)	Datadog APM
🔭 गोळा करणे	OpenTelemetry SDK + Collector (धडा 06)	ADOT (AWS Distro for OpenTelemetry)	Datadog Agent (OTLP स्वीकारतो)

कोणत्याही service ची पहिली प्रामाणिक तपासणी: ती आधीच जे बाहेर टाकते त्यावरून, पाच मिनिटांच्या आत तुम्ही हे तीन प्रश्न सोडवू शकता का?

1. What share of requests failed in the last 15 minutes?     (metrics)
2. For one failed request, which service spent the time?     (traces)
3. What did that service say about it, in detail?            (logs, joined by trace id)

🏭 Production मध्ये हे का महत्त्वाचे: आधी प्रश्न लिहा, मग telemetry. जी service हे तीन प्रश्न सोडवू शकत नाही ती निकालाच्या दिवसासाठी तयार नाही.

⏭️ पुढे

आरोग्य कक्षातले पहिले साधन: दैनंदिनी. प्रत्येक घटनेसाठी एक ओळ, अशी लिहिलेली की machine ती शोधू शकेल.

git checkout lesson-02-logs

🩺 Lesson 01 — Why observability: the school health room

📍 You are here: Lesson 01 of 12 · Next: lesson-02-logs

📦 What's in this branch

The one question this whole course answers: "Something is wrong — what, where, and why?" It is results day. At 11:40 parents say the results page is broken. Can the team answer why from what the system already tells them — without adding new code and waiting for it to break again? Real files you will use all the way through:

obs/signals.py — the health room's instruments: logs, metrics, traces and an OpenTelemetry-style Collector, as small honest models in pure Python (129 lines, zero dependencies)
obs/reliability.py — the health room's rulebook: alerts, burn rates, error budgets, incident clocks, suspect changes, five whys and the postmortem (69 lines)
obs/demo.py — one results day replayed minute by minute, one section per lesson (python3 obs/demo.py <section>; sections: why logs metrics percentiles tracing otel prometheus alerting slo incident rootcause postmortem)
obs/test_obs.py — 12 checks on the health room's rules
demo-output.txt — what a healthy run prints

🎒 Before you start: you need Python 3 and nothing else — no cloud account, no pip install. The lab holds teaching models of one 8-hour results day (1,000 requests a minute, a slow leak, a bad deploy). The real tools are Prometheus, Grafana, OpenTelemetry, CloudWatch and Datadog — the ideas and the maths are the same. Commands and files marked on a real account need those tools installed, and some of them cost money.

🧒 Explain like I'm 5

Every school has a health room 🩺. Katrina, the school nurse, works there.

When a pupil feels ill, Katrina does not guess. She has three tools:

a diary 📓 — one line for every visit: the time, the pupil, what happened;
a vital-signs chart 📊 — numbers over time: temperature every hour, how many pupils came in today;
a route card 🗺️ — a card the pupil carries from room to room; each room stamps the time in and out.

Now the old way. The head teacher, Dipika, once put a bell on the boiler. "If the boiler gets hot, ring the bell." The bell rang all morning. Nobody was ill. And the day a pupil really was ill, the bell was quiet — the boiler was fine.

That is the difference. Monitoring is the bell on the boiler: it watches the one thing you guessed in advance. Observability is the health room: the diary, the chart and the route cards together let Katrina answer new questions — "why is class 3A ill today?" — that nobody planned for.

🗺️ Diagram

flowchart LR
    q["❓ 11:40 — 'the results page is broken'"]
    subgraph room["🩺 the health room"]
      m["📊 metrics<br/>THAT something is wrong<br/>errors 0.1% → 6%"]
      t["🗺️ traces<br/>WHERE the time went<br/>grade service: 865 of 1040 ms"]
      l["📓 logs<br/>WHY, in detail<br/>'grade service timeout'"]
    end
    cpu["🔔 old bell: CPU > 80%<br/>64 noisy minutes"]
    q --> m --> t --> l
    q -.->|"quiet or wrong"| cpu

🗺️ Drawn version + a lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l01

❓ What

Monitoring — watching signals you chose in advance (CPU, disk, 5xx count) and alerting on limits you chose in advance. Good for failures you can predict.
Observability — how well you can understand a system's inside from what it outputs. A system is observable when you can answer a new question ("why only class 3A? why only since 11:30?") from data it already emits, without shipping new code.
Telemetry — the data a system emits about itself. The three main kinds, often called the three pillars:
- Metrics (lesson 03) — numbers over time: requests per second, error ratio, latency buckets. Cheap to keep and fast to query. They say that something is wrong.
- Traces (lesson 05) — the path of one request through many services, with the time spent in each. They say where.
- Logs (lesson 02) — one record per event, with details. They say why.
Signal vs noise — a signal tells you something users feel. Noise is an alarm that fires while users are fine (or stays quiet while they suffer).
The course in one line: emit good signals (lessons 02–07) → alert only when users are hurt (08) → agree what "good enough" means (09) → handle the fire drill (10) → find the real cause (11) → write it down and fix the system (12).

🤔 Why

Because modern systems fail in new ways. One page load on results day touches a gateway, an auth service, the results API, a database and a grade service. A dashboard built for last year's outage does not show this year's. With only monitoring, the team adds a new graph after the outage and waits for the next one. With observability, the data is already there — you only need to ask.

And the old signals are often noise. In the lab, CPU crosses 80% in 64 minutes of the day. None of those minutes are about what parents feel.

🔧 How (in this repo)

demo.py builds one results day of 480 minutes: REQ (1,000 requests a minute), ERR (errors per minute: 0.1% normally, 0.9% during a slow leak in minutes 100–379, 6% after a bad deploy in minutes 400–431) and CPU (busy, but not linked to errors). threshold_alerts(cpu_series, limit=80) in obs/reliability.py returns every minute where CPU is above the limit — the "bell on the boiler". why() prints the story and counts those minutes.

🧪 Try it

python3 obs/demo.py why
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from demo import CPU, ERR, REQ; from reliability import threshold_alerts
for limit in (70, 80, 90):
    print(f"alert when CPU > {limit}% → {len(threshold_alerts(CPU, limit)):>3} minutes of alerts")
bad = [m for m in range(len(ERR)) if ERR[m] / REQ[m] > 0.05]
cpu = threshold_alerts(CPU)
print(f"minutes when parents saw >5% errors: {len(bad)} · CPU alerts in those minutes: {len(set(bad) & set(cpu))}")
print(f"CPU during the bad deploy: min {min(CPU[400:432])}% · max {max(CPU[400:432])}% · on a calm minute (10): {CPU[10]}%")
EOF
python3 obs/test_obs.py

✅ Verify — what you should see

why prints:

── results day, 11:40: 'parents say the results page is broken' — can we answer WHY without adding code?
   monitoring: dashboards for failures you predicted (CPU, disk, 5xx)
   observability: ask NEW questions of the system from what it already emits — logs, metrics, traces
   this morning: CPU crossed 80% in 64 minutes — but CPU is not what parents feel
   the three signals: metrics say THAT something is wrong · traces say WHERE · logs say WHY

Your snippet prints:

alert when CPU > 70% → 224 minutes of alerts
alert when CPU > 80% →  64 minutes of alerts
alert when CPU > 90% →   0 minutes of alerts
minutes when parents saw >5% errors: 32 · CPU alerts in those minutes: 4
CPU during the bad deploy: min 55% · max 84% · on a calm minute (10): 65%

The tests end with 12/12 passed.

🏁 What you just proved

A CPU alarm is either too loud (224 minutes at 70%) or deaf (0 minutes at 90%). At 80% it fired in only 4 of the 32 minutes when parents really saw errors — and in 60 minutes when they did not. CPU during the bad deploy looked like any other hour. The signal that matters is the one users feel: errors and slowness. The rest of the course builds that signal, step by step.

⚠️ Common mistakes

building dashboards before asking "what questions will we need to answer?"
paging people on causes (CPU, memory, one pod restart) instead of symptoms (errors, latency)
keeping logs, metrics and traces in three tools with no shared id to join them (lesson 02)
"we have lots of data" — data you cannot query quickly during an outage does not help
treating observability as a tool you buy; it is a property of the system you build

🏭 In production

On a real account — the same three signals in the common stacks:

Signal	Open source	AWS	Datadog
📊 Metrics	Prometheus + Grafana	CloudWatch Metrics	Datadog Metrics
📓 Logs	Loki, OpenSearch	CloudWatch Logs + Logs Insights	Datadog Logs
🗺️ Traces	Tempo, Jaeger	AWS X-Ray (CloudWatch)	Datadog APM
🔭 Collecting	OpenTelemetry SDK + Collector (lesson 06)	ADOT (AWS Distro for OpenTelemetry)	Datadog Agent (accepts OTLP)

A first honest check on any service: can you answer these three questions in under five minutes, from what it already emits?

1. What share of requests failed in the last 15 minutes?     (metrics)
2. For one failed request, which service spent the time?     (traces)
3. What did that service say about it, in detail?            (logs, joined by trace id)

🏭 Why this matters in production: write the questions first, then the telemetry. A service that cannot answer those three questions is not ready for results day.

⏭️ Next

The first tool in the health room: the diary. One line per event, written so a machine can search it.

git checkout lesson-02-logs

← Course homeall lessons Next →logs

This page is the lesson's README from the lesson-01-why-observability branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.