🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

📈 धडा 15 — रचनेतूनच observability: "निरोगी" म्हणजे काय ते ठरवा

📍 तुम्ही इथे आहात: 18 पैकी धडा 15 · पुढे: lesson-16-cost-and-adrs

📦 या ब्रँचमध्ये काय आहे

धडे 01–14, आणि प्रत्येक box साठी तिसरा प्रश्न: ते चालते आहे हे आपल्याला कसे कळणार? तुम्ही शिकता SLI, SLO आणि SLA, error budgets (महिन्याला 432 / 43.2 / 4.3 मिनिटे), RED आणि USE, request ids सह structured logs, metrics, traces, आणि तुम्ही कारणांवर नाही तर लक्षणांवर alert का करता.

design/designs.py — error_budget(slo, days=30)
design/demo.py — python3 design/demo.py observability
design/test_design.py — check L15

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

app सुरू होण्याआधी दीपिका कार्यालयाला विचारते: "निरोगी म्हणजे काय?"

कतरिना म्हणते: "जलद." दीपिका म्हणते: "किती जलद, कोणासाठी, किती वेळा?" म्हणून त्या एक वाक्य लिहितात: "30 दिवसांत मोजले तर, प्रत्येक 1,000 पैकी 999 feed reads 200 ms पेक्षा कमी वेळात बरोबर उत्तर देतात."

मग ऐश्वर्या गणित करते. हजारात एक अपयशी ठरू शकतो. 30 दिवसांत ते 43.2 मिनिटांचा त्रास होतो. हीच मुभा — error budget. जर app ने या महिन्यात फक्त 10 मिनिटे वापरली, तर team धोका पत्करू शकते: नवी features, मोठे बदल. जर मुभा जवळजवळ संपली असेल, तर team वेग कमी करते आणि आधी गोष्टी दुरुस्त करते.

शेवटी कार्यालय ठरवते कोणाला कधी झोपेतून उठवायचे. server चा CPU जास्त असताना नाही — पालकांना CPU जाणवत नाही. फक्त तेव्हाच जेव्हा पालकांना जाणवते: errors किंवा हळू pages, जे मुभा खूप वेगाने संपवत आहेत.

🗺️ आकृती

flowchart LR
    sli["📏 SLI<br/>good reads ÷ all reads"] --> slo["🎯 SLO 99.9% / 30 days"]
    slo --> eb["⏳ error budget<br/>43.2 min per 30 days"]
    eb --> alert["🚨 alert on burn rate<br/>symptoms, not CPU"]
    subgraph sig["📡 three signals + one id"]
      logs["📜 logs<br/>JSON + request_id"]
      met["📊 metrics<br/>RED per API · USE per resource"]
      tr["🧵 traces<br/>one request across services"]
    end
    sig --> sli

🗺️ काढलेली आवृत्ती + एक lab: https://school-edh.pages.dev/system-design/lesson-diagrams.html#l15

❓ काय

SLI (service level indicator) — users ना जे जाणवते त्यावरून मोजलेला आकडा. बहुधा एक गुणोत्तर: चांगल्या घटना ÷ सर्व घटना. "200 ms पेक्षा कमी वेळात 2xx परत केलेले feed reads ÷ सर्व feed reads."
SLO (service level objective) — एका कालावधीसाठी SLI चे उद्दिष्ट: "30 दिवसांत 99.9%". team ने मान्य केलेले एक अंतर्गत ध्येय.
SLA (service level agreement) — दंडासह (पैसे परत) ग्राहकाला दिलेले वचन. नेहमी SLO पेक्षा सैल, म्हणजे करार मोडण्याआधीच team ला अडचण दिसते.
Error budget — कालावधीच्या 1 − SLO. 30 दिवसांसाठी: 99% → 432 min, 99.9% → 43.2 min, 99.99% → 4.3 min. प्रत्येक जास्तीचा nine म्हणजे दहा पट कमी जागा. Budget हे निर्णयाचे साधन आहे: budget शिल्लक → ship करा; budget संपले → स्थिर करा.
RED — प्रत्येक request-driven service साठी: Rate (सेकंदाला requests), Errors (अपयशी requests), Duration (latency, percentiles मध्ये: p50, p99).
USE — प्रत्येक resource साठी (CPU, disk, connection pool, queue): Utilisation (किती व्यस्त), Saturation (किती थांबलेले), Errors.
Structured logs — प्रत्येक घटनेसाठी ठरलेल्या fields सह एक JSON ओळ, आणि एक request id जो प्रत्येक service आपल्या logs मध्ये कॉपी करते आणि पुढे पाठवते. एका id ने पूर्ण कहाणी सापडते.
Metrics — वेळेनुसार स्वस्त आकडे (counters, histograms). Dashboards आणि alerts साठी चांगले. Labels लहान ठेवा: route, status — कधीही parent_id नाही.
Traces — एक request प्रत्येक service मधून, वेळ मोजलेल्या spans च्या झाडासारखी पाहिली जाते. 800 ms कुठे गेले ते त्या दाखवतात. OpenTelemetry हे सामान्य standard आहे.
लक्षणांवर alert — users ना त्रास होतो तेव्हा माणसाला page करा: SLO चे budget वेगाने जळते. कारणे (जास्त CPU, लवकरच भरणारा disk) ही tickets किंवा dashboard वरच्या गोष्टी, pages नाहीत. Burn-rate alert विचारतो: "या वेगाने budget कधी संपेल?"

🤔 का

कारण "ते चालते आहे का?" याची व्याख्या launch च्या आधी हवी, नाहीतर प्रत्येक incident वादाने सुरू होतो. SLO धडे 10–13 ला आकड्यांतही बदलते: रचनेला किती nines हवेत त्यावरून किती प्रती, zones आणि fallbacks लागतील ते ठरते. आणि प्रत्येक log ओळीत request id नसेल, तर कार्यालय एका पालकाची अपयशी request gateway, API, queue आणि SMS worker मधून शोधू शकत नाही.

🔧 कसे (या repo मध्ये)

design/designs.py मधले error_budget(slo, days=30) (1 − slo) × days × 24 × 60 मिनिटे परत करते, एका दशांशापर्यंत round केलेली. design/demo.py मधले observability() तीन SLOs छापते. snippet एक आठवडा आणि एक वर्ष असे कालावधी जोडतो, आणि एका महिन्याच्या वाईट reads ने budget चा किती भाग वापरला ते मोजतो.

🧪 करून पाहा

python3 design/demo.py observability
python3 - <<'EOF'
import sys; sys.path.insert(0, "design"); from designs import error_budget
for slo in (0.99, 0.995, 0.999, 0.9995, 0.9999):
    print(f"SLO {slo:.2%} → {error_budget(slo):>6} min / 30 days · {error_budget(slo, 7):>5} min / week · {error_budget(slo, 365):>7} min / year")
good, total = 99_950_000, 100_000_000                 # a month of feed reads
slo = 0.999; allowed_bad = total * (1 - slo); bad = total - good
print(f"SLI {good / total:.3%} · allowed bad {allowed_bad:,.0f} · bad so far {bad:,} → {bad / allowed_bad:.0%} of the budget spent")
EOF

✅ तपासा — तुम्हाला काय दिसायला हवे

observability हे छापते:

── decide what 'healthy' means BEFORE launch
   SLO 99.00% → error budget  432.0 minutes per 30 days
   SLO 99.90% → error budget   43.2 minutes per 30 days
   SLO 99.99% → error budget    4.3 minutes per 30 days
   RED per API (Rate, Errors, Duration) · USE per resource (Utilisation, Saturation, Errors)
   logs with a request id · metrics for alerts · traces across services · alert on the SLO, not on CPU

तुमचा snippet हे छापतो:

SLO 99.00% →  432.0 min / 30 days · 100.8 min / week ·  5256.0 min / year
SLO 99.50% →  216.0 min / 30 days ·  50.4 min / week ·  2628.0 min / year
SLO 99.90% →   43.2 min / 30 days ·  10.1 min / week ·   525.6 min / year
SLO 99.95% →   21.6 min / 30 days ·   5.0 min / week ·   262.8 min / year
SLO 99.99% →    4.3 min / 30 days ·   1.0 min / week ·    52.6 min / year
SLI 99.950% · allowed bad 100,000 · bad so far 50,000 → 50% of the budget spent

Tests मध्ये ✅ L15 a 99.9% SLO allows 43.2 minutes a month आहे.

🏁 तुम्ही आत्ताच काय सिद्ध केले

प्रत्येक जास्तीचा nine जागा दहाने भागतो: महिन्याला 432 → 43.2 → 4.3 मिनिटे. 99.99% वर team कडे आठवड्याला साधारण एक मिनिट असतो — चुकलेल्या एका deploy पेक्षाही कमी. म्हणूनच SLO गरजेशी जुळायला हवे (सूचना फलकासाठी 99.9% पुरेसे), इच्छेशी नाही. आणि budget मोजले जाते, अंदाजाने ठरत नाही: 100 million पैकी 50,000 वाईट reads म्हणजे 99.95% SLI — या महिन्याचे अर्धे budget खर्च झाले.

⚠️ नेहमीच्या चुका

100% चे SLO — काहीही बदलायला कधीच जागा नाही
आरोग्य user कडे ("read 200 ms मध्ये यशस्वी झाला") मोजण्याऐवजी server वर ("process चालू आहे") मोजणे
percentiles ऐवजी सरासरी — सरासरी हळू 1% लपवते
CPU, memory आणि disk वरचे alerts जे रात्री लोकांना उठवतात पण पालकांना काहीच जाणवत नाही
request id शिवाय मोकळ्या मजकुरातले logs — एक request services मधून शोधता येत नाही
metric label म्हणून parent_id — लाखो time series आणि मोठे बिल
SLO इतकेच SLA — पहिले चुकलेले उद्दिष्ट लगेच दंड ठरते

🏭 प्रत्यक्ष वापरात

खऱ्या account वर — प्रत्येक request साठी एक structured log ओळ (प्रत्येक service ला मिळालेला request_id कॉपी करते, किंवा edge वर एक तयार करते):

{"ts":"2026-09-27T09:00:01.234Z","level":"info","service":"notice-api","request_id":"req-7f3a9c",
 "route":"GET /parents/me/feed","status":200,"duration_ms":84,"cache":"hit"}

99.9% feed SLO साठी Prometheus burn-rate alert. एका तासासाठी 14.4 चा burn rate 30 दिवसांच्या budget चे 2% खर्च करतो; लहान window पण तपासल्याने अडचण थांबल्यावर alert लवकर थांबतो:

groups:
- name: feed-slo
  rules:
  - record: feed:error_ratio:rate1h
    expr: sum(rate(http_requests_total{route="/parents/me/feed",code=~"5.."}[1h]))
        / sum(rate(http_requests_total{route="/parents/me/feed"}[1h]))
  - record: feed:error_ratio:rate5m
    expr: sum(rate(http_requests_total{route="/parents/me/feed",code=~"5.."}[5m]))
        / sum(rate(http_requests_total{route="/parents/me/feed"}[5m]))
  - alert: FeedErrorBudgetBurn
    expr: feed:error_ratio:rate1h > (14.4 * 0.001) and feed:error_ratio:rate5m > (14.4 * 0.001)
    for: 2m
    labels: { severity: page }

OpenTelemetry सह traces — API automatic instrumentation सह सुरू करा आणि spans एका collector कडे पाठवा:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
OTEL_SERVICE_NAME=notice-api OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
    opentelemetry-instrument python app.py

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: प्रत्येक रचनेत एक "observability" विभाग जोडा: SLIs, SLO आणि त्याचे budget, RED आणि USE dashboards, आणि माणसाला उठवू शकणारे दोन-तीन alerts. बाकी सगळे dashboard किंवा ticket.

⏭️ पुढे

योजना बरोबर, सुरक्षित आणि मोजलेली आहे. शाळेला ती परवडेल का? आणि प्रत्येक box का आहे हे कोणाला लक्षात राहील का? Cost, trade-offs आणि ADRs.

git checkout lesson-16-cost-and-adrs

📈 Lesson 15 — Observability by design: decide what "healthy" means

📍 You are here: Lesson 15 of 18 · Next: lesson-16-cost-and-adrs

📦 What's in this branch

Lessons 01–14, plus the third question for every box: how will we know it works? You learn SLI, SLO and SLA, error budgets (432 / 43.2 / 4.3 minutes a month), RED and USE, structured logs with request ids, metrics, traces, and why you alert on symptoms, not on causes.

design/designs.py — error_budget(slo, days=30)
design/demo.py — python3 design/demo.py observability
design/test_design.py — check L15

🧒 Explain like I'm 5

Before the app opens, Dipika asks the office: "What does healthy mean?"

Katrina says: "Fast." Dipika says: "How fast, for whom, how often?" So they write one sentence: "999 of every 1,000 feed reads answer correctly in under 200 ms, counted over 30 days."

Then Aishwarya does the sum. One in a thousand may fail. Over 30 days that is 43.2 minutes of trouble. That is the allowance — the error budget. If the app uses only 10 minutes this month, the team may take risks: new features, big changes. If the allowance is nearly gone, the team slows down and fixes things first.

Last, the office decides when to wake someone up. Not when a server's CPU is high — parents do not feel CPU. Only when parents feel it: errors or slow pages, eating the allowance too fast.

🗺️ Diagram

flowchart LR
    sli["📏 SLI<br/>good reads ÷ all reads"] --> slo["🎯 SLO 99.9% / 30 days"]
    slo --> eb["⏳ error budget<br/>43.2 min per 30 days"]
    eb --> alert["🚨 alert on burn rate<br/>symptoms, not CPU"]
    subgraph sig["📡 three signals + one id"]
      logs["📜 logs<br/>JSON + request_id"]
      met["📊 metrics<br/>RED per API · USE per resource"]
      tr["🧵 traces<br/>one request across services"]
    end
    sig --> sli

🗺️ Drawn version + a lab: https://school-edh.pages.dev/system-design/lesson-diagrams.html#l15

❓ What

SLI (service level indicator) — a number measured from what users feel. Usually a ratio: good events ÷ all events. "Feed reads that returned 2xx in under 200 ms ÷ all feed reads."
SLO (service level objective) — the target for an SLI over a period: "99.9% over 30 days". An internal goal the team agrees on.
SLA (service level agreement) — a promise to a customer with a penalty (money back). Always looser than the SLO, so that the team sees trouble before the contract does.
Error budget — 1 − SLO of the period. For 30 days: 99% → 432 min, 99.9% → 43.2 min, 99.99% → 4.3 min. Each extra nine is ten times less room. The budget is a tool for decisions: budget left → ship; budget spent → stabilise.
RED — for every request-driven service: Rate (requests per second), Errors (failed requests), Duration (latency, as percentiles: p50, p99).
USE — for every resource (CPU, disk, a connection pool, a queue): Utilisation (how busy), Saturation (how much is waiting), Errors.
Structured logs — one JSON line per event with fixed fields, and a request id that every service copies into its own logs and passes on. One id finds the whole story.
Metrics — cheap numbers over time (counters, histograms). Good for dashboards and alerts. Keep labels small: route, status — never parent_id.
Traces — one request followed across every service, as a tree of timed spans. They show where the 800 ms went. OpenTelemetry is the common standard.
Alert on symptoms — page a person when users are hurt: the SLO's budget burns fast. Causes (high CPU, a full disk soon) are tickets or dashboard items, not pages. A burn-rate alert asks: "at this speed, when will the budget be gone?"

🤔 Why

Because "is it working?" needs a definition before launch, or every incident starts with an argument. The SLO also turns lessons 10–13 into numbers: how many nines the design needs decides how many copies, zones and fallbacks it needs. And without a request id in every log line, the office cannot follow one parent's failed request through the gateway, the API, the queue and the SMS worker.

🔧 How (in this repo)

error_budget(slo, days=30) in design/designs.py returns (1 − slo) × days × 24 × 60 minutes, rounded to one decimal. observability() in design/demo.py prints three SLOs. The snippet adds periods of a week and a year, and measures how much of the budget one month's bad reads used.

🧪 Try it

python3 design/demo.py observability
python3 - <<'EOF'
import sys; sys.path.insert(0, "design"); from designs import error_budget
for slo in (0.99, 0.995, 0.999, 0.9995, 0.9999):
    print(f"SLO {slo:.2%} → {error_budget(slo):>6} min / 30 days · {error_budget(slo, 7):>5} min / week · {error_budget(slo, 365):>7} min / year")
good, total = 99_950_000, 100_000_000                 # a month of feed reads
slo = 0.999; allowed_bad = total * (1 - slo); bad = total - good
print(f"SLI {good / total:.3%} · allowed bad {allowed_bad:,.0f} · bad so far {bad:,} → {bad / allowed_bad:.0%} of the budget spent")
EOF

✅ Verify — what you should see

observability prints:

── decide what 'healthy' means BEFORE launch
   SLO 99.00% → error budget  432.0 minutes per 30 days
   SLO 99.90% → error budget   43.2 minutes per 30 days
   SLO 99.99% → error budget    4.3 minutes per 30 days
   RED per API (Rate, Errors, Duration) · USE per resource (Utilisation, Saturation, Errors)
   logs with a request id · metrics for alerts · traces across services · alert on the SLO, not on CPU

Your snippet prints:

SLO 99.00% →  432.0 min / 30 days · 100.8 min / week ·  5256.0 min / year
SLO 99.50% →  216.0 min / 30 days ·  50.4 min / week ·  2628.0 min / year
SLO 99.90% →   43.2 min / 30 days ·  10.1 min / week ·   525.6 min / year
SLO 99.95% →   21.6 min / 30 days ·   5.0 min / week ·   262.8 min / year
SLO 99.99% →    4.3 min / 30 days ·   1.0 min / week ·    52.6 min / year
SLI 99.950% · allowed bad 100,000 · bad so far 50,000 → 50% of the budget spent

The tests include ✅ L15 a 99.9% SLO allows 43.2 minutes a month.

🏁 What you just proved

Each extra nine divides the room by ten: 432 → 43.2 → 4.3 minutes a month. At 99.99% the team has about one minute a week — less than one deploy that goes wrong. That is why the SLO must match the need (a notice board is fine at 99.9%), not a wish. And the budget is counted, not guessed: 50,000 bad reads of 100 million is an SLI of 99.95% — half of this month's budget is spent.

⚠️ Common mistakes

an SLO of 100% — no room to change anything, ever
measuring health on the server ("the process is up") instead of at the user ("the read succeeded in 200 ms")
averages instead of percentiles — the average hides the slow 1%
alerts on CPU, memory and disk that wake people at night while parents notice nothing
logs as free text with no request id — one request cannot be followed across services
parent_id as a metric label — millions of time series and a large bill
an SLA equal to the SLO — the first missed target is already a penalty

🏭 In production

On a real account — one structured log line per request (every service copies the request_id it receives, or creates one at the edge):

{"ts":"2026-09-27T09:00:01.234Z","level":"info","service":"notice-api","request_id":"req-7f3a9c",
 "route":"GET /parents/me/feed","status":200,"duration_ms":84,"cache":"hit"}

A Prometheus burn-rate alert for the 99.9% feed SLO. A burn rate of 14.4 for one hour spends 2% of a 30-day budget; checking a short window too makes the alert stop soon after the problem stops:

groups:
- name: feed-slo
  rules:
  - record: feed:error_ratio:rate1h
    expr: sum(rate(http_requests_total{route="/parents/me/feed",code=~"5.."}[1h]))
        / sum(rate(http_requests_total{route="/parents/me/feed"}[1h]))
  - record: feed:error_ratio:rate5m
    expr: sum(rate(http_requests_total{route="/parents/me/feed",code=~"5.."}[5m]))
        / sum(rate(http_requests_total{route="/parents/me/feed"}[5m]))
  - alert: FeedErrorBudgetBurn
    expr: feed:error_ratio:rate1h > (14.4 * 0.001) and feed:error_ratio:rate5m > (14.4 * 0.001)
    for: 2m
    labels: { severity: page }

Traces with OpenTelemetry — start the API with automatic instrumentation and send spans to a collector:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
OTEL_SERVICE_NAME=notice-api OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
    opentelemetry-instrument python app.py

🏭 Why this matters in production: add an "observability" section to every design: the SLIs, the SLO and its budget, the RED and USE dashboards, and the two or three alerts that may wake a person. Everything else is a dashboard or a ticket.

⏭️ Next

The plan is correct, safe and measured. Can the school pay for it? And will anyone remember why each box is there? Cost, trade-offs and ADRs.

git checkout lesson-16-cost-and-adrs

← Previoussecurity Next →cost and adrs

This page is the lesson's README from the lesson-15-observability branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.