🩺 शाळेच्या पद्धतीने observability शिका

# the 60-second wow — one results day, replayed:
git clone https://github.com/BaluRaut/learn-observability-school.git && cd learn-observability-school
python3 obs/demo.py                # 12 lessons: every signal, alert and minute
python3 obs/test_obs.py            # 12 checks across the lessons

तुम्हाला काय दिसायला हवे (छाटलेले — यश असे दिसते):

═══ why ═══
── results day, 11:40: 'parents say the results page is broken' — can we answer WHY without adding code?
   monitoring: dashboards for failures you predicted (CPU, disk, 5xx)
   observability: ask NEW questions of the system from what it already emits — logs, metrics, traces
   this morning: CPU crossed 80% in 64 minutes — but CPU is not what parents feel
   the three signals: metrics say THAT something is wrong · traces say WHERE · logs say WHY

═══ logs ═══
── structured logs: one JSON object per line, not free text
   {"ts": "11:40:02", "level": "ERROR", "msg": "grade service timeout", "route": "/results/3A", "status": 504, "ms": 1000, "trace_id": "a3ce929d", "user": "parent-9"}
── query level=ERROR route=/results/3A → 2 lines · trace ids ['a3ce929d', 'b7ad6b71']
   levels: DEBUG < INFO < WARN < ERROR · never log passwords, tokens or full personal data
   a request id / trace id in EVERY line is what joins logs to traces

═══ metrics ═══
── three kinds of metric — counter (only up), gauge (up and down), histogram (buckets)
   # HELP http_requests_total Requests served
   # TYPE http_requests_total counter
   http_requests_total{route="/results",status="200"} 9450
   http_requests_total{route="/results",status="404"} 30
   http_requests_total{route="/results",status="500"} 20
   # HELP print_queue_length Certificates waiting
   # TYPE print_queue_length gauge
   print_queue_length 137
   # HELP http_request_duration_seconds Request time
   # TYPE http_request_duration_seconds histogram
   http_request_duration_seconds_bucket{le="0.1"} 10
   http_request_duration_seconds_bucket{le="0.25"} 18
   http_request_duration_seconds_bucket{le="0.5"} 19
   http_request_duration_seconds_bucket{le="1.0"} 19
   http_request_duration_seconds_bucket{le="2.5"} 20
   http_request_duration_seconds_bucket{le="+Inf"} 20
   http_request_duration_seconds_sum 3.657
   http_request_duration_seconds_count 20
── labels route, status → 6 time series
── labels route, status, user_id → 300,000 time series
   never put user ids, emails or full URLs in labels — cardinality explodes the bill and the database

═══ percentiles ═══
── 20 requests (ms): 80 85 90 92 95 98 100 105 110 120 130 150 180 240 400 95 88 102 97 1200
   p50  true   100 ms · from buckets   100.0 ms
   p95  true   400 ms · from buckets   500.0 ms
   p99  true  1200 ms · from buckets  2200.0 ms
── two servers' p99: 2000 and 100 ms · their average 1050 ms · the real p99 of both: 2000 ms
   you cannot average percentiles — merge the histograms and compute again · buckets set the precision

═══ tracing ═══
── one parent's request, as a trace (one tree of spans):
   GET /results/3A [gateway] 0–1040 ms (1040 ms) ✗
     check parent login [auth] 5–45 ms (40 ms)
     load class list [results-api] 50–90 ms (40 ms)
     fetch grades [results-api] 95–1035 ms (940 ms) ✗
       SELECT grades [postgres] 100–160 ms (60 ms)
       grade service call [grade-service] 165–1030 ms (865 ms) ✗
   critical path: GET /results/3A → fetch grades → grade service call — 865 of 1040 ms is the grade service
   passed between services in a header: traceparent: 00-a3ce929d0e0e47364bf92f3577b34da6-00f067aa0ba902b7-01

═══ otel ═══
── OpenTelemetry: one SDK in the app → the Collector → any backend (Tempo, Jaeger, X-Ray, Datadog…)
   app (SDK: traces, metrics, logs) → OTLP → collector (receive → batch → sample → export) → backends
── 1,000 traces, tail sampling (keep every error and every trace > 500 ms, 1 in 10 of the rest) → kept 113, dropped 887
   all 4 error traces and all 10 slow traces survive — the boring ones are sampled
   semantic conventions: http.request.method, http.response.status_code, service.name — the same names everywhere

═══ prometheus ═══
── a counter scraped every 60 s: [(0, 1000), (60, 1600), (120, 2200), (180, 150), (240, 750)] — the app restarted at 180 s
   rate over 4 min = 8.1 requests/s (the reset is handled, not a negative spike)
── PromQL you will type every week:
   sum by (status) (rate(http_requests_total[5m]))
   sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
   histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
   Prometheus PULLS /metrics every 15–60 s and stores series · Grafana draws them · recording rules pre-compute

═══ alerting ═══
── 8 hours of traffic, SLO 99.9%: a slow leak (0.9% errors, minutes 100–379) and a bad deploy (6%, minutes 400–431)
   CPU > 80% alerts: 64 minutes of noise that do not track what users feel
   burn-rate TICKET (≥6× over 6 h AND 30 min): minutes 266–390, 401–408, 435–458 — first the slow leak (nobody woken at night), then around the deploy while the page is not firing
   burn-rate PAGE (≥14.4× over 1 h AND 5 min): minutes 409–434 — the bad deploy, 9 minutes after it began
   page for fast burns that hurt users now; open a ticket for slow burns; never page on a cause like CPU

═══ slo ═══
── SLI: good requests / all requests · SLO: 99.9% over 30 days · SLA: the contract with a penalty
   SLO 99.00% →  432.0 minutes of full outage per 30 days
   SLO 99.90% →   43.2 minutes of full outage per 30 days
   SLO 99.99% →    4.3 minutes of full outage per 30 days
── a 30-day window of 43,200,000 requests may have 43,200 failures · today's 8 hours failed 4,608
   one bad day used 10.7% of the month's budget → 89.3% left
   budget left → ship faster · budget gone → freeze risky changes and fix reliability first

═══ incident ═══
── roles: incident commander (Dipika) · ops lead (Katrina) · communications (Aishwarya) · scribe
   SEV1: most parents cannot see results — page everyone, status page, updates every 30 min
   SEV2: a feature is broken for many — page on-call, updates every hour
   SEV3: degraded or a small group affected — fix in working hours
── the timeline (minutes): {'started': 400, 'detected': 409, 'acknowledged': 411, 'mitigated': 432, 'resolved': 470}
   time_to_detect       9 min
   time_to_acknowledge  2 min
   time_to_mitigate     32 min
   time_to_resolve      70 min
   users_hurt_minutes   32 min
   first mitigate (roll back, switch off, add capacity), THEN find the cause

═══ rootcause ═══
── the error ratio first crosses 5% at minute 400
   changes in the 30 minutes before: [(398, 'deploy results-api v42')]
   why 1? parents saw 504 on /results
   why 2? the grade service call took 1 s and timed out
   why 3? v42 removed the grade cache, so every view hit the grade service
   why 4? the load test used 3 classes, not 900
   why 5? there is no production-like load test in the release checklist
   the root cause is usually a missing guard in the PROCESS, not a person

═══ postmortem ═══
   # Postmortem: results page 504s on results day
   
   Blameless: we look at the system, not at a person.
   
   ## Impact
   6% of results views failed for 32 minutes (minutes 400–431); ~1,900 parents saw an error
   
   ## Timeline
   - 398 deploy v42
   - 409 burn-rate page fired
   - 411 Dipika acknowledged, took command
   - 432 rollback to v41 — errors back to normal
   - 470 resolved; grade cache restored
   
   ## Root cause
   v42 removed the grade cache; the release load test did not match results-day traffic
   
   ## Action items
   - [Katrina] restore the grade cache with a test that fails without it (by 2 days)
   - [Aishwarya] results-day load test in the release checklist (by 1 week)
   - [Dipika] canary 5% for 15 min with burn-rate auto-rollback (by 2 weeks)
── the loop: signals → alerts on the SLO → incident → root cause → postmortem → action items → better signals

✅ done — the health room is watching

🗺️ संपूर्ण चित्र — एक आकृती, पूर्ण कोर्स

संपूर्ण कोर्स एका कॅनव्हासवर. 4K आवृत्तीसाठी क्लिक करा.

🩺 भाग 1 — signals (धडे 1–4)

एक git branch = एक कल्पना; branch 04 मध्ये धडे 01–04 आहेत. प्रत्येक आकडा obs/ मधून येतो — cloud account लागत नाही.

🧵 भाग 2 — services च्या पलीकडे (धडे 5–8)

एका request चा अनेक services मधून पाठलाग करा, सगळे एकाच पद्धतीने गोळा करा, आणि users ना जे जाणवते त्यावर alert करा.

🎯 भाग 3 — विश्वासार्हतेचे चक्र (धडे 9–12)

Signals चे निर्णयात रूपांतर करा: budgets, incidents, मूळ कारणे आणि टिकणारे उपाय.

📐 धड्यांच्या आकृत्या — क्रमांक अनुसरा

प्रत्येक धडा एका क्रमांकित बॉक्स-आणि-बाण आकृतीत — स्वतंत्र पानावरही.