🛠️ शाळेच्या पद्धतीने Site Reliability Engineering शिका

# the 60-second wow — one crew, twelve buildings:
git clone https://github.com/BaluRaut/learn-sre-school.git && cd learn-sre-school
python3 sre/demo.py               # 12 lessons: budgets, canaries, capacity, chaos
python3 sre/test_sre.py           # 12 checks across the lessons

तुम्हाला काय दिसायला हवे (छाटलेले — यश असे दिसते):

═══ job ═══
── the crew: Katrina, Dipika, Aishwarya · 3 × 40 h = 120 h a week · ops work (tickets, pages, manual fixes) per week:
   week 1:  48 h ops = 40.0%
   week 2:  54 h ops = 45.0%
   week 3:  63 h ops = 52.5%  ← over the 50% cap: extra tickets and pages go back to the dev team
   week 4:  71 h ops = 59.2%  ← over the 50% cap: extra tickets and pages go back to the dev team
   week 5:  58 h ops = 48.3%
   week 6:  45 h ops = 37.5%
   six-week average 47.1% · the rest of the week is ENGINEERING: tools that make next week's ops smaller
   ops team: carry more buckets as the school grows · SRE: fix the gutter, so the buckets are not needed
   DevOps is the culture (share ownership, automate, measure); SRE is one concrete way to practise it

═══ budget ═══
── timetable service · SLO 99.9% over a rolling 28 days · 1,000,000 requests a day → budget 28,000 failed requests
   example policy, signed in advance: < 75% used → normal · 75–100% → caution · ≥ 100% → freeze (P0 + security fixes only)
   day  1:   0.9% of the budget used → normal
   day 20:  83.9% of the budget used → caution
   day 24: 101.8% of the budget used → freeze
   day 34:  80.4% of the budget used → caution
   day 43:  57.1% of the budget used → normal
   day  6 incident burned 7,000 = 25.0% (> 20%) → its postmortem must carry a P0 action item
   day 15 incident burned 6,500 = 23.2% (> 20%) → its postmortem must carry a P0 action item
   the budget refills only as old failures leave the 28-day window — the freeze is lifted by time plus fixes, not by argument

═══ toil ═══
── the toil ledger: toil 24.2 h/week · overhead 3.0 h · engineering 8.0 h
   copy grades to the backup drive   5.00 h/week · build  6 h · upkeep 0.10 h/week → pays back in    1.2 weeks
   reset locked pupil accounts      10.00 h/week · build 16 h · upkeep 0.25 h/week → pays back in    1.6 weeks
   grow a full disk by hand          3.00 h/week · build  8 h · upkeep 0.10 h/week → pays back in    2.8 weeks
   restart the stuck print queue     6.00 h/week · build 20 h · upkeep 0.25 h/week → pays back in    3.5 weeks
   renew a TLS certificate by hand   0.19 h/week · build 12 h · upkeep 0.10 h/week → pays back in  137.1 weeks
   40 h of automation this quarter, best payback first: copy grades to the backup drive, reset locked pupil accounts, grow a full disk by hand (30 h)
   toil falls from 24.2 to 6.2 h/week, plus 0.45 h/week of upkeep for the new tools
   the print queue (20 h) waits for next quarter — better still, find out why it sticks and fix that
   the certificate never pays back by the ledger; renew it with ACME anyway, because an expired one is an outage

═══ oncall ═══
── 4 weeks of pages for the timetable service: 55 pages · 0.98 per 12-hour shift on average
   busiest shift 3 pages · shifts with more than 2 pages: 4 of 56
   one Pune crew, 24 h on call: 17 pages between 22:00 and 07:00
   follow-the-sun: Pune takes 07:00–19:00, a sister crew 12 h away takes the rest in its own daytime → night pages 0 + 0
   weekly rotation of 3: each person is on call 33.3% of weeks  ← over 25%: no time left for engineering
   weekly rotation of 5: each person is on call 20.0% of weeks
   weekly rotation of 8: each person is on call 12.5% of weeks
   the Google SRE book's numbers: at most 2 incidents per 12-hour shift, at most 25% of time on call, 8 people for one site

═══ canary ═══
── the baseline: the OLD version on a fresh copy, same size and traffic slice as the canary · 11 errors / 10,000 · p99 180 ms
   canary A: 13 errors / 10,000 · p99 185 ms → z   0.41 · p99 ×1.03 → PROMOTE
   canary B: 42 errors / 10,000 · p99 182 ms → z   4.26 · p99 ×1.01 → ROLLBACK
   canary C: 12 errors / 10,000 · p99 260 ms → z   0.21 · p99 ×1.44 → ROLLBACK
   canary D:  3 errors /    400 · p99 179 ms → z    n/a · p99 ×0.99 → EXTEND  (first minutes: baseline 1 / 400)
   rules: ROLLBACK if z > 3.0 (error rate clearly worse) or p99 > 1.2 × baseline · EXTEND under 1,000 requests each
   compare with a fresh baseline, not the whole old fleet: long-running machines differ (warm caches, leaks, old hosts)

═══ rollout ═══
── 1,000 requests/min · a bad change fails 30% of the requests it serves · noticed after 30 failures (at least 5 min)
   big bang, rollback = redeploy (15 min)     300 bad/min · noticed after  5 min → 6,000 failed requests
   big bang, flag off (1 min)                 300 bad/min · noticed after  5 min → 1,800 failed requests
   canary at 1%, rollback = redeploy            3 bad/min · noticed after 10 min →    75 failed requests
   canary at 1%, flag off                       3 bad/min · noticed after 10 min →    33 failed requests
   stages 1% → 5% → 25% → 100%, 30 min each: a full rollout takes 2 hours instead of 1 minute — that is the price
   exposure is the biggest lever, rollback speed the next; a flag only helps if the old path still works and the flag is removed later

═══ capacity ═══
── weekly peak requests/s, 12 weeks: [1206, 1226, 1276, 1373, 1365, 1448, 1474, 1533, 1600, 1613, 1684, 1707]
   straight-line fit: +47.8 req/s per week · forecast 26 weeks ahead: 2,963 req/s
   one server: 300 req/s at its load-test limit; plan at 70% = 210 → N = 15 servers for the forecast peak
   N+1 = 16 (one server can fail) · N+2 = 17 (one in maintenance AND one fails) · survive losing 1 of 3 zones: 24
   results day (2.5 × the normal peak = 7,408 req/s) needs 36 — plan it as an event, pre-scale, don't wait for autoscaling

═══ command ═══
── without roles: the timetable service is down at 23:00 · resolved after 110 min · 5 problem(s)
   ✗ 00:00 Katrina went off shift still holding ops — nobody holds it now
   ✗ no IC was ever named
   ✗ no comms was ever named
   ✗ no scribe was ever named
   ✗ 23:00 no status update for 110 min (rhythm: every 30)
── with incident command: the timetable service is down at 23:00 · resolved after 100 min · 0 problem(s)
   ✓ Dipika is IC (and scribe while it is small), Katrina fixes, Aishwarya talks; at 00:00 Dipika hands IC to crew-5 — acknowledged
   the detection clocks, the metrics and the postmortem are the Observability school's lessons 10–12

═══ nines ═══
── a request needs all three: gateway 99.99% × timetable app 99.95% × database 99.90% = 99.840%
   the chain is worse than its weakest part: 14.0 h a year of downtime allowed by the math
   two independent database copies: 1 − (1 − 0.999)² = 99.9999% → chain 99.940%
        99% →   87.60 h a year ·  403.2 min in 28 days
      99.9% →    8.76 h a year ·   40.3 min in 28 days
     99.95% →    4.38 h a year ·   20.2 min in 28 days
     99.99% →    0.88 h a year ·    4.0 min in 28 days
    99.999% →    0.09 h a year ·    0.4 min in 28 days
   each extra nine is 10 × less downtime: at 99.99% a 28-day window allows 4.0 min — about the time a paged person needs
   to wake up and log in, so at that level the fix must be automatic (failover, rollback), not a human

═══ overload ═══
── capacity 1,000 req/s · offered: critical 400 (log in, see timetable) + normal 500 + sheddable 600 (recommendations, prefetch)
   no priorities (everyone gets 1000/1500): {'critical': 267, 'normal': 333, 'sheddable': 400}
   shed by priority:                        {'critical': 400, 'normal': 500, 'sheddable': 100}
   + graceful degradation (normal and sheddable skip photo thumbnails: cost 0.7 each): {'critical': 400, 'normal': 500, 'sheddable': 357}
   3 retries per request           → 3,000 attempts/s arrive at a server that can do 1,000
   3 retries + a 10% retry budget  → 1,650 attempts/s arrive at a server that can do 1,000
   retries turn 1.5× overload into 3×; a retry budget caps the extra at 10% (bounded queues: Distributed Systems school, lesson 12)

═══ chaos ═══
── hypothesis: if zone B is lost, the cell's success rate stays ≥ 99.5% · abort if any minute < 99.0% · blast radius: 1 cell = 10% of users
   6 replicas in 3 zones: minutes 0–9 success 100% 100% 100% 100% 100% 100% 100% 100% 100% 100%
      → hypothesis held · no abort · 0 failed
   4 replicas in 2 zones: minutes 0–9 success 100% 100% 100% 57% 100% 100% 100% 100% 100% 100%
      → hypothesis refuted · ABORTED at minute 3, fault undone · 18,000 requests failed in the cell
   the same fault on all 10 cells at once would have failed 10 × as many — start small, widen only after a pass

═══ prr ═══
── production readiness review: the timetable service asks the crew to carry its pager
   ✅ L02  SLO agreed + error-budget policy signed
   ✅ L01  ops work under the 50% cap, toil ledger kept
   ✅ L04  on-call: 8+ people or two sites, a runbook per page
   ✅ L05  automated canary analysis gates every release
   ❌ L06  rollback in 5 minutes or less; risky features behind flags  [blocker]
   ✅ L07  capacity: 2-quarter forecast, survives a zone loss
   ✅ L08  incident roles trained, handoff template ready
   ❌ L09  dependency chain meets the SLO  [blocker]
   ✅ L10  load shedding by priority + a client retry budget
   ❌ L11  a game day in the last 90 days, abort conditions written
   ✅ obs  dashboards + burn-rate alerts (Observability school)
   ✅ ops  a backup restored in a test in the last 90 days
   → NOT READY — 2 blocker(s), 1 other item(s) to fix
   after fixes (flag-off rollback 1 min, two database copies, a game day): → READY — the crew takes the pager
── the whole picture: cap ops work → agree the budget → kill toil → sane on-call → judge canaries → roll out in stages
   → plan capacity → command incidents → do the availability math → shed load → test with chaos → review before launch

✅ done — the crew fixed the gutter

🗺️ संपूर्ण चित्र — एक आकृती, पूर्ण कोर्स

संपूर्ण कोर्स एका कॅनव्हासवर. 4K आवृत्तीसाठी क्लिक करा.

🪣 भाग 1 — काम (धडे 1–4)

एक git branch = एक कल्पना; branch 04 मध्ये धडे 01–04 आहेत. sre/ प्रत्येक वेळी चालवल्यावर तेच निकाल येतात — randomness ला seed दिलेला आहे.

🎚️ भाग 2 — सुरक्षितपणे बदल करणे (धडे 5–8)

बहुतेक outages एखाद्या बदलापासून सुरू होतात: तो छोट्या भागावर तपासा, तो कोणाला दिसतो ते मर्यादित ठेवा, capacity चे नियोजन करा, आणि काही बिघडले तर incident चे नेतृत्व करा.

🧯 भाग 3 — अपयश गृहीत धरून design (धडे 9–12)

वचन देण्याआधी हिशोब करा, overload मध्ये नीट अपयशी व्हा, chaos ने दाव्यांची तपासणी करा, आणि पथक pager हाती घेण्याआधी आढावा घ्या.

📐 धड्यांच्या आकृत्या — क्रमांक अनुसरा

प्रत्येक धडा एका क्रमांकित बॉक्स-आणि-बाण आकृतीत — स्वतंत्र पानावरही.