🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🛟 धडा 13 — बिघाडातून टिकणे: निकालाच्या दिवशी एखादा stall बिघडतो तेव्हा

📍 तुम्ही इथे आहात: 13 पैकी धडा 13 · मागील: lesson-12-queues-and-blueprint

📦 या ब्रँचमध्ये काय आहे

धडे 01–12, आणि scaling चा दुसरा अर्धा भाग: एखादा भाग बिघडला तरी जत्रा चालू ठेवणे. Servers ज्या service ला call करतात ती अडकली असेल, तर जास्त servers ने फायदा होत नाही. तुम्ही timeouts, exponential backoff + jitter सह retries, circuit breaker, health checks, graceful degradation, idempotency, dead-letter queue, multi-AZ, आणि dependencies च्या साखळीत availability का गुणली जाते हे शिकता. scale/demo.py मधले resilience() यातले प्रत्येक दाखवते, आणि त्याचे models scale/sim.py च्या "surviving failure" भागात आहेत.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

निकालाचा दिवस, सकाळचे 10 वाजलेत. जत्रेच्या मागच्या बाजूचा grades stall बिघडतो. त्याचा कारकून अडकला आहे आणि उत्तर देत नाही.

प्रत्येक खिडकी पालकांना grades stall कडे पाठवते आणि वाट पाहते. आणि वाट पाहते. लवकरच प्रत्येक कारकून वाट पाहत असतो, आणि संपूर्ण जत्रा ठप्प होते — एका stall मुळे. म्हणून दीपिका प्रत्येक खिडकीला काही साधने देते:

⏱️ खिडकीवर एक stopwatch. "Grades stall साठी एक सेकंद थांबा. मग थांबणे सोडा आणि पुढच्या पालकाला मदत करा." हा timeout.
🙏 सभ्य retries. "अपयश आले तर पुन्हा प्रयत्न करा — पण आधी थोडे थांबा, आणि प्रत्येक वेळी जास्त थांबा. आणि सगळ्यांनी एकाच क्षणी प्रयत्न करू नका: प्रत्येक खिडकी थोडा random विराम जोडते." हा jitter सह backoff.
🔌 एक breaker switch. ऐश्वर्या grades stall वर लक्ष ठेवते. सलग 5 अपयशांनंतर ती switch पाडते: "10 सेकंद तिथे कोणालाही पाठवू नका." मग ती एका पालकाला प्रयत्न करू देते. ते चालले, तर ती switch परत चालू करते.
🪧 बंद gate ऐवजी छोटी सूचना. Stall बिघडलेला असताना कार्यालय फलकावर शेवटचे माहीत असलेले निकाल लावते, "5 मिनिटे जुने" अशी खूण करून. परिपूर्ण नाही — पण gate उघडे राहते.
🔖 दुप्पट शुल्क थांबवणारा शिक्का. कतरिना फी भरते. मदतनीस चुकून तिची पावती दोनदा टाकतो. खजिनदार शिक्का पाहते — "आधीच भरले" — आणि तिच्याकडून फक्त एकदाच शुल्क घेते.
🗂️ खराब तिकिटांसाठी बाजूचा ट्रे. एक फाटलेले तिकीट प्रत्येक printer वर अपयशी ठरते. 3 प्रयत्नांनंतर ते बाजूच्या ट्रेमध्ये जाते, आणि printers बाकीची तिकिटे छापत राहतात.
🏢🏢🏢 तीन इमारती. जत्रा शाळेच्या तीन इमारतींमध्ये भरते. एकीची वीज गेली, तर उरलेल्या दोन चालू राहतात.

आणि आणखी एक नियम: पालकाची भेट gate, एक खिडकी, सूचना फलक आणि नोंदवही यांमधून जाते. त्यातले प्रत्येक 1,000 पैकी 999 वेळा चालत असेल, तर संपूर्ण भेट त्यातल्या कोणत्याही एकापेक्षा थोडी कमी वेळा यशस्वी होते.

🗺️ आकृती

flowchart LR
    u["👪 parent"] --> api["🍳 API"]
    api -->|"⏱️ timeout 1 s"| cb{"🔌 circuit breaker<br/>closed · open · half-open"}
    cb -->|"closed"| dep["📒 grades service"]
    cb -->|"open: fail fast"| fb["🪧 fallback<br/>cached page, '5 minutes old'"]
    dep -.->|"error"| rt["🙏 retry<br/>backoff + jitter<br/>500 at once → 81 per 10 ms"]
    rt -.-> cb
    q["📬 SQS"] -->|"at least once"| wk["🖨️ worker<br/>🔖 idempotency key"]
    q -.->|"after maxReceiveCount"| dlq["🗂️ DLQ"]
    az["🏢🏢🏢 3 AZs<br/>one AZ 99.5% → three 99.999988%"]

🗺️ रेखाटलेली आवृत्ती + एक lab: https://school-edh.pages.dev/scaling/lesson-diagrams.html#l13

❓ काय

बिघाड सामान्य आहे — मोठ्या प्रमाणावर, कोणता ना कोणता भाग नेहमी हळू किंवा बंद असतो: एखादा server, network link, database failover, संपूर्ण Availability Zone (AZ). ध्येय "काहीच बिघडत नाही" हे नाही. ध्येय आहे "एका बिघाडाने सगळे थांबत नाही".
Timeout — caller उत्तरासाठी जास्तीत जास्त किती वेळ थांबतो. तो नसेल तर अडकलेली service जोपर्यंत अडकलेली आहे तोपर्यंत एक thread, connection किंवा Lambda environment धरून ठेवते. Little's law नुसार (धडा 02), 2,000 req/s × 30 s = एकाच वेळी 60,000 requests थांबलेल्या. अनेक HTTP clients मध्ये default ने timeout नसतो, म्हणून प्रत्येक remote call वर तो ठेवा: request च्या एकूण वेळेच्या मर्यादेत एक connect timeout आणि एक read timeout.
Retry — अपयशी call पुन्हा करून पाहणे. छोट्या अडथळ्यांमध्ये (तुटलेले connection, 503, throttle) हे मदत करते. फक्त सुरक्षित calls retry करा: reads, किंवा idempotent असलेले writes. प्रयत्नांची संख्या मर्यादित ठेवा (3 नेहमीचे आहे).
Exponential backoff — प्रत्येक retry आधी जास्त थांबा: 100, 200, 400, 800 ms …, एका cap पर्यंत. आजारी service ला सावरायला वेळ मिळतो.
Jitter — प्रत्येक थांबण्यात एक random भाग. Full jitter 0 आणि backoff आकडा यांच्यामधला random वेळ थांबते. Jitter नसेल तर एकत्र अपयशी झालेले 500 clients एकत्रच retry करतात — एक retry storm, जो service ला पुन्हा ओव्हरलोड करू शकतो.
Circuit breaker — dependency समोरचा switch, ज्याच्या तीन अवस्था असतात:
- closed — calls पुढे जातात; अपयश मोजले जातात;
- open — सलग N अपयशांनंतर, काही वेळ calls न थांबता लगेच अपयशी होतात;
- half-open — तो वेळ संपल्यावर, एक चाचणी call: यश मिळाले तर breaker closed होतो, अपयश आले तर तो पुन्हा open होतो. तो दोन्ही बाजूंचे रक्षण करतो: तुमचे threads थांबणे सोडतात, आणि आजारी service ला मोकळीक मिळते.
Health checks — load balancer (धडा 05) किंवा DNS प्रत्येक target ला विचारतो "तू ठीक आहेस का?" आणि traffic फक्त निरोगी targets कडेच पाठवतो. Check उथळ ठेवा (ही process सेवा देऊ शकते का?). Database ला call करणारा खोल check database हळू असताना प्रत्येक server एकदम unhealthy ठरवू शकतो.
Graceful degradation — एखादा भाग बिघडला की error ऐवजी छोटे उत्तर द्या: "5 मिनिटे जुने" अशी खूण असलेले cached निकालाचे पान, photos शिवायचे पान, recommendations नाहीत. आवश्यक काय आणि काय बंद करता येईल हे आधीच ठरवा.
Idempotency — तीच request दोनदा केली तरी परिणाम एकदा केल्यासारखाच. Caller एक idempotency key पाठवतो (उदा. payment id); server ने पूर्ण केलेल्या keys लक्षात ठेवतो आणि पुन्हा आलेल्या request ला पहिलेच उत्तर परत देतो. यामुळे retries आणि SQS ची at-least-once delivery (धडा 12) सुरक्षित होते.
Dead-letter queue (DLQ) — maxReceiveCount वेळा अपयशी ठरलेला message एका वेगळ्या queue मध्ये जातो. Workers बाकीचे काम चालू ठेवतात; एखादी व्यक्ती खराब message पाहते आणि दुरुस्तीनंतर त्याला परत redrive करू शकते (धडा 12).
Multi-AZ — वेगवेगळ्या AZs मध्ये प्रती (एका Region मधल्या वेगळ्या इमारती, वीज आणि network). 3 AZs मध्ये प्रती असलेली system तिन्ही बंद असतील तरच बंद पडते: 1 − (1 − a)³. यात AZs स्वतंत्रपणे बिघडतात असे गृहीत धरले आहे — एखादा सामायिक bug किंवा वाईट deploy तरीही तिन्ही बंद पाडू शकतो.
Dependency failure — साखळीत availability गुणली जाते — एखाद्या request ला API आणि Redis आणि database लागत असेल, तर ती तिन्ही चालत असतील तेव्हाच चालते: a₁ × a₂ × a₃. 99.9% × 99.9% × 99.95% ≈ 99.75% — कोणत्याही एका भागापेक्षा कमी. प्रत्येक अतिरिक्त hard dependency एकूण आकडा कमी करते. Fallbacks आणि प्रती तो वाढवतात.

🤔 का

कारण scaling भाग वाढवते, आणि प्रत्येक भाग बिघडू शकतो. धडे 01–12 ने जत्रा मोठी केली: जास्त servers, pods, Lambdas, caches, replicas, partitions आणि queues. जास्त भाग असले की कोणत्याही दिवशी कुठेतरी बिघाड होण्याची शक्यता जास्त. Timeouts नसतील तर एक अडकलेली service प्रत्येक thread भरून टाकते. Jitter नसेल तर retries एका छोट्या अडथळ्याचे वादळ करतात. Idempotency नसेल तर retry पालकाकडून दोनदा शुल्क घेतो. या धड्यातली साधने मोठ्या system ला स्थिरही बनवतात.

🔧 कसे (या repo मध्ये)

scale/sim.py च्या "surviving failure" भागात छोटे models आहेत:

call_with_timeout(latency_ms, timeout_ms) हे ('ok', latency) किंवा ('timeout', timeout_ms) परत करते.
backoff_ms(attempt, base_ms=100, cap_ms=3200, jitter=None) म्हणजे min(cap, base × 2^attempt), किंवा jitter असेल तर त्याचा random वाटा.
retry_storm(clients, attempts, use_jitter) प्रत्येक client ला t=0 ला अपयशी करते, त्यांचे retries चालवते आणि सर्वात व्यस्त 10 ms slot परत करते — service वर एकदम किती retries आदळतात.
CircuitBreaker(threshold=5, open_s=10) — call(t, dependency_ok) हे ok, failed किंवा fast-fail परत करते, आणि closed, open आणि half-open मध्ये फिरते.
deliver_at_least_once() + process(deliveries, idempotent) घेतलेले शुल्क मोजतात.
redrive(messages, poison, max_receives=3) नेहमी अपयशी ठरणारा message max_receives receives नंतर DLQ मध्ये पाठवते.
parallel(a, copies) = 1 − (1 − a)^copies; serial(*a) साखळीचा गुणाकार करते.

scale/demo.py मधले resilience() प्रत्येकाचे एक उदाहरण चालवते. Breaker उदाहरणातली payments service सेकंद 0 पासून सेकंद 29 पर्यंत बंद असते, आणि demo तिला 40 सेकंद दर सेकंदाला एकदा call करते.

🧪 करून पाहा

python3 scale/demo.py resilience
python3 - <<'EOF'
import sys; sys.path.insert(0, "scale"); from sim import call_with_timeout, littles_law, backoff_ms, retry_storm
for lat in (120, 900, 1500, 30000):
    print(f"latency {lat:>5} ms, timeout 1,000 ms →", call_with_timeout(lat, 1000))
for wait in (30, 1, 0.2):
    print(f"2,000 req/s, each caller waits up to {wait:>4} s → {littles_law(2000, wait):>7,.0f} requests stuck at worst")
print("backoff (no jitter):", [backoff_ms(a) for a in range(7)], "ms")
for clients in (100, 500, 2000):
    print(f"{clients:>4} clients, 4 retries → busiest 10 ms: {retry_storm(clients, 4, False):>4} without jitter, {retry_storm(clients, 4, True):>3} with full jitter")
EOF
python3 - <<'EOF'
import sys; sys.path.insert(0, "scale"); from sim import CircuitBreaker, deliver_at_least_once, process, redrive, parallel, serial
for threshold, open_s in ((5, 10), (3, 10), (5, 2), (10, 30)):
    b = CircuitBreaker(threshold, open_s); log = [b.call(t, t >= 30) for t in range(40)]
    print(f"breaker {threshold:>2} failures, open {open_s:>2} s → reached the sick service {log.count('failed'):>2} · failed fast {log.count('fast-fail'):>2} · first ok at {log.index('ok')} s")
d = deliver_at_least_once(["pay-1", "pay-2", "pay-3", "pay-4"], ["pay-2", "pay-4", "pay-4"])
print(f"{len(d)} deliveries of 4 payments → plain {process(d, False)} charges · idempotent {process(d, True)}")
for mr in (1, 3, 5):
    done, dlq, rec = redrive([f"pdf-{i}" for i in range(10)], {"pdf-3", "pdf-7"}, max_receives=mr)
    print(f"maxReceiveCount {mr} → made {len(done)}, DLQ {dlq}, receives {rec}")
for a in (0.99, 0.995):
    print(f"one AZ {a:.1%} → two {parallel(a, 2):.4%} · three {parallel(a, 3):.6%}")
print(f"chain of 3 at 99.9% → {serial(0.999, 0.999, 0.999):.2%} · chain of 6 → {serial(*[0.999] * 6):.2%} · 6 links each on 3 AZs → {serial(*[parallel(0.999, 3)] * 6):.6%}")
EOF
python3 scale/test_scale.py

✅ तपासा — तुम्हाला काय दिसायला हवे

resilience हे छापते:

── timeouts: the results API usually answers in 120 ms; today it is stuck at 30 s
   latency    120 ms, timeout 1,000 ms → ('ok', 120)
   latency  30000 ms, timeout 1,000 ms → ('timeout', 1000)
   without a timeout every thread waits 30 s — 2,000 req/s × 30 s = 60,000 stuck requests (Little's law again)
── retries: 500 clients fail at once and retry 4 times (backoff 100 → 800 ms)
   no jitter   → busiest 10 ms:  500 retries hit the database together
   full jitter → busiest 10 ms:   81 retries hit the database together
── circuit breaker (5 failures → open for 10 s → one trial call); the payments service is down for 30 s
   calls that reached the sick service: 7 · failed fast without waiting: 27 · state at 40 s: closed
── graceful degradation: the grade service is down → show the results page from Redis, marked '5 minutes old'
   a slower, older page beats an error page; non-essential parts (photos, recommendations) can switch off
── at-least-once: 4 deliveries for 3 payments → plain worker charges 4, idempotent worker charges 3
── a poison message: 9 PDFs made, ['pdf-7'] moved to the DLQ after 3 receives (12 receives in total)
── multi-AZ: one AZ at 99.5% → two AZs 99.9975% · three AZs 99.999988%
   a chain API 99.9% × Redis 99.9% × database 99.95% = 99.75% — every dependency lowers the total
   scaling adds capacity; resilience keeps the fair open when one part of it breaks

तुमचा पहिला snippet हे छापतो:

latency   120 ms, timeout 1,000 ms → ('ok', 120)
latency   900 ms, timeout 1,000 ms → ('ok', 900)
latency  1500 ms, timeout 1,000 ms → ('timeout', 1000)
latency 30000 ms, timeout 1,000 ms → ('timeout', 1000)
2,000 req/s, each caller waits up to   30 s →  60,000 requests stuck at worst
2,000 req/s, each caller waits up to    1 s →   2,000 requests stuck at worst
2,000 req/s, each caller waits up to  0.2 s →     400 requests stuck at worst
backoff (no jitter): [100, 200, 400, 800, 1600, 3200, 3200] ms
 100 clients, 4 retries → busiest 10 ms:  100 without jitter,  21 with full jitter
 500 clients, 4 retries → busiest 10 ms:  500 without jitter,  81 with full jitter
2000 clients, 4 retries → busiest 10 ms: 2000 without jitter, 292 with full jitter

तुमचा दुसरा snippet हे छापतो:

breaker  5 failures, open 10 s → reached the sick service  7 · failed fast 27 · first ok at 34 s
breaker  3 failures, open 10 s → reached the sick service  5 · failed fast 27 · first ok at 32 s
breaker  5 failures, open  2 s → reached the sick service 17 · failed fast 13 · first ok at 30 s
breaker 10 failures, open 30 s → reached the sick service 10 · failed fast 29 · first ok at 39 s
7 deliveries of 4 payments → plain 7 charges · idempotent 4
maxReceiveCount 1 → made 8, DLQ ['pdf-3', 'pdf-7'], receives 10
maxReceiveCount 3 → made 8, DLQ ['pdf-3', 'pdf-7'], receives 14
maxReceiveCount 5 → made 8, DLQ ['pdf-3', 'pdf-7'], receives 18
one AZ 99.0% → two 99.9900% · three 99.999900%
one AZ 99.5% → two 99.9975% · three 99.999988%
chain of 3 at 99.9% → 99.70% · chain of 6 → 99.40% · 6 links each on 3 AZs → 99.999999%

Tests ✅ L13 timeouts, jittered retries, a circuit breaker and idempotent workers आणि 13/13 passed ने संपतात.

🏁 तुम्ही आत्ताच काय सिद्ध केले

Timeout "60,000 अडकलेल्या requests" चे जास्तीत जास्त 2,000 (1 s) किंवा 400 (0.2 s) करतो. Jitter नसेल तर प्रत्येक client चा retry त्याच 10 ms मध्ये येतो — एकदम 2,000; full jitter त्यांना 292 पर्यंत पसरवतो. Breaker आजारी service च्या 30 सेकंदांच्या बंदीत फक्त 7 calls तिच्यापर्यंत पोहोचू देतो; बाकीचे 23 लगेच अपयशी होतात. त्याची एक किंमत आहे: service सेकंद 30 ला परत आली, आणि 10 सेकंदांच्या open वेळेने ते फक्त 34 ला ओळखले. कमी open वेळ (2 s) लगेच सावरते पण आजारी service कडे 17 calls पाठवते. या दोघांमधून तुम्ही निवड करता. शिक्का (idempotency) 7 deliveries असतानाही 4 payments साठी 4 शुल्कच घेतो. maxReceiveCount काहीही असो, DLQ 2 खराब messages बाजूला ठेवते; जास्त count फक्त त्यांच्यावर जास्त receives खर्च करतो — पण 1 चा count एका छोट्या अडथळ्यानंतर चांगल्या message लाही बाजूला ठेवेल. आणि साखळी प्रत्येक दुव्यासोबत थोडे गमावते (6 दुवे: 99.40%), तर 3 AZs मधल्या प्रती तो आकडा पुन्हा वाढवतात — जोपर्यंत AZs स्वतंत्रपणे बिघडतात तोपर्यंत.

⚠️ नेहमीच्या चुका

HTTP किंवा database client वर timeout नाही — अनेक libraries मध्ये default म्हणजे "कायम थांबा"
caller च्या स्वतःच्या timeout पेक्षा मोठा timeout — caller सोडून देतो, आणि काम कोणासाठीच नसताना चालूच राहते
प्रत्येक layer वर retries (browser, API, service, SDK): एका click साठी 3 × 3 × 3 = 27 calls
backoff आणि jitter शिवाय retries — service सर्वात कमकुवत असतानाच retry storm
idempotency key शिवाय payment किंवा "send email" retry करणे — retry म्हणजे दुसरे शुल्क
fallback नसलेला circuit breaker — लगेच अपयशी होणे चांगले, पण user ला तरीही उत्तर हवे असते
database ला call करणारा खोल /health check, त्यामुळे प्रत्येक server एकदम check मध्ये नापास होतो
1 चा maxReceiveCount — एक छोटा अडथळा चांगला message DLQ मध्ये पाठवतो
कोणीच न पाहणारी DLQ — तिच्या depth वर alarm लावा
एकच NAT gateway, एकच cache node किंवा एकाच AZ मधला database असलेले "multi-AZ"

🏭 प्रत्यक्ष वापरात

खऱ्या account वर — प्रत्येक client वर timeouts आणि retries. Connect आणि read timeout, full-jitter backoff आणि एकूण वेळेची मर्यादा असलेले Python requests:

import random, time, requests

def get_results(cls, attempts=4, base=0.1, cap=3.2, budget=5.0):
    start = time.monotonic()
    for a in range(attempts):
        try:
            r = requests.get(f"https://grades.internal/results/{cls}", timeout=(0.5, 1.0))  # connect, read (s)
            if r.status_code < 500 and r.status_code != 429:
                return r
        except requests.RequestException:
            pass
        wait = random.uniform(0, min(cap, base * 2 ** a))          # full jitter
        if time.monotonic() - start + wait > budget:
            break
        time.sleep(wait)
    return None            # the caller shows the cached page, marked "5 minutes old"

(requests मध्ये read timeout म्हणजे bytes मधले सर्वात मोठे अंतर, संपूर्ण उत्तरावरची मर्यादा नाही — loop मधले budget ही एकूण मर्यादा आहे.)

AWS SDKs तुमच्यासाठी retry करतात. Retry mode निवडा: standard (jitter सह exponential backoff, default 3 प्रयत्न) किंवा adaptive (standard अधिक client-side rate limiter, जो AWS तुम्हाला throttle करते तेव्हा वेग कमी करतो). boto3 साठी:

import boto3
from botocore.config import Config
cfg = Config(connect_timeout=1, read_timeout=2, retries={"mode": "standard", "max_attempts": 3})
ddb = boto3.client("dynamodb", config=cfg)

किंवा कोणत्याही SDK आणि AWS CLI साठी: export AWS_RETRY_MODE=standard AWS_MAX_ATTEMPTS=3.

Circuit breakers — code मधली library (Java साठी Resilience4j, .NET साठी Polly, Node.js साठी opossum, Python साठी pybreaker), किंवा service mesh हे प्रत्येक service साठी करते. Istio: route वर timeout आणि retries, आणि आजारी pod ला काही वेळ pool मधून काढणारे outlier detection:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: { name: grades }
spec:
  hosts: [grades]
  http:
    - route: [{ destination: { host: grades } }]
      timeout: 2s
      retries: { attempts: 2, perTryTimeout: 1s, retryOn: "5xx,reset,connect-failure" }
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata: { name: grades }
spec:
  host: grades
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

Health checks — ALB target group /health तपासतो (धडा 05). Route 53 health checks संपूर्ण endpoint वर लक्ष ठेवतात, आणि पहिला अपयशी ठरला की failover record users ना दुसऱ्या endpoint कडे पाठवतो:

aws route53 create-health-check --caller-reference results-api-2026-05 \
    --health-check-config '{"Type":"HTTPS","FullyQualifiedDomainName":"api.school.example",
      "ResourcePath":"/health","RequestInterval":30,"FailureThreshold":3}'

SQS DLQ redrive policy + maxReceiveCount (धडा 12), आणि मग दुरुस्तीनंतर बाजूला ठेवलेले messages परत हलवा:

aws sqs set-queue-attributes --queue-url https://sqs.ap-south-1.amazonaws.com/111122223333/certificates \
    --attributes '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"arn:aws:sqs:ap-south-1:111122223333:certificates-dlq\",\"maxReceiveCount\":\"5\"}"}'
aws sqs start-message-move-task \
    --source-arn arn:aws:sqs:ap-south-1:111122223333:certificates-dlq \
    --destination-arn arn:aws:sqs:ap-south-1:111122223333:certificates

Idempotency keys — Powertools for AWS Lambda (Python) प्रत्येक key DynamoDB मध्ये साठवते आणि पुन्हा आलेल्या request ला साठवलेले उत्तर परत देते (table ला string partition key id, आणि expiration attribute वर TTL लागतो):

from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer, IdempotencyConfig, idempotent)

persistence = DynamoDBPersistenceLayer(table_name="school-idempotency")
config = IdempotencyConfig(event_key_jmespath="powertools_json(body).payment_id",
                           expires_after_seconds=3600)

@idempotent(config=config, persistence_store=persistence)
def handler(event, context):
    charge_fee(event)                     # runs once per payment_id, even if the event comes twice
    return {"statusCode": 200, "body": "paid"}

Database आणि cache साठी Multi-AZ — आणि निकालाच्या दिवसाआधी failover ची चाचणी घ्या:

aws rds modify-db-instance --db-instance-identifier school-db --multi-az --apply-immediately
aws rds reboot-db-instance --db-instance-identifier school-db --force-failover
aws elasticache modify-replication-group --replication-group-id school-sessions \
    --multi-az-enabled --automatic-failover-enabled --apply-immediately

AWS Fault Injection Service (FIS) सह chaos testing — एका AZ मधले API instances 10 मिनिटांसाठी बंद करा, आणि p99 alarm वाजला तर experiment लगेच थांबवा:

{
  "description": "results day drill: lose AZ a",
  "roleArn": "arn:aws:iam::111122223333:role/fis-school",
  "stopConditions": [{ "source": "aws:cloudwatch:alarm",
    "value": "arn:aws:cloudwatch:ap-south-1:111122223333:alarm:results-p99-high" }],
  "targets": { "api-in-az-a": {
    "resourceType": "aws:ec2:instance",
    "resourceTags": { "app": "school-api" },
    "filters": [{ "path": "Placement.AvailabilityZone", "values": ["ap-south-1a"] }],
    "selectionMode": "ALL" } },
  "actions": { "stop-az-a": {
    "actionId": "aws:ec2:stop-instances",
    "parameters": { "startInstancesAfterDuration": "PT10M" },
    "targets": { "Instances": "api-in-az-a" } } }
}

aws fis create-experiment-template --cli-input-json file://lose-az-a.json
aws fis start-experiment --experiment-template-id EXT1a2b3c4d5e6f7

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: निकालाच्या पानाची प्रत्येक dependency लिहून काढा, आणि प्रत्येकासाठी: तिचा timeout, तिचा retry नियम, तिचा breaker, आणि तिचा fallback. मग मुद्दाम एक तोडा (FIS, किंवा staging मध्ये एखादा container थांबवा) आणि p99 आणि error rate पाहा. सराव केलेला बिघाड ही छोटी घटना असते; सराव न केलेला बिघाड म्हणजे outage.

🎓 जत्रा आता तुमची आहे

Up किंवा out → बांधण्याआधी मोजा → प्रत्येक gate वर झेरॉक्स प्रती → live pages काही सेकंदांसाठी cache → कोणतीही खिडकी कोणत्याही पालकाला सेवा देते → रांग वाढली की जास्त खिडक्या → pods आणि ते बसतात ती टेबले → प्रत्येक पालकासाठी एक खिडकी → सूचना फलक → नोंदवहीच्या प्रती → पसरणाऱ्या key ने विभागलेले कप्पे → पेटीतले tokens → एक stopwatch, सभ्य retries, एक breaker switch आणि तीन इमारती. तुम्ही फक्त scaling शिकला नाहीत — तुम्ही निकालाच्या दिवसाचे नियोजन करू शकता, आणि एखादा stall बिघडला तरी जत्रा चालू ठेवू शकता. 📈🛟🎓

⏭️ पुढे

इतर शाळा. Kubernetes शाळा खिडक्या चालवते; Database शाळा नोंदवहीत आणखी खोलवर जाते; API Gateway शाळा API समोरचे पुढचे कार्यालय बांधते; School portal मध्ये बाकी सगळे आहे.

git checkout main
python3 scale/demo.py     # one last run, for fun

🛟 Lesson 13 — Surviving failure: when a stall breaks on results day

📍 You are here: Lesson 13 of 13 · Previous: lesson-12-queues-and-blueprint

📦 What's in this branch

Lessons 01–12, plus the other half of scaling: keeping the fair open when one part breaks. More servers do not help when a service they call is stuck. You learn timeouts, retries with exponential backoff + jitter, the circuit breaker, health checks, graceful degradation, idempotency, the dead-letter queue, multi-AZ, and why availability multiplies along a chain of dependencies. resilience() in scale/demo.py shows each one, with the models in the "surviving failure" part of scale/sim.py.

🧒 Explain like I'm 5

Results day, 10 o'clock. The grades stall at the back of the fair breaks. Its clerk is stuck and does not answer.

Every counter sends parents to the grades stall and waits. And waits. Soon every clerk is waiting, and the whole fair stands still — because of one stall. So Dipika gives each counter a few tools:

⏱️ A stopwatch at the counter. "Wait one second for the grades stall. Then stop waiting and help the next parent." That is a timeout.
🙏 Polite retries. "If it fails, try again — but wait a little first, and wait longer each time. And do not all try at the same moment: each counter adds a random pause." That is backoff with jitter.
🔌 A breaker switch. Aishwarya watches the grades stall. After 5 failures in a row she flips a switch: "Stop sending anyone there for 10 seconds." Then she lets one parent try. If it works, she flips the switch back.
🪧 A smaller notice instead of a closed gate. While the stall is broken, the office pins the last known results on the board, marked "5 minutes old". Not perfect — but the gate stays open.
🔖 A stamp that stops double-charging. Katrina pays the fee. The helper drops her slip twice by mistake. The cashier sees the stamp — "already paid" — and charges her only once.
🗂️ A side tray for bad tickets. One torn ticket fails at every printer. After 3 tries it goes into a side tray, and the printers go on with the other tickets.
🏢🏢🏢 Three buildings. The fair runs in three school buildings. If one loses power, the other two go on.

And one more rule: a parent's visit passes the gate, a counter, the notice board and the register. If each one works 999 times in 1,000, the whole visit works a little less often than any one of them.

🗺️ Diagram

flowchart LR
    u["👪 parent"] --> api["🍳 API"]
    api -->|"⏱️ timeout 1 s"| cb{"🔌 circuit breaker<br/>closed · open · half-open"}
    cb -->|"closed"| dep["📒 grades service"]
    cb -->|"open: fail fast"| fb["🪧 fallback<br/>cached page, '5 minutes old'"]
    dep -.->|"error"| rt["🙏 retry<br/>backoff + jitter<br/>500 at once → 81 per 10 ms"]
    rt -.-> cb
    q["📬 SQS"] -->|"at least once"| wk["🖨️ worker<br/>🔖 idempotency key"]
    q -.->|"after maxReceiveCount"| dlq["🗂️ DLQ"]
    az["🏢🏢🏢 3 AZs<br/>one AZ 99.5% → three 99.999988%"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/scaling/lesson-diagrams.html#l13

❓ What

Failure is normal — at scale, some part is always slow or down: a server, a network link, a database failover, a whole Availability Zone (AZ). The goal is not "nothing fails". The goal is "one failure does not stop everything".
Timeout — the longest a caller waits for an answer. Without one, a stuck service holds a thread, a connection or a Lambda environment for as long as it is stuck. By Little's law (lesson 02), 2,000 req/s × 30 s = 60,000 requests waiting at once. Many HTTP clients have no timeout by default, so set one on every remote call: a connect timeout and a read timeout, inside a total time budget for the request.
Retry — try a failed call again. It helps with short blips (a dropped connection, a 503, a throttle). Retry only safe calls: reads, or writes that are idempotent. Limit the number of attempts (3 is common).
Exponential backoff — wait longer before each retry: 100, 200, 400, 800 ms …, up to a cap. The sick service gets time to recover.
Jitter — a random part in each wait. Full jitter waits a random time between 0 and the backoff number. Without jitter, 500 clients that failed together retry together — a retry storm that can overload the service again.
Circuit breaker — a switch in front of a dependency with three states:
- closed — calls go through; failures are counted;
- open — after N failures in a row, calls fail fast for a while without waiting;
- half-open — after that time, one trial call: success closes the breaker, failure opens it again. It protects both sides: your threads stop waiting, and the sick service gets room.
Health checks — the load balancer (lesson 05) or DNS asks each target "are you OK?" and sends traffic only to healthy ones. Keep the check shallow (is this process able to serve?). A deep check that calls the database can mark every server unhealthy at once when the database is slow.
Graceful degradation — when a part fails, serve a smaller answer instead of an error: the cached results page marked "5 minutes old", the page without photos, no recommendations. Decide in advance what is essential and what can switch off.
Idempotency — doing the same request twice has the same effect as once. The caller sends an idempotency key (for example a payment id); the server remembers keys it has finished and returns the first answer for a repeat. It makes retries and SQS's at-least-once delivery (lesson 12) safe.
Dead-letter queue (DLQ) — a message that fails maxReceiveCount times moves to a separate queue. The workers go on with the rest; a person looks at the bad one and can redrive it back after a fix (lesson 12).
Multi-AZ — copies in different AZs (separate buildings, power and network in one Region). A system with copies in 3 AZs is down only if all 3 are down: 1 − (1 − a)³. This assumes the AZs fail independently — a shared bug or a bad deploy can still take all three down.
Dependency failure — availability multiplies along a chain — if a request needs the API and Redis and the database, it works only when all three work: a₁ × a₂ × a₃. 99.9% × 99.9% × 99.95% ≈ 99.75% — lower than any single part. Every extra hard dependency lowers the total. Fallbacks and copies raise it.

🤔 Why

Because scaling adds parts, and every part can fail. Lessons 01–12 made the fair bigger: more servers, pods, Lambdas, caches, replicas, partitions and queues. With more parts, a failure somewhere is more likely on any given day. Without timeouts, one stuck service fills every thread. Without jitter, retries turn a blip into a storm. Without idempotency, a retry charges a parent twice. The tools in this lesson make a large system also a steady one.

🔧 How (in this repo)

The "surviving failure" part of scale/sim.py holds small models:

call_with_timeout(latency_ms, timeout_ms) returns ('ok', latency) or ('timeout', timeout_ms).
backoff_ms(attempt, base_ms=100, cap_ms=3200, jitter=None) is min(cap, base × 2^attempt), or a random share of it with jitter.
retry_storm(clients, attempts, use_jitter) fails every client at t=0, plays their retries and returns the busiest 10 ms slot — how many retries hit the service at once.
CircuitBreaker(threshold=5, open_s=10) — call(t, dependency_ok) returns ok, failed or fast-fail, and moves between closed, open and half-open.
deliver_at_least_once() + process(deliveries, idempotent) count fees charged.
redrive(messages, poison, max_receives=3) sends a message that always fails to the DLQ after max_receives receives.
parallel(a, copies) = 1 − (1 − a)^copies; serial(*a) multiplies a chain.

resilience() in scale/demo.py runs one example of each. The payments service in the breaker example is down from second 0 to second 29, and the demo calls it once a second for 40 seconds.

🧪 Try it

python3 scale/demo.py resilience
python3 - <<'EOF'
import sys; sys.path.insert(0, "scale"); from sim import call_with_timeout, littles_law, backoff_ms, retry_storm
for lat in (120, 900, 1500, 30000):
    print(f"latency {lat:>5} ms, timeout 1,000 ms →", call_with_timeout(lat, 1000))
for wait in (30, 1, 0.2):
    print(f"2,000 req/s, each caller waits up to {wait:>4} s → {littles_law(2000, wait):>7,.0f} requests stuck at worst")
print("backoff (no jitter):", [backoff_ms(a) for a in range(7)], "ms")
for clients in (100, 500, 2000):
    print(f"{clients:>4} clients, 4 retries → busiest 10 ms: {retry_storm(clients, 4, False):>4} without jitter, {retry_storm(clients, 4, True):>3} with full jitter")
EOF
python3 - <<'EOF'
import sys; sys.path.insert(0, "scale"); from sim import CircuitBreaker, deliver_at_least_once, process, redrive, parallel, serial
for threshold, open_s in ((5, 10), (3, 10), (5, 2), (10, 30)):
    b = CircuitBreaker(threshold, open_s); log = [b.call(t, t >= 30) for t in range(40)]
    print(f"breaker {threshold:>2} failures, open {open_s:>2} s → reached the sick service {log.count('failed'):>2} · failed fast {log.count('fast-fail'):>2} · first ok at {log.index('ok')} s")
d = deliver_at_least_once(["pay-1", "pay-2", "pay-3", "pay-4"], ["pay-2", "pay-4", "pay-4"])
print(f"{len(d)} deliveries of 4 payments → plain {process(d, False)} charges · idempotent {process(d, True)}")
for mr in (1, 3, 5):
    done, dlq, rec = redrive([f"pdf-{i}" for i in range(10)], {"pdf-3", "pdf-7"}, max_receives=mr)
    print(f"maxReceiveCount {mr} → made {len(done)}, DLQ {dlq}, receives {rec}")
for a in (0.99, 0.995):
    print(f"one AZ {a:.1%} → two {parallel(a, 2):.4%} · three {parallel(a, 3):.6%}")
print(f"chain of 3 at 99.9% → {serial(0.999, 0.999, 0.999):.2%} · chain of 6 → {serial(*[0.999] * 6):.2%} · 6 links each on 3 AZs → {serial(*[parallel(0.999, 3)] * 6):.6%}")
EOF
python3 scale/test_scale.py

✅ Verify — what you should see

resilience prints:

── timeouts: the results API usually answers in 120 ms; today it is stuck at 30 s
   latency    120 ms, timeout 1,000 ms → ('ok', 120)
   latency  30000 ms, timeout 1,000 ms → ('timeout', 1000)
   without a timeout every thread waits 30 s — 2,000 req/s × 30 s = 60,000 stuck requests (Little's law again)
── retries: 500 clients fail at once and retry 4 times (backoff 100 → 800 ms)
   no jitter   → busiest 10 ms:  500 retries hit the database together
   full jitter → busiest 10 ms:   81 retries hit the database together
── circuit breaker (5 failures → open for 10 s → one trial call); the payments service is down for 30 s
   calls that reached the sick service: 7 · failed fast without waiting: 27 · state at 40 s: closed
── graceful degradation: the grade service is down → show the results page from Redis, marked '5 minutes old'
   a slower, older page beats an error page; non-essential parts (photos, recommendations) can switch off
── at-least-once: 4 deliveries for 3 payments → plain worker charges 4, idempotent worker charges 3
── a poison message: 9 PDFs made, ['pdf-7'] moved to the DLQ after 3 receives (12 receives in total)
── multi-AZ: one AZ at 99.5% → two AZs 99.9975% · three AZs 99.999988%
   a chain API 99.9% × Redis 99.9% × database 99.95% = 99.75% — every dependency lowers the total
   scaling adds capacity; resilience keeps the fair open when one part of it breaks

Your first snippet prints:

latency   120 ms, timeout 1,000 ms → ('ok', 120)
latency   900 ms, timeout 1,000 ms → ('ok', 900)
latency  1500 ms, timeout 1,000 ms → ('timeout', 1000)
latency 30000 ms, timeout 1,000 ms → ('timeout', 1000)
2,000 req/s, each caller waits up to   30 s →  60,000 requests stuck at worst
2,000 req/s, each caller waits up to    1 s →   2,000 requests stuck at worst
2,000 req/s, each caller waits up to  0.2 s →     400 requests stuck at worst
backoff (no jitter): [100, 200, 400, 800, 1600, 3200, 3200] ms
 100 clients, 4 retries → busiest 10 ms:  100 without jitter,  21 with full jitter
 500 clients, 4 retries → busiest 10 ms:  500 without jitter,  81 with full jitter
2000 clients, 4 retries → busiest 10 ms: 2000 without jitter, 292 with full jitter

Your second snippet prints:

breaker  5 failures, open 10 s → reached the sick service  7 · failed fast 27 · first ok at 34 s
breaker  3 failures, open 10 s → reached the sick service  5 · failed fast 27 · first ok at 32 s
breaker  5 failures, open  2 s → reached the sick service 17 · failed fast 13 · first ok at 30 s
breaker 10 failures, open 30 s → reached the sick service 10 · failed fast 29 · first ok at 39 s
7 deliveries of 4 payments → plain 7 charges · idempotent 4
maxReceiveCount 1 → made 8, DLQ ['pdf-3', 'pdf-7'], receives 10
maxReceiveCount 3 → made 8, DLQ ['pdf-3', 'pdf-7'], receives 14
maxReceiveCount 5 → made 8, DLQ ['pdf-3', 'pdf-7'], receives 18
one AZ 99.0% → two 99.9900% · three 99.999900%
one AZ 99.5% → two 99.9975% · three 99.999988%
chain of 3 at 99.9% → 99.70% · chain of 6 → 99.40% · 6 links each on 3 AZs → 99.999999%

The tests end with ✅ L13 timeouts, jittered retries, a circuit breaker and idempotent workers and 13/13 passed.

🏁 What you just proved

A timeout turns "60,000 stuck requests" into at most 2,000 (1 s) or 400 (0.2 s). Without jitter, every client's retry lands in the same 10 ms — 2,000 at once; full jitter spreads them to 292. The breaker lets only 7 calls reach the sick service during its 30-second outage; the other 23 fail fast. It has a price: the service was back at second 30, and a 10-second open time noticed it only at 34. A short open time (2 s) recovers at once but sends 17 calls to the sick service. You choose between the two. The stamp (idempotency) charges 4 fees for 4 payments, even with 7 deliveries. The DLQ parks the 2 bad messages whatever maxReceiveCount is; a higher count only spends more receives on them — but a count of 1 would also park a good message after a single blip. And the chain loses a little with every link (6 links: 99.40%), while copies in 3 AZs raise it again — as long as the AZs fail independently.

⚠️ Common mistakes

no timeout on an HTTP or database client — the default in many libraries is "wait forever"
a timeout longer than the caller's own timeout — the caller gives up, and the work goes on for nobody
retries at every layer (browser, API, service, SDK): 3 × 3 × 3 = 27 calls for one click
retries without backoff and jitter — a retry storm right when the service is weakest
retrying a payment or a "send email" without an idempotency key — a retry is a second charge
a circuit breaker with no fallback — failing fast is better, but the user still needs an answer
a deep /health check that calls the database, so every server fails its check at once
maxReceiveCount of 1 — one short blip sends a good message to the DLQ
a DLQ that nobody watches — add an alarm on its depth
"multi-AZ" with a single NAT gateway, a single cache node or the database in one AZ

🏭 In production

On a real account — timeouts and retries on every client. Python requests with a connect and a read timeout, full-jitter backoff, and a total time budget:

import random, time, requests

def get_results(cls, attempts=4, base=0.1, cap=3.2, budget=5.0):
    start = time.monotonic()
    for a in range(attempts):
        try:
            r = requests.get(f"https://grades.internal/results/{cls}", timeout=(0.5, 1.0))  # connect, read (s)
            if r.status_code < 500 and r.status_code != 429:
                return r
        except requests.RequestException:
            pass
        wait = random.uniform(0, min(cap, base * 2 ** a))          # full jitter
        if time.monotonic() - start + wait > budget:
            break
        time.sleep(wait)
    return None            # the caller shows the cached page, marked "5 minutes old"

(In requests, the read timeout is the longest gap between bytes, not a limit on the whole answer — the loop's budget is the total limit.)

The AWS SDKs retry for you. Choose the retry mode: standard (exponential backoff with jitter, 3 attempts by default) or adaptive (standard plus a client-side rate limiter that slows down when AWS throttles you). For boto3:

import boto3
from botocore.config import Config
cfg = Config(connect_timeout=1, read_timeout=2, retries={"mode": "standard", "max_attempts": 3})
ddb = boto3.client("dynamodb", config=cfg)

or for any SDK and the AWS CLI: export AWS_RETRY_MODE=standard AWS_MAX_ATTEMPTS=3.

Circuit breakers — a library in the code (Resilience4j for Java, Polly for .NET, opossum for Node.js, pybreaker for Python), or the service mesh does it for every service. Istio: a timeout and retries on the route, and outlier detection that removes a sick pod from the pool for a while:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: { name: grades }
spec:
  hosts: [grades]
  http:
    - route: [{ destination: { host: grades } }]
      timeout: 2s
      retries: { attempts: 2, perTryTimeout: 1s, retryOn: "5xx,reset,connect-failure" }
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata: { name: grades }
spec:
  host: grades
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

Health checks — the ALB target group checks /health (lesson 05). Route 53 health checks watch a whole endpoint, and a failover record sends users to a second one when the first fails:

aws route53 create-health-check --caller-reference results-api-2026-05 \
    --health-check-config '{"Type":"HTTPS","FullyQualifiedDomainName":"api.school.example",
      "ResourcePath":"/health","RequestInterval":30,"FailureThreshold":3}'

SQS DLQ redrive policy + maxReceiveCount (lesson 12), then move parked messages back after the fix:

aws sqs set-queue-attributes --queue-url https://sqs.ap-south-1.amazonaws.com/111122223333/certificates \
    --attributes '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"arn:aws:sqs:ap-south-1:111122223333:certificates-dlq\",\"maxReceiveCount\":\"5\"}"}'
aws sqs start-message-move-task \
    --source-arn arn:aws:sqs:ap-south-1:111122223333:certificates-dlq \
    --destination-arn arn:aws:sqs:ap-south-1:111122223333:certificates

Idempotency keys — Powertools for AWS Lambda (Python) stores each key in DynamoDB and returns the saved answer for a repeat (the table needs a string partition key id, and TTL on the expiration attribute):

from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer, IdempotencyConfig, idempotent)

persistence = DynamoDBPersistenceLayer(table_name="school-idempotency")
config = IdempotencyConfig(event_key_jmespath="powertools_json(body).payment_id",
                           expires_after_seconds=3600)

@idempotent(config=config, persistence_store=persistence)
def handler(event, context):
    charge_fee(event)                     # runs once per payment_id, even if the event comes twice
    return {"statusCode": 200, "body": "paid"}

Multi-AZ for the database and the cache — and test a failover before results day:

aws rds modify-db-instance --db-instance-identifier school-db --multi-az --apply-immediately
aws rds reboot-db-instance --db-instance-identifier school-db --force-failover
aws elasticache modify-replication-group --replication-group-id school-sessions \
    --multi-az-enabled --automatic-failover-enabled --apply-immediately

Chaos testing with AWS Fault Injection Service (FIS) — stop the API instances in one AZ for 10 minutes, and stop the experiment at once if the p99 alarm fires:

{
  "description": "results day drill: lose AZ a",
  "roleArn": "arn:aws:iam::111122223333:role/fis-school",
  "stopConditions": [{ "source": "aws:cloudwatch:alarm",
    "value": "arn:aws:cloudwatch:ap-south-1:111122223333:alarm:results-p99-high" }],
  "targets": { "api-in-az-a": {
    "resourceType": "aws:ec2:instance",
    "resourceTags": { "app": "school-api" },
    "filters": [{ "path": "Placement.AvailabilityZone", "values": ["ap-south-1a"] }],
    "selectionMode": "ALL" } },
  "actions": { "stop-az-a": {
    "actionId": "aws:ec2:stop-instances",
    "parameters": { "startInstancesAfterDuration": "PT10M" },
    "targets": { "Instances": "api-in-az-a" } } }
}

aws fis create-experiment-template --cli-input-json file://lose-az-a.json
aws fis start-experiment --experiment-template-id EXT1a2b3c4d5e6f7

🏭 Why this matters in production: write down every dependency of the results page, and for each one: its timeout, its retry rule, its breaker, and its fallback. Then break one on purpose (FIS, or stop a container in staging) and watch p99 and the error rate. A failure you have practised is a small event; one you have not is an outage.

🎓 The fair is yours

Up or out → count before you build → photocopies at every gate → live pages cached for seconds → any counter serves any parent → more counters when the queue grows → pods and the tables they sit at → a counter for every parent → the notice board → copies of the register → drawers split by a key that spreads → tokens in a box → a stopwatch, polite retries, a breaker switch and three buildings. You didn't just learn scaling — you can plan results day, and keep the fair open when a stall breaks. 📈🛟🎓

⏭️ Next

The other schools. The Kubernetes school runs the counters; the Database school goes deeper on the register; the API Gateway school builds the front office in front of the API; the School portal has the rest.

git checkout main
python3 scale/demo.py     # one last run, for fun

← Previousqueues and blueprint Finished! Take the quiz →check what stuck

This page is the lesson's README from the lesson-13-surviving-failure branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.