🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🛟 धडा 13 — Design मधूनच resilience: प्रत्येक बाणासाठी एक योजना

📍 तुम्ही इथे आहात: 18 पैकी धडा 13 · पुढील: lesson-14-security

📦 या ब्रँचमध्ये काय आहे

धडे 01–12, आणि कार्यालय आकृतीवरील प्रत्येक box ला विचारते तो पहिला प्रश्न: तो अपयशी झाला तर काय होते? तुम्ही शिकता timeout budgets, backoff आणि jitter सह retries, circuit breaker, bulkheads, graceful degradation, आणि multi-AZ चे गणित. हा धडा नियोजनाच्या नजरेतून आहे. Scaling शाळा, धडा 13 प्रत्येक साधन simulations सह खोलात बांधतो.

design/designs.py — call_chain(hops, ms_each, availability_each)
design/demo.py — python3 design/demo.py resilience
design/test_design.py — check L13

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

दीपिका सूचना-फलक app ची आकृती भिंतीवर लावते. त्यात boxes आणि बाण आहेत. ती कतरिनाला एक लाल पेन देते. "प्रत्येक बाणासाठी चार गोष्टी लिही."

⏱️ आपण किती वेळ थांबायचे? पूर्ण page 800 ms मध्ये तयार व्हायला हवे. म्हणून login तपासणीकडच्या बाणाला 50 ms, सूचना-नोंदवहीकडच्या बाणाला 150 ms, photo service कडच्या बाणाला 300 ms मिळतात. या सगळ्या प्रतीक्षा 800 च्या आत बसायला हव्यात.
🙏 आपण पुन्हा विचारायचे का? एक-दोनदा, वाढत्या विरामाने आणि थोड्या random जादा वेळाने, म्हणजे सगळे callers एकाच क्षणी पुन्हा विचारत नाहीत.
🔌 विचारणे कधी थांबवायचे? सलग अनेक अपयशांनंतर एक switch उघडतो: "10 seconds photo service ला call करू नका." मग एक चाचणी call.
🪧 त्याऐवजी काय दाखवायचे? Photo service बंद आहे? सूचना photos शिवाय दाखवा. Page तरीही उघडते.

ऐश्वर्या पाचवी कल्पना जोडते: वेगळ्या खोल्या. Photo मदतनीस त्यांच्या स्वतःच्या खोलीत स्वतःच्या खुर्च्यांवर काम करतात. ते सगळे अडकले तरी सूचना-मदतनीसांच्या खुर्च्या त्यांच्याकडेच राहतात. हेच bulkhead, जहाजाच्या आतल्या भिंतींसारखे.

🗺️ आकृती

flowchart LR
    u["👪 parent<br/>page budget 800 ms"] --> api["🧩 notice API"]
    api -->|"⏱️ 50 ms"| auth["🔑 auth"]
    api -->|"⏱️ 150 ms"| db["🗄️ notices DB"]
    api -->|"⏱️ 300 ms · 🔌 breaker"| ph["🖼️ photo service"]
    ph -.->|"down"| fb["🪧 degrade:<br/>notices without photos"]
    subgraph bh["🚪 bulkheads"]
      p1["pool for DB calls"]
      p2["pool for photo calls"]
    end
    az["🏢🏢🏢 3 AZs<br/>99.5% each → 99.99999%"]

🗺️ काढलेली आवृत्ती + एक lab: https://school-edh.pages.dev/system-design/lesson-diagrams.html#l13

❓ काय

Timeout budget — user च्या मर्यादेपासून सुरुवात करा (page: 800 ms) आणि प्रत्येक dependency ला त्यातला एक तुकडा द्या. हे तुकडे, अधिक तुमचे स्वतःचे काम, आत बसायला हवे. उरलेला वेळ request सोबत खाली पाठवा (deadline header, किंवा gRPC deadline) म्हणजे caller ने सोडून दिल्यानंतर खोलवरची service काम करत राहत नाही.
Retry + backoff + jitter — फक्त सुरक्षित calls retry करा (reads, किंवा idempotency key असलेले writes — धडा 11). प्रत्येक वेळी जास्त थांबा (100, 200, 400 ms …, एका मर्यादेपर्यंत), आणि एक random भाग (jitter) जोडा म्हणजे एकत्र अपयशी झालेले clients एकत्र retry करत नाहीत. एकाच थरावर retry करा: प्रत्येकी 3 वेळा प्रयत्न करणारे तीन थर 27 calls करतात.
Circuit breaker — closed (calls जातात, अपयश मोजले जाते) → N अपयशांनंतर open (पटकन fail, थांबणे नाही) → एका विरामानंतर half-open (एक चाचणी call). हा तुमचे threads वाचवतो आणि आजारी service ला सावरायला जागा देतो.
Bulkhead — प्रत्येक dependency किंवा प्रत्येक प्रकारच्या customer साठी वेगळे pools (threads, connections, queues, अगदी servers). अडकलेली dependency फक्त तिचाच pool संपवू शकते.
Graceful degradation — launch आधी ठरवा की काय अत्यावश्यक आहे (सूचनेचा मजकूर) आणि काय बंद होऊ शकते (photos, "312 जणांनी वाचली", शिफारसी). Error ऐवजी छोटे उत्तर द्या. "5 मिनिटे जुनी" अशी खूण असलेली cached प्रत अनेकदा चालते.
Multi-AZ चे गणित — n Availability Zones पैकी प्रत्येकात एक प्रत असेल, तर ती सगळ्या बंद असतील तेव्हाच बंद पडते: 1 − (1 − a)ⁿ, जर zones स्वतंत्रपणे अपयशी होत असतील तर. सामाईक bug किंवा वाईट deploy स्वतंत्र नसतो.
साखळ्या availability कमी करतात — ओळीतील hard dependencies चा गुणाकार होतो: a × a × a. प्रत्येक जास्तीची hard dependency एकूण availability कमी करते (धडा 09). प्रती आणि fallbacks ती वाढवतात.

🤔 का

कारण प्रत्येक बाणाला अपयशाची योजना मिळेपर्यंत आकृती अर्धीच असते. Timeout नसेल तर एक अडकलेली service प्रत्येक thread धरून ठेवते. Jitter शिवायचे retries छोट्या अडथळ्याचे वादळ करतात. Fallback नसलेले design "photos नाहीत" चे "app नाही" करते. हे design चे निर्णय आहेत — outage च्या वेळी जोडण्यापेक्षा ते योजनेवर लिहिणे खूप स्वस्त असते.

🔧 कसे (या repo मध्ये)

design/designs.py मधील call_chain(hops, ms_each, availability_each) calls च्या एका ओळीची latency (hops × ms_each) आणि availability (availability_each ** hops) परत करतो. design/demo.py मधील resilience() एका page चे timeout budget आणि multi-AZ बेरीज 1 − (1 − 0.995)³ छापतो. Snippet budget ची बेरीज करतो, साखळी वाढवतो, आणि थरांमधील retries मोजतो.

🧪 करून पाहा

python3 design/demo.py resilience
python3 - <<'EOF'
import sys; sys.path.insert(0, "design"); from designs import call_chain
budget, deps = 800, {"auth": 50, "notices DB": 150, "photo service": 300}
print("timeouts add up to", sum(deps.values()), "ms of an", budget, "ms budget → left for our own work:", budget - sum(deps.values()), "ms")
for hops in (1, 3, 5):
    c = call_chain(hops, 20, 0.999); print(f"{hops} hard dependencies in a line → {c['latency_ms']:>3} ms, {c['availability']:.3%}")
for a in (0.99, 0.995):
    print(f"one AZ {a:.1%} →", " · ".join(f"{n} AZs {1 - (1 - a) ** n:.6%}" for n in (1, 2, 3)))
for tries in (1, 2, 3):
    print(f"3 layers each trying {tries}x → up to {tries ** 3} calls reach the sick service for one click")
EOF

✅ तपासा — तुम्हाला काय दिसायला हवे

resilience हे छापतो:

── the budget for one page is 800 ms; each dependency gets a timeout inside it
   auth           timeout  50 ms
   notices DB     timeout 150 ms
   photo service  timeout 300 ms
   photo service down → show notices without photos (degrade), breaker open, retries with jitter
   one AZ 99.5% → three AZs 99.99999% · deeper dive: the Scaling school, lesson 13

तुमचा snippet हे छापतो:

timeouts add up to 500 ms of an 800 ms budget → left for our own work: 300 ms
1 hard dependencies in a line →  20 ms, 99.900%
3 hard dependencies in a line →  60 ms, 99.700%
5 hard dependencies in a line → 100 ms, 99.501%
one AZ 99.0% → 1 AZs 99.000000% · 2 AZs 99.990000% · 3 AZs 99.999900%
one AZ 99.5% → 1 AZs 99.500000% · 2 AZs 99.997500% · 3 AZs 99.999988%
3 layers each trying 1x → up to 1 calls reach the sick service for one click
3 layers each trying 2x → up to 8 calls reach the sick service for one click
3 layers each trying 3x → up to 27 calls reach the sick service for one click

Tests मध्ये ✅ L13 three copies beat one असते.

🏁 तुम्ही आत्ताच काय सिद्ध केले

तीन timeouts 800 ms च्या page पैकी 500 ms वापरतात, API च्या स्वतःच्या कामासाठी 300 ms उरतात — आणि photo call च्या retry साठी जागाच उरत नाही (आणखी 300 ms), म्हणून त्या call ला कोणताही retry मिळत नाही. 99.9% च्या पाच hard dependencies 99.501% पर्यंत घसरतात. 99.5% चा एक zone दोन zones सह 99.9975% आणि तीन सह 99.999988% होतो (demo ते गोल करून 99.99999% छापतो) — फक्त zones स्वतंत्रपणे अपयशी होत असेपर्यंत. आणि तीन थरांवर "फक्त 3 retries" म्हणजे आधीच आजारी असलेल्या service कडे 27 पर्यंत calls.

⚠️ नेहमीच्या चुका

user च्या budget वरून ठरवण्याऐवजी एक-एक करून निवडलेले timeouts ("30 s सुरक्षित आहे")
callee चा timeout त्याच्या caller पेक्षा मोठा — caller सोडून देतो, काम कोणासाठीच नसताना चालू राहते
प्रत्येक थरावर retries, आणि non-idempotent writes चे retries
fallback नसलेला breaker — पटकन fail झाले तरी पालकाला काहीतरी उत्तर हवेच
प्रत्येक dependency साठी एकच सामाईक connection pool — हळू dependency सगळी connections घेते
एकच NAT gateway, एकच cache node किंवा single-AZ database असलेले "multi-AZ"
कधीच तपासली न गेलेली अपयशाची योजना — staging मध्ये मुद्दाम एक dependency मोडा

🏭 प्रत्यक्ष वापरात

खऱ्या account वर — प्रत्येक call सोबत deadline खाली पाठवा, आणि खूप कमी वेळ उरला असेल तर लवकर थांबा:

import time, requests

def call(url, deadline):
    left = deadline - time.monotonic()
    if left < 0.05:                              # under 50 ms left: do not start
        raise TimeoutError("budget spent")
    return requests.get(url, timeout=(0.1, min(left, 0.3)),          # connect, read (s)
                        headers={"X-Request-Deadline": str(round(left * 1000))})

deadline = time.monotonic() + 0.8                # the page budget: 800 ms

Code मधला bulkhead — एका वेळी जास्तीत जास्त 20 photo calls, म्हणजे photo service कधीच सगळे workers घेऊ शकत नाही:

import threading
photo_slots = threading.BoundedSemaphore(20)

def photos_for(notice_id, deadline):
    if not photo_slots.acquire(timeout=0.01):
        return []                                # bulkhead full → degrade: no photos
    try:
        return call(f"https://photos.internal/notices/{notice_id}", deadline).json()
    except Exception:
        return []                                # photo service slow or down → degrade
    finally:
        photo_slots.release()

Backoff आणि jitter सह AWS SDK retries, आणि Multi-AZ database:

export AWS_RETRY_MODE=standard AWS_MAX_ATTEMPTS=3
aws rds modify-db-instance --db-instance-identifier notices-db --multi-az --apply-immediately

Service mesh मधील circuit breakers, outlier detection, DLQs, health checks आणि AWS FIS सह chaos tests साठी, पाहा Scaling शाळा, धडा 13.

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: प्रत्येक design मध्ये एक "failure" table जोडा: प्रत्येक बाणासाठी एक row, त्याचा timeout, retry नियम, breaker, bulkhead आणि fallback सह. रिकामा कप्पा म्हणजे भविष्यातला outage.

⏭️ पुढे

भाग अपघाताने अपयशी होतात. काही लोक ते मुद्दाम मोडण्याचा प्रयत्न करतात. Design मधूनच security.

git checkout lesson-14-security

🛟 Lesson 13 — Resilience by design: a plan for every arrow

📍 You are here: Lesson 13 of 18 · Next: lesson-14-security

📦 What's in this branch

Lessons 01–12, plus the first question the office asks of every box on the drawing: what happens when it fails? You learn timeout budgets, retries with backoff and jitter, the circuit breaker, bulkheads, graceful degradation, and the multi-AZ maths. This lesson is the planning view. The Scaling school, lesson 13 builds each tool in depth, with simulations.

design/designs.py — call_chain(hops, ms_each, availability_each)
design/demo.py — python3 design/demo.py resilience
design/test_design.py — check L13

🧒 Explain like I'm 5

Dipika pins the drawing of the notice-board app to the wall. It has boxes and arrows. She gives Katrina a red pen. "For every arrow, write four things."

⏱️ How long do we wait? The whole page must be ready in 800 ms. So the arrow to the login check gets 50 ms, the arrow to the notices book gets 150 ms, the arrow to the photo service gets 300 ms. The waits must fit inside the 800.
🙏 Do we ask again? Once or twice, with a growing pause and a little random extra, so that all callers do not ask again at the same moment.
🔌 When do we stop asking? After many failures in a row, a switch opens: "Do not call the photo service for 10 seconds." Then one test call.
🪧 What do we show instead? The photo service is down? Show the notices without photos. The page still opens.

Aishwarya adds a fifth idea: separate rooms. The photo helpers work in their own room with their own chairs. If they all get stuck, the notice helpers still have their chairs. That is a bulkhead, like the walls inside a ship.

🗺️ Diagram

flowchart LR
    u["👪 parent<br/>page budget 800 ms"] --> api["🧩 notice API"]
    api -->|"⏱️ 50 ms"| auth["🔑 auth"]
    api -->|"⏱️ 150 ms"| db["🗄️ notices DB"]
    api -->|"⏱️ 300 ms · 🔌 breaker"| ph["🖼️ photo service"]
    ph -.->|"down"| fb["🪧 degrade:<br/>notices without photos"]
    subgraph bh["🚪 bulkheads"]
      p1["pool for DB calls"]
      p2["pool for photo calls"]
    end
    az["🏢🏢🏢 3 AZs<br/>99.5% each → 99.99999%"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/system-design/lesson-diagrams.html#l13

❓ What

Timeout budget — start from the user's limit (the page: 800 ms) and give each dependency a slice of it. The slices, plus your own work, must fit. Pass the remaining time down with the request (a deadline header, or a gRPC deadline) so a deeper service does not work on after the caller has given up.
Retry + backoff + jitter — retry only safe calls (reads, or writes with an idempotency key — lesson 11). Wait longer each time (100, 200, 400 ms …, up to a cap), and add a random part (jitter) so clients that failed together do not retry together. Retry at one layer only: three layers that each try 3 times make 27 calls.
Circuit breaker — closed (calls pass, failures counted) → open after N failures (fail fast, no waiting) → half-open after a pause (one trial call). It protects your threads and gives the sick service room to recover.
Bulkhead — separate pools (threads, connections, queues, even servers) per dependency or per kind of customer. A stuck dependency can use up its pool only.
Graceful degradation — decide before launch what is essential (the notice text) and what can switch off (photos, "read by 312", recommendations). Serve a smaller answer instead of an error. A cached copy marked "5 minutes old" is often fine.
Multi-AZ maths — a copy in each of n Availability Zones is down only when all are down: 1 − (1 − a)ⁿ, if the zones fail independently. A shared bug or a bad deploy is not independent.
Chains lower availability — hard dependencies in a line multiply: a × a × a. Every extra hard dependency lowers the total (lesson 09). Copies and fallbacks raise it.

🤔 Why

Because the drawing is only half done until every arrow has a failure plan. A missing timeout lets one stuck service hold every thread. Retries without jitter turn a short blip into a storm. A design with no fallback turns "no photos" into "no app". These are design decisions — it is much cheaper to write them on the plan than to add them during an outage.

🔧 How (in this repo)

call_chain(hops, ms_each, availability_each) in design/designs.py returns the latency (hops × ms_each) and availability (availability_each ** hops) of a line of calls. resilience() in design/demo.py prints the timeout budget for one page and the multi-AZ sum 1 − (1 − 0.995)³. The snippet adds up the budget, grows the chain, and counts retries across layers.

🧪 Try it

python3 design/demo.py resilience
python3 - <<'EOF'
import sys; sys.path.insert(0, "design"); from designs import call_chain
budget, deps = 800, {"auth": 50, "notices DB": 150, "photo service": 300}
print("timeouts add up to", sum(deps.values()), "ms of an", budget, "ms budget → left for our own work:", budget - sum(deps.values()), "ms")
for hops in (1, 3, 5):
    c = call_chain(hops, 20, 0.999); print(f"{hops} hard dependencies in a line → {c['latency_ms']:>3} ms, {c['availability']:.3%}")
for a in (0.99, 0.995):
    print(f"one AZ {a:.1%} →", " · ".join(f"{n} AZs {1 - (1 - a) ** n:.6%}" for n in (1, 2, 3)))
for tries in (1, 2, 3):
    print(f"3 layers each trying {tries}x → up to {tries ** 3} calls reach the sick service for one click")
EOF

✅ Verify — what you should see

resilience prints:

── the budget for one page is 800 ms; each dependency gets a timeout inside it
   auth           timeout  50 ms
   notices DB     timeout 150 ms
   photo service  timeout 300 ms
   photo service down → show notices without photos (degrade), breaker open, retries with jitter
   one AZ 99.5% → three AZs 99.99999% · deeper dive: the Scaling school, lesson 13

Your snippet prints:

timeouts add up to 500 ms of an 800 ms budget → left for our own work: 300 ms
1 hard dependencies in a line →  20 ms, 99.900%
3 hard dependencies in a line →  60 ms, 99.700%
5 hard dependencies in a line → 100 ms, 99.501%
one AZ 99.0% → 1 AZs 99.000000% · 2 AZs 99.990000% · 3 AZs 99.999900%
one AZ 99.5% → 1 AZs 99.500000% · 2 AZs 99.997500% · 3 AZs 99.999988%
3 layers each trying 1x → up to 1 calls reach the sick service for one click
3 layers each trying 2x → up to 8 calls reach the sick service for one click
3 layers each trying 3x → up to 27 calls reach the sick service for one click

The tests include ✅ L13 three copies beat one.

🏁 What you just proved

The three timeouts use 500 ms of the 800 ms page, leaving 300 ms for the API's own work — and no room for a retry of the photo call (300 ms more), so that call gets no retry. Five hard dependencies at 99.9% fall to 99.501%. One zone at 99.5% becomes 99.9975% with two and 99.999988% with three (the demo prints it rounded to 99.99999%) — only while the zones fail independently. And "just 3 retries" at three layers is up to 27 calls into a service that is already sick.

⚠️ Common mistakes

timeouts chosen one by one ("30 s is safe") instead of from the user's budget
a callee's timeout longer than its caller's — the caller gives up, the work goes on for nobody
retries at every layer, and retries of non-idempotent writes
a breaker with no fallback — failing fast still needs an answer for the parent
one shared connection pool for every dependency — the slow one takes all connections
"multi-AZ" with one NAT gateway, one cache node or a single-AZ database
a failure plan that was never tested — break one dependency on purpose in staging

🏭 In production

On a real account — pass the deadline down with each call, and stop early when too little time is left:

import time, requests

def call(url, deadline):
    left = deadline - time.monotonic()
    if left < 0.05:                              # under 50 ms left: do not start
        raise TimeoutError("budget spent")
    return requests.get(url, timeout=(0.1, min(left, 0.3)),          # connect, read (s)
                        headers={"X-Request-Deadline": str(round(left * 1000))})

deadline = time.monotonic() + 0.8                # the page budget: 800 ms

A bulkhead in code — at most 20 photo calls at once, so the photo service can never take every worker:

import threading
photo_slots = threading.BoundedSemaphore(20)

def photos_for(notice_id, deadline):
    if not photo_slots.acquire(timeout=0.01):
        return []                                # bulkhead full → degrade: no photos
    try:
        return call(f"https://photos.internal/notices/{notice_id}", deadline).json()
    except Exception:
        return []                                # photo service slow or down → degrade
    finally:
        photo_slots.release()

AWS SDK retries with backoff and jitter, and a Multi-AZ database:

export AWS_RETRY_MODE=standard AWS_MAX_ATTEMPTS=3
aws rds modify-db-instance --db-instance-identifier notices-db --multi-az --apply-immediately

For circuit breakers in a service mesh, outlier detection, DLQs, health checks and chaos tests with AWS FIS, see the Scaling school, lesson 13.

🏭 Why this matters in production: add a "failure" table to every design: one row per arrow, with its timeout, retry rule, breaker, bulkhead and fallback. An empty cell is a future outage.

⏭️ Next

Parts fail by accident. Some people try to break them on purpose. Security by design.

git checkout lesson-14-security

← Previousrate limiting Next →security

This page is the lesson's README from the lesson-13-resilience branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.