🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🚦 धडा 10 — Overload: priority नुसार shed करा, नीट degrade करा, retries ना budget द्या

📍 तुम्ही इथे आहात: 12 पैकी धडा 10 · मागे: lesson-09-reliability-math · पुढे: lesson-11-chaos-engineering

📦 या ब्रँचमध्ये काय आहे

धडा 09, अधिक इमारत करू शकेल त्यापेक्षा जास्त काम आल्यावर काय करायचे ते. सगळे काम सारखेच महत्त्वाचे नसते: सर्वात कमी महत्त्वाचे आधी shed करा, प्रत्येक request स्वस्त करण्यासाठी features degrade करा, आणि retry budget ने clients ना overload चे वादळ बनवण्यापासून थांबवा. (मर्यादित queues आणि backpressure Distributed Systems school, धडा 12 मध्ये आहेत; backoff आणि circuit breakers Scaling school, धडा 13 मध्ये.) sre/design.py मधील Shedder आणि retry_load आणि sre/demo.py मधील overload().

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

शाळेचे उपाहारगृह मिनिटाला 1,000 ताटे वाढू शकते. आज रांगेत 1,500 जण आहेत. 🍽️

कोणताही plan नाही: सगळे सारखेच थांबतात. तीनपैकी फक्त दोघांना जेवण मिळते — आणि दहा मिनिटांत परीक्षा असलेल्या विद्यार्थिनींना (critical) उपाशी राहण्याची शक्यता, दुसऱ्यांदा गोड खायला आलेल्या पाहुण्यांइतकीच असते.

पहिला plan — priority: आधी परीक्षेच्या विद्यार्थिनी, मग नेहमीची जेवणाची रांग, मग दुसऱ्यांदा गोड. परीक्षेच्या प्रत्येक विद्यार्थिनीला आणि प्रत्येक नेहमीच्या जेवणाला वाढले जाते; दुसऱ्यांदा गोडाच्या 600 पैकी फक्त 100.

दुसरा plan — प्रत्येक ताट स्वस्त करा: नेहमीच्या रांगेसाठी आणि गोडासाठी सजावट वगळा. प्रत्येक ताटाला कमी वेळ लागतो, म्हणून 100 ऐवजी 357 गोड वाढले जातात.

आणि retry चा नियम: परत पाठवलेल्या सगळ्यांनी पुन्हा येऊन प्रत्येकी तीनदा घुसू नये — त्यामुळे रांग दुप्पट लांब होते. फक्त थोड्यांनाच पुन्हा प्रयत्न करता येतो.

🗺️ आकृती

flowchart LR
    in["📥 1,500 req/s offered<br/>capacity 1,000"] --> f["no priorities<br/>critical 267 of 400 ✗"]
    in --> p["shed by priority<br/>critical 400 · normal 500 · sheddable 100"]
    p --> g["+ degrade (cost 0.7)<br/>sheddable 357"]
    in --> r["3 retries each<br/>3,000 attempts/s"]
    in --> b["10% retry budget<br/>1,650 attempts/s"]

🗺️ काढलेली आकृती + एक lab: https://school-edh.pages.dev/sre/lesson-diagrams.html#l10

❓ काय

Overload — capacity पेक्षा जास्त आलेला load. Plan नसेल तर प्रत्येक request हळू होते, अनेक requests client च्या timeout नंतरही थांबलेल्या राहतात, आणि server असे काम करतो जे कोणालाच मिळत नाही — उपयुक्त throughput (goodput) capacity पेक्षाही खाली जाऊ शकतो. Lab मधील "no priorities" ओळ ही याचीच आशावादी आवृत्ती आहे.
Load shedding — बाकीच्या कामाचे रक्षण करण्यासाठी काही काम मुद्दाम, लवकर आणि स्वस्तात (जलद 429/503) नाकारणे.
Criticality — प्रत्येक request सोबत जाणारे priority चे लेबल. Google SRE book चार सांगते: CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE. खालून वर shed करा. लेबले overload च्या आधी, product owners सोबत ठरवा.
Graceful degradation — कामाची स्वस्त आवृत्ती करा: thumbnails वगळा, थोडे जुने cached वेळापत्रक द्या, recommendations बंद करा. Users ना कमी मिळते, पण काहीच नाही असे नाही. बहुतेक वेळा flag (धडा 06) ने किंवा load नुसार आपोआप चालवले जाते.
Retry amplification — प्रत्येक अयशस्वी request 3 वेळा retry झाली आणि retries सुद्धा अयशस्वी झाल्या, तर server ला तो हाताळू न शकलेल्या load च्या 4 × पर्यंत load दिसतो. अनेक layers वरील retries गुणाकाराने वाढतात (एका click साठी 3 × 3 × 3 = 27 प्रयत्न).
Retry budget — नेहमीच्या traffic च्या प्रमाणात retries ना मर्यादा घाला. SRE book चा pattern: प्रत्येक request वर जास्तीत जास्त 3 प्रयत्न, आणि प्रत्येक client चे retries त्याच्या requests च्या जास्तीत जास्त 10%. gRPC चे retry throttling आणि Envoy चे retry budget हीच कल्पना राबवतात.
फक्त सुरक्षित तेच retry करा — idempotent requests, "नंतर प्रयत्न करा" असा अर्थ असलेल्या errors वर, backoff आणि jitter सह.

🤔 का

कारण overload हीच ती वेळ जेव्हा छोटी अडचण संपूर्ण outage बनते: हळू dependency (धडा 09) capacity कमी करते, queues वाढतात, clients retry करतात, load दुप्पट होतो, आणि users चा प्रत्येक वर्ग एकत्र अयशस्वी होतो. Priority नुसार shedding सर्वात महत्त्वाचे काम पूर्ण ठेवते, degradation capacity ताणते, आणि retry budget clients ना परिस्थिती आणखी बिघडवण्यापासून थांबवते.

🔧 कसे (या repo मध्ये)

sre/design.py मधील Shedder(capacity) आलेला load priority च्या क्रमाने (class, requests/s, cost per request) असा घेते. fifo() प्रत्येक class ला तोच वाटा देते; by_priority() capacity वरच्या class पासून खाली भरते. retry_load(offered, capacity, max_retries, budget) सेकंदाला किती प्रयत्न होतात ते परत देते, जेव्हा नाकारलेली प्रत्येक request retry होते आणि overload टिकेपर्यंत प्रत्येक retry सुद्धा नाकारला जातो (सर्वात वाईट स्थिती).

🧪 करून पाहा

python3 sre/demo.py overload
python3 - <<'EOF'
import sys; sys.path.insert(0, "sre"); from design import Shedder, retry_load
s = Shedder(1000)
for crit in (400, 900, 1200):
    print(f"critical {crit:>4}/s →", s.by_priority([("critical", crit, 1.0), ("normal", 500, 1.0), ("sheddable", 600, 1.0)]))
for offered in (1000, 1200, 2000):
    print(f"offered {offered}/s → attempts with 3 retries {retry_load(offered, 1000, 3):>5,.0f} · with a 10% budget {retry_load(offered, 1000, 3, 0.1):>5,.0f}")
EOF

✅ तपासा — तुम्हाला काय दिसायला हवे

overload हे छापते:

   no priorities (everyone gets 1000/1500): {'critical': 267, 'normal': 333, 'sheddable': 400}
   shed by priority:                        {'critical': 400, 'normal': 500, 'sheddable': 100}
   + graceful degradation (normal and sheddable skip photo thumbnails: cost 0.7 each): {'critical': 400, 'normal': 500, 'sheddable': 357}
   3 retries per request           → 3,000 attempts/s arrive at a server that can do 1,000
   3 retries + a 10% retry budget  → 1,650 attempts/s arrive at a server that can do 1,000

तुमचा snippet हे छापतो:

critical  400/s → {'critical': 400, 'normal': 500, 'sheddable': 100}
critical  900/s → {'critical': 900, 'normal': 100, 'sheddable': 0}
critical 1200/s → {'critical': 1000, 'normal': 0, 'sheddable': 0}
offered 1000/s → attempts with 3 retries 1,000 · with a 10% budget 1,000
offered 1200/s → attempts with 3 retries 1,800 · with a 10% budget 1,320
offered 2000/s → attempts with 3 retries 5,000 · with a 10% budget 2,200

🏁 तुम्ही आत्ताच काय सिद्ध केले

Priorities नसताना एक तृतीयांश critical requests अयशस्वी झाल्या (400 पैकी 267 च मिळाल्या), तर 400 दुसऱ्यांदा-गोडाच्या requests ना सेवा मिळाली. Priority नुसार shedding ने प्रत्येक critical आणि normal request टिकवली, आणि degradation ने त्याच capacity वर sheddable class 100 वरून 357 वर नेला. पण फक्त critical traffic च 1,200/s असेल, तर shedding मदत करू शकत नाही — ती capacity ची अडचण आहे (धडा 07). आणि retries: 2× overload वर प्रत्येकी 3 retries मुळे सेकंदाला 2,000 चे 5,000 प्रयत्न होतात; 10% budget ते 2,200 वर रोखते.

⚠️ नेहमीच्या चुका

requests वर priority नाही, म्हणून एखादा batch job logins ना उपाशी ठेवू शकतो
दारातच नाही तर उशिरा shed करणे — अर्धे काम केल्यानंतर
प्रत्येक layer वर retries, प्रत्येक "फक्त 3"
timeout वर non-idempotent requests (payment) retry करणे
कधीच test न केलेले degradation चे मार्ग, जे पहिल्यांदा गरज पडल्यावरच अयशस्वी होतात
overload साठी 500 परत देणे, म्हणजे clients ना "busy" आणि "broken" मधला फरक कळत नाही

🏭 प्रत्यक्ष वापरात

Service config मधील gRPC retry throttling: प्रत्येक अपयश एक token खर्च करते, प्रत्येक यश tokenRatio परत मिळवते; tokens अर्ध्याखाली गेले की retries थांबतात. खऱ्या account वर:

{
  "methodConfig": [{
    "name": [{"service": "timetable.Timetable"}],
    "retryPolicy": {"maxAttempts": 3, "initialBackoff": "0.1s", "maxBackoff": "1s",
                    "backoffMultiplier": 2, "retryableStatusCodes": ["UNAVAILABLE"]}
  }],
  "retryThrottling": {"maxTokens": 10, "tokenRatio": 0.1}
}

Envoy retry budget — चालू retries cluster कडे जाणाऱ्या चालू requests च्या जास्तीत जास्त एका टक्केवारीइतक्याच असू शकतात:

clusters:
  - name: timetable
    circuit_breakers:
      thresholds:
        - priority: DEFAULT
          retry_budget:
            budget_percent: {value: 10.0}
            min_retry_concurrency: 3

Kubernetes वर API server स्वतः API Priority and Fairness ने priority नुसार shed करतो (FlowSchema आणि PriorityLevelConfiguration objects) आणि एखाद्या level च्या queues भरल्या की 429 उत्तर देतो.

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे आहे: तुमच्या पहिल्या पाच request प्रकारांना critical किंवा sheddable असे लेबल लावा, आणि retry होणारी प्रत्येक जागा शोधा. मग 1.5 × capacity वर load-test करा आणि कोणता class आधी अयशस्वी होतो ते तपासा. ते logins असतील, तर खरा overload येण्याआधी क्रम दुरुस्त करा.

⏭️ पुढे

अपयशासाठीचे designs तुमच्याकडे कागदावर आहेत. ते चालतात का? हे शोधण्यासाठी पुढचा धडा काळजीपूर्वक, मुद्दाम गोष्टी बिघडवतो.

git checkout lesson-11-chaos-engineering

🚦 Lesson 10 — Overload: shed by priority, degrade gracefully, budget the retries

📍 You are here: Lesson 10 of 12 · Previous: lesson-09-reliability-math · Next: lesson-11-chaos-engineering

📦 What's in this branch

Lesson 09, plus what to do when more work arrives than the building can do. Not everything is equally important: shed the least important first, degrade features to make each request cheaper, and stop clients from turning an overload into a storm with a retry budget. (Bounded queues and backpressure are in the Distributed Systems school, lesson 12; backoff and circuit breakers in the Scaling school, lesson 13.) Shedder and retry_load in sre/design.py and overload() in sre/demo.py.

🧒 Explain like I'm 5

The school canteen can serve 1,000 plates a minute. Today 1,500 people are in line. 🍽️

No plan: everybody waits the same. Only two in three get fed — and the pupils who have an exam in ten minutes (the critical ones) are just as likely to go hungry as the visitors who came for a second dessert.

Plan one — priority: exam pupils first, then the normal lunch line, then second desserts. Every exam pupil and every normal lunch is served; only 100 of 600 second desserts.

Plan two — make each plate cheaper: for the normal line and the desserts, skip the fancy garnish. Each plate takes less time, so 357 desserts get served instead of 100.

And the retry rule: people turned away must not all come back and push in again three times each — that makes the line twice as long. Only a few may try again.

🗺️ Diagram

flowchart LR
    in["📥 1,500 req/s offered<br/>capacity 1,000"] --> f["no priorities<br/>critical 267 of 400 ✗"]
    in --> p["shed by priority<br/>critical 400 · normal 500 · sheddable 100"]
    p --> g["+ degrade (cost 0.7)<br/>sheddable 357"]
    in --> r["3 retries each<br/>3,000 attempts/s"]
    in --> b["10% retry budget<br/>1,650 attempts/s"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/sre/lesson-diagrams.html#l10

❓ What

Overload — offered load above capacity. Without a plan, every request slows down, many wait past their client's timeout, and the server does work nobody receives — useful throughput (goodput) can fall below capacity. The lab's "no priorities" line is the optimistic version of this.
Load shedding — refusing some work on purpose, early and cheaply (a fast 429/503), to protect the rest.
Criticality — a priority label that travels with every request. The Google SRE book describes four: CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE. Shed from the bottom up. Decide the labels before the overload, with the product owners.
Graceful degradation — do a cheaper version of the work: skip thumbnails, serve a slightly stale cached timetable, turn off recommendations. Users get less, not nothing. Often driven by a flag (lesson 06) or automatically by load.
Retry amplification — if every failed request is retried 3 times and the retries also fail, the server sees up to 4 × the load it could not handle. Retries at several layers multiply (3 × 3 × 3 = 27 attempts for one click).
Retry budget — cap retries as a share of normal traffic. The SRE book's pattern: at most 3 attempts per request, and each client's retries at most 10% of its requests. gRPC's retry throttling and Envoy's retry budget implement the same idea.
Retry only what is safe — idempotent requests, on errors that mean "try later", with backoff and jitter.

🤔 Why

Because overload is when a small problem becomes a total outage: a slow dependency (lesson 09) cuts capacity, queues grow, clients retry, the load doubles, and every class of user fails together. Priority shedding keeps the most important work whole, degradation stretches capacity, and a retry budget stops clients from making it worse.

🔧 How (in this repo)

Shedder(capacity) in sre/design.py takes the offered load as (class, requests/s, cost per request) in priority order. fifo() serves every class the same fraction; by_priority() fills capacity from the top class down. retry_load(offered, capacity, max_retries, budget) returns attempts per second when every rejected request is retried and, while the overload lasts, every retry is rejected too (a worst case).

🧪 Try it

python3 sre/demo.py overload
python3 - <<'EOF'
import sys; sys.path.insert(0, "sre"); from design import Shedder, retry_load
s = Shedder(1000)
for crit in (400, 900, 1200):
    print(f"critical {crit:>4}/s →", s.by_priority([("critical", crit, 1.0), ("normal", 500, 1.0), ("sheddable", 600, 1.0)]))
for offered in (1000, 1200, 2000):
    print(f"offered {offered}/s → attempts with 3 retries {retry_load(offered, 1000, 3):>5,.0f} · with a 10% budget {retry_load(offered, 1000, 3, 0.1):>5,.0f}")
EOF

✅ Verify — what you should see

overload prints:

   no priorities (everyone gets 1000/1500): {'critical': 267, 'normal': 333, 'sheddable': 400}
   shed by priority:                        {'critical': 400, 'normal': 500, 'sheddable': 100}
   + graceful degradation (normal and sheddable skip photo thumbnails: cost 0.7 each): {'critical': 400, 'normal': 500, 'sheddable': 357}
   3 retries per request           → 3,000 attempts/s arrive at a server that can do 1,000
   3 retries + a 10% retry budget  → 1,650 attempts/s arrive at a server that can do 1,000

Your snippet prints:

critical  400/s → {'critical': 400, 'normal': 500, 'sheddable': 100}
critical  900/s → {'critical': 900, 'normal': 100, 'sheddable': 0}
critical 1200/s → {'critical': 1000, 'normal': 0, 'sheddable': 0}
offered 1000/s → attempts with 3 retries 1,000 · with a 10% budget 1,000
offered 1200/s → attempts with 3 retries 1,800 · with a 10% budget 1,320
offered 2000/s → attempts with 3 retries 5,000 · with a 10% budget 2,200

🏁 What you just proved

With no priorities, a third of the critical requests failed (267 of 400) while 400 second-dessert requests were served. Shedding by priority kept every critical and normal request, and degradation raised the sheddable class from 100 to 357 on the same capacity. But when critical traffic alone is 1,200/s, shedding cannot help — that is a capacity problem (lesson 07). And retries: at 2× overload, 3 retries each turn 2,000 into 5,000 attempts a second; a 10% budget holds it at 2,200.

⚠️ Common mistakes

no priority on requests, so a batch job can starve logins
shedding late — after doing half the work — instead of at the door
retries at every layer, each "only 3"
retrying non-idempotent requests (a payment) on a timeout
degradation paths that are never tested, and fail when first needed
returning 500 for overload, so clients cannot tell "busy" from "broken"

🏭 In production

gRPC retry throttling in the service config: each failure spends a token, each success earns back tokenRatio; below half the tokens, retries stop. On a real account:

{
  "methodConfig": [{
    "name": [{"service": "timetable.Timetable"}],
    "retryPolicy": {"maxAttempts": 3, "initialBackoff": "0.1s", "maxBackoff": "1s",
                    "backoffMultiplier": 2, "retryableStatusCodes": ["UNAVAILABLE"]}
  }],
  "retryThrottling": {"maxTokens": 10, "tokenRatio": 0.1}
}

Envoy retry budget — active retries may be at most a percentage of active requests to the cluster:

clusters:
  - name: timetable
    circuit_breakers:
      thresholds:
        - priority: DEFAULT
          retry_budget:
            budget_percent: {value: 10.0}
            min_retry_concurrency: 3

On Kubernetes, the API server itself sheds by priority with API Priority and Fairness (FlowSchema and PriorityLevelConfiguration objects) and answers 429 when a level's queues are full.

🏭 Why this matters in production: label your top five request types critical or sheddable, and find every place that retries. Then load-test at 1.5 × capacity and check which class fails first. If it is logins, fix the order before the real overload does.

⏭️ Next

You have designs for failure on paper. Do they work? The next lesson breaks things on purpose, carefully, to find out.

git checkout lesson-11-chaos-engineering

← Previousreliability math Next →chaos engineering

This page is the lesson's README from the lesson-10-overload branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.