🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🧯 धडा 11 — Chaos engineering आणि game days: मुद्दाम, पण काळजीपूर्वक बिघडवा

📍 तुम्ही इथे आहात: 12 पैकी धडा 11 · मागे: lesson-10-overload · पुढे: lesson-12-production-readiness

📦 या ब्रँचमध्ये काय आहे

धडा 10, अधिक धडे 07–10 मधील designs तपासणारा प्रयोग. Chaos experiment मध्ये एक hypothesis, छोटी blast radius आणि लिहून ठेवलेल्या abort च्या अटी असतात. Game day म्हणजे त्याचीच, खोलीत लोक असलेली, नियोजित आवृत्ती. sre/design.py मधील ChaosExperiment आणि sre/demo.py मधील chaos().

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

पथक म्हणते: "पूर्व बाजूची वीज गेली, तरी उरलेल्या बाजू सगळे वर्ग चालवू शकतात." 🏫 हे खरे आहे का? कोणालाच माहीत नाही. त्यांनी कधीच करून पाहिलेले नाही.

म्हणून ते एक test ठरवतात. पण संपूर्ण शाळेवर नाही — फक्त वर्गांच्या एका block वर, म्हणजे दहाव्या भागाच्या विद्यार्थिनींवर. ते आधी लिहून ठेवतात: "जवळजवळ प्रत्येक वर्ग चालू राहील अशी आमची अपेक्षा आहे. 100 पैकी 1 पेक्षा जास्त वर्ग थांबले, तर आम्ही लगेच वीज परत सुरू करू."

पहिल्या block चे वर्ग तीन बाजूंमध्ये आहेत. ते पूर्व बाजूची वीज बंद करतात. उरलेल्या दोन बाजू विद्यार्थिनींना सामावून घेतात. प्रत्येक वर्ग चालू राहतो. ✅ कल्पना बरोबर होती.

दुसऱ्या block चे वर्ग फक्त दोन बाजूंमध्ये आहेत. ते एक बंद करतात. जवळजवळ अर्धे वर्ग थांबतात! एका मिनिटाच्या आत कतरिना वीज परत सुरू करते. काही विद्यार्थिनींना एक मिनिट त्रास झाला — पण फक्त एका block मध्ये, आणि आता पथकाला माहीत आहे की दुसऱ्या block ला तिसरी बाजू हवी, खरी वीज गेल्यावर संपूर्ण शाळेसमोर हे शिकण्याआधीच.

🗺️ आकृती

flowchart LR
    h["📝 hypothesis<br/>lose zone B → success ≥ 99.5%"] --> x["💥 inject at minute 3<br/>one cell = 10% of users"]
    x --> a["cell A: 6 replicas, 3 zones<br/>800 ≥ 700 req/s → 100%<br/>hypothesis held"]
    x --> b["cell B: 4 replicas, 2 zones<br/>400 of 700 req/s → 57%"]
    b --> ab["🛑 abort (< 99%) at minute 3<br/>fault undone · 18,000 failed"]

🗺️ काढलेली आकृती + एक lab: https://school-edh.pages.dev/sre/lesson-diagrams.html#l11

❓ काय

Chaos engineering — "production मधील अस्थिर परिस्थिती सहन करण्याच्या system च्या क्षमतेवर विश्वास निर्माण करण्यासाठी system वर प्रयोग करण्याची शिस्त" (principlesofchaos.org). Netflix ने Chaos Monkey (2011) द्वारे ती लोकप्रिय केली, जे कामाच्या वेळेत यादृच्छिकपणे instances बंद करते.
Steady state — users ठीक आहेत याचे मोजता येणारे लक्षण: success rate, p99 latency, मिनिटाला orders. प्रयोगाच्या आधी निवडलेले.
Hypothesis — "X बिघडले तरी steady state Y च्या आत राहते". इथे: "zone B गेला तरी cell चा success rate ≥ 99.5% राहतो". प्रयोग ती खोटी ठरवू शकतो — तो यशस्वी प्रयोगच आहे, कारण तुम्ही काहीतरी शिकलात.
Fault injection — तुम्ही घडवलेला बिघाड: pod मारणे, zone गमावणे, 300 ms latency वाढवणे, disk भरणे, dependency अडवणे.
Blast radius — hypothesis चुकली तर किती नुकसान होऊ शकते. सर्वात छोट्यापासून सुरुवात करा (एक cell, 1–10% users, test environment), आणि pass झाल्यावरच विस्तार करा.
Abort (stop) च्या अटी — लिहून ठेवलेल्या मर्यादा ज्या प्रयोग संपवतात आणि बिघाड लगेच उलटवतात, शक्य असेल तर आपोआप. इथे: 99.0% खाली गेलेला कोणताही minute.
Game day — लोकांसह नियोजित सराव: बिघाड घडवला जातो (खरोखर, किंवा कागदावर tabletop म्हणून), on-call आणि IC (धडा 08) प्रतिसाद देतात, आणि runbooks तपासले जातात. तो फक्त software नाही, तर लोक आणि प्रक्रिया तपासतो.
आहात तिथून सुरुवात करा — आधी staging मध्ये, मग production मध्ये छोट्या blast radius ने आणि कामाच्या वेळेत, सगळे तयार असताना.

🤔 का

कारण या course मधील प्रत्येक design हा एक दावा आहे — "N+1 zones" (धडा 07), "failover चालतो" (धडा 09), "आम्ही sheddable आधी shed करतो" (धडा 10) — आणि न तपासलेले दावे पहाटे 3 वाजता अयशस्वी होतात. खरे outages सर्वात वाईट क्षण आणि संपूर्ण blast radius निवडतात. प्रयोग वेळ, आकार आणि थांबवण्याचे बटण निवडतो.

🔧 कसे (या repo मध्ये)

sre/design.py मध्ये ChaosExperiment(replicas_per_zone, zones, per_replica_rps, load_rps, hypothesis=0.995, abort=0.99). run(minutes=10, inject_at=3, fault_minutes=5) मिनिटा-मिनिटाने चालते: बिघाड चालू असताना एका zone चे replicas नसतात; success = min(1, capacity ÷ load). एखादा minute abort रेषेखाली गेला, तर पुढच्या minute पासून बिघाड उलटवला जातो. ते प्रत्येक minute चे rates, abort चा minute, अयशस्वी requests आणि निकाल परत देते.

🧪 करून पाहा

python3 sre/demo.py chaos
python3 - <<'EOF'
import sys; sys.path.insert(0, "sre"); from design import ChaosExperiment
for load in (600, 700, 800, 900):
    rates, ab, failed, verdict = ChaosExperiment(2, 3, 200, load).run()
    print(f"load {load} req/s, 6 replicas in 3 zones → worst minute {min(rates):.1%} · {verdict} · aborted at {ab} · failed {failed:,}")
rates, ab, failed, verdict = ChaosExperiment(2, 2, 200, 700, abort=0.5).run()
print(f"abort line lowered to 50% → aborted at {ab} · failed {failed:,} (the fault ran all 5 minutes)")
EOF

✅ तपासा — तुम्हाला काय दिसायला हवे

chaos हे छापते:

   6 replicas in 3 zones: minutes 0–9 success 100% 100% 100% 100% 100% 100% 100% 100% 100% 100%
      → hypothesis held · no abort · 0 failed
   4 replicas in 2 zones: minutes 0–9 success 100% 100% 100% 57% 100% 100% 100% 100% 100% 100%
      → hypothesis refuted · ABORTED at minute 3, fault undone · 18,000 requests failed in the cell

तुमचा snippet हे छापतो:

load 600 req/s, 6 replicas in 3 zones → worst minute 100.0% · hypothesis held · aborted at None · failed 0
load 700 req/s, 6 replicas in 3 zones → worst minute 100.0% · hypothesis held · aborted at None · failed 0
load 800 req/s, 6 replicas in 3 zones → worst minute 100.0% · hypothesis held · aborted at None · failed 0
load 900 req/s, 6 replicas in 3 zones → worst minute 88.9% · hypothesis refuted · aborted at 3 · failed 6,000
abort line lowered to 50% → aborted at None · failed 90,000 (the fault ran all 5 minutes)

🏁 तुम्ही आत्ताच काय सिद्ध केले

तेच design (3 zones मध्ये 6 replicas) 800 req/s वर pass होते आणि 900 वर fail: प्रयोगाचा निकाल फक्त तो ज्या load वर चालला त्यासाठीच खरा असतो — traffic वाढेल तसा तो पुन्हा करा. Abort च्या अटीने नुकसान 90,000 अयशस्वी requests (5 मिनिटे चालू ठेवलेला बिघाड) वरून 18,000 (1 मिनिट) वर आणले, आणि blast radius ने तेही दहापैकी एका cell पुरते मर्यादित ठेवले. हीच संपूर्ण पद्धत आहे: स्पष्ट hypothesis, छोटी blast radius, आणि "start" दाबण्याच्या आधी ठरवलेले थांबवण्याचे बटण.

⚠️ नेहमीच्या चुका

"चला, काहीतरी मारून बघू" — hypothesis नाही, steady state नाही, काहीच शिकायला मिळत नाही
abort ची अट नाही, किंवा अशी अट जिच्यावर कोणाचेच लक्ष नाही
पहिलीच blast radius म्हणून संपूर्ण production
कायम फक्त staging मध्येच चालवणे — staging कडे production चा traffic किंवा data नसतो
पहाटे 3 वाजता कोणीही आसपास नसताना, किंवा on-call ला न सांगता चालवणे
कमकुवतपणा सापडूनही तो दुरुस्त न करणे — हा शोध मालकासह backlog मध्ये जायला हवा

🏭 प्रत्यक्ष वापरात

Kubernetes वर Chaos Mesh — experiment namespace मधील timetable चा एक pod मारा. खऱ्या account वर (खरा cluster, सहमतीसह):

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata: {name: kill-one-timetable-pod, namespace: chaos-testing}
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces: [timetable-cell-1]
    labelSelectors: {app: timetable}

AWS Fault Injection Service — experiment template मध्ये त्याच्या स्वतःच्या stop conditions असतात, CloudWatch alarms शी जोडलेल्या, म्हणून abort आपोआप होतो:

{
  "description": "lose zone B for the timetable cell",
  "stopConditions": [{"source": "aws:cloudwatch:alarm",
                      "value": "arn:aws:cloudwatch:ap-south-1:111122223333:alarm:timetable-success-below-99"}],
  "targets": {"cell1": {"resourceType": "aws:ec2:instance", "selectionMode": "ALL",
                        "resourceTags": {"cell": "1"},
                        "filters": [{"path": "Placement.AvailabilityZone", "values": ["ap-south-1b"]}]}},
  "actions": {"stopZoneB": {"actionId": "aws:ec2:stop-instances", "targets": {"Instances": "cell1"},
                            "parameters": {"startInstancesAfterDuration": "PT5M"}}},
  "roleArn": "arn:aws:iam::111122223333:role/fis-experiments"
}

aws fis start-experiment --experiment-template-id EXT1a2b3c4d5e6f7

LitmusChaos आणि Gremlin ही इतर नेहमीची tools आहेत. Game day साठी फक्त calendar invite, लिहून ठेवलेली hypothesis, abort चा नियम, एक IC आणि एक scribe एवढेच लागते.

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे आहे: तुमच्या मागच्या architecture review मधील एक design दावा निवडा ("तो zone गेला तरी टिकतो"). Hypothesis, steady state, blast radius आणि abort ची अट एका पानावर लिहा. ते पान म्हणजे तुमचा पहिला game day.

⏭️ पुढे

शेवटचा धडा प्रत्येक धडा एका आढाव्यात आणतो — जो पथक एखाद्या service चा pager घेण्याआधी तिने पार करायलाच हवा.

git checkout lesson-12-production-readiness

🧯 Lesson 11 — Chaos engineering and game days: break it on purpose, carefully

📍 You are here: Lesson 11 of 12 · Previous: lesson-10-overload · Next: lesson-12-production-readiness

📦 What's in this branch

Lesson 10, plus the experiment that checks the designs from lessons 07–10. A chaos experiment has a hypothesis, a small blast radius and written abort conditions. A game day is the planned, people-in-the-room version. ChaosExperiment in sre/design.py and chaos() in sre/demo.py.

🧒 Explain like I'm 5

The crew says: "If the east wing loses power, the other wings can still run all the classes." 🏫 Is that true? Nobody knows. They have never tried.

So they plan a test. But not on the whole school — just on one block of classrooms, one tenth of the pupils. They write down first: "We expect nearly every class to keep going. If more than 1 class in 100 stops, we switch the power back on at once."

Block one has classrooms in three wings. They cut the east wing's power. The other two wings take the pupils. Every class keeps going. ✅ The idea was right.

Block two has classrooms in only two wings. They cut one. Nearly half the classes stop! Within a minute, Katrina switches the power back on. Some pupils were bothered for one minute — but only in one block, and now the crew knows block two needs a third wing, before a real power cut teaches them in front of the whole school.

🗺️ Diagram

flowchart LR
    h["📝 hypothesis<br/>lose zone B → success ≥ 99.5%"] --> x["💥 inject at minute 3<br/>one cell = 10% of users"]
    x --> a["cell A: 6 replicas, 3 zones<br/>800 ≥ 700 req/s → 100%<br/>hypothesis held"]
    x --> b["cell B: 4 replicas, 2 zones<br/>400 of 700 req/s → 57%"]
    b --> ab["🛑 abort (< 99%) at minute 3<br/>fault undone · 18,000 failed"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/sre/lesson-diagrams.html#l11

❓ What

Chaos engineering — "the discipline of experimenting on a system in order to build confidence in its capability to withstand turbulent conditions in production" (principlesofchaos.org). Netflix popularised it with Chaos Monkey (2011), which terminates instances at random in working hours.
Steady state — a measurable sign that users are fine: success rate, p99 latency, orders per minute. Chosen before the experiment.
Hypothesis — "when X fails, the steady state stays within Y". Here: "losing zone B keeps the cell's success rate ≥ 99.5%". An experiment can refute it — that is a successful experiment, because you learned something.
Fault injection — the failure you cause: kill a pod, lose a zone, add 300 ms of latency, fill a disk, block a dependency.
Blast radius — how much can be hurt if the hypothesis is wrong. Start with the smallest (one cell, 1–10% of users, a test environment), and widen only after a pass.
Abort (stop) conditions — written thresholds that end the experiment and undo the fault at once, automatically if possible. Here: any minute below 99.0%.
Game day — a planned exercise with people: a fault is injected (for real, or on paper as a tabletop), the on-call and IC (lesson 08) respond, and the runbooks are tested. It tests the people and process, not only the software.
Start where you are — first in staging, then in production with a small blast radius and during working hours with everyone ready.

🤔 Why

Because every design in this course is a claim — "N+1 zones" (lesson 07), "failover works" (lesson 09), "we shed sheddable first" (lesson 10) — and untested claims fail at 3 a.m. Real outages pick the worst moment and the whole blast radius. An experiment picks the time, the size and the stop button.

🔧 How (in this repo)

ChaosExperiment(replicas_per_zone, zones, per_replica_rps, load_rps, hypothesis=0.995, abort=0.99) in sre/design.py. run(minutes=10, inject_at=3, fault_minutes=5) goes minute by minute: while the fault is on, one zone's replicas are gone; success = min(1, capacity ÷ load). If a minute falls below the abort line, the fault is undone from the next minute. It returns the per-minute rates, the abort minute, the failed requests and the verdict.

🧪 Try it

python3 sre/demo.py chaos
python3 - <<'EOF'
import sys; sys.path.insert(0, "sre"); from design import ChaosExperiment
for load in (600, 700, 800, 900):
    rates, ab, failed, verdict = ChaosExperiment(2, 3, 200, load).run()
    print(f"load {load} req/s, 6 replicas in 3 zones → worst minute {min(rates):.1%} · {verdict} · aborted at {ab} · failed {failed:,}")
rates, ab, failed, verdict = ChaosExperiment(2, 2, 200, 700, abort=0.5).run()
print(f"abort line lowered to 50% → aborted at {ab} · failed {failed:,} (the fault ran all 5 minutes)")
EOF

✅ Verify — what you should see

chaos prints:

   6 replicas in 3 zones: minutes 0–9 success 100% 100% 100% 100% 100% 100% 100% 100% 100% 100%
      → hypothesis held · no abort · 0 failed
   4 replicas in 2 zones: minutes 0–9 success 100% 100% 100% 57% 100% 100% 100% 100% 100% 100%
      → hypothesis refuted · ABORTED at minute 3, fault undone · 18,000 requests failed in the cell

Your snippet prints:

load 600 req/s, 6 replicas in 3 zones → worst minute 100.0% · hypothesis held · aborted at None · failed 0
load 700 req/s, 6 replicas in 3 zones → worst minute 100.0% · hypothesis held · aborted at None · failed 0
load 800 req/s, 6 replicas in 3 zones → worst minute 100.0% · hypothesis held · aborted at None · failed 0
load 900 req/s, 6 replicas in 3 zones → worst minute 88.9% · hypothesis refuted · aborted at 3 · failed 6,000
abort line lowered to 50% → aborted at None · failed 90,000 (the fault ran all 5 minutes)

🏁 What you just proved

The same design (6 replicas in 3 zones) passes at 800 req/s and fails at 900: an experiment's result is only true for the load it ran under — repeat it as traffic grows. The abort condition cut the damage from 90,000 failed requests (fault left on for 5 minutes) to 18,000 (1 minute), and the blast radius kept even that to one cell of ten. That is the whole method: a clear hypothesis, a small blast radius, and a stop button you decided on before you pressed "start".

⚠️ Common mistakes

"let's just kill something and see" — no hypothesis, no steady state, nothing learned
no abort condition, or one nobody is watching
the whole of production as the first blast radius
running only in staging forever — staging does not have production's traffic or data
running at 3 a.m. with nobody around, or without telling on-call
finding a weakness and not fixing it — the finding goes into the backlog with an owner

🏭 In production

Chaos Mesh on Kubernetes — kill one timetable pod in the experiment namespace. On a real account (a real cluster, with agreement):

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata: {name: kill-one-timetable-pod, namespace: chaos-testing}
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces: [timetable-cell-1]
    labelSelectors: {app: timetable}

AWS Fault Injection Service — the experiment template carries its own stop conditions, tied to CloudWatch alarms, so the abort is automatic:

{
  "description": "lose zone B for the timetable cell",
  "stopConditions": [{"source": "aws:cloudwatch:alarm",
                      "value": "arn:aws:cloudwatch:ap-south-1:111122223333:alarm:timetable-success-below-99"}],
  "targets": {"cell1": {"resourceType": "aws:ec2:instance", "selectionMode": "ALL",
                        "resourceTags": {"cell": "1"},
                        "filters": [{"path": "Placement.AvailabilityZone", "values": ["ap-south-1b"]}]}},
  "actions": {"stopZoneB": {"actionId": "aws:ec2:stop-instances", "targets": {"Instances": "cell1"},
                            "parameters": {"startInstancesAfterDuration": "PT5M"}}},
  "roleArn": "arn:aws:iam::111122223333:role/fis-experiments"
}

aws fis start-experiment --experiment-template-id EXT1a2b3c4d5e6f7

LitmusChaos and Gremlin are other common tools. A game day needs only a calendar invite, a written hypothesis, an abort rule, an IC and a scribe.

🏭 Why this matters in production: pick one design claim from your last architecture review ("it survives a zone loss"). Write the hypothesis, the steady state, the blast radius and the abort condition on one page. That page is your first game day.

⏭️ Next

The last lesson puts every lesson into one review — the one a service must pass before the crew takes its pager.

git checkout lesson-12-production-readiness

← Previousoverload Next →production readiness

This page is the lesson's README from the lesson-11-chaos-engineering branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.