🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🧯 धडा 10 — Incident response: अग्निशमन सराव

📍 तुम्ही इथे आहात: 12 पैकी धडा 10 · मागील: lesson-09-slos · पुढील: lesson-11-root-cause

📦 या ब्रँचमध्ये काय आहे

धडे 01–09, आणि page वाजल्यावर काय होते: severity levels, incident च्या भूमिका (incident commander, operations lead, communications lead, scribe), आधी mitigate, status page, आणि चार घड्याळे — detect, acknowledge, mitigate, resolve (MTTD, MTTA, MTTR). incident_metrics() आणि SEVERITY obs/reliability.py मध्ये आहेत; obs/demo.py मधले incident() निकालाच्या दिवसाचे incident चालवते.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

आगीची घंटा वाजते. काय व्हायला हवे? 400 विद्यार्थी 400 दिशांना पळणे नव्हे.

शाळा अग्निशमन सराव 🧯 करते, आणि सरावात प्रत्येक मोठ्या व्यक्तीकडे एकच काम असते:

दीपिका नेतृत्व करते. ती बादल्या वाहत नाही. ती निर्णय घेते: "सगळे पूर्वेच्या दरवाजाने बाहेर." संपूर्ण चित्र ती लक्षात ठेवते.
कतरिना प्रत्यक्ष हाताने काम करते. ती खोल्या तपासते आणि दरवाजे बंद करते.
ऐश्वर्या फाटकावर पालकांशी बोलते, दर काही मिनिटांनी: काय घडत आहे, काय करायचे, पुढची माहिती कधी मिळेल.
एक मदतनीस प्रत्येक वेळ आणि प्रत्येक निर्णय clipboard वर लिहून ठेवतो.

आणि सरावाचा पहिला नियम: आधी सगळ्यांना सुरक्षित करा. Corridor धुराने भरलेला असताना कोणीही थांबून "toaster कोणी चालू ठेवला?" असे विचारत नाही. तो प्रश्न नंतर येतो — धडे 11 आणि 12 मध्ये.

🗺️ आकृती

flowchart LR
    s["⏱️ 400<br/>errors start"] -->|"detect 9 min"| d["🚨 409<br/>page fires"]
    d -->|"acknowledge 2 min"| a["👋 411<br/>Dipika takes command"]
    a --> m["↩️ 432<br/>rollback — mitigated<br/>(32 min from start)"]
    m --> r["✅ 470<br/>resolved<br/>(70 min from start)"]
    subgraph roles["🧑‍🚒 roles"]
      ic["🎖️ incident commander — Dipika"]
      ops["🔧 operations lead — Katrina"]
      com["📣 communications lead — Aishwarya"]
      sc["📝 scribe"]
    end

🗺️ काढलेली आकृती + एक lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l10

❓ काय

Incident — अनियोजित घटना जी users ना किंवा व्यवसायाला त्रास देते (किंवा लवकरच देईल), आणि जिला समन्वित प्रतिसाद लागतो. ते लवकर आणि मोठ्याने declare करा — नंतर लहान ठरलेले incident बंद करायला फारसे काही लागत नाही.
Severity levels — आधीच ठरवलेले, म्हणजे पहाटे 3 वाजता कोणी त्यावर वाद घालत नाही. या lab चे:
- SEV1 — बहुतेक पालकांना निकाल दिसत नाहीत: लागणाऱ्या सगळ्यांना page करा, status page, दर 30 मिनिटांनी updates;
- SEV2 — अनेकांसाठी एक feature तुटले आहे: on-call ला page करा, दर तासाला updates;
- SEV3 — कामगिरी घसरली आहे, किंवा छोट्या गटावर परिणाम: कामाच्या वेळेत दुरुस्त करा. (काही companies उलट्या क्रमाने मोजतात, P1–P5, किंवा SEV0 जोडतात. महत्त्वाचे हे की सगळे एकच तक्ता वापरतात.)
भूमिका (Incident Command System मधून, जशा Google SRE आणि PagerDuty च्या incident response मध्ये वापरल्या जातात):
- incident commander (IC) — नेतृत्व करते आणि निर्णय घेते; संपूर्ण चित्र लक्षात ठेवते; काम वाटून देते; स्वतः debug करत नाही;
- operations lead (ops lead) — systems मध्ये बदल करणारी एकमेव व्यक्ती (किंवा team);
- communications lead — प्रतिसाद देणाऱ्यांच्या बाहेरच्या लोकांना updates: status page, support, leadership — ठरलेल्या लयीत;
- scribe — timeline ठेवते: वेळा, निर्णय, काय करून पाहिले. लहान incident मध्ये एकच व्यक्ती अनेक भूमिका सांभाळू शकते; IC ची भूमिका नेहमी नावाने ठरलेली असते.
आधी mitigate — समजण्यापूर्वी हानी थांबवा: शेवटचा बदल roll back करा, feature flag बंद करा, दुसऱ्या region ला fail over करा, capacity वाढवा, वाईट client block करा. Root cause नंतर येते (धडा 11).
Status page — सार्वजनिक (किंवा अंतर्गत) page जे सांगते की users ना काय दिसते, काय केले जात आहे, आणि पुढचे update कधी. त्यामुळे "बंद आहे का?" प्रश्नांचा पूरही कमी होतो.
घड्याळे — timeline वरून मोजलेली:
- MTTD (mean time to detect) — हानीची सुरुवात → detection (page);
- MTTA (mean time to acknowledge) — page → एखादी व्यक्ती म्हणते "मी बघते";
- time to mitigate — सुरुवात → users चा त्रास थांबतो;
- MTTR — सुरुवात → resolved. सावध: "R" repair, recover, restore किंवा resolve साठी वापरला जातो. तुमच्या team ला कोणता अर्थ अभिप्रेत आहे ते लिहून ठेवा. "Mean" = अनेक incidents ची सरासरी; एका incident मधून प्रत्येकाचा एक नमुना मिळतो.
Handover — लांब incidents मध्ये IC आणि ops lead ठरलेल्या वेळापत्रकानुसार, लेखी सारांशासह बदलतात.

🤔 का

कारण गोंधळ हीच मूळ स्थिती असते. भूमिका नसतील तर, पाच engineers एकाच वेळी पाच गोष्टी बदलतात, support ला कोणी सांगत नाही, आणि काय करून पाहिले हे माहीत असलेली एकमेव व्यक्ती जेवायला जाते. "आधी mitigate" नसेल तर, team 40 मिनिटे cause शोधण्यात घालवते आणि पालकांना errors दिसत राहतात — जेव्हा rollback ने ते 2 मिनिटांत संपले असते. आणि users ना mitigate ची वेळ जाणवते, समजून घेण्याची वेळ नाही.

🔧 कसे (या repo मध्ये)

obs/reliability.py मधले incident_metrics(tl) minute stamps घेते — started, detected, acknowledged, mitigated, resolved — आणि time_to_detect (detected − started), time_to_acknowledge (acknowledged − detected), time_to_mitigate आणि time_to_resolve (दोन्ही started पासून), आणि users_hurt_minutes (started → mitigated) देते. SEVERITY मध्ये तीन levels आहेत. obs/demo.py मधले incident() निकालाच्या दिवसाची खरी timeline वापरते: 400 ला errors, 409 ला page (धडा 08), 411 ला acknowledged, 432 ला rollback, 470 ला resolved.

🧪 करून पाहा

घड्याळे हलवा. Bad deploy मध्ये मिनिटाला सुमारे 60 requests fail झाल्या (1,000 चे 6%), म्हणून हानीची मिनिटे × 60 ≈ fail झालेले निकाल views:

python3 obs/demo.py incident
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from reliability import incident_metrics
base = dict(started=400, detected=409, acknowledged=411, mitigated=432, resolved=470)
for name, change in (("today", {}), ("page after 3 min", dict(detected=403, acknowledged=405, mitigated=426)),
                     ("auto-rollback at 404", dict(detected=403, acknowledged=405, mitigated=404)),
                     ("found by parents on social media", dict(detected=425, acknowledged=427, mitigated=448, resolved=486))):
    m = incident_metrics({**base, **change})
    print(f"{name:<34} detect {m['time_to_detect']:>2} · ack {m['time_to_acknowledge']} · mitigate {m['time_to_mitigate']:>2} · resolve {m['time_to_resolve']:>2} · failed views ≈ {m['users_hurt_minutes'] * 60:,}")
EOF

✅ तपासा — तुम्हाला काय दिसायला हवे

incident हे print करते:

── roles: incident commander (Dipika) · ops lead (Katrina) · communications (Aishwarya) · scribe
   SEV1: most parents cannot see results — page everyone, status page, updates every 30 min
   SEV2: a feature is broken for many — page on-call, updates every hour
   SEV3: degraded or a small group affected — fix in working hours
── the timeline (minutes): {'started': 400, 'detected': 409, 'acknowledged': 411, 'mitigated': 432, 'resolved': 470}
   time_to_detect       9 min
   time_to_acknowledge  2 min
   time_to_mitigate     32 min
   time_to_resolve      70 min
   users_hurt_minutes   32 min
   first mitigate (roll back, switch off, add capacity), THEN find the cause

तुमचा snippet हे print करतो:

today                              detect  9 · ack 2 · mitigate 32 · resolve 70 · failed views ≈ 1,920
page after 3 min                   detect  3 · ack 2 · mitigate 26 · resolve 70 · failed views ≈ 1,560
auto-rollback at 404               detect  3 · ack 2 · mitigate  4 · resolve 70 · failed views ≈ 240
found by parents on social media   detect 25 · ack 2 · mitigate 48 · resolve 86 · failed views ≈ 2,880

🏁 तुम्ही आत्ताच काय सिद्ध केले

पालकांना mitigate चे घड्याळ जाणवले, resolve चे नाही: पहिल्या तीन ओळींमध्ये resolve ची वेळ 70 मिनिटे होती, पण fail झालेले views 1,920 वरून 240 वर आले. जलद detection ने 6 मिनिटे (360 views) वाचवली; burn-rate signal वरच्या automatic rollback ने 28 मिनिटे (1,680 views) वाचवली. Social media वर पालकांकडून कळणे — कोणताही alert नाही — याची किंमत 16 जादा मिनिटे आणि 960 जास्त fail झालेले views. सर्वोत्तम incident तो, जो व्यक्ती जागी होण्यापूर्वीच machine mitigate करते.

⚠️ नेहमीच्या चुका

rollback करून पाहण्यापूर्वी 30 मिनिटे debugging करणे
कोणालाही incident commander म्हणून नाव न देणे — किंवा IC नेच commands type करणे
अनेक लोक एकाच वेळी production बदलत आहेत, आणि कोणीच ते लिहून ठेवत नाही
status page update नाही, त्यामुळे support आणि पालकांना एक तास काहीच कळत नाही
incident उशिरा declare करणे, "कदाचित काहीच नसेल म्हणून"
SLI खरोखर परत आला आहे का ते तपासण्यापूर्वी, page थांबताच resolved म्हणणे

🏭 प्रत्यक्ष वापरात

खऱ्या account वर — Kubernetes वर bad deploy साठी सगळ्यात जलद mitigation:

kubectl rollout history deployment/results-api
kubectl rollout undo deployment/results-api            # back to the previous version
kubectl rollout status deployment/results-api --timeout=5m

PagerDuty Events API v2 मधून कोणत्याही script वरून page करा (routing key एका service integration मधून मिळते):

curl -s -X POST https://events.pagerduty.com/v2/enqueue -H 'Content-Type: application/json' -d '{
  "routing_key": "<integration key>",
  "event_action": "trigger",
  "dedup_key": "results-api-burn-rate",
  "payload": {"summary": "results-api fast error-budget burn (14.4x)", "source": "prometheus", "severity": "critical"}
}'

Atlassian Statuspage page वर incident उघडा:

curl -s -X POST "https://api.statuspage.io/v1/pages/<page id>/incidents" \
  -H "Authorization: OAuth <api key>" -H 'Content-Type: application/json' -d '{
  "incident": {"name": "Results page errors", "status": "investigating",
               "body": "Some parents see an error on the results page. We are rolling back a change. Next update in 30 minutes."}
}'

PagerDuty, incident.io, FireHydrant, Rootly आणि Jira Service Management सारखी incident tools तुमच्यासाठी chat channel उघडतात, भूमिका वाटतात आणि timeline ठेवतात. Incident channel साठी पहिल्या message चा नमुना:

🔴 SEV1 declared 11:49 · IC: Dipika · Ops: Katrina · Comms: Aishwarya · Scribe: on-call #2
Impact: ~6% of results views fail (504) since 11:40.
Current action: rolling back results-api v42 → v41 (Katrina).
Next update: 12:19, or sooner if something changes.

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: सराव करा. दर तिमाहीत एक game day घ्या — staging मध्ये मुद्दाम काहीतरी मोडा, on-call ला page करा, भूमिका पार पाडा — म्हणजे पहिल्या खऱ्या SEV1 मध्ये कोणीतरी पहिल्यांदाच IC ची भूमिका सांभाळत नसेल.

⏭️ पुढे

Rollback ने हानी थांबवली. पण v42 का मोडले — आणि पालकांच्या आधी कशानेच ते का पकडले नाही? Root-cause analysis.

git checkout lesson-11-root-cause

🧯 Lesson 10 — Incident response: the fire drill

📍 You are here: Lesson 10 of 12 · Previous: lesson-09-slos · Next: lesson-11-root-cause

📦 What's in this branch

Lessons 01–09, plus what happens when the page fires: severity levels, the incident roles (incident commander, operations lead, communications lead, scribe), mitigate first, the status page, and the four clocks — detect, acknowledge, mitigate, resolve (MTTD, MTTA, MTTR). incident_metrics() and SEVERITY live in obs/reliability.py; incident() in obs/demo.py runs the results-day incident.

🧒 Explain like I'm 5

The fire alarm rings. What should happen? Not 400 pupils running in 400 directions.

The school practises a fire drill 🧯, and in the drill every adult has one job:

Dipika leads. She does not carry buckets. She decides: "Everyone out by the east door." She keeps the whole picture.
Katrina does the hands-on work. She checks the rooms and closes the doors.
Aishwarya talks to the parents at the gate, every few minutes: what is happening, what to do, when the next update comes.
A helper writes down every time and every decision on a clipboard.

And the first rule of the drill: get everyone safe first. Nobody stops to ask "who left the toaster on?" while the corridor is full of smoke. That question comes later — in lessons 11 and 12.

🗺️ Diagram

flowchart LR
    s["⏱️ 400<br/>errors start"] -->|"detect 9 min"| d["🚨 409<br/>page fires"]
    d -->|"acknowledge 2 min"| a["👋 411<br/>Dipika takes command"]
    a --> m["↩️ 432<br/>rollback — mitigated<br/>(32 min from start)"]
    m --> r["✅ 470<br/>resolved<br/>(70 min from start)"]
    subgraph roles["🧑‍🚒 roles"]
      ic["🎖️ incident commander — Dipika"]
      ops["🔧 operations lead — Katrina"]
      com["📣 communications lead — Aishwarya"]
      sc["📝 scribe"]
    end

🗺️ Drawn version + a lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l10

❓ What

Incident — an unplanned event that hurts (or will soon hurt) users or the business, and needs a coordinated response. Declare one early and loudly — it costs little to close an incident that turns out small.
Severity levels — agreed in advance, so nobody debates them at 3 a.m. The lab's:
- SEV1 — most parents cannot see results: page everyone needed, status page, updates every 30 minutes;
- SEV2 — a feature is broken for many: page on-call, updates every hour;
- SEV3 — degraded, or a small group affected: fix in working hours. (Some companies count the other way, P1–P5, or add SEV0. What matters is that everyone uses the same table.)
Roles (from the Incident Command System, as used in Google SRE and PagerDuty's incident response):
- incident commander (IC) — leads and decides; keeps the big picture; assigns work; does not debug;
- operations lead (ops lead) — the only person (or team) changing the systems;
- communications lead — updates to people outside the response: the status page, support, leadership — at a fixed rhythm;
- scribe — keeps the timeline: times, decisions, what was tried. In a small incident one person may hold several roles; the IC role is always named.
Mitigate first — stop the harm before you understand it: roll back the last change, switch off a feature flag, fail over to another region, add capacity, block a bad client. Root cause comes after (lesson 11).
Status page — a public (or internal) page that says what users see, what is being done, and when the next update is. It also cuts the flood of "is it down?" questions.
The clocks — measured from the timeline:
- MTTD (mean time to detect) — start of harm → detection (the page);
- MTTA (mean time to acknowledge) — page → a person says "I have it";
- time to mitigate — start → users stop being hurt;
- MTTR — start → resolved. Careful: "R" is used for repair, recover, restore or resolve. Write down which one your team means. "Mean" = the average over many incidents; one incident gives one sample of each.
Handover — long incidents change IC and ops lead on a schedule, with a written summary.

🤔 Why

Because chaos is the default. Without roles, five engineers change five things at once, nobody tells support, and the one person who knows what was tried goes to lunch. Without "mitigate first", the team spends 40 minutes finding the cause while parents keep seeing errors — when a rollback would have ended it in 2 minutes. And users feel the time to mitigate, not the time to understand.

🔧 How (in this repo)

incident_metrics(tl) in obs/reliability.py takes minute stamps — started, detected, acknowledged, mitigated, resolved — and returns time_to_detect (detected − started), time_to_acknowledge (acknowledged − detected), time_to_mitigate and time_to_resolve (both from started), and users_hurt_minutes (started → mitigated). SEVERITY holds the three levels. incident() in obs/demo.py uses the real results-day timeline: errors at 400, page at 409 (lesson 08), acknowledged at 411, rollback at 432, resolved at 470.

🧪 Try it

Move the clocks. The bad deploy failed about 60 requests a minute (6% of 1,000), so minutes of harm × 60 ≈ failed results views:

python3 obs/demo.py incident
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from reliability import incident_metrics
base = dict(started=400, detected=409, acknowledged=411, mitigated=432, resolved=470)
for name, change in (("today", {}), ("page after 3 min", dict(detected=403, acknowledged=405, mitigated=426)),
                     ("auto-rollback at 404", dict(detected=403, acknowledged=405, mitigated=404)),
                     ("found by parents on social media", dict(detected=425, acknowledged=427, mitigated=448, resolved=486))):
    m = incident_metrics({**base, **change})
    print(f"{name:<34} detect {m['time_to_detect']:>2} · ack {m['time_to_acknowledge']} · mitigate {m['time_to_mitigate']:>2} · resolve {m['time_to_resolve']:>2} · failed views ≈ {m['users_hurt_minutes'] * 60:,}")
EOF

✅ Verify — what you should see

incident prints:

── roles: incident commander (Dipika) · ops lead (Katrina) · communications (Aishwarya) · scribe
   SEV1: most parents cannot see results — page everyone, status page, updates every 30 min
   SEV2: a feature is broken for many — page on-call, updates every hour
   SEV3: degraded or a small group affected — fix in working hours
── the timeline (minutes): {'started': 400, 'detected': 409, 'acknowledged': 411, 'mitigated': 432, 'resolved': 470}
   time_to_detect       9 min
   time_to_acknowledge  2 min
   time_to_mitigate     32 min
   time_to_resolve      70 min
   users_hurt_minutes   32 min
   first mitigate (roll back, switch off, add capacity), THEN find the cause

Your snippet prints:

today                              detect  9 · ack 2 · mitigate 32 · resolve 70 · failed views ≈ 1,920
page after 3 min                   detect  3 · ack 2 · mitigate 26 · resolve 70 · failed views ≈ 1,560
auto-rollback at 404               detect  3 · ack 2 · mitigate  4 · resolve 70 · failed views ≈ 240
found by parents on social media   detect 25 · ack 2 · mitigate 48 · resolve 86 · failed views ≈ 2,880

🏁 What you just proved

Parents felt the mitigate clock, not the resolve clock: the resolve time was 70 minutes in the first three rows, but failed views went from 1,920 to 240. Faster detection saved 6 minutes (360 views); an automatic rollback on the burn-rate signal saved 28 minutes (1,680 views). Being told by parents on social media — no alert — cost 16 extra minutes and 960 more failed views. The best incident is the one a machine mitigates before a person wakes up.

⚠️ Common mistakes

debugging for 30 minutes before trying the rollback
nobody named as incident commander — or the IC also typing commands
several people changing production at once, with nobody writing it down
no status page update, so support and parents hear nothing for an hour
declaring the incident late, "in case it is nothing"
calling it resolved when the page stops, before checking the SLI is really back

🏭 In production

On a real account — the fastest mitigation for a bad deploy on Kubernetes:

kubectl rollout history deployment/results-api
kubectl rollout undo deployment/results-api            # back to the previous version
kubectl rollout status deployment/results-api --timeout=5m

Page from any script through the PagerDuty Events API v2 (the routing key comes from a service integration):

curl -s -X POST https://events.pagerduty.com/v2/enqueue -H 'Content-Type: application/json' -d '{
  "routing_key": "<integration key>",
  "event_action": "trigger",
  "dedup_key": "results-api-burn-rate",
  "payload": {"summary": "results-api fast error-budget burn (14.4x)", "source": "prometheus", "severity": "critical"}
}'

Open an incident on an Atlassian Statuspage page:

curl -s -X POST "https://api.statuspage.io/v1/pages/<page id>/incidents" \
  -H "Authorization: OAuth <api key>" -H 'Content-Type: application/json' -d '{
  "incident": {"name": "Results page errors", "status": "investigating",
               "body": "Some parents see an error on the results page. We are rolling back a change. Next update in 30 minutes."}
}'

Incident tools such as PagerDuty, incident.io, FireHydrant, Rootly and Jira Service Management open the chat channel, assign roles and keep the timeline for you. A first message template for the incident channel:

🔴 SEV1 declared 11:49 · IC: Dipika · Ops: Katrina · Comms: Aishwarya · Scribe: on-call #2
Impact: ~6% of results views fail (504) since 11:40.
Current action: rolling back results-api v42 → v41 (Katrina).
Next update: 12:19, or sooner if something changes.

🏭 Why this matters in production: practise. Run a game day each quarter — break something on purpose in staging, page the on-call, run the roles — so the first real SEV1 is not the first time anyone has held the IC role.

⏭️ Next

The rollback stopped the harm. But why did v42 break — and why did nothing catch it before parents did? Root-cause analysis.

git checkout lesson-11-root-cause

← Previousslos Next →root cause

This page is the lesson's README from the lesson-10-incident-response branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.