🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🦺 धडा 08 — Incident command: भूमिका, लय आणि handoff

📍 तुम्ही इथे आहात: 12 पैकी धडा 08 · मागे: lesson-07-capacity-planning · पुढे: lesson-09-reliability-math

📦 या ब्रँचमध्ये काय आहे

धडा 07, अधिक incident दरम्यान पथक स्वतःला कसे संघटित करते. Observability school चा धडा 10 severity, incident ची घड्याळे (detect, acknowledge, mitigate, resolve) शिकवतो, आणि त्याचे धडे 11–12 root cause आणि postmortem. हा धडा म्हणजे command ची रचना: चार भूमिका, status-update ची लय, आणि shift संपल्यावरचे handoff. sre/crew.py मधील IncidentLog आणि sre/demo.py मधील command().

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

रात्री 23:00 ला वेळापत्रकाची इमारत अंधारात जाते. 🌑

पहिली रात्र — कोणताही plan नाही. कतरिना आणि दीपिका दोघीही दुरुस्ती सुरू करतात. त्या एकमेकींचे काम उलटवतात. जवळजवळ दोन तास कोणीच मुख्याध्यापकांना काहीही सांगत नाही. मध्यरात्री कतरिना घरी जाते — आणि ती नेमके काय करत होती ते कोणालाच माहीत नसते.

दुसरी रात्र — एक plan. दीपिका म्हणते: "मी incident commander आहे. मी दुरुस्ती करत नाही — मी नेतृत्व करते." ती बोट दाखवते: "कतरिना, तू दुरुस्ती कर. ऐश्वर्या, तू दर अर्ध्या तासाने लोकांना काय चालले आहे ते सांगत राहा. आणि लहान आहे तोपर्यंत काय घडते ते मीच लिहून ठेवते." मध्यरात्री दीपिकाची shift संपते. ती अशीच निघून जात नाही. ती crew-5 ला सगळे सांगते — काय बिघडले आहे, काय करून पाहिले, कोण काय करत आहे — आणि crew-5 म्हणते: "I have command." त्यानंतरच दीपिका जाते.

बिघाड तोच. दुसरी रात्र शांत असते, आणि लवकर संपते.

🗺️ आकृती

flowchart LR
    ic["🦺 IC — Dipika<br/>leads, decides, does NOT fix"] --> ops["🔧 ops — Katrina<br/>hands on the system"]
    ic --> comms["📣 comms — Aishwarya<br/>update every 30 min"]
    ic --> scribe["📝 scribe<br/>the timeline"]
    ic -->|"00:00 handoff:<br/>state, actions, roles"| ic2["🦺 IC — crew-5<br/>'I have command' ✓"]
    bad["without roles: 5 problems<br/>no IC · 110 min with no update<br/>Katrina left holding ops"]

🗺️ काढलेली आकृती + एक lab: https://school-edh.pages.dev/sre/lesson-diagrams.html#l08

❓ काय

Incident Command System (ICS) — 1970 च्या दशकात California मधील जंगलातील आगी विझवण्यासाठी तयार केलेली रचना, आता आपत्कालीन सेवा ती वापरतात. Google ची incident प्रक्रिया (IMAG), PagerDuty ची प्रकाशित प्रक्रिया आणि इतर अनेक तिच्यावर आधारित आहेत.
Incident commander (IC) — incident ची मालकी घेते: प्राधान्ये ठरवते, भूमिका वाटते, निर्णय घेते (उदाहरणार्थ "आत्ताच roll back करा"), आणि संपूर्ण चित्र लक्षात ठेवते. IC ने स्वतः commands टाइप करू नयेत — नाहीतर संपूर्ण incident वर कोणाचेच लक्ष राहत नाही. अगदी छोट्या incident मध्ये एक व्यक्ती अनेक भूमिका सांभाळू शकते, पण ते मोठ्याने सांगा.
Operations lead (ops) — system मध्ये बदल करणारे लोक: शोध घेणे, mitigate करणे. Incident दरम्यान फक्त ops च production बदलते.
Communications lead (comms) — users, support आणि नेतृत्वाला नियमित status updates पाठवते, म्हणजे ते दुरुस्ती करणाऱ्यांना मध्ये मध्ये त्रास देणे थांबवतात. बातमी नसतानाही एक ठरलेली लय (इथे दर 30 मिनिटांनी): "अजून शोध सुरू आहे, पुढचा update 00:30 ला".
Scribe (planning) — timeline ठेवते: काय दिसले, काय ठरले आणि काय केले, केव्हा. त्यावरूनच postmortem बनतो.
Incident घोषित करणे — "हे incident आहे, मी IC आहे" असे लवकर म्हणणे. छोटे incident घोषित करून मग बंद करणे स्वस्त आहे; उशिरा केलेली घोषणा महाग पडते.
Handoff — जेव्हा एखादी भूमिका दुसऱ्याकडे जाते (shift संपते, IC थकली आहे), तेव्हा जाणारी व्यक्ती सारांश देते — सध्याची स्थिती, काय करून पाहिले, बाकी असलेल्या कृती, कोणाकडे कोणती भूमिका — आणि येणारी व्यक्ती स्पष्टपणे "I have command" म्हणते. Acknowledge न केलेले handoff म्हणजे दोघींना वाटते की आपण नेतृत्व करतो, किंवा कोणीच करत नाही.
आधी mitigate — root cause शोधण्याआधी नुकसान थांबवा (roll back, fail over, load shed करा). कारण शोधणे postmortem चे काम आहे (Observability धडा 11).

🤔 का

कारण incident मध्ये अडचण बहुतेक वेळा बिघाड नसते — ती त्याच्या भोवतीचे लोक असतात: दुहेरी काम, परस्परविरोधी बदल, दर पाच मिनिटांनी "काही बातमी?" विचारणारे नेते, आणि कोणी घरी गेल्यावर हरवलेली माहिती. भूमिका आणि लय स्वस्त आहेत, आणि त्यांच्यामुळे तेच लोक ताणाखाली खूप चांगले काम करतात.

🔧 कसे (या repo मध्ये)

sre/crew.py मधील IncidentLog(events, update_every=30) events घेते (minute, kind, …) — role, update, handoff (role, from, to, acknowledged), leave, resolve — मिनिटे 23:00 पासून मोजलेली. check() त्याला सापडलेल्या अडचणी परत देते: कधीच नेमलेली नसलेली भूमिका, लयीपेक्षा मोठे दोन updates मधील अंतर, भूमिका हातात असतानाच निघून गेलेली व्यक्ती, acknowledge न केलेले handoff, ops सुद्धा करणारी IC. command() दोन्ही रात्री तपासते.

🧪 करून पाहा

python3 sre/demo.py command
python3 - <<'EOF'
import sys; sys.path.insert(0, "sre"); from crew import IncidentLog
log = IncidentLog([(0, "role", "Dipika", "IC"), (2, "role", "Dipika", "ops"), (3, "role", "Aishwarya", "comms"),
                   (4, "role", "Katrina", "scribe"), (5, "update"), (50, "update"),
                   (60, "handoff", "IC", "Dipika", "crew-5", False), (61, "leave", "Dipika"), (90, "resolve")])
probs, mins = log.check()
print(f"{mins} min, {len(probs)} problems:")
for p in probs: print(" -", p)
EOF

✅ तपासा — तुम्हाला काय दिसायला हवे

command हे छापते:

── without roles: the timetable service is down at 23:00 · resolved after 110 min · 5 problem(s)
   ✗ 00:00 Katrina went off shift still holding ops — nobody holds it now
   ✗ no IC was ever named
   ✗ no comms was ever named
   ✗ no scribe was ever named
   ✗ 23:00 no status update for 110 min (rhythm: every 30)
── with incident command: the timetable service is down at 23:00 · resolved after 100 min · 0 problem(s)

तुमचा snippet हे छापतो:

90 min, 6 problems:
 - 00:00 IC handoff Dipika → crew-5 never acknowledged — two people think they lead
 - 00:01 Dipika went off shift still holding IC — nobody holds it now
 - 00:01 Dipika went off shift still holding ops — nobody holds it now
 - 23:05 no status update for 45 min (rhythm: every 30)
 - 23:50 no status update for 40 min (rhythm: every 30)
 - the IC is also hands-on in ops — nobody is watching the whole incident

🏁 तुम्ही आत्ताच काय सिद्ध केले

भूमिकांची फक्त नावे असणे पुरेसे नाही. तुमच्या snippet मध्ये प्रत्येक भूमिका नेमली होती, आणि तरीही त्यात सहा अडचणी होत्या: IC दुरुस्तीही करत होती, updates 45 आणि 40 मिनिटांपर्यंत लांबले, आणि handoff कधीच acknowledge झाले नाही — म्हणून दीपिका गेल्यावर command किंवा ops कोणाकडेच नव्हते. शांत रात्र प्रत्येक नियमात उत्तीर्ण झाली, त्यात 00:00 चे handoff सुद्धा, जे crew-5 ने स्वीकारले. (या runs मधील resolve होण्याचे वेळ गोष्टीचा भाग आहेत, model ते गणत नाही.)

⚠️ नेहमीच्या चुका

सर्वात वरिष्ठ engineer IC होते आणि debugging सुद्धा सुरू करते
"कदाचित काहीच नसेल" म्हणून incident उशिरा घोषित करणे
बातमी असेल तेव्हाच status updates — शांततेमुळे नेते दुरुस्ती करणाऱ्यांना फोन करतात
handoff म्हणून "मी झोपायला जाते"
अनेक लोक एकाच वेळी production बदलतात, आणि कोणी काय बदलले ते कोणालाच माहीत नसते
scribe नाही, म्हणून postmortem ची timeline आठवणीतून आणि chat scrollback मधून पुन्हा बनवली जाते

🏭 प्रत्यक्ष वापरात

Handoff चा template सगळ्यांना सापडेल अशा ठिकाणी ठेवा (incident channel चा pinned message):

IC HANDOFF — timetable outage — 00:00
state:     timetable API 40% errors, database failover done at 23:40, errors falling
tried:     rolled back release 2026.09.27-2 (no change); failed over DB (helped)
next:      confirm replica lag < 5 s, then raise read traffic
roles:     IC crew-5 (from Dipika) · ops Katrina · comms Aishwarya (next update 00:30) · scribe crew-5
incoming:  "crew-5: I have command."

incident.io, PagerDuty, FireHydrant आणि Rootly सारखी tools incident channel तयार करतात, भूमिका वाटतात आणि chat मधून timeline ठेवतात. खऱ्या account वर — PagerDuty च्या API (Events API v2) ने command line वरून incident उघडा:

curl -s -X POST https://events.pagerduty.com/v2/enqueue -H "Content-Type: application/json" -d '{
  "routing_key": "'"$PD_ROUTING_KEY"'", "event_action": "trigger",
  "payload": {"summary": "timetable API 40% errors", "source": "timetable", "severity": "critical"}}'

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे आहे: या महिन्यात 30 मिनिटांचा tabletop exercise करा: एक काल्पनिक outage, नेमलेली IC, updates टाकणारी comms lead, आणि मध्येच एक handoff. पहिल्याच वेळी नेहमी एखादी उणीव सापडते.

⏭️ पुढे

भाग 3: अपयश गृहीत धरून design. आधी गणित: इमारतींची साखळी किती विश्वासार्ह असू शकते?

git checkout lesson-09-reliability-math

🦺 Lesson 08 — Incident command: roles, rhythm and the handoff

📍 You are here: Lesson 08 of 12 · Previous: lesson-07-capacity-planning · Next: lesson-09-reliability-math

📦 What's in this branch

Lesson 07, plus how the crew organises during an incident. The Observability school's lesson 10 teaches severity, the incident clocks (detect, acknowledge, mitigate, resolve) and its lessons 11–12 the root cause and the postmortem. This lesson is the command structure: the four roles, the status-update rhythm, and the handoff when a shift ends. IncidentLog in sre/crew.py and command() in sre/demo.py.

🧒 Explain like I'm 5

At 23:00 the timetable building goes dark. 🌑

Night one — no plan. Katrina and Dipika both start fixing things. They undo each other's work. Nobody tells the head teacher anything for almost two hours. At midnight Katrina goes home — and nobody knows what she was in the middle of.

Night two — a plan. Dipika says: "I am the incident commander. I don't fix — I lead." She points: "Katrina, you fix. Aishwarya, you tell people what is happening, every half hour. I'll also write down what happens, while it is small." At midnight Dipika's shift ends. She doesn't just leave. She tells crew-5 everything — what is broken, what was tried, who is doing what — and crew-5 says: "I have command." Only then does Dipika go.

Same fault. The second night is calm, and it ends sooner.

🗺️ Diagram

flowchart LR
    ic["🦺 IC — Dipika<br/>leads, decides, does NOT fix"] --> ops["🔧 ops — Katrina<br/>hands on the system"]
    ic --> comms["📣 comms — Aishwarya<br/>update every 30 min"]
    ic --> scribe["📝 scribe<br/>the timeline"]
    ic -->|"00:00 handoff:<br/>state, actions, roles"| ic2["🦺 IC — crew-5<br/>'I have command' ✓"]
    bad["without roles: 5 problems<br/>no IC · 110 min with no update<br/>Katrina left holding ops"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/sre/lesson-diagrams.html#l08

❓ What

Incident Command System (ICS) — a structure created for fighting wildfires in California in the 1970s, now used by emergency services. Google's incident process (IMAG), PagerDuty's published process and many others are based on it.
Incident commander (IC) — owns the incident: sets priorities, assigns roles, decides (for example "roll back now"), and keeps the big picture. The IC should not also be typing commands — then nobody watches the whole incident. In a tiny incident, one person may hold several roles, but say so out loud.
Operations lead (ops) — the people changing the system: investigating, mitigating. Only ops changes production during the incident.
Communications lead (comms) — sends regular status updates to users, support and leaders, so they stop interrupting the people fixing things. A fixed rhythm (every 30 minutes here) even when there is no news: "still investigating, next update 00:30".
Scribe (planning) — keeps the timeline: what was seen, decided and done, when. The postmortem is built from it.
Declaring an incident — saying "this is an incident, I am IC" early. Declaring and then closing a small one is cheap; a late declaration is expensive.
Handoff — when a role changes hands (a shift ends, the IC is tired), the outgoing person gives a summary — current state, what was tried, open actions, who holds each role — and the incoming one says clearly "I have command". An unacknowledged handoff means two people think they lead, or nobody does.
Mitigate first — stop the harm (roll back, fail over, shed load) before finding the root cause. The cause is for the postmortem (Observability lesson 11).

🤔 Why

Because in an incident, the problem is often not the fault — it is the people around it: duplicated work, conflicting changes, leaders asking "any news?" every five minutes, and knowledge lost when someone goes home. Roles and a rhythm are cheap, and they make the same people work much better under stress.

🔧 How (in this repo)

IncidentLog(events, update_every=30) in sre/crew.py takes events (minute, kind, …) — role, update, handoff (role, from, to, acknowledged), leave, resolve — minutes counted from 23:00. check() returns the problems it finds: a role never named, a gap between updates longer than the rhythm, a person who left still holding a role, an unacknowledged handoff, the IC also doing ops. command() checks the two nights.

🧪 Try it

python3 sre/demo.py command
python3 - <<'EOF'
import sys; sys.path.insert(0, "sre"); from crew import IncidentLog
log = IncidentLog([(0, "role", "Dipika", "IC"), (2, "role", "Dipika", "ops"), (3, "role", "Aishwarya", "comms"),
                   (4, "role", "Katrina", "scribe"), (5, "update"), (50, "update"),
                   (60, "handoff", "IC", "Dipika", "crew-5", False), (61, "leave", "Dipika"), (90, "resolve")])
probs, mins = log.check()
print(f"{mins} min, {len(probs)} problems:")
for p in probs: print(" -", p)
EOF

✅ Verify — what you should see

command prints:

── without roles: the timetable service is down at 23:00 · resolved after 110 min · 5 problem(s)
   ✗ 00:00 Katrina went off shift still holding ops — nobody holds it now
   ✗ no IC was ever named
   ✗ no comms was ever named
   ✗ no scribe was ever named
   ✗ 23:00 no status update for 110 min (rhythm: every 30)
── with incident command: the timetable service is down at 23:00 · resolved after 100 min · 0 problem(s)

Your snippet prints:

90 min, 6 problems:
 - 00:00 IC handoff Dipika → crew-5 never acknowledged — two people think they lead
 - 00:01 Dipika went off shift still holding IC — nobody holds it now
 - 00:01 Dipika went off shift still holding ops — nobody holds it now
 - 23:05 no status update for 45 min (rhythm: every 30)
 - 23:50 no status update for 40 min (rhythm: every 30)
 - the IC is also hands-on in ops — nobody is watching the whole incident

🏁 What you just proved

Having the names of roles is not enough. In your snippet every role was named, and it still had six problems: the IC was also fixing, updates slipped to 45 and 40 minutes, and the handoff was never acknowledged — so when Dipika left, nobody held command or ops. The calm night passed every rule, including a handoff at 00:00 that crew-5 accepted. (The resolve times in these runs are part of the story, not something the model computes.)

⚠️ Common mistakes

the most senior engineer becomes IC and also starts debugging
declaring the incident late, "because it might be nothing"
status updates only when there is news — silence makes leaders call the fixers
"I'm going to bed" as a handoff
several people changing production at once, with nobody knowing who changed what
no scribe, so the postmortem timeline is rebuilt from memory and chat scrollback

🏭 In production

Keep a handoff template where everyone can find it (the incident channel's pinned message):

IC HANDOFF — timetable outage — 00:00
state:     timetable API 40% errors, database failover done at 23:40, errors falling
tried:     rolled back release 2026.09.27-2 (no change); failed over DB (helped)
next:      confirm replica lag < 5 s, then raise read traffic
roles:     IC crew-5 (from Dipika) · ops Katrina · comms Aishwarya (next update 00:30) · scribe crew-5
incoming:  "crew-5: I have command."

Tools like incident.io, PagerDuty, FireHydrant and Rootly create the incident channel, assign roles and keep the timeline from chat. On a real account — open an incident from the command line with PagerDuty's API (Events API v2):

curl -s -X POST https://events.pagerduty.com/v2/enqueue -H "Content-Type: application/json" -d '{
  "routing_key": "'"$PD_ROUTING_KEY"'", "event_action": "trigger",
  "payload": {"summary": "timetable API 40% errors", "source": "timetable", "severity": "critical"}}'

🏭 Why this matters in production: run a 30-minute tabletop exercise this month: a made-up outage, a named IC, a comms lead posting updates, and a handoff in the middle. The first one always finds a gap.

⏭️ Next

Part 3: designing for failure. First, the arithmetic: how reliable can a chain of buildings be?

git checkout lesson-09-reliability-math

← Previouscapacity planning Next →reliability math

This page is the lesson's README from the lesson-08-incident-command branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.