🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🚨 धडा 10 — Incidents आणि कठीण निर्णयांमध्ये नेतृत्व

📍 तुम्ही इथे आहात: 12 पैकी धडा 10 · मागचा: lesson-09-team-topologies · पुढचा: lesson-11-writing

📦 या ब्रँचमध्ये काय आहे

धडा 09, आणि असे क्षण जेव्हा lead सर्वात जास्त दिसते: एखादे incident, आणि एखादा कठीण निर्णय. उलटता येणारे विरुद्ध न उलटता येणारे निर्णय (two-way आणि one-way doors), revisit तारखांसह निर्णयांची नोंदवही (decision log), incident roles, आणि पुढे debug करत राहण्यापेक्षा आधी mitigate करणे का चांगले. incidents() lead/demo.py मध्ये; classify(), DecisionLog आणि timeline() lead/models.py मध्ये. Operations ची बाजू SRE school मध्ये आहे.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

परीक्षेची सकाळ. काल रात्री update केलेले नवे online gradebook प्रत्येक विद्यार्थ्याचे गुण शून्य दाखवते. 😱 पालकांचे फोन येत आहेत.

सगळेजण bug शोधायला screen भोवती गर्दी करतात. कतरिना त्यांना थांबवते. ती प्रत्येकीला एकच काम देते:

कतरिना प्रमुख आहे. ती स्वतः काहीही दुरुस्त करत नाही — ती सगळ्यांना एकाच दिशेने ठेवते. 🧭
दीपिका कालची आवृत्ती परत लावते. "bug शोधते" नाही — जी चालत होती ती परत लावते. ⏪
ऐश्वर्या पालकांना आणि मुख्याध्यापिकेला काय चालले आहे ते दर 15 मिनिटांनी सांगते. 📣

कालची आवृत्ती 15 मिनिटांत परत येते. Bug नंतर शांतपणे शोधता येतो. जर त्यांनी आधी bug शोधला असता, तर पालकांना 52 मिनिटे वाट पाहावी लागली असती.

त्याच आठवड्यात नंतर कतरिना दोन निर्णय घेते. "Standup ची नवी पद्धत करून पाहायची?" उलटणे सोपे — ती एका मिनिटात ठरवते. "गुण देण्याच्या tool साठी 3 वर्षांचा contract करायचा?" उलटणे कठीण — ती ते लिहून काढते, लोकांना विचारते, आणि एक आठवडा घेते. वेगळे दरवाजे, वेगळे वेग. 🚪

🗺️ आकृती

flowchart LR
    i["🚨 impact starts, minute 0"] --> d["🔔 detected, minute 6"]
    d --> r["⏪ mitigate first: roll back<br/>users hurt 15 min"]
    d --> f["🔍 debug forward: find and fix<br/>users hurt 52 min"]
    t["🚪 two-way door<br/>decide now, log it, revisit"] --- o["🔒 one-way door<br/>write it up, consult, deadline"]
    l["📒 decision log<br/>due by week 8: standup format,<br/>exam-week freeze"]

🗺️ काढलेली आवृत्ती + एक lab: https://school-edh.pages.dev/engineering-leadership/lesson-diagrams.html#l10

❓ काय

One-way आणि two-way doors — Jeff Bezos यांच्या Amazon shareholders ना लिहिलेल्या 2015 च्या पत्रात यांना Type 1 आणि Type 2 निर्णय म्हटले आहे. Two-way doors मधून स्वस्तात परत येता येते: प्रश्नाच्या सर्वात जवळ असलेल्या व्यक्तीसोबत पटकन ठरवा. One-way doors उलटणे कठीण किंवा अशक्य असते: सावकाश जा, लिहून काढा, सल्ला घ्या. नेहमीची चूक म्हणजे प्रत्येक निर्णयाला one-way समजणे, आणि सगळीकडेच हळू चालणे.
उलटता येणे ही एक श्रेणी आहे — "reversible" चा खरा अर्थ "उलटण्याइतके स्वस्त". हा lab एक ढोबळ मर्यादा वापरतो (reversible आणि 5 दिवसांत उलटता येणारे); तुमची स्वतःची निवडा.
निर्णयांची नोंदवही (decision log) — प्रत्येक निर्णयाची छोटी नोंद: तारीख, काय ठरले, विचारात घेतलेले पर्याय, कोणी ठरवले, आणि पुन्हा कधी पाहायचे. Architecture decision records (ADRs) हा त्याचा एक सुप्रसिद्ध प्रकार आहे.
Incident roles — आपत्कालीन सेवांच्या incident command system वरून घेतलेले आणि Google च्या SRE book मध्ये वर्णन केलेले: एक incident commander (समन्वय करते आणि ठरवते, debug करत नाही), एक operations lead (system बदलते), एक communications lead (ठरलेल्या वेळापत्रकाने लोकांना update देते). छोट्या टीम्स roles एकत्र करतात, पण त्यांना नाव दिल्याने पाच जण debug करत आहेत आणि users शी कोणीच बोलत नाही अशी स्थिती टळते.
आधी mitigate करा — service पूर्ववत करा (roll back, fail over, feature flag बंद करा), मग कारण शोधा. एखाद्या बदलामुळे समस्या आली असेल तर roll back हा सहसा सर्वात जलद सुरक्षित उपाय असतो — अर्थात rollback शक्य असेल तर.
Time to detect / time to mitigate — users वर परिणाम सुरू झाल्यापासून मोजलेले.
Blameless review (postmortem) — दोषारोपविरहित आढावा: failure कशामुळे शक्य झाले आणि ते कमी शक्य कसे करायचे हे विचारतो, दोष कोणाला द्यायचा हे नाही. दोषारोपाच्या संस्कृतीत लोक समस्या लपवतात.

🤔 का

कारण incident मध्ये lead ची सहज प्रवृत्ती — "इथे मीच सर्वोत्तम engineer आहे, मला पाहू दे" — सहसा चुकीची असते: टीमला keyboard वर आणखी एक जोडी हात नको, तर समन्वय करणारी कोणीतरी हवी. आणि बहुतेक निर्णय two-way doors असल्याने, lead तिच्या टीमला देऊ शकणारा सर्वात मोठा वेग म्हणजे हे स्पष्ट सांगणे आणि मालकाला ठरवू देणे — आणि जे काही थोडे निर्णय तसे नसतात त्यांवर खरी काळजी घेणे.

🔧 कसे (या repo मध्ये)

lead/models.py मधील classify(reversible, cost_to_undo_days) निर्णय reversible असेल आणि 5 दिवसांत उलटता येत असेल तर "two-way door" परत देते, नाहीतर "one-way door". DecisionLog (week, decision, owner, door, revisit) नोंदवते आणि due(week) ज्यांचा revisit आठवडा आला आहे असे निर्णय दाखवते. timeline(events) परिणाम सुरू झाल्यापासून detect आणि mitigate करायला लागलेली मिनिटे परत देते. lead/demo.py मधील incidents() चार निर्णयांचे वर्गीकरण करते आणि incident ला दिलेल्या दोन प्रतिसादांची तुलना करते. 5 दिवसांची रेषा मनमानी आहे: विचार करण्याचे साधन, लोकांसाठीचे सूत्र नव्हे.

🧪 करून पाहा

python3 lead/demo.py incidents
python3 - <<'EOF'
import sys; sys.path.insert(0, "lead"); from models import classify, DecisionLog, timeline
for rev, undo in ((True, 0), (True, 5), (True, 6), (False, 1)):
    print(f"reversible={rev!s:<5} undo {undo} days → {classify(rev, undo).split(' →')[0]}")
log = DecisionLog()
log.add(3, "new standup format", "Dipika", "two-way", 5)
log.add(3, "PostgreSQL migration RFC accepted", "Katrina", "one-way", 12)
log.add(4, "freeze releases in exam week", "Aishwarya", "two-way", 8)
for w in (4, 5, 8, 12):
    print(f"week {w:>2}: due for review {log.due(w)}")
for det in (2, 6, 20):
    print(f"alert fires after {det:>2} min, rollback takes 9 → users hurt for {timeline({'impact': 0, 'detected': det, 'mitigated': det + 9})[1]} min")
EOF

✅ तपासा — तुम्हाला काय दिसायला हवे

incidents हे छापते:

── roll back last night's gradebook release     two-way door → decide now, the owner decides, log it, set a revisit date
── move the gradebook from MySQL to PostgreSQL  one-way door → slow down: write it up, consult, name a decider and a deadline
   decision log: 3 entries · due for review by week 8: ['new standup format', 'freeze releases in exam week']
── incident, mitigate first (roll back)    detected after 6 min · users hurt for 15 min
── incident, debug forward (find and fix)  detected after 6 min · users hurt for 52 min

तुमचा snippet हे छापतो:

reversible=True  undo 0 days → two-way door
reversible=True  undo 5 days → two-way door
reversible=True  undo 6 days → one-way door
reversible=False undo 1 days → one-way door
week  4: due for review []
week  5: due for review ['new standup format']
week  8: due for review ['new standup format', 'freeze releases in exam week']
week 12: due for review ['new standup format', 'PostgreSQL migration RFC accepted', 'freeze releases in exam week']
alert fires after  2 min, rollback takes 9 → users hurt for 11 min
alert fires after  6 min, rollback takes 9 → users hurt for 15 min
alert fires after 20 min, rollback takes 9 → users hurt for 29 min

🏁 तुम्ही आत्ताच काय सिद्ध केले

आधी roll back केल्याने users ना 15 मिनिटे त्रास झाला; पुढे debug करत राहिल्यास 52. Detection चा वेळ थेट नुकसानात भर घालतो: तोच 9 मिनिटांचा rollback 2 मिनिटांच्या alert सह 11 मिनिटांचा परिणाम देतो आणि 20 मिनिटांच्या alert सह 29. निर्णयांची नोंदवही "बघू कसे होते" ला एका तारखेत बदलते: आठवडा 8 पर्यंत दोन निर्णयांचा आढावा घ्यायचा आहे, आणि one-way door ला आठवडा 12 मध्ये त्याची स्वतःची तारीख आहे. आणि दरवाजा ही तुम्ही निवडलेली मर्यादा आहे — 5 दिवसांत उलटता येणारा "two-way" होता, 6 दिवसांचा नाही.

⚠️ नेहमीच्या चुका

lead debug करत आहे आणि कोणीच समन्वय करत नाही किंवा users ना update देत नाही
users ना अजून त्रास होत असताना root cause शोधत बसणे
कोणीही test न केलेला rollback मार्ग
प्रत्येक निर्णयाला one-way door समजणे, म्हणजे टीम प्रत्येक गोष्टीसाठी lead ची वाट पाहते
one-way door ला (contract, data migration, hire) झटपट निर्णय समजणे
meeting मध्ये घेतलेले आणि कधीच न लिहिलेले निर्णय — सहा महिन्यांनंतर का ते कोणालाच माहीत नसते
शेवटी एखाद्या नावानेच संपणारा "blameless" review

🏭 प्रत्यक्ष वापरात

एक decision record, 10 मिनिटांत लिहिता येईल इतके छोटे:

# 2026-09-14 — Freeze gradebook releases in exam week
- Door: two-way (we can lift it any day)
- Decider: Aishwarya (release owner) · Consulted: Katrina, Dipika
- Options: (1) no freeze (2) freeze Mon–Fri of exam week (3) freeze two weeks
- Decision: (2). Reason: last year's exam-week incident; one week limits the backlog
- Revisit: week 8, after the first exam week

Mitigation एका command ची असेल तेव्हा सर्वात सोपी असते. खऱ्या cluster वर, उदाहरणार्थ (Kubernetes school हे समजावते):

kubectl rollout undo deployment/gradebook      # back to the previous ReplicaSet
kubectl rollout status deployment/gradebook

Incident channel चा पहिला message roles ठरवतो:

🚨 INC-231 · gradebook shows 0 for all marks · started ~09:02
IC: Katrina · Ops: Dipika (rolling back) · Comms: Aishwarya (next update 09:30)

Service-level objectives, error budgets, on-call आणि postmortems हे SRE school चे विषय आहेत — हा धडा त्याच क्षणांची lead ची बाजू आहे.

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: समन्वय कोण करणार, roll back कसे करायचे, आणि लोकांना किती वेळाने update द्यायचा हे आधीच ठरवा. आणि प्रत्येक निर्णय त्याच्या दरवाजानुसार वर्गीकृत करा: अनेक two-way निर्णयांसाठी जलद, मोजक्या one-way निर्णयांसाठी सावकाश आणि लिखित.

⏭️ पुढे

निर्णयांची नोंदवही हे एक लेखनच होते. Lead मोठ्या प्रमाणावर जे काही चांगले करते ते जवळजवळ सगळे असेच असते. पुढे: लेखन — design docs, status updates, आणि एका meeting ची किंमत.

git checkout lesson-11-writing

🚨 Lesson 10 — Leading through incidents and hard decisions

📍 You are here: Lesson 10 of 12 · Previous: lesson-09-team-topologies · Next: lesson-11-writing

📦 What's in this branch

Lesson 09, plus the moments when a lead is most visible: an incident, and a hard decision. Reversible vs irreversible decisions (two-way and one-way doors), a decision log with revisit dates, incident roles, and why mitigating first beats debugging forward. incidents() in lead/demo.py; classify(), DecisionLog and timeline() in lead/models.py. The operations side lives in the SRE school.

🧒 Explain like I'm 5

Exam morning. The new online gradebook, updated last night, shows every pupil's mark as zero. 😱 Parents are phoning.

Everyone crowds around the screen trying to find the bug. Katrina stops them. She gives each person one job:

Katrina is in charge. She does not fix anything herself — she keeps everyone pointed the same way. 🧭
Dipika puts back yesterday's version. Not "finds the bug" — puts back the one that worked. ⏪
Aishwarya tells the parents and the head teacher what is happening, every 15 minutes. 📣

Yesterday's version is back in 15 minutes. The bug can be found calmly afterwards. If they had hunted for the bug first, parents would have waited 52 minutes.

Later that week Katrina makes two decisions. "Try a new standup format?" Easy to undo — she decides in a minute. "Sign a 3-year contract for a marking tool?" Hard to undo — she writes it up, asks people, and takes a week. Different doors, different speeds. 🚪

🗺️ Diagram

flowchart LR
    i["🚨 impact starts, minute 0"] --> d["🔔 detected, minute 6"]
    d --> r["⏪ mitigate first: roll back<br/>users hurt 15 min"]
    d --> f["🔍 debug forward: find and fix<br/>users hurt 52 min"]
    t["🚪 two-way door<br/>decide now, log it, revisit"] --- o["🔒 one-way door<br/>write it up, consult, deadline"]
    l["📒 decision log<br/>due by week 8: standup format,<br/>exam-week freeze"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/engineering-leadership/lesson-diagrams.html#l10

❓ What

One-way and two-way doors — Jeff Bezos's 2015 letter to Amazon shareholders called them Type 1 and Type 2 decisions. Two-way doors can be walked back through cheaply: decide fast, with the person closest to the problem. One-way doors are hard or impossible to undo: slow down, write it up, consult. The common failure is treating every decision as one-way, and moving slowly everywhere.
Reversibility is a spectrum — "reversible" really means "cheap enough to undo". This lab uses a crude threshold (reversible and undoable within 5 days); pick your own.
Decision log — a short record per decision: date, what was decided, options considered, who decided, and when to revisit. Architecture decision records (ADRs) are one well-known form.
Incident roles — adapted from emergency services' incident command system and described in Google's SRE book: an incident commander (coordinates and decides, does not debug), an operations lead (changes the system), a communications lead (updates people on a schedule). Small teams combine roles, but naming them avoids five people debugging and nobody talking to users.
Mitigate first — restore service (roll back, fail over, turn a feature flag off), then find the cause. Rolling back is usually the fastest safe move when a change caused the problem — when a rollback is possible at all.
Time to detect / time to mitigate — measured from the start of user impact.
Blameless review (postmortem) — asks what made the failure possible and how to make it less likely, not who to blame. People hide problems in a culture of blame.

🤔 Why

Because in an incident, a lead's instinct — "I am the best engineer here, let me look" — is usually wrong: the team needs someone coordinating, not another pair of hands on the keyboard. And because most decisions are two-way doors, the biggest speed gain a lead can give her team is to say so, and let the owner decide — while spending real care on the few that are not.

🔧 How (in this repo)

classify(reversible, cost_to_undo_days) in lead/models.py returns "two-way door" if the decision is reversible and can be undone within 5 days, and "one-way door" otherwise. DecisionLog records (week, decision, owner, door, revisit) and due(week) lists decisions whose revisit week has come. timeline(events) returns minutes to detect and to mitigate from the start of impact. incidents() in lead/demo.py classifies four decisions and compares two incident responses. The 5-day line is arbitrary: a thinking tool, not a formula for people.

🧪 Try it

python3 lead/demo.py incidents
python3 - <<'EOF'
import sys; sys.path.insert(0, "lead"); from models import classify, DecisionLog, timeline
for rev, undo in ((True, 0), (True, 5), (True, 6), (False, 1)):
    print(f"reversible={rev!s:<5} undo {undo} days → {classify(rev, undo).split(' →')[0]}")
log = DecisionLog()
log.add(3, "new standup format", "Dipika", "two-way", 5)
log.add(3, "PostgreSQL migration RFC accepted", "Katrina", "one-way", 12)
log.add(4, "freeze releases in exam week", "Aishwarya", "two-way", 8)
for w in (4, 5, 8, 12):
    print(f"week {w:>2}: due for review {log.due(w)}")
for det in (2, 6, 20):
    print(f"alert fires after {det:>2} min, rollback takes 9 → users hurt for {timeline({'impact': 0, 'detected': det, 'mitigated': det + 9})[1]} min")
EOF

✅ Verify — what you should see

incidents prints:

── roll back last night's gradebook release     two-way door → decide now, the owner decides, log it, set a revisit date
── move the gradebook from MySQL to PostgreSQL  one-way door → slow down: write it up, consult, name a decider and a deadline
   decision log: 3 entries · due for review by week 8: ['new standup format', 'freeze releases in exam week']
── incident, mitigate first (roll back)    detected after 6 min · users hurt for 15 min
── incident, debug forward (find and fix)  detected after 6 min · users hurt for 52 min

Your snippet prints:

reversible=True  undo 0 days → two-way door
reversible=True  undo 5 days → two-way door
reversible=True  undo 6 days → one-way door
reversible=False undo 1 days → one-way door
week  4: due for review []
week  5: due for review ['new standup format']
week  8: due for review ['new standup format', 'freeze releases in exam week']
week 12: due for review ['new standup format', 'PostgreSQL migration RFC accepted', 'freeze releases in exam week']
alert fires after  2 min, rollback takes 9 → users hurt for 11 min
alert fires after  6 min, rollback takes 9 → users hurt for 15 min
alert fires after 20 min, rollback takes 9 → users hurt for 29 min

🏁 What you just proved

Rolling back first kept users hurt for 15 minutes; debugging forward, 52. Detection time adds directly to the damage: the same 9-minute rollback costs 11 minutes of impact with a 2-minute alert and 29 with a 20-minute one. The decision log turns "we'll see how it goes" into a date: by week 8, two decisions are due for review, and the one-way door has its own date in week 12. And the door is a threshold you choose — undoable in 5 days was "two-way", 6 days was not.

⚠️ Common mistakes

the lead debugging while nobody coordinates or updates users
hunting for the root cause while users are still hurt
a rollback path nobody has tested
treating every decision as a one-way door, so the team waits on the lead for everything
treating a one-way door (a contract, a data migration, a hire) as a quick call
decisions made in a meeting and never written down — six months later nobody knows why
a "blameless" review that still ends with a name

🏭 In production

A decision record, short enough to write in 10 minutes:

# 2026-09-14 — Freeze gradebook releases in exam week
- Door: two-way (we can lift it any day)
- Decider: Aishwarya (release owner) · Consulted: Katrina, Dipika
- Options: (1) no freeze (2) freeze Mon–Fri of exam week (3) freeze two weeks
- Decision: (2). Reason: last year's exam-week incident; one week limits the backlog
- Revisit: week 8, after the first exam week

Mitigation is easiest when it is one command. On a real cluster, for example (the Kubernetes school explains it):

kubectl rollout undo deployment/gradebook      # back to the previous ReplicaSet
kubectl rollout status deployment/gradebook

An incident channel's first message sets the roles:

🚨 INC-231 · gradebook shows 0 for all marks · started ~09:02
IC: Katrina · Ops: Dipika (rolling back) · Comms: Aishwarya (next update 09:30)

Service-level objectives, error budgets, on-call and postmortems are the SRE school's subject — this lesson is the lead's side of the same moments.

🏭 Why this matters in production: decide in advance who coordinates, how to roll back, and how often to update people. And sort every decision by its door: fast for the many two-way ones, slow and written for the few one-way ones.

⏭️ Next

The decision log was a piece of writing. So is almost everything a lead does well at scale. Next: writing — design docs, status updates, and the cost of a meeting.

git checkout lesson-11-writing

← Previousteam topologies Next →writing

This page is the lesson's README from the lesson-10-incidents-decisions branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.