🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

📝 धडा 12 — Postmortems आणि चक्र: शाळा अधिक सुरक्षित करणारा अहवाल

📍 तुम्ही इथे आहात: 12 पैकी धडा 12 · मागील: lesson-11-root-cause

📦 या ब्रँचमध्ये काय आहे

धडे 01–11, आणि चक्र पूर्ण करणारी पायरी: blameless postmortem — impact, timeline, root cause आणि contributing factors, आणि owner आणि date असलेले action items — आणि signals पासून परत अधिक चांगल्या signals पर्यंतचे चक्र. postmortem() obs/reliability.py मध्ये आहे; obs/demo.py मधले postmortem_demo() (section postmortem) निकालाच्या दिवसाचा अहवाल लिहिते.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

सरावानंतर एका आठवड्याने, दीपिका बैठक बोलावते आणि एक अहवाल 📝 लिहिते. त्याचे चार भाग आहेत:

विद्यार्थ्यांचे काय झाले — किती जणांना, किती वेळ.
घड्याळ — minute-minute ने, clipboard वरून.
का — धडा 11 मधले पाच का.
आम्ही काय बदलू — आणि प्रत्येक बदलासमोर एक नाव आणि एक तारीख: "कतरिना — प्रत्येक toaster ला timer — शुक्रवारपर्यंत."

अहवालाच्या वर मोठ्या अक्षरांत एक नियम आहे: "आम्ही शाळेकडे पाहतो, एखाद्या व्यक्तीकडे नाही." कारण खरे सांगितल्याबद्दल लोक अडचणीत आले, तर पुढच्या वेळी कोणीच खरे सांगणार नाही — आणि toaster पुन्हा जळेल.

मग अहवाल सगळ्यांना वाचता यावा म्हणून staff-room च्या भिंतीवर लावला जातो. आणि दर शुक्रवारी, दीपिका यादी तपासते: झाले, की अजून नाही? बदलांवर कुणाचेच नाव नसलेला अहवाल म्हणजे फक्त एक गोष्ट.

🗺️ आकृती

flowchart LR
    sig["📊📓🗺️ signals<br/>L02–L07"] --> al["🚨 alert on the SLO<br/>L08–L09"]
    al --> inc["🧯 incident<br/>L10"]
    inc --> rc["🔍 root cause<br/>L11"]
    rc --> pm["📝 blameless postmortem<br/>impact · timeline · cause"]
    pm --> ai["✅ action items<br/>owner + date"]
    ai -->|"new tests, guards, alerts"| sig

🗺️ काढलेली आकृती + एक lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l12

❓ काय

Postmortem (incident review, post-incident review) — incident ची लेखी नोंद: काय घडले, impact, timeline, causes, आणि काय बदलणार. प्रत्येक SEV1 आणि SEV2 साठी, error budget चा मोठा भाग वापरणाऱ्या कोणत्याही incident साठी (धडा 09), आणि थोडक्यात टळलेल्या रोचक घटनांसाठी एक लिहा.
Blameless (दोषारोपविरहित) — review systems आणि processes कडे पाहतो, एखाद्या व्यक्तीच्या चुकीकडे नाही. तो गृहीत धरतो की त्या वेळी असलेल्या माहिती आणि tools सह प्रत्येकाने योग्य रीतीने वागले. मग लोक खरोखर काय घडले ते सांगतात — स्वतःच्या चुकांसह — खऱ्या पोकळ्या शोधण्याचा हाच एकमेव मार्ग आहे. Blameless म्हणजे कोणीच जबाबदार नाही असे नाही; याचा अर्थ fix system मध्ये जातो.
चांगल्या postmortem चे भाग:
- summary — कोणालाही वाचता येतील अशी दोन-तीन वाक्ये;
- impact — कोणावर परिणाम झाला, किती जणांवर, किती वेळ, किती error budget वापरला;
- timeline — बदल, detection, acknowledgement, mitigation आणि resolution च्या वेळा (धडा 10 ची घड्याळे), scribe च्या नोंदींमधून घेतलेल्या;
- root cause आणि contributing factors — धडा 11 मधून;
- काय चांगले झाले, काय वाईट झाले, कुठे आपण नशीबवान ठरलो;
- action items.
Action items — प्रत्येक नेमका, एका owner सह (एक व्यक्ती, team नाही) आणि एका due date सह, इतर कामासारख्याच system मध्ये track केलेला. चांगले items system बदलतात:
- prevent — एक test, एक संरक्षण, checklist ची एक पायरी (निकालाच्या दिवसाच्या data सह load test);
- detect — नवा किंवा अधिक जलद alert, नवा SLI;
- mitigate — अधिक जलद किंवा automatic rollback (burn-rate auto-rollback सह canary). "अधिक काळजी घ्या" हा action item नाही.
Review आणि share — एक छोटी review बैठक, मग postmortem अशा ठिकाणी प्रकाशित करा जिथे प्रत्येक team तो वाचू शकेल. उघडे items पूर्ण होईपर्यंत track करा; किती उशीर झालेले आहेत ते मोजा.
चक्र — signals → SLO वर alerts → incident → root cause → postmortem → action items → अधिक चांगले signals. प्रत्येक outage पुढचे outage लहान किंवा अशक्य करते.

🤔 का

कारण incident महाग असते, आणि postmortem हा त्या पैशाच्या बदल्यात काहीतरी मिळवण्याचा मार्ग आहे. त्याशिवाय, धडे तीन लोकांच्या डोक्यात राहतात आणि ते गेले की तेही जातात. Owners आणि dates नसतील, तर action items कोणीच न वाचणारी यादी बनतात, आणि तेच outage तीन महिन्यांत परत येते. "Blameless" नसेल, तर लोक थोडक्यात टळलेल्या घटना सांगणे बंद करतात — जे सगळ्यात स्वस्त धडे असतात.

🔧 कसे (या repo मध्ये)

obs/reliability.py मधले postmortem(title, impact, timeline, root_cause, actions) एक Markdown document देते: title, blameless ओळ, Impact, Timeline, Root cause आणि Action items, प्रत्येक action - [owner] what (by due) असे लिहिलेले. obs/demo.py मधले postmortem_demo() bad-deploy चा अहवाल लिहिते: 32 मिनिटे, सुमारे 1,900 fail झालेले views, कतरिना, ऐश्वर्या आणि दीपिका यांच्यासाठी तीन action items. शेवटी ते चक्र print करते.

🧪 करून पाहा

ज्या incident साठी कुणालाच page झाले नाही त्याचा postmortem लिहा — धडा 11 ने cache TTL बदलाशी जोडलेला slow leak — खऱ्या dates सह:

python3 obs/demo.py postmortem
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from reliability import postmortem
print(postmortem("slow results page from a flaky grade dependency",
                 "0.9% of results views failed for 280 minutes (minutes 100–379); no page, one ticket at minute 266",
                 ["090 config: cache TTL 60 s → 5 s", "100 errors rise to 0.9%", "266 burn-rate ticket opened",
                  "380 dependency recovered; errors back to 0.1%"],
                 "a 5 s cache TTL sent 12× more calls to a grade service that could not take them",
                 [("alert on the grade service's own SLO", "Aishwarya", "2026-10-02"),
                  ("config changes go through the same canary as deploys", "Dipika", "2026-10-09")]))
EOF
python3 obs/test_obs.py

✅ तपासा — तुम्हाला काय दिसायला हवे

postmortem हे print करते:

   # Postmortem: results page 504s on results day

   Blameless: we look at the system, not at a person.

   ## Impact
   6% of results views failed for 32 minutes (minutes 400–431); ~1,900 parents saw an error

   ## Timeline
   - 398 deploy v42
   - 409 burn-rate page fired
   - 411 Dipika acknowledged, took command
   - 432 rollback to v41 — errors back to normal
   - 470 resolved; grade cache restored

   ## Root cause
   v42 removed the grade cache; the release load test did not match results-day traffic

   ## Action items
   - [Katrina] restore the grade cache with a test that fails without it (by 2 days)
   - [Aishwarya] results-day load test in the release checklist (by 1 week)
   - [Dipika] canary 5% for 15 min with burn-rate auto-rollback (by 2 weeks)
── the loop: signals → alerts on the SLO → incident → root cause → postmortem → action items → better signals

तुमचा snippet हे print करतो:

# Postmortem: slow results page from a flaky grade dependency

Blameless: we look at the system, not at a person.

## Impact
0.9% of results views failed for 280 minutes (minutes 100–379); no page, one ticket at minute 266

## Timeline
- 090 config: cache TTL 60 s → 5 s
- 100 errors rise to 0.9%
- 266 burn-rate ticket opened
- 380 dependency recovered; errors back to 0.1%

## Root cause
a 5 s cache TTL sent 12× more calls to a grade service that could not take them

## Action items
- [Aishwarya] alert on the grade service's own SLO (by 2026-10-02)
- [Dipika] config changes go through the same canary as deploys (by 2026-10-09)

Tests ✅ L12 the postmortem is blameless and has action items with owners आणि 12/12 passed ने संपतात.

🏁 तुम्ही आत्ताच काय सिद्ध केले

प्रत्येक action item मध्ये एका व्यक्तीचे नाव आणि एक deadline आहे, आणि प्रत्येक item system बदलतो: एक test, checklist मध्ये load test, canary, alert, config बदलांसाठी एक नियम. कोणताच "काळजी घ्या" म्हणत नाही. तुमचा दुसरा अहवाल चक्र चालत असल्याचे दाखवतो: slow leak ने कधीच कुणाला उठवले नाही (धडा 08 ने त्याचे काम केले), तरीही त्याची किंमत 2,520 fail झालेले views होती — bad deploy च्या ~1,920 पेक्षा जास्त — म्हणून त्याचाही postmortem व्हायला हवा, आणि त्याचे fixes अधिक चांगले signals बनतात: एक नवा SLO आणि config साठी canary. 2026-10-02 सारखी तारीख तपासता येते; demo च्या अहवालातले "2 days" फक्त तो लिहिला त्या दिवशीच स्पष्ट असते.

⚠️ नेहमीच्या चुका

"root cause: human error", "action item: engineer ला पुन्हा प्रशिक्षण द्या"
"the team" च्या मालकीचे, किंवा date नसलेले action items — ते कधीच बंद होत नाहीत
लिहिलेला, एकदा review केलेला, आणि पुन्हा कधीच न वाचलेला postmortem; items सामान्य कामासोबत track न करणे
फक्त मोठ्या outages साठी postmortems, त्यामुळे थोडक्यात टळलेल्या घटना (स्वस्त धडे) हरवतात
scribe च्या नोंदी आणि graphs ऐवजी एका आठवड्यानंतर आठवणीतून बनवलेली timeline
timeline मधली नावे लोकांना जोखण्यासाठी वापरणे; लोकांना असुरक्षित वाटत असेल तर भूमिका वापरा ("the on-call engineer")

🏭 प्रत्यक्ष वापरात

खऱ्या account वर — एक postmortem template (Markdown, wiki किंवा repo साठी):

# Postmortem: <title>  ·  SEV<1-3>  ·  <date>  ·  status: draft | reviewed | closed
Blameless: we look at the system, not at a person.

## Summary
<2–3 sentences: what users saw, for how long, what fixed it>

## Impact
- Users affected: <number / share>   · Duration: <start → mitigated>
- Error budget used: <% of the 30-day budget>   · SLA breached: yes / no

## Timeline (all times UTC)
| time | event |
|---|---|
| 06:08 | deploy results-api v42 |
| 06:10 | errors start (SLI) |
| 06:19 | burn-rate page fires — detected |
| 06:21 | on-call acknowledges, becomes IC — acknowledged |
| 06:42 | rollback to v41 — mitigated |
| 07:20 | cache restored, SLI normal — resolved |

## Root cause and contributing factors
<five whys from lesson 11; list every factor that lined up>

## What went well · what went badly · where we got lucky

## Action items
| type | action | owner | due | ticket |
|---|---|---|---|---|
| prevent | test that fails when the grade cache is removed | Katrina | 2026-09-29 | RES-101 |
| prevent | results-day load test in the release checklist | Aishwarya | 2026-10-04 | RES-102 |
| mitigate | canary 5% for 15 min with burn-rate auto-rollback | Dipika | 2026-10-11 | RES-103 |

Team जिथे काम track करते तिथे action items तयार करा, म्हणजे ते planning मध्ये दिसतील (GitHub CLI):

gh issue create --title "Test that fails when the grade cache is removed" \
    --body "Postmortem: results page 504s on results day (SEV1). Due 2026-09-29." \
    --assignee katrina --label postmortem,prevent
gh issue list --label postmortem --state open        # review this list every week

PagerDuty, incident.io, FireHydrant, Rootly आणि Jira Service Management सारखी tools incident channel मधून timeline तयार करतात आणि follow-ups track करतात.

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: फक्त outages नव्हे, चक्र मोजा: किती action items उघडे आहेत, किती उशीर झालेले आहेत, आणि तेच cause किती वेळा परत येते. आपले items बंद करणाऱ्या team च्या रात्री दर तिमाहीत अधिक शांत होतात.

🎓 आरोग्य कक्ष आता तुमचा आहे

आरोग्य कक्ष का → दैनंदिनी → तपासणी तक्ता → हळू म्हणजे किती हळू → एका विद्यार्थ्याचे मार्ग कार्ड → प्रत्येक खोलीसाठी एकच form → scrape करा, विचारा, काढा → विद्यार्थ्यांना त्रास होत असेल तेव्हाच उठवणारी धोक्याची घंटा → आजारी-रजेची मुभा → अग्निशमन सराव → का, पाच वेळा → नावे आणि तारखांसह अहवाल. तुम्ही फक्त observability शिकला नाहीत — system काय करत आहे आणि का, हे तुम्ही सांगू शकता, आणि पुढचे incident लहान करू शकता. 🩺📊🎓

⏭️ पुढे

इतर शाळा. System Design शाळा प्रत्येक box मध्ये observability आधीच ठरवते; Scaling शाळा load खाली काय पाहायचे ते दाखवते; Kubernetes शाळा ते pods चालवते ज्यांवर आता तुम्ही लक्ष ठेवू शकता; School portal मध्ये बाकी सगळे आहे.

git checkout main
python3 obs/demo.py     # one last run, for fun

📝 Lesson 12 — Postmortems & the loop: the report that makes the school safer

📍 You are here: Lesson 12 of 12 · Previous: lesson-11-root-cause

📦 What's in this branch

Lessons 01–11, plus the step that closes the loop: the blameless postmortem — impact, timeline, root cause and contributing factors, and action items with an owner and a date — and the loop from signals back to better signals. postmortem() lives in obs/reliability.py; postmortem_demo() (section postmortem) in obs/demo.py writes the results-day report.

🧒 Explain like I'm 5

A week after the drill, Dipika calls a meeting and writes a report 📝. It has four parts:

What happened to the pupils — how many, for how long.
The clock — minute by minute, from the clipboard.
Why — the five whys from lesson 11.
What we will change — and next to each change, a name and a date: "Katrina — timers on every toaster — by Friday."

There is one rule at the top of the report, in big letters: "We look at the school, not at a person." Because if people get into trouble for telling the truth, next time nobody tells the truth — and the toaster burns again.

Then the report goes on the staff-room wall for everyone to read. And every Friday, Dipika checks the list: done, or not yet? A report with nobody's name on the changes is just a story.

🗺️ Diagram

flowchart LR
    sig["📊📓🗺️ signals<br/>L02–L07"] --> al["🚨 alert on the SLO<br/>L08–L09"]
    al --> inc["🧯 incident<br/>L10"]
    inc --> rc["🔍 root cause<br/>L11"]
    rc --> pm["📝 blameless postmortem<br/>impact · timeline · cause"]
    pm --> ai["✅ action items<br/>owner + date"]
    ai -->|"new tests, guards, alerts"| sig

🗺️ Drawn version + a lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l12

❓ What

Postmortem (incident review, post-incident review) — a written record of an incident: what happened, the impact, the timeline, the causes, and what will change. Write one for every SEV1 and SEV2, for any incident that used a big part of the error budget (lesson 09), and for interesting near misses.
Blameless — the review looks at systems and processes, not at a person's fault. It assumes everyone acted reasonably with the information and tools they had at the time. People then share what really happened — including their own mistakes — which is the only way to find the real gaps. Blameless does not mean nobody is responsible; it means the fix goes into the system.
The parts of a good postmortem:
- summary — two or three sentences anyone can read;
- impact — who was affected, how many, for how long, how much error budget was used;
- timeline — times of the change, detection, acknowledgement, mitigation and resolution (the lesson 10 clocks), taken from the scribe's notes;
- root cause and contributing factors — from lesson 11;
- what went well, what went badly, where we got lucky;
- action items.
Action items — each one specific, with one owner (a person, not a team) and a due date, tracked in the same system as other work. Good items change the system:
- prevent — a test, a guard, a checklist step (a load test with results-day data);
- detect — a new or faster alert, a new SLI;
- mitigate — a faster or automatic rollback (a canary with burn-rate auto-rollback). "Be more careful" is not an action item.
Review and share — a short review meeting, then publish the postmortem where every team can read it. Track open items until they are done; count how many are overdue.
The loop — signals → alerts on the SLO → incident → root cause → postmortem → action items → better signals. Each outage makes the next one shorter or impossible.

🤔 Why

Because an incident is expensive, and the postmortem is how you get something for the money. Without it, the lessons live in three people's heads and leave when they do. Without owners and dates, action items become a list nobody reads, and the same outage returns in three months. Without "blameless", people stop reporting near misses — the cheapest lessons of all.

🔧 How (in this repo)

postmortem(title, impact, timeline, root_cause, actions) in obs/reliability.py returns a Markdown document: a title, the blameless line, Impact, Timeline, Root cause and Action items, with each action written as - [owner] what (by due). postmortem_demo() in obs/demo.py writes the bad-deploy report: 32 minutes, about 1,900 failed views, three action items for Katrina, Aishwarya and Dipika. It ends by printing the loop.

🧪 Try it

Write the postmortem for the incident nobody was paged for — the slow leak that lesson 11 tied to the cache TTL change — with real dates:

python3 obs/demo.py postmortem
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from reliability import postmortem
print(postmortem("slow results page from a flaky grade dependency",
                 "0.9% of results views failed for 280 minutes (minutes 100–379); no page, one ticket at minute 266",
                 ["090 config: cache TTL 60 s → 5 s", "100 errors rise to 0.9%", "266 burn-rate ticket opened",
                  "380 dependency recovered; errors back to 0.1%"],
                 "a 5 s cache TTL sent 12× more calls to a grade service that could not take them",
                 [("alert on the grade service's own SLO", "Aishwarya", "2026-10-02"),
                  ("config changes go through the same canary as deploys", "Dipika", "2026-10-09")]))
EOF
python3 obs/test_obs.py

✅ Verify — what you should see

postmortem prints:

   # Postmortem: results page 504s on results day

   Blameless: we look at the system, not at a person.

   ## Impact
   6% of results views failed for 32 minutes (minutes 400–431); ~1,900 parents saw an error

   ## Timeline
   - 398 deploy v42
   - 409 burn-rate page fired
   - 411 Dipika acknowledged, took command
   - 432 rollback to v41 — errors back to normal
   - 470 resolved; grade cache restored

   ## Root cause
   v42 removed the grade cache; the release load test did not match results-day traffic

   ## Action items
   - [Katrina] restore the grade cache with a test that fails without it (by 2 days)
   - [Aishwarya] results-day load test in the release checklist (by 1 week)
   - [Dipika] canary 5% for 15 min with burn-rate auto-rollback (by 2 weeks)
── the loop: signals → alerts on the SLO → incident → root cause → postmortem → action items → better signals

Your snippet prints:

# Postmortem: slow results page from a flaky grade dependency

Blameless: we look at the system, not at a person.

## Impact
0.9% of results views failed for 280 minutes (minutes 100–379); no page, one ticket at minute 266

## Timeline
- 090 config: cache TTL 60 s → 5 s
- 100 errors rise to 0.9%
- 266 burn-rate ticket opened
- 380 dependency recovered; errors back to 0.1%

## Root cause
a 5 s cache TTL sent 12× more calls to a grade service that could not take them

## Action items
- [Aishwarya] alert on the grade service's own SLO (by 2026-10-02)
- [Dipika] config changes go through the same canary as deploys (by 2026-10-09)

The tests end with ✅ L12 the postmortem is blameless and has action items with owners and 12/12 passed.

🏁 What you just proved

Every action item names one person and a deadline, and each one changes the system: a test, a load test in the checklist, a canary, an alert, a rule for config changes. None says "be careful". Your second report shows the loop working: the slow leak never woke anyone (lesson 08 did its job), yet it still cost 2,520 failed views — more than the bad deploy's ~1,920 — so it earns a postmortem too, and its fixes become better signals: a new SLO and a canary for config. A date like 2026-10-02 can be checked; "2 days" in the demo's report is only clear on the day it was written.

⚠️ Common mistakes

"root cause: human error", "action item: retrain the engineer"
action items owned by "the team", or with no date — they never close
a postmortem written, reviewed once, and never read again; items not tracked with normal work
postmortems only for huge outages, so the near misses (the cheap lessons) are lost
a timeline built from memory a week later instead of from the scribe's notes and the graphs
names in the timeline used to judge people; use roles ("the on-call engineer") if people feel exposed

🏭 In production

On a real account — a postmortem template (Markdown, for a wiki or a repo):

# Postmortem: <title>  ·  SEV<1-3>  ·  <date>  ·  status: draft | reviewed | closed
Blameless: we look at the system, not at a person.

## Summary
<2–3 sentences: what users saw, for how long, what fixed it>

## Impact
- Users affected: <number / share>   · Duration: <start → mitigated>
- Error budget used: <% of the 30-day budget>   · SLA breached: yes / no

## Timeline (all times UTC)
| time | event |
|---|---|
| 06:08 | deploy results-api v42 |
| 06:10 | errors start (SLI) |
| 06:19 | burn-rate page fires — detected |
| 06:21 | on-call acknowledges, becomes IC — acknowledged |
| 06:42 | rollback to v41 — mitigated |
| 07:20 | cache restored, SLI normal — resolved |

## Root cause and contributing factors
<five whys from lesson 11; list every factor that lined up>

## What went well · what went badly · where we got lucky

## Action items
| type | action | owner | due | ticket |
|---|---|---|---|---|
| prevent | test that fails when the grade cache is removed | Katrina | 2026-09-29 | RES-101 |
| prevent | results-day load test in the release checklist | Aishwarya | 2026-10-04 | RES-102 |
| mitigate | canary 5% for 15 min with burn-rate auto-rollback | Dipika | 2026-10-11 | RES-103 |

Create the action items where the team tracks work, so they show up in planning (GitHub CLI):

gh issue create --title "Test that fails when the grade cache is removed" \
    --body "Postmortem: results page 504s on results day (SEV1). Due 2026-09-29." \
    --assignee katrina --label postmortem,prevent
gh issue list --label postmortem --state open        # review this list every week

Tools such as PagerDuty, incident.io, FireHydrant, Rootly and Jira Service Management build the timeline from the incident channel and track the follow-ups.

🏭 Why this matters in production: measure the loop, not only the outages: how many action items are open, how many are overdue, and how often the same cause comes back. A team that closes its items gets quieter nights every quarter.

🎓 The health room is yours

Why a health room → the diary → the vital-signs chart → how slow is slow → one pupil's route card → one form for every room → scrape, ask, draw → an alarm that wakes you only when pupils are hurting → a sick-day allowance → the fire drill → why, five times → the report, with names and dates. You didn't just learn observability — you can tell what the system is doing, why, and make the next incident shorter. 🩺📊🎓

⏭️ Next

The other schools. The System Design school plans observability into every box; the Scaling school shows what to watch under load; the Kubernetes school runs the pods you now know how to watch; the School portal has the rest.

git checkout main
python3 obs/demo.py     # one last run, for fun

← Previousroot cause Finished! Take the quiz →check what stuck

This page is the lesson's README from the lesson-12-postmortems branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.