🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🔍 धडा 11 — Root-cause analysis: हे खरोखर का घडले

📍 तुम्ही इथे आहात: 12 पैकी धडा 11 · मागील: lesson-10-incident-response · पुढील: lesson-12-postmortems

📦 या ब्रँचमध्ये काय आहे

धडे 01–10, आणि खरे कारण शोधणे: change correlation (अगदी आधी काय बदलले?), पाच का, आणि उत्तर जवळजवळ नेहमीच व्यक्ती नव्हे तर process मधले नसलेले संरक्षण का असते. first_bad_minute(), suspect_changes() आणि five_whys() obs/reliability.py मध्ये आहेत; obs/demo.py मधले rootcause() त्यांना निकालाच्या दिवसावर चालवते.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

सरावानंतर सगळे सुरक्षित आहेत. आता कतरिना विचारते: हे का घडले?

आधी ती घड्याळ आणि भेट नोंदवही पाहते. धूर 11:40 ला सुरू झाला. 11:40 च्या अगदी आधी कोण आत आले किंवा कोणी काहीतरी बदलले? नोंदवही सांगते: 11:38 ला, staff room मध्ये एक नवा toaster लावला गेला. तो पहिला संशयित.

मग ती विचारते "का?" — आणि पुन्हा विचारते, पुन्हा पुन्हा:

धूर का आला? Toaster ने पाव जाळला.
तो का जळला? Toaster 20 मिनिटे चालू राहिला.
तो चालू का राहिला? त्याला बंद करणारा timer नाही.
Timer नसलेल्या toaster ला परवानगी का मिळाली? स्वयंपाकघराच्या नव्या वस्तू कोणीच तपासत नाही.
कोणीच का तपासत नाही? स्वयंपाकघराच्या नव्या वस्तूंसाठी कोणतीही checklist नाही.

कतरिना "कुणीतरी तो चालू ठेवला" इथे थांबत नाही. माणसे गोष्टी विसरतात; ते उद्याही पुन्हा घडेल. Timer आणि checklist हे सगळ्यांसाठी दुरुस्त करतात.

🗺️ आकृती

flowchart TB
    spike["📈 error ratio first above 5% at minute 400"]
    ch["🗂️ changes in the 30 minutes before<br/>398 deploy results-api v42 ✅ suspect<br/>(90 config · 250 v41 · 430 rollback — outside)"]
    spike --> ch
    ch --> w1["why 1? parents saw 504 on /results"]
    w1 --> w2["why 2? the grade service call took 1 s and timed out"]
    w2 --> w3["why 3? v42 removed the grade cache"]
    w3 --> w4["why 4? the load test used 3 classes, not 900"]
    w4 --> w5["why 5? no production-like load test in the release checklist"]
    w5 --> fix["🔧 fix the PROCESS, not the person"]

🗺️ काढलेली आकृती + एक lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l11

❓ काय

Root-cause analysis (RCA) — incident का घडले ते इतक्या खोलवर शोधणे की एखादा fix ते (आणि त्याच्यासारखे इतर) परत येण्यापासून थांबवेल.
Change correlation — बहुतेक incidents एखाद्या बदलानंतर येतात: deploy, config बदल, feature flag, dependency upgrade, infrastructure बदल, certificate किंवा data migration. म्हणून पहिला प्रश्न आहे "हानी सुरू होण्याच्या अगदी आधी काय बदलले?". तुम्हाला लागते: हानीची नेमकी सुरुवात (SLI मधून, page मधून नाही — page 9 मिनिटांनी उशिरा आले) आणि वेळेसह प्रत्येक बदलाची नोंद.
Lookback — सुरुवातीच्या किती आधीपर्यंत तुम्ही पाहता. खूप कमी ठेवले तर हळू परिणाम सुटतो (एक cache TTL ज्याचा परिणाम दिसायला एक तास लागतो); खूप जास्त ठेवले तर सगळ्याच गोष्टींची यादी मिळते.
Correlation म्हणजे पुरावा नव्हे — spike च्या अगदी आधीचा बदल हा संशयित असतो. त्याची खात्री करा: rollback ने हानी थांबली का? Diff ने symptom स्पष्ट होतो का? तुम्ही ते staging मध्ये पुन्हा घडवू शकता का?
पाच का — प्रत्येक उत्तराबद्दल "का?" विचारा, सुमारे पाच वेळा, जोपर्यंत तुम्ही संस्था बदलू शकेल अशा गोष्टीपर्यंत पोहोचत नाही (test, संरक्षण, checklist, alert, limit). पाच ही सवय आहे, कायदा नाही: पुढचा "का" तुमच्या नियंत्रणाबाहेर गेला की थांबा.
Contributing factors — खऱ्या incidents मध्ये सहसा अनेक causes एकत्र जुळलेले असतात (deploy आणि खूपच लहान load test आणि canary नाही). म्हणूनच अनेक teams एकाच "root cause" ऐवजी "contributing factors" म्हणतात.
"Human error" हे कधीच root cause नसते — तिथून शोध सुरू होतो. विचारा की एका व्यक्तीची सामान्य चूक users पर्यंत system ने का पोहोचू दिली.

🤔 का

कारण तुम्ही निवडलेला fix तुम्ही किती खोल जाता यावर अवलंबून असतो. "v42 वाईट होते" इथे थांबलात तर तुम्ही roll back करता आणि v43 ने तेच करण्याची वाट पाहता. "engineer cache विसरला" इथे थांबलात तर तुम्ही एका व्यक्तीला दोष देता, लोक पुढची चूक लपवतात, आणि काहीच बदलत नाही. "release checklist मध्ये production सारखी load test नाही" इथपर्यंत पोहोचा आणि तुम्ही अशी पोकळी भरता जिने हा deploy पकडला असता — आणि पुढचे पाचही.

🔧 कसे (या repo मध्ये)

obs/reliability.py मधले first_bad_minute(errors, requests, ratio=0.05) error ratio ratio पेक्षा जास्त असलेला पहिला minute देते — हानीची सुरुवात. suspect_changes(changes, spike_minute, lookback=30) त्या minute पर्यंत आणि तो धरून, lookback मिनिटांतील प्रत्येक (minute, what) बदल देते (दिलेल्या क्रमाने). five_whys(chain) उत्तरांच्या साखळीला क्रमांक देते. obs/demo.py मधले CHANGES दिवसातले चार बदल नोंदवते: 90 ला एक cache TTL config, 250 आणि 398 ला deploys, आणि rollback.

🧪 करून पाहा

Minute 385 ला एक feature flag जोडा (दुसरा संशयित), आणखी मागे पाहा — मग तीच tools slow leak कडे वळवा (0.5% ची रेषा), आणि खूप लवकर थांबणारी साखळी लिहा:

python3 obs/demo.py rootcause
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from demo import ERR, REQ, CHANGES; from reliability import first_bad_minute, suspect_changes, five_whys
changes = sorted(CHANGES + [(385, "feature flag: new PDF certificates ON")])
for ratio in (0.05, 0.005):
    spike = first_bad_minute(ERR, REQ, ratio)
    print(f"first minute above {ratio:.1%}: {spike}")
    for lookback in (30, 180):
        print(f"   changes in the {lookback} min before → {suspect_changes(changes, spike, lookback)}")
for w in five_whys(["the results page returned 504", "the grade service was slow", "Katrina pressed deploy"]): print(w)
EOF
python3 obs/test_obs.py

✅ तपासा — तुम्हाला काय दिसायला हवे

rootcause हे print करते:

── the error ratio first crosses 5% at minute 400
   changes in the 30 minutes before: [(398, 'deploy results-api v42')]
   why 1? parents saw 504 on /results
   why 2? the grade service call took 1 s and timed out
   why 3? v42 removed the grade cache, so every view hit the grade service
   why 4? the load test used 3 classes, not 900
   why 5? there is no production-like load test in the release checklist
   the root cause is usually a missing guard in the PROCESS, not a person

तुमचा snippet हे print करतो:

first minute above 5.0%: 400
   changes in the 30 min before → [(385, 'feature flag: new PDF certificates ON'), (398, 'deploy results-api v42')]
   changes in the 180 min before → [(250, 'deploy results-api v41'), (385, 'feature flag: new PDF certificates ON'), (398, 'deploy results-api v42')]
first minute above 0.5%: 100
   changes in the 30 min before → [(90, 'config: cache TTL 60 s → 5 s')]
   changes in the 180 min before → [(90, 'config: cache TTL 60 s → 5 s')]
why 1? the results page returned 504
why 2? the grade service was slow
why 3? Katrina pressed deploy

Tests मध्ये ✅ L10/L11 incident maths and the suspect change असतो.

🏁 तुम्ही आत्ताच काय सिद्ध केले

Window मध्ये एकच बदल असेल तर संशयित सहज मिळतो. एक flag जोडा आणि दोन होतात, आणि 180-minute lookback तिसरा जोडतो — correlation यादी लहान करते; ते निवड करत नाही. v42 च्या rollback ने errors संपले, यानेच खात्री होते. Slow leak वर तीच tools काहीतरी नवे दाखवतात: minute 90 चा cache TTL बदल, 0.9% errors सुरू होण्याच्या दहा मिनिटे आधी — incident ने कधीच न पाहिलेला धागा, कारण leak ने कधीच page केले नाही. आणि तीन-का ची साखळी एका व्यक्तीवर थांबली. त्यातले काहीच कतरिनाला दोष दिल्याशिवाय दुरुस्त करता येत नाही. विचारत राहा: कोणतीही test fail न होता एका deploy ने cache कसे काढून टाकले?

⚠️ नेहमीच्या चुका

हानीची सुरुवात SLI (400) ऐवजी page च्या वेळेवरून (409) घेणे
config बदल आणि feature flags ची नोंद नाही — फक्त deploys दिसतात
तो बदल उलटवल्याने हानी थांबली का हे न तपासता पहिल्या संशयिताला सिद्ध मानणे
"root cause: human error" — मग "action item: अधिक काळजी घ्या"
तीन contributing factors असलेल्या incident वर जबरदस्तीने एकच root cause लादणे
विश्लेषण एकट्याने करणे; तिथे हजर असलेल्या लोकांना graphs न दाखवणाऱ्या गोष्टी माहीत असतात

🏭 प्रत्यक्ष वापरात

खऱ्या account वर — प्रत्येक बदल प्रत्येक graph वर टाका. Grafana annotation जोडणारी deploy pipeline step (वेळ epoch milliseconds मध्ये):

curl -s -X POST https://grafana.example.school/api/annotations \
  -H "Authorization: Bearer $GRAFANA_TOKEN" -H 'Content-Type: application/json' \
  -d "{\"time\": $(date +%s000), \"tags\": [\"deploy\", \"results-api\"], \"text\": \"deploy results-api v42\"}"

हानीच्या सुरुवातीच्या आसपास काय बदलले ते शोधा:

kubectl rollout history deployment/results-api
git log --since="2026-09-27 11:00" --until="2026-09-27 11:45" --oneline -- services/results-api
aws cloudtrail lookup-events --start-time 2026-09-27T05:30:00Z --end-time 2026-09-27T06:15:00Z \
    --lookup-attributes AttributeKey=ReadOnly,AttributeValue=false \
    --query 'Events[].[EventTime,EventName,Username]' --output table

Error ratio version नुसार तुलना करा — फक्त v42 fail होत असेल, तर संशयिताची खात्री होते (यासाठी metric वर version label लागतो, जो low-cardinality आहे आणि ठेवण्यासारखा आहे):

sum by (version) (rate(http_requests_total{job="results-api", status=~"5.."}[5m]))
/
sum by (version) (rate(http_requests_total{job="results-api"}[5m]))

Datadog graphs वर deploys events म्हणून दाखवतो आणि APM च्या Deployment Tracking मध्ये versions ची तुलना करतो; CloudWatch Amazon DevOps Guru किंवा change events असलेल्या dashboard मधून CloudTrail बदल वर मांडू शकतो. Feature flag tools (LaunchDarkly, Unleash, OpenFeature providers) audit log ठेवतात — तोही शोधात समाविष्ट करा.

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: एखाद्या बदलाची वेळेसह नोंद नसेल, तर त्याची correlation करता येत नाही. प्रत्येक deploy, config बदल आणि flag flip ने असा timestamped event सोडला पाहिजे जो graphs दाखवू शकतील.

⏭️ पुढे

Cause सापडले. आता ते लिहून ठेवा — दोष न देता — अशा fixes सह ज्यांना owner आणि date आहे, म्हणजे तीच आग पुन्हा कधीच लागणार नाही.

git checkout lesson-12-postmortems

🔍 Lesson 11 — Root-cause analysis: why did it really happen

📍 You are here: Lesson 11 of 12 · Previous: lesson-10-incident-response · Next: lesson-12-postmortems

📦 What's in this branch

Lessons 01–10, plus finding the real cause: change correlation (what changed just before?), the five whys, and why the answer is almost always a missing guard in the process, not a person. first_bad_minute(), suspect_changes() and five_whys() live in obs/reliability.py; rootcause() in obs/demo.py runs them on results day.

🧒 Explain like I'm 5

After the drill, everyone is safe. Now Katrina asks: why did it happen?

First she looks at the clock and the visitors' book. The smoke started at 11:40. Who came in or changed something just before 11:40? The book says: 11:38, a new toaster was plugged in the staff room. That is the first suspect.

Then she asks "why?" — and asks again, and again:

Why was there smoke? The toaster burned the bread.
Why did it burn? It was left on for 20 minutes.
Why was it left on? It has no timer that switches it off.
Why was a toaster with no timer allowed? Nobody checks new kitchen things.
Why does nobody check? There is no checklist for new kitchen things.

Katrina does not stop at "Mr. Somebody left it on". People forget things; that will happen again tomorrow. A timer and a checklist fix it for everyone.

🗺️ Diagram

flowchart TB
    spike["📈 error ratio first above 5% at minute 400"]
    ch["🗂️ changes in the 30 minutes before<br/>398 deploy results-api v42 ✅ suspect<br/>(90 config · 250 v41 · 430 rollback — outside)"]
    spike --> ch
    ch --> w1["why 1? parents saw 504 on /results"]
    w1 --> w2["why 2? the grade service call took 1 s and timed out"]
    w2 --> w3["why 3? v42 removed the grade cache"]
    w3 --> w4["why 4? the load test used 3 classes, not 900"]
    w4 --> w5["why 5? no production-like load test in the release checklist"]
    w5 --> fix["🔧 fix the PROCESS, not the person"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l11

❓ What

Root-cause analysis (RCA) — finding why an incident happened, deeply enough that a fix stops it (and its cousins) coming back.
Change correlation — most incidents follow a change: a deploy, a config change, a feature flag, a dependency upgrade, an infrastructure change, a certificate or a data migration. So the first question is "what changed just before the harm started?". You need: the exact start of the harm (from the SLI, not from the page — the page came 9 minutes later) and a record of every change with its time.
Lookback — how far before the start you look. Too short misses a slow effect (a cache TTL that takes an hour to matter); too long gives a list of everything.
Correlation is not proof — a change just before the spike is a suspect. Confirm it: did the rollback end the harm? Does the diff explain the symptom? Can you repeat it in staging?
Five whys — ask "why?" about each answer, about five times, until you reach something the organisation can change (a test, a guard, a checklist, an alert, a limit). Five is a habit, not a law: stop when the next "why" leaves your control.
Contributing factors — real incidents usually have several causes that lined up (the deploy and a load test that was too small and no canary). Many teams say "contributing factors" rather than one "root cause" for that reason.
"Human error" is never the root cause — it is where the search starts. Ask why the system let one person's normal mistake reach users.

🤔 Why

Because the fix you choose depends on how deep you go. Stop at "v42 was bad" and you roll back and wait for v43 to do the same. Stop at "the engineer forgot the cache" and you blame a person, people hide the next mistake, and nothing changes. Go down to "the release checklist has no production-like load test" and you fix a gap that would have caught this deploy — and the next five.

🔧 How (in this repo)

first_bad_minute(errors, requests, ratio=0.05) in obs/reliability.py returns the first minute where the error ratio is above ratio — the start of the harm. suspect_changes(changes, spike_minute, lookback=30) returns every (minute, what) change in the lookback minutes up to and including that minute (in the order given). five_whys(chain) numbers a chain of answers. CHANGES in obs/demo.py records the day's four changes: a cache TTL config at 90, deploys at 250 and 398, and the rollback.

🧪 Try it

Add a feature flag at minute 385 (a second suspect), look back further — then point the same tools at the slow leak (a 0.5% line), and write a chain that stops too early:

python3 obs/demo.py rootcause
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from demo import ERR, REQ, CHANGES; from reliability import first_bad_minute, suspect_changes, five_whys
changes = sorted(CHANGES + [(385, "feature flag: new PDF certificates ON")])
for ratio in (0.05, 0.005):
    spike = first_bad_minute(ERR, REQ, ratio)
    print(f"first minute above {ratio:.1%}: {spike}")
    for lookback in (30, 180):
        print(f"   changes in the {lookback} min before → {suspect_changes(changes, spike, lookback)}")
for w in five_whys(["the results page returned 504", "the grade service was slow", "Katrina pressed deploy"]): print(w)
EOF
python3 obs/test_obs.py

✅ Verify — what you should see

rootcause prints:

── the error ratio first crosses 5% at minute 400
   changes in the 30 minutes before: [(398, 'deploy results-api v42')]
   why 1? parents saw 504 on /results
   why 2? the grade service call took 1 s and timed out
   why 3? v42 removed the grade cache, so every view hit the grade service
   why 4? the load test used 3 classes, not 900
   why 5? there is no production-like load test in the release checklist
   the root cause is usually a missing guard in the PROCESS, not a person

Your snippet prints:

first minute above 5.0%: 400
   changes in the 30 min before → [(385, 'feature flag: new PDF certificates ON'), (398, 'deploy results-api v42')]
   changes in the 180 min before → [(250, 'deploy results-api v41'), (385, 'feature flag: new PDF certificates ON'), (398, 'deploy results-api v42')]
first minute above 0.5%: 100
   changes in the 30 min before → [(90, 'config: cache TTL 60 s → 5 s')]
   changes in the 180 min before → [(90, 'config: cache TTL 60 s → 5 s')]
why 1? the results page returned 504
why 2? the grade service was slow
why 3? Katrina pressed deploy

The tests include ✅ L10/L11 incident maths and the suspect change.

🏁 What you just proved

One change in the window makes an easy suspect. Add a flag and there are two, and a 180-minute lookback adds a third — correlation narrows the list; it does not choose. The rollback of v42 ending the errors is what confirms it. The same tools on the slow leak point at something new: the cache TTL change at minute 90, ten minutes before the 0.9% errors began — a lead the incident never looked at, because the leak never paged. And the three-why chain stopped at a person. Nothing in it can be fixed except by blaming Katrina. Keep asking: why could one deploy remove a cache with no test failing?

⚠️ Common mistakes

taking the start of the harm from the page time (409) instead of the SLI (400)
no record of config changes and feature flags — only deploys are visible
treating the first suspect as proven without checking that undoing it fixed the harm
"root cause: human error" — then "action item: be more careful"
one single root cause forced onto an incident that had three contributing factors
doing the analysis alone; the people who were there know what the graphs do not show

🏭 In production

On a real account — put every change on every graph. A deploy pipeline step that adds a Grafana annotation (time in epoch milliseconds):

curl -s -X POST https://grafana.example.school/api/annotations \
  -H "Authorization: Bearer $GRAFANA_TOKEN" -H 'Content-Type: application/json' \
  -d "{\"time\": $(date +%s000), \"tags\": [\"deploy\", \"results-api\"], \"text\": \"deploy results-api v42\"}"

Find what changed around the start of the harm:

kubectl rollout history deployment/results-api
git log --since="2026-09-27 11:00" --until="2026-09-27 11:45" --oneline -- services/results-api
aws cloudtrail lookup-events --start-time 2026-09-27T05:30:00Z --end-time 2026-09-27T06:15:00Z \
    --lookup-attributes AttributeKey=ReadOnly,AttributeValue=false \
    --query 'Events[].[EventTime,EventName,Username]' --output table

Compare the error ratio by version — if only v42 fails, the suspect is confirmed (this needs a version label on the metric, which is low-cardinality and worth having):

sum by (version) (rate(http_requests_total{job="results-api", status=~"5.."}[5m]))
/
sum by (version) (rate(http_requests_total{job="results-api"}[5m]))

Datadog shows deploys as events on graphs and compares versions in APM's Deployment Tracking; CloudWatch can overlay CloudTrail changes through Amazon DevOps Guru or a dashboard with the change events. Feature flag tools (LaunchDarkly, Unleash, OpenFeature providers) keep an audit log — include it in the search.

🏭 Why this matters in production: if a change is not recorded with a time, it cannot be correlated. Make every deploy, config change and flag flip leave a timestamped event where the graphs can show it.

⏭️ Next

The cause is found. Now write it down — without blame — with fixes that have an owner and a date, so the same fire never starts twice.

git checkout lesson-12-postmortems

← Previousincident response Next →postmortems

This page is the lesson's README from the lesson-11-root-cause branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.