🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🚨 धडा 08 — Alerting: विद्यार्थ्यांना त्रास होत असेल तेव्हाच कुणाला उठवा

📍 तुम्ही इथे आहात: 12 पैकी धडा 08 · मागील: lesson-07-prometheus-grafana · पुढील: lesson-09-slos

📦 या ब्रँचमध्ये काय आहे

धडे 01–07, आणि भाग 2 चा शेवट: alerting. तुम्ही symptoms वर (users ना जे जाणवते) alert करायला शिकता, causes वर (CPU) नाही; burn rate म्हणजे काय, आणि Google SRE Workbook चे multi-window, multi-burn-rate alerts. burn_rate() आणि burn_alerts() obs/reliability.py मध्ये आहेत; obs/demo.py मधले alerting() पूर्ण 8 तासांचा दिवस त्यांच्यामधून पुन्हा चालवते.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

शाळेत एक धोक्याची घंटा 🚨 आहे जी रात्री कतरिनाला उठवते. धडा 01 मधली boiler वरची घंटा आठवते? ती 64 मिनिटे वाजली आणि आजारी विद्यार्थ्यासाठी एकदाही नाही. कतरिना तिच्याकडे दुर्लक्ष करू लागली. घंटेकडून होऊ शकणारी ही सगळ्यात वाईट गोष्ट आहे.

म्हणून दीपिका नवे नियम लिहिते. घंटा boiler चे नव्हे, विद्यार्थ्यांचे ऐकते:

"अनेक विद्यार्थी झपाट्याने आजारी पडत आहेत" — असेच चालू राहिले, तर पूर्ण महिन्याची आजारी-रजेची मुभा काही दिवसांतच संपेल. कतरिनाला आत्ता उठवा. 🚨
"नेहमीपेक्षा थोडे जास्त विद्यार्थी, हळूहळू, दुपारभर" — आज रात्रीची आणीबाणी नाही. सकाळसाठी तिच्या टेबलावर एक चिठ्ठी ठेवा 📝.

आणि प्रत्येक नियम दोन घड्याळे तपासतो. मोठे घड्याळ (शेवटचा एक तास) विचारते: "हे महत्त्वाचे ठरण्याइतके मोठे आहे का?" छोटे घड्याळ (शेवटची 5 मिनिटे) विचारते: "हे अजूनही घडत आहे का?" दोघांनीही हो म्हटले पाहिजे. त्यामुळे एका छोट्या शिंकेने घंटा वाजत नाही, आणि विद्यार्थी बरे झाल्यावर लगेचच घंटा थांबते.

🗺️ आकृती

flowchart TB
    err["📈 error ratio per minute<br/>SLO 99.9% → budget rate 0.1%"]
    br["burn rate = error ratio ÷ (1 − SLO)<br/>0.9% → 9× · 6% → 60×"]
    err --> br
    br --> pg{"🚨 PAGE<br/>≥ 14.4× over 1 h<br/>AND over 5 min"}
    br --> tk{"📝 TICKET<br/>≥ 6× over 6 h<br/>AND over 30 min"}
    pg -->|"bad deploy"| p1["minutes 409–434<br/>9 min after it began"]
    tk -->|"slow leak"| t1["minutes 266–390<br/>nobody woken"]
    cpu["🔕 CPU > 80%: 64 minutes of noise"]

🗺️ काढलेली आकृती + एक lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l08

❓ काय

Alert — signal चे notification मध्ये रूपांतर करणारा नियम. Notification चे दोन प्रकार:
- page — एखाद्या व्यक्तीला आत्ता उठवतो (PagerDuty, Opsgenie, phone call). फक्त अशा गोष्टींसाठी ज्या users ना आत्ता त्रास देतात आणि ज्यांना आत्ता माणसाची गरज आहे;
- ticket — कामाच्या वेळेसाठी एक काम (chat message, issue).
Symptom विरुद्ध cause — symptom म्हणजे users ना जे जाणवते: errors, हळूपणा, हरवलेला data. Cause म्हणजे संभाव्य कारण: CPU, memory, pod restart, भरलेली queue. Page फक्त symptoms वर करा. Causes dashboards वर असतात, आणि ते लवकरच त्रास देणार असतील तेव्हा tickets मध्ये (4 तासांत भरणारी disk).
प्रत्येक page actionable असले पाहिजे — त्यावर कुणीतरी काहीतरी करायला हवे, आणि त्यात runbook ची link असते (घ्यायची पहिली पावले).
Error budget — 99.9% च्या SLO मध्ये, 0.1% requests fail होऊ शकतात (धडा 09). तो 0.1% म्हणजे budget.
Burn rate — "अगदी budget प्रमाणे" च्या तुलनेत तुम्ही budget किती जलद खर्च करता: burn rate = error ratio ÷ (1 − SLO). 99.9% ला, 0.1% errors म्हणजे 1× (budget बरोबर 30 दिवस टिकतो), 0.9% म्हणजे 9×, 6% म्हणजे 60× (30 दिवसांचा budget 12 तासांत संपतो).
Alert ने खर्च केलेला budget — burn rate × window ÷ SLO period. 30 दिवसांच्या period मध्ये (720 तास):
- 1 तासासाठी 14.4× = 14.4 ÷ 720 = महिन्याच्या budget चा 2%;
- 6 तासांसाठी 6× = 36 ÷ 720 = 5%.
Multi-window, multi-burn-rate alerts (Google SRE Workbook, प्रकरण "Alerting on SLOs") — प्रत्येक alert ला त्याच threshold वर एक long window आणि एक short window लागते:
- 1 h मध्ये AND 5 min मध्ये 14.4× → page;
- 6 h मध्ये AND 30 min मध्ये 6× → Workbook मध्ये page;
- 3 दिवसांत AND 6 h मध्ये 1× → Workbook मध्ये ticket. Long window सांगते की burn महत्त्वाचा ठरण्याइतका मोठा आहे; short window (long window चा 1/12) सांगते की तो अजूनही घडत आहे, त्यामुळे fix नंतर काही मिनिटांतच alert reset होतो. हा lab दोन स्तर ठेवतो: 14.4× page करतो आणि 6× ticket उघडतो — अनेक teams अशीच सुरुवात करतात, आणि कुणाला न उठवताही slow leak सापडतो.
Alert fatigue — खूप जास्त alerts, बहुतेक actionable नसलेले, त्यामुळे लोक प्रतिसाद देणे थांबवतात. उपाय म्हणजे कमी, पण अधिक चांगले alerts.
Alertmanager — संबंधित alerts एका notification मध्ये group करतो, labels नुसार route करतो (जसे severity), त्याच service चे page चालू असताना ticket inhibit करतो, आणि नियोजित कामाच्या वेळी silences लावतो.

🤔 का

कारण page महाग असते: ते एका व्यक्तीला उठवते, आणि प्रत्येक खोटे page तिला पुढच्याकडे दुर्लक्ष करायला शिकवते. CPU हे cause आहे: ते 64 मिनिटे 80% च्या वर होते आणि पालकांना काहीच त्रास नव्हता. Burn-rate alerts पालकांना जे जाणवते त्यावर वाजतात, budget किती जलद संपत आहे त्या प्रमाणात. मोठा जलद burn काही मिनिटांत कुणाला तरी उठवतो. छोटा हळू burn ticket बनतो. छोटासा blip काहीच करत नाही — कारण त्याने खरोखरच फारसा फरक पडला नाही.

🔧 कसे (या repo मध्ये)

obs/reliability.py मधले burn_rate(error_ratio, slo) budget rate ने भागाकार करते. window_ratio(errors, requests, end, minutes) म्हणजे end पूर्वीच्या शेवटच्या minutes चा error ratio. burn_alerts(errors, requests, slo=0.999) प्रत्येक minute तपासते: जर 1 h आणि 5 min चे burn rates दोन्ही ≥ 14.4 असतील तर ते page करते; नाहीतर जर 6 h आणि 30 min चे burn rates दोन्ही ≥ 6 असतील तर ते ticket उघडते. (else Alertmanager च्या inhibit rule ची भूमिका बजावते: page चालू असताना ticket नाही. दिवस minute 0 ला सुरू होतो, म्हणून दिवसाच्या सुरुवातीला 6 h window फक्त आतापर्यंतची मिनिटे व्यापते.) alerting() दिवस पुन्हा चालवते: एक slow leak (0.9%, minutes 100–379) आणि एक bad deploy (6%, minutes 400–431). demo.py मधले spans() minute ranges print करते.

🧪 करून पाहा

Bad deploy छोटा करा, मग एकसारखी समस्या दोन आकारांत करून पाहा:

python3 obs/demo.py alerting
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from demo import ERR, REQ, spans; from reliability import burn_alerts, burn_rate
print(f"burn rate of 0.9% errors on a 99.9% SLO: {burn_rate(0.009, 0.999):.0f}× · of 6%: {burn_rate(0.06, 0.999):.0f}×")
for bad_minutes in (3, 10, 32):
    err = [1] * 480
    for m in range(400, 400 + bad_minutes): err[m] = 60
    pages, tickets = burn_alerts(err, REQ)
    print(f"6% errors for {bad_minutes:>2} min → page: {spans(pages)} · ticket: {spans(tickets)}")
for pct in (1.5, 3.0):
    err = [1] * 480
    for m in range(100, 480): err[m] = int(pct * 10)
    pages, tickets = burn_alerts(err, REQ)
    print(f"{pct}% errors from minute 100 → first page {pages[0] if pages else 'never'} · first ticket {tickets[0] if tickets else 'never'}")
EOF
python3 obs/test_obs.py

✅ तपासा — तुम्हाला काय दिसायला हवे

alerting हे print करते:

── 8 hours of traffic, SLO 99.9%: a slow leak (0.9% errors, minutes 100–379) and a bad deploy (6%, minutes 400–431)
   CPU > 80% alerts: 64 minutes of noise that do not track what users feel
   burn-rate TICKET (≥6× over 6 h AND 30 min): minutes 266–390, 401–408, 435–458 — first the slow leak (nobody woken at night), then around the deploy while the page is not firing
   burn-rate PAGE (≥14.4× over 1 h AND 5 min): minutes 409–434 — the bad deploy, 9 minutes after it began
   page for fast burns that hurt users now; open a ticket for slow burns; never page on a cause like CPU

तुमचा snippet हे print करतो:

burn rate of 0.9% errors on a 99.9% SLO: 9× · of 6%: 60×
6% errors for  3 min → page: never · ticket: never
6% errors for 10 min → page: never · ticket: never
6% errors for 32 min → page: minutes 413–434 · ticket: minutes 435–458
1.5% errors from minute 100 → first page 157 · first ticket 155
3.0% errors from minute 100 → first page 127 · first ticket 120

Tests मध्ये ✅ L08 the bad deploy pages within 10 minutes; the slow leak only opens a ticket असतो.

🏁 तुम्ही आत्ताच काय सिद्ध केले

Slow leak (9×) ने कधीच page केले नाही — त्याने minute 266 ला ticket उघडले. Bad deploy ने minute 409 ला page केले, आणि page 434 ला थांबले, errors 431 ला संपल्यानंतर तीन मिनिटांनी — हे 5-minute window चे काम. 6% चा 10-minute blip काहीच करू शकला नाही: त्याने 600 ÷ 43,200 ≈ महिन्याच्या budget चा 1.4% खर्च केला, page ला योग्य ठरवणाऱ्या 2% पेक्षा कमी. आधी leak नसलेल्या त्याच 32-minute deploy ने 409 ऐवजी 413 ला page केले — leak ने आधीच शेवटच्या तासाच्या budget चा काही भाग वापरला होता. आणि एकसारख्या 3% (30×) ने 27 मिनिटांत page केले; एकसारख्या 1.5% (15×) ला 57 मिनिटे लागली: burn जितका लहान, तितका 1-hour window ला जास्त वेळ लागतो.

⚠️ नेहमीच्या चुका

CPU, memory किंवा एका pod restart वर page करणे
एकच short window (14.4× वर 5 min): प्रत्येक blip वर page होते
फक्त एकच long window (1 h): fix नंतर एक तासापर्यंत alert वाजत राहतो
ratio ऐवजी raw error count वर threshold — गर्दीचे तास outages सारखे दिसतात
page वर runbook link नसणे
ज्यांचा कुणीच मालक नाही असे alerts, किंवा कुणीच न वाचणाऱ्या channel वर जाणारे alerts
महिनोन् महिने कोणतीही action न घडवलेला alert कधीच delete न करणे

🏭 प्रत्यक्ष वापरात

खऱ्या account वर — प्रत्येक window साठी error ratio चे Prometheus recording rules, मग 99.9% SLO साठी Workbook चे burn-rate alerts (0.001 हा budget rate आहे):

groups:
  - name: results-api-slo
    rules:
      - record: job:slo_errors_per_request:ratio_rate5m
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (job) (rate(http_requests_total[5m]))
      - record: job:slo_errors_per_request:ratio_rate30m
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[30m])) / sum by (job) (rate(http_requests_total[30m]))
      - record: job:slo_errors_per_request:ratio_rate1h
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[1h])) / sum by (job) (rate(http_requests_total[1h]))
      - record: job:slo_errors_per_request:ratio_rate6h
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[6h])) / sum by (job) (rate(http_requests_total[6h]))

      - alert: ResultsApiErrorBudgetFastBurn
        expr: |
          job:slo_errors_per_request:ratio_rate1h{job="results-api"} > (14.4 * 0.001)
          and
          job:slo_errors_per_request:ratio_rate5m{job="results-api"} > (14.4 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "results-api is burning its 30-day error budget 14.4× too fast (2% per hour)"
          runbook_url: https://runbooks.example.school/results-api/error-budget-burn

      - alert: ResultsApiErrorBudgetSlowBurn
        expr: |
          job:slo_errors_per_request:ratio_rate6h{job="results-api"} > (6 * 0.001)
          and
          job:slo_errors_per_request:ratio_rate30m{job="results-api"} > (6 * 0.001)
        labels:
          severity: ticket          # the Workbook pages here too; choose per service
        annotations:
          summary: "results-api has used 5% of its 30-day error budget in 6 hours"

Alertmanager — pages PagerDuty ला, tickets chat ला, service नुसार group केलेले, आणि त्याच service चे page चालू असताना ticket नाही:

route:
  receiver: team-chat
  group_by: [alertname, job]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['severity="page"']
      receiver: pagerduty-results
receivers:
  - name: pagerduty-results
    pagerduty_configs:
      - routing_key: <PagerDuty Events API v2 integration key>
  - name: team-chat
    slack_configs:
      - api_url: <Slack incoming webhook URL>
        channel: "#results-alerts"
inhibit_rules:
  - source_matchers: ['severity="page"']
    target_matchers: ['severity="ticket"']
    equal: [job]

Datadog चे SLO monitors हीच कल्पना burn rate alert म्हणून देतात (burn_rate("<slo id>").over("30d").long_window("1h").short_window("5m") > 14.4). CloudWatch composite alarm ने alarms एकत्र करू शकतो (ALARM(long) AND ALARM(short)), आणि CloudWatch Application Signals मध्ये burn-rate alarms सह SLOs आधीच असतात.

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: मागच्या महिन्यातील pages मोजा. प्रत्येकासाठी विचारा: "user ला ते जाणवले का, आणि कुणाला action घ्यावी लागली का?" जिथे उत्तर नाही होते ते प्रत्येक page delete करा किंवा खालच्या स्तरावर न्या.

⏭️ पुढे

भाग 2 पूर्ण झाला. इथल्या प्रत्येक alert ने "99.9%" वापरले. हा आकडा कुठून येतो, आणि budget संपल्यावर तुम्ही काय करता? भाग 3 सुरू होतो.

git checkout lesson-09-slos

🚨 Lesson 08 — Alerting: wake someone only when pupils are hurting

📍 You are here: Lesson 08 of 12 · Previous: lesson-07-prometheus-grafana · Next: lesson-09-slos

📦 What's in this branch

Lessons 01–07, plus the end of Part 2: alerting. You learn to alert on symptoms (what users feel) and not causes (CPU), what a burn rate is, and the Google SRE Workbook's multi-window, multi-burn-rate alerts. burn_rate() and burn_alerts() live in obs/reliability.py; alerting() in obs/demo.py replays the whole 8-hour day through them.

🧒 Explain like I'm 5

The school has an alarm 🚨 that wakes Katrina at night. Remember the bell on the boiler from lesson 01? It rang for 64 minutes and never once for an ill pupil. Katrina started to ignore it. That is the worst thing an alarm can do.

So Dipika writes new rules. The alarm listens to pupils, not the boiler:

"Many pupils are getting ill fast" — if this goes on, the whole month's sick-day allowance is gone in a few days. Wake Katrina now. 🚨
"A few more pupils than usual, slowly, all afternoon" — not an emergency tonight. Put a note on her desk 📝 for the morning.

And each rule checks two clocks. The long clock (the last hour) asks: "is this big enough to matter?" The short clock (the last 5 minutes) asks: "is it still happening?" Both must say yes. So a short sneeze does not ring the alarm, and the alarm stops soon after the pupils are well again.

🗺️ Diagram

flowchart TB
    err["📈 error ratio per minute<br/>SLO 99.9% → budget rate 0.1%"]
    br["burn rate = error ratio ÷ (1 − SLO)<br/>0.9% → 9× · 6% → 60×"]
    err --> br
    br --> pg{"🚨 PAGE<br/>≥ 14.4× over 1 h<br/>AND over 5 min"}
    br --> tk{"📝 TICKET<br/>≥ 6× over 6 h<br/>AND over 30 min"}
    pg -->|"bad deploy"| p1["minutes 409–434<br/>9 min after it began"]
    tk -->|"slow leak"| t1["minutes 266–390<br/>nobody woken"]
    cpu["🔕 CPU > 80%: 64 minutes of noise"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l08

❓ What

Alert — a rule that turns a signal into a notification. Two kinds of notification:
- page — wakes a person now (PagerDuty, Opsgenie, a phone call). Only for things that hurt users now and need a human now;
- ticket — a task for working hours (a chat message, an issue).
Symptom vs cause — a symptom is what users feel: errors, slowness, missing data. A cause is a possible reason: CPU, memory, a pod restart, a full queue. Page on symptoms. Causes belong on dashboards, and in tickets when they will hurt soon (a disk that fills in 4 hours).
Every page must be actionable — it needs a person to do something, and it links a runbook (the first steps to take).
Error budget — with an SLO of 99.9%, 0.1% of requests may fail (lesson 09). That 0.1% is the budget.
Burn rate — how fast you spend the budget compared with "exactly on budget": burn rate = error ratio ÷ (1 − SLO). At 99.9%, 0.1% errors is 1× (the budget lasts exactly the 30 days), 0.9% is 9×, 6% is 60× (a 30-day budget is gone in 12 hours).
Budget spent by an alert — burn rate × window ÷ SLO period. Over a 30-day period (720 hours):
- 14.4× for 1 hour = 14.4 ÷ 720 = 2% of the month's budget;
- 6× for 6 hours = 36 ÷ 720 = 5%.
Multi-window, multi-burn-rate alerts (Google SRE Workbook, chapter "Alerting on SLOs") — each alert needs a long window and a short window over the same threshold:
- 14.4× over 1 h AND over 5 min → page;
- 6× over 6 h AND over 30 min → page in the Workbook;
- 1× over 3 days AND over 6 h → ticket in the Workbook. The long window says the burn is big enough to matter; the short window (1/12 of the long one) says it is still happening, so the alert resets within minutes of the fix. This lab keeps two tiers: 14.4× pages and 6× opens a ticket — many teams start that way, and a slow leak is still found without waking anyone.
Alert fatigue — too many alerts, most not actionable, so people stop reacting. The cure is fewer, better alerts.
Alertmanager — groups related alerts into one notification, routes by labels (such as severity), inhibits a ticket while the page for the same service fires, and applies silences during planned work.

🤔 Why

Because a page is expensive: it wakes a person, and every false one teaches them to ignore the next. CPU is a cause: it was above 80% for 64 minutes and parents were fine. Burn-rate alerts fire on what parents feel, scaled by how fast the budget is going. A big fast burn wakes someone within minutes. A small slow burn becomes a ticket. A short blip does nothing — because it truly did not matter much.

🔧 How (in this repo)

burn_rate(error_ratio, slo) in obs/reliability.py divides by the budget rate. window_ratio(errors, requests, end, minutes) is the error ratio of the last minutes before end. burn_alerts(errors, requests, slo=0.999) checks every minute: if the 1 h and 5 min burn rates are both ≥ 14.4 it pages; else if the 6 h and 30 min burn rates are both ≥ 6 it opens a ticket. (The else plays the part of an Alertmanager inhibit rule: no ticket while the page fires. The day starts at minute 0, so early in the day the 6 h window covers only the minutes so far.) alerting() replays the day: a slow leak (0.9%, minutes 100–379) and a bad deploy (6%, minutes 400–431). spans() in demo.py prints minute ranges.

🧪 Try it

Shorten the bad deploy, then try a steady problem at two sizes:

python3 obs/demo.py alerting
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from demo import ERR, REQ, spans; from reliability import burn_alerts, burn_rate
print(f"burn rate of 0.9% errors on a 99.9% SLO: {burn_rate(0.009, 0.999):.0f}× · of 6%: {burn_rate(0.06, 0.999):.0f}×")
for bad_minutes in (3, 10, 32):
    err = [1] * 480
    for m in range(400, 400 + bad_minutes): err[m] = 60
    pages, tickets = burn_alerts(err, REQ)
    print(f"6% errors for {bad_minutes:>2} min → page: {spans(pages)} · ticket: {spans(tickets)}")
for pct in (1.5, 3.0):
    err = [1] * 480
    for m in range(100, 480): err[m] = int(pct * 10)
    pages, tickets = burn_alerts(err, REQ)
    print(f"{pct}% errors from minute 100 → first page {pages[0] if pages else 'never'} · first ticket {tickets[0] if tickets else 'never'}")
EOF
python3 obs/test_obs.py

✅ Verify — what you should see

alerting prints:

── 8 hours of traffic, SLO 99.9%: a slow leak (0.9% errors, minutes 100–379) and a bad deploy (6%, minutes 400–431)
   CPU > 80% alerts: 64 minutes of noise that do not track what users feel
   burn-rate TICKET (≥6× over 6 h AND 30 min): minutes 266–390, 401–408, 435–458 — first the slow leak (nobody woken at night), then around the deploy while the page is not firing
   burn-rate PAGE (≥14.4× over 1 h AND 5 min): minutes 409–434 — the bad deploy, 9 minutes after it began
   page for fast burns that hurt users now; open a ticket for slow burns; never page on a cause like CPU

Your snippet prints:

burn rate of 0.9% errors on a 99.9% SLO: 9× · of 6%: 60×
6% errors for  3 min → page: never · ticket: never
6% errors for 10 min → page: never · ticket: never
6% errors for 32 min → page: minutes 413–434 · ticket: minutes 435–458
1.5% errors from minute 100 → first page 157 · first ticket 155
3.0% errors from minute 100 → first page 127 · first ticket 120

The tests include ✅ L08 the bad deploy pages within 10 minutes; the slow leak only opens a ticket.

🏁 What you just proved

The slow leak (9×) never paged — it opened a ticket at minute 266. The bad deploy paged at minute 409, and the page stopped at 434, three minutes after the errors ended at 431 — the 5-minute window at work. A 10-minute blip at 6% did nothing: it spent 600 ÷ 43,200 ≈ 1.4% of the month's budget, below the 2% that justifies a page. The same 32-minute deploy with no leak before it paged at 413 instead of 409 — the leak had already used part of the last hour's budget. And a steady 3% (30×) paged 27 minutes in; a steady 1.5% (15×) took 57 minutes: the smaller the burn, the longer the 1-hour window needs.

⚠️ Common mistakes

paging on CPU, memory or one pod restart
a single short window (5 min at 14.4×): pages on every blip
a single long window (1 h) only: the alert keeps firing for up to an hour after the fix
a threshold on the raw error count instead of the ratio — busy hours look like outages
no runbook link on the page
alerts that nobody owns, or that go to a channel nobody reads
never deleting an alert that has not led to an action in months

🏭 In production

On a real account — Prometheus recording rules for the error ratio over each window, then the Workbook's burn-rate alerts for a 99.9% SLO (0.001 is the budget rate):

groups:
  - name: results-api-slo
    rules:
      - record: job:slo_errors_per_request:ratio_rate5m
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (job) (rate(http_requests_total[5m]))
      - record: job:slo_errors_per_request:ratio_rate30m
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[30m])) / sum by (job) (rate(http_requests_total[30m]))
      - record: job:slo_errors_per_request:ratio_rate1h
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[1h])) / sum by (job) (rate(http_requests_total[1h]))
      - record: job:slo_errors_per_request:ratio_rate6h
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[6h])) / sum by (job) (rate(http_requests_total[6h]))

      - alert: ResultsApiErrorBudgetFastBurn
        expr: |
          job:slo_errors_per_request:ratio_rate1h{job="results-api"} > (14.4 * 0.001)
          and
          job:slo_errors_per_request:ratio_rate5m{job="results-api"} > (14.4 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "results-api is burning its 30-day error budget 14.4× too fast (2% per hour)"
          runbook_url: https://runbooks.example.school/results-api/error-budget-burn

      - alert: ResultsApiErrorBudgetSlowBurn
        expr: |
          job:slo_errors_per_request:ratio_rate6h{job="results-api"} > (6 * 0.001)
          and
          job:slo_errors_per_request:ratio_rate30m{job="results-api"} > (6 * 0.001)
        labels:
          severity: ticket          # the Workbook pages here too; choose per service
        annotations:
          summary: "results-api has used 5% of its 30-day error budget in 6 hours"

Alertmanager — pages to PagerDuty, tickets to chat, grouped per service, and no ticket while the page for the same service fires:

route:
  receiver: team-chat
  group_by: [alertname, job]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['severity="page"']
      receiver: pagerduty-results
receivers:
  - name: pagerduty-results
    pagerduty_configs:
      - routing_key: <PagerDuty Events API v2 integration key>
  - name: team-chat
    slack_configs:
      - api_url: <Slack incoming webhook URL>
        channel: "#results-alerts"
inhibit_rules:
  - source_matchers: ['severity="page"']
    target_matchers: ['severity="ticket"']
    equal: [job]

Datadog SLO monitors support the same idea as a burn rate alert (burn_rate("<slo id>").over("30d").long_window("1h").short_window("5m") > 14.4). CloudWatch can combine alarms with a composite alarm (ALARM(long) AND ALARM(short)), and CloudWatch Application Signals has SLOs with burn-rate alarms built in.

🏭 Why this matters in production: count last month's pages. For each one, ask: "did a user feel it, and did a person need to act?" Delete or demote every page where the answer was no.

⏭️ Next

Part 2 is done. Every alert here used "99.9%". Where does that number come from, and what do you do when the budget runs out? Part 3 begins.

git checkout lesson-09-slos

← Previousprometheus grafana Next →slos

This page is the lesson's README from the lesson-08-alerting branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.