🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🎯 धडा 09 — SLIs, SLOs आणि error budgets: पुरेसे चांगले म्हणजे काय ते ठरवा

📍 तुम्ही इथे आहात: 12 पैकी धडा 09 · मागील: lesson-08-alerting · पुढील: lesson-10-incident-response

📦 या ब्रँचमध्ये काय आहे

धडे 01–08, आणि भाग 3 ची सुरुवात, विश्वासार्हतेचे चक्र: SLIs, SLOs, SLAs आणि error budget — आणि आकड्याचे निर्णयात रूपांतर करणारी error budget policy. error_budget() obs/reliability.py मध्ये आहे; obs/demo.py मधले slo() निकालाच्या पूर्ण दिवसाची किंमत एका महिन्याच्या तुलनेत मोजते.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

एखादी शाळा वचन देऊ शकते का की कोणताही विद्यार्थी कधीच आजारी पडणार नाही? नाही. विद्यार्थ्यांना सर्दी होते. "कधीच नाही" साठी प्रयत्न करणारी शाळा आपले दरवाजे बंद करेल आणि कुणालाच शिकवणार नाही.

म्हणून दीपिका आणि कतरिना त्याऐवजी एका आकड्यावर सहमत होतात: "सामान्य महिन्यात, प्रति-विद्यार्थी शाळेच्या दिवसांपैकी प्रत्येक 1,000 पैकी 999 दिवस निरोगी असावेत." ते वचन म्हणजे ध्येय. जे 1,000 पैकी 1 चुकू शकतात ती मुभा — आजारी-रजेची मुभा.

आता ही मुभा एक साधन बनते. भरपूर मुभा उरलेली असताना, शाळा नव्या गोष्टी करून पाहू शकते: सहल, क्रीडा दिन, नवे खेळाचे मैदान. मुभा जवळजवळ संपत आली की दीपिका म्हणते: "या महिन्यात नवे साहस नाही. आधी गळणारे छत आणि तुटलेला नळ दुरुस्त करू."

कोणीही भावनांवरून वाद घालत नाही. त्या मुभेकडे पाहतात.

🗺️ आकृती

flowchart LR
    sli["📏 SLI<br/>good requests ÷ all requests"]
    slo["🎯 SLO<br/>99.9% over 30 days"]
    sla["📜 SLA<br/>a contract, with a penalty<br/>(looser, e.g. 99.5%)"]
    bud["💰 error budget = 1 − SLO<br/>43,200 of 43,200,000 requests<br/>≈ 43.2 min of full outage"]
    day["📅 results day used 4,608 → 10.7%<br/>89.3% left"]
    pol{"⚖️ budget policy"}
    sli --> slo --> bud --> day --> pol
    slo -.-> sla
    pol -->|"budget left"| ship["🚀 ship features"]
    pol -->|"budget gone"| fix["🧰 freeze risky changes<br/>fix reliability first"]

🗺️ काढलेली आकृती + एक lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l09

❓ काय

SLI (service level indicator) — service users ना किती चांगली सेवा देते याचे मोजमाप, सहसा एक ratio: चांगल्या events ÷ वैध events. उदाहरणे:
- availability — fail न झालेल्या (5xx नसलेल्या) requests ÷ सगळ्या requests;
- latency — 500 ms पेक्षा जलद requests ÷ सगळ्या requests (धडा 04 मधला bucket ratio);
- freshness — 1 मिनिटापेक्षा कमी जुना data दाखवणारी results pages ÷ सगळी pages. शक्य तितके user च्या जवळ मोजा (load balancer वर, एका service च्या आत खोलवर नाही).
SLO (service level objective) — एका window मध्ये SLI साठी ध्येय: "rolling 30 दिवसांत 99.9% results requests यशस्वी होतात". हे team ने निवडलेले अंतर्गत ध्येय आहे.
SLA (service level agreement) — customers बरोबरचा करार, जो चुकला तर परिणाम होतो (पैसे परत, service credits). SLA सहसा SLO पेक्षा ढिले असते (SLO 99.9%, SLA 99.5%), म्हणजे करार मोडण्याच्या खूप आधी team ला घंटा ऐकू येते. अनेक अंतर्गत services कडे SLOs असतात पण SLA नसते.
Error budget — 1 − SLO: SLO किती अपयशाला परवानगी देतो. 30 दिवसांत 99.9% साठी: requests चा 0.1%, किंवा पूर्ण outage ची सुमारे 43.2 मिनिटे. प्रत्येक जादा "nine" budget ला 10 ने भागतो: 99% → 432 min, 99.9% → 43.2 min, 99.99% → 4.3 min.
Rolling विरुद्ध calendar window — rolling 30-day window नेहमी शेवटचे 30 दिवस पाहते (सुरळीत, alerting साठी वापरली जाते); calendar month 1 तारखेला reset होतो (reports साठी सोपे).
Error budget policy — product आणि engineering ने सही केलेला लेखी करार, budget वापरला जाईल तसे काय होईल याचा. उदाहरणार्थ:
- budget उरला आहे → सामान्य वेगाने ship करा; तो experiments आणि नियोजित धोक्यावर खर्च करा;
- budget जवळजवळ संपला → गती कमी करा: लहान releases, जास्त canary वेळ;
- budget संपला → SLI परत येईपर्यंत धोकादायक बदल (features) freeze करा; फक्त reliability fixes आणि security fixes ship होतात;
- एका incident ने budget चा मोठा भाग वापरला (उदाहरणार्थ 20% पेक्षा जास्त) → postmortem (धडा 12), ज्याच्या action items वर आधी काम होते.
100% हे चुकीचे ध्येय आहे — users ना त्यांच्या घरच्या networks आणि phones मधून 99.99% आणि 100% मधला फरक कळत नाही, आणि 100% चे ध्येय म्हणजे कोणताच बदल कधीच सुरक्षित नाही.

🤔 का

कारण "विश्वसनीय" हा शब्द आकडा बनेपर्यंत काहीच अर्थ नसतो, आणि "जलद ship करा" आणि "काळजी घ्या" यांतला वाद आकड्याशिवाय कधीच संपत नाही. Budget तो संपवतो: reliability एक संसाधन बनते जे team खर्च करते. Budget उरलेला असताना, बदलाची भीती विनाकारण पैसा खर्च करते. Budget संपल्यावर, आणखी features users ना दिलेले वचन मोडतील. हाच आकडा धडा 08 मधले burn-rate alerts चालवतो.

🔧 कसे (या repo मध्ये)

obs/reliability.py मधल्या error_budget(slo, total_events=None, bad_events=0, days=30) चे दोन modes आहेत. फक्त SLO दिला तर ते budget पूर्ण outage च्या minutes म्हणून देते: (1 − slo) × days × 24 × 60. Event counts दिले तर ते allowed_bad (total × (1 − slo)), used, आणि left_pct देते. obs/demo.py मधले slo() मिनिटाला 1,000 requests प्रमाणे 30 दिवसांचा महिना (43,200,000) आणि दिवसभरातील failed requests (sum(ERR) = 4,608) घेते.

🧪 करून पाहा

आणखी SLOs आणि 7-day window; मग वेगवेगळ्या वाईट दिवसांनंतरचा budget; मग तोच दिवस अधिक कडक SLO विरुद्ध मोजलेला:

python3 obs/demo.py slo
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from reliability import error_budget
for slo in (0.99, 0.995, 0.999, 0.9995, 0.9999):
    print(f"SLO {slo:.2%} → {error_budget(slo)['minutes']:>6} min per 30 days · {error_budget(slo, days=7)['minutes']:>5} min per 7 days")
month = 1000 * 60 * 24 * 30
for bad in (4_608, 21_600, 43_200):
    b = error_budget(0.999, month, bad)
    print(f"{bad:>6,} bad of {month:,} → allowed {b['allowed_bad']:,} · left {b['left_pct']}%")
print("same day on a 99.99% SLO →", error_budget(0.9999, month, 4_608))
EOF
python3 obs/test_obs.py

✅ तपासा — तुम्हाला काय दिसायला हवे

slo हे print करते:

── SLI: good requests / all requests · SLO: 99.9% over 30 days · SLA: the contract with a penalty
   SLO 99.00% →  432.0 minutes of full outage per 30 days
   SLO 99.90% →   43.2 minutes of full outage per 30 days
   SLO 99.99% →    4.3 minutes of full outage per 30 days
── a 30-day window of 43,200,000 requests may have 43,200 failures · today's 8 hours failed 4,608
   one bad day used 10.7% of the month's budget → 89.3% left
   budget left → ship faster · budget gone → freeze risky changes and fix reliability first

तुमचा snippet हे print करतो:

SLO 99.00% →  432.0 min per 30 days · 100.8 min per 7 days
SLO 99.50% →  216.0 min per 30 days ·  50.4 min per 7 days
SLO 99.90% →   43.2 min per 30 days ·  10.1 min per 7 days
SLO 99.95% →   21.6 min per 30 days ·   5.0 min per 7 days
SLO 99.99% →    4.3 min per 30 days ·   1.0 min per 7 days
 4,608 bad of 43,200,000 → allowed 43,200 · left 89.3%
21,600 bad of 43,200,000 → allowed 43,200 · left 50.0%
43,200 bad of 43,200,000 → allowed 43,200 · left 0.0%
same day on a 99.99% SLO → {'allowed_bad': 4320, 'used': 4608, 'left_pct': -6.7}

Tests मध्ये ✅ L09 a 99.9% SLO allows 43.2 minutes a month असतो.

🏁 तुम्ही आत्ताच काय सिद्ध केले

99.9% ला, निकालाच्या दिवसाने — एक slow leak आणि एक bad deploy — महिन्याचा 10.7% वापरला: वाईट, पण policy म्हणते "ship करत राहा, काळजीपूर्वक". तोच दिवस 99.99% विरुद्ध मोजला तर त्याने पूर्ण महिन्यापेक्षा जास्त वापरले (−6.7% उरले): एकाच दिवसाने freeze सुरू होईल. प्रत्येक जादा nine budget ला दहाने भागतो, म्हणून users ना जो SLO लागतो तो निवडा, सगळ्यात छान वाटणारा नव्हे. आणि 99.95% ची 7-day window फक्त 5 मिनिटे देते — लहान windows जलद प्रतिसाद देतात आणि कमी माफ करतात.

⚠️ नेहमीच्या चुका

100% चा SLO — budget नाही, म्हणून प्रत्येक बदल असा धोका ठरतो जो कोणीच घेऊ शकत नाही
user च्या जवळ मोजण्याऐवजी server च्या CPU वर किंवा एका service च्या आत मोजलेले SLIs
SLA मधून copy केलेला SLO, त्यामुळे पहिली घंटा करार आधीच मोडल्यावर वाजते
कुणीच सही न केलेली error budget policy — budget संपला तरी काहीच बदलत नाही
प्रत्येक service साठी खूप जास्त SLOs; एक availability आणि एक latency SLO ने सुरुवात करा
users ना service वापरता येत नसतानाही नियोजित maintenance "वाईट नाही" म्हणून मोजणे

🏭 प्रत्यक्ष वापरात

खऱ्या account वर — 30 दिवसांवरचा availability SLI Prometheus recording rule म्हणून, आणि उरलेला budget Grafana panel वर लावता येईल अशा series म्हणून (SLO 99.9%):

groups:
  - name: results-api-slo-budget
    rules:
      - record: job:slo_availability:ratio_rate30d
        expr: |
          1 - (
            sum by (job) (increase(http_requests_total{job="results-api", status=~"5.."}[30d]))
            /
            sum by (job) (increase(http_requests_total{job="results-api"}[30d]))
          )
      - record: job:slo_error_budget_remaining:ratio
        expr: 1 - ((1 - job:slo_availability:ratio_rate30d) / (1 - 0.999))

Sloth आणि Pyrra सारखी tools हे rules — आणि धडा 08 चे burn-rate alerts — एका छोट्या SLO file मधून तयार करतात:

version: prometheus/v1
service: results-api
slos:
  - name: requests-availability
    objective: 99.9
    sli:
      events:
        error_query: sum(rate(http_requests_total{job="results-api",status=~"5.."}[{{.window}}]))
        total_query: sum(rate(http_requests_total{job="results-api"}[{{.window}}]))
    alerting:
      name: ResultsApiAvailability
      page_alert: {labels: {severity: page}}
      ticket_alert: {labels: {severity: ticket}}

Grafana (Grafana Cloud SLO), Datadog SLOs आणि CloudWatch Application Signals सुद्धा त्यांच्या UI मध्ये SLOs ठरवतात आणि उरलेला budget दाखवतात.

सहमत होऊन सही करायची एका पानाची error budget policy:

# Error budget policy — results-api (SLO 99.9% availability, rolling 30 days)
- Budget > 25% left: normal releases.
- Budget < 25% left: releases need a canary of 30 minutes; no big-bang changes.
- Budget spent: feature freeze. Only reliability and security fixes ship until the 30-day SLI is back above 99.9%.
- Any single incident that used > 20% of the budget: postmortem within 5 working days; its P0 action items come before features.
- Disagreements go to: the head of engineering and the product owner, together.
Signed: Dipika (engineering) · Aishwarya (product) · date: 2026-09-27

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: policy शिवायचा SLO म्हणजे फक्त एक graph. Budget संपल्यावर team काय करणे थांबवेल ते आधीच लिहून ठेवा — आणि engineering व product दोघांनीही त्यावर सही करू द्या.

⏭️ पुढे

Minute 409 ला page वाजते. आता काय? नेतृत्व कोण करते, दुरुस्ती कोण करते, पालकांना कोण सांगते? अग्निशमन सराव.

git checkout lesson-10-incident-response

🎯 Lesson 09 — SLIs, SLOs & error budgets: decide what good enough means

📍 You are here: Lesson 09 of 12 · Previous: lesson-08-alerting · Next: lesson-10-incident-response

📦 What's in this branch

Lessons 01–08, plus the start of Part 3, the reliability loop: SLIs, SLOs, SLAs and the error budget — and the error budget policy that turns a number into a decision. error_budget() lives in obs/reliability.py; slo() in obs/demo.py prices the whole results day against a month.

🧒 Explain like I'm 5

Can a school promise that no pupil is ever ill? No. Pupils catch colds. A school that tries for "never" would close its doors and teach nobody.

So Dipika and Katrina agree on a number instead: "In a normal month, 999 of every 1,000 school days-per-pupil should be healthy ones." That promise is the target. The 1 in 1,000 that may go wrong is the allowance — the sick-day allowance.

Now the allowance becomes a tool. While there is plenty left, the school can try new things: a trip, a sports day, a new playground. When the allowance is nearly gone, Dipika says: "No new adventures this month. We fix the leaking roof and the broken tap first."

Nobody argues about feelings. They look at the allowance.

🗺️ Diagram

flowchart LR
    sli["📏 SLI<br/>good requests ÷ all requests"]
    slo["🎯 SLO<br/>99.9% over 30 days"]
    sla["📜 SLA<br/>a contract, with a penalty<br/>(looser, e.g. 99.5%)"]
    bud["💰 error budget = 1 − SLO<br/>43,200 of 43,200,000 requests<br/>≈ 43.2 min of full outage"]
    day["📅 results day used 4,608 → 10.7%<br/>89.3% left"]
    pol{"⚖️ budget policy"}
    sli --> slo --> bud --> day --> pol
    slo -.-> sla
    pol -->|"budget left"| ship["🚀 ship features"]
    pol -->|"budget gone"| fix["🧰 freeze risky changes<br/>fix reliability first"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l09

❓ What

SLI (service level indicator) — a measurement of how well the service serves users, usually a ratio: good events ÷ valid events. Examples:
- availability — requests that did not fail (not 5xx) ÷ all requests;
- latency — requests faster than 500 ms ÷ all requests (lesson 04's bucket ratio);
- freshness — results pages showing data less than 1 minute old ÷ all pages. Measure as close to the user as you can (the load balancer, not deep inside one service).
SLO (service level objective) — a target for an SLI over a window: "99.9% of results requests succeed, over a rolling 30 days". It is an internal goal the team chooses.
SLA (service level agreement) — a contract with customers, with a consequence (money back, service credits) if it is missed. An SLA is usually looser than the SLO (SLO 99.9%, SLA 99.5%), so the team hears the alarm long before the contract is broken. Many internal services have SLOs and no SLA.
Error budget — 1 − SLO: how much failure the SLO allows. For 99.9% over 30 days: 0.1% of requests, or about 43.2 minutes of complete outage. Each extra "nine" divides the budget by 10: 99% → 432 min, 99.9% → 43.2 min, 99.99% → 4.3 min.
Rolling vs calendar window — a rolling 30-day window always looks at the last 30 days (smooth, used for alerting); a calendar month resets on the 1st (easier for reports).
Error budget policy — a written agreement, signed by product and engineering, of what happens as the budget is used. For example:
- budget left → ship at normal speed; spend it on experiments and planned risk;
- budget nearly gone → slow down: smaller releases, more canary time;
- budget gone → freeze risky changes (features) until the SLI is back; only reliability fixes and security fixes ship;
- one incident used a large part of the budget (for example over 20%) → a postmortem (lesson 12) with action items that are worked on first.
100% is the wrong target — users cannot tell 99.99% from 100% through their own home networks and phones, and a 100% target means no change is ever safe.

🤔 Why

Because "reliable" means nothing until it is a number, and the argument between "ship faster" and "be careful" never ends without one. The budget ends it: reliability becomes a resource that the team spends. With budget left, fear of change costs money for no reason. With the budget gone, more features would break a promise to users. The same number drives the burn-rate alerts in lesson 08.

🔧 How (in this repo)

error_budget(slo, total_events=None, bad_events=0, days=30) in obs/reliability.py has two modes. With only an SLO it returns the budget as minutes of complete outage: (1 − slo) × days × 24 × 60. With event counts it returns allowed_bad (total × (1 − slo)), used, and left_pct. slo() in obs/demo.py takes a 30-day month at 1,000 requests a minute (43,200,000) and the day's failed requests (sum(ERR) = 4,608).

🧪 Try it

More SLOs and a 7-day window; then the budget after different bad days; then the same day measured against a stricter SLO:

python3 obs/demo.py slo
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from reliability import error_budget
for slo in (0.99, 0.995, 0.999, 0.9995, 0.9999):
    print(f"SLO {slo:.2%} → {error_budget(slo)['minutes']:>6} min per 30 days · {error_budget(slo, days=7)['minutes']:>5} min per 7 days")
month = 1000 * 60 * 24 * 30
for bad in (4_608, 21_600, 43_200):
    b = error_budget(0.999, month, bad)
    print(f"{bad:>6,} bad of {month:,} → allowed {b['allowed_bad']:,} · left {b['left_pct']}%")
print("same day on a 99.99% SLO →", error_budget(0.9999, month, 4_608))
EOF
python3 obs/test_obs.py

✅ Verify — what you should see

slo prints:

── SLI: good requests / all requests · SLO: 99.9% over 30 days · SLA: the contract with a penalty
   SLO 99.00% →  432.0 minutes of full outage per 30 days
   SLO 99.90% →   43.2 minutes of full outage per 30 days
   SLO 99.99% →    4.3 minutes of full outage per 30 days
── a 30-day window of 43,200,000 requests may have 43,200 failures · today's 8 hours failed 4,608
   one bad day used 10.7% of the month's budget → 89.3% left
   budget left → ship faster · budget gone → freeze risky changes and fix reliability first

Your snippet prints:

SLO 99.00% →  432.0 min per 30 days · 100.8 min per 7 days
SLO 99.50% →  216.0 min per 30 days ·  50.4 min per 7 days
SLO 99.90% →   43.2 min per 30 days ·  10.1 min per 7 days
SLO 99.95% →   21.6 min per 30 days ·   5.0 min per 7 days
SLO 99.99% →    4.3 min per 30 days ·   1.0 min per 7 days
 4,608 bad of 43,200,000 → allowed 43,200 · left 89.3%
21,600 bad of 43,200,000 → allowed 43,200 · left 50.0%
43,200 bad of 43,200,000 → allowed 43,200 · left 0.0%
same day on a 99.99% SLO → {'allowed_bad': 4320, 'used': 4608, 'left_pct': -6.7}

The tests include ✅ L09 a 99.9% SLO allows 43.2 minutes a month.

🏁 What you just proved

At 99.9%, results day — a slow leak and a bad deploy — used 10.7% of the month: bad, but the policy says "keep shipping, carefully". The same day against 99.99% used more than the whole month (−6.7% left): one day would trigger a freeze. Every extra nine divides the budget by ten, so choose the SLO users need, not the one that sounds best. And a 7-day window at 99.95% allows only 5 minutes — shorter windows react faster and forgive less.

⚠️ Common mistakes

an SLO of 100% — no budget, so every change is a risk nobody can take
SLIs measured on the server's CPU or inside one service instead of close to the user
the SLO copied from the SLA, so the first alarm arrives when the contract is already broken
an error budget policy nobody signed — when the budget runs out, nothing changes
too many SLOs per service; start with one availability and one latency SLO
counting planned maintenance as "not bad" when users could not use the service

🏭 In production

On a real account — the availability SLI as a Prometheus recording rule over 30 days, and the budget left as a series you can put on a Grafana panel (SLO 99.9%):

groups:
  - name: results-api-slo-budget
    rules:
      - record: job:slo_availability:ratio_rate30d
        expr: |
          1 - (
            sum by (job) (increase(http_requests_total{job="results-api", status=~"5.."}[30d]))
            /
            sum by (job) (increase(http_requests_total{job="results-api"}[30d]))
          )
      - record: job:slo_error_budget_remaining:ratio
        expr: 1 - ((1 - job:slo_availability:ratio_rate30d) / (1 - 0.999))

Tools such as Sloth and Pyrra generate these rules — and the lesson 08 burn-rate alerts — from a short SLO file:

version: prometheus/v1
service: results-api
slos:
  - name: requests-availability
    objective: 99.9
    sli:
      events:
        error_query: sum(rate(http_requests_total{job="results-api",status=~"5.."}[{{.window}}]))
        total_query: sum(rate(http_requests_total{job="results-api"}[{{.window}}]))
    alerting:
      name: ResultsApiAvailability
      page_alert: {labels: {severity: page}}
      ticket_alert: {labels: {severity: ticket}}

Grafana (Grafana Cloud SLO), Datadog SLOs and CloudWatch Application Signals also define SLOs in their UI and show the budget left.

A one-page error budget policy to agree and sign:

# Error budget policy — results-api (SLO 99.9% availability, rolling 30 days)
- Budget > 25% left: normal releases.
- Budget < 25% left: releases need a canary of 30 minutes; no big-bang changes.
- Budget spent: feature freeze. Only reliability and security fixes ship until the 30-day SLI is back above 99.9%.
- Any single incident that used > 20% of the budget: postmortem within 5 working days; its P0 action items come before features.
- Disagreements go to: the head of engineering and the product owner, together.
Signed: Dipika (engineering) · Aishwarya (product) · date: 2026-09-27

🏭 Why this matters in production: an SLO without a policy is a graph. Write down, in advance, what the team will stop doing when the budget is gone — and have both engineering and product sign it.

⏭️ Next

The page fires at minute 409. Now what? Who leads, who fixes, who tells the parents? The fire drill.

git checkout lesson-10-incident-response

← Previousalerting Next →incident response

This page is the lesson's README from the lesson-09-slos branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.