🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🔺 धडा 08 — End-to-end tests: pyramid विरुद्ध trophy

📍 तुम्ही इथे आहात: 12 पैकी धडा 08 · मागील: lesson-07-contract-tests · पुढील: lesson-09-property-based

📦 या ब्रँचमध्ये काय आहे

धडे 01–07, आणि त्याशिवाय end-to-end (e2e) tests — संपूर्ण system, बाहेरून, एखादा user जसा वापरेल तसा — आणि प्रत्येक team ज्यावर वाद घालते तो प्रश्न: प्रत्येक प्रकारचे किती tests? Test pyramid, testing trophy आणि ice-cream cone यांची एका cost model ने तुलना. suite_cost() आणि LAYERS exam/lab.py मध्ये; pyramid() exam/demo.py मध्ये.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

शाळा तपासण्याचे तीन मार्ग आहेत.

Unit प्रश्न 📝 — कतरिना एका बाकावर एका विद्यार्थिनीला एक प्रश्न विचारते. प्रत्येकी दोन सेकंद.
Integration प्रश्न 🔌 — ती marks book ची बेरीज खरोखर जुळते का ते तपासते. थोडे हळू.
शाळेचा पूर्ण दिवस 🏫 — ऐश्वर्या फाटकातून आत येते, वर्ग शोधते, परीक्षा देते, आणि report card घरी पाठवले जाते. हे सगळे एकत्र तपासते. पण याला पूर्ण दिवस लागतो, आणि कधी कधी bus उशिरा येते — आणि परीक्षेशी काहीही संबंध नसलेल्या कारणाने तपासणी "fail" होते.

कतरिनाने सगळे पूर्ण शाळेच्या दिवसांनी तपासायचे ठरवले, तर तिला शेकडो दिवस लागतील. प्रत्येक run ला पाव तास लागेल, आणि जवळजवळ प्रत्येक run मध्ये कुठे ना कुठे bus उशिरा येईलच. 🚌

म्हणून ती पूर्ण दिवस थोडे ठेवते, छोटे प्रश्न खूप ठेवते, आणि मधल्या भागावर दीपिकाशी वाद घालते: कतरिनाला खूप unit प्रश्न आवडतात (pyramid 🔺), दीपिकाला खूप integration तपासण्या आवडतात (trophy 🏆). उलटा आकार — बहुतेक पूर्ण दिवस — म्हणजे गोंधळ आहे यावर दोघींचे एकमत आहे (ice-cream cone 🍦).

🗺️ आकृती

flowchart TB
    subgraph pyr["🔺 pyramid 400 / 80 / 10"]
      p["60.0 s per run · flaky 17%"]
    end
    subgraph tro["🏆 trophy 150 / 300 / 10"]
      t["80.8 s per run · flaky 23%"]
    end
    subgraph ice["🍦 ice-cream cone 50 / 50 / 200"]
      i["1005.2 s per run · flaky 87%"]
    end
    note["model: unit 0.005 s · integration 0.1 s · e2e 5 s per test<br/>flake chance 0.01% · 0.05% · 1%"]

🗺️ काढलेली आकृती + एक lab: https://school-edh.pages.dev/testing/lesson-diagrams.html#l08

❓ काय

End-to-end (e2e) test — खरी system बाहेरून चालवतो: browser web app मधून click करत जातो, app खरा API बोलवते, आणि API खरा database वापरतो. हे सिद्ध करते की भाग एकमेकांशी जोडलेले आहेत. हा सर्वात हळू, सर्वात नाजूक प्रकार आहे, आणि तो fail झाला की कुठे चुकले ते क्वचितच सांगतो.
Test pyramid — Mike Cohn, Succeeding with Agile (2009): तळाशी खूप unit tests, त्यावर कमी service/integration tests, टोकाला अगदी थोडे UI tests.
Testing trophy — Kent C. Dodds (2018): पायाशी static checks (types, linters), त्यावर integration tests चा मोठा थर, कमी unit tests, थोडे e2e. युक्तिवाद असा: खऱ्या वापरासारखे दिसणारे tests प्रत्येक test मागे जास्त विश्वास देतात, आणि आधुनिक tools मुळे integration tests पुरेसे वेगवान होतात.
Ice-cream cone — एक anti-pattern: बहुतेक manual किंवा e2e tests, unit tests थोडे. हळू, flaky, सांभाळायला महाग.
Flake — code न बदलता fail होणारा test (धडा 11). प्रत्येक e2e test 1% वेळा flake होत असेल, तर अशा 200 tests चा suite बहुतेक runs मध्ये विनाकारण लाल होतो: एकही flake न होण्याची शक्यता 0.99²⁰⁰ ≈ 13% आहे.
Cost model — demo मधले आकडे (प्रत्येक e2e test साठी 5 s आणि 1% flake शक्यता, वगैरे) उदाहरणादाखल आहेत, पटण्याजोगे म्हणून निवडलेले, मोजलेले नाहीत. तुमच्या स्वतःच्या suite चे आकडे महत्त्वाचे; निकालाचा आकार हाच मुद्दा आहे.

🤔 का

कारण suite चा वेळ आणि त्यावरचा विश्वास यावरच लोक तो चालवतात की नाही हे ठरते. जो suite पाव तास घेतो आणि बहुतेक runs मध्ये विनाकारण लाल होतो, तो वगळला जातो, हिरवा होईपर्यंत पुन्हा चालवला जातो, किंवा delete केला जातो. Pyramid आणि trophy मधल्या भागावर असहमत आहेत पण टोकावर सहमत: e2e tests थोडे ठेवा, आणि ते फक्त पैसे मिळवून देणाऱ्या किंवा मोडल्यास बातमी होईल अशा प्रवासांसाठी.

🔧 कसे (या repo मध्ये)

exam/lab.py मधील LAYERS मध्ये unit, integration आणि e2e साठी (प्रत्येक test चे seconds, test flake होण्याची शक्यता) आहे. suite_cost(counts) वेळ बेरीज करते आणि 1 − Π(1 − p)ⁿ काढते, म्हणजे run मधील किमान एक test flake होण्याची शक्यता. exam/demo.py मधील pyramid() teaching app मधून एक खरा end-to-end pass सुद्धा चालवते: SQLite मध्ये save केलेले marks, fake service कडून attendance, आणि बाहेर एक report card.

🧪 करून पाहा

python3 exam/demo.py pyramid
python3 - <<'EOF'
from exam.lab import suite_cost, LAYERS
print("model:", LAYERS)
for e2e in (0, 10, 50, 200):
    secs, flake = suite_cost({"unit": 400, "integration": 80, "e2e": e2e})
    print(f"400 unit + 80 integration + {e2e:>3} e2e → {secs:>6.1f} s · red for no reason {flake:.1%} of runs")
secs, flake = suite_cost({"unit": 0, "integration": 0, "e2e": 1})
print(f"one e2e test alone: {secs} s, flakes {flake:.1%} · one unit test: {suite_cost({'unit': 1})[0]} s")
EOF

✅ तपासा — तुम्हाला काय दिसायला हवे

pyramid हे छापते:

── one end-to-end run through the teaching app: database → attendance → report card → Aishwarya: average 74.33 · B · attendance 86.0% · promoted True
   pyramid         400 / 80 / 10 tests →   60.0 s per run · a run hits at least one flaky failure 17% of the time
   trophy          150 / 300 / 10 tests →   80.8 s per run · a run hits at least one flaky failure 23% of the time
   ice-cream cone  50 / 50 / 200 tests → 1005.2 s per run · a run hits at least one flaky failure 87% of the time

तुमचा snippet हे छापतो:

model: {'unit': (0.005, 0.0001), 'integration': (0.1, 0.0005), 'e2e': (5.0, 0.01)}
400 unit + 80 integration +   0 e2e →   10.0 s · red for no reason 7.7% of runs
400 unit + 80 integration +  10 e2e →   60.0 s · red for no reason 16.5% of runs
400 unit + 80 integration +  50 e2e →  260.0 s · red for no reason 44.2% of runs
400 unit + 80 integration + 200 e2e → 1010.0 s · red for no reason 87.6% of runs
one e2e test alone: 5.0 s, flakes 1.0% · one unit test: 0.005 s

🏁 तुम्ही आत्ताच काय सिद्ध केले

या model मध्ये एक e2e test 1000 unit tests इतका वेळ घेतो. e2e tests 10 वरून 200 केल्यावर run चा वेळ सुमारे 17 पट झाला आणि बहुतेक runs विनाकारण लाल झाले. तरीही app मधून गेलेला तो e2e pass एका test इतका मोलाचा होता: database, attendance client आणि report card एकमेकांत बसतात हे त्याने दाखवले (74.33 → B, 86.0% → promoted) — जे कोणताही unit test एकट्याने दाखवू शकत नाही. थोडे, पण शून्य नाही.

⚠️ नेहमीच्या चुका

प्रत्येक edge case browser मधून तपासणे (boundary table unit tests मध्ये ठेवा)
एकही e2e test नसणे — प्रत्येक भाग एकटा चालतो आणि login page मात्र मोडलेले असते
element ची वाट पाहण्याऐवजी ठरावीक वेळ sleep करणे (धडा 11)
e2e tests नी एकच test account वापरणे आणि एकमेकांच्या data वर पाय देणे
स्वतःच्या suite चा वेळ आणि flake rate मोजण्याऐवजी pyramid विरुद्ध trophy असा वाद घालणे

🏭 प्रत्यक्ष वापरात

खऱ्या project मध्ये — Python साठी Playwright वापरून एक e2e test (pip install pytest-playwright, मग playwright install chromium). Playwright elements तयार होईपर्यंत आपोआप वाट पाहते, त्यामुळे बरेच flaky sleeps निघून जातात:

from playwright.sync_api import Page, expect

def test_report_card_page(page: Page):
    page.goto("https://staging.school.example/report/42")
    expect(page.get_by_role("heading", name="Report card")).to_be_visible()
    expect(page.get_by_test_id("grade")).to_have_text("B")

pytest tests/e2e --browser chromium --tracing retain-on-failure   # keep a trace to replay failures

JavaScript runner म्हणजे npx playwright test; Cypress आणि Selenium हे इतर नेहमीचे पर्याय आहेत. तुमचे खरे थर मोजा:

pytest --durations=10              # the 10 slowest tests

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: तुमचे 5–10 महत्त्वाचे user प्रवास लिहून काढा (log in, marks submit करणे, report cards publish करणे) आणि प्रत्येकाला एक e2e test द्या. बाकी सगळे pyramid मध्ये खालच्या थरात — आणि suite चा run वेळ व flake rate इतर कोणत्याही production metric सारखा track करा.

⏭️ पुढे

आत्तापर्यंत कतरिना प्रत्येक प्रश्न स्वतः निवडते. संगणकानेच एका नियमावरून शेकडो प्रश्न तयार केले — आणि मग failure शक्य तितका लहान केला तर? Property-based testing.

git checkout lesson-09-property-based

🔺 Lesson 08 — End-to-end tests: the pyramid vs the trophy

📍 You are here: Lesson 08 of 12 · Previous: lesson-07-contract-tests · Next: lesson-09-property-based

📦 What's in this branch

Lessons 01–07, plus end-to-end (e2e) tests — the whole system, from the outside, as a user would use it — and the question every team argues about: how many tests of each kind? The test pyramid, the testing trophy and the ice-cream cone, compared with a cost model. suite_cost() and LAYERS in exam/lab.py; pyramid() in exam/demo.py.

🧒 Explain like I'm 5

There are three ways to check the school.

Unit questions 📝 — Katrina asks one pupil one question at a desk. Two seconds each.
Integration questions 🔌 — she checks the marks book really adds up. A bit slower.
The full school day 🏫 — Aishwarya walks in at the gate, finds the classroom, sits the exam, gets the report card posted home. It checks everything together. But it takes all day, and sometimes the bus is late — and the check "fails" for a reason that has nothing to do with the exam.

If Katrina tried to test everything with full school days, she would need hundreds of them. Every run would take a quarter of an hour, and almost every run would have a late bus somewhere. 🚌

So she keeps few full days, many small questions, and argues with Dipika about the middle: Katrina likes lots of unit questions (the pyramid 🔺), Dipika likes lots of integration checks (the trophy 🏆). Both agree the upside-down shape — mostly full days — is a mess (the ice-cream cone 🍦).

🗺️ Diagram

flowchart TB
    subgraph pyr["🔺 pyramid 400 / 80 / 10"]
      p["60.0 s per run · flaky 17%"]
    end
    subgraph tro["🏆 trophy 150 / 300 / 10"]
      t["80.8 s per run · flaky 23%"]
    end
    subgraph ice["🍦 ice-cream cone 50 / 50 / 200"]
      i["1005.2 s per run · flaky 87%"]
    end
    note["model: unit 0.005 s · integration 0.1 s · e2e 5 s per test<br/>flake chance 0.01% · 0.05% · 1%"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/testing/lesson-diagrams.html#l08

❓ What

End-to-end (e2e) test — drives the real system from the outside: a browser clicks through the web app, which calls the real API, which uses the real database. It proves the parts are wired together. It is the slowest, most fragile kind, and when it fails it rarely says where.
Test pyramid — Mike Cohn, Succeeding with Agile (2009): many unit tests at the bottom, fewer service/integration tests, very few UI tests at the top.
Testing trophy — Kent C. Dodds (2018): static checks (types, linters) at the base, a larger layer of integration tests, fewer unit tests, few e2e. The argument: tests that resemble real use give more confidence per test, and modern tools make integration tests fast enough.
Ice-cream cone — an anti-pattern: mostly manual or e2e tests, few unit tests. Slow, flaky, expensive to keep.
Flake — a test that fails with no code change (lesson 11). If each e2e test flakes 1% of the time, a suite of 200 of them is red for no reason in most runs: the chance of no flake is 0.99²⁰⁰ ≈ 13%.
The cost model — the demo's numbers (5 s and a 1% flake chance per e2e test, and so on) are illustrative, chosen to be plausible, not measured. Your own suite's numbers are what matter; the shape of the result is the point.

🤔 Why

Because the time and trust of a suite decide whether people run it. A suite that takes a quarter of an hour and goes red for no reason in most runs gets skipped, rerun until green, or deleted. The pyramid and the trophy disagree on the middle but agree on the top: keep e2e tests few, covering the journeys that make money or would make the news if they broke.

🔧 How (in this repo)

LAYERS in exam/lab.py holds (seconds per test, chance a test flakes) for unit, integration and e2e. suite_cost(counts) adds up the time and computes 1 − Π(1 − p)ⁿ, the chance that at least one test in the run flakes. pyramid() in exam/demo.py also runs one real end-to-end pass through the teaching app: marks saved in SQLite, attendance from the fake service, a report card out.

🧪 Try it

python3 exam/demo.py pyramid
python3 - <<'EOF'
from exam.lab import suite_cost, LAYERS
print("model:", LAYERS)
for e2e in (0, 10, 50, 200):
    secs, flake = suite_cost({"unit": 400, "integration": 80, "e2e": e2e})
    print(f"400 unit + 80 integration + {e2e:>3} e2e → {secs:>6.1f} s · red for no reason {flake:.1%} of runs")
secs, flake = suite_cost({"unit": 0, "integration": 0, "e2e": 1})
print(f"one e2e test alone: {secs} s, flakes {flake:.1%} · one unit test: {suite_cost({'unit': 1})[0]} s")
EOF

✅ Verify — what you should see

pyramid prints:

── one end-to-end run through the teaching app: database → attendance → report card → Aishwarya: average 74.33 · B · attendance 86.0% · promoted True
   pyramid         400 / 80 / 10 tests →   60.0 s per run · a run hits at least one flaky failure 17% of the time
   trophy          150 / 300 / 10 tests →   80.8 s per run · a run hits at least one flaky failure 23% of the time
   ice-cream cone  50 / 50 / 200 tests → 1005.2 s per run · a run hits at least one flaky failure 87% of the time

Your snippet prints:

model: {'unit': (0.005, 0.0001), 'integration': (0.1, 0.0005), 'e2e': (5.0, 0.01)}
400 unit + 80 integration +   0 e2e →   10.0 s · red for no reason 7.7% of runs
400 unit + 80 integration +  10 e2e →   60.0 s · red for no reason 16.5% of runs
400 unit + 80 integration +  50 e2e →  260.0 s · red for no reason 44.2% of runs
400 unit + 80 integration + 200 e2e → 1010.0 s · red for no reason 87.6% of runs
one e2e test alone: 5.0 s, flakes 1.0% · one unit test: 0.005 s

🏁 What you just proved

In this model, one e2e test costs as much time as 1000 unit tests. Going from 10 to 200 e2e tests multiplied the run time by about 17 and made most runs red for no reason. The e2e pass through the app was still worth one test: it showed that the database, the attendance client and the report card fit together (74.33 → B, 86.0% → promoted) — which no unit test alone can show. Few, not none.

⚠️ Common mistakes

testing every edge case through the browser (put the boundary table in unit tests)
no e2e tests at all — every part works alone and the login page is broken
sleeping a fixed time instead of waiting for an element (lesson 11)
e2e tests sharing one test account and stepping on each other's data
arguing pyramid vs trophy instead of measuring your own suite's time and flake rate

🏭 In production

On a real project — an e2e test with Playwright for Python (pip install pytest-playwright, then playwright install chromium). Playwright waits automatically for elements to be ready, which removes many flaky sleeps:

from playwright.sync_api import Page, expect

def test_report_card_page(page: Page):
    page.goto("https://staging.school.example/report/42")
    expect(page.get_by_role("heading", name="Report card")).to_be_visible()
    expect(page.get_by_test_id("grade")).to_have_text("B")

pytest tests/e2e --browser chromium --tracing retain-on-failure   # keep a trace to replay failures

The JavaScript runner is npx playwright test; Cypress and Selenium are the other common choices. Measure your real layers:

pytest --durations=10              # the 10 slowest tests

🏭 Why this matters in production: write down your 5–10 critical user journeys (log in, submit marks, publish report cards) and give each one e2e test. Everything else goes lower in the pyramid — and track the suite's run time and flake rate like any other production metric.

⏭️ Next

So far Katrina chooses every question herself. What if the computer invented hundreds of questions from a rule — and then made a failure as small as possible? Property-based testing.

git checkout lesson-09-property-based

← Previouscontract tests Next →property based

This page is the lesson's README from the lesson-08-e2e-pyramid branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.