🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🧬 धडा 10 — Coverage विरुद्ध mutation testing: 100% जी फक्त काही mutants मारते

📍 तुम्ही इथे आहात: 12 पैकी धडा 10 · मागील: lesson-09-property-based · पुढील: lesson-11-flaky-tests

📦 या ब्रँचमध्ये काय आहे

धडे 01–09, आणि त्याशिवाय tests चीच परीक्षा घेण्याचे दोन मार्ग: line coverage (suite चालताना grade() च्या कोणत्या lines चालल्या) आणि mutation testing (grade() मध्ये छोटे bugs — mutants — पेरा आणि suite त्यातले किती पकडतो ते मोजा). 100% line coverage असलेला suite 10 पैकी फक्त 4 mutants मारतो. line_coverage(), MUTANTS, mutation_run(), WEAK आणि STRONG exam/lab.py मध्ये; mutation() exam/demo.py मध्ये.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

कतरिनाला फक्त विद्यार्थिनी नव्हे, तर तिची प्रश्नपत्रिका चांगली आहे का हे जाणून घ्यायचे आहे.

पहिली कल्पना: coverage. 📋 पाठ्यपुस्तकाच्या प्रत्येक पानाला किमान एका प्रश्नाने स्पर्श केला का हे ती तपासते. 7 प्रश्नांची कमकुवत प्रश्नपत्रिका प्रत्येक पानाला स्पर्श करते: 100%! 🎉

दुसरी कल्पना: mutants. 🧬 ऐश्वर्या गुपचूप विद्यार्थिनीच्या 10 प्रती बनवते, प्रत्येकीत एक छोटी चूक: एकीला वाटते pass होण्यासाठी किमान 35 नव्हे तर 35 पेक्षा जास्त हवेत; एक अर्धे गुण जुन्या चुकीच्या पद्धतीने round करते; एक जिथे "A" बरोबर होता तिथे "B" देते. मग कतरिनाची प्रश्नपत्रिका प्रत्येक प्रतीला दिली जाते.

प्रश्नपत्रिकेने चूक पकडली, तर तो mutant मारला गेला (killed). ✅
प्रत प्रश्नपत्रिका तरीही pass झाली, तर mutant टिकला (survived) — प्रश्नपत्रिकेला चुकीची विद्यार्थिनी आणि बरोबर विद्यार्थिनी यांतला फरक ओळखता आला नाही. ❌

100% coverage असलेली कमकुवत प्रश्नपत्रिका 10 पैकी फक्त 4 मारते. तिने प्रत्येक पानाला स्पर्श केला, पण कडांबद्दल कधीच विचारले नाही. धडा 03 मधल्या boundary प्रश्नांसह असलेली मजबूत प्रश्नपत्रिका 9 मारते.

आणि 10वा? त्याने "किमान 75" चे "74 पेक्षा जास्त" केले. पूर्ण संख्यांसाठी हा तोच नियम आहे. कोणताही प्रश्न त्याला कधीच पकडू शकत नाही, कारण ती चूकच नाही. त्यालाच equivalent mutant म्हणतात.

🗺️ आकृती

flowchart LR
    subgraph weak["📄 WEAK suite: 7 questions"]
      w["coverage 12/12 = 100%<br/>mutants killed 4/10"]
    end
    subgraph strong["📑 STRONG suite: 21 questions"]
      s["coverage 12/12 = 100%<br/>mutants killed 9/10"]
    end
    weak -->|"add boundary questions<br/>34.5 · 35 · 60 · 74.5 · 75 · 100.1"| strong
    strong --> eq["the survivor: s >= 75 → s > 74<br/>EQUIVALENT for whole numbers"]

🗺️ काढलेली आकृती + एक lab: https://school-edh.pages.dev/testing/lesson-diagrams.html#l10

❓ काय

Line (statement) coverage — tests दरम्यान किमान एकदा चाललेल्या executable lines चा वाटा. grade() मध्ये 12 statement lines आहेत.
Branch coverage — घेतलेल्या branch दिशांचा वाटा (प्रत्येक if true आणि false दोन्ही). Line coverage पेक्षा कडक, पण तरीही asserts बद्दल काहीच सांगत नाही.
Coverage काय सांगते — ज्या lines कधीच चालल्या नाहीत त्या नक्कीच test झालेल्या नाहीत. ज्या चालल्या त्याही तपासलेल्या नसू शकतात: assert नसलेली test सुद्धा पूर्ण coverage देते.
Mutant — mutation operator ने केलेला एक छोटा बदल असलेली code ची copy: >= → >, and ↔ or, एखादा constant ± 1, वेगळी return value, बदललेला call.
Killed / survived — mutant वर किमान एक test fail झाली तर killed; पूर्ण suite तरीही pass झाला तर survived. टिकलेला mutant एखाद्या नसलेल्या किंवा कमकुवत test कडे बोट दाखवतो.
Equivalent mutant — वागणूक न बदलणारा बदल (इथे, s पूर्ण संख्या असताना s >= 75 विरुद्ध s > 74). कोणतीही test त्याला मारू शकत नाही; tools ना ते खऱ्या survivors पासून नेहमी वेगळे ओळखता येत नाहीत, म्हणून एखाद्या माणसाने पाहावे लागते.
Mutation score — killed ÷ (सर्व mutants − equivalent mutants). इथे मजबूत suite चा score 9 ÷ 9 = 100%, कमकुवतचा 4 ÷ 9 ≈ 44%.
किंमत — प्रत्येक mutant साठी suite पुन्हा चालतो. खरे tools हजारो mutants बनवतात, म्हणून ते प्रत्येक बदललेली line cover करणाऱ्या tests च चालवतात, आणि बऱ्याचदा रात्री किंवा फक्त बदललेल्या code वर चालवले जातात.

🤔 का

कारण coverage मोजायला सोपी आणि फसवायलाही सोपी आहे, आणि teams असे coverage targets ठरवतात जे assert नसलेल्या tests सुद्धा गाठू शकतात. Mutation testing तुम्हाला खरोखर हवे ते मोजते: suite ला bug लक्षात येईल का? टिकलेले mutants म्हणजे नेमके कोणते प्रश्न राहिले आहेत याची यादी — score > 100 → score > 101 टिकला म्हणजे "100.1 बद्दल कोणीच विचारले नाही".

🔧 कसे (या repo मध्ये)

exam/lab.py मधील line_coverage(fn, suite) sys.settrace ने एक trace function लावते, suite चालताना fn च्या code object मधील प्रत्येक line event नोंदवते, आणि fn चा source ast ने parse करून सापडलेल्या statement lines शी तुलना करते. MUTANTS मध्ये 10 (जुना मजकूर, नवा मजकूर) जोड्या आहेत; mutant(old, new) grade() चा source पुन्हा लिहिते आणि नव्या namespace मध्ये exec करते. mutation_run(suite) प्रत्येक mutant वर suite चालवते आणि कोणता case fail झाला का ते नोंदवते. WEAK मध्ये वर्गाच्या मधल्या भागातले 7 cases आहेत; STRONG त्यात धडा 03 मधला boundary table जोडतो.

🧪 करून पाहा

python3 exam/demo.py mutation
python3 - <<'EOF'
from exam.lab import line_coverage, mutation_run, statement_lines, WEAK, STRONG
from exam.grades import grade
print("statement lines of grade():", statement_lines(grade))
for name, suite in (("one question", [(90, "A")]), ("WEAK", WEAK), ("WEAK + 3 edges", WEAK + [(34.5, "D"), (75, "A"), (100.1, ValueError)]), ("STRONG", STRONG)):
    hit, total = line_coverage(grade, suite)
    res = mutation_run(suite)
    print(f"{name:<15} coverage {hit:>2}/{total} · killed {sum(k for *_, k in res):>2}/10 · survivors {[n for o, n, k in res if not k]}")
EOF

✅ तपासा — तुम्हाला काय दिसायला हवे

mutation हे छापते:

── WEAK suite (7 questions): line coverage of grade() 12/12 = 100% · mutants killed 4/10
   s >= 75               → s > 75         SURVIVED
   s >= 35               → s > 35         SURVIVED
   s >= 60               → s >= 61        SURVIVED
   round_half_up(score)  → round(score)   SURVIVED
   return "A"            → return "B"     killed
── STRONG suite (21 questions): line coverage of grade() 12/12 = 100% · mutants killed 9/10

तुमचा snippet हे छापतो:

statement lines of grade(): [27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38]
one question    coverage  4/12 · killed  1/10 · survivors ['s > 75', 's > 35', 's >= 61', 'round(score)', 'return "D"', 'score < 0 and', 's <= 45', 'score > 101', 's > 74']
WEAK            coverage 12/12 · killed  4/10 · survivors ['s > 75', 's > 35', 's >= 61', 'round(score)', 'score > 101', 's > 74']
WEAK + 3 edges  coverage 12/12 · killed  8/10 · survivors ['s >= 61', 's > 74']
STRONG          coverage 12/12 · killed  9/10 · survivors ['s > 74']

🏁 तुम्ही आत्ताच काय सिद्ध केले

7 प्रश्नांनंतर coverage ची मदत थांबली: WEAK पासून STRONG पर्यंत ती 100% वरच राहिली, तर मारलेल्यांची संख्या 4 वरून 9 झाली. WEAK suite ने धडा 01 मध्ये ship झालेला नेमका bug — round_half_up(score) → round(score) — पूर्ण coverage असूनही टिकू दिला. नीट निवडलेल्या तीन कडेच्या प्रश्नांनी आणखी चार mutants मारले, आणि उरलेल्या प्रत्येक survivor ने अजून कोणता प्रश्न राहिला आहे ते सांगितले (s >= 61 साठी नेमक्या 60 वर प्रश्न हवा). शेवटचा, s > 74, equivalent आहे: तिथे थांबा.

⚠️ नेहमीच्या चुका

coverage target (समजा 80%) ला quality target मानणे — ती फक्त किमान पातळी आहे
coverage वाढवण्यासाठी लिहिलेल्या, code बोलवणाऱ्या पण assert न करणाऱ्या tests
प्रत्येक commit वर पूर्ण code base वर mutation testing चालवणे (बदललेल्या code वर, किंवा रात्री चालवा)
equivalent mutants च्या मागे तासन्तास धावणे
line coverage 100% असताना branch coverage कडे दुर्लक्ष करणे

🏭 प्रत्यक्ष वापरात

खऱ्या project मध्ये — coverage.py किंवा pytest-cov plugin ने coverage; -m आणि term-missing कधीच न चाललेल्या lines ची यादी देतात:

pip install coverage pytest pytest-cov
coverage run --branch -m pytest && coverage report -m
pytest --cov=exam --cov-branch --cov-report=term-missing --cov-fail-under=90

mutmut ने mutation testing (Python; ते तुमचा pytest suite प्रत्येक mutant वर चालवते):

pip install mutmut
mutmut run            # generate mutants and run the tests against each one
mutmut results        # list the survivors
mutmut show <mutant>  # the diff of one survivor

इतर tools: Java साठी PIT (PITest), JavaScript/TypeScript, C# आणि Scala साठी Stryker, Python साठी cosmic-ray. Google ने सांगितले आहे की ते टिकलेले mutants project-wide score म्हणून नव्हे, तर code review मध्ये comments म्हणून, फक्त बदललेल्या lines वर दाखवतात.

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: test न झालेल्या code साठी coverage एक स्वस्त धोक्याची घंटा म्हणून ठेवा, आणि जिथे चुकीचे उत्तर महाग पडते अशा थोड्या modules वर mutation testing चालवा — grading, billing, permissions. प्रत्येक survivor म्हणजे एकतर नसलेली test किंवा equivalent mutant; दोन्ही माहीत असणे फायद्याचे आहे.

⏭️ पुढे

चांगला suite सुद्धा दुसऱ्या प्रकारे खोटे बोलू शकतो: code मध्ये काहीही बदल न करता सोमवारी pass आणि मंगळवारी fail होणारी test. भिंतीवरचे डुगडुगणारे घड्याळ — flaky tests.

git checkout lesson-11-flaky-tests

🧬 Lesson 10 — Coverage vs mutation testing: 100% that kills only some mutants

📍 You are here: Lesson 10 of 12 · Previous: lesson-09-property-based · Next: lesson-11-flaky-tests

📦 What's in this branch

Lessons 01–09, plus two ways to grade the tests themselves: line coverage (which lines of grade() ran while the suite ran) and mutation testing (plant small bugs — mutants — in grade() and count how many the suite catches). A suite with 100% line coverage kills only 4 of 10 mutants. line_coverage(), MUTANTS, mutation_run(), WEAK and STRONG in exam/lab.py; mutation() in exam/demo.py.

🧒 Explain like I'm 5

Katrina wants to know if her exam paper is good, not just the pupil.

First idea: coverage. 📋 She checks that every page of the textbook was touched by at least one question. The weak paper, with 7 questions, touches every page: 100%! 🎉

Second idea: mutants. 🧬 Aishwarya secretly makes 10 copies of the pupil, each with one small mistake: one thinks a pass needs more than 35 instead of at least 35; one rounds halves the old wrong way; one gives "B" where "A" was right. Then Katrina's paper is given to each copy.

If the paper catches the mistake, that mutant is killed. ✅
If the copy passes the paper anyway, the mutant survived — the paper could not tell a wrong pupil from a right one. ❌

The weak paper, with its 100% coverage, kills only 4 of the 10. It touched every page but never asked about the edges. The strong paper, with the boundary questions from lesson 03, kills 9.

And the 10th? It changed "at least 75" to "more than 74". For whole numbers, that is the same rule. No question can ever catch it, because it is not a mistake. That is an equivalent mutant.

🗺️ Diagram

flowchart LR
    subgraph weak["📄 WEAK suite: 7 questions"]
      w["coverage 12/12 = 100%<br/>mutants killed 4/10"]
    end
    subgraph strong["📑 STRONG suite: 21 questions"]
      s["coverage 12/12 = 100%<br/>mutants killed 9/10"]
    end
    weak -->|"add boundary questions<br/>34.5 · 35 · 60 · 74.5 · 75 · 100.1"| strong
    strong --> eq["the survivor: s >= 75 → s > 74<br/>EQUIVALENT for whole numbers"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/testing/lesson-diagrams.html#l10

❓ What

Line (statement) coverage — the share of executable lines that ran at least once during the tests. grade() has 12 statement lines.
Branch coverage — the share of branch directions taken (each if both true and false). Stricter than line coverage, still says nothing about the asserts.
What coverage tells you — lines that never ran are certainly untested. Lines that ran may still be unchecked: a test with no assert gives full coverage.
Mutant — a copy of the code with one small change made by a mutation operator: >= → >, and ↔ or, a constant ± 1, a different return value, a call swapped.
Killed / survived — killed if at least one test fails on the mutant; survived if the whole suite still passes. A survivor points at a missing or weak test.
Equivalent mutant — a change that does not change behaviour (here, s >= 75 vs s > 74 when s is a whole number). No test can kill it; tools cannot always tell them apart from real survivors, so a person has to look.
Mutation score — killed ÷ (all mutants − equivalent ones). Here the strong suite scores 9 ÷ 9 = 100%, the weak one 4 ÷ 9 ≈ 44%.
The cost — every mutant runs the suite again. Real tools make thousands of mutants, so they run only the tests that cover each mutated line, and are often run nightly or on changed code only.

🤔 Why

Because coverage is easy to measure and easy to game, and teams set coverage targets that tests without asserts can meet. Mutation testing measures what you actually want: would the suite notice a bug? The survivors are a to-do list of exactly which questions are missing — score > 100 → score > 101 surviving says "nobody asked about 100.1".

🔧 How (in this repo)

line_coverage(fn, suite) in exam/lab.py sets a trace function with sys.settrace, records every line event inside fn's code object while the suite runs, and compares with the statement lines found by parsing fn's source with ast. MUTANTS lists 10 (old text, new text) pairs; mutant(old, new) rewrites the source of grade() and execs it into a fresh namespace. mutation_run(suite) runs the suite against each mutant and records whether any case failed. WEAK has 7 middle-of-class cases; STRONG adds the boundary table from lesson 03.

🧪 Try it

python3 exam/demo.py mutation
python3 - <<'EOF'
from exam.lab import line_coverage, mutation_run, statement_lines, WEAK, STRONG
from exam.grades import grade
print("statement lines of grade():", statement_lines(grade))
for name, suite in (("one question", [(90, "A")]), ("WEAK", WEAK), ("WEAK + 3 edges", WEAK + [(34.5, "D"), (75, "A"), (100.1, ValueError)]), ("STRONG", STRONG)):
    hit, total = line_coverage(grade, suite)
    res = mutation_run(suite)
    print(f"{name:<15} coverage {hit:>2}/{total} · killed {sum(k for *_, k in res):>2}/10 · survivors {[n for o, n, k in res if not k]}")
EOF

✅ Verify — what you should see

mutation prints:

── WEAK suite (7 questions): line coverage of grade() 12/12 = 100% · mutants killed 4/10
   s >= 75               → s > 75         SURVIVED
   s >= 35               → s > 35         SURVIVED
   s >= 60               → s >= 61        SURVIVED
   round_half_up(score)  → round(score)   SURVIVED
   return "A"            → return "B"     killed
── STRONG suite (21 questions): line coverage of grade() 12/12 = 100% · mutants killed 9/10

Your snippet prints:

statement lines of grade(): [27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38]
one question    coverage  4/12 · killed  1/10 · survivors ['s > 75', 's > 35', 's >= 61', 'round(score)', 'return "D"', 'score < 0 and', 's <= 45', 'score > 101', 's > 74']
WEAK            coverage 12/12 · killed  4/10 · survivors ['s > 75', 's > 35', 's >= 61', 'round(score)', 'score > 101', 's > 74']
WEAK + 3 edges  coverage 12/12 · killed  8/10 · survivors ['s >= 61', 's > 74']
STRONG          coverage 12/12 · killed  9/10 · survivors ['s > 74']

🏁 What you just proved

Coverage stopped helping at 7 questions: from WEAK to STRONG it stayed at 100% while the kill count went from 4 to 9. The WEAK suite let the exact bug that shipped in lesson 01 — round_half_up(score) → round(score) — survive with full coverage. Three well-chosen edge questions killed four more mutants, and each remaining survivor named the question still missing (s >= 61 needs a question at exactly 60). The last one, s > 74, is equivalent: stop there.

⚠️ Common mistakes

a coverage target (say 80%) treated as a quality target — it is only a floor
tests that call code without asserting, written to lift coverage
running mutation testing on the whole code base every commit (run it on changed code, or nightly)
chasing equivalent mutants for hours
ignoring branch coverage when line coverage is 100%

🏭 In production

On a real project — coverage with coverage.py or the pytest-cov plugin; -m and term-missing list the lines that never ran:

pip install coverage pytest pytest-cov
coverage run --branch -m pytest && coverage report -m
pytest --cov=exam --cov-branch --cov-report=term-missing --cov-fail-under=90

Mutation testing with mutmut (Python; it runs your pytest suite against each mutant):

pip install mutmut
mutmut run            # generate mutants and run the tests against each one
mutmut results        # list the survivors
mutmut show <mutant>  # the diff of one survivor

Other tools: PIT (PITest) for Java, Stryker for JavaScript/TypeScript, C# and Scala, cosmic-ray for Python. Google has described showing surviving mutants as comments in code review, on changed lines only, rather than a project-wide score.

🏭 Why this matters in production: keep coverage as a cheap alarm for untested code, and run mutation testing on the few modules where a wrong answer is expensive — grading, billing, permissions. Every survivor is either a missing test or an equivalent mutant; both are worth knowing.

⏭️ Next

A good suite can still lie in another way: a test that passes on Monday and fails on Tuesday with no code change. The wobbly clock on the wall — flaky tests.

git checkout lesson-11-flaky-tests

← Previousproperty based Next →flaky tests

This page is the lesson's README from the lesson-10-coverage-mutation branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.