🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🏁 धडा 02 — प्रामाणिक benchmarking: warm-up फेऱ्या आणि अनेक heats

📍 तुम्ही इथे आहात: 12 पैकी धडा 02 · मागे: lesson-01-latency-percentiles · पुढे: lesson-03-profiling

📦 या ब्रँचमध्ये काय आहे

धडा 01, अधिक एका program च्या दोन versions ची प्रामाणिक तुलना. एक run खोटे बोलतो: पहिले runs थंड असतात, machine मध्ये noise असतो, आणि छोटे बदल त्या noise मध्ये लपतात. यावर उपाय: warm up करा, पुन्हा पुन्हा चालवा, median आणि पसारा (spread) सांगा, आणि फायदा तेव्हाच जाहीर करा जेव्हा तो noise पेक्षा मोठा असेल. perf/demo.py मधले bench() आणि perf/sim.py मधले simulated_runs, spread आणि verdict.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

नवीन धावण्याच्या बुटांमुळे ऐश्वर्या जलद धावते का, हे दीपिकाला जाणून घ्यायचे आहे. 👟

कतरिना जुन्या बुटांत ऐश्वर्याची वेळ एकदाच घेते. भल्या सकाळी ऐश्वर्या थंड आणि आखडलेली असते: 42 seconds. नंतर, नवीन बुटांत: 12 seconds. "नवीन बूट 3 पट जलद आहेत!" नाही — ऐश्वर्या फक्त warm up होत होती.

म्हणून त्या हे नीट करतात:

आधी काही warm-up फेऱ्या, ज्या मोजल्या जात नाहीत 🔥
मग एक नव्हे, अनेक heats 🔁
मग मधली वेळ (median) आणि वेळा किती पसरलेल्या आहेत ते पाहतात
ज्या heat मध्ये कुत्रा track वर धावत आला 🐕 ती एक विचित्र वेळ आहे — median ला त्याची फारशी पर्वा नसते

मग त्या तुलना करतात. जुन्या आणि नव्या वेळा खूप overlap होत असतील, तर बुटांनी कोणताही स्पष्ट फरक केला नाही. नव्या वेळा स्पष्टपणे कमी असतील तरच तो खरा फायदा आहे.

🗺️ आकृती

flowchart LR
    cold["🥶 3 cold runs<br/>42.1 · 31.7 · 21.6 ms"] -->|"drop them"| warm["🔁 17 warm runs"]
    warm --> med["median 12.4 ms<br/>middle half 11.7–12.7"]
    b1["B1 tweak: middle half 11.3–12.3"] --> cmp{"do the middle<br/>halves overlap?"}
    b2["B2 fix: middle half 7.8–8.4"] --> cmp
    med --> cmp
    cmp -->|"B1 overlaps"| no["no clear difference"]
    cmp -->|"B2 is clear"| yes["B is faster (1.53x)"]

🗺️ काढलेली आकृती + एक lab: https://school-edh.pages.dev/performance/lesson-diagrams.html#l02

❓ काय

Benchmark — code चा एखादा तुकडा (किंवा संपूर्ण program) नियंत्रित परिस्थितीत अनेक वेळा चालवून मोजणे. Micro-benchmark एखाद्या अगदी छोट्या गोष्टीची वेळ घेतो, जसे एक function; macro-benchmark एखाद्या खऱ्या task ची सुरुवातीपासून शेवटपर्यंत वेळ घेतो.
Warm-up — पहिल्या runs ना एकदाच येणारे खर्च भरावे लागतात: थंड CPU caches, operating system च्या page cache मध्ये अजून न आलेल्या files, lazy imports, connection set-up, आणि काही runtimes मध्ये असा JIT compiler ज्याने अजून code optimise केलेला नाही. ते runs टाकून द्या (किंवा users ना cold start दिसत असेल तर ते मुद्दाम मोजा).
Noise — इतर programs, CPU frequency मधले बदल (turbo boost, गरम laptop हळू होणे), interrupts आणि memory layout हे सगळे आकडे हलवतात. Simulated स्टॉपवॉच ±8% jitter आणि अधूनमधून एक spike जोडते.
Median आणि पसारा — median आणि मधला अर्धा भाग (p25 ते p75, म्हणजे interquartile range) सांगा, किंवा min/median/max. काही tools minimum सांगतात, कारण noise नेहमी फक्त वेळच वाढवतो असा त्यांचा युक्तिवाद असतो; हे छोट्या, फक्त-CPU code ला शोभते, पण खरा बदल लपवते.
दोन versions ची तुलना — lab चे verdict एक अंदाजी नियम वापरते: B ला जलद म्हणा फक्त तेव्हा जेव्हा B चा p75 हा A च्या p25 पेक्षा कमी असेल (मधले अर्धे भाग overlap होत नाहीत). खरी tools statistics वापरतात (उदाहरणार्थ, pyperf चे compare_to फरक significant आहे की नाही ते सांगते).
time.perf_counter() — एका machine वर वेळ मोजण्यासाठीचे घड्याळ: monotonic आणि high-resolution. time.time() ने code ची वेळ घेऊ नका (system clock दुरुस्त केल्यावर ते उडी मारू शकते).

🤔 का

कारण पुढचा प्रत्येक धडा "जलद करण्यासाठी" code बदलतो. प्रामाणिक मोजमापाशिवाय तुम्ही काहीच न करणारे बदल ठेवाल, मदत करणारे बदल फेकून द्याल, आणि noise असलेल्या आकड्यांवर वाद घालाल. जो benchmark 3% बदल पाहू शकत नाही, तो तसा दावा करण्यासाठी वापरू नये.

🔧 कसे (या repo मध्ये)

perf/sim.py मधले simulated_runs(base_ms, n, seed) हे एक simulated स्टॉपवॉच आहे: प्रत्येक run ला base_ms ± 8% (seeded) लागतो, पहिले 3 runs warm-up खर्च जोडतात (30, 20, 10 ms), आणि 5% runs एक spike जोडतात. spread(xs) min, p25, median, p75 आणि max परत देते. verdict(a, b) वरचा अंदाजी नियम लागू करते. perf/demo.py मधले bench() version A (12 ms), छोटा tweak B1 (11.7 ms) आणि खरा fix B2 (8 ms) यांची वेळ घेते. करून पाहा ची शेवटची ओळ Python चे खरे timeit वापरते — खरे घड्याळ.

🧪 करून पाहा

python3 perf/demo.py bench
python3 - <<'EOF'
import sys; sys.path.insert(0, "perf"); from sim import simulated_runs, spread, verdict
a = simulated_runs(12.0)[3:]
for base in (12.0, 11.5, 11.0, 10.5, 10.0, 9.0):
    b = simulated_runs(base, seed=6)[3:]
    print(f"B at {base:>4} ms → median {spread(b)['median']:>4} · {verdict(a, b)}")
for n in (3, 5, 10, 20):
    print(f"{n:>2} runs, warm-up kept → median {spread(simulated_runs(12.0, n=n))['median']:>4} ms")
EOF
python3 -m timeit -n 1000 -r 5 "sorted(range(1000, 0, -1))"

शेवटचा command तुमच्या machine वर खऱ्या sort ची वेळ घेतो — काहीसे असे: 1000 loops, best of 5: 5.62 usec per loop. तुमचे आकडे वेगळे असतील, प्रत्येक run ला आणि प्रत्येक machine वर; हाच तर या धड्याचा मुद्दा आहे.

✅ तपासा — तुम्हाला काय दिसायला हवे

bench हे छापते:

── Katrina times version A 20 times (simulated stopwatch): first three runs [42.1, 31.7, 21.6] ms — cold, warming up
   one run says 42.1 ms · the mean of all 20 says 15.5 ms · after 3 warm-up runs, the median of 17 says 12.4 ms
   spread of the 17: min 11.2 · p25 11.7 · median 12.4 · p75 12.7 · max 19.9 (one spike: another program woke up)
── A vs B1 (a small tweak): medians 12.4 vs 12.0 ms · middle halves 11.7–12.7 vs 11.3–12.3 → no clear difference: the change is smaller than the noise
── A vs B2 (a real fix): medians 12.4 vs 8.1 ms · middle halves 11.7–12.7 vs 7.8–8.4 → B is faster (1.53x)

तुमचा snippet हे छापतो:

B at 12.0 ms → median 12.3 · no clear difference: the change is smaller than the noise
B at 11.5 ms → median 11.8 · no clear difference: the change is smaller than the noise
B at 11.0 ms → median 11.3 · B is faster (1.10x)
B at 10.5 ms → median 10.8 · B is faster (1.15x)
B at 10.0 ms → median 10.2 · B is faster (1.22x)
B at  9.0 ms → median  9.2 · B is faster (1.35x)
 3 runs, warm-up kept → median 31.7 ms
 5 runs, warm-up kept → median 21.6 ms
10 runs, warm-up kept → median 12.9 ms
20 runs, warm-up kept → median 12.6 ms

🏁 तुम्ही आत्ताच काय सिद्ध केले

एका थंड run ने 42.1 ms सांगितले; प्रामाणिक उत्तर 12.4 ms आहे — 3 पटीपेक्षा जास्त कमी. सगळ्या 20 चा mean (15.5) warm-up आणि spike मुळे अजूनही वर ओढला गेला होता. इतक्या noise मध्ये, सुमारे 4% चा बदल (12.0 → 11.5) दिसतच नाही; सुमारे 8% (→ 11.0) हा यापैकी तो पाहू शकणारा सर्वात लहान टप्पा आहे. आणि फक्त 3 किंवा 5 runs आणि warm-up न काढता, median स्वतःच चुकीचा येतो. जास्त runs आणि warm-up benchmark अधिक धारदार करतात.

⚠️ नेहमीच्या चुका

एका run ची, किंवा पहिल्या run ची वेळ घेऊन तेच उत्तर म्हणणे
वेगवेगळ्या machines, दिवस किंवा power settings वरच्या दोन आकड्यांची तुलना करणे
असा benchmark जो compiler किंवा runtime वगळू शकतो (result कधीच वापरला जात नाही)
खऱ्या वेळेच्या फक्त 1% असलेल्या function चा micro-benchmark करणे (आधी profile करा — धडा 03)
वेळ मोजताना timeit garbage collector बंद करते — भरपूर garbage बनवणारा code प्रत्यक्ष वापरात असेल त्यापेक्षा जलद दिसतो
ज्याचे runs 8% ने बदलतात अशा benchmark वरून "2% जलद" असा दावा करणे

🏭 प्रत्यक्ष वापरात

खऱ्या machine वर — pyperf प्रत्येक benchmark अनेक नव्या processes मध्ये चालवतो, loop count calibrate करतो, warm-ups काढून टाकतो आणि mean ± standard deviation सांगतो:

python3 -m pip install pyperf
python3 -m pyperf timeit -o old.json -s "data = list(range(1000, 0, -1))" "sorted(data)"
# change the code, then:
python3 -m pyperf timeit -o new.json -s "data = list(range(1000, 0, -1))" "sorted(data)"
python3 -m pyperf compare_to old.json new.json     # says if the difference is significant
python3 -m pyperf system tune                       # (Linux, root) quieter CPU settings for benchmarking

hyperfine संपूर्ण commands साठी हेच करतो, warm-up runs सह:

hyperfine --warmup 3 --runs 20 'python3 old_report.py' 'python3 new_report.py'

तुमच्या स्वतःच्या code मध्ये, monotonic घड्याळ वापरा:

import time
t0 = time.perf_counter()
build_results()
print(f"{(time.perf_counter() - t0) * 1000:.1f} ms")

🏭 Production मध्ये हे का महत्त्वाचे आहे: benchmark results ते बनवणाऱ्या commit सोबत ठेवा, ते त्याच शांत machine वर चालवा, आणि प्रत्येक आकड्याशेजारी पसारा लिहा. पसाऱ्यापेक्षा लहान speed-up हा result नाही.

⏭️ पुढे

आता तुम्ही एखादा program मोजू शकता. पण त्याचा कोणता भाग हळू आहे? पुढे: प्रशिक्षिकेचा clipboard — profiling.

git checkout lesson-03-profiling

🏁 Lesson 02 — Benchmarking honestly: warm-up laps and many heats

📍 You are here: Lesson 02 of 12 · Previous: lesson-01-latency-percentiles · Next: lesson-03-profiling

📦 What's in this branch

Lesson 01, plus comparing two versions of a program honestly. One run lies: the first runs are cold, the machine is noisy, and small changes hide inside that noise. The cure: warm up, repeat, report the median and the spread, and only call a win when it is bigger than the noise. bench() in perf/demo.py and simulated_runs, spread and verdict in perf/sim.py.

🧒 Explain like I'm 5

Dipika wants to know if new running shoes make Aishwarya faster. 👟

Katrina times Aishwarya once in the old shoes. It is early morning, Aishwarya is cold and stiff: 42 seconds. Later, in the new shoes: 12 seconds. "The new shoes are 3 times faster!" No — Aishwarya was just warming up.

So they do it properly:

a few warm-up laps first, not counted 🔥
then many heats, not one 🔁
then look at the middle time (the median) and how spread out the times are
a heat where a dog ran onto the track 🐕 is one odd time — the median does not care much

Then they compare. If the old and new times overlap a lot, the shoes made no clear difference. Only if the new times are clearly lower is it a real win.

🗺️ Diagram

flowchart LR
    cold["🥶 3 cold runs<br/>42.1 · 31.7 · 21.6 ms"] -->|"drop them"| warm["🔁 17 warm runs"]
    warm --> med["median 12.4 ms<br/>middle half 11.7–12.7"]
    b1["B1 tweak: middle half 11.3–12.3"] --> cmp{"do the middle<br/>halves overlap?"}
    b2["B2 fix: middle half 7.8–8.4"] --> cmp
    med --> cmp
    cmp -->|"B1 overlaps"| no["no clear difference"]
    cmp -->|"B2 is clear"| yes["B is faster (1.53x)"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/performance/lesson-diagrams.html#l02

❓ What

Benchmark — running a piece of code (or a whole program) many times under controlled conditions to measure it. A micro-benchmark times something tiny, like one function; a macro-benchmark times a real task end to end.
Warm-up — the first runs pay one-time costs: cold CPU caches, files not yet in the operating system's page cache, lazy imports, connection set-up, and in some runtimes a JIT compiler that has not optimised the code yet. Throw those runs away (or measure them on purpose, if a cold start is what users see).
Noise — other programs, CPU frequency changes (turbo boost, a hot laptop slowing down), interrupts and memory layout all move the numbers. The simulated stopwatch adds ±8% jitter and an occasional spike.
Median and spread — report the median and the middle half (p25 to p75, the interquartile range), or min/median/max. Some tools report the minimum, arguing that noise only ever adds time; that suits tiny, CPU-only code, but hides real variation.
Comparing two versions — the lab's verdict uses a rule of thumb: call B faster only if B's p75 is below A's p25 (the middle halves do not overlap). Real tools use statistics (for example, pyperf's compare_to reports whether a difference is significant).
time.perf_counter() — the clock for timing on one machine: monotonic and high-resolution. Do not time code with time.time() (it can jump when the system clock is corrected).

🤔 Why

Because every later lesson changes code "to make it faster". Without honest measuring you will keep changes that do nothing, throw away changes that help, and argue about numbers that were noise. A benchmark that cannot see a 3% change should not be used to claim one.

🔧 How (in this repo)

simulated_runs(base_ms, n, seed) in perf/sim.py is a simulated stopwatch: each run costs base_ms ± 8% (seeded), the first 3 runs add a warm-up cost (30, 20, 10 ms), and 5% of runs add a spike. spread(xs) returns min, p25, median, p75 and max. verdict(a, b) applies the rule of thumb above. bench() in perf/demo.py times version A (12 ms), a small tweak B1 (11.7 ms) and a real fix B2 (8 ms). The last line of Try it uses Python's real timeit — a real clock.

🧪 Try it

python3 perf/demo.py bench
python3 - <<'EOF'
import sys; sys.path.insert(0, "perf"); from sim import simulated_runs, spread, verdict
a = simulated_runs(12.0)[3:]
for base in (12.0, 11.5, 11.0, 10.5, 10.0, 9.0):
    b = simulated_runs(base, seed=6)[3:]
    print(f"B at {base:>4} ms → median {spread(b)['median']:>4} · {verdict(a, b)}")
for n in (3, 5, 10, 20):
    print(f"{n:>2} runs, warm-up kept → median {spread(simulated_runs(12.0, n=n))['median']:>4} ms")
EOF
python3 -m timeit -n 1000 -r 5 "sorted(range(1000, 0, -1))"

The last command times a real sort on your machine — something like 1000 loops, best of 5: 5.62 usec per loop. Your numbers will differ, run to run and machine to machine; that is the point of this lesson.

✅ Verify — what you should see

bench prints:

── Katrina times version A 20 times (simulated stopwatch): first three runs [42.1, 31.7, 21.6] ms — cold, warming up
   one run says 42.1 ms · the mean of all 20 says 15.5 ms · after 3 warm-up runs, the median of 17 says 12.4 ms
   spread of the 17: min 11.2 · p25 11.7 · median 12.4 · p75 12.7 · max 19.9 (one spike: another program woke up)
── A vs B1 (a small tweak): medians 12.4 vs 12.0 ms · middle halves 11.7–12.7 vs 11.3–12.3 → no clear difference: the change is smaller than the noise
── A vs B2 (a real fix): medians 12.4 vs 8.1 ms · middle halves 11.7–12.7 vs 7.8–8.4 → B is faster (1.53x)

Your snippet prints:

B at 12.0 ms → median 12.3 · no clear difference: the change is smaller than the noise
B at 11.5 ms → median 11.8 · no clear difference: the change is smaller than the noise
B at 11.0 ms → median 11.3 · B is faster (1.10x)
B at 10.5 ms → median 10.8 · B is faster (1.15x)
B at 10.0 ms → median 10.2 · B is faster (1.22x)
B at  9.0 ms → median  9.2 · B is faster (1.35x)
 3 runs, warm-up kept → median 31.7 ms
 5 runs, warm-up kept → median 21.6 ms
10 runs, warm-up kept → median 12.9 ms
20 runs, warm-up kept → median 12.6 ms

🏁 What you just proved

One cold run said 42.1 ms; the honest answer is 12.4 ms — more than 3 times less. The mean of all 20 (15.5) was still pulled up by the warm-up and the spike. With this much noise, a change of about 4% (12.0 → 11.5) is invisible; about 8% (→ 11.0) is the smallest of these steps it can see. And with only 3 or 5 runs and no warm-up, the median itself is wrong. More runs and a warm-up make the benchmark sharper.

⚠️ Common mistakes

timing one run, or the first run, and calling it the answer
comparing two numbers from different machines, days or power settings
a benchmark the compiler or runtime can skip (the result is never used)
micro-benchmarking a function that is 1% of the real time (profile first — lesson 03)
timeit turns off the garbage collector while it times — code that makes lots of garbage looks faster than it will be in real use
claiming "2% faster" from a benchmark whose runs vary by 8%

🏭 In production

On a real machine — pyperf runs each benchmark in several fresh processes, calibrates the loop count, drops warm-ups and reports mean ± standard deviation:

python3 -m pip install pyperf
python3 -m pyperf timeit -o old.json -s "data = list(range(1000, 0, -1))" "sorted(data)"
# change the code, then:
python3 -m pyperf timeit -o new.json -s "data = list(range(1000, 0, -1))" "sorted(data)"
python3 -m pyperf compare_to old.json new.json     # says if the difference is significant
python3 -m pyperf system tune                       # (Linux, root) quieter CPU settings for benchmarking

hyperfine does the same for whole commands, with warm-up runs:

hyperfine --warmup 3 --runs 20 'python3 old_report.py' 'python3 new_report.py'

In your own code, use a monotonic clock:

import time
t0 = time.perf_counter()
build_results()
print(f"{(time.perf_counter() - t0) * 1000:.1f} ms")

🏭 Why this matters in production: keep benchmark results with the commit that produced them, run them on the same quiet machine, and write the spread next to every number. A speed-up smaller than the spread is not a result.

⏭️ Next

You can now measure a program. But which part of it is slow? Next: the coach's clipboard — profiling.

git checkout lesson-03-profiling

← Previouslatency percentiles Next →profiling

This page is the lesson's README from the lesson-02-benchmarking branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.