🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

📚 धडा 05 — Replication: नोंदवहीच्या प्रती

📍 तुम्ही इथे आहात: 12 पैकी धडा 05 · मागे: lesson-04-failure-detection · पुढे: lesson-06-quorums

📦 या ब्रँचमध्ये काय आहे

धडे 01–04, अधिक भाग 2 सुरू होतो: प्रती. एक शाखा leader असते आणि प्रत्येक write घेते; बाकीच्या followers असतात आणि तिचा log copy करतात. Synchronous, asynchronous आणि semi-synchronous replication, replication lag, failover मध्ये काय हरवते, आणि split brain. dist/demo.py मधले replication() आणि dist/sim.py मधले Leader.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

पुणे मुख्य नोंदवही 📒 ठेवते. फक्त पुणेच त्यात गुण लिहिते. नाशिक आणि नागपूर प्रती 📚 ठेवतात, म्हणजे पुण्यात पूर आला तरी सगळे गुण नष्ट होत नाहीत.

पुण्यातली कतरिना चार गुण लिहिते: g1, g2, g3, g4. प्रतींना ते कसे कळतात?

जलद मार्ग (asynchronous): कतरिना गुण लिहिते, शिक्षिकेला लगेच "झाले!" सांगते, आणि ते पान प्रतींना नंतर पाठवते. जलद. पण एके दिवशी फक्त g1 आणि g2 पाठवले गेल्यावरच पुण्यात पूर येतो. नागपूर नवी मुख्य नोंदवही बनते. g3 आणि g4 गेले — आणि शिक्षिकेला "झाले!" असे सांगितले गेले होते. 😱
काळजीपूर्वक मार्ग (synchronous): एखाद्या प्रतीकडेही ते पान पोहोचल्यावरच कतरिना "झाले!" सांगते. पुण्यात पूर आला तरी काहीच हरवत नाही. पण प्रत्येक गुण निरोप्याची वाट पाहतो. प्रतीकडे जाणारा रस्ता हळू असेल, तर प्रत्येक गुण हळू होतो.
मधला मार्ग (semi-synchronous): सगळ्या नव्हे, एका प्रतीची वाट पाहा.

आणि आणखी एक धोका: पुरानंतर पुणे कोरडे होते आणि अजूनही स्वतःला मुख्य नोंदवही समजते. आता दोन शाखा writes घेतात. यालाच split brain म्हणतात.

🗺️ आकृती

flowchart LR
    t["👩‍🏫 teacher"] -->|"write g1 g2 g3 g4"| lead["📒 Pune — leader<br/>says 'acknowledged' at once"]
    lead -->|"shipped: g1 g2"| f1["📚 Nashik — follower"]
    lead -->|"shipped: g1 g2"| f2["📚 Nagpur — follower"]
    lead --> x["💥 Pune dies"]
    x --> nl["new leader has g1 g2<br/>lost: g3 g4 — already acknowledged"]
    s["synchronous: 'acknowledged' only after a follower has it<br/>lost: nothing"]

🗺️ काढलेली आवृत्ती + एक lab: https://school-edh.pages.dev/distributed-systems/lesson-diagrams.html#l05

❓ काय

Replication — एकाच data ची प्रत अनेक nodes वर ठेवणे: एखादा node हरवला तरी टिकण्यासाठी, जास्त ठिकाणांहून reads देण्यासाठी, आणि users च्या जवळ राहण्यासाठी.
Leader-follower (primary-replica) — एक leader writes स्वीकारतो आणि त्यांची log मध्ये नोंद करतो; followers तोच log त्याच क्रमाने लागू करतात. Reads leader कडे किंवा followers कडे जाऊ शकतात.
Synchronous replication — किमान एका follower कडे write पोहोचल्यावरच leader तो acknowledge करतो. Acknowledged write leader हरवला तरी टिकतो. किंमत: प्रत्येक write एका round trip ची वाट पाहतो, आणि हळू किंवा बंद पडलेला follower writes अडवतो.
Asynchronous replication — leader लगेच acknowledge करतो आणि log नंतर पाठवतो. जलद. किंमत: failover वर, leader ने acknowledge केलेले पण अजून न पाठवलेले writes हरवतात.
Semi-synchronous — सगळ्या नव्हे, एका (किंवा काही) followers ची वाट पाहा. सामान्य मधला मार्ग: एक synchronous follower, बाकीचे asynchronous. (MySQL त्याच्या आवृत्तीला "semisynchronous replication" म्हणते; PostgreSQL हे synchronous_standby_names ने करते.)
Replication lag — follower leader च्या किती मागे आहे. सहसा milliseconds; जास्त load खाली सेकंद किंवा मिनिटे. मागे पडलेल्या follower कडून आलेले reads stale (जुने) असतात (धडा 07).
Failover — leader बंद पडल्याचा संशय आला की (धडा 04) एका follower ला leader बनवले जाते. Async replication मध्ये, सर्वात अद्ययावत follower कडूनही काही writes सुटलेले असू शकतात.
Split brain — दोन nodes दोघेही leader सारखे वागतात आणि दोघेही writes स्वीकारतात; त्यांचा data दुभंगतो. जुना leader फक्त हळू होता, बंद पडलेला नव्हता, आणि परत आला की असे होते. उपाय: बहुमताचे मतदान (धडा 09), fencing (धडा 10), किंवा जुन्या leader ला तोडून टाकणे (कधी कधी त्याला STONITH म्हणतात — "shoot the other node in the head", म्हणजे त्याची वीज बंद करणे).

🤔 का

कारण "database ला replica आहे" याचा अर्थ अनेकदा "आपण data गमावू शकत नाही" असा घेतला जातो — आणि asynchronous replication व automatic failover असताना ते खोटे आहे. 2012 ची एक प्रसिद्ध GitHub घटना: जुना झालेला एक MySQL follower promote केला गेला; त्याने जुन्या leader ने आधीच दिलेल्या primary keys पुन्हा वापरल्या, आणि त्याच keys एका Redis store मध्येही वापरल्या होत्या, त्यामुळे काही खासगी data चुकीच्या users ना दिसला. Sync आणि async मधली निवड म्हणजे write latency आणि acknowledged writes गमावणे यांच्यातली निवड, आणि ती तुम्ही जाणीवपूर्वक करायला हवी.

🔧 कसे (या repo मध्ये)

dist/sim.py मधले Leader(followers, sync=False) एक log आणि प्रत्येक follower साठी एक list ठेवते. write(item) log मध्ये जोडते आणि लगेच 'acknowledged' परत देते; sync=True असेल तर ते आधी प्रत्येक follower मध्येही जोडते. ship(upto) log मधल्या पहिल्या upto entries followers कडे copy करते (async निरोप्या). fail_over() सर्वात लांब प्रत असलेल्या follower ला promote करते आणि (its copy, the writes it lacks) परत देते.

🧪 करून पाहा

python3 dist/demo.py replication
python3 - <<'EOF'
import sys; sys.path.insert(0, "dist"); from sim import Leader
for shipped in (0, 2, 3, 4):
    L = Leader(["nashik", "nagpur"])
    for g in ("g1", "g2", "g3", "g4"): L.write(g)
    L.ship(shipped)
    survivor, lost = L.fail_over()
    print(f"async, {shipped} of 4 shipped before the crash → new leader has {survivor}, lost {lost}")
EOF

✅ तपासा — तुम्हाला काय दिसायला हवे

replication हे print करते:

── asynchronous replication: 4 grades acknowledged, then Pune (leader) dies → new leader has ['g1', 'g2'], lost ['g3', 'g4']
── synchronous replication: 4 grades acknowledged, then Pune (leader) dies → new leader has ['g1', 'g2', 'g3', 'g4'], lost []

तुमचा snippet हे print करतो:

async, 0 of 4 shipped before the crash → new leader has [], lost ['g1', 'g2', 'g3', 'g4']
async, 2 of 4 shipped before the crash → new leader has ['g1', 'g2'], lost ['g3', 'g4']
async, 3 of 4 shipped before the crash → new leader has ['g1', 'g2', 'g3'], lost ['g4']
async, 4 of 4 shipped before the crash → new leader has ['g1', 'g2', 'g3', 'g4'], lost []

🏁 तुम्ही आत्ताच काय सिद्ध केले

Async replication मध्ये, failover मध्ये जे हरवते ते म्हणजे crash च्या क्षणीचा नेमका replication lag: 0 पाठवले → चारही acknowledged गुण हरवले; 4 पाठवले → एकही नाही. शिक्षिकेने चारही वेळा "acknowledged" ऐकले होते. Synchronous replication मध्ये काहीच हरवले नाही, कारण "acknowledged" ने एका प्रतीची वाट पाहिली होती.

⚠️ नेहमीच्या चुका

गमावू न शकणाऱ्या data साठी (पैसे, गुण) async replication + automatic failover
कोणताही राखीव नसताना एकाच follower कडे synchronous replication — तो बंद पडला की writes थांबतात
जुन्या leader ला fencing नाही, त्यामुळे तो परत येतो आणि writes घेतो (split brain)
leader वर लिहिल्यानंतर लगेच follower कडून वाचणे (धडा 07)
failover कधीच test न करणे, त्यामुळे पहिला failover थेट outage दरम्यान होतो

🏭 प्रत्यक्ष वापरात

PostgreSQL — streaming replication default नुसार asynchronous असते. On a real account, primary वरच्या postgresql.conf मध्ये दोन standbys पैकी एक synchronous करा:

synchronous_standby_names = 'FIRST 1 (nashik, nagpur)'   # wait for the first one in the list that is connected
synchronous_commit = on                                  # a commit waits until that standby has flushed the WAL to disk

किंवा 'ANY 1 (nashik, nagpur)' — जो आधी उत्तर देईल त्याची वाट पाहा (quorum commit). synchronous_commit हे remote_apply (standby ने ते लागू करेपर्यंत थांबा, म्हणजे तिथले reads ते पाहतात), remote_write, local आणि off सुद्धा स्वीकारते, आणि ते प्रत्येक transaction साठी वेगळे ठरवता येते — उदाहरणार्थ गमावली तरी चालेल अशा log line साठी SET LOCAL synchronous_commit = off;. Lag वर लक्ष ठेवा:

SELECT application_name, state, sync_state, write_lag, flush_lag, replay_lag
FROM pg_stat_replication;

MySQL — semisynchronous replication हे एक plugin आहे: किमान एका replica ला transaction मिळेपर्यंत source थांबतो (rpl_semi_sync_source_wait_for_replica_count).

Managed failover — Amazon RDS Multi-AZ दुसऱ्या Availability Zone मध्ये एक synchronous standby ठेवते आणि त्यावर fail over करते; RDS read replicas asynchronous असतात आणि आपोआप fail over करत नाहीत. Patroni सारखी tools एकच leader कोण हे ठरवण्यासाठी etcd किंवा ZooKeeper (धडे 09–10) वापरून PostgreSQL failover चालवतात.

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: प्रत्येक database साठी लिहून ठेवा: sync की async, किती synchronous प्रती, failover कोण ठरवतो, आणि जुन्या leader ला writes घेण्यापासून कसे रोखले जाते. मग failover drill चालवा आणि काय हरवले ते मोजा.

⏭️ पुढे

एकच leader म्हणजे bottleneck आणि निर्णयाचे एकच ठिकाण. दुसरा मार्ग: अनेक प्रतींवर लिहा, अनेक प्रतींकडून वाचा, आणि आकड्यांनाच हमी देऊ द्या की तुम्हाला सर्वात नवे दिसेल. Quorums.

git checkout lesson-06-quorums

📚 Lesson 05 — Replication: copies of the register

📍 You are here: Lesson 05 of 12 · Previous: lesson-04-failure-detection · Next: lesson-06-quorums

📦 What's in this branch

Lessons 01–04, plus Part 2 begins: copies. One branch is the leader and takes every write; the others are followers and copy its log. Synchronous, asynchronous and semi-synchronous replication, replication lag, what a failover loses, and split brain. replication() in dist/demo.py and Leader in dist/sim.py.

🧒 Explain like I'm 5

Pune keeps the main register 📒. Only Pune writes grades in it. Nashik and Nagpur keep copies 📚, so a flood in Pune does not destroy every grade.

Katrina in Pune writes four grades: g1, g2, g3, g4. How do the copies learn about them?

The fast way (asynchronous): Katrina writes the grade, says "done!" to the teacher at once, and sends the page to the copies later. Quick. But one day Pune floods after only g1 and g2 have been sent. Nagpur becomes the new main register. g3 and g4 are gone — and the teacher was told "done!". 😱
The careful way (synchronous): Katrina says "done!" only after a copy has the page too. Nothing is lost when Pune floods. But every grade waits for the messenger. If the road to the copy is slow, every grade is slow.
The middle way (semi-synchronous): wait for one copy, not all of them.

And one more danger: after the flood, Pune dries out and still thinks it is the main register. Now two branches take writes. That is split brain.

🗺️ Diagram

flowchart LR
    t["👩‍🏫 teacher"] -->|"write g1 g2 g3 g4"| lead["📒 Pune — leader<br/>says 'acknowledged' at once"]
    lead -->|"shipped: g1 g2"| f1["📚 Nashik — follower"]
    lead -->|"shipped: g1 g2"| f2["📚 Nagpur — follower"]
    lead --> x["💥 Pune dies"]
    x --> nl["new leader has g1 g2<br/>lost: g3 g4 — already acknowledged"]
    s["synchronous: 'acknowledged' only after a follower has it<br/>lost: nothing"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/distributed-systems/lesson-diagrams.html#l05

❓ What

Replication — keeping a copy of the same data on several nodes: to survive a lost node, to serve reads from more places, and to be close to users.
Leader-follower (primary-replica) — one leader accepts writes and records them in a log; followers apply the same log in the same order. Reads can go to the leader or to followers.
Synchronous replication — the leader acknowledges a write only after at least one follower has it. An acknowledged write survives the loss of the leader. Cost: every write waits for a round trip, and a slow or dead follower blocks writes.
Asynchronous replication — the leader acknowledges at once and ships the log later. Fast. Cost: on failover, writes the leader acknowledged but had not shipped are lost.
Semi-synchronous — wait for one (or some) followers, not all. The common middle: one synchronous follower, the rest asynchronous. (MySQL calls its version "semisynchronous replication"; PostgreSQL does it with synchronous_standby_names.)
Replication lag — how far a follower is behind the leader. Usually milliseconds; seconds or minutes under heavy load. Reads from a lagging follower are stale (lesson 07).
Failover — a follower is promoted to leader when the leader is suspected dead (lesson 04). With async replication, the most up-to-date follower still may miss some writes.
Split brain — two nodes both act as leader and both accept writes; their data splits. It happens when the old leader was only slow, not dead, and comes back. Fixes: a majority vote (lesson 09), fencing (lesson 10), or cutting the old leader off (sometimes called STONITH — "shoot the other node in the head", that is, power it off).

🤔 Why

Because "the database has a replica" is often read as "we cannot lose data" — and with asynchronous replication and automatic failover, that is false. A known 2012 GitHub incident: an out-of-date MySQL follower was promoted; it reused primary keys the old leader had already given out, and those keys were also used in a Redis store, so some private data was shown to the wrong users. The choice between sync and async is a choice between write latency and losing acknowledged writes, and you should make it on purpose.

🔧 How (in this repo)

Leader(followers, sync=False) in dist/sim.py keeps a log and one list per follower. write(item) appends to the log and returns 'acknowledged' at once; with sync=True it also appends to every follower first. ship(upto) copies the first upto log entries to the followers (the async messenger). fail_over() promotes the follower with the longest copy and returns (its copy, the writes it lacks).

🧪 Try it

python3 dist/demo.py replication
python3 - <<'EOF'
import sys; sys.path.insert(0, "dist"); from sim import Leader
for shipped in (0, 2, 3, 4):
    L = Leader(["nashik", "nagpur"])
    for g in ("g1", "g2", "g3", "g4"): L.write(g)
    L.ship(shipped)
    survivor, lost = L.fail_over()
    print(f"async, {shipped} of 4 shipped before the crash → new leader has {survivor}, lost {lost}")
EOF

✅ Verify — what you should see

replication prints:

── asynchronous replication: 4 grades acknowledged, then Pune (leader) dies → new leader has ['g1', 'g2'], lost ['g3', 'g4']
── synchronous replication: 4 grades acknowledged, then Pune (leader) dies → new leader has ['g1', 'g2', 'g3', 'g4'], lost []

Your snippet prints:

async, 0 of 4 shipped before the crash → new leader has [], lost ['g1', 'g2', 'g3', 'g4']
async, 2 of 4 shipped before the crash → new leader has ['g1', 'g2'], lost ['g3', 'g4']
async, 3 of 4 shipped before the crash → new leader has ['g1', 'g2', 'g3'], lost ['g4']
async, 4 of 4 shipped before the crash → new leader has ['g1', 'g2', 'g3', 'g4'], lost []

🏁 What you just proved

With async replication, what a failover loses is exactly the replication lag at the moment of the crash: 0 shipped → all 4 acknowledged grades lost; 4 shipped → none. The teacher heard "acknowledged" all four times. Synchronous replication lost nothing, because "acknowledged" waited for a copy.

⚠️ Common mistakes

async replication + automatic failover for data you cannot lose (money, grades)
synchronous replication to one follower with no spare — when it dies, writes stop
no fencing of the old leader, so it comes back and takes writes (split brain)
reading from a follower right after writing to the leader (lesson 07)
never testing a failover, so the first one happens during an outage

🏭 In production

PostgreSQL — streaming replication is asynchronous by default. On a real account, make one of two standbys synchronous in postgresql.conf on the primary:

synchronous_standby_names = 'FIRST 1 (nashik, nagpur)'   # wait for the first one in the list that is connected
synchronous_commit = on                                  # a commit waits until that standby has flushed the WAL to disk

Or 'ANY 1 (nashik, nagpur)' — wait for whichever answers first (a quorum commit). synchronous_commit also accepts remote_apply (wait until the standby has applied it, so reads there see it), remote_write, local and off, and it can be set per transaction — for example SET LOCAL synchronous_commit = off; for a log line you can afford to lose. Watch the lag:

SELECT application_name, state, sync_state, write_lag, flush_lag, replay_lag
FROM pg_stat_replication;

MySQL — semisynchronous replication is a plugin: the source waits until at least one replica has received the transaction (rpl_semi_sync_source_wait_for_replica_count).

Managed failover — Amazon RDS Multi-AZ keeps a synchronous standby in another Availability Zone and fails over to it; RDS read replicas are asynchronous and do not fail over on their own. Tools such as Patroni run PostgreSQL failover using etcd or ZooKeeper (lessons 09–10) to decide who the one leader is.

🏭 Why this matters in production: for each database, write down: sync or async, how many synchronous copies, who decides a failover, and how the old leader is kept from taking writes. Then run a failover drill and count what was lost.

⏭️ Next

A single leader is a bottleneck and a single point of decision. Another way: write to several copies, read from several, and let the numbers guarantee you see the newest. Quorums.

git checkout lesson-06-quorums

← Previousfailure detection Next →quorums

This page is the lesson's README from the lesson-05-replication branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.