🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🎯 धडा 11 — Exactly-once चे मिथक: at-least-once + idempotency

📍 तुम्ही इथे आहात: 12 पैकी धडा 11 · मागे: lesson-10-locks-leases · पुढे: lesson-12-backpressure

📦 या ब्रँचमध्ये काय आहे

धडे 01–10, आणि delivery guarantees: at-most-once, at-least-once, आणि systems ज्याची जाहिरात करतात ते "exactly-once". Network पलीकडे message नेमका एकदाच पोहोचवता येत नाही — पण त्याचा परिणाम एकदाच घडू शकतो: **at-least-once delivery

idempotent processing = effectively once**. dist/demo.py मधील exactlyonce() आणि dist/sim.py मधील deliver().

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

मुख्य office फी परतावा निरोप्यामार्फत पाठवते: कतरिनाला पैसे द्या, दीपिकाला पैसे द्या, ऐश्वर्याला पैसे द्या. 💸

निरोप्या "दीपिकाला पैसे द्या" पोहोचवतो. दीपिकाची शाखा तिला पैसे देते. शाखा परत "झाले" असे कळवते — आणि ती चिठ्ठी हरवते (धडा 01). मुख्य office ला काहीच कळत नाही. म्हणून ते "दीपिकाला पैसे द्या" पुन्हा पाठवते. दीपिकाला दोनदा पैसे मिळतात. 😬

मुख्य office ने ते फक्त एकदाच पाठवले, आणि पुन्हा कधीच नाही, तर? मग पहिली चिठ्ठीच हरवली तर दीपिकाला शून्य वेळा पैसे मिळतात. तेही वाईटच.

म्हणून शाखा काहीतरी हुशारीचे करतात. प्रत्येक आदेशाला एक नाव असते: pay-dipika. शाखा आधीच केलेल्या नावांची यादी ठेवते. 📋 "pay-dipika" दुसऱ्यांदा आला की कारकून यादी तपासते, ते नाव तिथे पाहते, आणि काहीच करत नाही — पण तरीही "झाले" असे सांगते.

हवे तितक्या वेळा पाठवा. पैसे एकदाच जातात.

🗺️ आकृती

flowchart LR
    q["📬 queue<br/>at-least-once: may deliver again"] -->|"pay-katrina, pay-dipika,<br/>pay-aishwarya, pay-dipika again"| c1["plain consumer"]
    q -->|"same four deliveries"| c2["idempotent consumer<br/>checks a list of done keys"]
    c1 --> e1["4 effects<br/>Dipika paid twice"]
    c2 --> e2["3 effects<br/>every payment once"]

🗺️ रेखाटलेली आवृत्ती + एक lab: https://school-edh.pages.dev/distributed-systems/lesson-diagrams.html#l11

❓ काय

At-most-once — एकदा पाठवा, कधीच retry करू नका. Duplicates नाहीत, पण हरवलेला message कायमचा हरवतो. जे चुकले तरी चालेल अशा metrics साठी ठीक.
At-least-once — acknowledgement मिळेपर्यंत retry करा. काहीच हरवत नाही, पण message एकापेक्षा जास्त वेळा येऊ शकतो (काम झाले, "झाले" हरवले). बहुतेक queues तुम्हाला हेच देतात — SQS standard queues, processing नंतर offsets commit करणारे Kafka consumers, acknowledgements सह RabbitMQ.
Exactly-once delivery — message एकदा आणि फक्त एकदाच येतो. अविश्वसनीय network पलीकडे याची हमी देता येत नाही: पाठवणाऱ्याला "हरवला" आणि "झाले, पण उत्तर हरवले" यांतला फरक सांगता येत नाही (धडा 01), म्हणून त्याने एकतर retry करायला हवा (duplicates चा धोका) किंवा नाही (हरवण्याचा धोका).
Idempotent — दोनदा केले तरी परिणाम एकदा केल्यासारखाच. "Grade A set करा" हे idempotent आहे; "100 रुपये जोडा" हे नाही — जोपर्यंत त्यावर तपासली जाणारी key नसते.
Effectively once ("exactly-once" चा खरा अर्थ) — at-least-once delivery + idempotent processing. साधने:
- प्रत्येक message किंवा request वर idempotency key / dedupe key (pay-dipika, किंवा client एकदा तयार करून retries मध्ये पुन्हा वापरतो तो UUID)
- processed keys ची dedupe table, परिणामाच्या त्याच transaction मध्ये लिहिलेली
- आंधळ्या increments ऐवजी upserts आणि conditional writes ("नसेल तर insert करा", "अजून 4 असेल तर version 5 set करा")
- outbox pattern — बदल आणि पाठवायचा message एकाच database transaction मध्ये लिहा; एक वेगळी process तो message नंतर पाठवते (at-least-once, key सह)
Kafka ची exactly-once semantics (EOS) — खरी, आणि Kafka च्या आत: idempotent producer (broker producer ID आणि प्रत्येक partition साठी sequence number वापरून duplicate retries टाकून देतो) अधिक transactions (consumer चे read, output topics वरचे त्याचे writes आणि त्याचा offset commit एकत्र यशस्वी किंवा अयशस्वी होतात; read_committed असलेल्या readers ना फक्त committed results दिसतात). Kafka च्या बाहेरील परिणाम — database row, email, payment — यांना अजूनही idempotency लागते.

🤔 का

कारण retries सगळीकडे असतात: clients, load balancers, SDKs, queues, आणि timeout नंतर तुमचा स्वतःचा code. प्रत्येक जण स्वतःच्या जागी बरोबर असतो, आणि सगळे मिळून duplicates पोहोचवतात. Idempotent नसलेले payment, email किंवा stock count production मध्ये दोनदा घडणारच; प्रश्न फक्त किती वेळा इतकाच. Product च्या page वर छापलेल्या "Exactly once" चा अर्थ सहसा "आमच्या system च्या आत" असा असतो — त्या system ची कड म्हणजेच जिथे तुमची idempotency key जाते.

🔧 कसे (या repo मध्ये)

dist/sim.py मधील deliver(messages, redeliver, idempotent) आधी messages आणि मग redeliveries process करते (at-least-once). साधा consumer प्रत्येक message लागू करतो. Idempotent consumer message keys चा एक seen set ठेवतो आणि आधीच लागू केलेली कोणतीही key वगळतो. ते परिणामांची यादी परत करते.

🧪 करून पाहा

python3 dist/demo.py exactlyonce
python3 - <<'EOF'
import sys; sys.path.insert(0, "dist"); from sim import deliver
msgs = ["pay-katrina", "pay-dipika", "pay-aishwarya"]
again = ["pay-dipika", "pay-dipika", "pay-katrina"]            # a bad night: three redeliveries
for idem in (False, True):
    eff = deliver(msgs, again, idem)
    print(f"{'idempotent' if idem else 'plain     '} → {len(eff)} payments: {eff}")
print("at-most-once (never redeliver) and a lost message →", deliver(["pay-katrina", "pay-aishwarya"], [], True))
EOF

✅ तपासा — तुम्हाला काय दिसायला हवे

exactlyonce हे print करते:

── the queue redelivers pay-dipika (at-least-once) · plain consumer      → effects ['pay-katrina', 'pay-dipika', 'pay-aishwarya', 'pay-dipika']
── the queue redelivers pay-dipika (at-least-once) · idempotent consumer  → effects ['pay-katrina', 'pay-dipika', 'pay-aishwarya']

तुमचा snippet हे print करतो:

plain      → 6 payments: ['pay-katrina', 'pay-dipika', 'pay-aishwarya', 'pay-dipika', 'pay-dipika', 'pay-katrina']
idempotent → 3 payments: ['pay-katrina', 'pay-dipika', 'pay-aishwarya']
at-most-once (never redeliver) and a lost message → ['pay-katrina', 'pay-aishwarya']

🏁 तुम्ही आत्ताच काय सिद्ध केले

तीन redeliveries मुळे साध्या consumer साठी 3 payments चे 6 झाले, आणि idempotent consumer साठी काहीच बदलले नाही. शेवटची ओळ दुसरी बाजू दाखवते: at-most-once मध्ये हरवलेला pay-dipika सरळ नाहीसा होतो — duplicates नाहीत, आणि payment ही नाही. या दोघांपैकी at-least-once + key हाच असा आहे जो तुम्ही दुरुस्त करू शकता.

⚠️ नेहमीच्या चुका

queue चे "exactly once" तुमच्या consumer ने केलेला database write किंवा email सुद्धा कव्हर करते असे मानणे
key परिणामानंतर वेगळ्या step मध्ये नोंदवणे — दोघांच्या मध्ये crash झाला तर परिणाम पुन्हा घडतो
expiry नसलेली dedupe table, किंवा सर्वात लांब शक्य redelivery पेक्षा कमी expiry
प्रत्येक attempt साठी server वर बनवलेल्या keys, त्यामुळे retry ला नवी key मिळते
जिथे "version 4 असेल तर 600 set करा" idempotent ठरले असते तिथे "100 जोडा" updates

🏭 प्रत्यक्ष वापरात

SQL मध्ये idempotent consumer — key नोंदवणे आणि परिणाम लागू करणे एकाच transaction मध्ये करा. On a real account (PostgreSQL):

BEGIN;
INSERT INTO processed_messages (message_id) VALUES ('pay-dipika')
    ON CONFLICT (message_id) DO NOTHING;            -- 0 rows inserted → already done: ROLLBACK and ack
UPDATE fees SET refunded = true WHERE student = 'dipika';
COMMIT;                                             -- then delete the message from the queue

Amazon SQS — standard queues at-least-once असतात (आणि क्रम बदलून देऊ शकतात). जो message त्याच्या visibility timeout मध्ये delete होत नाही तो पुन्हा दिला जातो. FIFO queues 5-minute च्या window मध्ये त्याच MessageDeduplicationId सह पाठवलेला message टाकून देतात — यामुळे duplicate sends थांबतात, पण तुमचा consumer message delete करण्याआधी crash झाला तर तो अजूनही message दोनदा process करू शकतो:

aws sqs send-message --queue-url https://sqs.ap-south-1.amazonaws.com/111122223333/refunds.fifo \
    --message-body '{"student":"dipika","amount":600}' \
    --message-group-id dipika --message-deduplication-id pay-dipika

Kafka — शब्दांत: idempotent producer (enable.idempotence=true, Kafka 3.0 पासून default, acks=all सह) producer च्या स्वतःच्या retries मुळे होणारे duplicates थांबवतो. Transactions (producer वर transactional.id; consumers वर isolation.level=read_committed) Kafka च्या आत read-process-write सर्व-किंवा-काहीच नाही असे करतात, consumer offsets सकट. Kafka Streams हे processing.guarantee=exactly_once_v2 ने चालू करते. Database मध्ये लिहिणाऱ्या consumer ला तिथे अजूनही key किंवा upsert लागतो.

HTTP APIs — POST requests वर Idempotency-Key header स्वीकारा (Stripe असे करते) आणि पुन्हा आलेल्या key साठी जतन केलेला response परत करा.

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: पैसे हलवणारा, messages पाठवणारा किंवा counts बदलणारा प्रत्येक consumer आणि प्रत्येक POST endpoint यादीत लिहा. प्रत्येकासाठी त्याची idempotency key आणि ती key कुठे साठवली जाते ते सांगा. "Queue exactly-once करते" हे उत्तर नव्हे.

⏭️ पुढे

शेवटची अडचण: खूप जास्त काम. Retries, bursts आणि हळू प्रती साचत जातात, आणि सगळे स्वीकारणारी system सगळ्यांसाठीच हळू होते. पुढे: backpressure — आणि संपूर्ण नकाशा.

git checkout lesson-12-backpressure

🎯 Lesson 11 — The exactly-once myth: at-least-once + idempotency

📍 You are here: Lesson 11 of 12 · Previous: lesson-10-locks-leases · Next: lesson-12-backpressure

📦 What's in this branch

Lessons 01–10, plus delivery guarantees: at-most-once, at-least-once, and the "exactly-once" that systems advertise. Across a network, a message cannot be delivered exactly once — but its effect can happen once: **at-least-once delivery

idempotent processing = effectively once**. exactlyonce() in dist/demo.py and deliver() in dist/sim.py.

🧒 Explain like I'm 5

The head office sends fee refunds by messenger: pay Katrina, pay Dipika, pay Aishwarya. 💸

The messenger delivers "pay Dipika". Dipika's branch pays her. The branch sends back "done" — and that note is lost (lesson 01). The head office hears nothing. So it sends "pay Dipika" again. Dipika is paid twice. 😬

Could the head office send it only once, and never again? Then if the first note was lost, Dipika is paid zero times. Also bad.

So the branches do something smarter. Each order has a name: pay-dipika. The branch keeps a list of names it has already done. 📋 When "pay-dipika" comes a second time, the clerk checks the list, sees it, and does nothing — but still says "done".

Send as many times as you like. The money moves once.

🗺️ Diagram

flowchart LR
    q["📬 queue<br/>at-least-once: may deliver again"] -->|"pay-katrina, pay-dipika,<br/>pay-aishwarya, pay-dipika again"| c1["plain consumer"]
    q -->|"same four deliveries"| c2["idempotent consumer<br/>checks a list of done keys"]
    c1 --> e1["4 effects<br/>Dipika paid twice"]
    c2 --> e2["3 effects<br/>every payment once"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/distributed-systems/lesson-diagrams.html#l11

❓ What

At-most-once — send once, never retry. No duplicates, but a lost message is lost for good. Fine for metrics you can afford to miss.
At-least-once — retry until you get an acknowledgement. Nothing is lost, but a message can arrive more than once (the work was done, the "done" was lost). This is what most queues give you — SQS standard queues, Kafka consumers that commit offsets after processing, RabbitMQ with acknowledgements.
Exactly-once delivery — a message arrives once and only once. Across an unreliable network this cannot be guaranteed: the sender cannot tell "lost" from "done, answer lost" (lesson 01), so it must either retry (risk duplicates) or not (risk loss).
Idempotent — doing it twice has the same effect as doing it once. "Set grade to A" is idempotent; "add 100 rupees" is not — unless it carries a key that is checked.
Effectively once (what "exactly-once" really means) — at-least-once delivery + idempotent processing. Tools:
- an idempotency key / dedupe key on every message or request (pay-dipika, or a UUID the client makes once and reuses on retries)
- a dedupe table of processed keys, written in the same transaction as the effect
- upserts and conditional writes ("insert if not present", "set version 5 if still 4") instead of blind increments
- the outbox pattern — write the change and the message to send in one database transaction; a separate process sends the message later (at-least-once, with a key)
Kafka's exactly-once semantics (EOS) — real, and within Kafka: an idempotent producer (the broker drops duplicate retries using a producer ID and a sequence number per partition) plus transactions (a consumer's read, its writes to output topics and its offset commit succeed or fail together; readers with read_committed see only committed results). Effects outside Kafka — a database row, an email, a payment — still need idempotency.

🤔 Why

Because retries are everywhere: clients, load balancers, SDKs, queues, and your own code after a timeout. Each one is correct on its own, and together they deliver duplicates. A payment, an email or a stock count that is not idempotent will happen twice in production; the only question is how often. "Exactly once" printed on a product's page usually means "within our system" — the edge of that system is where your idempotency key goes.

🔧 How (in this repo)

deliver(messages, redeliver, idempotent) in dist/sim.py processes the messages and then the redeliveries (at-least-once). A plain consumer applies every one. An idempotent consumer keeps a seen set of message keys and skips any key it has already applied. It returns the list of effects.

🧪 Try it

python3 dist/demo.py exactlyonce
python3 - <<'EOF'
import sys; sys.path.insert(0, "dist"); from sim import deliver
msgs = ["pay-katrina", "pay-dipika", "pay-aishwarya"]
again = ["pay-dipika", "pay-dipika", "pay-katrina"]            # a bad night: three redeliveries
for idem in (False, True):
    eff = deliver(msgs, again, idem)
    print(f"{'idempotent' if idem else 'plain     '} → {len(eff)} payments: {eff}")
print("at-most-once (never redeliver) and a lost message →", deliver(["pay-katrina", "pay-aishwarya"], [], True))
EOF

✅ Verify — what you should see

exactlyonce prints:

── the queue redelivers pay-dipika (at-least-once) · plain consumer      → effects ['pay-katrina', 'pay-dipika', 'pay-aishwarya', 'pay-dipika']
── the queue redelivers pay-dipika (at-least-once) · idempotent consumer  → effects ['pay-katrina', 'pay-dipika', 'pay-aishwarya']

Your snippet prints:

plain      → 6 payments: ['pay-katrina', 'pay-dipika', 'pay-aishwarya', 'pay-dipika', 'pay-dipika', 'pay-katrina']
idempotent → 3 payments: ['pay-katrina', 'pay-dipika', 'pay-aishwarya']
at-most-once (never redeliver) and a lost message → ['pay-katrina', 'pay-aishwarya']

🏁 What you just proved

Three redeliveries turned 3 payments into 6 for the plain consumer, and changed nothing for the idempotent one. The last line is the other side: with at-most-once, a lost pay-dipika is simply gone — no duplicates, and no payment. Of the two, at-least-once + a key is the one you can fix.

⚠️ Common mistakes

believing a queue's "exactly once" covers the database write or the email your consumer makes
recording the key after the effect in a separate step — a crash between them repeats the effect
a dedupe table with no expiry, or an expiry shorter than the longest possible redelivery
keys made on the server for each attempt, so a retry gets a new key
"add 100" updates where "set to 600 if version is 4" would be idempotent

🏭 In production

An idempotent consumer in SQL — record the key and apply the effect in one transaction. On a real account (PostgreSQL):

BEGIN;
INSERT INTO processed_messages (message_id) VALUES ('pay-dipika')
    ON CONFLICT (message_id) DO NOTHING;            -- 0 rows inserted → already done: ROLLBACK and ack
UPDATE fees SET refunded = true WHERE student = 'dipika';
COMMIT;                                             -- then delete the message from the queue

Amazon SQS — standard queues are at-least-once (and may deliver out of order). A message that is not deleted within its visibility timeout is delivered again. FIFO queues drop a message with the same MessageDeduplicationId sent within a 5-minute window — this stops duplicate sends, but your consumer can still process a message twice if it crashes before deleting it:

aws sqs send-message --queue-url https://sqs.ap-south-1.amazonaws.com/111122223333/refunds.fifo \
    --message-body '{"student":"dipika","amount":600}' \
    --message-group-id dipika --message-deduplication-id pay-dipika

Kafka — in words: the idempotent producer (enable.idempotence=true, the default since Kafka 3.0, with acks=all) stops duplicates caused by the producer's own retries. Transactions (a transactional.id on the producer; isolation.level=read_committed on consumers) make read-process-write inside Kafka all-or-nothing, including the consumer offsets. Kafka Streams turns this on with processing.guarantee=exactly_once_v2. A consumer that writes to a database still needs a key or an upsert there.

HTTP APIs — accept an Idempotency-Key header on POST requests (Stripe does this) and return the saved response for a repeated key.

🏭 Why this matters in production: list every consumer and every POST endpoint that moves money, sends messages or changes counts. For each, name its idempotency key and where the key is stored. "The queue does exactly-once" is not an answer.

⏭️ Next

The last problem: too much work. Retries, bursts and slow copies pile up, and a system that accepts everything slows down for everyone. Next: backpressure — and the whole map.

git checkout lesson-12-backpressure

← Previouslocks leases Next →backpressure

This page is the lesson's README from the lesson-11-exactly-once branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.