🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🗂️ धडा 11 — Partitioning + DynamoDB: पसरणाऱ्या key ने नोंदवहीची विभागणी करा

📍 तुम्ही इथे आहात: 13 पैकी धडा 11 · मागील: lesson-10-replicas-and-pools · पुढील: lesson-12-queues-and-blueprint

📦 या ब्रँचमध्ये काय आहे

धडे 01–10, आणि data चीच विभागणी: hash partitioning, partition key, एका key ला बहुतेक traffic मिळते तेव्हाचा hot partition, तो पसरवण्यासाठी write-sharding, आणि DynamoDB चे per-partition capacity units (RCU आणि WCU). scale/demo.py मधले shards() 1,000 keys 4 partitions वर hash करते.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

नोंदवही एका कारकुनासाठी खूप मोठी आहे, म्हणून दीपिका ती 4 कारकुनांसह 4 कप्प्यांमध्ये 🗄️ विभागते. कोणता विद्यार्थी कोणत्या कप्प्यात? एक ठरलेला नियम: एक यंत्र प्रत्येक विद्यार्थ्याच्या नावाचे एका आकड्यात रूपांतर करते, आणि तो आकडा कप्पा निवडतो. एकाच नावाला नेहमी एकच कप्पा मिळतो. 1,000 विद्यार्थी असतील तर प्रत्येक कप्प्याला सुमारे 250. प्रत्येक कारकून सारखाच व्यस्त.

मग कोणीतरी म्हणते: "त्याऐवजी वर्गानुसार लावूया." निकालाच्या दिवशी 1,000 पैकी 900 नवीन नोंदी 3A वर्गाच्या असतात. 3A चे सगळे एकाच कप्प्यात जाते. त्या कारकुनाकडे प्रचंड काम, आणि बाकीचे तिघे रिकामे. यालाच hot partition म्हणतात.

उपाय: 3A च्या नोंदी "3A#0" ते "3A#39" खाली लिहा — 40 छोटी लेबले. यंत्र ती चारही कप्प्यांत पसरवते. सगळा 3A वाचायचा असेल तर सगळ्या 40 लेबलांना विचारा.

🗺️ आकृती

flowchart LR
    w["✏️ 1,000 writes"] --> h["#️⃣ hash(partition key) mod 4"]
    h --> p0["🗄️ drawer 0"]
    h --> p1["🗄️ drawer 1"]
    h --> p2["🗄️ drawer 2"]
    h --> p3["🗄️ drawer 3"]
    good["pupil-N → 245 · 262 · 249 · 244"]
    hot["class → 19 · 31 · 928 · 22 🔥"]
    fix["class-3A#0…#39 → 179 · 253 · 277 · 291"]

🗺️ रेखाटलेली आवृत्ती + एक lab: https://school-edh.pages.dev/scaling/lesson-diagrams.html#l11

❓ काय

Partitioning (sharding) — एक data set अनेक machines वर विभागणे, प्रत्येकीकडे एक भाग. Replicas च्या उलट, हे फक्त reads नाही तर write क्षमता आणि storage वाढवते.
Partition key — row कुठे राहील ते ठरवणारे field. Hash partitioning key चा hash काढून त्या निकालाने partition निवडते, म्हणजे सारख्या keys पसरतात.
Hot key / hot partition — एका key ला (किंवा थोड्या keys ना) बहुतेक traffic मिळते. तो partition त्याच्या मर्यादेला पोहोचतो आणि बाकीचे रिकामे राहतात. जास्त partitions ने फायदा होत नाही: एक key नेहमी एकाच partition वर जाते.
Write-sharding — hot key ला suffix जोडा (class-3A#0 … #39, random निवडलेला किंवा दुसऱ्या field वरून) म्हणजे तिचे writes पसरतात. मग reads ना प्रत्येक suffix ला विचारून एकत्र करावे लागते.
Amazon DynamoDB — एक key-value आणि document database जो तुमच्यासाठी partitioning करतो:
- प्रत्येक partition जास्तीत जास्त 3,000 RCU + 1,000 WCU देतो — read आणि write capacity units, फक्त "reads" आणि "writes" नाही:
  - 1 RCU = 4 KB पर्यंतच्या item चा दर सेकंदाला एक strongly consistent read, किंवा त्याच आकाराचे दोन eventually consistent reads;
  - 1 WCU = 1 KB पर्यंतच्या item चा दर सेकंदाला एक write;
  - मोठे items जास्त units वापरतात: 3 KB चा write 3 WCU खर्च करतो, म्हणून एक partition सेकंदाला 1,000 नाही तर सुमारे 333 असे writes घेतो;
- adaptive capacity — मदत करते: ती व्यस्त partition ला table च्या क्षमतेतला जास्त वाटा देते आणि hot items भोवती partitions विभागू शकते. पण ती एका item ला एका partition च्या मर्यादेपेक्षा जलद करू शकत नाही. खरा उपाय म्हणजे पसरणारी key.
- on-demand mode — प्रत्येक request नुसार पैसे, क्षमतेचे नियोजन नाही; provisioned mode — RCU/WCU ठरवा (auto scaling सह), स्थिर, माहीत असलेल्या load साठी स्वस्त.
Cross-partition query — प्रत्येक partition लागणारा प्रश्न ("सगळे विद्यार्थी grade नुसार क्रमाने"). Keys अशा design करा की नेहमीच्या प्रश्नांना एकच partition लागेल.

🤔 का

कारण replicas आणि caches (धडे 09–10) फक्त reads ला मदत करतात. निकालाचा दिवस सेकंदाला हजारो writes आणतो, किंवा data एका machine साठी खूप मोठा होतो, तेव्हा पुढे जाण्याचा एकच मार्ग म्हणजे त्याची विभागणी. विभागणी key इतकीच चांगली असते: पसरणारी key असेल तर N partitions N पट काम करतात; एकच hot value असलेली key असेल तर N partitions एकाचे काम करतात.

🔧 कसे (या repo मध्ये)

scale/sim.py मधले shard_of(key, shards) म्हणजे md5(key) mod shards — एक स्थिर hash, म्हणजे एकच key नेहमी एकाच ठिकाणी जाते. spread(keys, shards) प्रत्येक partition मध्ये किती keys जातात ते मोजते. shards() आधी 1,000 pupil ids पसरवते, मग class ने key केलेले 1,000 writes (त्यातले 900 3A), मग तेच writes 3A चे 40 भाग करून.

🧪 करून पाहा

python3 scale/demo.py shards
python3 - <<'EOF'
import sys; sys.path.insert(0, "scale"); from sim import spread, shard_of
keys = [f"pupil-{i}" for i in range(1000)]
for n in (2, 4, 8):
    print(f"{n} partitions →", spread(keys, n))
print("class-3A always lands on partition", shard_of("class-3A", 4))
hot = ["class-3A"] * 900 + [f"class-{c}" for c in range(100)]
for suffixes in (4, 10, 40):
    fixed = [f"class-3A#{i % suffixes}" for i in range(900)] + [f"class-{c}" for c in range(100)]
    print(f"write-sharding with {suffixes:>2} suffixes →", spread(fixed, 4))
share = max(spread(hot, 4)) / 1000
print(f"at 2,000 writes/s the hot partition gets ~{2000 * share:.0f} writes/s (limit 1,000 WCU per partition)")
EOF

✅ तपासा — तुम्हाला काय दिसायला हवे

shards हे छापते:

── 1,000 pupils hashed across 4 partitions → [245, 262, 249, 244]
   partition key = class, results day (900 of 1,000 writes are 3A) → [19, 31, 928, 22] — one HOT partition
   write-sharding the hot key (class-3A#0 … #39) → [179, 253, 277, 291]
   DynamoDB: a partition gives up to 3,000 RCU + 1,000 WCU (1 RCU = one strong 4 KB read/s, 1 WCU = one 1 KB write/s)
   bigger items use more units; adaptive capacity helps a bit — a key that spreads is still the real fix

तुमचा snippet हे छापतो:

2 partitions → [494, 506]
4 partitions → [245, 262, 249, 244]
8 partitions → [106, 133, 129, 128, 139, 129, 120, 116]
class-3A always lands on partition 2
write-sharding with  4 suffixes → [244, 31, 253, 472]
write-sharding with 10 suffixes → [109, 121, 478, 292]
write-sharding with 40 suffixes → [179, 253, 277, 291]
at 2,000 writes/s the hot partition gets ~1856 writes/s (limit 1,000 WCU per partition)

🏁 तुम्ही आत्ताच काय सिद्ध केले

अनेक values असलेली key 2, 4 किंवा 8 partitions वर सारखी पसरते. Class key 1,000 पैकी 928 writes partition 2 मध्ये टाकते — सेकंदाला 2,000 writes असताना ते सेकंदाला सुमारे 1,856 होतात, DynamoDB च्या प्रत्येक partition च्या 1,000 WCU पेक्षा खूप जास्त (आणि हे 1 KB किंवा त्याहून लहान items साठी; मोठे items परिस्थिती आणखी बिघडवतात). आणि थोडे suffixes पुरेसे नाहीत: 4 किंवा 10 असतील तर hash अजूनही दोन किंवा जास्त suffixes एकाच कप्प्यात टाकतो (472, 478). 40 असतील तर पसरण पुरेशी सारखी होते.

⚠️ नेहमीच्या चुका

थोड्याच values असलेली partition key (status, class, date) जिच्यावर सगळे एकाच वेळी writes करतात
date ही key: आजचे सगळे writes एकाच partition वर जातात
खूप कमी suffixes सह write-sharding, किंवा आता reads ना प्रत्येक suffix लागतो हे विसरणे
adaptive capacity एका खूप hot item ला वाचवेल अशी अपेक्षा
WCU ऐवजी "writes" मोजणे — 3 KB चा item प्रत्येक write ला 3 WCU खर्च करतो
tables SQL सारखे design करून मग scan करणे — DynamoDB मध्ये keys प्रश्नांवरून design करा
caches, replicas आणि चांगले indexes वापरून पाहण्याआधीच relational database हाताने विभागणे

🏭 प्रत्यक्ष वापरात

खऱ्या account वर — sharded class आणि विद्यार्थी यांनी key केलेले on-demand DynamoDB table:

aws dynamodb create-table --table-name results \
    --attribute-definitions AttributeName=pk,AttributeType=S AttributeName=sk,AttributeType=S \
    --key-schema AttributeName=pk,KeyType=HASH AttributeName=sk,KeyType=RANGE \
    --billing-mode PAY_PER_REQUEST

Suffix सह write करा; सगळे suffixes वाचून एकत्र करा (Python, boto3):

import random, boto3
from boto3.dynamodb.conditions import Key
table = boto3.resource("dynamodb").Table("results")
SUFFIXES = 40

def save(cls, pupil, grade):
    table.put_item(Item={"pk": f"class-{cls}#{random.randrange(SUFFIXES)}", "sk": f"pupil#{pupil}", "grade": grade})

def class_results(cls):
    items = []
    for n in range(SUFFIXES):                   # one query per suffix — run them in parallel in real code
        items += table.query(KeyConditionExpression=Key("pk").eq(f"class-{cls}#{n}"))["Items"]
    return items

DynamoDB साठी CloudWatch Contributor Insights ने hot keys शोधा:

aws dynamodb update-contributor-insights --table-name results --contributor-insights-action ENABLE

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: key निवडण्याआधी app विचारत असलेले पाच मुख्य प्रश्न लिहून काढा, आणि निकालाच्या दिवसाच्या data ने load test चालवा — जिथे एक value बाकीच्यांपेक्षा खूपच लोकप्रिय असते — सारख्या random test data ने नाही.

⏭️ पुढे

जे काम आत्ताच व्हायची गरज नाही — प्रमाणपत्रे, emails, PDFs — ते token रांगेत जाते. आणि मग संपूर्ण जत्रा, स्तरानुसार.

git checkout lesson-12-queues-and-blueprint

🗂️ Lesson 11 — Partitioning + DynamoDB: split the register by a key that spreads

📍 You are here: Lesson 11 of 13 · Previous: lesson-10-replicas-and-pools · Next: lesson-12-queues-and-blueprint

📦 What's in this branch

Lessons 01–10, plus splitting the data itself: hash partitioning, the partition key, the hot partition when one key gets most of the traffic, write-sharding to spread it, and DynamoDB's per-partition capacity units (RCU and WCU). shards() in scale/demo.py hashes 1,000 keys across 4 partitions.

🧒 Explain like I'm 5

The register is too big for one clerk, so Dipika splits it into 4 drawers 🗄️ with 4 clerks. Which drawer holds which pupil? A fixed rule: a machine turns each pupil's name into a number, and the number picks the drawer. The same name always gives the same drawer. With 1,000 pupils, each drawer gets about 250. Every clerk is equally busy.

Then someone says: "Let's file by class instead." On results day, 900 of 1,000 new entries are for class 3A. All of 3A goes to one drawer. That clerk has far too much work, and the other three are idle. That is a hot partition.

The fix: write 3A's entries under "3A#0" to "3A#39" — 40 little labels. The machine spreads them over all four drawers. To read all of 3A, you ask all 40 labels.

🗺️ Diagram

flowchart LR
    w["✏️ 1,000 writes"] --> h["#️⃣ hash(partition key) mod 4"]
    h --> p0["🗄️ drawer 0"]
    h --> p1["🗄️ drawer 1"]
    h --> p2["🗄️ drawer 2"]
    h --> p3["🗄️ drawer 3"]
    good["pupil-N → 245 · 262 · 249 · 244"]
    hot["class → 19 · 31 · 928 · 22 🔥"]
    fix["class-3A#0…#39 → 179 · 253 · 277 · 291"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/scaling/lesson-diagrams.html#l11

❓ What

Partitioning (sharding) — splitting one data set across many machines, each holding a part. Unlike replicas, it adds write capacity and storage, not only reads.
Partition key — the field that decides where a row lives. Hash partitioning hashes the key and uses the result to pick a partition, so similar keys spread out.
Hot key / hot partition — one key (or a few) gets most of the traffic. That partition hits its limit while the others idle. More partitions do not help: one key always lands on one partition.
Write-sharding — add a suffix to a hot key (class-3A#0 … #39, chosen at random or from another field) to spread its writes. Reads must then ask every suffix and merge.
Amazon DynamoDB — a key-value and document database that partitions for you:
- each partition gives up to 3,000 RCU + 1,000 WCU — read and write capacity units, not simply "reads" and "writes":
  - 1 RCU = one strongly consistent read per second of an item up to 4 KB, or two eventually consistent reads of that size;
  - 1 WCU = one write per second of an item up to 1 KB;
  - bigger items use more units: a 3 KB write costs 3 WCU, so one partition takes about 333 of them a second, not 1,000;
- adaptive capacity — helps: it gives a busy partition more of the table's capacity and can split partitions around hot items. But it cannot make one item faster than one partition allows. The real fix is a key that spreads.
- on-demand mode — pay per request, no capacity planning; provisioned mode — set RCU/WCU (with auto scaling), cheaper for steady, known load.
Cross-partition query — a question that needs every partition ("all pupils sorted by grade"). Design keys so the common questions need one partition.

🤔 Why

Because replicas and caches (lessons 09–10) only help reads. When results day brings thousands of writes a second, or the data is too big for one machine, the only way forward is to split it. The split is only as good as the key: a key that spreads makes N partitions do N times the work; a key with one hot value makes N partitions do the work of one.

🔧 How (in this repo)

shard_of(key, shards) in scale/sim.py is md5(key) mod shards — a stable hash, so the same key always lands in the same place. spread(keys, shards) counts how many keys land in each partition. shards() spreads 1,000 pupil ids, then 1,000 writes keyed by class (900 of them 3A), then the same writes with 3A split 40 ways.

🧪 Try it

python3 scale/demo.py shards
python3 - <<'EOF'
import sys; sys.path.insert(0, "scale"); from sim import spread, shard_of
keys = [f"pupil-{i}" for i in range(1000)]
for n in (2, 4, 8):
    print(f"{n} partitions →", spread(keys, n))
print("class-3A always lands on partition", shard_of("class-3A", 4))
hot = ["class-3A"] * 900 + [f"class-{c}" for c in range(100)]
for suffixes in (4, 10, 40):
    fixed = [f"class-3A#{i % suffixes}" for i in range(900)] + [f"class-{c}" for c in range(100)]
    print(f"write-sharding with {suffixes:>2} suffixes →", spread(fixed, 4))
share = max(spread(hot, 4)) / 1000
print(f"at 2,000 writes/s the hot partition gets ~{2000 * share:.0f} writes/s (limit 1,000 WCU per partition)")
EOF

✅ Verify — what you should see

shards prints:

── 1,000 pupils hashed across 4 partitions → [245, 262, 249, 244]
   partition key = class, results day (900 of 1,000 writes are 3A) → [19, 31, 928, 22] — one HOT partition
   write-sharding the hot key (class-3A#0 … #39) → [179, 253, 277, 291]
   DynamoDB: a partition gives up to 3,000 RCU + 1,000 WCU (1 RCU = one strong 4 KB read/s, 1 WCU = one 1 KB write/s)
   bigger items use more units; adaptive capacity helps a bit — a key that spreads is still the real fix

Your snippet prints:

2 partitions → [494, 506]
4 partitions → [245, 262, 249, 244]
8 partitions → [106, 133, 129, 128, 139, 129, 120, 116]
class-3A always lands on partition 2
write-sharding with  4 suffixes → [244, 31, 253, 472]
write-sharding with 10 suffixes → [109, 121, 478, 292]
write-sharding with 40 suffixes → [179, 253, 277, 291]
at 2,000 writes/s the hot partition gets ~1856 writes/s (limit 1,000 WCU per partition)

🏁 What you just proved

A key with many values spreads evenly at 2, 4 or 8 partitions. A class key puts 928 of 1,000 writes in partition 2 — at 2,000 writes a second that is about 1,856 per second, far over DynamoDB's 1,000 WCU per partition (and that is for items of 1 KB or less; bigger items make it worse). And a few suffixes are not enough: with 4 or 10, the hash still puts two or more suffixes in the same drawer (472, 478). With 40 the spread is even enough.

⚠️ Common mistakes

a partition key with few values (status, class, date) that everyone writes at once
a date as the key: all of today's writes go to one partition
write-sharding with too few suffixes, or forgetting that reads now need every suffix
expecting adaptive capacity to rescue one very hot item
counting "writes" instead of WCU — a 3 KB item costs 3 WCU per write
designing tables like SQL and then scanning — in DynamoDB, design keys from the questions
splitting a relational database by hand before trying caches, replicas and better indexes

🏭 In production

On a real account — an on-demand DynamoDB table keyed by a sharded class and the pupil:

aws dynamodb create-table --table-name results \
    --attribute-definitions AttributeName=pk,AttributeType=S AttributeName=sk,AttributeType=S \
    --key-schema AttributeName=pk,KeyType=HASH AttributeName=sk,KeyType=RANGE \
    --billing-mode PAY_PER_REQUEST

Write with a suffix; read all suffixes and merge (Python, boto3):

import random, boto3
from boto3.dynamodb.conditions import Key
table = boto3.resource("dynamodb").Table("results")
SUFFIXES = 40

def save(cls, pupil, grade):
    table.put_item(Item={"pk": f"class-{cls}#{random.randrange(SUFFIXES)}", "sk": f"pupil#{pupil}", "grade": grade})

def class_results(cls):
    items = []
    for n in range(SUFFIXES):                   # one query per suffix — run them in parallel in real code
        items += table.query(KeyConditionExpression=Key("pk").eq(f"class-{cls}#{n}"))["Items"]
    return items

Find hot keys with CloudWatch Contributor Insights for DynamoDB:

aws dynamodb update-contributor-insights --table-name results --contributor-insights-action ENABLE

🏭 Why this matters in production: write down the top five questions the app asks before you choose a key, and run a load test with results-day data — where one value is far more popular than the rest — not with evenly random test data.

⏭️ Next

Work that does not need to happen now — certificates, emails, PDFs — goes into a token queue. And then the whole fair, tier by tier.

git checkout lesson-12-queues-and-blueprint

← Previousreplicas and pools Next →queues and blueprint

This page is the lesson's README from the lesson-11-partitioning branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.