🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🎬 धडा 18 — सोडवलेल्या रचना: file storage · video · RAG

📍 तुम्ही इथे आहात: 18 पैकी धडा 18 · मागे: lesson-17-designs-short-notify-chat

📦 या ब्रँचमध्ये काय आहे

संपूर्ण course, आणि तीन जड रचना, प्रत्येक त्याच नऊ पायऱ्यांतून: requirements → numbers → API → data → blocks → failure → security → observability → cost. File storage (chunks, content hashes, dedupe, metadata vs blobs, sync), video (upload → transcode ladder → CDN; egress सर्वात मोठा) आणि RAG (ingest offline, ask online; आकार आणि token budgets).

design/designs.py — chunks(), sync_upload(), LADDER, video_storage_gb(), cdn_egress_tb(), rag()
design/demo.py — python3 design/demo.py designs2
design/test_design.py — check L18, आणि सगळे 18

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

दीपिकाच्या planning कार्यालयात शेवटच्या तीन मागण्या येतात.

📁 सामायिक files. शिक्षिका शाळेचा वार्षिक अहवाल एका सामायिक folder मध्ये ठेवतात. एक शब्द बदलतो. पूर्ण file पुन्हा प्रवास करायला हवी का? नाही. कार्यालय प्रत्येक file चे छोटे तुकडे करते आणि प्रत्येक तुकड्याला त्याच्या content वरून बनलेला ठसा देते. फक्त नव्या ठशाचे तुकडेच प्रवास करतात. दोन शिक्षिकांनी तोच form upload केला? कार्यालय तो एकदाच ठेवते.

🎬 वार्षिक दिवसाचे videos. Upload केलेले एक मिनिट चार आकारांत कॉपी केले जाते, स्पष्टपासून लहानपर्यंत, म्हणजे प्रत्येक phone ला त्याचे network वाहू शकेल असा आकार मिळतो. ते साठवणे स्वस्त आहे. ते लाखभर पालकांना पाठवणे हे मोठे बिल आहे.

🤖 शाळेच्या documents ना विचारा. एक पालक विचारते: "सहलीची फी कधी भरायची?" कार्यालयाने आधीच 10,000 documents चे छोटे परिच्छेद केले आहेत आणि प्रत्येक त्याच्या अर्थानुसार लावून ठेवला आहे — रात्री, कोणी विचारण्याआधी. प्रश्न आला की कार्यालय सर्वात जवळचे पाच परिच्छेद शोधते आणि प्रश्नासह ते AI ला देते. AI उत्तर देते आणि उत्तर कुठे सापडले ते दाखवते.

🗺️ आकृती

flowchart LR
    subgraph fs["📁 file storage"]
      f["file"] --> ck["✂️ chunks + SHA-256"] --> have{"server has it?"}
      have -->|"no"| s3[("🪣 object storage")]
      md[("🗄️ metadata DB<br/>file → chunk list")]
    end
    subgraph vd["🎬 video"]
      up["⬆️ upload"] --> tq["📬 queue"] --> tr["🎞️ transcode<br/>1080p · 720p · 480p · 240p"] --> cdn["🌍 CDN<br/>egress = the bill"]
    end
    subgraph rg["🤖 RAG"]
      ing["offline: chunk → embed → index"] --> vx[("🧭 vector index")]
      q["online: question → embed"] --> vx --> top["top-5"] --> llm["🧠 prompt → answer + sources"]
    end

🗺️ काढलेली आवृत्ती + एक lab: https://school-edh.pages.dev/system-design/lesson-diagrams.html#l18

❓ काय

📁 रचना 4 — file storage आणि sync

पायरी	आराखडा
requirements	upload, download, share, devices मध्ये sync, versions; अनेक GB पर्यंतच्या files; व्याप्तीबाहेर: real time मध्ये एकत्र editing
numbers	file च्या आकारापेक्षा बदलाचा आकार जास्त महत्त्वाचा: एका शब्दाच्या बदलाला एका chunk ची किंमत लागावी, पूर्ण file ची नाही
API	`POST /files/{id}/versions {chunk_hashes}` → `{missing}` · `PUT /chunks/{hash}` · `GET /files/{id}` → chunk list
data	database मध्ये metadata: `files`, `versions(file_id, version, chunk_hashes[])`, `shares`; object storage मध्ये blobs (chunks), hash ही key
blocks	client बाजूचा chunker; pre-signed URLs ने थेट object storage वर upload; नवी version आली की इतर devices ना notification
failure	uploads प्रत्येक chunk पासून पुन्हा सुरू होतात; प्रत्येक chunk साठवल्यावरच version दिसते (metadata शेवटी commit करा); conflicts → दोन्ही ठेवा ("conflicted copy")
security	प्रत्येक file आणि प्रत्येक share link साठी authorization; at rest encryption; फक्त chunk hash ने access मिळू नये
observability	upload success rate, dedupe ने वाचलेले bytes, devices मधला sync delay
cost	storage प्रतींनी नाही, unique chunks ने वाढते — dedupe म्हणजे पैसा

Chunking — file चे तुकडे करा (खऱ्या systems मध्ये: साधारण 4 MB; lab मध्ये 4 अक्षरे, म्हणजे तुम्हाला ते दिसते).
Content hash — तुकड्याचा SHA-256 हेच त्याचे नाव. सारखा content → सारखे नाव, कोणत्याही device वर, कोणत्याही user साठी.
Dedupe — upload करण्याआधी client hashes ची यादी पाठवतो; server उत्तर देतो त्याच्याकडे कोणते नाहीत. फक्त तेवढेच प्रवास करतात.
Fixed-size vs content-defined chunks — fixed-size chunks मध्ये सुरुवातीला दोन अक्षरे घातली तर प्रत्येक chunk सरकतो, आणि सगळे पुन्हा नवे होते. खरी sync tools content ने निवडलेल्या जागी कापतात (rolling hash), म्हणजे insert फक्त जवळचे chunks बदलतो.
Metadata vs blobs — छोटे, वारंवार query होणारे, transactions लागणारे → database. मोठे, एकदा लिहिलेले, key ने वाचलेले → object storage.

🎬 रचना 5 — video

पायरी	आराखडा
requirements	video upload करा; कोणत्याही phone आणि network वर पाहा; साधारण 2 सेकंदांत play सुरू
numbers	एक मिनिट → 4 renditions → 0.07 GB साठवले; 1M views × 10 min, 2.5 Mb/s ने → 187.5 TB egress
API	`POST /videos` → एक pre-signed upload URL · `GET /videos/{id}` → एक playlist URL (HLS / DASH)
data	`videos(id, owner, status, duration, renditions)`; files object storage मध्ये
blocks	upload → object storage → event → queue → transcode workers → segments + playlists → CDN
failure	transcoding हा queue मधला, retry होणारा, idempotent job आहे (धडा 07); एक rendition अपयशी झाली तर बाकीच्या अडत नाहीत
security	फक्त शाळेची कुटुंबेच पाहू शकतात: CDN वर signed URLs किंवा signed cookies; upload वर malware आणि content तपासणी
observability	पहिल्या frame पर्यंतचा वेळ, rebuffering ratio, transcode queue depth, CDN cache hit ratio
cost	egress सर्वात मोठा: उदाहरणादाखल किमतीला 187.5 TB पाठवायला साधारण $16,875; 1,000 तास साठवायला महिन्याला साधारण $91

Transcode ladder — तोच video अनेक bitrates वर: 1080p (5 Mb/s), 720p (2.5), 480p (1.0), 240p (0.3). network बदलतो तसा player त्यांच्यात बदलत राहतो (adaptive bitrate), छोटे segments वापरून.
CDN — पाहणाऱ्यांजवळ segments च्या प्रती (Scaling शाळेचा धडा 03). पालक पाहत असलेला प्रत्येक byte म्हणजे egress — दर GB ला पैसे.

🤖 रचना 6 — RAG (retrieval-augmented generation)

पायरी	आराखडा
requirements	पालक शाळेच्या documents बद्दल प्रश्न विचारतात; उत्तरे आपले sources सांगतात; ज्यांना पाहायची परवानगी नाही ते documents कोणालाच दिसत नाहीत
numbers	10,000 documents × 20 pages × 500 words, प्रत्येक chunk मध्ये 250 words → 400,000 chunks; 1,024 dimensions × 4 bytes → 1.64 GB index; प्रत्येक प्रश्नाला ~1,675 prompt tokens
API	`POST /ask {question}` → `{answer, sources[]}`
data	`chunks(id, doc_id, text, embedding, allowed_classes)`; documents object storage मध्ये
blocks	ingest (offline, queue मधून): parse → chunk → embed → index. Ask (online): प्रश्न embed करा → permission filter सह top-k search → prompt → sources सह उत्तर
failure	model हळू किंवा बंद → AI उत्तराशिवाय top passages दाखवा (degrade); प्रत्येक request साठी timeouts आणि token budget
security	permission नुसार filter search मध्येच करा, नंतर नाही; document चा मजकूर data म्हणून हाताळा, कधीही instructions म्हणून नाही (prompt injection); तुमच्या नियंत्रणाबाहेर जाणाऱ्या prompts मध्ये वैयक्तिक data नको
observability	retrieval quality (योग्य passage परत आला का?), source सह उत्तरे, प्रत्येक प्रश्नाचे tokens, पूर्ण ask चा p99
cost	embeddings साठी ingest वेळी एकदाच पैसे; tokens साठी प्रत्येक प्रश्नाला पैसे — prompt चा आकार हा चालू खर्च

Chunk size — मोठे chunks: कमी vectors, छोटा index, पण प्रत्येक प्रश्नाला जास्त tokens (500 words → 0.82 GB, 3,300 tokens). छोटे chunks: उलट.
Top-k — prompt मध्ये किती passages जातात. जास्त passages: जास्त संदर्भ, जास्त tokens (top-10 → 3,300 tokens).
Index size — chunks × dimensions × 4 bytes (float32), index च्या स्वतःच्या overhead आधी.
AI शाळेत (prompts, embeddings, evaluation) आणि Vector DB शाळेत (HNSW सारखे indexes, filters, recall) अधिक खोलात जा.

🤔 का

कारण जड data मुळे रचनेची कोणती ओळ महत्त्वाची ते बदलते. Files साठी बदलाचा आकार bandwidth ठरवतो, म्हणून content hashes हा गाभा. Video साठी egress बिल ठरवतो, म्हणून CDN आणि ladder हा गाभा. RAG साठी offline/online विभागणी latency ठरवते, आणि प्रत्येक प्रश्नाचे tokens चालू खर्च ठरवतात. त्याच नऊ पायऱ्या यातले प्रत्येक शोधतात — पद्धतीचा हाच तर मुद्दा आहे.

🔧 कसे (या repo मध्ये)

design/designs.py मध्ये:

chunks(data, size=4) — fixed-size तुकडे, प्रत्येकाचे नाव त्याच्या SHA-256 चे पहिले 8 hex अंक.
sync_upload(old, new, size=4) — नव्या chunks पैकी किती server कडे अजून नाहीत.
LADDER, video_storage_gb(minutes) आणि cdn_egress_tb(views, minutes_watched, mbps=2.5).
rag(documents, pages_each, words_per_page=500, chunk_words=250, dims=1024, question_tokens=50, top_k=5) — chunks, index GB, आणि prompt tokens (प्रत्येक word ला 1.3 tokens).

design/demo.py मधले designs2() तिन्ही चालवते. snippet धडा 16 मधल्या monthly_cost() ने video ची किंमतही काढतो.

🧪 करून पाहा

python3 design/demo.py designs2
python3 - <<'EOF'
import sys; sys.path.insert(0, "design"); from designs import chunks, sync_upload, video_storage_gb, cdn_egress_tb, rag, monthly_cost
old = "the school annual report for 2026 version one"
for new, what in ((old, "nothing changed"), (old.replace("one", "two"), "one word replaced"), ("A " + old, "two letters added at the start")):
    up, total = sync_upload(old, new); print(f"{what:<32} → upload {up:>2} of {total} chunks")
print("same content, same chunk id:", chunks("dipika") == chunks("dipika"), chunks("dipika"))
for minutes in (1, 10, 60):
    print(f"{minutes:>2} min uploaded → {video_storage_gb(minutes):>5} GB stored (all renditions)")
tb = cdn_egress_tb(1_000_000, 10)
print(f"1M views × 10 min → {tb} TB egress · at the example $0.09/GB →", monthly_cost(0, 0, tb * 1000, 0)["egress"], "$ · storing 1,000 hours →", monthly_cost(0, video_storage_gb(60) * 1000, 0, 0)["storage"], "$ a month")
for docs, k, words in ((10_000, 5, 250), (10_000, 10, 250), (100_000, 5, 250), (10_000, 5, 500)):
    print(f"{docs:>7,} docs, top-{k:<2}, {words} words per chunk → {rag(docs, 20, chunk_words=words, top_k=k)}")
EOF
python3 design/test_design.py

✅ तपासा — तुम्हाला काय दिसायला हवे

designs2 हे छापते:

── file storage: a file split into chunks by content hash; one word changed → upload 2 of 12 chunks
   metadata (who, which chunks, which version) in a database · chunks in object storage · dedupe for free
── video: one uploaded minute becomes 4 renditions (1080p, 720p, 480p, 240p) → 0.07 GB per minute stored
   1M views × 10 minutes at 2.5 Mb/s → 187.5 TB from the CDN — egress, not storage, is the bill
── RAG over 10,000 school documents × 20 pages → 400,000 chunks · vector index ~1.64 GB · ~1675 prompt tokens per question
   ingest (chunk → embed → index) runs offline; ask (embed question → top-5 → prompt → answer with sources) runs online
── the method, every time: requirements → numbers → API → data → blocks → failure → security → observability → cost → ADR

तुमचा snippet हे छापतो:

nothing changed                  → upload  0 of 12 chunks
one word replaced                → upload  2 of 12 chunks
two letters added at the start   → upload 12 of 12 chunks
same content, same chunk id: True ['0f12ca8a', '3ebb1c99']
 1 min uploaded →  0.07 GB stored (all renditions)
10 min uploaded →  0.66 GB stored (all renditions)
60 min uploaded →  3.96 GB stored (all renditions)
1M views × 10 min → 187.5 TB egress · at the example $0.09/GB → 16875.0 $ · storing 1,000 hours → 91.08 $ a month
 10,000 docs, top-5 , 250 words per chunk → {'chunks': 400000, 'index_gb': 1.64, 'prompt_tokens': 1675}
 10,000 docs, top-10, 250 words per chunk → {'chunks': 400000, 'index_gb': 1.64, 'prompt_tokens': 3300}
100,000 docs, top-5 , 250 words per chunk → {'chunks': 4000000, 'index_gb': 16.38, 'prompt_tokens': 1675}
 10,000 docs, top-5 , 500 words per chunk → {'chunks': 200000, 'index_gb': 0.82, 'prompt_tokens': 3300}

Tests ✅ L18 changing one word re-uploads only the changed chunks आणि 18/18 passed ने संपतात.

🏁 तुम्ही आत्ताच काय सिद्ध केले

Content hashes मुळे sync स्वस्त होते: बदल नाही → 0 chunks, एक शब्द → 12 पैकी 2. पण fixed-size chunks ची एक कमजोरी तुम्ही आत्ताच पाहिली: सुरुवातीच्या दोन अक्षरांनी प्रत्येक chunk सरकवला (12 पैकी 12) — म्हणूनच खरी tools content-defined chunks वापरतात. Video साठी, एका महिन्याचे views पाठवणे ($16,875) हे 1,000 तास साठवण्यापेक्षा ($91.08) साधारण 185 पट महाग आहे: आधी CDN आणि ladder आखा. RAG साठी, दहा पट documents म्हणजे दहा पट index (16.38 GB) पण prompt तोच (1,675 tokens) — तर top-k किंवा chunk size दुप्पट केल्यास प्रत्येक प्रश्नाला मोजावे लागणारे tokens दुप्पट होतात.

⚠️ नेहमीच्या चुका

प्रत्येक बदलाला पूर्ण files upload करणे, किंवा ठरलेल्या offsets वर chunking करून inserts मुळे सगळे पुन्हा upload का होते याचे आश्चर्य वाटणे
files database मध्ये, किंवा metadata object storage मध्ये साठवणे
नव्या version चे सगळे chunks साठवण्याआधीच ती दाखवणे
uploads थेट object storage ऐवजी API servers मधून पाठवणे
सगळ्यांसाठी एकच video आकार — हळू networks थांबून वाट पाहतात, जलद networks काही वाया घालवत नाहीत पण तरीही निकृष्ट दिसतात
video ची किंमत storage वरून काढणे आणि egress विसरणे
top-k नंतर permissions filter करणारे RAG — पालकाला पाहायची परवानगी नसलेल्या passages मधून उत्तर बनते, किंवा filter नंतर काहीच उरत नाही
प्रत्येक प्रश्नाला प्रत्येक document पुन्हा embed करणे, किंवा प्रत्येक prompt मध्ये 50 passages घालणे

🏭 प्रत्यक्ष वापरात

खऱ्या account वर — थेट S3 वर multipart upload (प्रत्येक part एकट्याने retry होतो; parts समांतर जाऊ शकतात):

aws s3api create-multipart-upload --bucket school-files --key chunks/3ebb1c99
aws s3api upload-part --bucket school-files --key chunks/3ebb1c99 --part-number 1 \
    --body part-1.bin --upload-id "EXAMPLE-UPLOAD-ID"
aws s3api complete-multipart-upload --bucket school-files --key chunks/3ebb1c99 \
    --upload-id "EXAMPLE-UPLOAD-ID" --multipart-upload file://parts.json

एक pre-signed URL, म्हणजे phone तुमच्या API मधून न जाता object storage वर upload करतो:

aws s3 presign s3://school-videos/uploads/annual-day.mp4 --expires-in 900

Video: AWS Elemental MediaConvert job upload चे HLS ladder मध्ये रूपांतर करतो (job settings file मध्ये चार renditions असतात), आणि CloudFront segments पुरवतो:

aws mediaconvert create-job --endpoint-url https://abcd1234.mediaconvert.ap-south-1.amazonaws.com \
    --role arn:aws:iam::111122223333:role/mediaconvert-school \
    --settings file://hls-ladder-1080-720-480-240.json

pgvector सह PostgreSQL वर RAG: HNSW index, आणि permission filter query च्या आत:

CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE chunks (id bigserial PRIMARY KEY, doc_id bigint, body text,
                     allowed_classes text[], embedding vector(1024));
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

SELECT doc_id, body
FROM chunks
WHERE allowed_classes && ARRAY['3A']               -- only what this parent may see
ORDER BY embedding <=> $1                           -- $1 = the question's embedding
LIMIT 5;

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: प्रत्येक जड रचनेसाठी बिलाशेजारी dashboard वर एक आकडा ठेवा: dedupe ने वाचलेले bytes, दररोजचा CDN egress, प्रत्येक प्रश्नाचे tokens. ते कोणत्याही server पेक्षा जास्त खर्च हलवतात.

🎓 आता planning कार्यालय तुमचे आहे

काढण्याआधी विचारा → मोजा → आधी करार → प्रश्नांवरून data → प्रती → पसरवणे → नंतर → अनेक ऐकणारे असलेली तथ्ये → एक deploy की अनेक → प्रतींमधले एकमत → प्रत्येक पायरीला एक undo → बरणीतले tokens → प्रत्येक बाणाला एक योजना → प्रत्येक box साठी सहा प्रश्न → निरोगी म्हणजे काय → एक बिल आणि एक कारण → सहा रचना, सुरुवातीपासून शेवटपर्यंत. तुम्ही फक्त system design शिकला नाहीत — कोणी बांधण्याआधीच तुम्ही system आखू शकता, आणि भिंतीवरचा प्रत्येक box समजावू शकता. 📐🏫🎓

⏭️ पुढे

इतर शाळा, जिथे तुमच्या आराखड्यातले boxes बांधले जातात:

Scaling शाळा — traffic: CDN, load balancers, auto scaling, replicas, partitions आणि failure;
Database शाळा — data: models, indexes, transactions आणि replication;
API शाळा — करार: REST, errors, pagination, versioning आणि auth;
AI शाळा — RAG रचनेमागचे models;
School portal — प्रत्येक course एकाच ठिकाणी.

git checkout main
python3 design/demo.py     # one last run, for fun

🎬 Lesson 18 — Worked designs: file storage · video · RAG

📍 You are here: Lesson 18 of 18 · Previous: lesson-17-designs-short-notify-chat

📦 What's in this branch

The whole course, plus three heavy designs, each walked through the same nine steps: requirements → numbers → API → data → blocks → failure → security → observability → cost. File storage (chunks, content hashes, dedupe, metadata vs blobs, sync), video (upload → transcode ladder → CDN; egress dominates) and RAG (ingest offline, ask online; sizes and token budgets).

design/designs.py — chunks(), sync_upload(), LADDER, video_storage_gb(), cdn_egress_tb(), rag()
design/demo.py — python3 design/demo.py designs2
design/test_design.py — check L18, and all 18

🧒 Explain like I'm 5

Three last requests come to Dipika's planning office.

📁 Shared files. Teachers keep the school's annual report in a shared folder. One word changes. Must the whole file travel again? No. The office cuts every file into small pieces and gives each piece a fingerprint made from its content. Only pieces with a new fingerprint travel. Two teachers upload the same form? The office keeps it once.

🎬 Videos of the annual day. One uploaded minute is copied into four sizes, from sharp to small, so every phone gets a size its network can carry. Storing them is cheap. Sending them to a million parents is the big bill.

🤖 Ask the school documents. A parent asks: "When is the fee due for the trip?" The office has already cut 10,000 documents into small passages and filed each one by its meaning — at night, before anyone asks. When the question comes, the office finds the five closest passages and gives them to the AI with the question. The AI answers and shows where it found the answer.

🗺️ Diagram

flowchart LR
    subgraph fs["📁 file storage"]
      f["file"] --> ck["✂️ chunks + SHA-256"] --> have{"server has it?"}
      have -->|"no"| s3[("🪣 object storage")]
      md[("🗄️ metadata DB<br/>file → chunk list")]
    end
    subgraph vd["🎬 video"]
      up["⬆️ upload"] --> tq["📬 queue"] --> tr["🎞️ transcode<br/>1080p · 720p · 480p · 240p"] --> cdn["🌍 CDN<br/>egress = the bill"]
    end
    subgraph rg["🤖 RAG"]
      ing["offline: chunk → embed → index"] --> vx[("🧭 vector index")]
      q["online: question → embed"] --> vx --> top["top-5"] --> llm["🧠 prompt → answer + sources"]
    end

🗺️ Drawn version + a lab: https://school-edh.pages.dev/system-design/lesson-diagrams.html#l18

❓ What

📁 Design 4 — file storage and sync

step	the plan
requirements	upload, download, share, sync across devices, versions; files up to many GB; out of scope: editing together in real time
numbers	the size of change matters more than file size: a one-word change should cost one chunk, not the file
API	`POST /files/{id}/versions {chunk_hashes}` → `{missing}` · `PUT /chunks/{hash}` · `GET /files/{id}` → chunk list
data	metadata in a database: `files`, `versions(file_id, version, chunk_hashes[])`, `shares`; blobs (the chunks) in object storage, keyed by hash
blocks	client-side chunker; upload directly to object storage with pre-signed URLs; a notification to other devices when a new version exists
failure	uploads resume per chunk; a version is visible only when every chunk is stored (commit the metadata last); conflicts → keep both ("conflicted copy")
security	authorization per file and per share link; encryption at rest; a chunk hash alone must not grant access
observability	upload success rate, bytes saved by dedupe, sync delay between devices
cost	storage grows with unique chunks, not with copies — dedupe is money

Chunking — split a file into pieces (real systems: about 4 MB; the lab uses 4 characters so you can see it).
Content hash — SHA-256 of the piece is its name. Same content → same name, on any device, for any user.
Dedupe — before uploading, the client sends the list of hashes; the server answers which ones it is missing. Only those travel.
Fixed-size vs content-defined chunks — with fixed-size chunks, inserting two letters at the start shifts every chunk, and everything is new again. Real sync tools cut at points chosen by the content (a rolling hash), so an insert changes only nearby chunks.
Metadata vs blobs — small, queried often, needs transactions → a database. Large, written once, read by key → object storage.

🎬 Design 5 — video

step	the plan
requirements	upload a video; watch on any phone and network; start playing in about 2 seconds
numbers	one minute → 4 renditions → 0.07 GB stored; 1M views × 10 min at 2.5 Mb/s → 187.5 TB egress
API	`POST /videos` → a pre-signed upload URL · `GET /videos/{id}` → a playlist URL (HLS / DASH)
data	`videos(id, owner, status, duration, renditions)`; the files in object storage
blocks	upload → object storage → event → queue → transcode workers → segments + playlists → CDN
failure	transcoding is a queued, retried, idempotent job (lesson 07); a failed rendition does not block the others
security	only the school's families may watch: signed URLs or signed cookies on the CDN; malware and content checks on upload
observability	time to first frame, rebuffering ratio, transcode queue depth, CDN cache hit ratio
cost	egress dominates: sending 187.5 TB costs about $16,875 at the example price; storing 1,000 hours costs about $91 a month

Transcode ladder — the same video at several bitrates: 1080p (5 Mb/s), 720p (2.5), 480p (1.0), 240p (0.3). The player switches between them as the network changes (adaptive bitrate), using short segments.
CDN — copies of the segments near the viewers (the Scaling school's lesson 03). Every byte a parent watches is egress — paid per GB.

🤖 Design 6 — RAG (retrieval-augmented generation)

step	the plan
requirements	parents ask questions about school documents; answers cite their sources; nobody sees documents they may not see
numbers	10,000 documents × 20 pages × 500 words, 250 words per chunk → 400,000 chunks; 1,024 dimensions × 4 bytes → 1.64 GB index; ~1,675 prompt tokens per question
API	`POST /ask {question}` → `{answer, sources[]}`
data	`chunks(id, doc_id, text, embedding, allowed_classes)`; the documents in object storage
blocks	ingest (offline, queued): parse → chunk → embed → index. Ask (online): embed the question → top-k search with a permission filter → prompt → answer with sources
failure	the model is slow or down → show the top passages without an AI answer (degrade); timeouts and a token budget per request
security	filter by permission in the search, not after; treat document text as data, never as instructions (prompt injection); no personal data in prompts that leave your control
observability	retrieval quality (did the right passage come back?), answers with a source, tokens per question, p99 of the whole ask
cost	embeddings are paid once at ingest; tokens are paid on every question — the prompt size is the running cost

Chunk size — bigger chunks: fewer vectors, a smaller index, but more tokens per question (500 words → 0.82 GB, 3,300 tokens). Smaller chunks: the opposite.
Top-k — how many passages go into the prompt. More passages: more context, more tokens (top-10 → 3,300 tokens).
Index size — chunks × dimensions × 4 bytes (float32), before the index's own overhead.
Go deeper in the AI school (prompts, embeddings, evaluation) and the Vector DB school (indexes such as HNSW, filters, recall).

🤔 Why

Because heavy data changes which line of the design matters. For files, the size of a change decides the bandwidth, so content hashes are the core. For video, egress decides the bill, so the CDN and the ladder are the core. For RAG, the offline/online split decides the latency, and tokens per question decide the running cost. The same nine steps find each of these — which is the point of a method.

🔧 How (in this repo)

In design/designs.py:

chunks(data, size=4) — fixed-size pieces, each named by the first 8 hex digits of its SHA-256.
sync_upload(old, new, size=4) — how many of the new chunks the server does not have yet.
LADDER, video_storage_gb(minutes) and cdn_egress_tb(views, minutes_watched, mbps=2.5).
rag(documents, pages_each, words_per_page=500, chunk_words=250, dims=1024, question_tokens=50, top_k=5) — chunks, index GB, and prompt tokens (1.3 tokens per word).

designs2() in design/demo.py runs all three. The snippet also prices video with monthly_cost() from lesson 16.

🧪 Try it

python3 design/demo.py designs2
python3 - <<'EOF'
import sys; sys.path.insert(0, "design"); from designs import chunks, sync_upload, video_storage_gb, cdn_egress_tb, rag, monthly_cost
old = "the school annual report for 2026 version one"
for new, what in ((old, "nothing changed"), (old.replace("one", "two"), "one word replaced"), ("A " + old, "two letters added at the start")):
    up, total = sync_upload(old, new); print(f"{what:<32} → upload {up:>2} of {total} chunks")
print("same content, same chunk id:", chunks("dipika") == chunks("dipika"), chunks("dipika"))
for minutes in (1, 10, 60):
    print(f"{minutes:>2} min uploaded → {video_storage_gb(minutes):>5} GB stored (all renditions)")
tb = cdn_egress_tb(1_000_000, 10)
print(f"1M views × 10 min → {tb} TB egress · at the example $0.09/GB →", monthly_cost(0, 0, tb * 1000, 0)["egress"], "$ · storing 1,000 hours →", monthly_cost(0, video_storage_gb(60) * 1000, 0, 0)["storage"], "$ a month")
for docs, k, words in ((10_000, 5, 250), (10_000, 10, 250), (100_000, 5, 250), (10_000, 5, 500)):
    print(f"{docs:>7,} docs, top-{k:<2}, {words} words per chunk → {rag(docs, 20, chunk_words=words, top_k=k)}")
EOF
python3 design/test_design.py

✅ Verify — what you should see

designs2 prints:

── file storage: a file split into chunks by content hash; one word changed → upload 2 of 12 chunks
   metadata (who, which chunks, which version) in a database · chunks in object storage · dedupe for free
── video: one uploaded minute becomes 4 renditions (1080p, 720p, 480p, 240p) → 0.07 GB per minute stored
   1M views × 10 minutes at 2.5 Mb/s → 187.5 TB from the CDN — egress, not storage, is the bill
── RAG over 10,000 school documents × 20 pages → 400,000 chunks · vector index ~1.64 GB · ~1675 prompt tokens per question
   ingest (chunk → embed → index) runs offline; ask (embed question → top-5 → prompt → answer with sources) runs online
── the method, every time: requirements → numbers → API → data → blocks → failure → security → observability → cost → ADR

Your snippet prints:

nothing changed                  → upload  0 of 12 chunks
one word replaced                → upload  2 of 12 chunks
two letters added at the start   → upload 12 of 12 chunks
same content, same chunk id: True ['0f12ca8a', '3ebb1c99']
 1 min uploaded →  0.07 GB stored (all renditions)
10 min uploaded →  0.66 GB stored (all renditions)
60 min uploaded →  3.96 GB stored (all renditions)
1M views × 10 min → 187.5 TB egress · at the example $0.09/GB → 16875.0 $ · storing 1,000 hours → 91.08 $ a month
 10,000 docs, top-5 , 250 words per chunk → {'chunks': 400000, 'index_gb': 1.64, 'prompt_tokens': 1675}
 10,000 docs, top-10, 250 words per chunk → {'chunks': 400000, 'index_gb': 1.64, 'prompt_tokens': 3300}
100,000 docs, top-5 , 250 words per chunk → {'chunks': 4000000, 'index_gb': 16.38, 'prompt_tokens': 1675}
 10,000 docs, top-5 , 500 words per chunk → {'chunks': 200000, 'index_gb': 0.82, 'prompt_tokens': 3300}

The tests end with ✅ L18 changing one word re-uploads only the changed chunks and 18/18 passed.

🏁 What you just proved

Content hashes make sync cheap: no change → 0 chunks, one word → 2 of 12. But fixed-size chunks have a weakness you just saw: two letters at the start shifted every chunk (12 of 12) — that is why real tools use content-defined chunks. For video, sending one month of views ($16,875) costs about 185 times more than storing 1,000 hours ($91.08): design the CDN and the ladder first. For RAG, ten times the documents gives ten times the index (16.38 GB) but the same prompt (1,675 tokens) — while doubling top-k or the chunk size doubles the tokens paid on every question.

⚠️ Common mistakes

uploading whole files on every change, or chunking at fixed offsets and wondering why inserts re-upload everything
storing files in the database, or metadata in object storage
making a new version visible before all its chunks are stored
sending uploads through the API servers instead of directly to object storage
one video size for everyone — slow networks stop and wait, fast ones waste nothing but still look poor
pricing video by storage and forgetting egress
RAG that filters permissions after top-k — the answer is built from passages the parent may not see, or the filter leaves nothing
re-embedding every document on every question, or putting 50 passages in each prompt

🏭 In production

On a real account — a multipart upload straight to S3 (each part is retried alone; parts can go in parallel):

aws s3api create-multipart-upload --bucket school-files --key chunks/3ebb1c99
aws s3api upload-part --bucket school-files --key chunks/3ebb1c99 --part-number 1 \
    --body part-1.bin --upload-id "EXAMPLE-UPLOAD-ID"
aws s3api complete-multipart-upload --bucket school-files --key chunks/3ebb1c99 \
    --upload-id "EXAMPLE-UPLOAD-ID" --multipart-upload file://parts.json

A pre-signed URL, so the phone uploads to object storage without going through your API:

aws s3 presign s3://school-videos/uploads/annual-day.mp4 --expires-in 900

Video: an AWS Elemental MediaConvert job turns the upload into an HLS ladder (the job settings file holds the four renditions), and CloudFront serves the segments:

aws mediaconvert create-job --endpoint-url https://abcd1234.mediaconvert.ap-south-1.amazonaws.com \
    --role arn:aws:iam::111122223333:role/mediaconvert-school \
    --settings file://hls-ladder-1080-720-480-240.json

RAG on PostgreSQL with pgvector: an HNSW index, and the permission filter inside the query:

CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE chunks (id bigserial PRIMARY KEY, doc_id bigint, body text,
                     allowed_classes text[], embedding vector(1024));
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

SELECT doc_id, body
FROM chunks
WHERE allowed_classes && ARRAY['3A']               -- only what this parent may see
ORDER BY embedding <=> $1                           -- $1 = the question's embedding
LIMIT 5;

🏭 Why this matters in production: for each heavy design, keep one number on a dashboard next to the bill: bytes saved by dedupe, CDN egress per day, tokens per question. They move the cost more than any server.

🎓 The planning office is yours

Ask before you draw → count → the contract first → data from its questions → copies → spreading → later → facts with many listeners → one deploy or many → agreement between copies → an undo for every step → tokens in a jar → a plan for every arrow → six questions for every box → what healthy means → a bill and a reason → six designs, end to end. You didn't just learn system design — you can plan a system before anyone builds it, and explain every box on the wall. 📐🏫🎓

⏭️ Next

The other schools, where the boxes on your plan are built:

the Scaling school — the traffic: CDN, load balancers, auto scaling, replicas, partitions and failure;
the Database school — the data: models, indexes, transactions and replication;
the API school — the contract: REST, errors, pagination, versioning and auth;
the AI school — the models behind the RAG design;
the School portal — every course in one place.

git checkout main
python3 design/demo.py     # one last run, for fun

← Previousdesigns short notify chat Finished! Take the quiz →check what stuck

This page is the lesson's README from the lesson-18-designs-files-video-rag branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.