✂️ धडा 06 — Chunking आणि metadata: कार्डे आणि रंगीत स्टिकर

📍 तुम्ही इथे आहात: 8 पैकी धडा 06 · मागे: lesson-05-build-a-vector-db · पुढे: lesson-07-rag-wiring

📦 या ब्रँचमध्ये काय आहे

धडे 01–05, आणि कोणत्याही index पेक्षा retrieval चा दर्जा जास्त बदलणारे साधेसुधे निर्णय: तुम्ही कसे कापता (chunking) आणि तुम्ही कसे label करता (metadata) — शिवाय hybrid search.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

400 पानांची handbook तुम्ही एका खुर्चीत बसवू शकत नाही. 📚 Hall मध्ये नेण्याआधी ग्रंथपाल पुस्तकांचे index cards मध्ये तुकडे करते ✂️ — आणि कापणे ही एक कला आहे:

खूप मोठी cards → एका card मध्ये पाच विषय मिसळतात; त्याची जागा कुठेच नसलेल्या धूसर मध्यभागी पडते (अर्थांची सरासरी बहुतेक वेळा काहीच अर्थ नसते). शिवाय: जाड cards नंतर छोट्या बाकावर पूर आणतात (AI शाळा L08).
खूप छोटी cards → "…ते परत केलेच पाहिजे." काय परत? जागा नेमकी असते पण शेजाऱ्यांशिवाय card निरुपयोगी असते.
कौशल्य: नैसर्गिक शिवणींवर कापा (परिच्छेद, विभाग, headings — वाक्याच्या मध्ये कधीच नाही), chunks जवळपास स्वयंपूर्ण ठेवा, overlap 🔁 जोडा (प्रत्येक card मागच्या card च्या शेवटच्या ओळी पुन्हा लिहिते) म्हणजे सीमेवर पसरलेल्या कल्पना किमान एका card वर पूर्ण राहतात. Chunk size आणि overlap document ची रचना आणि retrieval च्या गरजांवर अवलंबून असतात — थोड्या overlap सह काहीशे tokens ही सुरुवातीची सामान्य जागा आहे, नियम नाही; तुमच्या data वर TUNE करा.

आणि प्रत्येक card ला रंगीत stickers 🏷️ (metadata) मिळतात: room: 3A, kind: rules, year: 2026, source: handbook-p12. Stickers मुळे दोन महाशक्ती मिळतात:

Filtered search — "फक्त kind=rules मधली सर्वात जवळची cards" (आपले search(query, kind="rules") — तुम्ही demo मध्ये चालवले होते!). अर्थ उमेदवार शोधतो; stickers तथ्ये लागू करतात (tenant! permissions! — k8s namespaces ची सहजप्रवृत्ती).
पावत्या — source stickers मुळेच RAG पाने सांगू (cite करू) शकते (धडा 07).

Hybrid search 🤝: अर्थ-search नेमक्या strings चुकवतो ("error E-4012", part numbers, नावे); keyword search समानार्थी शब्द चुकवतो. प्रौढ systems दोन्ही चालवतात आणि एकत्र करतात (RRF) — ग्रंथपाल card catalog पण तपासते आणि hall मध्ये फिरतेही.

🗺️ आकृती

flowchart LR
    book["📚 400-page handbook"]
    cut["✂️ chunking<br/>natural seams · self-contained ·<br/>size & overlap fit the docs (e.g. ~hundreds of tokens) 🔁"]
    cards["🗂️ index cards<br/>+ stickers 🏷️ room/kind/source"]
    hall["🗺️ the hall<br/>filtered search: stickers FIRST,<br/>then nearest-by-meaning"]
    hybrid["🤝 hybrid: + keyword catalog<br/>for exact strings (E-4012) — merge results"]
    book --> cut --> cards --> hall
    hall <--> hybrid

❓ काय

Chunking रणनीती: fixed-size (सोपी, शिवणी न पाहणारी) · by-structure (headings/परिच्छेद — सहसा सर्वोत्तम) · semantic (embed करून विषय बदलतो तिथे विभागणे — भारी, कधी कधी फायद्याचे). Citations साठी नेहमी source + स्थान साठवा.
Overlap storage देऊन सीमेवरची सुरक्षितता मिळवतो. माफक overlap ही समजूतदार सुरुवात आहे; किती ते document च्या रचनेवर अवलंबून असते.
Scale वर metadata filtering हे खरे feature आहे, for-loop नाही: pre-filter ANN search ची जागा लहान करतो (वेगवान, पण index ने ते support केले पाहिजे); post-filter तुमच्या top-k ला उपाशी ठेवू शकतो. धडा 08 मधली products नेमकी इथेच वेगळी पडतात.
Evals चा प्रतिध्वनी (Agents शाळा L06): retrieval चा स्वतःचा metric असतो — "योग्य card परत आले का?" प्रश्नोत्तरांचा एक छोटा golden-questions संच बनवा आणि chunking बदलताना प्रत्येक वेळी तो पुन्हा चालवा. Chunking बदलांनी retrieval चा दर्जा नाट्यमयरीत्या बदलू शकतो; index tuning मुख्यतः speed बदलते. त्यानुसार मेहनत खर्च करा.

🤔 का

"RAG वाईट उत्तरे देतो" अशा दहापैकी नऊ tickets प्रत्यक्षात retrieval tickets असतात, आणि बहुतेक retrieval tickets म्हणजे chunking tickets: model ने वाईट card वरून बरोबर उत्तर दिलेले असते. खरा नॉब कुठे आहे ते आता तुम्हाला माहीत आहे — आणि तो कात्रीने धरला जातो, indexes ने नाही. ✂️

🧪 करून पाहा

python3 - <<'EOF'
import sys; sys.path.insert(0,'vectordb')
from vectordb import MiniVectorDB
db = MiniVectorDB()
# ONE fat card mixing topics vs two clean cards:
db.add("fat",  "Library books are due in two weeks. The picnic needs one pizza per three students.")
db.add("liba", "Library books are due in two weeks.", kind="rules")
db.add("picn", "The picnic needs one pizza per three students.", kind="rules")
for s, i, t, m in db.search("how many pizzas for the picnic?", k=3):
    print(f"{s:.2f}  [{i}] {t}")
EOF
# the clean picnic card outranks the fat mixed one — chunking IS ranking

⏭️ पुढे

हे सगळे प्रत्येक जण बनवतो त्या pipeline मध्ये जोडा: RAG, सुरुवातीपासून शेवटपर्यंत — आपल्याच database ला page-finder बनवून.

git checkout lesson-07-rag-wiring

✂️ Lesson 06 — Chunking & metadata: cards and colored stickers

📍 You are here: Lesson 06 of 8 · Previous: lesson-05-build-a-vector-db · Next: lesson-07-rag-wiring

📦 What's in this branch

Lessons 01–05, plus the unglamorous decisions that move retrieval quality more than any index: how you cut (chunking) and how you label (metadata) — plus hybrid search.

🧒 Explain like I'm 5

You can't seat a 400-page handbook in ONE chair. 📚 Before the hall, the librarian cuts books into index cards ✂️ — and the cutting is an art:

Cards too big → one card mixes five topics; its seat lands in the murky middle of nowhere (an average of meanings is often no meaning). Also: fat cards flood the small desk later (AI school L08).
Cards too small → "…it must be returned." Returned WHAT? The seat is precise but the card is useless without its neighbors.
The craft: cut on natural seams (paragraphs, sections, headings — never mid-sentence), keep chunks self-contained-ish, add overlap 🔁 (each card repeats the last lines of the previous one) so ideas spanning a boundary live fully on at least one card. Chunk size and overlap depend on document structure and retrieval needs — a few hundred tokens with some overlap is a common starting point, not a rule; TUNE on your data.

And every card gets colored stickers 🏷️ (metadata): room: 3A, kind: rules, year: 2026, source: handbook-p12. Stickers make two superpowers:

Filtered search — "nearest cards among kind=rules only" (our search(query, kind="rules") — you ran it in the demo!). Meaning finds candidates; stickers enforce facts (tenant! permissions! — the k8s namespaces instinct).
Receipts — source stickers are what lets RAG cite pages (lesson 07).

Hybrid search 🤝: meaning-search misses exact strings ("error E-4012", part numbers, names); keyword search misses synonyms. Grown-up systems run BOTH and merge (RRF) — the librarian checks the card catalog AND walks the hall.

🗺️ Diagram

flowchart LR
    book["📚 400-page handbook"]
    cut["✂️ chunking<br/>natural seams · self-contained ·<br/>size & overlap fit the docs (e.g. ~hundreds of tokens) 🔁"]
    cards["🗂️ index cards<br/>+ stickers 🏷️ room/kind/source"]
    hall["🗺️ the hall<br/>filtered search: stickers FIRST,<br/>then nearest-by-meaning"]
    hybrid["🤝 hybrid: + keyword catalog<br/>for exact strings (E-4012) — merge results"]
    book --> cut --> cards --> hall
    hall <--> hybrid

❓ What

Chunking strategies: fixed-size (simple, seam-blind) · by-structure (headings/paragraphs — usually best) · semantic (embed-and-split at topic shifts — fancy, sometimes worth it). Always store source + position for citations.
Overlap trades storage for boundary-safety. A modest overlap is a sensible starting point; how much depends on the document structure.
Metadata filtering at scale is a real feature, not a for-loop: pre-filter shrinks the ANN search space (fast, but the index must support it); post-filter can starve your top-k. Lesson 08's products differ exactly here.
The evals echo (Agents school L06): retrieval has its own metric — "did the right card come back?" Build a tiny golden-questions set and re-run it whenever you change chunking. Chunking changes can move retrieval quality dramatically; index tuning mostly moves speed. Spend accordingly.

🤔 Why

Nine out of ten "RAG gives bad answers" tickets are retrieval tickets, and most retrieval tickets are chunking tickets: the model answered correctly from a bad card. You now know where the real dial is — and that it's held with scissors, not indexes. ✂️

🧪 Try it

python3 - <<'EOF'
import sys; sys.path.insert(0,'vectordb')
from vectordb import MiniVectorDB
db = MiniVectorDB()
# ONE fat card mixing topics vs two clean cards:
db.add("fat",  "Library books are due in two weeks. The picnic needs one pizza per three students.")
db.add("liba", "Library books are due in two weeks.", kind="rules")
db.add("picn", "The picnic needs one pizza per three students.", kind="rules")
for s, i, t, m in db.search("how many pizzas for the picnic?", k=3):
    print(f"{s:.2f}  [{i}] {t}")
EOF
# the clean picnic card outranks the fat mixed one — chunking IS ranking

⏭️ Next

Wire it all into the pipeline everyone builds: RAG, end to end — with our own database as the page-finder.

git checkout lesson-07-rag-wiring

← Previousbuild a vector db Next →rag wiring

This page is the lesson's README from the lesson-06-chunking-metadata branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.