🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🔑 धडा 08 — Secrets आणि config as a service: किल्ल्यांचे कपाट

📍 तुम्ही इथे आहात: 12 पैकी धडा 08 · मागील: lesson-07-paved-pipelines · पुढील: lesson-09-policy-guardrails

📦 या ब्रँचमध्ये काय आहे

धडे 01–07, आणि secrets आणि configuration as a service. Secrets एकाच store मध्ये, प्रत्येक team च्या स्वतंत्र path खाली राहतात; team फक्त स्वतःचेच वाचू शकते; प्रत्येक बदलातून नवे version बनते; rotation नंतर store दाखवते की कोणते consumers अजून जुने version वापरत आहेत. Configuration layers मध्ये येते — आधी defaults, मग environment, मग service — आणि प्रत्येक अंतिम value सांगते की ते कोणत्या layer ने ठरवले. SecretStore आणि layered_config() idp/selfserve.py मध्ये, secrets() idp/demo.py मध्ये.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

सुविधा कार्यालयात एक किल्ल्यांचे कपाट आहे. 🔑 प्रत्येक विभागाचा स्वतःचा खुंटा आहे: science च्या किल्ल्या science च्या खुंट्यावर, maths च्या किल्ल्या maths च्या खुंट्यावर.

कतरिना science lab ची किल्ली मागते: ✅ "ही घ्या — किल्ली version 1." ती maths च्या भांडारखोलीची किल्ली मागते: ❌ "ती maths च्या खुंट्यावर आहे. maths ला ती द्यायला सांगा."

अधूनमधून दीपिका कुलूप बदलते (म्हणजे rotation) आणि किल्ली version 2 टांगते. जुनी किल्ली कोणी कोणी नेली याची यादी ती ठेवते. Exam-results चा कारकून नवी किल्ली घ्यायला परत येतो ✅. Lab worker अजून परत आलेला नाही ⚠️. म्हणून lab worker कडेही नवी किल्ली येईपर्यंत दीपिका जुने कुलूप फेकून देत नाही.

आणि कार्यालय प्रत्येक खोलीसाठी एक settings पत्रक ठेवते: आधी शाळेचा default ("दिवे नेहमीसारखे"), मग इमारतीचा नियम ("परीक्षा हॉलमध्ये 2 नव्हे, 3 सफाई कर्मचारी"), मग खोलीची स्वतःची इच्छा ("नवा grades board: चालू"). कोणीही पत्रक वाचून प्रत्येक ओळ कोणत्या नियमाने ठरवली ते पाहू शकतो.

🗺️ आकृती

flowchart LR
    subgraph store["🔑 secret store"]
      s1["teams/science/results-db-password<br/>v1 → v2 (rotated)"]
      s2["teams/maths/timetable-api-key"]
    end
    sci["🧪 science"] -->|"✅ own path"| s1
    sci -.->|"❌ denied"| s2
    s1 --> er["exam-results: reloaded v2"]
    s1 --> lw["lab-worker: still on v1 ⚠️"]
    subgraph cfg["📄 config layers"]
      d["defaults: replicas 2"] --> p["production: replicas 3"] --> sv["exam-results: feature_new_grades on"]
    end

🗺️ काढलेली आवृत्ती + एक lab: https://school-edh.pages.dev/platform-engineering/lesson-diagrams.html#l08

❓ काय

Secret — प्रवेश देणारे value: password, API key, private key, token. ते Git मध्ये, logs मध्ये किंवा container image मध्ये असता कामा नये.
Secret store / secrets manager — secrets encrypt करून ठेवणारी, ती कोण वाचू शकते हे नियंत्रित करणारी आणि प्रत्येक access ची नोंद ठेवणारी service. उदाहरणे: HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, Azure Key Vault.
Path-based access — secrets path नुसार मांडलेले असतात (teams/science/...) आणि एक policy प्रत्येक team ला फक्त तिच्या स्वतःच्या subtree वर read access देते. प्रत्येक team साठी least privilege.
Versioning — प्रत्येक write जुने value आधीच्या version म्हणून जपते (Vault चे KV version 2 आणि AWS Secrets Manager दोन्ही versions ठेवतात), त्यामुळे चुकीचा बदल मागे घेता येतो.
Rotation — secret ठरलेल्या वेळापत्रकानुसार किंवा leak नंतर बदलणे. सुरक्षित क्रम: आधी नवे value बनवा, प्रत्येक consumer ला त्यावर जाऊ द्या, मग जुने revoke करा. आधी revoke केले तर जुने value वापरणारे सगळे बंद पडतात.
Platform मध्ये sync करणे — अनेक teams code मधून store ला थेट call करत नाहीत. External Secrets Operator सारखे tool value ला Kubernetes Secret मध्ये copy करते आणि ठराविक अंतराने ते refresh करते.
Layered configuration — secret नसलेल्या settings (log level, replica count, feature flags) layers मधून बनतात, जिथे नंतरचा layer जिंकतो: organisation defaults → environment → service. Helm values files आणि Kustomize overlays याच पद्धतीने काम करतात.

🤔 का

कारण हाताने सांभाळलेले secrets leak होतात: chat मध्ये paste केले जातात, "थोड्या वेळासाठी" commit केले जातात, laptop वर copy करून विसरले जातात. आणि हाताने केलेले rotation क्वचितच होते, कारण ते भीतीदायक असते — जुने value कोण वापरते हे कोणालाच माहीत नसते. प्रत्येक team साठी store, versions आणि consumers ची यादी यांमुळे rotation नेहमीचे काम बनते. "कोणी ठरवले" हे उत्तर देणाऱ्या layered config मुळे "production मध्ये हे 3 का आहे?" हा प्रश्न तपासाचा विषय न राहता फक्त पाहण्याचा विषय होतो.

🔧 कसे (या repo मध्ये)

idp/selfserve.py मधले SecretStore प्रत्येक path साठी values ची यादी ठेवते. get(team, path, consumer) teams/<team>/ च्या बाहेरचा कोणताही path नाकारते, फक्त version आणि length परत देते (value कधीच छापले जात नाही), आणि प्रत्येक consumer कडे कोणते version आहे याची नोंद ठेवते. rotate() नवे version जोडते; stale() सर्वात नव्या version पेक्षा जुने version धरून ठेवलेल्या consumers ची यादी देते. layered_config(*layers) (name, dict) layers डावीकडून उजवीकडे merge करते आणि प्रत्येक key साठी value आणि ते ठरवणारा layer जपून ठेवते.

🧪 करून पाहा

python3 idp/demo.py secrets
python3 - <<'EOF'
import sys; sys.path.insert(0, "idp"); from selfserve import SecretStore, layered_config
st = SecretStore()
st.put("teams/library/search-token", "tok-one-for-the-search")
print(st.get("library", "teams/library/search-token", consumer="library-search"))
print(st.get("library", "teams/library/missing"))
print(st.get("maths", "teams/library/search-token"))
st.rotate("teams/library/search-token", "tok-two"); st.rotate("teams/library/search-token", "tok-three")
print("after two rotations, stale:", st.stale(), "·", st.get("library", "teams/library/search-token", consumer="library-search")[1])
cfg = layered_config(("defaults", {"log_level": "info", "replicas": 2}), ("preview", {"replicas": 1, "log_level": "debug"}),
                     ("library-search", {"log_level": "warning"}))
for k, (v, src) in cfg.items(): print(f"{k} = {v} (from {src})")
EOF

✅ तपासा — तुम्हाला काय दिसायला हवे

secrets हे छापते:

   science reads teams/science/results-db-password → ok · teams/science/results-db-password v1 (22 characters, value hidden)
   science reads teams/maths/timetable-api-key     → denied · science may read teams/science/* only — ask the owner of teams/maths/timetable-api-key to share it
── rotation: the password is now v2; consumers still on the old version: ['exam-results', 'lab-worker']
   exam-results reloads → still stale: ['lab-worker'] · keep v1 valid until nobody uses it, then revoke
   replicas            = 3     (from production)

तुमचा snippet हे छापतो:

('ok', 'teams/library/search-token v1 (22 characters, value hidden)')
('missing', 'no secret at teams/library/missing')
('denied', 'maths may read teams/maths/* only — ask the owner of teams/library/search-token to share it')
after two rotations, stale: ['library-search'] · teams/library/search-token v3 (9 characters, value hidden)
log_level = warning (from library-search)
replicas = 1 (from preview)

🏁 तुम्ही आत्ताच काय सिद्ध केले

Store ने तीन वेगवेगळी उत्तरे दिली — ok, missing, denied — आणि कधीच value छापले नाही. दोन rotations मुळे v3 बनले; library-search पुन्हा वाचेपर्यंत stale होते, आणि v3 आणणाऱ्या त्याच call ने ते साफ केले. Config मध्ये log_level तीन वेळा ठरवले गेले आणि शेवटचा layer जिंकला; replicas preview मधून आले, आणि output तसे सांगतो. "Stale consumers" ची यादीच जुने value revoke करणे सुरक्षित बनवते.

📏 शिकवण्यासाठीचा नमुना: खरा store values encrypt करतो, प्रत्येक read ची नोंद ठेवतो आणि शक्य तिथे थोड्या काळाचे credentials देतो. हा lab फक्त paths, versions आणि कोणाकडे कोणते version आहे एवढेच दाखवतो.

⚠️ नेहमीच्या चुका

Kubernetes Secret encrypted असतो असे समजणे — default मध्ये त्याचा data फक्त base64-encoded असतो; encryption at rest चालू करावे लागते, आणि RBAC ने तो कोण वाचू शकते हे मर्यादित करावे लागते
प्रत्येक consumer कडे नवे secret येण्याआधीच जुने revoke करणे
प्रत्येक team वाचू शकेल असा एकच सामायिक "platform" secret path
CI logs किंवा error messages मध्ये secrets छापणे
environments मध्ये वेगळे असलेले config values, आणि ते का वेगळे आहेत याची कोणतीही नोंद नसणे

🏭 प्रत्यक्ष वापरात

HashiCorp Vault (KV version 2), खऱ्या server वर:

vault kv put secret/teams/science/results-db-password value='…'
vault kv get -version=1 secret/teams/science/results-db-password     # an older version

Science ला फक्त तिचाच subtree वाचू देणारी policy (KV v2 API paths मध्ये जोडतो तो data/ भाग लक्षात घ्या):

path "secret/data/teams/science/*" {
  capabilities = ["read"]
}

External Secrets Operator Kubernetes Secret ला store शी sync ठेवतो (जुन्या releases मध्ये apiVersion: external-secrets.io/v1beta1 वापरतात):

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: results-db
  namespace: science
spec:
  refreshInterval: 1h
  secretStoreRef: {name: vault-science, kind: SecretStore}
  target: {name: results-db}
  data:
    - secretKey: password
      remoteRef: {key: teams/science/results-db-password, property: value}

AWS Secrets Manager rotation function वापरून काही database secrets ठरलेल्या वेळापत्रकाने rotate करू शकतो, यात "alternating users" strategy पण आहे जी एक credentials संच चालू ठेवते आणि दुसरा बदलते.

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: गरज पडण्याआधीच rotation चा सराव करा. Secret कोण वापरते हे कोणालाच सांगता आले नाही, तर पहिला खरा leak हा outage पण ठरतो.

⏭️ पुढे

भाग 2 पूर्ण झाला: teams स्वतःची मदत स्वतः करू शकतात. भाग 3 विचारतो की 'नाही' ची भिंत न बनता हे सुरक्षित कसे ठेवायचे — दुरुस्ती समजावून सांगणारे guardrails.

git checkout lesson-09-policy-guardrails

🔑 Lesson 08 — Secrets & config as a service: the key cabinet

📍 You are here: Lesson 08 of 12 · Previous: lesson-07-paved-pipelines · Next: lesson-09-policy-guardrails

📦 What's in this branch

Lessons 01–07, plus secrets and configuration as a service. Secrets live in one store, under a path per team; a team can read only its own; every change makes a new version; after a rotation the store shows which consumers still hold the old one. Configuration comes in layers — defaults, then environment, then service — and every final value says which layer set it. SecretStore and layered_config() in idp/selfserve.py, secrets() in idp/demo.py.

🧒 Explain like I'm 5

The facilities office has a key cabinet. 🔑 Each department has its own hook: science keys on the science hook, maths keys on the maths hook.

Katrina asks for the science lab key: ✅ "Here — key version 1." She asks for the maths store-room key: ❌ "That is on the maths hook. Ask maths to share it."

Now and then Dipika changes a lock (a rotation) and hangs up key version 2. She keeps a list of who took the old key. The exam-results clerk comes back for the new one ✅. The lab worker has not come back yet ⚠️. So Dipika does not throw the old lock away until the lab worker has the new key too.

And the office keeps one settings sheet per room: the school's default ("lights at normal"), then the building's rule ("the exam hall has 3 cleaners, not 2"), then the room's own wish ("new grades board: on"). Anyone can read the sheet and see which rule set each line.

🗺️ Diagram

flowchart LR
    subgraph store["🔑 secret store"]
      s1["teams/science/results-db-password<br/>v1 → v2 (rotated)"]
      s2["teams/maths/timetable-api-key"]
    end
    sci["🧪 science"] -->|"✅ own path"| s1
    sci -.->|"❌ denied"| s2
    s1 --> er["exam-results: reloaded v2"]
    s1 --> lw["lab-worker: still on v1 ⚠️"]
    subgraph cfg["📄 config layers"]
      d["defaults: replicas 2"] --> p["production: replicas 3"] --> sv["exam-results: feature_new_grades on"]
    end

🗺️ Drawn version + a lab: https://school-edh.pages.dev/platform-engineering/lesson-diagrams.html#l08

❓ What

Secret — a value that grants access: a password, an API key, a private key, a token. It must not be in Git, in logs, or in a container image.
Secret store / secrets manager — a service that keeps secrets encrypted, controls who may read them, and records access. Examples: HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, Azure Key Vault.
Path-based access — secrets are organised by path (teams/science/...) and a policy grants each team read access to its own subtree. Least privilege, per team.
Versioning — each write keeps the old value as an earlier version (Vault's KV version 2 and AWS Secrets Manager both keep versions), so a bad change can be undone.
Rotation — replacing a secret on a schedule or after a leak. The safe order: create the new value, let every consumer switch, then revoke the old one. Revoking first breaks whoever is still on the old value.
Syncing into the platform — many teams do not call the store from code. A tool such as the External Secrets Operator copies the value into a Kubernetes Secret and refreshes it on an interval.
Layered configuration — non-secret settings (log level, replica count, feature flags) built from layers where later layers win: organisation defaults → environment → service. Helm values files and Kustomize overlays work this way.

🤔 Why

Because secrets handled by hand leak: pasted into chat, committed "for a moment", copied to a laptop and forgotten. And rotation done by hand is done rarely, because it is scary — nobody knows who uses the old value. A store per team, with versions and a list of consumers, makes rotation routine. Layered config with a "set by" answer makes the question "why is this 3 in production?" a lookup instead of an investigation.

🔧 How (in this repo)

SecretStore in idp/selfserve.py keeps a list of values per path. get(team, path, consumer) refuses any path outside teams/<team>/, returns only the version and length (the value is never printed), and records which version each consumer holds. rotate() appends a new version; stale() lists consumers holding an older version than the newest. layered_config(*layers) merges (name, dict) layers left to right and keeps, for every key, the value and the layer that set it.

🧪 Try it

python3 idp/demo.py secrets
python3 - <<'EOF'
import sys; sys.path.insert(0, "idp"); from selfserve import SecretStore, layered_config
st = SecretStore()
st.put("teams/library/search-token", "tok-one-for-the-search")
print(st.get("library", "teams/library/search-token", consumer="library-search"))
print(st.get("library", "teams/library/missing"))
print(st.get("maths", "teams/library/search-token"))
st.rotate("teams/library/search-token", "tok-two"); st.rotate("teams/library/search-token", "tok-three")
print("after two rotations, stale:", st.stale(), "·", st.get("library", "teams/library/search-token", consumer="library-search")[1])
cfg = layered_config(("defaults", {"log_level": "info", "replicas": 2}), ("preview", {"replicas": 1, "log_level": "debug"}),
                     ("library-search", {"log_level": "warning"}))
for k, (v, src) in cfg.items(): print(f"{k} = {v} (from {src})")
EOF

✅ Verify — what you should see

secrets prints:

   science reads teams/science/results-db-password → ok · teams/science/results-db-password v1 (22 characters, value hidden)
   science reads teams/maths/timetable-api-key     → denied · science may read teams/science/* only — ask the owner of teams/maths/timetable-api-key to share it
── rotation: the password is now v2; consumers still on the old version: ['exam-results', 'lab-worker']
   exam-results reloads → still stale: ['lab-worker'] · keep v1 valid until nobody uses it, then revoke
   replicas            = 3     (from production)

Your snippet prints:

('ok', 'teams/library/search-token v1 (22 characters, value hidden)')
('missing', 'no secret at teams/library/missing')
('denied', 'maths may read teams/maths/* only — ask the owner of teams/library/search-token to share it')
after two rotations, stale: ['library-search'] · teams/library/search-token v3 (9 characters, value hidden)
log_level = warning (from library-search)
replicas = 1 (from preview)

🏁 What you just proved

The store answered three different ways — ok, missing, denied — and never printed a value. Two rotations made v3; library-search was stale until it read again, and the same call that fetched v3 cleared it. In the config, log_level was set three times and the last layer won; replicas came from preview, and the output says so. The "stale consumers" list is what makes it safe to revoke the old value.

📏 Teaching model: a real store encrypts values, logs every read and hands out short-lived credentials where it can. This lab only models paths, versions and who holds which version.

⚠️ Common mistakes

thinking a Kubernetes Secret is encrypted — by default its data is only base64-encoded; encryption at rest must be turned on, and RBAC must limit who can read it
revoking the old secret before every consumer has the new one
one shared "platform" secret path that every team can read
printing secrets in CI logs or error messages
config values that differ between environments with no record of why

🏭 In production

HashiCorp Vault (KV version 2), on a real server:

vault kv put secret/teams/science/results-db-password value='…'
vault kv get -version=1 secret/teams/science/results-db-password     # an older version

A policy that lets science read only its own subtree (note the data/ segment KV v2 adds to API paths):

path "secret/data/teams/science/*" {
  capabilities = ["read"]
}

The External Secrets Operator keeps a Kubernetes Secret in sync with the store (older releases use apiVersion: external-secrets.io/v1beta1):

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: results-db
  namespace: science
spec:
  refreshInterval: 1h
  secretStoreRef: {name: vault-science, kind: SecretStore}
  target: {name: results-db}
  data:
    - secretKey: password
      remoteRef: {key: teams/science/results-db-password, property: value}

AWS Secrets Manager can rotate some database secrets on a schedule with a rotation function, including an "alternating users" strategy that keeps one set of credentials valid while the other is changed.

🏭 Why this matters in production: run a rotation drill before you need one. If nobody can tell who uses a secret, the first real leak becomes an outage too.

⏭️ Next

Part 2 is done: teams can help themselves. Part 3 asks how to keep that safe without becoming a wall of "no" — guardrails that explain the fix.

git checkout lesson-09-policy-guardrails

← Previouspaved pipelines Next →policy guardrails

This page is the lesson's README from the lesson-08-secrets-config branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.