🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🔭 धडा 06 — OpenTelemetry: सगळे गोळा करण्याचा एकच मार्ग

📍 तुम्ही इथे आहात: 12 पैकी धडा 06 · मागे: lesson-05-tracing · पुढे: lesson-07-prometheus-grafana

📦 या ब्रँचमध्ये काय आहे

धडे 01–05, आणि तिन्ही signals तयार करण्याची आणि पुढे नेण्याची standard पद्धत: OpenTelemetry (OTel). तुम्ही API आणि SDK, OTLP protocol, Collector (receivers → processors → exporters), head विरुद्ध tail sampling, आणि semantic conventions शिकता. obs/signals.py मधील Collector class tail sampling चे model आहे; obs/demo.py मधील otel() 1,000 traces त्यामधून चालवते.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

पूर्वी शाळेतली प्रत्येक खोली आपापल्या पद्धतीने नोंदी ठेवायची. Office पेन्सिलने लिहायचे, ग्रंथालय cards वापरायचे, शिक्षक कक्ष whiteboard वापरायचा. कतरिनाला त्यांची बेरीज करता येत नव्हती.

म्हणून दीपिका प्रत्येक खोलीला एकच form 📋 देते, त्याच चौकटी आणि तेच शब्द असलेला: "वर्ग", "आत येण्याची वेळ", "काय घडले". प्रत्येक खोली आपले forms व्हरांड्यातल्या एकाच पत्रपेटीत 📮 टाकते. एक मदतनीस पत्रपेटी रिकामी करते: ती forms चे गठ्ठे बांधते, बहुतेक कंटाळवाणे forms टाकून देते ("विद्यार्थी ठीक, 2 मिनिटे"), आजारी असलेल्या किंवा खूप वेळ वाट पाहिलेल्या विद्यार्थ्याबद्दलचा प्रत्येक form ठेवते, आणि गठ्ठा जिथे गरज आहे तिथे पाठवते — आरोग्य कक्ष, मुख्याध्यापिकेचे office, किंवा दोन्हीकडे.

तोच एक form म्हणजे semantic conventions. पत्रपेटी आणि मदतनीस म्हणजे Collector. Forms वाचल्यानंतर रोचक forms ठेवणे म्हणजे tail sampling.

🗺️ आकृती

flowchart LR
    subgraph app["🍳 results-api"]
      api["OTel API<br/>(your code calls this)"] --> sdk["OTel SDK<br/>resource: service.name<br/>batch + export"]
    end
    sdk -->|"OTLP<br/>gRPC 4317 · HTTP 4318"| rx
    subgraph col["📮 OpenTelemetry Collector"]
      rx["receivers<br/>otlp, prometheus"] --> pr["processors<br/>memory_limiter → tail_sampling → batch"] --> exp["exporters<br/>otlp, prometheusremotewrite"]
    end
    exp --> t["🗺️ Tempo / Jaeger / X-Ray"]
    exp --> m["📊 Prometheus / Mimir"]
    exp --> dd["🐶 Datadog"]
    s["1,000 traces → keep 113<br/>all 4 errors · all 10 slow · 1 in 10 of the rest"]

🗺️ काढलेली आवृत्ती + एक lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l06

❓ काय

OpenTelemetry (OTel) — traces, metrics आणि logs तयार करण्यासाठी आणि पुढे नेण्यासाठी एक open-source, कोणत्याही vendor ला न बांधलेले standard (एक CNCF project). तुम्ही एकदाच instrument करता; backend हा फक्त configuration चा निर्णय असतो.
API — तुमचा code आणि libraries ज्यांना call करतात ती interfaces (tracer.start_as_current_span, meter.create_counter). SDK install नसेल तर API काहीच करत नाही — म्हणून libraries ते सुरक्षितपणे समाविष्ट करू शकतात.
SDK — तुमच्या process मधली अंमलबजावणी: ते spans आणि metric points बनवते, sampling लागू करते, resource जोडते (कोण पाठवत आहे: service.name, service.version, deployment.environment), batch करते आणि export करते.
Auto-instrumentation — नेहमीच्या frameworks आणि clients साठी तयार instrumentation (HTTP servers, HTTP clients, database drivers), सहसा code न बदलता चालू करता येते.
OTLP — OpenTelemetry Protocol, SDKs, Collectors आणि backends यांच्यातला wire format. Default ports: 4317 (gRPC) आणि 4318 (HTTP).
Collector — telemetry घेणारी, त्यावर प्रक्रिया करणारी आणि export करणारी एक वेगळी process. ती pipelines पासून बनलेली असते (प्रत्येक signal साठी एक: traces, metrics, logs), प्रत्येकात:
- receivers — data आत कसा येतो (otlp, prometheus, filelog…);
- processors — त्याचे काय होते (memory_limiter, batch, fields जोडण्यासाठी किंवा काढण्यासाठी attributes, tail_sampling…);
- exporters — तो कुठे जातो (otlp, otlphttp, prometheusremotewrite, vendor exporters). ती agent म्हणून (प्रत्येक app किंवा node शेजारी) आणि/किंवा gateway म्हणून (एक मध्यवर्ती pool) चालते.
Head sampling — trace record करायचा की नाही हे सुरुवातीलाच, पहिल्या service मध्ये ठरवणे (उदाहरणार्थ "trace id नुसार 10% ठेवा"); हा निर्णय traceparent flags मधून पुढे जातो. स्वस्त आणि सोपे, पण trace अयशस्वी होईल की हळू होईल हे कोणालाही कळण्याआधीच ते ठरवते.
Tail sampling — trace पूर्ण झाल्यानंतर, Collector मध्ये ठरवणे: प्रत्येक error, प्रत्येक हळू trace, आणि बाकीच्यांचा एक नमुना ठेवा. त्यासाठी एका trace चे सगळे spans एकाच Collector वर लागतात (trace id नुसार load-balancing करणारा layer वापरा) आणि traces संपेपर्यंत ठेवण्यासाठी memory लागते.
Semantic conventions — attributes ची सामायिक नावे: service.name, http.request.method, http.response.status_code, http.route, url.path, db.system… प्रत्येक service आणि प्रत्येक tool तीच नावे वापरते, म्हणून एक query सगळीकडे चालते. (जुन्या आवृत्त्या http.method आणि http.status_code वापरत; stable HTTP conventions नी त्यांची नावे बदलली.)

🤔 का

कारण standard नसताना, प्रत्येक vendor चा स्वतःचा agent आणि स्वतःची library होती. Backend बदलणे म्हणजे प्रत्येक service पुन्हा instrument करणे. OTel सह, app Collector शी OTLP मध्ये बोलते, आणि Collector मध्येच तुम्ही backends निवडता, गुपिते काढून टाकता, आणि खर्च नियंत्रित करता — मुख्यतः sampling ने. व्यस्त मार्गावरच्या प्रत्येक request चा trace घेणे महाग असते; यादृच्छिक 10% चा trace घेतला तर तुम्हाला हवा असलेला एकमेव error trace गळू शकतो. Tail sampling रोचक traces ठेवते.

🔧 कसे (या repo मध्ये)

obs/signals.py मधील Collector(slow_ms=500, keep_1_in=10) हा एक tail sampler आहे: export(traces) प्रत्येक पूर्ण झालेला trace पाहतो आणि त्यात error असेल, किंवा तो slow_ms पेक्षा हळू असेल, किंवा तो दर keep_1_in-वा trace असेल, तर तो ठेवतो; बाकीचे टाकले जातात. तो (kept, dropped) परत करतो. otel() 1,000 traces बनवते: बहुतेकांना 90–389 ms लागतात, दर 97 वा हळू असतो (+900 ms) आणि दर 250 व्यात error असते.

🧪 करून पाहा

Tail-sampling चा नियम बदला, मग head sampling शी तुलना करा (सुरुवातीलाच, काहीही न कळता 10 पैकी 1 ठरवणे):

python3 obs/demo.py otel
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from signals import Collector
traces = [dict(id=i, ms=90 + (i * 53) % 300 + (900 if i % 97 == 0 else 0), error=(i % 250 == 0)) for i in range(1, 1001)]
for slow_ms, k in ((500, 10), (500, 100), (300, 10), (1000, 1000)):
    kept, dropped = Collector(slow_ms=slow_ms, keep_1_in=k).export(traces)
    print(f"keep errors + traces > {slow_ms:>4} ms + 1 in {k:<4} → kept {kept:>3}, dropped {dropped}")
head = [t for t in traces if t["id"] % 10 == 3]
print(f"head sampling 1 in 10 (decided at the start) → kept {len(head)} · errors kept {sum(t['error'] for t in head)} of 4 · slow kept {sum(t['ms'] > 500 for t in head)} of 10")
EOF
python3 obs/test_obs.py

✅ तपासा — तुम्हाला काय दिसायला हवे

otel हे print करते:

── OpenTelemetry: one SDK in the app → the Collector → any backend (Tempo, Jaeger, X-Ray, Datadog…)
   app (SDK: traces, metrics, logs) → OTLP → collector (receive → batch → sample → export) → backends
── 1,000 traces, tail sampling (keep every error and every trace > 500 ms, 1 in 10 of the rest) → kept 113, dropped 887
   all 4 error traces and all 10 slow traces survive — the boring ones are sampled
   semantic conventions: http.request.method, http.response.status_code, service.name — the same names everywhere

तुमचा snippet हे print करतो:

keep errors + traces >  500 ms + 1 in 10   → kept 113, dropped 887
keep errors + traces >  500 ms + 1 in 100  → kept  24, dropped 976
keep errors + traces >  300 ms + 1 in 10   → kept 377, dropped 623
keep errors + traces > 1000 ms + 1 in 1000 → kept  15, dropped 985
head sampling 1 in 10 (decided at the start) → kept 100 · errors kept 0 of 4 · slow kept 1 of 10

Tests मध्ये ✅ L06 tail sampling keeps every error and every slow trace असते.

🏁 तुम्ही आत्ताच काय सिद्ध केले

100 पैकी 1 वरच्या tail sampling ने 1,000 ऐवजी 24 traces साठवले — आणि तरीही प्रत्येक error आणि प्रत्येक हळू trace ठेवला. "हळू" ची रेषा 300 ms पर्यंत खाली आणल्याने storage तिप्पट झाले (377), कारण अनेक सामान्य traces 300 ms ओलांडतात: मर्यादा हा खर्चाचा निर्णय आहे. Head sampling ने पहिल्या नियमाइतकीच संख्या ठेवली (100 विरुद्ध 113) पण 4 पैकी एकही error नाही आणि 10 पैकी फक्त 1 हळू trace ठेवला — त्याने कळण्याआधीच निवड केली.

⚠️ नेहमीच्या चुका

कमी दराने head sampling करणे आणि मग error trace का गायब आहे याचे आश्चर्य वाटणे
trace id नुसार routing न करता अनेक Collectors वर tail sampling — प्रत्येकाला अर्धाच trace दिसतो
Collector मध्ये memory_limiter नाही — traffic चा एक spike out-of-memory crash बनतो
semantic conventions वापरण्याऐवजी प्रत्येक team ने स्वतःची attribute names शोधून काढणे
service.name resource attribute नाही — प्रत्येक span unknown_service म्हणून दिसतो
प्रत्येक app मधून थेट vendor कडे export करणे, त्यामुळे backend बदलणे म्हणजे प्रत्येक app पुन्हा deploy करणे

🏭 प्रत्यक्ष वापरात

On a real account — Collector कडे OTLP exporter सह Python SDK (packages opentelemetry-sdk आणि opentelemetry-exporter-otlp):

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.trace.sampling import ParentBased, ALWAYS_ON
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider(
    resource=Resource.create({"service.name": "results-api", "service.version": "v41",
                              "deployment.environment": "production"}),
    sampler=ParentBased(ALWAYS_ON),        # send everything; the Collector tail-samples
)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317", insecure=True)))
trace.set_tracer_provider(provider)

किंवा code न बदलता (auto-instrumentation):

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
OTEL_SERVICE_NAME=results-api OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
    opentelemetry-instrument python app.py

Tail sampling असलेली Collector pipeline — errors ठेवा, हळू traces ठेवा, बाकीचे 10% ठेवा (tail_sampling Collector च्या contrib distribution मध्ये आहे):

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  tail_sampling:
    decision_wait: 10s              # wait for late spans before deciding
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow
        type: latency
        latency: {threshold_ms: 500}
      - name: the-rest
        type: probabilistic
        probabilistic: {sampling_percentage: 10}
  batch: {}

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]

AWS वर, ADOT Collector (AWS Distro for OpenTelemetry) X-Ray आणि CloudWatch साठी exporters जोडतो; Datadog Agent आणि Datadog चा Collector exporter सुद्धा OTLP स्वीकारतात.

🏭 Production मध्ये हे का महत्त्वाचे: पहिल्या दिवसापासूनच तुमचे apps आणि प्रत्येक backend यांच्यामध्ये एक Collector ठेवा. तिथेच तुम्ही गुपिते काढता, sample करता, आणि vendors बदलता — एकाही service ला हात न लावता.

⏭️ पुढे

Signals वाहू लागले आहेत. आता metrics साठवा, त्यांना प्रश्न विचारा आणि त्यांचे चित्र काढा: Prometheus आणि Grafana.

git checkout lesson-07-prometheus-grafana

🔭 Lesson 06 — OpenTelemetry: one way to collect everything

📍 You are here: Lesson 06 of 12 · Previous: lesson-05-tracing · Next: lesson-07-prometheus-grafana

📦 What's in this branch

Lessons 01–05, plus the standard way to produce and move all three signals: OpenTelemetry (OTel). You learn the API and SDK, the OTLP protocol, the Collector (receivers → processors → exporters), head vs tail sampling, and semantic conventions. The Collector class in obs/signals.py models tail sampling; otel() in obs/demo.py runs 1,000 traces through it.

🧒 Explain like I'm 5

Every room in the school used to keep notes its own way. The office wrote in pencil, the library used cards, the staff room used a whiteboard. Katrina could not add them up.

So Dipika gives every room the same form 📋 with the same boxes, in the same words: "class", "time in", "what happened". Every room drops its forms in one post box 📮 in the corridor. A helper empties the post box: she bundles the forms, throws away most of the boring ones ("pupil fine, 2 minutes"), keeps every form about a pupil who was ill or waited a long time, and sends the bundle wherever it is needed — the health room, the head's office, or both.

The same form is semantic conventions. The post box and helper are the Collector. Keeping the interesting forms after reading them is tail sampling.

🗺️ Diagram

flowchart LR
    subgraph app["🍳 results-api"]
      api["OTel API<br/>(your code calls this)"] --> sdk["OTel SDK<br/>resource: service.name<br/>batch + export"]
    end
    sdk -->|"OTLP<br/>gRPC 4317 · HTTP 4318"| rx
    subgraph col["📮 OpenTelemetry Collector"]
      rx["receivers<br/>otlp, prometheus"] --> pr["processors<br/>memory_limiter → tail_sampling → batch"] --> exp["exporters<br/>otlp, prometheusremotewrite"]
    end
    exp --> t["🗺️ Tempo / Jaeger / X-Ray"]
    exp --> m["📊 Prometheus / Mimir"]
    exp --> dd["🐶 Datadog"]
    s["1,000 traces → keep 113<br/>all 4 errors · all 10 slow · 1 in 10 of the rest"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/observability/lesson-diagrams.html#l06

❓ What

OpenTelemetry (OTel) — an open-source, vendor-neutral standard (a CNCF project) for producing and moving traces, metrics and logs. You instrument once; the backend is a configuration choice.
API — the interfaces your code and libraries call (tracer.start_as_current_span, meter.create_counter). With no SDK installed, the API does nothing — so libraries can include it safely.
SDK — the implementation inside your process: it creates spans and metric points, applies sampling, adds the resource (who is sending: service.name, service.version, deployment.environment), batches and exports.
Auto-instrumentation — ready-made instrumentation for common frameworks and clients (HTTP servers, HTTP clients, database drivers), usually switched on without code changes.
OTLP — the OpenTelemetry Protocol, the wire format between SDKs, Collectors and backends. Default ports: 4317 (gRPC) and 4318 (HTTP).
Collector — a separate process that receives, processes and exports telemetry. It is built from pipelines (one per signal: traces, metrics, logs), each made of:
- receivers — how data gets in (otlp, prometheus, filelog…);
- processors — what happens to it (memory_limiter, batch, attributes to add or remove fields, tail_sampling…);
- exporters — where it goes (otlp, otlphttp, prometheusremotewrite, vendor exporters). It runs as an agent (next to each app or node) and/or a gateway (a central pool).
Head sampling — decide whether to record a trace at the start, in the first service (for example "keep 10% by trace id"); the decision travels in the traceparent flags. Cheap and simple, but it decides before anyone knows whether the trace will fail or be slow.
Tail sampling — decide after the trace is complete, in the Collector: keep every error, every slow trace, and a sample of the rest. It needs all spans of one trace on the same Collector (use a load-balancing layer keyed by trace id) and memory to hold traces until they finish.
Semantic conventions — the shared names for attributes: service.name, http.request.method, http.response.status_code, http.route, url.path, db.system… Every service and every tool uses the same names, so one query works everywhere. (Older versions used http.method and http.status_code; the stable HTTP conventions renamed them.)

🤔 Why

Because without a standard, each vendor had its own agent and its own library. Changing backend meant re-instrumenting every service. With OTel, the app speaks OTLP to a Collector, and the Collector is where you choose backends, drop secrets, and control cost — mostly by sampling. Tracing every request on a busy path is expensive; tracing a random 10% can drop the one error trace you needed. Tail sampling keeps the interesting ones.

🔧 How (in this repo)

Collector(slow_ms=500, keep_1_in=10) in obs/signals.py is a tail sampler: export(traces) looks at each finished trace and keeps it if it has an error, or is slower than slow_ms, or is every keep_1_in-th trace; the rest are dropped. It returns (kept, dropped). otel() makes 1,000 traces: most take 90–389 ms, every 97th is slow (+900 ms) and every 250th has an error.

🧪 Try it

Change the tail-sampling rule, then compare with head sampling (decide 1 in 10 at the start, knowing nothing):

python3 obs/demo.py otel
python3 - <<'EOF'
import sys; sys.path.insert(0, "obs"); from signals import Collector
traces = [dict(id=i, ms=90 + (i * 53) % 300 + (900 if i % 97 == 0 else 0), error=(i % 250 == 0)) for i in range(1, 1001)]
for slow_ms, k in ((500, 10), (500, 100), (300, 10), (1000, 1000)):
    kept, dropped = Collector(slow_ms=slow_ms, keep_1_in=k).export(traces)
    print(f"keep errors + traces > {slow_ms:>4} ms + 1 in {k:<4} → kept {kept:>3}, dropped {dropped}")
head = [t for t in traces if t["id"] % 10 == 3]
print(f"head sampling 1 in 10 (decided at the start) → kept {len(head)} · errors kept {sum(t['error'] for t in head)} of 4 · slow kept {sum(t['ms'] > 500 for t in head)} of 10")
EOF
python3 obs/test_obs.py

✅ Verify — what you should see

otel prints:

── OpenTelemetry: one SDK in the app → the Collector → any backend (Tempo, Jaeger, X-Ray, Datadog…)
   app (SDK: traces, metrics, logs) → OTLP → collector (receive → batch → sample → export) → backends
── 1,000 traces, tail sampling (keep every error and every trace > 500 ms, 1 in 10 of the rest) → kept 113, dropped 887
   all 4 error traces and all 10 slow traces survive — the boring ones are sampled
   semantic conventions: http.request.method, http.response.status_code, service.name — the same names everywhere

Your snippet prints:

keep errors + traces >  500 ms + 1 in 10   → kept 113, dropped 887
keep errors + traces >  500 ms + 1 in 100  → kept  24, dropped 976
keep errors + traces >  300 ms + 1 in 10   → kept 377, dropped 623
keep errors + traces > 1000 ms + 1 in 1000 → kept  15, dropped 985
head sampling 1 in 10 (decided at the start) → kept 100 · errors kept 0 of 4 · slow kept 1 of 10

The tests include ✅ L06 tail sampling keeps every error and every slow trace.

🏁 What you just proved

Tail sampling at 1 in 100 stored 24 traces instead of 1,000 — and still kept every error and every slow one. Lowering the "slow" line to 300 ms tripled the storage (377), because many normal traces cross 300 ms: the threshold is a cost decision. Head sampling kept the same number as the first rule (100 vs 113) but none of the 4 errors and only 1 of the 10 slow traces — it chose before it knew.

⚠️ Common mistakes

head sampling at a low rate and then wondering why the error trace is missing
tail sampling on several Collectors without routing by trace id — each sees half a trace
no memory_limiter in the Collector — a traffic spike becomes an out-of-memory crash
every team inventing its own attribute names instead of using the semantic conventions
no service.name resource attribute — every span shows up as unknown_service
exporting straight from every app to a vendor, so changing backend means redeploying every app

🏭 In production

On a real account — the Python SDK with an OTLP exporter to a Collector (packages opentelemetry-sdk and opentelemetry-exporter-otlp):

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.trace.sampling import ParentBased, ALWAYS_ON
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider(
    resource=Resource.create({"service.name": "results-api", "service.version": "v41",
                              "deployment.environment": "production"}),
    sampler=ParentBased(ALWAYS_ON),        # send everything; the Collector tail-samples
)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317", insecure=True)))
trace.set_tracer_provider(provider)

Or with no code changes (auto-instrumentation):

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
OTEL_SERVICE_NAME=results-api OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
    opentelemetry-instrument python app.py

A Collector pipeline with tail sampling — keep errors, keep slow traces, keep 10% of the rest (tail_sampling is in the Collector contrib distribution):

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  tail_sampling:
    decision_wait: 10s              # wait for late spans before deciding
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow
        type: latency
        latency: {threshold_ms: 500}
      - name: the-rest
        type: probabilistic
        probabilistic: {sampling_percentage: 10}
  batch: {}

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]

On AWS, the ADOT Collector (AWS Distro for OpenTelemetry) adds exporters for X-Ray and CloudWatch; the Datadog Agent and Datadog's Collector exporter accept OTLP too.

🏭 Why this matters in production: put a Collector between your apps and every backend from day one. It is where you drop secrets, sample, and switch vendors — without touching a single service.

⏭️ Next

The signals are flowing. Now store the metrics, ask them questions and draw them: Prometheus and Grafana.

git checkout lesson-07-prometheus-grafana

← Previoustracing Next →prometheus grafana

This page is the lesson's README from the lesson-06-opentelemetry branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.