🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

📈 धडा 11 — Monitoring: dashboard आणि stopwatch

📍 तुम्ही इथे आहात: 12 पैकी धडा 11 · मागे: lesson-10-custom-domains · पुढे: lesson-12-private-waf-limits

📦 या ब्रँचमध्ये काय आहे

धडे 01–10, आणि office च्या आत कसे पाहायचे: CloudWatch metrics (Count, 4XXError, 5XXError, Latency, IntegrationLatency, CacheHitCount, CacheMissCount), $context variables असलेले access logs, आणि X-Ray traces. मुख्य कौशल्य: वेळ कुठे जातो ते शोधण्यासाठी Latency विरुद्ध IntegrationLatency वाचणे. apigw/demo.py मधले monitor() सहा requests पाठवते आणि dashboard व visitor book छापते.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

Front office च्या भिंतीवर एक dashboard 📊 आहे: आज किती visitors आले, किती जणांना त्यांच्याच चुकीमुळे परत पाठवले (4XX), किती जणांची kitchen मुळे निराशा झाली (5XX), किती उत्तरे सूचना फलकावरून आली.

Clerk कडे दोन stopwatches ⏱️ सुद्धा आहेत. पहिले visitor आत आल्यावर सुरू होते आणि तो बाहेर पडल्यावर थांबते — संपूर्ण भेट (Latency). दुसरे फक्त visitor kitchen ची वाट पाहत असतानाच चालते (IntegrationLatency).

संपूर्ण भेटीला 42 सेकंद लागले आणि kitchen ला 38, तर kitchen सावकाश आहे. संपूर्ण भेटीला 42 लागले आणि kitchen ला 2, तर समस्या desk वर आहे — कदाचित badge तपासणी सावकाश आहे.

आणि प्रत्येक भेट म्हणजे visitor book 📒 मधली एक ओळ — कोण, केव्हा, कोणते दार, काय उत्तर, किती वेळ.

🗺️ आकृती

flowchart LR
    in["🙋 request in"] --> office["🛎️ office work<br/>auth · throttle · mapping<br/>≈ 4 ms in the model"]
    office --> kitchen["🍳 kitchen<br/>IntegrationLatency 38 ms"]
    kitchen --> out["📤 answer out"]
    lat["⏱️ Latency = 42 ms<br/>whole trip"] -.-> in
    lat -.-> out
    out --> cw["📊 CloudWatch<br/>Count · 4XXError · 5XXError · Cache*"]
    out --> log["📒 access log<br/>$context.requestId · status · latencies"]

🗺️ काढलेली आकृती + एक lab: https://school-edh.pages.dev/apigateway/lesson-diagrams.html#l11

❓ काय

CloudWatch metrics (REST APIs, namespace AWS/ApiGateway, dimensions ApiName आणि Stage, हवे असल्यास प्रत्येक method साठी):
- Count — requests;
- 4XXError / 5XXError — client errors / server errors (Sum किंवा Average वापरा);
- Latency — office ला request मिळाल्या क्षणापासून ते उत्तर पाठवेपर्यंत;
- IntegrationLatency — फक्त back end चा भाग;
- CacheHitCount / CacheMissCount — सूचना फलक (धडा 08). HTTP APIs मध्ये लहान अक्षरांतील नावांचे (4xx, 5xx) असेच metrics असतात, आणि cache चे metrics नसतात.
Latency − IntegrationLatency = office चा स्वतःचा वेळ (authorizers, mapping, throttling). मोठे अंतर → authorizer पाहा; छोटे अंतर → back end पाहा.
Access logs — प्रत्येक request ची CloudWatch Logs (किंवा Firehose) मध्ये एक ओळ, तुम्ही $context variables ने निवडलेल्या format मध्ये: $context.requestId, $context.status, $context.responseLatency, $context.integrationLatency, $context.identity.sourceIp, $context.authorizer.principalId, $context.error.message.
Execution logs (REST) — office च्या स्वतःच्या पायरी-पायरीच्या नोंदी (ERROR किंवा INFO level). Debugging करताना उपयोगी; prod मध्ये संपूर्ण request bodies log करू नका.
X-Ray (REST stages) — office, Lambda आणि पुढच्या AWS calls मधून request चा माग काढतो, आणि वेळेसह मार्ग काढून दाखवतो.

🤔 का

कारण "API सावकाश आहे" किंवा "users ना errors येतात" हे तुम्ही दुरुस्त करू शकत नाही. तुम्हाला कोणता route, कोणता status आणि प्रवासाचा कोणता भाग हे माहीत असायला हवे. दोन latency metrics काही सेकंदांत सांगतात की back-end team ला बोलवायचे की authorizer पाहायचा. requestId असलेले access logs user ची तक्रार एका नेमक्या request शी जोडतात.

🔧 कसे (या repo मध्ये)

apigw/gateway.py मधला Gateway.handle() gw.metrics मध्ये Count, 4XXError, 5XXError, CacheHitCount आणि CacheMissCount मोजतो, आणि प्रत्येक request साठी requestId, stage, httpMethod, path, status, principal, latencyMs आणि integrationLatencyMs असलेली एक access-log dict gw.logs मध्ये जोडतो. Kitchen चा वेळ म्हणजे route ची latency (/students/{id} साठी 38 ms; /report साठी 31 s, 29 s timeout ला कापला जातो), आणि model स्वतःच्या कामासाठी OFFICE_MS = 4 जोडते — म्हणून इथे latencyMs - integrationLatencyMs नेहमी 4 असतो.

🧪 करून पाहा

python3 apigw/demo.py monitor
python3 - <<'EOF'
import sys; sys.path.insert(0, "apigw"); from demo import office, Clock
c = Clock(); gw, _ = office(c)
for p in ("/students/7", "/students/7", "/students/9", "/students/404", "/broken", "/report", "/teachers", "/files/a.pdf"):
    gw.handle("GET", p)
m = gw.metrics
print(m)
print(f"4XX rate {m['4XXError'] / m['Count']:.0%} · 5XX rate {m['5XXError'] / m['Count']:.0%}")
for l in gw.logs:
    print(f"{l['requestId']} {l['path']:<14} {l['status']}  Latency {l['latencyMs']:>5} ms  Integration {l['integrationLatencyMs']:>5} ms  office {l['latencyMs'] - l['integrationLatencyMs']} ms")
EOF

✅ तपासा — तुम्हाला काय दिसायला हवे

monitor छापते Count=6 · 4XXError=2 · 5XXError=2 · CacheHitCount=1 · CacheMissCount=2 आणि सहा log ओळी, उदा. {'requestId': 'req-0001', 'stage': 'prod', 'httpMethod': 'GET', 'path': '/students/7', 'status': 200, 'principal': None, 'latencyMs': 42, 'integrationLatencyMs': 38} आणि, सावकाश report साठी, {'requestId': 'req-0005', …, 'path': '/report', 'status': 504, …, 'latencyMs': 29004, 'integrationLatencyMs': 29000}.

तुमचा snippet छापतो {'Count': 8, '4XXError': 2, '5XXError': 2, 'CacheHitCount': 1, 'CacheMissCount': 3}, 4XX rate 25% · 5XX rate 25%, आणि एक table:

req-0001 /students/7    200  Latency    42 ms  Integration    38 ms  office 4 ms
req-0002 /students/7    200  Latency     4 ms  Integration     0 ms  office 4 ms
req-0003 /students/9    200  Latency    42 ms  Integration    38 ms  office 4 ms
req-0004 /students/404  404  Latency    42 ms  Integration    38 ms  office 4 ms
req-0005 /broken        502  Latency     4 ms  Integration     0 ms  office 4 ms
req-0006 /report        504  Latency 29004 ms  Integration 29000 ms  office 4 ms
req-0007 /teachers      404  Latency     4 ms  Integration     0 ms  office 4 ms
req-0008 /files/a.pdf   200  Latency     4 ms  Integration     0 ms  office 4 ms

Cache hit (req-0002) ला kitchen चा वेळ अजिबात लागला नाही; 504 ने 29 s kitchen मध्ये घालवले.

🏁 तुम्ही आत्ताच काय सिद्ध केले

दोन आकड्यांवरून तुम्ही वेळ कुठे गेला ते सांगू शकता, आणि log ओळीवरून user ज्या एका request बद्दल तक्रार करतो ती शोधू शकता — तिच्या requestId ने.

⚠️ नेहमीच्या चुका

फक्त Average latency वर alarm — एक सावकाश route अनेक जलद routes मागे लपतो; p99 वापरा
4XXError म्हणजे "आमचा bug" असे वाचणे — ते बहुतेक callers चे असतात (401, 403, 429); access log मध्ये status नुसार वेगळे करा
access logs अजिबात नसणे, त्यामुळे तक्रार कोणत्याही request शी जुळवता येत नाही
prod मध्ये संपूर्ण request data सह INFO level वर execution logs — खर्च, आणि logs मध्ये वैयक्तिक data

🏭 प्रत्यक्ष वापरात

खऱ्या account वर — JSON मध्ये access logs आणि X-Ray चालू (Terraform), आणि 5XXError वर alarm (CLI):

resource "aws_api_gateway_stage" "prod" {
  rest_api_id          = aws_api_gateway_rest_api.school.id
  deployment_id        = aws_api_gateway_deployment.school.id
  stage_name           = "prod"
  xray_tracing_enabled = true
  access_log_settings {
    destination_arn = aws_cloudwatch_log_group.school_api_access.arn
    format = jsonencode({
      requestId          = "$context.requestId"
      ip                 = "$context.identity.sourceIp"
      method             = "$context.httpMethod"
      path               = "$context.resourcePath"
      status             = "$context.status"
      latency            = "$context.responseLatency"
      integrationLatency = "$context.integrationLatency"
      principal          = "$context.authorizer.principalId"
    })
  }
}

aws cloudwatch put-metric-alarm --alarm-name school-api-prod-5xx \
    --namespace AWS/ApiGateway --metric-name 5XXError \
    --dimensions Name=ApiName,Value=school-api Name=Stage,Value=prod \
    --statistic Sum --period 60 --evaluation-periods 3 --threshold 5 \
    --comparison-operator GreaterThanOrEqualToThreshold \
    --alarm-actions arn:aws:sns:ap-south-1:111122223333:on-call

गेल्या तासासाठी दोन्ही stopwatches ची तुलना करा:

for m in Latency IntegrationLatency; do
  aws cloudwatch get-metric-statistics --namespace AWS/ApiGateway --metric-name $m \
      --dimensions Name=ApiName,Value=school-api Name=Stage,Value=prod \
      --start-time $(date -u -v-1H +%FT%TZ) --end-time $(date -u +%FT%TZ) \
      --period 3600 --extended-statistics p99
done

(date -v-1H हे macOS चे आहे; Linux वर date -u -d '-1 hour' +%FT%TZ वापरा.)

🏭 Production मध्ये हे का महत्त्वाचे आहे: साध्या API dashboard मध्ये चार panels असतात — requests, 4XX, 5XX, आणि p99 IntegrationLatency च्या शेजारी p99 Latency — आणि प्रत्येक stage साठी 5XX व latency वर alarms. तुम्ही परत देत असलेल्या प्रत्येक error message मध्ये requestId घाला.

⏭️ पुढे

तुम्हाला आता सगळे दिसते. शेवटचा धडा: कडक मर्यादा — 29-second timeout, 10 MB limit, private APIs, WAF आणि bill.

git checkout lesson-12-private-waf-limits

📈 Lesson 11 — Monitoring: the dashboard and the stopwatch

📍 You are here: Lesson 11 of 12 · Previous: lesson-10-custom-domains · Next: lesson-12-private-waf-limits

📦 What's in this branch

Lessons 01–10, plus how to see inside the office: the CloudWatch metrics (Count, 4XXError, 5XXError, Latency, IntegrationLatency, CacheHitCount, CacheMissCount), access logs with $context variables, and X-Ray traces. The key skill: read Latency vs IntegrationLatency to find where the time goes. monitor() in apigw/demo.py sends six requests and prints the dashboard and the visitor book.

🧒 Explain like I'm 5

The front office has a dashboard 📊 on the wall: how many visitors today, how many were sent away for their own mistakes (4XX), how many were let down by a kitchen (5XX), how many answers came from the notice board.

The clerk also has two stopwatches ⏱️. The first starts when the visitor walks in and stops when they leave — the whole visit (Latency). The second runs only while the visitor waits for the kitchen (IntegrationLatency).

If the whole visit took 42 seconds and the kitchen took 38, the kitchen is slow. If the whole visit took 42 and the kitchen took 2, the problem is at the desk — maybe the badge check is slow.

And every visit is one line in the visitor book 📒 — who, when, which door, what answer, how long.

🗺️ Diagram

flowchart LR
    in["🙋 request in"] --> office["🛎️ office work<br/>auth · throttle · mapping<br/>≈ 4 ms in the model"]
    office --> kitchen["🍳 kitchen<br/>IntegrationLatency 38 ms"]
    kitchen --> out["📤 answer out"]
    lat["⏱️ Latency = 42 ms<br/>whole trip"] -.-> in
    lat -.-> out
    out --> cw["📊 CloudWatch<br/>Count · 4XXError · 5XXError · Cache*"]
    out --> log["📒 access log<br/>$context.requestId · status · latencies"]

🗺️ Drawn version + a lab: https://school-edh.pages.dev/apigateway/lesson-diagrams.html#l11

❓ What

CloudWatch metrics (REST APIs, namespace AWS/ApiGateway, dimensions ApiName and Stage, optionally per method):
- Count — requests;
- 4XXError / 5XXError — client errors / server errors (use the Sum or Average);
- Latency — from the moment the office receives the request until it sends the answer;
- IntegrationLatency — the back end's part only;
- CacheHitCount / CacheMissCount — the notice board (lesson 08). HTTP APIs have similar metrics with lower-case names (4xx, 5xx) and no cache ones.
Latency − IntegrationLatency = the office's own time (authorizers, mapping, throttling). Large gap → look at the authorizer; small gap → look at the back end.
Access logs — one line per request to CloudWatch Logs (or Firehose), in a format you choose with $context variables: $context.requestId, $context.status, $context.responseLatency, $context.integrationLatency, $context.identity.sourceIp, $context.authorizer.principalId, $context.error.message.
Execution logs (REST) — the office's own step-by-step notes (ERROR or INFO level). Useful while debugging; do not log full request bodies in prod.
X-Ray (REST stages) — traces a request through the office, the Lambda and further AWS calls, and draws the path with timings.

🤔 Why

Because "the API is slow" or "users get errors" is not something you can fix. You need to know which route, which status and which part of the trip. The two latency metrics tell you in seconds whether to call the back-end team or look at the authorizer. Access logs with the requestId connect a user's complaint to one exact request.

🔧 How (in this repo)

Gateway.handle() in apigw/gateway.py counts Count, 4XXError, 5XXError, CacheHitCount and CacheMissCount in gw.metrics, and appends one access-log dict per request to gw.logs with requestId, stage, httpMethod, path, status, principal, latencyMs and integrationLatencyMs. The kitchen's time is the route's latency (38 ms for /students/{id}; 31 s for /report, cut at the 29 s timeout), and the model adds OFFICE_MS = 4 for its own work — so latencyMs - integrationLatencyMs is always 4 here.

🧪 Try it

python3 apigw/demo.py monitor
python3 - <<'EOF'
import sys; sys.path.insert(0, "apigw"); from demo import office, Clock
c = Clock(); gw, _ = office(c)
for p in ("/students/7", "/students/7", "/students/9", "/students/404", "/broken", "/report", "/teachers", "/files/a.pdf"):
    gw.handle("GET", p)
m = gw.metrics
print(m)
print(f"4XX rate {m['4XXError'] / m['Count']:.0%} · 5XX rate {m['5XXError'] / m['Count']:.0%}")
for l in gw.logs:
    print(f"{l['requestId']} {l['path']:<14} {l['status']}  Latency {l['latencyMs']:>5} ms  Integration {l['integrationLatencyMs']:>5} ms  office {l['latencyMs'] - l['integrationLatencyMs']} ms")
EOF

✅ Verify — what you should see

monitor prints Count=6 · 4XXError=2 · 5XXError=2 · CacheHitCount=1 · CacheMissCount=2 and six log lines, e.g. {'requestId': 'req-0001', 'stage': 'prod', 'httpMethod': 'GET', 'path': '/students/7', 'status': 200, 'principal': None, 'latencyMs': 42, 'integrationLatencyMs': 38} and, for the slow report, {'requestId': 'req-0005', …, 'path': '/report', 'status': 504, …, 'latencyMs': 29004, 'integrationLatencyMs': 29000}.

Your snippet prints {'Count': 8, '4XXError': 2, '5XXError': 2, 'CacheHitCount': 1, 'CacheMissCount': 3}, 4XX rate 25% · 5XX rate 25%, and a table:

req-0001 /students/7    200  Latency    42 ms  Integration    38 ms  office 4 ms
req-0002 /students/7    200  Latency     4 ms  Integration     0 ms  office 4 ms
req-0003 /students/9    200  Latency    42 ms  Integration    38 ms  office 4 ms
req-0004 /students/404  404  Latency    42 ms  Integration    38 ms  office 4 ms
req-0005 /broken        502  Latency     4 ms  Integration     0 ms  office 4 ms
req-0006 /report        504  Latency 29004 ms  Integration 29000 ms  office 4 ms
req-0007 /teachers      404  Latency     4 ms  Integration     0 ms  office 4 ms
req-0008 /files/a.pdf   200  Latency     4 ms  Integration     0 ms  office 4 ms

The cache hit (req-0002) has no kitchen time at all; the 504 spent 29 s in the kitchen.

🏁 What you just proved

From two numbers you can say where the time went, and from the log line you can find the one request a user complains about — by its requestId.

⚠️ Common mistakes

alarming on Average latency only — one slow route hides behind many fast ones; use p99
reading 4XXError as "our bug" — it is mostly callers (401, 403, 429); split by status in the access log
no access logs at all, so a complaint cannot be matched to a request
execution logs at INFO with full request data in prod — cost and personal data in logs

🏭 In production

On a real account — access logs in JSON and X-Ray on (Terraform), and an alarm on 5XXError (CLI):

resource "aws_api_gateway_stage" "prod" {
  rest_api_id          = aws_api_gateway_rest_api.school.id
  deployment_id        = aws_api_gateway_deployment.school.id
  stage_name           = "prod"
  xray_tracing_enabled = true
  access_log_settings {
    destination_arn = aws_cloudwatch_log_group.school_api_access.arn
    format = jsonencode({
      requestId          = "$context.requestId"
      ip                 = "$context.identity.sourceIp"
      method             = "$context.httpMethod"
      path               = "$context.resourcePath"
      status             = "$context.status"
      latency            = "$context.responseLatency"
      integrationLatency = "$context.integrationLatency"
      principal          = "$context.authorizer.principalId"
    })
  }
}

aws cloudwatch put-metric-alarm --alarm-name school-api-prod-5xx \
    --namespace AWS/ApiGateway --metric-name 5XXError \
    --dimensions Name=ApiName,Value=school-api Name=Stage,Value=prod \
    --statistic Sum --period 60 --evaluation-periods 3 --threshold 5 \
    --comparison-operator GreaterThanOrEqualToThreshold \
    --alarm-actions arn:aws:sns:ap-south-1:111122223333:on-call

Compare the two stopwatches for the last hour:

for m in Latency IntegrationLatency; do
  aws cloudwatch get-metric-statistics --namespace AWS/ApiGateway --metric-name $m \
      --dimensions Name=ApiName,Value=school-api Name=Stage,Value=prod \
      --start-time $(date -u -v-1H +%FT%TZ) --end-time $(date -u +%FT%TZ) \
      --period 3600 --extended-statistics p99
done

(date -v-1H is macOS; on Linux use date -u -d '-1 hour' +%FT%TZ.)

🏭 Why this matters in production: a basic API dashboard has four panels — requests, 4XX, 5XX, p99 Latency next to p99 IntegrationLatency — and alarms on 5XX and latency per stage. Put the requestId in every error message you return.

⏭️ Next

You can see everything. Last lesson: the hard edges — the 29-second timeout, the 10 MB limit, private APIs, WAF and the bill.

git checkout lesson-12-private-waf-limits

← Previouscustom domains Next →private waf limits

This page is the lesson's README from the lesson-11-monitoring branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.