🖼️ See the drawing + lab 🏠 Course home 🌿 Branch on GitHub ✏️ View source

🖼️ आकृती आणि labThe drawing + lab पूर्ण पानावर उघडा ↗Open full page ↗

🔎 धडा 08 — फाटकाचे troubleshooting

📍 तुम्ही इथे आहात: 8 पैकी धडा 08 · मागे: lesson-07-alternatives

📦 या ब्रँचमध्ये काय आहे

धडे 01–07, आणि तुम्हाला प्रत्यक्षात दिसणारे दोन बिघाड: port exhaustion (एका destination साठी नोंदवही भरली → ErrorPortAllocation) आणि 350-second idle timeout (शांत ओळ विसरली जाते → connection reset होते). आणि दोन्ही दाखवणारे CloudWatch metrics. nat/demo.py मधील troubleshoot() मुद्दाम फाटक बिघडवते.

🧒 5 वर्षांच्या मुलाला समजावल्यासारखे

फाटकावर दोन गोष्टी चुकू शकतात:

📒 नोंदवहीचे पान भरले. प्रत्येक पान एका दुकानासाठी असते, आणि त्यावर ठराविक संख्येचेच तिकीट क्रमांक असतात. एकाच वेळी खूप विद्यार्थिनी एकाच दुकानात गेल्या, तर रखवालदाराकडे तिकीटच उरत नाही आणि तो म्हणतो "थांबा". दुसऱ्या दुकानात जाणाऱ्या विद्यार्थिनींना तरीही तिकिटे मिळतात. उपाय: फाटकाचा दुसरा पत्ता (तिकिटांचे नवे पान), किंवा एकाच वेळी त्या दुकानात जाणाऱ्या विद्यार्थिनी कमी करणे.
🕰️ रखवालदार शांत ओळी विसरतो. एखाद्या फेरीवरून सुमारे सहा मिनिटे (350 seconds) कोणी एकही शब्द पाठवला नाही, तर रखवालदार ती ओळ खोडून टाकतो. दुकान शेवटी उत्तर देते तेव्हा ओळच नसते — उत्तर फेकून दिले जाते, आणि विद्यार्थिनीला सांगितले जाते की फेरी संपली. उपाय: दर काही मिनिटांनी छोटी "मी अजून आहे" चिठ्ठी (keepalive) पाठवणे.

🗺️ आकृती

flowchart TB
    subgraph full["📒 port exhaustion"]
      c["connections 1–5 → pypi"] --> ok["✅ ports 1024–1028"]
      c6["connection 6 → pypi"] --> err["❌ ErrorPortAllocation"]
      gh["connection → github"] --> ok2["✅ different destination"]
    end
    subgraph idle["🕰️ idle timeout"]
      t0["t=0 row written"] --> t400["t=400 s quiet > 350 s → row wiped"]
      t400 --> late["late answer → dropped, app sees a reset"]
    end

🗺️ काढलेली आकृती + एक lab: https://school-edh.pages.dev/nat/lesson-diagrams.html#l08

❓ काय

Port exhaustion — NAT gateway प्रत्येक assigned IP ला प्रत्येक unique destination कडे (destination IP, destination port आणि protocol) 55,000 पर्यंत एकाच वेळचे connections चालवतो. त्याच destination कडचे 55,001 वे fail होते. CloudWatch ते ErrorPortAllocation म्हणून मोजते. इतर destinations वर परिणाम होत नाही.
Port exhaustion वरचे उपाय — जास्त IPs जोडा (प्रत्येक NAT gateway ला 8 पर्यंत; धडा 05), traffic जास्त destinations वर पसरवा (उदाहरणार्थ, जास्त backend IPs), connections पुन्हा वापरा (HTTP keep-alive, connection pools), आणि AWS-service traffic endpoints कडे हलवा (धडा 07).
Idle timeout — traffic शिवाय 350 seconds गेल्यावर NAT gateway connection विसरतो. त्यानंतर कोणत्याही बाजूने ते वापरण्याचा प्रयत्न केला, तर NAT gateway त्याच्यामागच्या resource ला RST परत पाठवतो; application ला "connection reset" दिसते. CloudWatch हे IdleTimeoutCount मध्ये मोजते.
Idle timeout वरचा उपाय — instances वर (किंवा client library मध्ये) 350 seconds पेक्षा कमी interval ने TCP keepalive चालू करा, किंवा खूप वेळ शांत असलेली connections बंद करून पुन्हा उघडा.
इतर उपयोगी metrics (namespace AWS/NATGateway) — PacketsDropCount (NAT ने drop केलेले packets — वाढता आकडा NAT मधील अडचणीकडे निर्देश करू शकतो), ActiveConnectionCount (नोंदवहीचा आकार), BytesOutToDestination (बाहेर जाणारा traffic, GB bill चा पाया), आणि ConnectionAttemptCount / ConnectionEstablishedCount.
Private subnet मधून "internet चालत नाही" तेव्हाची तपासणी यादी — route table मध्ये 0.0.0.0/0 → nat-… आहे; NAT available आहे; NAT चे subnet 0.0.0.0/0 → igw-… route करते; NACLs traffic आणि परतीचे ephemeral ports दोन्हींना परवानगी देतात; instance चा security group outbound traffic ला परवानगी देतो; DNS resolve होते.

🤔 का

कारण दोन्ही बिघाड अधूनमधून येणाऱ्या application errors सारखे दिसतात: "load खाली timeouts" आणि "शांत कालावधीनंतर connection reset". नोंदवही माहीत नसेल तर teams servers restart करतात किंवा मोठी instances जोडतात, आणि काहीच बदलत नाही. Metrics असतील तर कारण काही मिनिटांत दिसते.

🔧 कसे (या repo मध्ये)

nat/gateway.py मध्ये, प्रत्येक IP ने त्या destination कडे ports_per_dest ports वापरले की send() हे self.errors += 1 मोजते (model चे ErrorPortAllocation). tick(now) ज्या ओळींची last वेळ idle_timeout seconds पेक्षा जास्त जुनी आहे त्या पुसते, आणि ओळीवरचे प्रत्येक send() किंवा reply() हे last ताजे करते — keepalive नेमके हेच करतो. nat/demo.py मधील troubleshoot() नोंदवही 5 ports पर्यंत लहान करते, म्हणजे ती screen वरच भरते. (Model मध्ये खरे 55,000 भरण्याचा प्रयत्न करू नका — ते प्रत्येक send ला संपूर्ण table scan करते, त्यामुळे त्याला खूप वेळ लागतो. त्याऐवजी ports_per_dest लहान करा.)

🧪 करून पाहा

python3 nat/demo.py troubleshoot
python3 - <<'EOF'
import sys; sys.path.insert(0, "nat"); from gateway import NatGateway
PYPI = ("151.101.0.223", 443, "tcp")
for now in (350, 351):
    g = NatGateway("nat-a", ["13.232.10.20"]); ip, port = g.send(("10.20.48.25", 50001), PYPI, t=0)
    print(f"quiet until t={now}: wiped {g.tick(now)} · reply → {g.reply(ip, port, PYPI, now)}")
g = NatGateway("nat-a", ["13.232.10.20"]); ip, port = g.send(("10.20.48.25", 50001), PYPI, t=0)
for t in (300, 600, 900):                       # a keepalive every 300 s
    g.send(("10.20.48.25", 50001), PYPI, t=t); g.tick(t)
print("with keepalive every 300 s, at t=1000:", g.tick(1000), "wiped · reply →", g.reply(ip, port, PYPI, 1000))
EOF
python3 nat/test_nat.py

✅ तपासा — तुम्हाला काय दिसायला हवे

troubleshoot हे connection 1 → 13.232.10.20:1024 पासून connection 5 → 13.232.10.20:1028 पर्यंत छापते, मग connection 6 → ❌ ErrorPortAllocation आणि connection 7 → ❌ ErrorPortAllocation, मग a DIFFERENT destination still works → rows: 6 · ErrorPortAllocation = 2, आणि t=400 s the server answers late → wiped 1 row → reply dropped (the app sees a reset). तुमचा snippet हे छापतो:

quiet until t=350: wiped 0 · reply → ('10.20.48.25', 50001)
quiet until t=351: wiped 1 · reply → None
with keepalive every 300 s, at t=1000: 0 wiped · reply → ('10.20.48.25', 50001)

बरोबर 350 seconds ला ओळ टिकते; एक second जास्त झाला की ती जाते. दर 300 seconds ला keepalive असेल तर 1,000 seconds नंतरही ओळ तिथेच असते. Tests 8/8 passed ने संपतात.

🏁 तुम्ही आत्ताच काय सिद्ध केले

"Load खाली connect होत नाही" म्हणजे एका destination साठी भरलेली नोंदवही असू शकते, आणि "शांत कालावधीनंतर reset" म्हणजे विसरलेली ओळ असू शकते — आणि प्रत्येकाचा metric आणि उपाय तुम्हाला माहीत आहे.

⚠️ नेहमीच्या चुका

ErrorPortAllocation साठी app servers restart करणे — मर्यादा NAT च्या बाजूला आहे; IPs जोडा किंवा destinations पसरवा
pool पुन्हा वापरण्याऐवजी एकाच व्यस्त API कडे प्रत्येक request ला नवे connection उघडणे
NAT मधून जाणारी database किंवा SSH connections 10 मिनिटे शांत बसतात, keepalive नसताना
outbound traffic ला परवानगी देणारे पण ephemeral ports (1024–65535) वरचा परतीचा traffic अडवणारे NACLs
alarm च नाही — ErrorPortAllocation ऐवजी users कडून कळणे

🏭 प्रत्यक्ष वापरात

खऱ्या account वर — port allocation errors वर alarm लावा, idle-timeout count वाचा, आणि Linux instance वर 350 seconds पेक्षा कमी keepalive ठेवा:

aws cloudwatch put-metric-alarm --alarm-name nat-a-port-allocation --namespace AWS/NATGateway --metric-name ErrorPortAllocation --dimensions Name=NatGatewayId,Value=nat-0abc --statistic Sum --period 300 --evaluation-periods 1 --threshold 0 --comparison-operator GreaterThanThreshold --alarm-actions arn:aws:sns:us-east-1:111122223333:ops
aws cloudwatch get-metric-statistics --namespace AWS/NATGateway --metric-name IdleTimeoutCount --dimensions Name=NatGatewayId,Value=nat-0abc --start-time 2026-09-26T00:00:00Z --end-time 2026-09-27T00:00:00Z --period 3600 --statistics Sum
aws cloudwatch get-metric-statistics --namespace AWS/NATGateway --metric-name PacketsDropCount --dimensions Name=NatGatewayId,Value=nat-0abc --start-time 2026-09-26T00:00:00Z --end-time 2026-09-27T00:00:00Z --period 3600 --statistics Sum
sudo sysctl -w net.ipv4.tcp_keepalive_time=300    # on the instance; the app must enable SO_KEEPALIVE

resource "aws_cloudwatch_metric_alarm" "nat_ports" {
  for_each            = aws_nat_gateway.this
  alarm_name          = "nat-${each.key}-port-allocation"
  namespace           = "AWS/NATGateway"
  metric_name         = "ErrorPortAllocation"
  dimensions          = { NatGatewayId = each.value.id }
  statistic           = "Sum"
  period              = 300
  evaluation_periods  = 1
  threshold           = 0
  comparison_operator = "GreaterThanThreshold"
}

🏭 प्रत्यक्ष वापरात हे का महत्त्वाचे: प्रत्येक NAT gateway वर ErrorPortAllocation चा alarm असायला हवा आणि ActiveConnectionCount, IdleTimeoutCount, PacketsDropCount व BytesOutToDestination असलेला dashboard असायला हवा. हे चार graphs कोणी ticket उघडण्याआधीच बहुतेक egress प्रश्नांची उत्तरे देतात.

🎓 फाटक आता तुमचे आहे

शाळेचे फाटक → नोंदवही → managed फाटक की स्वतःचे → प्रत्येक इमारतीला एक फाटक → स्थिर परतीचा पत्ता → दोन मीटर → फाटक टाळणाऱ्या फेऱ्या → भरलेल्या नोंदवह्या आणि विसरलेल्या ओळी. तुम्ही फक्त NAT शिकला नाहीत — तुम्ही शाळेचे फाटक चालवू शकता. 🚪🎓

शाळेतली पुढची दारे: VPC शाळा हे फाटक ज्या संपूर्ण campus चा भाग आहे तो बांधते; Networking शाळा IP, ports आणि TCP मध्ये अधिक खोलवर जाते; School portal मध्ये बाकी सगळे आहे.

git checkout main
python3 nat/demo.py     # one last run, for fun

🔎 Lesson 08 — Troubleshooting the gate

📍 You are here: Lesson 08 of 8 · Previous: lesson-07-alternatives

📦 What's in this branch

Lessons 01–07, plus the two failures you will actually see: port exhaustion (the register is full for one destination → ErrorPortAllocation) and the 350-second idle timeout (a quiet row is forgotten → the connection is reset). And the CloudWatch metrics that show both. troubleshoot() in nat/demo.py breaks the gate on purpose.

🧒 Explain like I'm 5

Two things can go wrong at the gate:

📒 The register page is full. Each page is for one shop, and it has a fixed number of ticket numbers. If too many pupils go to the same shop at the same time, the guard has no ticket left and says "wait". Pupils going to a different shop still get tickets. Fixes: a second gate address (a new page of tickets), or fewer pupils going to that shop at once.
🕰️ The guard forgets quiet rows. If nobody on a trip sends a word for about six minutes (350 seconds), the guard crosses out the row. When the shop finally answers, there is no row — the answer is thrown away, and the pupil is told the trip is over. Fix: send a small "still here" note (a keepalive) every few minutes.

🗺️ Diagram

flowchart TB
    subgraph full["📒 port exhaustion"]
      c["connections 1–5 → pypi"] --> ok["✅ ports 1024–1028"]
      c6["connection 6 → pypi"] --> err["❌ ErrorPortAllocation"]
      gh["connection → github"] --> ok2["✅ different destination"]
    end
    subgraph idle["🕰️ idle timeout"]
      t0["t=0 row written"] --> t400["t=400 s quiet > 350 s → row wiped"]
      t400 --> late["late answer → dropped, app sees a reset"]
    end

🗺️ Drawn version + a lab: https://school-edh.pages.dev/nat/lesson-diagrams.html#l08

❓ What

Port exhaustion — a NAT gateway supports up to 55,000 simultaneous connections to each unique destination (destination IP, destination port and protocol) per assigned IP. The 55,001st to that same destination fails. CloudWatch counts it as ErrorPortAllocation. Other destinations are not affected.
Fixes for port exhaustion — associate more IPs (up to 8 per NAT gateway; lesson 05), spread traffic over more destinations (for example, more backend IPs), reuse connections (HTTP keep-alive, connection pools), and move AWS-service traffic to endpoints (lesson 07).
Idle timeout — a NAT gateway forgets a connection after 350 seconds without traffic. If either side then tries to use it, the NAT gateway returns a RST to the resource behind it; the application sees "connection reset". CloudWatch counts this in IdleTimeoutCount.
Fix for idle timeout — enable TCP keepalive at an interval under 350 seconds on the instances (or in the client library), or close and reopen long-quiet connections.
Other useful metrics (namespace AWS/NATGateway) — PacketsDropCount (packets the NAT dropped — a rising value can point to a NAT problem), ActiveConnectionCount (the register's size), BytesOutToDestination (traffic out, the base of the GB bill), and ConnectionAttemptCount / ConnectionEstablishedCount.
Checklist when "the internet does not work" from a private subnet — route table has 0.0.0.0/0 → nat-…; the NAT is available; the NAT's subnet routes 0.0.0.0/0 → igw-…; NACLs allow the traffic and the return ephemeral ports; the instance's security group allows the outbound traffic; DNS resolves.

🤔 Why

Because both failures look like random application errors: "timeouts under load" and "connection reset after a quiet period". Without knowing the register, teams restart servers or add bigger instances, and nothing changes. With the metrics, the cause is visible in minutes.

🔧 How (in this repo)

In nat/gateway.py, send() counts self.errors += 1 (the model's ErrorPortAllocation) when every IP has used ports_per_dest ports to that destination. tick(now) wipes rows whose last time is more than idle_timeout seconds old, and every send() or reply() on a row refreshes last — which is exactly what a keepalive does. troubleshoot() in nat/demo.py shrinks the register to 5 ports so it fills on screen. (Do not try to fill the real 55,000 in the model — it scans the whole table on every send, so that takes a very long time. Shrink ports_per_dest instead.)

🧪 Try it

python3 nat/demo.py troubleshoot
python3 - <<'EOF'
import sys; sys.path.insert(0, "nat"); from gateway import NatGateway
PYPI = ("151.101.0.223", 443, "tcp")
for now in (350, 351):
    g = NatGateway("nat-a", ["13.232.10.20"]); ip, port = g.send(("10.20.48.25", 50001), PYPI, t=0)
    print(f"quiet until t={now}: wiped {g.tick(now)} · reply → {g.reply(ip, port, PYPI, now)}")
g = NatGateway("nat-a", ["13.232.10.20"]); ip, port = g.send(("10.20.48.25", 50001), PYPI, t=0)
for t in (300, 600, 900):                       # a keepalive every 300 s
    g.send(("10.20.48.25", 50001), PYPI, t=t); g.tick(t)
print("with keepalive every 300 s, at t=1000:", g.tick(1000), "wiped · reply →", g.reply(ip, port, PYPI, 1000))
EOF
python3 nat/test_nat.py

✅ Verify — what you should see

troubleshoot prints connection 1 → 13.232.10.20:1024 up to connection 5 → 13.232.10.20:1028, then connection 6 → ❌ ErrorPortAllocation and connection 7 → ❌ ErrorPortAllocation, then a DIFFERENT destination still works → rows: 6 · ErrorPortAllocation = 2, and t=400 s the server answers late → wiped 1 row → reply dropped (the app sees a reset). Your snippet prints:

quiet until t=350: wiped 0 · reply → ('10.20.48.25', 50001)
quiet until t=351: wiped 1 · reply → None
with keepalive every 300 s, at t=1000: 0 wiped · reply → ('10.20.48.25', 50001)

At exactly 350 seconds the row survives; one second more and it is gone. With a keepalive every 300 seconds, the row is still there after 1,000 seconds. The tests end with 8/8 passed.

🏁 What you just proved

"Cannot connect under load" can be a full register for one destination, and "reset after a quiet period" can be a forgotten row — and you know the metric and the fix for each.

⚠️ Common mistakes

restarting app servers for ErrorPortAllocation — the limit is on the NAT's side; add IPs or spread destinations
opening a new connection per request to one busy API instead of reusing a pool
database or SSH connections through the NAT that sit quiet for 10 minutes, with no keepalive
NACLs that allow outbound traffic but block the return traffic on ephemeral ports (1024–65535)
no alarm at all — finding out from users instead of from ErrorPortAllocation

🏭 In production

On a real account — alarm on port allocation errors, read the idle-timeout count, and set a keepalive under 350 seconds on a Linux instance:

aws cloudwatch put-metric-alarm --alarm-name nat-a-port-allocation --namespace AWS/NATGateway --metric-name ErrorPortAllocation --dimensions Name=NatGatewayId,Value=nat-0abc --statistic Sum --period 300 --evaluation-periods 1 --threshold 0 --comparison-operator GreaterThanThreshold --alarm-actions arn:aws:sns:us-east-1:111122223333:ops
aws cloudwatch get-metric-statistics --namespace AWS/NATGateway --metric-name IdleTimeoutCount --dimensions Name=NatGatewayId,Value=nat-0abc --start-time 2026-09-26T00:00:00Z --end-time 2026-09-27T00:00:00Z --period 3600 --statistics Sum
aws cloudwatch get-metric-statistics --namespace AWS/NATGateway --metric-name PacketsDropCount --dimensions Name=NatGatewayId,Value=nat-0abc --start-time 2026-09-26T00:00:00Z --end-time 2026-09-27T00:00:00Z --period 3600 --statistics Sum
sudo sysctl -w net.ipv4.tcp_keepalive_time=300    # on the instance; the app must enable SO_KEEPALIVE

resource "aws_cloudwatch_metric_alarm" "nat_ports" {
  for_each            = aws_nat_gateway.this
  alarm_name          = "nat-${each.key}-port-allocation"
  namespace           = "AWS/NATGateway"
  metric_name         = "ErrorPortAllocation"
  dimensions          = { NatGatewayId = each.value.id }
  statistic           = "Sum"
  period              = 300
  evaluation_periods  = 1
  threshold           = 0
  comparison_operator = "GreaterThanThreshold"
}

🏭 Why this matters in production: every NAT gateway should have an alarm on ErrorPortAllocation and a dashboard with ActiveConnectionCount, IdleTimeoutCount, PacketsDropCount and BytesOutToDestination. Those four graphs answer most egress questions before anyone opens a ticket.

🎓 The gate is yours

The school gate → the register → a managed gate or your own → one gate per building → a fixed return address → two meters → trips that skip the gate → full registers and forgotten rows. You didn't just learn NAT — you can run the school gate. 🚪🎓

Next doors in the school: the VPC school builds the whole campus this gate belongs to; the Networking school goes deeper into IP, ports and TCP; the School portal has the rest.

git checkout main
python3 nat/demo.py     # one last run, for fun

← Previousalternatives Finished! Take the quiz →check what stuck

This page is the lesson's README from the lesson-08-troubleshooting branch, shown here so the whole School stays on one site. Code files open on GitHub at the same branch.