Troubleshooting VPC Connectivity

An application server in a private subnet cannot open a connection to an RDS instance in the same VPC. The database is running, the credentials are unchanged, and the connection just times out. You have four suspects and no evidence: a security group, a network ACL, a route table, and the possibility that the database is not listening at all.

Most engineers start clicking through the console in whatever order the tabs happen to be in. That works eventually. It also burns the first twenty minutes of an outage on components that were never involved. The skill this lesson teaches is not memorizing more services. It is walking the path in the order a packet walks it, so that the first thing you find wrong is actually the thing that broke.

The path has an order, and so should you

A packet leaving an EC2 instance passes through a fixed sequence of gates. Every one of them can deny it, and each denial looks identical from the application: a timeout. So work the sequence, not your intuition.

1. Source security group, outbound. Does an egress rule allow the destination address and port? Default groups allow all outbound, so this is only a suspect in a locked-down environment. When it is the cause, it is because someone replaced the default egress rule and forgot a port.
2. Source subnet network ACL, outbound. Does an outbound rule allow the request, and does an inbound rule allow the reply on ephemeral ports? Network ACLs are stateless, so both directions are separate questions.
3. Source route table. Is there a route whose destination CIDR contains the target address, and does its target exist and work? No route means the packet is dropped before any firewall is consulted.
4. The intermediate. NAT gateway, internet gateway, VPC endpoint, peering connection, or transit gateway. Each has its own failure modes, and each was covered in the earlier topics of this domain.
5. Destination subnet network ACL, inbound. Same stateless pair of questions from the far side.
6. Destination security group, inbound. Does an ingress rule allow the source, whether by CIDR or by referencing the source's security group?
7. The resource policy. An endpoint policy, an S3 bucket policy, or a service-side policy can reject a request whose packets arrived perfectly. This one produces an error rather than a timeout, which is a useful signal on its own.

Then walk the return path. That sounds like a formality until you meet the failure it catches: the request is accepted, the destination replies, and the reply is denied by an outbound network ACL rule that nobody thought about because the application only ever initiates in one direction.

The reason this order pays is that each gate hides the ones behind it. If the route table has no match, the security group at the far end could be wide open or closed and you would never know. Checking in path order means the first denial you find is the one that matters, and everything after it is noise.

Read the symptom before you read the config

Three of the failures above produce distinguishable symptoms if you look at the right thing.

That second row deserves its own sentence, because it saves entire investigations. A "connection refused" error means the network worked. The SYN reached the host, the host had nothing listening on that port, and it answered honestly. No security group produces that error, because a security group denial produces silence. When an application logs connection refused, stop looking at the VPC and go look at whether the process is running and bound to the address you think it is.

Reachability Analyzer reads the whole path for you

Walking seven gates by hand across two accounts takes real time. Reachability Analyzer does it in one call.

The important thing to understand about it is what it actually looks at. It builds a model of your network configuration and reasons over that model. It does not send packets and it does not touch the data plane. So it can tell you that a security group would drop this traffic, and it can tell you the exact hop-by-hop path a permitted packet would take, without any traffic existing at all. That is why it works on a resource that is completely broken, and why it works before you deploy anything real.

You define a path: a source, a destination, and optionally a protocol, a destination port, and intermediate components to include or exclude. Then you run an analysis on that path. Supported sources and destinations are EC2 instances, internet gateways, network interfaces, transit gateways, transit gateway attachments, virtual private gateways, VPC endpoint services, VPC endpoints, and VPC peering connections, plus a plain IP address as a destination.

# 1. Define the path once
aws ec2 create-network-insights-path \
  --source i-0a1b2c3d4e5f67890 \
  --destination i-09876fedcba543210 \
  --destination-port 3306 \
  --protocol tcp

# 2. Run an analysis against it, as often as you like
aws ec2 start-network-insights-analysis \
  --network-insights-path-id nip-0abc123def456789a

# 3. Read the verdict and the explanations
aws ec2 describe-network-insights-analyses \
  --network-insights-analysis-ids nia-0abc123def456789a \
  --query 'NetworkInsightsAnalyses[0].[NetworkPathFound,Explanations]'

Three operational facts about it are worth carrying:

• You are charged per analysis run, not per path, so the path object is free to keep and re-run after every change. That is what makes it useful as a regression check and not just an incident tool.
• Source and destination must be in the same Region, and in the same VPC or in VPCs connected by peering or a transit gateway. They may be in different accounts in the same AWS Organizations organization if you enable trusted access.
• Analyses are deleted automatically 120 days after creation. If a result is evidence for an audit, export it.

Explanation codes name the guilty component

When a path is not reachable, the analysis returns one or more explanation codes. You do not need to memorize the full list, but recognizing the families turns a wall of output into a one-line diagnosis.

Two of those codes teach something beyond their own message. MORE_SPECIFIC_ROUTE is the routing lesson restated as a diagnosis: your route is present and correct and still unused, because something longer matched first. And TGW_ATTACH_MISSING_TGW_RTB_ASSOCIATION is the association-versus-propagation distinction from the transit gateway lesson showing up as a concrete failure, since an attachment that propagates routes but is associated with nothing has no route table to consult.

Where the model stops and the packets start

Reachability Analyzer is a configuration checker, so it is blind to everything that is not configuration. Knowing its blind spots is what keeps a green result from misleading you.

• It does not consider the health of registered targets. A load balancer whose targets are all failing health checks will still analyze as reachable.
• It supports only IPv4. If a resource has both address families, only the IPv4 side is analyzed. An IPv6-only failure shows up as NO_PATH.
• It has no view of DNS. If your application resolves a name to the wrong address, the path to the right address is still perfectly reachable.
• It stops at transit gateway Connect attachments, and paths through a Gateway Load Balancer endpoint exclude the Gateway Load Balancer and its targets, which need their own analysis.
• Network Firewall support is partial. It handles stateful and stateless 5-tuple rules, but not domain lists, Suricata rules, rule options, or tag-based resource groups, and it says so in the path details when it meets one.
• It says nothing about the application. A listening process, a TLS handshake, a database that rejects the credentials: all outside the model.

So the honest rule is: Reachability Analyzer proves the path is permitted, not that the call will succeed. When it says not reachable, you have your answer and you can stop. When it says reachable and the application still fails, you have learned something valuable too, which is that the problem is above the network layer, and the next lesson's logs are where you go.

The failures that are not really network failures

Four causes account for a large share of the tickets that arrive labeled "VPC connectivity" and never touch a security group.

DNS resolving to the wrong thing. A private hosted zone needs enableDnsSupport and enableDnsHostnames on the VPC to be usable, and an interface endpoint with private DNS disabled leaves the public service name resolving to a public address that a private subnet cannot reach. The connection fails at the network layer, but the fix is a DNS setting.

Path MTU. A connection that opens cleanly and then stalls the moment a large payload starts is almost never a permission problem. Handshakes are small and fit anywhere. Tunnels reduce the usable packet size, and if a firewall drops the ICMP fragmentation-needed messages, path MTU discovery cannot tell the sender to send smaller packets, so the transfer just hangs. Check MTU before you check rules whenever the symptom depends on size.

Source/destination check. Any instance forwarding traffic on behalf of others, such as a NAT instance or an appliance, must have this disabled. Left on, the interface discards forwarded packets and everything about the routing looks correct.

Ephemeral ports on a stateless network ACL. Covered in full in the security groups lesson, and it keeps reappearing here because it is the single most common reason a network ACL allows the request and kills the reply. If the traffic is outbound and the network ACL's inbound rules do not allow 1024 to 65535 back in, nothing works and every rule reads as reasonable.

Exam tips

• Ordered walk, every time. Source security group, source network ACL, route table, intermediate, destination network ACL, destination security group, resource policy, then the return path. Scenario questions are built by breaking exactly one of these.
• "Connection refused" is not a network problem. Timeout means dropped, refused means delivered. A question that says the client gets an immediate refusal is pointing at the service, not the VPC.
• Reachability Analyzer analyzes configuration, not packets. When a question asks how to find the blocking component without generating traffic or changing anything, this is the answer. When it asks what actually happened to real traffic, it is flow logs.
• Reachability Analyzer is IPv4 only, same Region, and ignores target health. Those three constraints are the most likely thing an exam question tests about it.
• MORE_SPECIFIC_ROUTE and NO_ROUTE_TO_DESTINATION are routing; SUBNET_ACL_RESTRICTION is the network ACL; ENI_SG_RULES_MISMATCH is the security group. Being able to map a code to a component is enough.
• Overlapping CIDRs cannot be peered, because the local route always wins inside your own VPC CIDR and cannot be overridden.
• Forwarding appliance plus silent drops equals source/destination check.
• Stalls only on big transfers equals MTU, not firewalls.

The habit to carry out of this lesson is smaller than the tool list: when a connection fails, name the first gate on the path that you have not verified, and check that one. Reachability Analyzer is how you do that walk in one call instead of seven. What it cannot tell you is what actually happened to real traffic last Tuesday at 03:00, and that is what the next lesson's logs are for.