VPC Endpoints and AWS PrivateLink

An instance in a private subnet uploads 40 TB a month to an S3 bucket in the same Region. The route table sends 0.0.0.0/0 to a NAT gateway, so every one of those gigabytes is metered by the NAT gateway's data processing charge on the way out. The data never touches the public internet: it goes out through the internet gateway and stays on the AWS network the whole way. You are paying a translation device to forward traffic between 2 AWS services.

There is a route table entry that removes the NAT gateway from that path entirely, and it costs nothing. There is also a second kind of endpoint that costs money and does something the first one cannot do at all. Knowing which is which is most of this lesson.

Why the traffic leaves your VPC in the first place

S3, DynamoDB, CloudWatch, and the rest of the AWS API surface are reached through public service endpoints: regional DNS names like s3.us-east-1.amazonaws.com that resolve to public IP addresses. Your instance is not talking to something inside your VPC, so the packet needs a way out, and by definition a private subnet has no route to an internet gateway. That is why the NAT gateway is there.

Worth being precise about one thing, because it changes how you argue about this with a security team: traffic to an AWS service through an internet gateway does not leave the AWS network. AWS says so directly. The reason to build a VPC endpoint is not usually "the data is exposed on the internet". The reasons are that a private subnet should not need an internet path at all, that you want to pin access to a specific endpoint in a policy, and that NAT gateway data processing on high-volume traffic is money spent on nothing.

A VPC endpoint is the fix. It connects your VPC to a service without an internet gateway or a NAT device in the path. There are 2 shapes of it, and they are built on different mechanisms.

Gateway endpoints: a route, not an address

A gateway endpoint serves exactly 2 services: Amazon S3 and DynamoDB. Nothing else. It is not built on PrivateLink, unlike every other endpoint type, and it is free.

When you create one, you select the route tables that should use it. AWS then adds this route to each selected table:

You can look at that route but you cannot edit or delete it. It is removed when you disassociate the route table or delete the endpoint. That is the entire mechanism: a more specific route that beats 0.0.0.0/0.

Three consequences follow directly, and each shows up in scenarios:

Association is by route table, not by VPC. Instances in subnets whose route tables you associated use the endpoint. Instances in other subnets keep using the public service endpoint through whatever path they had. A "some instances work, some do not" report about S3 access is usually a route table that was never associated.

Longest prefix match decides everything. The prefix list is more specific than 0.0.0.0/0, so same-Region S3 traffic goes to the endpoint while traffic to any other AWS service still goes to the internet gateway. Prefix lists are per-Region, so traffic to S3 in a different Region falls back to the internet gateway. If someone adds a route with the exact service IP range and a different target, that route wins over the endpoint route.

One endpoint route per service per route table. A single route table can hold an S3 endpoint route and a DynamoDB endpoint route, and you can point several route tables at the same endpoint. You cannot put 2 S3 endpoint routes in one table.

Security is where gateway endpoints surprise people. Your instances still reach the service at its public IP addresses, so the controls that apply are the ones that filter by address:

# Outbound security group rule for instances using a gateway endpoint.
# The source is a prefix list ID, not a CIDR.
aws ec2 authorize-security-group-egress \
  --group-id sg-0app11111111111111 \
  --ip-permissions IpProtocol=tcp,FromPort=443,ToPort=443,PrefixListIds=[{PrefixListId=pl-63a5400a}]

Network ACLs cannot reference a prefix list, so if the subnet has a restrictive network ACL you have to read the service CIDR blocks out of the prefix list and write them as rules. There is no security group on a gateway endpoint, because a gateway endpoint has no network interface to attach one to.

Interface endpoints: a private address in your subnet

An interface endpoint is a different object. For each subnet you select, AWS creates an endpoint network interface in that subnet and gives it a private IP address from the subnet range. That interface is requester-managed: you can see it, you cannot manage it, and its IP address does not change for the lifetime of the endpoint. You pick one subnet per Availability Zone, never 2 in the same zone.

Because it is a real network interface with a real private address, the things you would expect to work do work:

• It carries security groups, and those rules control which resources in your VPC may talk to it. Forget to allow inbound 443 from your application's security group and every SDK call hangs.
• Its address is reachable from anywhere that can route to your subnet: a peered VPC, a transit gateway, a Site-to-Site VPN, a Direct Connect connection.
• It is billed. You pay an hourly charge for the endpoint in each Availability Zone where it is provisioned, plus a charge per GB processed.

The service is reached over DNS. Creating the endpoint gives you a regional name and one zonal name per zone:

vpce-099deb00b40f00e22.monitoring.us-east-2.vpce.amazonaws.com
vpce-099deb00b40f00e22-us-east-2a.monitoring.us-east-2.vpce.amazonaws.com

Nobody wants to rewrite every SDK client to use those. That is what private DNS is for. Enable it and AWS creates a hidden, AWS-managed private hosted zone containing a record for the service's normal public name, pointing at the private IP addresses of your endpoint interfaces. Your existing code calling monitoring.us-east-2.amazonaws.com now reaches the endpoint with no change at all.

Private DNS has one hard prerequisite that generates a steady stream of tickets: the VPC must have both enableDnsSupport and enableDnsHostnames turned on. Without them the option does nothing visible. And because the record lives in a private hosted zone served by the Route 53 Resolver, it works only inside the VPC. On-premises callers either use the endpoint-specific DNS names, which resolve publicly to the private addresses, or reach the Route 53 Resolver through an inbound Resolver endpoint.

Availability follows from where you put the interfaces. If you enable a single Availability Zone, the regional name resolves to that one interface for the entire VPC, including instances in other zones. That works fine right up until the zone holding the interface is impaired, and then the whole VPC loses the service. AWS recommends at least 2 zones per endpoint, and with more than one healthy interface it alternates between them round robin.

The boundary that decides most questions

Both endpoint types keep traffic on the AWS network. They differ on who can use them and what they cost.

The single question that separates them in a scenario is where the caller sits. Inside this VPC, and the destination is S3 or DynamoDB? A gateway endpoint is the cheap answer. Anywhere else, or any other service? Interface endpoint.

They are not exclusive, and AWS documents the combination as a cost pattern for S3: keep the gateway endpoint so in-VPC traffic stays free, add an interface endpoint so on-premises applications can reach S3 privately, and point the on-premises clients at the endpoint-specific DNS names. The console does this for you with the Enable private DNS only for inbound endpoint option, which routes only the queries arriving through an inbound Resolver endpoint to the interface endpoint and leaves in-VPC traffic on the free gateway path. Selecting it requires that a gateway endpoint already exist in the VPC, and you cannot delete that gateway endpoint while the option is on.

Endpoint policies: a gate, not a grant

Every endpoint for an AWS service can carry an endpoint policy: an IAM-language resource policy attached to the endpoint that decides which principals and actions may pass through it. If you do not attach one, AWS attaches the default, which allows everything:

{
  "Statement": [
    { "Effect": "Allow", "Principal": "*", "Action": "*", "Resource": "*" }
  ]
}

The mental model that keeps people out of trouble: an endpoint policy is a filter on the pipe, not a source of permission. It never overrides or replaces an identity-based policy or a resource-based policy. A request that crosses the endpoint needs a yes from the IAM policy on the caller, a yes from any resource policy such as an S3 bucket policy, and a yes from the endpoint policy. Replace the default with a narrow policy and you can produce AccessDenied for a role that has AmazonS3FullAccess, which is exactly the confusing ticket this design creates.

A useful pairing goes the other way as well. The endpoint policy limits which buckets can be reached through the endpoint; a bucket policy with an aws:sourceVpce condition limits which endpoints can reach the bucket. Use both and the bucket is only reachable from your network, and your network can only reach that bucket:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "Access-to-specific-VPCE-only",
    "Principal": "*",
    "Action": "s3:*",
    "Effect": "Deny",
    "Resource": ["arn:aws:s3:::finance-reports", "arn:aws:s3:::finance-reports/*"],
    "Condition": { "StringNotEquals": { "aws:sourceVpce": "vpce-1a2b3c4d" } }
  }]
}

Details that bite in practice:

• The policy must contain a Principal element. For gateway endpoints that element must be *, and you narrow the principal with an aws:PrincipalArn condition instead.
• Maximum size is 20,480 characters including whitespace.
• Not every AWS service supports endpoint policies. Where a service does not, full access is allowed through the endpoint and nothing you write changes that.
• Changes take a few minutes to take effect, so a test immediately after saving can mislead you in either direction.

PrivateLink for a service you own

Everything so far consumed an AWS service. PrivateLink also runs in the other direction: you can publish a service from your VPC and let other accounts consume it as an interface endpoint, with no peering, no shared address space, and no route between the 2 VPCs.

As the service provider you put a Network Load Balancer in front of your service, then create an endpoint service configuration that points at that load balancer. By default nobody can connect: you add permissions naming the specific AWS principals allowed to request a connection. AWS generates a service name like com.amazonaws.vpce.us-east-2.vpce-svc-071afff70666e61e0 that you share with consumers.

As the consumer, you create an interface endpoint for that service name. The connection request lands on the provider, who accepts or rejects it, either manually or automatically. The endpoint becomes usable when it reaches the available state, and the possible states are worth recognizing in a troubleshooting question: pendingAcceptance means the provider has not acted yet, rejected means they refused, expired means the request timed out.

Two things make this pattern work at scale where peering does not. The 2 VPCs never exchange routes, so overlapping CIDR blocks do not matter. And the connection is one-directional by construction: the consumer initiates, and the service cannot open connections back through the endpoint.

For high availability the provider enables the load balancer in at least 2 Availability Zones, since the endpoint service is only available in zones the load balancer is enabled in. Cross-zone load balancing is an alternative, with the caveat that a zone failure then takes out access from both zones, plus EC2 data transfer charges.

If the provider associates a private DNS name with the endpoint service and verifies domain ownership, consumers can keep calling the service by its existing name. Without that, consumers change their applications to use the endpoint DNS name.

When it does not work

Endpoint failures cluster into a small number of causes. Walk them in this order:

The security group on the endpoint interface. Interface endpoints get the VPC's default security group unless you pick another, and the default allows inbound only from resources in the same group. If your application instances are in a different security group, every call to the endpoint hangs until you add an inbound rule allowing 443 from them.

Ping proves nothing. Interface endpoints do not respond to ICMP echo requests. AWS says to use nc or nmap instead. Rebuilding a healthy endpoint because ping failed is a real and avoidable outage.

# Does the endpoint answer on the service port?
nc -zv vpce-099deb00b40f00e22.monitoring.us-east-2.vpce.amazonaws.com 443

# What is the service name actually resolving to?
dig +short monitoring.us-east-2.amazonaws.com

If that dig returns public addresses, private DNS is either off or the VPC is missing the DNS attributes.

The endpoint policy. An AccessDenied that survives a correct IAM policy, and that only happens for calls from inside the VPC, points at the endpoint policy every time.

The wrong endpoint type. On-premises callers, peered VPCs, and other Regions cannot use a gateway endpoint. This is not a configuration you can fix; it is a property of the mechanism.

Network ACLs. Traffic between your resources and the endpoint interfaces still crosses the subnet boundary. A restrictive network ACL needs rules in both directions, including the ephemeral port range for return traffic.

Quotas worth remembering

Exam tips

• "S3 or DynamoDB, from inside the VPC, at no extra cost" is a gateway endpoint. Any other service, or any caller outside the VPC, is an interface endpoint.
• The phrase "from our on-premises data center" or "from the peered VPC" rules out gateway endpoints entirely. That is the fastest elimination in this topic.
• Gateway endpoint means a route table entry. Interface endpoint means an ENI with a private IP. Every other difference follows from that one sentence.
• Only interface endpoints carry security groups. A question about restricting which instances may use an endpoint is an interface endpoint question.
• An endpoint policy never grants permission. When a scenario shows AccessDenied despite an over-broad IAM policy, look for the endpoint policy or a bucket policy aws:sourceVpce condition.
• Private DNS requires DNS hostnames and DNS resolution enabled on the VPC. This is the answer whenever "we enabled private DNS but nothing changed".
• Hosting your own PrivateLink service needs a Network Load Balancer plus explicit principal permissions. Gateway Load Balancer endpoints are for routing traffic to inspection appliances, which is a different job.
• One subnet per Availability Zone for an interface endpoint, and at least 2 zones in production or you have built a single-zone dependency for the whole VPC.
• Interface endpoints do not answer ping.

The decision rule to carry forward is short: if the caller is inside this VPC and the destination is S3 or DynamoDB, take the free route; otherwise pay for a private address. The next lesson keeps the same goal of private traffic but changes the destination from an AWS service to another network, where the question stops being "which endpoint" and becomes "how many connections am I willing to manage".