Traditional Tools

In this section, we discuss traditional network troubleshooting tools, many of which you may be already familiar with.

AWS-Native Tools

In this section, we discuss AWS-native tools for troubleshooting, which provide additional insight to augment traditional troubleshooting tools.

Internet Connectivity

New VPCs do not have public Internet connectivity by default. User action is required to set up the appropriate requirements for Internet connectivity.

There are five requirements for connectivity to the Internet from an Amazon EC2 instance:

• A public IP address is assigned to an instance (note that for IPv6, all addresses are assigned by AWS and are public) or a Network Address Translation (NAT) gateway with a public IP in a public subnet.
• An Internet gateway is attached to the VPC.
• There is a default route to an Internet gateway in the route table on the public subnet. If a NAT gateway is used, then the default route on private subnets should be the NAT gateway instance.
• Outbound ports are open in the instance security group (80 and 443 for web traffic).
• Inbound and outbound ports are open in the subnet network ACL (80 and 443 outbound and ephemeral port range inbound).

Virtual Private Network

AWS provides a managed Virtual Private Network (VPN) service to allow for easy connectivity between on-premises environments and VPCs. After you have created your VPN, you can download the IP Security (IPsec) VPN configuration from the VPC console to configure the firewall or device in your local network that will connect to the VPN.

The following should be checked if there are issues with traffic over a VPN tunnel:

1. Verify that the VPN tunnels are connected. (See the following section for more details on Internet Key Exchange [IKE] phase 1 and phase 2 troubleshooting.)
2. Verify that the proper VPC is attached to the Virtual Private Gateway (VGW).
3. Verify that there are either propagated or static routes to the VPN subnets, with the VGW as the destination, in each subnet route table.
4. Verify that the subnet network ACLs and instance security groups are set to allow the traffic that you would like to flow over the VPN.

These steps can be optimized using a top-down approach. For example, if some traffic is traversing a VPN connection and other traffic is not, then you can skip the steps on troubleshooting the VPN tunnel connectivity and start looking at routing and security groups/network ACLs.

Internet Key Exchange (IKE) Phase 1 and Phase 2 Troubleshooting

In the event that the VPN tunnels are not established, IKE phase 1 followed by IKE phase 2 of the IPsec tunnel should be investigated.

If there are issues establishing an IKE phase 1 connection, then the following should be checked:

1. Verify that IKEv1 is being used instead of IKEv2; AWS only supports IKEv1.
2. Verify that Diffie-Hellman (DH) group 2 is being used.
3. Verify that the phase 1 lifetime is set to 28,800 seconds (480 minutes or 8 hours).
4. Verify that phase 1 is using the Secure Hash Algorithm (SHA) 1 hashing algorithm.
5. Verify that phase 1 is using Advanced Encryption Standard (AES) 128 as the encryption algorithm.
6. Verify that the customer gateway device is configured with the correct Preshared Key (PSK) specified in the downloaded AWS VPN configuration for the tunnels.
7. If the customer gateway endpoint is behind a NAT device, verify that IKE traffic leaving the customer on-premises network is sourced from the configured customer gateway IP address and on User Datagram Protocol (UDP) port 500. Also test by disabling NAT traversal on the customer gateway device.
8. Verify that UDP packets on port 500 (and port 4500 if using NAT traversal) are allowed to pass to and from your network to the AWS VPN endpoints. Ensure that there is no device in place between your customer gateway and the VGW that could be blocking UDP port 500; this includes checking Internet Service Providers (ISPs) that could be blocking UDP port 500.

If there are issues establishing an IKE phase 2 connection, then the following should be checked:

1. Verify that Encapsulating Security Payload (ESP) protocol 50 is not blocked inbound or outbound.
2. Verify that the security association lifetime is 3,600 seconds (60 minutes).
3. Verify that there are no firewall ACLs interfering with IPsec traffic.
4. Verify that phase 2 is using the SHA-1 hashing algorithm.
5. Verify that phase 2 is using AES-128 as the encryption algorithm.
6. Verify that Perfect Forward Secrecy (PFS) is enabled and that DH group 2 is being used for key generation.
7. Enhanced AWS VPN endpoints support some additional advanced encryption and hashing algorithms, such as AES 256; SHA-2(256); and DH groups 5, 14–18, 22, 23, and 24 for phase 2. If your VPN connection requires any of these additional features, contact AWS to verify that you are using the enhanced VPN endpoints. Typically, you must re-create the VGW of your VPC to move to the enhanced VPN endpoints.
8. If you are using policy-based routing, verify that you have correctly defined the source and destination networks in your encryption domain.

AWS Direct Connect

AWS Direct Connect lets you establish a dedicated network connection between your network and one of the AWS Direct Connect locations. Using industry standard 802.1q Virtual Local Area Networks (VLANs), this dedicated connection can be partitioned into multiple Virtual Interfaces (VIFs). This allows you to use the same connection to access public resources, such as objects stored in Amazon Simple Storage Service (Amazon S3) using public IP address space and private resources such as Amazon EC2 instances running within a VPC using private IP space, all while maintaining network separation between the public and private environments. VIFs can be reconfigured at any time to meet your changing needs.

An AWS Direct Connect connection can either be established directly within an AWS Direct Connect location or extended to your location through an AWS Partner Network (APN) partner. Some APN partners also offer hosted VIFs at sub-1 Gbps speeds. Note that these hosted VIFs are not full AWS Direct Connect connections and only support a single VIF.

The following are items to consider when troubleshooting an AWS Direct Connect connection:

• AWS Direct Connect requires single mode fiber.
• A VIF must have a public or private Border Gateway Protocol (BGP) Autonomous System Number (ASN). If you are using a public ASN, you must own it.
• For a public VIF, you must specify public IPv4 addresses (/30) that you own.
• For IPv6, regardless of the type of VIF, AWS automatically allocates a /125 IPv6 Classless Inter-Domain Routing (CIDR) for you. You cannot specify your own peer IPv6 addresses.
• There is a limit of 50 VIFs per AWS Direct Connect connection.
• There is a limit of 100 routes per BGP session. Advertising more routes than this can cause port flapping.
• AWS Direct Connect supports a Maximum Transmission Unit (MTU) up to 1,522 bytes at the physical connection layer. If your network path does not support this, then you should set an MTU value below 1,500 bytes on your router to avoid issues.
• Security groups and network ACLs must be configured to allow access.
• Route propagation needs to be enabled on each subnet route table for the routes learned through BGP to show up.

Security Groups

Security groups are implicit DENY. Unless a rule allowing incoming or outgoing traffic is created, traffic will not flow. This means that even if two instances are in the same subnet, they will not be able to communicate with each other unless there is a rule created allowing such traffic. Security groups are also stateful, meaning that if an inbound or outbound rule is created, it will allow the return traffic.

The following are items to consider when troubleshooting security groups:

• There is a limit on the number of inbound and outbound rules of 50.
• Up to five security groups can be added per network interface.
• Only ALLOW rules can be added to a security group.

Amazon VPC Flow Logs can be useful for troubleshooting security group-related issues. Traffic will be recorded as a rejected packet if there is not a rule in place to allow it.

Network Access Control Lists

By default, the network access control list (ACL) on a VPC is set to allow all inbound and outbound traffic. If network ACLs are set to be more restrictive, care must be taken to allow all required traffic. Note that network ACLs are not stateful like security groups—return traffic to an outbound port must be explicitly permitted with an ALLOW rule. For example, locking down outbound to port 80 and port 443 only in a subnet would also require an inbound ALLOW rule for ephemeral ports (1024-65535). A good understanding of network traffic flows and ports and protocols in use should be established prior to implementing network ACLs on a subnet.

The following are items to consider when troubleshooting network ACLs:

• Network ACLs are not stateful like security groups.
• Return traffic may need the entire ephemeral port range of 1024-65535 to be open.
• There is a limit of 20 inbound and outbound rules per ACL.
• By default, each custom network ACL denies all inbound and outbound traffic until you add rules.
• Rules are evaluated starting with the lowest numbered rule. As soon as a rule matches traffic, it is applied regardless of any higher-numbered rule that may contradict it.

Applications commonly communicate over a number of ports and often require an inbound rule for return traffic. Amazon VPC Flow Logs can be useful for troubleshooting network ACL-related issues. Traffic will be recorded as a rejected packet if there is not a rule to allow it or there is an explicit rule to block it.

Routing

Routing within a VPC is controlled by a route table attached to each subnet. Note that unless a route table is explicitly associated with a subnet, the main route table for the VPC will be used for each subnet. Knowing the caveats of routing and how routing within VPC works is beneficial to troubleshooting. The following are some common considerations when troubleshooting route tables:

• The most specific route that matches traffic in a route table is used.
• IPv4 and IPv6 routes are handled separately.
• The default local route of the VPC CIDR cannot be modified.
• When you add an Internet gateway, an egress-only Internet gateway (for IPv6), a VGW, a NAT device, a peering connection, or a VPC endpoint in your VPC, you must update the route table for any subnet that uses these gateways or connections.
• There is a limit of 50 non-propagated routes that you can have in a route table.
• There is a hard limit of 100 routes that can be propagated to a VPC route table. More general routes or a default route should be used if this limit is reached.
• If a custom Amazon EC2 instance is used as a router, then its elastic network interface needs to be added in the route table as a destination. Note that the instance must also be set to disable source/destination checking for traffic to flow.

Virtual Private Cloud (VPC) Peering Connections

A VPC peering connection is a networking connection between two VPCs that enables you to route traffic between them using private IPv4 addresses or IPv6 addresses. Instances in either VPC can communicate with each other as if they are within the same network. You can create a VPC peering connection between your own VPCs or with a VPC in another AWS account. In both cases, the VPCs must be in the same AWS Region.

AWS uses the existing infrastructure of a VPC to create a VPC peering connection; it is neither a gateway nor a VPN connection, and it does not rely on a separate piece of physical hardware. There is no single point of failure for communication or a bandwidth bottleneck.

The following are items to consider when troubleshooting VPC peering connections:

• VPC peering connections are not transitive. Traffic not in a destination VPC CIDR will not flow over a peering link.
• A VPC peering connection cannot be used to reach out to the Internet or to a VPN connection.
• A route needs to be added to each subnet route table with the remote VPC CIDR and the peering connection as the destination. This needs to be done on both sides of the VPC peering connection.
• There cannot be conflicting or overlapping VPC CIDR ranges.
• There is a limit of 50 VPC peering connections per VPC. This limit can be increased to maximum of 125 by opening an AWS Support ticket.
• Security groups and network ACLs need to be set to allow traffic to flow on the source and destination instances and subnets.

Connectivity to AWS Cloud Services

AWS Cloud services that reside outside of a VPC require a public IP address for access. This can be accomplished through a NAT gateway, public IP address, proxy server, or by setting up an endpoint on a VPC (if an endpoint is available for the service).

The following considerations should be checked when there are issues accessing AWS Cloud services:

1. There should be an Internet gateway, proxy, or VPC endpoint that enables connectivity from private IP addresses with the VPC.
2. If a VPC endpoint is used, there should be a route in the route table that has the VPC endpoint as the destination.
3. Security groups and network ACLs should be checked to confirm they are properly set to allow the instances and services to communicate.
4. IAM roles (if in use) allow communication with the service. If a VPC endpoint is used, then the IAM policy on the endpoint must be set to allow access. The IAM Policy Simulator tool is helpful in troubleshooting permissions issues.

Amazon CloudFront Connectivity

Amazon CloudFront is a global Content Delivery Network (CDN) service that securely delivers data, videos, applications, and APIs to viewers with low latency and high transfer speeds. Amazon CloudFront is integrated with AWS—both physical locations which are directly connected to the AWS global infrastructure, as well as software that works seamlessly with services including AWS Shield for Distributed Denial of Service (DDoS) mitigation, Amazon S3, Elastic Load Balancing, or Amazon EC2 as origins for your applications, and AWS Lambda to run custom code close to your viewers.

The following are some common items to consider when troubleshooting Amazon CloudFront connectivity problems:

• If you are using a custom DNS entry, then it must be a CNAME that points to your Amazon CloudFront distribution’s domain name.
• If the Amazon CloudFront origin is an Amazon S3 bucket, the objects must either be publically readable or have an Origin Access Identity (OAI) created and attached to the distribution with permissions assigned to the Amazon S3 objects.
• HTTP 502 status codes (Bad Gateway) indicates that Amazon CloudFront was not able to serve the requested object because it could not connect to the origin server. If 502 errors are seen, connectivity to the origin server should be confirmed.
• HTTP 503 Status Codes (Service Unavailable) status code typically indicates a lack of capacity on the origin server. If 503 errors are seen, capacity at the origin server should be confirmed.

Elastic Load Balancing Functionality

Elastic Load Balancing automatically distributes incoming application traffic across multiple Amazon EC2 instances. It enables you to achieve fault tolerance in your applications, seamlessly providing the required amount of load balancing capacity needed to route application traffic.

Due to the scalable nature of Elastic Load Balancing, the fleet of AWS-managed Elastic Load Balancing instances will grow and shrink to meet demand. This scaling requires allocating a sufficient amount of available IP addresses to the Elastic Load Balancing subnet. Failure to account for the scaling of an Elastic Load Balancing fleet can result in errors and the inability of the load balancer to balance traffic.

The following are some common considerations to take into account when troubleshooting Elastic Load Balancing:

• Verify that a public load balancer resides only in public subnets.
• Verify that the load balancer security group and network ACLs allow inbound traffic from the clients and outbound traffic to the clients.
• Verify that the load balancer target’s security groups and network ACLs allow inbound traffic from the load balancer subnet and outbound traffic to the load balancer subnet.
• Verify that the default success code for a health check is 200. (This should be modified if your application uses an alternate success code.)
• Verify that sticky sessions are enabled on the load balancer; stateful connections will not work correctly otherwise.

Domain Name System

Domain Name System (DNS) provides hostname to IP resolution. AWS creates a DNS server by default within a VPC. (This option can be disabled.) AWS has a managed DNS service called Amazon Route 53 that provides the ability to create public and private hosted zones. Private hosted zones are only available within a VPC, while public hosted zones are globally accessible throughout the world.

DNS entries have a TTL value for each DNS record within a domain that specifies how long a client can cache the values of a DNS query. These TTL values are only a suggestion, and they can be ignored by caching at intermediary DNS servers, operating systems, and individual applications. For this reason, DNS queries may take some time to resolve to the correct values, even after the TTL period has expired.

The following are items to consider when troubleshooting DNS issues:

• AWS allocates the second IP address in a VPC CIDR as a DNS server. If a custom DNS server is needed, then an alternate Dynamic Host Configuration Protocol (DHCP) option set can be created.
• If a DNS entry is not resolving properly after an entry update, then the TTL of a DNS entry may not have expired.
• DNS values can be cached by ISPs, operating systems, and applications, and they can return incorrect values even after a TTL has expired.
• Only CNAME records should be used to point to AWS Cloud services. Using A records can cause errors.
• EnableDnsHostnames and enableDnsSupport must be set to true for Amazon Route 53 private hosted zones.

The nslookup command is a useful tool for troubleshooting DNS issues to determine to which IP address a hostname resolves.

Hitting Service Limits

Every AWS Cloud service has some type of limit. There are hard limits, which cannot be increased, and soft limits, which can be increased with an AWS Support ticket. It is very important to understand these limits. A lot of troubleshooting time can be saved by recognizing when a service limit is the root cause of an issue. Each service’s page on the AWS website lists the limits. A subset of network limits can also be seen in AWS Trusted Advisor:

• Elastic IP address
• Amazon VPC
• Subnets per security group
• Internet gateway
• Active load balancers

Many services will show an error in the API response or AWS Management Console when trying to create or allocate resources once a limit has been reached.