802.1Q Virtual Local Area Networks (VLANs)

802.1Q is an Ethernet standard as defined by the Institute of Electrical and Electronics Engineers (IEEE) that enables Virtual Local Area Networks (VLANs) on an Ethernet network. It uses the addition of a VLAN tag to the header of an Ethernet frame to define membership of a particular VLAN. Networking equipment that supports this standard is then able to maintain separation for the associated traffic at Layer 2.

Border Gateway Protocol

Border Gateway Protocol (BGP) is a routing protocol used to exchange network routing and reachability information, either within the same or a different autonomous system. This is known as Internal BGP (iBGP) and External BGP (eBGP), respectively. BGP is defined by RFC 4271 - A Border Gateway Protocol 4 (BGP-4).

Bidirectional Forwarding Detection

Bidirectional Forwarding Detection (BFD) is a mechanism used to support fast failover of connections in the event of a failure in the forwarding path between two routers. If a failure occurs, then BFD notifies the associated routing protocols to cause a recalculation of available routes, which reduces convergence time.

Requesting a Connection

The process to request a connection is self-service and fully documented in the AWS Direct Connect User Guide. You must first identify the AWS Region to which you wish to connect and the specific AWS Direct Connect location. You can then use any of the supported methods—AWS Management Console, Application Programming Interface (API), or AWS Command Line Interface (AWS CLI)—to request a new connection. AWS may require additional information at this stage, so you should monitor the primary email address for your AWS account for such an email and provide an appropriate reply.

Download Your Letter of Authorization

AWS will provision your port within 72 hours, after which you will be able to download the Letter of Authorization – Connecting Facility Assignment (LOA-CFA or simply LOA). This document provides specific demarcation details for your assigned port within the facility that will provide connectivity to AWS.

Cross-Connect to the AWS Port

The AWS Direct Connect locations each use their own defined process for arranging physical connectivity between different organizations within their facilities. These connections are known as cross-connects, and they are typically passive-fiber connections between the AWS equipment and either your equipment or that of your AWS Direct Connect partner. In some cases, the AWS Direct Connect location is listed as a campus of buildings. In that case, a cross-connect can provide connectivity from any site on the campus back to the AWS equipment. Note that a campus site does not imply that AWS has equipment within every building in the campus, so the campus should be viewed as a single “location” from an architectural and risk- assessment perspective.

Cross-connects must typically be ordered by an organization that has a physical presence within the AWS Direct Connect location and a commercial agreement with the location provider. If you are using an AWS Direct Connect partner to provide this remote connectivity, it will be the partner that places the order for the cross-connect. You will need to provide them with a copy of the LOA-CFA document to enable them to place the order.

Once the connection is in place, you should see a “link up” and good light levels being received at your equipment. The AWS Management Console will also reflect the status of the connection as either down or available. The AWS Direct Connect monitoring tools in the AWS Management Console can be used to validate the appropriate transmit and receive light levels for 10 Gbps connections. Figure 5.1 illustrates the components involved in establishing the physical connection to AWS.

Multiple Connections

You may choose to use multiple AWS Direct Connect connections to increase the resilience and bandwidth of your environment. These connections can be one of the following:

• At the same location, on the same AWS device
• At the same location, on a different AWS device
• At a different location

When provisioning multiple connections at the same AWS Direct Connect location, AWS will automatically provision these additional ports on different devices where possible. This enables a level of resilience to interface failure, device failure, or planned maintenance. You may also choose the “Associate with Link Aggregation Group” (LAG) option when creating the additional connections. This will ensure that the connection is allocated on the same AWS device such that it can be used with a LAG.

When additional connections are provisioned on different AWS devices or in different AWS Direct Connection locations, multiple connections provide a level of resilience against a potential location-level failure in addition to the benefits of connecting via different devices.

Link Aggregation Groups

A Link Aggregation Group (LAG) is a logical interface that uses the Link Aggregation Control Protocol (LACP) to aggregate multiple 1 Gbps or 10 Gbps connections at a single AWS Direct Connect location, allowing you to treat them as a single, managed connection. You can create a LAG from existing connections, or you can provision new connections. After you’ve created the LAG, you can associate existing connections (whether standalone or part of another LAG) with the LAG.

The following rules apply to LAGs:

• All connections in the LAG must use the same bandwidth. The following bandwidths are supported: 1 Gbps and 10 Gbps.
• You can have a maximum of four connections in a LAG. Each connection in the LAG counts toward your overall connection limit for the region.
• All connections in the LAG must terminate at the same AWS Direct Connect location and on the same AWS device.

All LAGs have an attribute that determines the minimum number of connections in the LAG that must be operational for the LAG itself to be operational. By default, new LAGs have this attribute set to 0. You can update your LAG to specify a different value. Doing so means that your entire LAG becomes non-operational if the number of operational connections falls below this threshold. This attribute can be used to prevent overutilization of the remaining connections.

For example, if you provision a LAG with four connections and you configure the minimum number of connections to be two, should there be a failure on two of the connections, the overall LAG operation status remains “Up” and passing traffic. Should a third connection fail, then the operational status will be changed to “Down,” even though a single connection remains available. This will enable any failover paths that you have in place to take over, avoiding the potential of the single remaining circuit becoming congested and impacting performance.

Hosted Connections

A hosted connection is available via AWS Direct Connect partners at bandwidth increments less than those available from a dedicated 1 Gbps or 10 Gbps connection. These are provided using a physical partner interconnect on which these hosted connections are provisioned. Partner interconnects have similar characteristics to dedicated connections; however, they can only be ordered by approved partners using a specifically-enabled partner console.

Hosted connections can be provided at bandwidths of 50 Mbps, 100 Mbps, 200 Mbps, 300 Mbps, 400 Mbps, and 500 Mbps. The provisioning process differs from dedicated connections. Because the physical interconnect is already in place, there is no need for you to download the LOA-CFA or arrange for cross-connects to AWS. The partner interconnects are shared and support multiple AWS customers’ hosted connections.

The process for ordering a hosted connection begins with speaking to an AWS Direct Connect partner. They will establish how the hosted connection will be delivered to your own location and with the appropriate interface types. Hosted connections are often used by AWS Direct Connect partners to enable rapid delivery of AWS connectivity to existing Multiprotocol Label Switching (MPLS) Virtual Private Network (VPN)-based solutions.

Once the method of delivery has been agreed on, you must provide your 12-digit AWS account number to the AWS Direct Connect partner, who will use this to provision the hosted connection on their interconnect. The hosted connection will be presented in your AWS account for the appropriate AWS Region in preparation for you to accept it. As part of the provisioning process, the AWS Direct Connect partner will have chosen an 802.1Q VLAN identifier on the interconnect. This may not be relevant depending on how you are consuming the service, and some partners may translate this VLAN to a different one at the point of service presentation to customers.

Accepting the hosted connection in your AWS account enables the billing for the associated port-hours and any data transfer to commence. This is charged on your monthly AWS bill and may be in addition to any fees the AWS Direct Connect partner charges for the service they are providing.

Public Virtual Interfaces

Public Virtual Interfaces (VIFs) enable your network to reach all of the AWS public IP addresses for the AWS Region with which your AWS Direct Connect connection is associated. In addition, public VIFs are also enabled for “Global” capabilities, which allows you to receive BGP announcements for all AWS public IPs globally. BGP communities are then used both to identify the source of the AWS prefix announcements and for you to control where your announcements are propagated within the AWS backbone network.

Public VIFs are typically used to enable direct network access to services that are not reachable via a private IP address within your own VPC. These include, but are not limited to, Amazon Simple Storage Service (Amazon S3), Amazon DynamoDB, Amazon Simple Queue Services (Amazon SQS), and the public endpoints used to provide AWS-managed VPN services.

When creating a public VIF with an address family type of IPv4, you must specify public IP addresses for both the Amazon router peer IP and your router peer IP. When creating (or adding a peering) to your VIF for the address family IPv6, AWS will always auto-generate the peering IP addresses using Amazon-owned IPv6 address space.

You must also specify the IP address prefixes you plan to announce to AWS over this type of VIF. This enables AWS to verify that you are the owner of these IP addresses and that you are authorized to announce them. If the IP addresses are listed in the regional Internet registries as belonging to another entity, then you will receive an email to the primary email address on your AWS account with further instructions. If you do not have your own public IP addresses to use for the peering and to announce, you should raise a case with AWS support.

The number of prefixes that AWS will announce to you can vary by region and whether your account is enabled for inter-region capabilities. In return, AWS will accept up to 1,000 prefixes from you.

Once AWS receives a BGP announcement from you, all network traffic from AWS destined to the announced prefix will be routed via AWS Direct Connect. This includes traffic from other AWS customers using public or Elastic IP addresses on their Amazon Elastic Compute Cloud (Amazon EC2) instances, traffic routed via Network Address Translation (NAT) gateways, AWS Lambda functions that make outbound connections, and more. You should configure your routers and firewalls appropriately to accept or reject this traffic per your own routing policies. AWS does not re-advertise customer prefixes to other customers that have been received over AWS Direct Connect public VIFs. The prefixes that AWS announces do change, and a current list can be obtained using the public ip-ranges.json file maintained and available at https://ip-ranges .amazonaws.com/ip-ranges.json.

Private Virtual Interfaces

Private Virtual Interfaces (VIFs) enable your network to reach resources that have been provisioned within your VPC via their private IP address. A private VIF is associated with the VGW for your VPC to enable this connectivity. Because this is a private VIF, the BGP peer IP addresses do not need to be public and therefore can be defined by you at the time of creation for IPv4 or automatically generated for both IPv4 and IPv6.

Private VIFs are used to enable direct network access to services that are reachable via an IP address within your own VPC. These include, but are not limited to, Amazon EC2, Amazon Relational Database Service (Amazon RDS), and Amazon Redshift.

When the BGP session comes up, your peer router will receive announcements for all Classless Inter-Domain Routing (CIDR) address ranges associated with your VPC. You are able to announce up to 100 prefixes to AWS over a private VIF, including a default (0.0.0.0/0) route. These routes are used by the VGW and can optionally be propagated into route tables within your VPC. The routes also contribute to CloudHub within the VGW, which enables you to route between multiple AWS Direct Connect private VIFs, VPN connections, and the attached VPC.

Because this is a private VIF, it is acceptable to announce any prefix regardless of whether it is considered public or private address space and irrespective of IP ownership. In a VPC, more specific routes always take preference over default routes, hence propagation or use of these routes in your VPC will take precedence over the default route to the Internet via the Internet gateway. Take care if you’re using non-RFC1918 (or other private address space) that might otherwise also exist somewhere on the Internet.

Direct Connect Gateway

A Direct Connect Gateway enables you to combine Private VIFs with multiple VGWs in the local or in remote regions. You can use this feature to establish connectivity from an AWS Direct Connect location in one geographical zone to an AWS Region in a different geographical zone. This is in addition to being able to use a single private VIF to access multiple VPCs in multiple AWS Regions (in the same account). Your router will establish a single BGP session with the Direct Connect Gateway and, from there, receive announcements for all associated VPCs. Note that the Direct Connect Gateway does not enable CloudHub between the associated private VIFs and VGWs. This is shown in Figure 5.2.

Hosted Virtual Interfaces

When creating a VIF (both public and private) you are able to choose the VIF owner. This can be “My AWS Account” or “Another AWS Account.” When choosing “Another AWS Account,” you are prompted to provide a 12-digit account number. All of the BGP configuration is still completed in the account that owns the AWS Direct Connect connection; however, when choosing another account, that VIF becomes a hosted VIF. The recipient of the hosted VIF must choose to accept it and, in the case of a private VIF, choose which VGW to associate it with. A hosted VIF results in all related data transfer charges being charged to the recipient account holder’s AWS bill. Note that the port-hour charges are always charged to the owner of the AWS Direct Connect connections.

Public Virtual Interface Configuration

For public VIFs, each one will have unique peer IP addresses but be configured to announce the same prefixes from your router. Depending on your network architecture, you may want AWS to prefer one connection for a specific range of IPs over the other for traffic flowing from AWS toward your routers. You can influence this using standard BGP configurations such as announcing a more specific prefix or by using AS_PATH prepending (if you are using a public ASN). More specific prefixes are preferred on the Amazon network as longest prefix match is the first criteria examined when processing a packet for path selection. A common configuration is to announce a supernet (such as /23) on both public VIFs and then a more specific subnet (for example, /24) on each individual VIF, depending on where that particular connection terminates on your network.

AS_PATH prepending is a mechanism where you artificially make the AS_PATH longer on one connection compared to the other by adding your own ASN multiple times to the path. AWS will always prefer the shortest path for a prefix if there are multiple options.

Private Virtual Interface Configuration

The same configuration options can be applied on private VIFs to influence how network traffic flows from a VPC to your network. You will create two or more private VIFs (one VIF per AWS Direct Connect connection) and associate them with the same VGW or Direct Connect Gateway to enable the different paths to the VPC. You can then use AS_PATH prepending or announcements of more specific prefixes to influence which path traffic will take traffic flowing from AWS to your network. On a private VIF, you are not required to use a public ASN to use AS_PATH prepending.

For influencing traffic flowing from your routers toward AWS within your own network, you will need to configure the routing appropriately using either local preference or similar options.

The options described all lead to an active/passive setup of some kind where one of the connections is chosen specifically for the network traffic from AWS. An alternative option is to run an active/active setup where both connections are used to carry network traffic. To enable this, you should ensure that the BGP configuration on both VIFs is the same. Within AWS, if the same prefix is seen with multiple identical paths via the same location, Equal-Cost Multi-Path (ECMP) is performed and individual traffic flows are hashed to one particular connection in turn. The result, given a reasonable mix of source and destination IPs or ports, is an effective method of load balancing across the various paths available.

Backup Virtual Private Network (VPN)

Backup VPN is covered in more detail in Chapter 4, “Virtual Private Networks,” but it is often used as a backup for AWS Direct Connect connections that use a private VIF. A VPN connection provides encrypted access over the Internet to resources within your VPC using their private IP address. Both AWS Direct Connect and VPN connections terminate on the VGW for your VPC.

When combining a VPN connection as a backup for AWS Direct Connect, AWS recommends that you use a dynamic BGP-based VPN to enable the most flexible and consistent behavior for all network traffic flows.

Routes received via AWS Direct Connect and a VPN are consolidated within the VGW. From here, path selection is applied to an outbound packet with AWS Direct Connect always prioritized over a VPN for any given prefix that is advertised over both connections.

It is possible to prioritize the VPN by advertising a more specific prefix over the VPN and a supernet over AWS Direct Connect. For example, you want to use a VPN for traffic destined to non-production servers in your data center that belong to a /27 range, but you want to have AWS Direct Connect as a backup. You can advertise the /27 over a VPN while advertising a supernet such as a /24 on AWS Direct Connect. In this scenario, VPN will be the primary and AWS Direct Connect will be the backup.

Virtual Private Network Over AWS Direct Connect

For some customers, there is an additional need to encrypt their network traffic over AWS Direct Connect due to their applications not natively encrypting in transit. The preferred way to do this is to enable encryption at Layer 4, for example, using Transport Layer Security (TLS). If you want to implement encryption at Layer 3, the simplest way to do this is to combine a public VIF with a managed VPN connection.

Public VIFs enable announcements of all Amazon public IPs. Within those public IPs are the endpoints that are provisioned when you create a VPN connection. Within your network, the public VIF over AWS Direct Connect provides you with an optimal path to those AWS endpoints (compared to routing over the Internet) and enables your VPN customer gateway to establish a connection using that path. In the event of an outage on AWS Direct Connect, your customer gateway should use an alternative path to reach the VPN endpoints such as over the Internet. This solution is illustrated in Figure 5.6, which shows the public VIF and two VPN tunnels.

Integration with the Transit Virtual Private Cloud Solution

AWS Direct Connect can be used with the Transit VPC Solution most easily by using a detached VGW. This is a VGW that has been created but not attached to a specific VPC. Within the VGW, CloudHub provides the ability to receive routes via BGP from both VPN and AWS Direct Connect and then re-advertise/reflect them back to each other. This enables the VGW to form the hub for your connectivity. Tagging the detached VGW appropriately enables it to be added automatically to the Transit VPC Solution configuration, and appropriate VPN tunnels are configured from the Amazon EC2 software routers.

Once these VPN tunnels are established, you should attach one or multiple private VIFs to the same VGW, which will cause the routes received from the Amazon EC2-based software routers to be reflected to your remote network over AWS Direct Connect. Conversely, your routes advertised over AWS Direct Connect will be reflected by CloudHub into the transit VPC Amazon EC2-based software routers via the VPN tunnels. An example deployment is shown in Figure 5.7.

Border Gateway Protocol Path Selection

Routing decisions for network traffic leaving your VPC occur first within the VPC based on entries in your route tables and then within the VGW. The VGW consolidates the available paths from all of the associated private VIFs and VPN connections before making a selection.

The path selection order is as follows:

1. Local routes to the VPC (no override with more specific routing)
2. Longest prefix match first
3. Static route table entries preferred over dynamic
4. Dynamic routes:
   1. Prefer AWS Direct Connect BGP routes
      1. Shorter AS_PATH
      2. Considered equivalent and will balance traffic per flow
   2. VPN static routes (defined on VPN connection)
   3. BGP routes from VPN
      1. Shorter AS_PATH

Private Virtual Interface Data Transfer

A private VIF is always associated with a Direct Connect Gateway or a VGW, which is then attached to a VPC. As a result, when network traffic flows from an Amazon EC2 instance in your VPC via the VGW, you are charged at the reduced AWS Direct Connect data transfer rate. This can be viewed as simply being metered on the VIF itself and the charges applied to the AWS account that owns the VPC and VIF. In the case of a hosted VIF, the ownership of the private VIF is separate from the ownership of the actual AWS Direct Connect connection. Hence, the data transfer charges are charged to the owner of the VPC/VIF while the port fees are charged to the owner of the AWS Direct Connect connection. The rate charged for this data transfer is calculated by referencing the pricing matrix on the AWS Website. It is a combination of the AWS Region and the AWS Direct Connect location being used.

Public Virtual Interface Data Transfer

Public VIFs on AWS Direct Connect provide a network path that applies to all network traffic that leaves the AWS non-VPC network. This will include both your network traffic as well as traffic potentially flowing from other AWS resources that you do not own. For example, if you browse a website that is hosted on AWS by a third party, the traffic from your network to and from that website will flow over your public VIF. In this situation, you will not be charged for any data transfer—the owner of the website will receive any related charges.

If your public VIF is used to access AWS resources that are owned by you, then the reduced data transfer charge is applied. This is applicable to any resources owned by the same account as the public VIF as well as to any linked accounts within the same billing family or AWS organization structure. In this way, a single public VIF can provide reduced data transfer pricing benefits to all of your related AWS accounts.

Consider the following example:

• Account A is the payer account for your organization.
• Account B is the owner of a public VIF.
• Account C has an Amazon S3 bucket containing a large volume of objects available for download.
• Account D retrieves those objects from your company network routing over the public VIF.

Assuming that Accounts B, C, and D are all linked to the payer Account A or are in the same AWS organization, Account C will observe data transfer charges at the AWS Direct Connect rate on their monthly bill.