The main charter of this case study is to find cost-effective alternatives to dedicated circuits connecting data centers with all remote Cisco locations, partner locations, and regional locations for day-to-day operations as well as in a disaster-recovery scenario.
This case study focuses on building a VPN solution that allows control of costs, yet incorporates requirements of business continuity.
Cisco IT has deployed cost-effective WAN VPNs to provide secure WAN connectivity to remote Cisco sites and partner locations, and for disaster-recovery connectivity on critical WAN links.
Cisco IT has deployed WAN VPNs in every region of the world and continues to evaluate its connectivity needs worldwide and employ WAN VPN technology when and where it makes business sense.
Cisco IT has chosen to build the VPN solution in deploying VPN services for WAN connectivity and disaster recovery.
Note
According to Craig Huegen, Cisco IT network architect, in the Cisco On Cisco Case Study Series, “Internet VPN is ideal where dedicated WAN connectivity is cost-prohibitive, or as a final disaster recovery mechanism between major sites, or to use common infrastructure between Cisco partners, and where Cisco maintains a presence at partner sites.”
Note
The source of the majority of the material in this chapter is the Cisco case-study document, “Cisco Uses WAN VPNs to Provide Cost-Effective Remote Site and Disaster Recovery Connectivity.” You can find it at http://www.cisco.com/web/about/ciscoitatwork/case_studies/routing_dl7.html. Much of the material in this chapter is verbatim from that source, but there are many areas where the source material has been adapted to apply to this particular Short Cut. Any inaccuracies introduced into the original work as a result of this adaptation are solely the responsibility of the author of this chapter.
WAN connectivity in the industry generally is achieved over leased circuits. Depending on the businesses’ need of connectivity, the leased lines are sized to provide adequate bandwidth bits per second (bps), availability and response times, as well as quality of service (QoS).
Several issues are associated with connecting global locations over the leased-lines network. High cost, unreliability, poor infrastructure, inconsistent costs for the same service, and time to get a leased line are some of these issues because the carrier infrastructure in the world is at different maturity levels.
Cisco Systems, Inc., is a global organization with more than 250 company sites worldwide. It is essential that Cisco IT provide and maintain adequate connectivity to every site and location. Sites can range from large campuses such as San Jose, California, and Research Triangle Park (RTP), North Carolina, with thousands of employees, to small remote field offices such as Anchorage, Alaska, with eight employees. Cisco IT’s goal is to provide high-performance, high-availability network connectivity as soon as it is needed at each site. The challenge is finding the right technical solution at the best price at each location. Internet VPN circuits for either primary or backup connectivity is the solution of choice for a small but increasing number of Cisco locations. There are three main areas where selection of Internet VPN replacing traditional WAN circuits is examined:
VPN connections as primary WAN links between Cisco offices
VPN connections as a disaster recovery for WAN links between global regions
VPN connections as WAN links between Cisco offices and partner locations
The main challenges are reliability and cost-effectiveness together for the transport types available. Adding disaster recovery requirements makes the challenge even harder.
The following are the requirements from a VPN solution that needs to incorporate disaster recovery:
Cost-effective and dependable alternative to dedicated circuits at remote company sites
Reliable disaster recovery capability for the most critical WAN links
Cost-effective alternative to dedicated circuits between Cisco and its partners that could be deployed quickly, using existing partner Internet access infrastructure
Cisco IT has deployed Internet (or WAN) VPNs for three types of applications:
WAN links between Cisco offices
Disaster recovery between global regions
Links between Cisco and partner locations
Cisco has deployed WAN VPNs as a cost-effective, dependable alternative to dedicated circuits at remote company sites. This would be site-to-site VPNs with IPsec providing secure WAN connectivity. Running compression on these links is a common practice as well.
Several alternatives exist for designing the topology for WAN link connectivity. Following are some of the options that can be deployed:
Primary connectivity with backup
Primary connectivity without backup
Backup connectivity with dedicated primary circuits
Figure 4-1 shows an example of WAN VPN connectivity with backup.
Cisco IT has deployed VPN connectivity with backup when high availability is critical, yet dedicated circuits are cost prohibitive.
Primary connectivity with backup consists of dual remote VPN routers and dual IPsec/GRE tunnels connecting to the VPN routers. Full redundancy is offered here because either the right or left leg of the VPN connectivity can experience outage and VPN connectivity still remains.
Costs for this still remain high, and as a result it is not widely deployed by Cisco IT.
Figure 4-2 shows an example of WAN VPN connectivity without backup.
Cisco IT deployed WAN VPN to provide primary connectivity to several remote field locations. Mainly because of cost reasons, as well as uninterrupted availability not being a critical requirement, a single VPN router with primary connectivity with a single remote ISP has been deployed. This situation
Is appropriate where uninterrupted availability is not as critical
Consists of a single remote VPN router connected to two head-end routers at the regional hub using two IPsec/Generic Routing Encapsulation (GRE) tunnels
Offers primary connectivity with limited redundancy
Has been deployed in several locations, including Anchorage and Honolulu in the United States, and in Costa Rica
For example, at the Cisco site in Costa Rica, a dedicated circuit back to San Jose would cost thousands of U.S. dollars, so the single WAN VPN with dual head-end is selected. Although the Internet connectivity available creates high latency levels, the WAN VPN is capable of supporting reliable service, even with voice. The Internet connectivity is a fraction of the cost of dedicated circuits, resulting in multifold savings in circuit costs with acceptable connectivity services. From a location such as Nairobi, Kenya, a WAN VPN link over satellite connection is used, which also creates high latency. Bandwidth is limited to 512 kbps, but it provides essential network connectivity.
Figure 4-3 shows an example of backup connectivity with dedicated primary circuits.
Unlike a traditional WAN link that requires a dedicated point-to-point circuit, a WAN VPN utilizes a local connection to the nearest ISP’s Point of Presence (POP). From there, the public Internet infrastructure carries the VPN connection to the other endpoint. Such site-to-site encrypted WAN VPNs offer the same benefits as dedicated WANs by ensuring private communications from one trusted site to another and providing multiprotocol support, high reliability, and extensive scalability. In addition, site-to-site encrypted WAN VPNs are cost effective, secure, and allow for greater administrative flexibility than legacy private WANs.
The following are the criteria for choosing backup connectivity with a dedicated primary circuit:
Is appropriate where dedicated primary circuit can be cost justified, but the cost of dedicated backup circuit cannot
Consists of single remote VPN router connected to a single head-end router at the regional hub through a single IPsec/GRE tunnel and VPN
Offers backup connectivity to dedicated primary circuit
Has been deployed in several locations, including Mexico City, Buenos Aires, and Rio de Janeiro
Cisco IT uses IPsec to provide data encryption over the WAN VPNs. However, IPsec does not support IP multicast over the VPN. IP Multicast is a bandwidth-conserving technology that allows applications such as video conferencing, corporate communication, and distance learning. For example, John Chambers, Cisco chief executive Office, announces quarterly results in company meetings; Cisco IP/TV streams live video, audio, and slides to WAN VPN sites. To allow multicast broadcasts to WAN VPN sites, Cisco IT uses GRE tunneling technology. This Cisco IOS software-based GRE over IPsec VPN design was chosen for maximum configuration flexibility that closely mimics traditional private lines, Frame Relay, or ATM services.
Cisco IT also deploys hardware acceleration because of application and traffic latency requirements.
Cisco IT continually evaluates the Cisco All Packet Network for reliability and fault tolerance. WAN links connecting major Cisco locations have been designed with redundancies to help ensure failsafe operations. In catastrophic situations, however, additional measures may be justified to protect these links. The same WAN VPN technology deployed for connectivity to smaller remote sites also can be utilized to provide disaster-recovery capabilities for WAN links between regional hub sites.
A WAN VPN is unlikely to be an appropriate disaster-recovery solution for all major Cisco locations because of cost of the high-bandwidth VPN connections and other factors. The business case for WAN VPN disaster recovery can vary considerably. The expense incurred to mitigate the risk of a complete WAN failure must be evaluated on a site-by-site basis. Among the business factors that should be considered when evaluating a site for WAN VPN disaster recovery are the diversity of existing primary and backup facilities, the impact of a catastrophic WAN failure, and the cost to implement or upgrade an ISP POP to accommodate the bandwidth needs.
Because connectivity during unpredicted disasters of unknown scale can affect the health or survival of a business, the following design considerations are important:
WAN links between major Cisco locations have been designed with redundancies.
Additional measures may be justified to protect links in catastrophic situations.
Cisco IT has deployed WAN VPN for disaster recovery between the United States and Europe, using existing Internet access.
Cisco IT identified the transatlantic WAN links connecting the United States and Europe as vital communication routes. Two OC-3 (155 Mbps) circuits provide redundant, diverse routes between New York and London and between Research Triangle Park and Amsterdam. Each circuit has sufficient bandwidth to handle the total traffic load between the United States and Europe if a circuit fails. In the unlikely event that both circuits fail, Europe would be isolated from the rest of the network. The potential impact of this event convinced Cisco IT that a WAN VPN disaster-recovery solution was justified. Furthermore, the cost to implement a WAN VPN solution would be minimal because the existing Internet connections in both Amsterdam and RTP were large enough (STM-3 and OC-3, respectively) to provide sufficient bandwidth for the disaster recovery VPN connection. A WAN VPN was deployed between these sites, as shown in Figure 4-4.
Cisco has deployed WAN VPNs as a cost-effective, dependable alternative to dedicated circuits between Cisco and partner locations. Some of the benefits of VPN extranet connectivity include
Eliminating the cost of WAN circuits used in traditional extranet connectivity
Eliminating hardware costs for internal clients
Accelerating implementation
Facilitating short-term extranet connectivity or fast location moves
Supporting partner telecommuters with user-based VPNs
Interestingly, mainly because of cost-effectiveness, the ratio of connectivity requests that Cisco IT receives for VPN versus leased-line connectivity is 5:1. However, partner connectivity is individually assessed based on price and performance trade-offs given the partner locations.
VPN links between Cisco and partner locations are deployed in approximately 50 locations, in one of two models:
Extranet remote LAN model
Interconnect model
A remote LAN is an extension of the Cisco network at a partner site. A managed Cisco router at the partner site terminates the transport connectivity from Cisco and connections to one or more managed switches at the partner location, as depicted in Figure 4-5.
Acting as an extension of the Cisco network at the partner site, the extranet remote LAN model allows printing of the files on the printers installed on the partner sites.
Currently, an extranet remote LAN model is used to connect about 20 partner sites where the router is managed by Cisco at the partner site terminating transport connectivity from Cisco. As Cisco IT utilizes the existing VPN connection, there is no additional cost of circuit leasing and managing.
With the Interconnect model, partners connect using their corporate LAN, which interconnects with the Cisco LAN, as shown in Figure 4-6.
A firewall at each side protects each company’s respective resources. Because of security policy enforcement, Cisco IT does not advertise or allow a partner internal network into the Cisco network; Cisco translates the partner IP addresses into Cisco addresses using Network Address Translation (NAT). The use of NAT provides obfuscation and provides an additional layer of security as well as manageability because each partner is uniquely addressed and identified, whereas in the remote LAN models, access is not at the IP subnet level but is limited to desktops that are physically connected to the remote LAN. Some sites incorporate the interconnect model as well as extranet remote LAN technologies, depending on the requirements of a connection.
The following is how connectivity is designed for the extranet remote LAN model:
Partners connect through their corporate LAN, which connects with the Cisco LAN.
Respective firewalls protect each company’s resources.
Cisco translates partner IP-addressing into Cisco addresses using NAT.
More than 30 partners currently use Interconnect VPN. Partners provide VPN routers and own and manage all equipment on their site. Cisco is responsible only for supporting the equipment at the Cisco location and for troubleshooting the connection between the two sites when needed.
Cisco IT has deployed WAN VPNs in every region of the world. Some of the advantages are outlined in this section, along with some important deployment pointers.
The following are benefits when deploying WAN VPN:
To provide cost-effective primary or backup connectivity, or both, to remote sites.
To deliver disaster recovery capabilities along the most critical WAN routes.
To connect partner sites to the Cisco internal network in a speedy and cost-effective manner.
VPNs are useful where they cost less than the dedicated circuits.
Sometimes a trade-off exists with more frequent outages and a greater difficulty in troubleshooting outages, especially when there are multiple ISPs involved in providing service suitable for business connectivity requirements.
Consider the following pointers when you are thinking about deploying a VPN WAN:
Monitoring considerations
Standardized configuration
Rely on trusted partners
Set expectations with extranet partners
WAN VPN may not always be the best choice
Total cost of ownership should determine choice
Latency issues
Network management systems that monitor physical interfaces are always recommended, but it is necessary to monitor IPsec tunnels or logical interfaces as well. Enabling keepalive with technology features such as Dead Peer Detection (DPD) is also essential in that it allows IPsec tunnel malfunction or up/down scenarios. DPD monitoring allows you to overcome the issue that an IPsec tunnel will remain up as long as the tunnel endpoint interface remains up. Other workarounds include utilizing features such as Virtual Tunnel Interface (VTI) as well.
Note
The Cisco Feature Navigator tool provides information on Cisco IOS features such as DPD and VTI:
Utilizing a Secure Shell (SSH) connection for troubleshooting (as long as the ISP connectivity is not disrupted) makes partner-side troubleshooting more manageable because secure access via backup is available when the primary connectivity link is disrupted. However, partners do need to monitor the private LAN where the IPsec tunnel terminates. Cisco IT does not request access privileges to the private LAN segments of the partner network.
Because partners might connect with less-reliable ISPs, given what is available geographically, it is important to set monitoring metrics inline with the SLA that the local ISP can provide. Having a uniform set of SLA monitoring is not prudent given the geographical diversity that exists in the ISPs around the globe.
It is important to have standardized configuration for the partner VPN routers. It helps with troubleshooting as well as security. Monitoring the checksum of partner routers alerts against any type of configuration alteration and is very important.
Providing global connectivity to a variety of partners almost mandates establishing relationships with ISPs that provide service across the globe as well. It may be advantageous to have a trusted and reliable ISP partner, or even their regional small ISP consortium members, provide connectivity to the partner network.
Cisco IT has benefited from leveraging regional managed service provider partnership in Latin America that helps with local escalation, language, and other regional intricacies.
Identifying key personnel at the partner site also helps with faster resolutions.
Cisco IT provides VPN connections only when the connection does not require continuous 100 percent availability but can tolerate occasional small outages. Hence, extranet partners may be eager to use VPN connectivity for cost reasons, but Cisco IT offers it when the business needs permit the VPN usage.
Cisco IT also monitors the change in business requirements from partners, and should the need arise, Cisco IT recommends changes to suit the new needs.
VPN may or may not be the best method for WAN connectivity, depending on the needs of the business. For example, VPNs are most cost effective where traditional leased lines (Frame Relay or ATM) are more expensive, but VPNs may be less reliable in those regions. Another example relates to Cisco IT individual network needs. Although VPN is stable in the continental United States, it is not currently a candidate for Cisco U.S. WAN connectivity because Cisco IT has built an extensive backbone where field locations are close to the nearest POP. The cost of a T1 Internet connection is approximately the same as a dedicated circuit to the nearest local hub, and Cisco IT is unlikely to reduce costs by using VPNs as backup within the United States. However, VPNs remain advantageous for extranet and disaster recovery applications. Extranet connections within the United States have been successful because the Internet infrastructure is robust and reliable. Global regions where Tier 1 or global providers have few or no locations, such as Latin America and Africa, provide greater challenges.
The general perception is that VPN is free, but the real capital cost of a VPN connection depends on the current Internet capacity at both endpoints. If both endpoints have large bandwidth capacity and router resources to handle IPsec encryption and decryption, VPN implementation does not result in a huge capital expense. If the bandwidth or endpoint routers need to be upgraded, capital costs need to be considered for the upgrade required. Failing to do that upgrade results in disruption of services that worked prior to IPsec VPN implementation.
In the case of Cisco IT, hardware installed where VPN tunnels would terminate already had adequate hardware resources with minor upgrades. As a result, building a VPN solution resulted in a cost-beneficial scenario. Additional costs, however, in provisioning and monitoring VPN solutions, were also included while calculating the total cost of ownership. Having a highly skilled IT staff familiar with the command-line interface (CLI) of IOS, troubleshooting tools, and proven techniques impacted the decision to build the VPN solution versus buying it.
Deployment costs of a VPN solution are applicable to both building and buying a VPN solution, and here project management available to Cisco IT proved to be a pivotal success point for timely and cost-efficient completion of the CPN solution.
Latency can vary widely, depending on situation and region. Latency is particularly high in Costa Rica and Nairobi, where satellite facilities are used. Even with excessive latency, however, users still can conduct business sending and receiving e-mail messages, accessing the web, and even viewing webcast, although at a slower rate. Costa Rica, for example, also supports voice traffic over WAN VPN.
Cisco IT realized that latency can sometimes generate help desk calls claiming that the connectivity is down or unacceptably slow. Hence, reliable monitoring and availability measurements were put in place to correctly address the remote user concern to the help desk calls.
Cisco IT continues to evaluate connectivity needs worldwide and employ WAN VPN technology when and where it makes business sense.
Upgrading remote WAN VPN sites from Cisco 3640 to newer Cisco 3745 routers as budget and time permits allows Cisco IT to meet more aggressive resource requirements for newer applications and increases in the user base at the remote locations.
Cisco IT is considering, on a case-by-case basis, extending disaster-recovery WAN VPNs to other regional hub sites, but no immediate deployment plans are in place.
Regional regulatory issues and concerns about quality of service will likely limit the use of WAN VPNs throughout the Asia-Pacific region in the near future and to more infrastructure-challenged areas and to extranet applications.
Cisco IT is building the business case for adding new extranet VPN endpoints to Cisco locations other than San Jose, California, where appropriate.
Note
For more information about the Cisco IT experience on WAN VPN and other technologies, visit http://www.cisco.com/go/ciscoitatwork
WAN VPNs have been shown to provide cost-effective, reasonably reliable connectivity. They are a good choice for remote field locations where dedicated WAN connectivity is cost prohibitive or availability is limited. WAN VPNs also can deliver economical connectivity as a final disaster-recovery mechanism between major sites. They can be a viable alternative to dedicated private-line service by using common infrastructure between Cisco and its partners, as well as where Cisco maintains a presence at partner sites.
Cisco IT made a decision to build the VPN solution as the resources and skill levels available, combined with project management that is key to successful completion, allowed them to do so. The business case of building or do-it-yourself was strengthened because of the existing hardware available as well as the ability of Cisco IT to dictate what hardware must be installed to connect into the VPN solution from anywhere in the globe. The Cisco IT help desk, provisioning and monitoring tools, techniques, and resources can scale in a cost-effective manner because of standardization of the solution architecture, connectivity models, hardware, software, and a well-defined security policy specific to each remote user group type.