In 2000, Cisco information technology (IT) began developing a consistent and supported global wireless networking architecture. During this process, IT recognized a growing number of non-IT deployments throughout the company, led by user demand for the benefits offered by wireless networking. These WLANs were purchased, deployed, and supported by local teams without IT support or supervision. This situation resulted in many inconsistent “gray IT” deployments, often with poor security and sometimes involving ready-to-use wireless solutions with no security. Most of these “DIY” networks used Cisco Aironet access points, but wireless products from other manufacturers were also identified. Even when the same products were used, software versions and configurations were often different.
The business model for deploying enterprise-class WLANs in the Cisco internal environment was based upon two underlying fundamentals:
The desire to embrace and showcase new technology where Cisco Systems led the industry
The realization of the real and measurable benefits that wireless networks would provide to the Cisco global workforce, a workforce that was already partly “mobilized” by the provision of laptops to all staff
The issue for Cisco IT was not whether WLANs should be deployed, because Cisco Systems had long since identified the many benefits offered by the technology, but rather determining how to cost-effectively maintain control, reduce overall support costs, ensure a secure wireless infrastructure, and still provide benefits to Cisco employees. The project team realized that WLANs would deliver productivity benefits. Additionally, Cisco already had a highly mobile workforce where almost every employee (in excess of 37,000) was issued a laptop computer. Most onsite vendors and contractors were also similarly equipped.
Potential mobility (and therefore additional productivity) was limited, however, because of one simple fact: Laptops users—indeed all users—were “tethered” to their desks by the traditional Ethernet cable. This simple fact alone negatively impacted the vast potential benefit that user mobility offered to the corporation.
The strategic value of wireless networking was characterized by five guiding principles:
Business value—. WLANs should be productivity tools, enabling greater mobility for Cisco employees.
Comprehensive entitlement—. Each Cisco employee, regardless of position, should have access to the global wireless network.
Ubiquitous coverage—. The WLANs should be built on global, scalable standards to provide a single, worldwide wireless network.
Security—. IT had to design a secure architecture that did not rely upon the then-prevalent, yet insecure, static Wired Equivalent Privacy (WEP) shared-key framework.
Ease of use—. User friendliness and a common user experience across all Cisco sites were essential for widespread adoption.
Cisco IT identified additional security principles, including these:
WLANs should support both privacy and access control through enterprise-class authentication and encryption capabilities.
Network attacks must be mitigated.
Rogue access points must be detected and remediated.
The selection of a suitable WLAN technology was an easy one. As the world’s leader in the manufacture of enterprise-class WLAN equipment, Cisco did not have difficulty in choosing the products to deploy. Cisco did, however, need to define, deploy, and provision a robust end-to-end solution.
When considering the architecture of your WLAN, your assessment must encompass many points. This section examines some of the factors that affected the enterprise WLAN deployment at Cisco Systems, as follows:
Topology
802.11 wireless networking standards
Client-to-AP ratio
Signal strength
Roaming
Radio cell architecture
Global naming standards
Cisco Aironet access points
Cisco Secure Access Control Server (ACS)
Early in the planning stage, the Cisco IT WLAN Architecture team decided that the WLAN would be a secondary network complementing the existing wired network (that is, a separate “overlay” network). Each large building would use a single Layer 3 domain within each building to help ensure session integrity for wireless devices moving within or between floors. Effectively, each building had a unique wireless subnet, where both the access points and the wireless devices shared IP addresses from a common Class C address pool. However, in line with prudent IP address management, smaller buildings with fewer than 20 or 30 users shared a common VLAN for both wired and wireless devices.
Additionally, at the time of deployment, the Cisco Aironet product line was based solely on a distributed, autonomous access point (or so-called “Intelligent AP”) model. Each access point was a unique, managed host with full intelligence and configurability. As such, the current global WLAN is a distributed model with over 3000 intelligent IOS access points in production. Figure 9-1 shows a basic topological diagram of the initial enterprise WLAN. The access points are connected directly to standard Layer 2 switches, and network management is provided by the Wireless LAN Solution Engine (WLSE) and the internally developed Enterprise Management (EMAN) toolset.
In 2000, the architecture standard called for Cisco Aironet 350 Series access points to be connected to the nearest access-layer switch, as shown in Figure 9-2. A separate cable provides console access to each access point to mitigate a loss of network connectivity, a practice that Cisco IT has standardized for all network devices. The console network is used for out-of-band (OOB) network management, configuration, and troubleshooting. Figure 9-2 shows how each access point is connected to the production data network and via a separate cable to the console network.
Because of ongoing developments in WLAN technologies, Cisco decided to redesign its enterprise wireless network in 2005. This project, known internally as the NexGen WLAN, will feature a combination of autonomous (IOS-based) access points and new centrally managed (LWAPP-based) access points, controlled and managed by WLAN controllers. Further information on the Cisco IT strategy can be found in the section “What the Future Holds” later in this chapter.
Note
Lightweight Access Point Protocol (LWAPP) is a protocol used to allow WLAN controllers to configure, manage, and control access points in the Cisco Centralized WLAN Solution. LWAPP introduces a split MAC, which allows real-time frame exchange and certain real-time portions of MAC management to be accomplished within the access point, while WLAN controllers handle authentication, security management, and mobility.
More detailed information on LWAPP and the Cisco Centralized WLAN Solution can be found at http://www.cisco.com/en/US/netsol/ns340/ns394/ns348/ns337/networking_solutions_white_paper0900aecd802c18ee.shtml or by going to Cisco.com and searching for “Understanding the Lightweight Access Point Protocol (LWAPP).”
At the time that the architecture team was designing the global WLAN, the only ratified standard was 802.11b, providing raw data rates of up to 11 Mbps in the 2.4-GHz frequency range. Therefore, this standard was adopted for the global enterprise wireless network.
Based upon internal Cisco IT policies and procedures, new products and standards must first undergo prudent and comprehensive testing and certification before they are used in the production environment. Shortly after ratification, the 802.11g standard was internally certified for use within Cisco by the architecture team. Today, therefore, Cisco is deploying 802.11g Cisco Aironet access points and client devices in its WLANs. 802.11g was selected over 802.11a because it also works in the 2.4-GHz frequency band and therefore offers seamless backward compatibility with the existing 802.11b network. Apart from limited lab and showcase sites, the 802.11a standard was not deployed in a widespread manner, but it will form part of the NexGen WLAN that is currently being designed by the architecture team; see the “What the Future Holds” section for more details.
Although 802.11g supports data rates of up to 54 Mbps in the 2.4-GHz band, these higher speeds are only available to 802.11g clients. Furthermore, 802.11g access points “step down” their speed when older 802.11b clients are associated to ensure backward compatibility. As such, it is not uncommon to find many 802.11g access points working at a maximum of 11 Mbps (effectively in 802.11b, or “legacy” mode). Such circumstances will decrease as the number of older 802.11b clients diminishes in line with the introduction of new laptops and replacement of the older devices.
After careful traffic analysis, Cisco IT built its architecture on a user-to-AP ratio of 25:1 would provide acceptable performance. At that time (early 2000), it was deemed unlikely that all 25 users would be accessing the WLAN at the same time and even more unlikely that they would all be simultaneously sending or receiving large amounts of data. Because the WLAN was an overlay network, those users who needed to use bandwidth-intensive applications such as network backups or video streaming were encouraged to use the wired network and not depend on the wireless network for these functions.
However, Cisco IT has found that adoption has been extremely high. Within 12 months of deployment, Cisco IT commissioned an internal “Voice of the Client” survey, which showed that 92 percent of staff were using the WLAN on a weekly basis; furthermore, 27 percent of users were relying upon the WLAN as their “primary or only network access medium.” Even with the limitation of the 802.11b data rate of 11 Mbps (or actual throughput of 6 Mbps), day-to-day performance has not been adversely affected and is deemed perfectly acceptable for the vast majority of user activity. Comments from users have been overwhelmingly positive.
Some Cisco buildings use wireless connectivity almost exclusively. This includes network backups, software downloads, video unicast, and Cisco IP Communicator (a software-based IP phone), in addition to standard web browsing, e-mail, and calendars. Rich Gore, Cisco IT project manager, says, “With quality of service now supported over wireless, I’ve been taking all my phone calls over the wireless network using Cisco IP Communicator, and it’s been working perfectly.”
Note
Users always have the option of manually connecting their laptops to the wired network if they so wish, but this practice is by no means standard for most users.
Moving forward, a lower user to AP ratio (approximately 12:1) has been recommended as reliance upon the WLAN increases and adoption has proven to be widespread. This topic is covered in more detail in the “What the Future Holds” section later in this chapter.
Cisco Aironet access points can broadcast up to 100mW (depending on the regulatory domain). When such high transmission power is used, it is possible for the WLAN coverage to extend beyond the originally desired areas, potentially reaching out into parking lots and public areas. After conducting tests, the architecture team established standards that call for using the minimum power to reach all areas within buildings, but never exceeding 20mW. That is, the “less is best” approach is taken. Access points are ideally configured to use 1mW, 2mW, 5mW, and so on, but never more than 20mW.
In some instances, directional antennas have been used to more narrowly focus the signal, reducing the power required to achieve full coverage. Where necessary, rather than increasing transmit power to exceed 20mW, additional access points are installed to cover “dead” spots.
To more accurately control roaming, the WLAN client software (in this case, the Cisco Aironet Client Utility [ACU]) was configured to roam only under certain circumstances—that is, when the current signal strength has dropped below a specified threshold or number of retries. This configuration reduces the tendency to reassociate to a new access point and helps avoid flip-flopping.
Each time the user switches from one access point to another, connectivity is momentarily lost, necessitating reauthentication. Numerous reauthentication requests can increase load on the authentication server, which can adversely affect service. This situation can be particularly notable in wireless voice applications, with clearly discernable “stutter” as the client reassociates and authenticates.
If cells overlap too much, continual switching (“flip-flopping”) is possible. Cisco adopted an overlap of about 15 percent (roughly 10 feet in most buildings) to minimize this possibility.
As we mentioned in Chapter 5, the 802.11 standard allows for devices to connect at various data rates depending on the RF environment. To minimize this effect, the architecture team locked the data rate at 11 Mbps. Thus, the user’s wireless connection will never “step down” but rather will associate to a different access point when it is far enough away from the original access point. This solution controls the roaming and avoids flip-flopping between access points, which in turn greatly assists in troubleshooting and predicting client behavior.
The policy for 802.11g cells is to permit data speeds as high as possible, but never less than 802.11b (11 Mbps). This results in the ability of newer 802.11g clients to associate with the latest model 802.11g access points at higher than 11-Mbps speeds in some circumstances. However, association rates at lower than 11 Mbps are never permitted.
Cisco uses a clear, concise, and consistent naming standard for all access points. This standard aids greatly in troubleshooting and also provides users and network engineers with useful information about their current access point.
The naming standard is as follows:
<site name>-AP<floor><AP letter>.cisco.com
For example, for the third access point on the second floor of a New York office, the access point name could be NYC-AP2c.cisco.com
Cisco IT has found that a consistent naming standard allows for easier management.
When originally deployed, the Cisco Aironet 350 Series Access Point was selected as the standard access point. The Cisco Aironet 350 Series was the most advanced, fully featured wireless access point available. It supported the 802.11b protocol standard (the most advanced at that time), which provides data rates of up to 11 Mbps. The Cisco Aironet 350 Series also supported inline Power over Ethernet (PoE), which greatly simplifies installation and reduces costs by eliminating the need for separate, dedicated power cabling to the main supply.
PoE allows the access point to draw power through its Ethernet cable, from the switch to which it is connected. In some circumstances, where certain sites did not have switches that supported PoE, Cisco IT used standalone “power injectors.” These devices sit inline between the network switch and the access point and “inject” DC power into the cable. This allowed Cisco IT to continue using PoE at all locations, even where they had older switches that did not provide PoE or that did not have sufficient power capacity to power all the access points required. Figure 9-3 shows how a power injector sits in between the access point and the switch.
Today, Cisco IT is expanding and enhancing its initial Cisco Aironet 350 Series deployments by installing Cisco Aironet 1000, 1100, and 1200 Series access points. These access points support new 802.11 standards and additional feature enhancements and options for modular and flexible WLAN deployments, including the centralized, controller-based architecture or the distributed autonomous access point architecture. At the time of writing, approximately 25 percent of the access points were the 1200 series. This percentage will rise to 100 percent with the NexGen WLAN.
The Cisco Secure ACS is used as the standard AAA server for the global WLAN and for other recently introduced services such as 802.1x-based port authentication for wired Ethernet ports in public areas and Network Access Control (NAC), part of the Cisco Self-Defending Network security strategy. Pairs of Cisco Secure ACSs were deployed at strategic locations worldwide.
The value of using a globally distributed AAA architecture instead of a single AAA server was highlighted by the WLAN deployment. Because of the greater load that a WLAN creates for AAA, due to authentications and reauthentications (as the client device roams from AP to AP), it was important to ensure that all users did not have to rely upon a single, centralized server. This would have introduced unacceptable delays for users in geographically remote areas. As such, at 13 different locations around the world, Cisco placed two ACS servers, in a load-balanced configuration, that served as AAA servers for that local geographical region.
The ACS servers are fully integrated with the Cisco Active Directory domain structure, enabling a single sign-on (SSO) capability. Effectively, AD user credentials are used not only for access to their laptops and wired network but also to provide transparent authentication to the wireless network. SSO has greatly reduced the client impact for users and has helped ensure a common, user-friendly experience across platforms and transport media. Users need only remember their normal ID and password for access to their laptop, the wired network, and the wireless network, and they only have to enter their credentials once each session regardless of the transport medium they are using.
To date, more than 3100 Cisco Aironet access points have been deployed worldwide, supporting more than 50,000 users. This includes over 37,000 full-time Cisco employees, as well as over 10,000 temporary, contractor, and vendor staff. A WLAN as widely used as this requires a robust management capability. Because a dedicated wireless management system was not available in 2000, the Cisco wireless network was managed through EMAN, an internally developed web-based enterprise-management framework. Today, Cisco IT also uses the CiscoWorks WLSE, a Cisco appliance for managing WLAN deployments.
Client management is a challenging area, and Cisco has implemented robust business processes to address it. Before 2004, all client devices were based upon Cisco-manufactured client adaptors, radios, and devices. However, the Cisco Client Extensions (CCX) is a technology licensing scheme that allows third-party manufacturers to produce equipment that supports Cisco value-added capabilities. With CCX, many third-party client devices and platforms have been introduced within the production environment.
To address this issue, Cisco made the decision to adopt third-party wireless software for all platforms. This adoption ensures that a common software application is used for all operating systems (Windows 2000, Windows XP, Linux, MacOS, and so on), regardless of the particular adaptor used in the relevant laptop (Cisco adaptors, Intel Centrino laptops, Macintosh PowerBooks, and so on).
The third-party supplicant also provides a consistent management toolset to allow for centralized profile management and configuration.
A centralized client management solution is also used to facilitate software distribution and updates.
Service dashboards, which are internal intranet websites, also provide service information, user communication, software, and self-service configuration utilities for all users. All Cisco staff can use dashboards for instructions on how to manually configure or update their systems. Because dashboards are based on standard HTML pages, they are platform agnostic and suitable for all platforms and clients that support HTTP.
Network devices, systems, and applications on the Cisco global network are managed according to levels of impact to the business. Service or support levels fall into four categories:
Priority 1 (P1)—. Immediate and severe business impact including revenue loss (actual, not postponed); inability to make or ship product; inability to develop code or product; inability to meet contractual, legal, or government-imposed processing deadlines; impact to external Cisco customers, partners, or supplier processes with negative implications for relations, market perception, or revenue; or engineering groups unable to work on a critical customer build or fix other critical account issues.
Priority 2 (P2)—. Adverse business impact including the inability of an organization (or organizations) within Cisco to perform daily operations such that it is essentially idle; or direct and critical impact to executives within the company, or to development, test, disaster-recovery, or staging environment for a P1 service or system.
Priority 3 (P3)—. Low business impact including the inability of multiple users to perform their daily tasks such that they are essentially idle; or impact to a single user under an approved, documented Service Level Agreement (SLA) requirement, or to a development, test, disaster-recovery, or staging environment for a P2 service or system.
Priority 4 (P4)—. Minor or no business impact to Cisco such as a question or new service request, or a problem that keeps one employee from performing part of a job function.
Within this support-level structure, Cisco Secure ACSs are managed as a P1 device because they are critical not only for WLAN access, but also for NAC, an element of the Cisco Self-Defending Network security strategy. The wireless network was originally managed as a P4 because it was considered a secondary network to the wired network. However, because of widespread adoption and usage within Cisco, support for the WLAN has become equivalent to P2. Cisco envisions that the NexGen WLAN, based upon more advanced and intelligent wireless networking technologies, will be formally supported on a P2 basis.
Cisco has a four-tier support model, as follows:
Tier 1: Frontline Global Technical Resource Center (GTRC)—. This is equivalent to a standard internal helpdesk. Agents are familiar with the most common problems and work from prepared scripts and troubleshooting guides. Each GTRC hub has a nominated wireless LAN expert who is more familiar with the solution than his colleagues.
Cases that are handled at this level are usually client configuration issues or the initial reports of service outages. Problems that cannot be solved by the GTRC are escalated to Tier 2 support.
Tier 2: Cisco IT WLAN network operations team—. These Cisco IT engineers are responsible for ongoing network and infrastructure support. The WLAN subteam is made up of engineers who usually have several years of experience supporting the solution and, in many cases, were directly involved in the original deployment and design. The IT WLAN network operations team has access to the access points, switches, routers, AAA servers, and WLAN controllers that make up the solution. This team also includes virtual members from the Cisco dedicated security organization and hosting teams (responsible for the AAA and Active Directory servers).
Cases that are handled at this level are usually AP or controller configuration issues, service outage problems, requests for enhanced coverage, and so on. Problems that the IT WLAN network operations team cannot solve are escalated to Tier 3 support.
Tier 3: Cisco IT WLAN architecture team—. The IT WLAN architecture team is made up of several senior design engineers and solutions architects. Members of this team designed the original solution and have continued their work on evolutionary change and development over the past five years. This team holds the most technical, business, security, and program management experience on the Cisco solution.
Cases that are handled at this level are usually fundamental design or architecture issues, requests for new services or capabilities, and new product or solution implementation. If a problem cannot be handled at this level, it is usually a result of a product bug and is escalated to Tier 4 support. This is a rare occurrence because most issues that are escalated this high relate to solution development rather than bug fixes.
Tier 4: Technical Assistance Center (TAC) and Wireless Networking Business Unit (WNBU)—. The TAC is the top level of support within Cisco and for Cisco customers. Cisco IT can also escalate directly to the WNBU within Cisco. Only officially noted bugs are escalated to this level.
A team of three and a half full time equivalent (FTE) staff makes up the Tier 2 IT WLAN network operations staff. Note that this effort is spread over several people in several countries but that the combined total is equivalent to 3.5 FTE.
A team of two and a half FTE makes up the Tier 3 IT WLAN architecture team. This includes the global program manager responsible for enterprise wireless strategy and architecture.
Cisco prices each GTRC support call at US$25 per call. This results in annualized cost of frontline Tier 1 support of US$318,900.
Cisco budgets US$120,000 per annum as the fully loaded cost of an FTE. This cost includes salary, assets, workplace costs, business costs, and so on, and is not indicative of salary alone. This results in annualized cost of second-line Tier 2 support of US$420,000.
Because of the nature of the Cisco business and the maintenance of a Tier 3 architecture team, Cisco does not include these costs in the day-to-day annualized support costs. Cisco believes the maintenance of a dedicated architecture team is not indicative of a typical enterprise because not all corporations are based in the networking industry.
This results in a total annualized cost of support as reflected in Table 9-1.
Several enhanced services are available today, including support for wireless voice services and global guest networking. The enhanced services are facilitated by the use of several SSIDs and wireless VLANs, with differing security settings based upon the target devices. Figure 9-4 displays the various SSIDs used by Cisco to provide enhanced services, such as wireless voice and guest WLAN networking. Two production SSIDs are also used with different encryption methods: one with WPA and one with Cisco TKIP. This ensures that older devices that cannot support WPA are still provided with an SSID that they can use.
Wireless voice services are provided to Cisco employees by the use of the Cisco Wireless IP Phone 7920, a key component of the Cisco AVVID Wireless Solution and Cisco IP Communicator, a Windows-based Softphone application.
The Cisco Wireless IP Phone 7920 is a WiFi-based (802.11b) phone that offers employees the ability to carry their extension with them as they move about Cisco premises. Many highly mobile users have adopted this device because it allows them to keep abreast of their voice communication services, even when away from their desk.
Cisco IP Communicator is similar in concept to the Cisco Wireless IP Phone 7920, but it uses a virtual software-based IP phone that is set up and configured on the user’s laptop. This allows users to access their extension, regardless of location and even when outside of Cisco sites by the use of VPN technology.
Wireless voice services are provided by a dedicated SSID and wireless VLAN, configured with support for QoS (802.11e and WMM, or Wireless MultiMedia, protocols) and fast secure Layer 2 roaming (provided by Cisco Centralized Key Management [CCKM]).
Wireless Guest Networking services are provided by Cisco IT to enable visitors to access the Internet while at Cisco sites. The solution is based upon a combination of the existing WLAN infrastructure, the Cisco Building Broadband Services Manager (BBSM), and an internally developed web portal for employee self-service access code generation.
All access points within the Cisco global network broadcast a guest networking SSID, which is configured with open authentication and no encryption to ensure that any client device can associate. When a visitor associates to the guest networking SSID and launches his browser, his HTTP session is automatically routed to a welcome page and portal. The visitor must read and accept the Cisco acceptable use policy and enter a preprovisioned access code. Upon validation, the visitor is then rerouted the Cisco demilitarized zone (DMZ) via a GRE tunnel to protect the production network. Effectively, from the guest’s point of view, he is immediately provided with seamless Internet access, and he has no awareness of the intervening and transporting network between his location and the DMZ.
Note
DMZ is originally a military term denoting a semi-safe area around a base or border where military (and therefore enemy) activity is controlled. In the networking world, this term was adopted to describe the area of an enterprise network that lies between the Internet and the internal enterprise network. It is where the enterprise typically places its security apparatus and gateways to the Internet. A firewall or a router usually protects this zone.
GRE is a tunneling protocol developed by Cisco Systems that can encapsulate a wide variety of protocol packet types inside IP tunnels.
Access codes are required to satisfy the Cisco requirement for auditing and IT forensics. Were a visitor to undertake illegal or unfriendly activity, the behavior could be tracked back to a particular IP address, which in turn is associated with a particular access code. Because each access code is generated on an as-needed basis and associated with a particular visitor, Cisco security and legal departments can ascertain who was using a particular IP address at any particular time.
Access control, and the use of access codes, is provided by the BBSM. However, to avoid unnecessary administrative overhead, Cisco IT developed an internal tool to allow Cisco users and staff to generate access codes for their own visitors. Effectively, Cisco has empowered its own staff with the ability to create access codes when they expect visitors. This removes unnecessary support burden from IT and administrative staff.
Access codes are therefore available for creation at the internal intranet page hotspot.cisco.com. Any Cisco employee can access this page and, after authenticating oneself as a Cisco employee, generate one or more access codes. Figure 9-5 illustrates how a Cisco staff member can generate the access codes, which are in turn provisioned on the BBSM, which in turn acts as an access portal to the Internet for the guest.
In 2000, during the initial deployment, the Cisco security architecture was based upon a combination of Cisco LEAP, for authentication, and Cisco Key Integrity Protocol (CKIP), for data integrity (encryption). However, as the industry, solutions, and threats evolved, Cisco further strengthened the security of its internal WLAN.
In 2005, Cisco replaced LEAP with Extensible Authentication Protocol-Flexible Authentication via Secure Tunneling (EAP-FAST). EAP-FAST further secures authentication by ensuring that all user credentials and passwords are passed from the client to the authenticators via a strongly encrypted tunnel. For more information about EAP-FAST, visit http://www.cisco.com/en/US/netsol/ns339/ns395/ns176/ns178/netqa09186a00802030dc.html or visit Cisco.com and search for the keyword EAP-FAST.
Additionally, and in line with Cisco IT’s policy of adopting open, cross-industry standards (where applicable and where Cisco does not provide enhanced value-added alternatives), WiFi Protected Access (WPA) was adopted as the encryption protocol for data integrity.
The Wireless LAN Solution Engine (WLSE) provides radio-based rogue AP detection and has been integrated into Cisco IT’s help desk case generation system. Additionally, an internally developed tool is used for network-based (that is, wired) scanning. This tool regularly scans Class C IP subnets, searching for devices that satisfy certain criteria and may be rogue access points. Based upon so-called “TCP port fingerprinting” and other holistic logic, the tool compares all devices it detects with the database of Cisco IT installed access points. Where a device is not already listed as a Cisco IT device, it is flagged as “interesting,” and a case is automatically generated. This case, in turn, is routed to the Tier 2 support team for investigation.
To facilitate the deployment, Cisco IT assembled a global program management team under the direction of a global program manager. Representatives were selected from each of the four regions worldwide: Americas, APAC (Asia Pacific), EMEA (Europe, Middle East, Africa), and SJ (San Jose Headquarters).
Responsibility for deployment within each region was delegated to a regional project manager and local team. Region-specific project managers determined and managed an implementation schedule within their own region. Local teams communicated progress at a weekly global deployment meeting. Several sites were deployed concurrently across and within regions. Serial installation by one global team might have taken years. Instead, almost all sites were deployed within a four-month timeframe in 2000, with the exception of India, which was delayed because of local regulatory issues with 802.11 standards.
The global program management team recognized that using Cisco employees exclusively to perform WLAN installation tasks was not the most cost-effective use of resources. Instead, where possible, vendors were hired for the bulk of the work. These vendors had to meet a minimum set of requirements established by the global program management team. The local contractors had to have a previous trusted relationship with Cisco and had to have wireless certified engineers. An emphasis on competitive bidding helped to minimize capital investment. Each local team selected their contractors based on their familiarity with the local market.
The global program management team established a guideline for the deployment process to be followed worldwide. The first step in deployment was the site survey. A formal and well-defined site survey was undertaken at each site, or on large campuses with several buildings with identical floorplans, at one building only, with the same results being applied to each identical building.
In many locations, trusted vendors performed the site surveys, while in some locations, such as San Jose, Cisco IT employees undertook the process themselves.
After the site survey was complete, local contractors (different from the site survey firm) installed the cabling and physically placed, secured, and connected the Cisco Aironet access points. Each access point was provided with two cables: one for data connectivity and one for console access.
After the access points were cabled and powered (via PoE), they were configured. Before distribution to each site, the access points were preconfigured with a “generic” configuration that allowed Cisco IT to connect and push the final production configuration. This was known as the staging phase, and it allowed Cisco IT to preconfigure and update firmware before shipping equipment to each site. This configuration was in compliance with the global design specifications established by the architecture team. Most critical were the IP address, channel assignment, and transmit power settings. Using generic and standardized access-point configurations helped to ensure consistent access-point settings across the entire deployment, simplified troubleshooting, and provided Cisco IT with greater control of individual access points.
Following configuration, the same contractor who performed the site surveys returned to conduct post-installation acceptance tests in each building. Dummy user accounts with limited access rights were provided, which enabled the contractors to test basic WLAN authentication and services. The globally consistent and clearly defined acceptance tests included the ability to roam from access point to access point and transfer a file at a minimum designated speed. Tests also helped to ensure the correct overlap between access point cells and verified that there were no dead spots.
At the time of the global WLAN deployment, distribution of Cisco Aironet 802.11b WLAN client adapters (wireless cards) presented a significant challenge. Cisco had to smoothly distribute more than 35,000 wireless cards in a timely and controlled manner across 400 sites in approximately 100 countries. Not only did the cards represent a significant percentage of the total program cost, but Cisco IT also needed to update each card with the latest firmware before delivering it to each user. Furthermore, it was important that the cards were not distributed to users until their sites were completed and had successfully passed the post-installation acceptance test.
In most locations, the task of inventorying, verifying firmware level, and distributing cards and operating instructions to users was performed during the staging phase and assigned to the same vendor that performed acceptance tests. Cards were shipped to the local vendor’s distribution center where the correct firmware level would be verified or updated as appropriate. When vendors carried out the post-installation acceptance tests, they distributed the cards to local users, along with user instructions, FAQs (Frequently Asked Questions), and introductory collateral from Cisco IT. Setting user expectations and providing comprehensive information were critical to minimizing support calls.
Today, Cisco uses both Cisco Aironet access points and Cisco compatible client devices in its network. Using these devices provides Cisco employees with a variety of licensed Cisco infrastructure innovations and enhancements for its WLANs, including advanced enterprise-class security, extended air RF management, and enhanced interoperability.
Cisco maintains a full-time global solutions program manager for enterprise wireless strategy and architecture. This individual is responsible for leading the IT WLAN architecture team (which also provides Tier 3 support where necessary). The team meets on a weekly basis and discusses ongoing solutions development and enhancement, operational issues, and any current projects in the wireless technology domain.
Cisco IT believes that the global WLAN solution provides positive productivity benefits in excess of US$50M per annum. Internal Cisco IT studies show average productive time savings of over 30 minutes per day. However, to ensure a more conservative approach, Cisco has calculated the business value of the WLAN on the assumption of 50 percent of users saving only 10 minutes per day.
Cisco Systems, Inc. commissioned an independent study on WLAN benefits in 2003. Based on a survey of over 300 U.S. organizations with more than 100 employees, it was found that WLANs typically provide average daily time savings (and therefore productivity) of 90 minutes. This in turn is equivalent to US$14,000 per employee, per annum[1]
Rather than adopting these figures (which would have resulted in projected savings of several hundred million dollars when considering over 38,000 regular employees), Cisco IT took a considerably more conservative and financially prudent approach.
First, Cisco calculated the cost of employee time, as follows:
230 work days per year
96,600 work minutes per year (7 hours per day)
Average cost of $120,000 per employee per year (salary, workplace resources costs, and so on)
120,000 / 96,600 = 1.2422
Average cost per work minute: $1.24 / minute
Cisco IT then calculated the benefits that would result if each employee were to save 10 minutes per day. This resulted in a figure of over 100 million dollars:
10 minutes saved * $1.24 * 230 work days = $2,852 per employee
38,000 employees, productivity improvement =US$108,376,000
This figure is, of course, unreliable. Cisco IT therefore undertook a normalization process and reduced the projected time savings. To ensure financial prudence, Cisco IT revised their estimates downward. They now assume only 50 percent of their daily WLAN users saved and used 10 minutes of productive time per day:
10 minutes saved * $1.24 * 230 work days = $2,852 per employee
19,000 employees * 2,852 = US$54,188,000
Experience, user reports, and observed behavior show that these figures err on the side of caution. However, even with this statistically conservative and financially prudent approach, Cisco IT has shown that the global WLAN solution not only paid for itself within six months of deployment but also has been positively affecting the Cisco bottom line to the tune of tens of millions of dollars for the past five years. Cisco IT expects this trend to continue as users’ reliance on the WLAN increases and additionally enhanced services are added to the solution.
Cisco IT is currently undertaking a major upgrade of the global WLAN. As the business has come to realize the benefit of wireless connectivity, the WLAN is being redesigned from the bottom up, adopting a proactive business value philosophy rather than a simple technology deployment approach. The team has been tasked to tackle the wireless LAN in a holistic manner, taking into account not only new products (such as the Cisco WLAN controllers and LWAPP access points), but also concepts such as fully integrated security with the Cisco self-defending network security strategy, a converged management solution, improved stability, and resilience in addition to data and voice capacity, outdoor coverage to ensure seamless roaming on campus sites, and a raft of additional features, enhancements, and evolutionary development.
The number of access points will be approximately doubled, providing a 100 percent improvement in user to AP ratio, from the current 25:1 ratio to approximately 14:1. This is essential for robust wireless voice services, increased granularity in wireless rogue AP detection and Intrusion Detection Systems, and greater wireless traffic load—all features and characteristics of Cisco’s internal WLAN today.
This section describes other changes introduced by Cisco IT’s NexGen WLAN:
Modular architecture
Enhanced security
Location-based services
Outdoor wireless
Because of the large number of Cisco sites and their varying sizes (from large multibuilding campuses with thousands of users to small, shared-tenancy sales offices with five or fewer staff), the Cisco WLAN upgrade plan includes a combination of the Cisco centralized WLAN solution (based on LWAPP access points and WLAN controllers) for large- and medium-sized sites, along with the Cisco distributed WLAN solution (based on intelligent, IOS-based access points) for small and very small sites where local controllers are uneconomical. The flexibility of this solution allows Cisco to tailor its internal global solution to all kinds of sites, from campus sites with thousands of staff to small, regional sales offices with five or fewer users.
Figure 9-6 provides a snapshot of the NexGen WLAN architecture. Large campus buildings are fitted with LWAPP access points. Buildings are logically grouped into clusters, and dual redundant WLAN controllers are used to manage the access points in each cluster. Wireless coverage is provided outside using Cisco outdoor mesh access points. The outdoor mesh network is provided between buildings on large campus sites to allow seamless roaming from building to building and to support enhanced wireless voice services. Medium to large remote offices are also fitted with LWAPP access points, and dual redundant WLAN controller appliances are installed locally. For small offices, IOS access points are used. Finally, WLAN management is provided by both the WCS and Wireless LAN Solution Engine (WLSE) that are centrally located at regional data centers.
The security framework for the Cisco internal NexGen WLAN will be based on the recently ratified 802.11i protocol. Authentication will continue to be provided by EAP-FAST, a tunneled authentication protocol that protects authentication exchanges in a strongly encrypted tunnel. Data integrity will be provided by WPA and WiFi Protected Access 2 (WPA2), with the incremental introduction of Advanced Encryption Standard (AES) capable devices.
The integrated Wireless Intrusion Detection System will be used to proactively monitor, detect, and isolate wireless security threats, including rogue access points and well-known wireless hacking attacks. The latter is a fundamental feature of the Cisco centralized WLAN solution, itself part of the Cisco Unified Wireless Network solutions family. To learn more, visit http://www.cisco.com/en/US/products/ps6306/prod_brochure09186a0080184925.html or go to Cisco.com and search for the keyphrase Cisco Unified Wireless Network.
Finally, third-party scanning utilities will be used for wired network scanning; this is especially important as a tool to reduce false positives and to assist with rogue AP detection in smaller sites and “air gapped” locations, where there are fewer access points to undertake active over-the-air scanning.
The Cisco WLAN Location Appliance will provide robust location-based services (LBS) such as asset tracking to assist in E911 applications. Combined with the use of 802.11-based wireless asset tags, this will allow Cisco IT to identify, locate, and track high-value assets in real time, down to a particular room and usually within five meters of accuracy.
Cisco plans to extend the enterprise WLAN such that it will provide coverage outdoor between buildings in its large campus sites. This coverage will be achieved with the use of the Cisco new Aironet 1510 outdoor mesh access point. The use of mesh technology will avoid the necessity of cabling each outdoor access point and will ensure seamless self-configuration and optimization.
The outdoor coverage will be a logical extension of the indoor WLAN and will be protected with the same level of robust security features.
Outdoor coverage will extend the capabilities of the enterprise WLAN and also ensure seamless, building-to-building roaming, which is especially important for wireless voice features.
Cisco Systems, Inc. deployed a global WLAN in 2000, and within 18 months, 27 percent of their staff were using it as their primary access medium. Ubiquitous coverage and comprehensive entitlement dramatically increased the uptake of the solution. Careful project and program management were adopted during the deployment, and the global network was deployed in 400 sites in four months. Security has continued to evolve in line with industry trends. Cisco believes the global WLAN has resulted in real productivity benefits in the tens of millions of dollars. Enhanced services such as wireless voice and guest WLAN networking have added to the success of the solution.
Cisco IT is undertaking a major global redesign of its solution in late 2005, and the NexGen WLAN will be based on the Cisco integrated wireless network family, including the Cisco centralized WLAN solution for large and medium sites and the Cisco distributed WLAN solution for small sites. Significant additional enhanced services are being planned, including outdoor mesh wireless coverage, location-based services for asset tracking, and significantly improved security with integrated wireless intrusion detection services.
The Cisco solution continues to evolve and provide real-world, tangible, everyday benefits to every Cisco employee in every office around the globe.
| 1. | Cisco Systems, Inc. New Study Points to Substantial Financial Returns from Broad-Based Wireless LAN Deployments. 2003. http://newsroom.cisco.com/dlls/hd_111203b.html. |