22
DATA CENTER SITE SEARCH AND SELECTION
Ken Baudry
K.J. Baudry, Inc., Atlanta, Georgia, United States of America
22.1 INTRODUCTION
Almost all data center disasters can be traced back to poor decisions in the selection, design, construction, or maintenance of the facility. This chapter will help you find the right combination and eliminate poor site selection as a cause of failure. It begins with setting objectives and building a team, examining the process, and selection considerations. The chapter concludes with a look at industry trends and how they may affect site selection.
Site selection is the process of identification, evaluation, and, ultimately, selection of a single site. In this context, a “site” is a community, city, or other populated area with a base of infrastructure (streets and utilities) and core services such as police, fire, safety, education, and parks and recreation. This definition is not meant to eliminate a location in the middle of a nowhere, the proverbial “corn field” location. However, experience indicates that unless an organization is really set on that idea, there are few such locations that have the key utilities (power and fiber) and core required to support a data center. Most organizations back away from this idea as they estimate the cost and logistics of operating such a facility.
Site search and selection can be as comprehensive or as simple as necessary to meet the goals of the organization. In practice, it consists of asking a lot of questions, gathering information, visiting potential communities, and perhaps entertainment by local and economic development officials.
The motivation behind conducting an extensive site search tends to be economic (i.e., find the site with the least total cost of ownership). While the site search and selection process attempts to establish an objective basis for the decision, it relies on estimates and assumptions about future conditions and often includes criteria that have an economic value but are not easily fit into an economic model. These criteria include marketing and political aspects; social responsibility, as in giving back to the community; and quality‐of‐life issues. These issues tend to be subjective and are often weighted heavier in the decision matrix than the economics suggests. The end result is that the site selection process tends to be a process of site elimination based on economic criteria until the list is pared down to a few sites with similar characteristics. The final decision is often subjective.
There is no such thing as a single “best site,” and the goal of a site search is to select the site that meets the requirements, does not violate any constraints, and is a reasonable fit against the selection criteria.
In general, the goal in site search and selection is to ensure that there are a sufficient number of development opportunities in the selected community. However, there are good reasons to search for specific opportunities (existing data centers that are suitable or that can be refurbished, buildings that might be converted, pad sites, or raw land) as part of the site search and selection process. There are often negotiating advantages prior to the site selection announcement, and the identification of a specific facility or property might become the deciding factor between the short‐listed sites.
22.2 SITE SEARCHES VERSUS FACILITY SEARCHES
Most of the discussion contained in this chapter is based on selection of a site for development of a data center from the ground up a.k.a. “a brown field site.” However, there are other viable data center acquisition strategies such as buying or leasing an existing single‐tenant data center or leasing a co‐location center. With co‐location and existing facilities, much of the investigative work should have already been done for you. You will be able to ask the prospective landlords to provide answers to your questions. It is likely that some aspects, such as power company rates and tax incentives, have already been negotiated by the developer.
You will still need to understand what you want to achieve and what your requirements are, and it would be a mistake to assume that the developer had the uncanny ability to anticipate your requirements and has taken them all into consideration. Data centers are not a one‐size‐fits‐all proposition, and existing centers may have been located where they are for reasons that aren't related to the data center business at all. It could be that the original tenant selected the location because it is where someone's grandchildren live.
You will still need to compare the information you are given against your requirements and analyze the total occupancy cost for each prospective facility. In many cases, the lowest cost alternative may not be the one with the best lease rate. It might be the one with the best power company rate and the most opportunities for energy savings driven by the local climate.
22.3 GLOBALIZATION AND THE SPEED OF LIGHT
Globalization has been around since Christopher Columbus’s voyage, when a group of adventurers left one area to seek better fortunes in another area. Today, it is largely driven by economic forces as organizations seek out competitive sources of raw materials and labor and new markets for their products. Globalization and technology, in particular air travel and voice and data communications, have made it possible for organizations, more than ever before, to consider sites outside of their country of origin.
Global strategic location plan can be divided in macro, mid and micro levels (see chapter 1). Amazon Web Services have built a hierarchy of “Regions” located around the world. Within a “Region”, there are “Available Zones” (AZ) and each AZ has a cluster of data centers.
Historically, organizations have located facilities overseas to secure raw materials or inexpensive labor and to avoid taxes and safety and regulatory policies of their country of origin. The question for us is: “Are there economic benefits to locating a data center outside of one's country of origin?”
The motivation for locating overseas, today, is very different from the raw materials and cheap labor considerations of the past. Data centers don't need a large pool of unskilled labor like manufacturing and call centers, and they are not tied to location by raw materials. Data centers can be located almost anywhere that can provide significant amounts of power and connectivity to high speed national and international communication networks.
Including sites in foreign countries adds a layer of complexity as the differences in tax structure, laws, ownership of real property, security of data, political unrest, etc. need to be considered. There is, however, one difference that is not easily eliminated by technology or money: communication signals cannot propagate any faster than the speed of light.
As an example of how this figures into site selection constraints, let us consider a route from Bombay, India, to New York City, New York, United States. The speed of light in free space is 299,792 km/s or approximately 186,282 mi/s. It is slightly slower in optical fiber, but we are going to ignore that for simplicity. The distance is about 7,800 miles, and assuming that a fiber route would be 50% longer than a direct path, it takes about 70 ms (1 ms is equal to 0.001 s) for the signal to propagate one way. This figure does not include any latency for network interfaces, transmission gear, etc. So unless Einstein was wrong, 70 ms is the best possible signal latency.
Today, the expected latency between the United States and India is between 215 and 305 ms. Network improvements will likely drop this figure over time to about 200 ms. How this limits site selection depends largely on the applications that you use.
One of the most common latency‐sensitive applications is storage array replication. In data replication, an application will write data to a disk or storage array, and the array will replicate the same to a remote array. When confirmation comes back that the data has been successfully written, the transaction is complete. If the latency is high, then the performance is seriously degraded. Other latency‐sensitive applications include transaction‐oriented applications like banking and retail sales where, due to a very high transaction rate, the latency must be very low and burst capabilities are required to meet high demand.
Without a doubt, some organizations will realize significant benefits from selecting an overseas location. But there are challenges, and it will not be a suitable option for everyone.
22.3.1 The Site Selection Team
Like any process with many moving parts, site search and selection requires diligence, clear expectations, schedules with defined deliverables and due dates, and good communication between all stakeholders. This doesn't happen by itself.
Project management is the art of managing a process from beginning to end. It concerns itself with reaching the end state by defining and organizing the work effort, communicating and leading stakeholders, and driving decision making consistent with requirements, within constraints, and in a timely manner.
Project management sets metrics for constant feedback on performance and adjusts accordingly to keep on track. It is a profession, and the role of the project manager cannot be overstated. If you do not have someone in‐house who has successfully demonstrated their competence, you should look outside of the organization.
Some key reasons why projects fail are similar across almost all endeavors are as follows:
- Lack of user involvement
- Unrealistic expectations
- Incomplete requirements and unsupportable criteria
- Lack of planning
- Lack of executive support
Before your organization can make good progress with the site search, it will be important to identify potential team members and start building the team. The timing of “when” you bring your team on board is as important to your success as “who” you bring on board. Assumptions about cost and feasibility are often made early in the program, before subject matter experts (SMEs) are traditionally on board, and often spell doom for a project when it is determined that they are not realistic and, more often than not, need to be adjusted upward. You will need your team on board early in the process to avoid this sort of pitfall and to create plausible criteria and constraint list.
As you build your site selection team, you should talk with key executives and ask for recommendations. By involving them early, reporting progress in a concise and effective manner, you will gain their support. If you are the key executive, then you should involve your board, investment committee, and your key subordinates as you will need their support as you push your search forward. As a minimum, this effort should lead to a better understanding of the project and identification of who will cooperatively back the project and who won't.
One of the key consultants will be a site search consultant. A good site search consultant will not only be an SME but also be an advisor. He will not only guide you through the site selection process but also will know what to expect from economic development agencies. He will know where to get answers, know other professional resources in the industry, and understand how to evaluate proposals. He will understand when it is time to bring the search to a close and how to bring it to a close. He will render advice and help you make good decisions. In many cases, he will be able to save you time and money, by eliminating some sites, based on recent relevant experiences prior to beginning a search.
Your team will need to include SMEs such as data center planners, data center consultants, architects, engineers, lawyers, tax accountants, and real estate brokers. Any specialty that can vary in cost between sites will need some level of representation on your team. The greater the expected cost variance between sites, the greater the need for the SME.
An architect might only be needed to create an initial estimate of the space required and nothing more if construction costs are believed to be the same regardless of site selection. An engineer might only be needed to create a power consumption profile and energy consumption and demand for the proposed facility, while a utility expert might be retained to evaluate energy costs for each prospective site.
When building a team, there is often a distinction between consultants and vendors. Consultants typically charge a fee for their services. Vendors typically provide preliminary support for free or for a minimal fee in hopes of making the big sale later on in the project. This includes real estate brokerage firms that will assist with site selection, general contractors that will perform preliminary design and develop budgets, and others. While you should surround yourself with the best resources that you can afford, the key is to surround yourself with resources that have the experience, competence, and that you can trust to act in your best interest. All three are important!
22.3.2 The Nature of the Site Search and Selection Process
There are a few characteristics of site search that require special mentioning; it's generally performed in secrecy, and it's not a search process but an elimination process, and there comes a time when continuing the process will no longer produce further value.
22.3.2.1 Secrecy
More often than not, a site search is conducted in secret. There are many reasons including the following:
- Site searches don't always result in moves or in the development of new facilities. Announcing a site search and then not following through may be perceived as a failure.
- News of a site search may cause concerns among employees over potential facility closures.
- Most businesses don't find it competitive to telegraph future plans to their competition.
- Once the word is out, you will be inundated by vendors wanting to get a piece of the action.
Regardless of your reasons, it is likely that management will expect that the search be conducted in secret. Most site search consultants are going to be aware that this will be the case. As the size of your team grows and many aspects of the work will require team members to consult with resources outside of the team, such as equipment vendors, it is likely that by the end of a lengthy search, there the number of individuals aware of your search will be quite large.
If secrecy is important to your organization, then you will need to take precautions. The first step should be execution of simple confidentiality and nondisclosure agreements with every team member. Many of the consultants will be looking to perform additional services once the site is selected, and well‐timed reminders about disclosure can be very effective.
Some companies give their site selection projects a code name. This works well as it gives everyone a name to reference and serves as a constant reminder that the project is secretive.
When it comes to secrecy, we are often our own worst enemy, and many individuals unknowingly let out the secret by wearing their company access card with picture ID, wearing clothing or carrying pens or clipboards with a corporate logo, handing out business cards, providing phone numbers that can be easily traced back to a company, etc. Even discussing your travel arrangements and where home is can provide significant clues to the curious. While all of these things are obvious, it's the most obvious things that we forget to deal with. If you really want secrecy, you will leave nothing to chance, and let your site search consultant handle all of the communications.
22.3.2.2 Process of Elimination
Site search and selection is a bit of a misnomer as it is more a process of elimination than selection. It starts with a broad universe of possibilities and moves through rounds of elimination (Fig. 22.1). The broadest criteria are applied in the first round to eliminate the most number of sites and narrow the search. The broadest criteria are usually geographic. For example, North America, Europe, cities of five million or more in population, within my service region, etc.
FIGURE 22.1 Site search and selection process.
Source: Courtesy of K.J. Baudry, Inc.
Each successive round uses more selective criteria until all but one site is eliminated. This may sound a bit daunting, but in practice, the number of possibilities falls rapidly and narrows to a dozen or so very quickly.
By the time it gets down to two or three of sites, the remaining sites are all very similar in terms of economics, and the search becomes as subjective as it is objective. At this stage, organizations may eliminate a prospective site based on perception, and previously unknown criteria may spring up as a way to differentiate between the remaining sites.
One thing is certain, once you make your announcement as to final selection, the game is over. There are no more concessions to be won and no more dealing to be done. It is important to keep two or three candidates actively involved until the very end.
22.3.2.3 Relative Costs and the Law of Diminishing Returns
Estimating all of the cost may be important to the funding decision and may be developed as part of the overall project strategy, but as long as there isn't a difference between sites, it's not critical to the decision. For the purpose of site selection, it is only necessary to compare costs that differ between sites. For example, if license fees for operating systems and applications are the same regardless of where deployed, then this is not a cost that needs to be evaluated as part of the site selection process.
Site search and selection involves many forecasts of future events: how much power you will need, how fast you will grow, how many people you will employ, how the political and economic climates will change over time, etc. There is a cost to the site search and selection process, and at some point the cost of obtaining more information, performing more analysis, and perfecting the estimates starts to exceed any benefits that may be derived from the effort.
In the end, worrying about costs that aren't significant or key to making the site selection decision and overanalyzing the costs tend to only delay the decision. There may be a clear choice, but more often than not, there will be two or three sites that are all well suited, and if you have done a good job, all very similar. At this point, additional analysis is not likely to improve your decision, and it is time to bring the process to a close.
22.4 THE SITE SELECTION PROCESS
Site search and selection can be boiled down to several key activities as follows (Fig. 1):
- Develop business requirements and constraints.
- Define geographic search area.
- Define site selection criteria.
- Screen opportunities.
- Evaluate “the short list.”
- Close the deal.
22.4.1 Develop Business Requirements and Constraints
The first activity of a site search is to define the business requirements. Typical motivations for conducting a site search may include supporting growth, reducing cost, expanding into new markets, etc. Requirements are the items that we believe are necessary to conduct business including space, power, cooling, communications, etc. Some might add “in a profitable manner,” while others might argue that profitability is not part of a requirement.
The industry has created its own rules and vernacular that include requirements, constraints, criteria, etc. Don't get hung up on the subtle difference between these terms. What you need to end up with is a set of statements that define what results are wanted and by whom segregated into “must have” (deal breakers) and “must have if available and affordable” (negotiable), a ranking by priority, and finally rough expectations of cost and practicality.
By definition, you would think that a “requirement” must be satisfied. But reality is that requirements come from a collection of stakeholders with different motivations, fears, and concerns. Some are more perceived than real. They often conflict with one another, and some are more important than others. As part of cataloging and ranking the requirements, you will need to resolve the conflicts.
You can catalog and track this kind of information in a spreadsheet, database, or an intranet Web‐based collaboration tool. There are some specialized “requirement management” type products available as well. How you do this is not important, but it is important that it is well done and documented.
In all probability, by the time an organization funds a site search, they have a pretty good idea of what they want to accomplish and have something akin to a requirement list. Key to being successful will be flushing out this list, eliminating requirements that are not supportable by the business, adding requirements that are missing, and building a consensus among the decision makers.
A good place to start is to look at why you are searching for a new site and establishing if expectations of the benefits that will be gained from the endeavor are reasonable. Even if the reasons for the search seem obvious, there are always alternatives, and all too often, when the final cost estimate is put together, the program changes. An organization may elect to refurbish an existing data center rather than relocate, may decide that a new data center cannot be supported by the customer base (sales revenues), or may find that despite tall tales of cheap electrical power deals, cheap power is a fleeting opportunity at best.
This first high‐level pass should be based on what we know today and on any additional information that can be gathered in a timely and inexpensive manner. If the key business requirement is to reduce operating expenses and if the belief is that cheaper power is available elsewhere, then a purchased power survey/report and industry average construction costs might be enough for a first‐round estimate of the potential annual energy savings and a potential payback. This effort might confirm, change, or disprove the savings expectation and reason for the site search, saving the organization countless hours and expense.
Each requirement and expectation should be challenged in this manner, as well as assumptions about executive and business unit support. Don't assume that what is in the interest of one business unit is in the best interest of all business units. Spend some time and effort confirming the initial project program with the stakeholders (i.e., anyone who has a credible interest in the outcome). This may include senior management, business units that will use the facility, customers, consultants, etc. The goal is not to kill the site search before it starts, but a search that starts badly usually ends up badly.
It is surprising, and all too frequent, that halfway through the site search selection process, the question of what the organization really needs or wants gets raised as it was never suitably addressed in the beginning. This kind of questioning down the home stretch will lead to delays and missed opportunities. It can also cause confusion among your stakeholders and with the local and economic development leaders that are after your business.
The key here is “plausible.” Beginning the search with too specific or unrealistic criteria risks eliminating potential sites and tends to get the search bogged down with detail that often isn't available until further along in the process. On the other hand, beginning the search with too few or incomplete criteria wastes a lot of everyone's time. When you don't sound like you know what you are doing, people are less responsive and don't offer their best.
No matter how hard you try to build a great team (Table 22.1), you will have some team members that are not of your choosing, some of whom will be uncooperative and will present site selection criteria that ensure the search will not be successful. It is important that inappropriate criteria do not make it to your site selection criteria list. Also, once you begin the search process, it is not uncommon to find that a certain requirement is not easily met or that it is prohibitively expensive to meet, and there may be other reasons to redress the requirements.
TABLE 22.1 Typical search team members and criteria
Source: Courtesy of K.J. Baudry, Inc.
Typical site selection team members:
|
Be cautious and exercise due diligence when a change is imitated, but keep in mind that the goal is not to adhere to a statement of requirements at all cost but to make the right decision for the business. Business conditions do change, and the original requirements may have been misdirected in the first place or are perhaps no longer appropriate.
22.4.2 Round 1: Define Geographic Search Area
Geographic area is generally the first requirement, or a constraint, that the search team will use to narrow the universe of possibilities. Why? First, because the process is one of elimination and geographic area generally eliminates more potential sites than any other constraints. Second, the research and effort required are practically nil. Connecting these two points, there is simply more bang for the buck with this approach.
The geographical search area can be anything reasonable. Keep in mind that there are often legitimate requirements or constraints that may not be directly cost related and aren't easily estimated or accounted for:
- A power company or other regulated utility might define their search area as their service territory because locating outside their territory might be considered both politically incorrect and akin to suicide with the public service commission or other regulators.
- A bank with sensitive customer credit data might constrain their search to the locations in their home country based on regulatory requirements that certain data be maintained in‐country.
- An international business such as Coca‐Cola or Home Depot might look to any country where sales are substantial and contribute to their annual income, in an effort to give something back to the community.
- A regional organization that will relocate a number of employees to the new site may limit the search to cities in their region with a population between 1,000,000 and 5,000,000, in an effort to make the relocation more palatable to their existing employees.
- A business in a very competitive environment where being the lowest cost provider is key to winning business might not have any geographic limitation. It simply comes down to the least expensive site available.
Geographic constraints carry a lot of risk if the impact is not fully thought out. Many organizations set geographic constraints based on proximity to existing facilities, the assumption being that there are practical benefits (cost savings) to be gained. However, there is little basis for this assumption. Any operational support or convenience that might be available from the headquarters, office, warehouse, or other facilities in the immediate vicinity is often insignificant when compared with other savings such as energy cost and tax incentives that might be available outside the immediate vicinity. There also may be existing facilities in locations outside of the immediate vicinity that are available because of an unrelated merger or acquisition, bankruptcy, etc. that might be leased or purchased at substantial discount to the cost of new construction.
Once you have the search area, you will want to develop a list of potential communities. There aren't a magical number of prospective communities. Given the costs involved, more than 20 opportunities are probably excessive. Your site search consultant may suggest eliminating some communities based on relevant experience in dealing with them. With fewer than a dozen, you might not end up with any acceptable candidates as you cut the list down, applying more detailed criteria.
22.4.3 Round 2: Site Selection Criteria
We have opted to present this information as taking place in a serial manner for ease of presentation. Depending on your schedule and how committed (willing to spend money) the organization is, there is some benefit to developing the statement of requirements and selection criteria as a contiguous event in advance of starting the search. With this in mind, selection criteria that require little research might be incorporated as part of round one. The idea being to eliminate as many possibilities before the in‐depth (expensive) research is required.
The following sections identify concerns that are common to many data center users when searching for a site. The list is not comprehensive but covers the major areas. Some of the items may not be applicable, depending on the type of organization and industry which you operate in.
The idea is that for each of these items you assess how they affect your organization and if they are potentially impacting identify how they can best be dealt with.
22.4.3.1 Political Environment
Most national governments, and especially emerging economies, have reduced barriers to market entry, property ownership, and deregulated privatized industries and encourage capital investment. When looking overseas, the chances are that you will receive a warm welcome. However, there may be significant challenges in the following areas:
- Security
- Laws
- Regulatory
- Employment
- Property ownership and investment
- Censorship
22.4.3.2 Quality of Life for Employees
Data centers can operate with a few employees. In many cases, quality‐of‐life issues might not be important, as employees will be hired locally. However, if an organization plans on relocating employees, quality‐of‐life issues will become important in retaining employees. What are people looking for? Some combination of the following:
- Low home price, taxes, and energy cost
- Short commute times
- Employment and educational opportunities for spouses and children
- An environment that provides residents a year‐round playground with plenty to do—mountains, lakes, rivers, and beaches
- Entertainment including major league sports and theaters
- World‐class cultural attractions
- Exciting nightlife
- Convenience of travel etc.
There are numerous publications such as Forbes, Business Week, and Time Magazine that create “top ten” lists. Best cities for college graduates, best cities for technology, best cities for entrepreneurs, most high‐tech employment, etc. make good sources for this kind of information.
22.4.3.3 Business Environment
Site selection is largely about taxes and other costs that are a large part of operating expenses. Taxes come in all shapes and sizes, vary in how they are calculated, and are applied from state to state. Historically, some communities have looked at the economic benefits that data centers provide in terms of direct and indirect jobs created (payroll), investment, and taxes and have identified data centers as good sources of tax revenue. Data centers pay a lot in taxes and demand very little in terms of publicly provided services. They do not use significant amounts of sewage capacity, generate trash, fill up the local schools, and don't require new roads or extra police services. For a politician, it's a significant source of new “unencumbered” income.
The largest single category of taxes tends to be property taxes. In many communities, both the value of the real property, land, and buildings and the value of the fixtures, furnishings, and equipment are taxed. When you consider the cost of the facility is often well over $1,000/sf and that the IT equipment installed can easily exceed this figure, even a low tax rate can result in a significant annual outlay.
Local communities typically have offered incentives in terms of tax abatement on property taxes and reduced sales taxes on equipment purchased for installation in the facility. Programs will vary, but almost all communities phase the incentives out over a period of years. Other state and local incentives may include an expedited permit process, land grants, improved road access, extension of utilities, tax rebates, and financing through industrial revenue bonds. There may also be community development zones and associated development grants. Many states offer job credits and training programs through local community colleges.
You will need to calculate the economic benefit of the incentive package and include it in your overall analysis. A word of caution: a significant percentage of incentives, well over 50%, are never collected due to failure on the part of the organization to follow through after the facility is opened or due to changed economic conditions, overenthusiastic rates of growth, etc. For example, a delay in growth that pushes large purchases of equipment beyond the first couple of years could result in significant reduction in tax savings if the incentive was highest in the first year and fell off over successive years.
When considering overseas locations, there are differences in the way taxes are levied, and it will affect your cost structure. This is one of the reasons for having an accountant on your team who can evaluate the implications for your economic modeling.
Politicians, local officials, and economic development groups love to make a splash in the news headlines by announcing big deals, and data centers tend to be big deals in terms of dollars invested. But it is a two‐way street, and it will be up to you to show them the value that you bring to the community. In short, the better you sell yourself, the more successful you will be at winning incentives.
22.4.3.4 Infrastructure and Services
While airports, roads, water, sewage, and other utilities are all important, two utilities are showstoppers. A data center must have electrical power and telecommunications.
Telecommunications engineers often represent networks as a cloud with an access circuit leading in/out of the cloud at the “A end” and in/out of the cloud at the “Z end.” It's a bit of a simplification but is a good representation. The in/out circuits are called end or tail circuits and are provided by a local exchange carrier (LEC). They typically run to an exchange where traffic can be passed to long‐haul carriers.
Depending on how you purchase bandwidth, you may make a single purchase and get a single bill, but it is very likely that your traffic is carried over circuits owned by several different carriers. Having more than one carrier available means that there is always a competitive alternate carrier who wants your business. At one time, network connections were priced on bandwidth miles, but today, supply and demand, number and strength of competitors in the local market, and available capacity all factor into the cost of bandwidth. The only way to compare bandwidth cost between site options is to solicit proposals.
While small users may be able to use copper circuits, T1s and T3s, many organizations will require fiber medium services, or the latest carrier grade Ethernet. LECs typically build fiber networks using self‐healing networks, such as synchronous digital hierarchy (SDH) or synchronous optical network (SONET) rings. These are very reliable. Typically arrangements can be made to exchange traffic to long‐haul networks at more than one exchange making the system very reliable and resilient.
Many organizations require that there be two LECs available. Reasons for this include reliability, pricing, and perhaps a lack of understanding about networks. In general, site search and selection tends to be initiated and led by the financial side of the organization, and most organizations do not involve their network engineers in the site search and selection process. However, network connectivity is one of the fundamental requirements of a data center. It is a key cost and performance issue, and a network engineer should be considered a critical part of one's team.
Electrical power is the other big piece of infrastructure that is a must have. Smaller data centers with less than 5 MW of load can generally be accommodated in most large industrial and office parks where three‐phase service exists. Larger data centers require significant amount of power and may require planning with the power company and the upgrading of distribution lines and substations and, in some cases, construction of dedicated substations. All of this must be addressed with the power company prior to the site selection.
Energy can make up as much as a third of the total occupancy cost and more if the rates are high. Almost all power companies use the same basic formula for setting rates: recover the capital cost of serving the customer, recover the cost to produce the energy, and make a reasonable return for the stockholders. Understanding this is important to negotiating the best rates. Yes, most utilities can negotiate rates even if they are subject to public service commission regulation, and regulated utilities can be every bit as competitive as nonregulated utilities!
If you want to get the best rate, you need to know your load profile, and you need to share it with the power company. Your design may be 200 W/ft2, and that is great, but that's not your load profile. Your load profile has to do with how much energy you will actually consume and the rate at which you will consume it. Retaining a consultant who understands rate tariffs and can analyze your load is important to negotiating the best rates.
Negotiating rates is best done in a cooperative manner, sharing information as opposed to the more traditional negotiation stance of sharing little, demanding a lot, and constantly threatening to take your business elsewhere. Demanding oversized service only results in the utility spending more money to serve you, and more money has to be recovered from you in upfront capital contribution or through higher rates; therefore, properly sizing the service results in the most competitive rates. Further, most utility companies, either because of regulation or because of board governance, will not knowingly provide you with below cost rates and then make it up by charging another customer higher rates.
Climate is not part of the infrastructure but makes the checklist twice: first, as a factor that affects your energy cost and second as a quality‐of‐life issue. Your energy cost is dependent on the rate, but it is also dependent on how much energy you use. Recent thinking has led air‐conditioning engineers to consider ways of reducing cooling costs such as air‐side and water‐side economization. The potential savings are greatly increased when the outside air is cool and dry for substantial parts of the year.
However, there is more that should be considered than just cost; the capability of the utility and responsiveness when and if an emergency should occur is also important. A short example is probably worth more than a dozen paragraphs. In September 2005, Hurricane Katrina devastated the Mississippi coast. Mississippi Power, a small operating unit of Southern Company with 1,250 employees, restored power to almost 90% of their customers within 11 days (10% were too decimated to receive power). They rebuilt 300 transmission towers, over 8,000 poles, and 1,000 miles of overhead lines against all odds by bringing in a workforce of over 10,000, providing temporary housing in large circus tents, food, and water; over 8,000 tetanus shots; and burning 140,000 gallons of diesel fuel a day. A small municipal utility, affiliated with a regional utility, may not be able to come through when the going gets tough. The preparation and logistics necessary to maintain operations during a disaster get exponentially more difficult as the duration becomes longer.
There are three concepts in business continuity that need to be considered in the site selection process: walk to, drive to, and fly to. In major regional outages such as earthquakes and hurricanes and after 9/11, it became apparent very quickly that accessing to a primary or fall back site can be challenging if not impossible. Roads become blocked with traffic or debris, and even air travel may become curtailed. If your business continuity planning requires that your data center continues operating in an emergency, then it becomes important to have multiple means of transportation.
It is a bit unusual to think of maintenance vendors as part of the local infrastructure. However, the role of preventive maintenance is of as much importance as it is to have redundant systems, and perhaps even more important. While it is important that a vendor know everything there is to know about their equipment, they must also be familiar with and have the discipline to work in data center environments. If you are the only data center in the region, you may not find suitable maintenance vendors locally. If qualified maintenance vendors are not available, you will need to consider how this will affect your normal maintenance program as well as your ability to recover from a failure in a timely manner. Operator errors, including errors made by maintenance vendors, account for a significant percentage of all failures. Having other significant data center operations in the area is a good indication that maintenance vendors are available.
Finally, if there are other data centers in the area, ask the local economic development authority to introduce you. The odds are that they played some role in their site selection decision and will already have connections within the organization. Buying one of these contact's lunch is probably the best investment you can make in the search process.
22.4.3.5 Real Estate and Construction Opportunities
Depending on your requirements, you may be looking for an existing facility to purchase or lease, for pad sites within a development, or for raw land. Regardless of the need, you will want to make sure that there are a sufficient number of opportunities.
During the dot‐com boom, a new breed of business, co‐location, was created. Aiming at the outsourcing of information technology needs by corporate America, and armed with easy money, these companies built large, state‐of‐the‐art facilities across the country, some at a cost of $125 M or more and as large as 300,000 ft2. Many of these companies failed, some because of a poor business plans and others because they were simply ahead of their time. These facilities were placed on the market at substantial discounts to construction cost. Some were practically given away.
Today, these types of opportunities are rare, but occasionally companies merge or change computing strategies to find that they now have excess data center facilities. Other facilities become available because of lease expirations, growth, changing technologies, etc. While these may be great opportunities for some companies, they are often based on outdated standards and were built prior to the current focus on energy costs. It may not be necessary that a facility meet “current standards,” but it must meet your standards and needs. Great care should be taken to properly assess the opportunity and cost to upgrade the facility if necessary.
It is not uncommon to find industrial or mixed‐use parks advertising themselves as a “data center opportunity.” This may mean that the developer has already worked with the local power utility, brought multiple fiber carriers to the park, negotiated for tax abatement, and taken other steps to prepare the site for a new data center. However, it often doesn't signify anything more than someone could build a data center on this site if they choose to.
When selecting a community, it is important that there are multiple sites available. These sites should be competitive (owed by different investors) and suitable for constructing a data center. It is important that your team verify any claims made by the owner, economic development agency, or other organizations trying to attract your business.
If there is a single existing facility or only one site available, then the negotiations with the various parties involved need to be tied together for a simultaneous close or with dependencies written into any purchase agreements so that the failure of any one part invalidates any other agreements.
Construction costs tend to be relatively uniform, and there are construction indexes readily available that can be used to estimate differences between communities or geographic regions. Development costs, permitting process, and construction requirements may vary between communities and, in some cases, might represent a significant impediment to your project, especially in terms of schedule. If you have a target date in mind, then finding a community that is willing to expedite the permit process may be important.
22.4.3.6 Geography, Forces of Nature, and Climate
Avoiding forces of nature is one area where there is often a discrepancy between what people say and do. This is due in part to a checklist mentality: the idea that by checking off a generic list of standard issues, we can avoid during our own legwork.
The general recommendation is to avoid risk due to forces of nature. Yet, California is home to a large number of data centers and continues to capture its share of growth despite an elevated risk of seismic activity. Almost half of the continental United States falls within a 200 mph or greater wind speed rating and is subject to tornado activity. Yet these areas continue to see investment in new data center facilities. Finally, East Coast areas with elevated risk of hurricane activity such as around New York and Washington, DC, continue to be desirable.
There are a couple of reasons why this happens. First, if there are compelling reasons to be located in a specific area, the risk may be substantially mitigated through good design practices and construction. Second, many individuals in key decision‐making positions cling to the idea that they need to be able to see and touch their IT equipment. For these people, sometimes referred to as “server huggers,” the idea of a remote lights‐out operation entails more risk than the local forces of nature.
Regardless of the reason, it is important to assess the level of risk, identify steps to mitigate the risk, estimate the cost of risk mitigation, and include that cost in your financial modeling. Depending on whom you ask, natural disasters account for anywhere from 1% to almost 50% of all major data center outages. Another report puts power‐related causes at 31%, weather and flooding (including broken water lines) at 36%, fire at 9%, and earthquake at 7%. A lot depends on the definition of a disaster, the size of data center under consideration, and what product or service is being promoted. The fact is that many data center outages are avoidable through good site selection, proper planning, design, and maintenance.
Design and construction techniques that effectively mitigate risk of damage from earthquakes, tornados, hurricanes, and flooding are well known and, while expensive, can be economically feasible when considered as part of the total cost.
While protecting your facility may be feasible, it is generally not possible to change the local utility systems and infrastructure. Many seemingly well‐prepared organizations have found out the hard way that it is easy to end up being an island after a region‐wide event such as a hurricane. An operating data center is worthless without outside communications and without its resupply chain in place (fuel, food, and water).
22.4.3.7 Man‐made Risks
Avoiding man‐made risks ranks up there with avoiding natural disaster, and the variance between what one asks for and what one does is even greater with man‐made risks. Requirements that we often find on checklists include the following:
- Two miles from an airport
- Two miles from a broadcast facility
- Four miles from a major state highway or interstate highway
- Five miles from a railroad
- Ten miles from a chemical plant
- One hundred miles from a nuclear facility
Many of the items on this list appear obvious and reasonable on first review. However, trains and tankers full of hazardous material traverse our railroad tracks and highways every day of the week and at all hours of the day and night. So being a minimum distance from potential accidents makes sense. But most accidents are not contained within the highway right of way, and when you consider that released toxic chemicals can be transported by wind for miles, you realize that one‐half mile, two miles, or four miles do not substantially change the risk.
Now consider where fiber and power are readily available. Utility providers built their networks where they expect customers to be, in industrial, mixed‐use, and industrial parks (i.e., where there is commerce), and where there is commerce, there is man‐made hazards.
It is important that all risks be considered and they should be considered from both short‐ and long‐term points of view. It is even more important that the nature of the risk, relationship between the risk and distance, and the potential impact be understood.
22.4.4 The Short List: Analyzing and Eliminating Opportunities
The site search and selection process is built around elimination, and, to be most effective, we apply broad brush strokes in order to eliminate as many sites as possible. In doing this, we take a risk that we might eliminate some good opportunities. We eliminate some communities because they have significant shortcomings and others because they might have simply lacked the preparation and coordinated effort to provide a credible response to our request for information.
We are at a critical point in the process; we have eliminated most of the sites and are down to the short list. Each remaining community will need to be visited, the promotional claims and statements made during the initial rounds researched and supported, specific proposals requested, high‐level costs estimates brought down to specific costs, and agreements negotiated. If there is a specific facility that figures into the site selection decision, you may want to have an architectural and engineering team evaluate the facility, create a test fit, identify the scope of work, and prepare a construction cost estimate (Table 22.2). This is a considerable effort for each site, and most organizations will want to limit the number of sites to three, the two front‐runners and a replacement.
TABLE 22.2 Simple visual presentation of sites
Source: Courtesy of K.J. Baudry, Inc.
| Comparison of sites | ||||||||
|---|---|---|---|---|---|---|---|---|
| Selection criteria | Weighting | Criteria ratings | ||||||
| Option A | Option B | Option C | Option D | Option E | ||||
| 1 | Ability to meet schedule | 4 | 2 | 5 | 2 | 4 | 1 | |
| 2 | Availability of property | 3 | 3 | 5 | 3 | 3 | 1 | |
| 3 | Cost of property | 3 | 3 | 4 | 4 | 3 | 1 | |
| 4 | Business environment—economic incentives | 4 | 4 | 1 | 4 | 3 | 1 | |
| 5 | Availability of low‐cost financing | 3 | 4 | 1 | 4 | 3 | 1 | |
| 6 | Availability of power | 5 | 4 | 3 | 5 | 4 | 4 | |
| 7 | Reliability of power system | 4 | 4 | 4 | 5 | 4 | 4 | |
| 8 | Low‐cost energy | 5 | 4 | 3 | 5 | 4 | 4 | |
| 9 | Environmental energy savings opportunity | 3 | 2 | 5 | 3 | 4 | 5 | |
| 10 | Availability of fiber networks | 5 | 3 | 4 | 2 | 4 | 2 | |
| 11 | Availability of skilled vendors | 3 | 2 | 4 | 1 | 4 | 2 | |
| 12 | Availability of IT labor | 3 | 2 | 4 | 1 | 4 | 2 | |
| 13 | Easy to “fly to” | 4 | 1 | 3 | 4 | 4 | 2 | |
| 14 | Easy to “drive to” | 4 | 1 | 3 | 4 | 4 | 1 | |
| 15 | Proximity to major technical university | 3 | 1 | 4 | 3 | 4 | 1 | |
| 16 | Local job opportunities for family members | 3 | 1 | 3 | 3 | 4 | 0 | |
| 17 | Quality of life | 4 | 1 | 4 | 3 | 4 | 1 | |
| 18 | 0 | 0 | 0 | 0 | 0 | 0 | ||
| 19 | 0 | 0 | 0 | 0 | 0 | 0 | ||
| 20 | 0 | 0 | 0 | 0 | 0 | 0 | ||
Weighting—The default weighting is 2.5 and is the mean rating.
Rating—Ratings are high (5) to low (1).
Score—Score is calculated as “weighting × rating.” For example, a weighting of 3 and a rating of 2, produces a score of 6 (3 × 2). The resulting score is an indication of how well a vendor meets the weighted selection criteria. The higher the score, the better the vendor meets the criteria.
At this stage, it is inappropriate to eliminate a site without identifying a significant shortcoming. Earlier in the chapter, we stated that there was no such thing as the best site, just the best combination of site characteristics. In light of that idea, sites can be flawed but fixable or flawed and unfixable. In most cases, the reason for conducting an extensive site search is to minimize costs, and a site that has a fixable flaw, even if it is expensive to fix, might be the least expensive opportunity due to other line items. Consider a community without a competitive LEC. Could another carrier be attracted to the area because of your business? Could you negotiate a long‐term agreement with the existing carrier to assure good service and pricing?
If you identify an unfixable flaw early enough in the process, it may be worthwhile eliminating the opportunity from the shortlist and moving one of the lower‐ranked sites up to the list. It may be that a promise or claim made by the development agency was not correct or turns out to be incomplete. This can and does occur because of deception, but for legitimate reasons as well. The community may have expected a bond referendum to pass that didn't; another company may step in front of you and contract for the available power; etc.
If a specific facility is part of the selection decision, some of the business requirements will need to be translated into technical or construction requirements. This can be a challenge. The bulk of our efforts will have been financial. This does not change with technical requirements but does take on a new complexion. It is not uncommon for the engineering members of your team to designate specific properties as unacceptable, especially when dealing with upgrading an existing data center or retrofitting an existing structure. In many cases, this will mean that the engineer perceives one or more aspects as too difficult or expensive.
It is important that questions of suitability, especially when they potentially remove a site from the list, be questioned and thoroughly reviewed. Often the cost, when taken into perspective, is not significant, or that the many cost savings features of the site easily offset the added cost. Not every member of your team will be fully aware of the overall cost and potential benefits that a community or facility brings to the program.
22.4.5 Closing the Deal
The process is complete when the site selection is announced. The site selection choice may be made long before it is announced and long before the process is complete, and it is important that at least two communities be kept in the running until all concession and agreements are in place and the deals are executed. The reason is very simple; you lose all negotiating power once all the other sites are eliminated. So despite the effort involved, it is important to continue to pursue your number one and two choices until the very end. Just like secrecy, the team members are the ones most likely to let out the decision through innocent conversations with power company employees, local consultants, permitting agencies, and other entities that are involved in your due diligence.
22.5 INDUSTRY TRENDS AFFECTING SITE SELECTION
22.5.1 Globalization and Political and Economic Reforms
Globalization has been a consistent force since the beginning of mankind. It has moved slowly at times and with great speed at other times. The past 30 years have seen great advancements. Trade barriers have fallen, and governments have sought out investment from other countries. We may be on the verge of a temporary slowdown as governments move to protect their turf during economic downturns.
Globalization will continue as long as there are opportunities to sell more products and services and as long as there are differences in the cost of key resources such as utilities, differences in energy policies, and taxes.
22.5.2 Global Strategic Locations
While the world keeps getting smaller and smaller, it is still a pretty big place. Every search needs a starting point. Traditional economic criteria such as per capita income, cost of living, and population might be applicable. Perhaps, the location where your competitor already has facilities. Or maybe the location already has major international IT companies such as Facebook, Google, HP, and IBM located.
Google has facilities in many U.S. cities as well as cities outside the United States. It is important to note that these locations are not all major data centers. Some may simply be peering points. For more information on Google Data Centers, see http://www.google.com/about/datacenters/locations/index.html#.
In addition to traditional criteria, we offer one more that might be worthwhile considering volume or other measures of IP exchange traffic.
The largest exchanges in the United States are the New York Area, Northern Virginia, Chicago, San Francisco Bay Area, Los Angeles Area, Dallas, Atlanta, and Miami.
The largest exchanges outside the United States include Frankfurt, Amsterdam, London, Moscow, and Tokyo. A full list of exchanges can be found on Wikipedia by searching on “peering” or “list of IP exchanges.”
22.5.3 Future Data Centers
Today, we design and build data centers from a facility‐centric approach. They are conceived, designed, and operated by facility groups, not by the real customer IT groups. Facilities design has had hard time keeping up with the rapidly changing IT environment. Our facilities designs have changed little in the past 20 years. Sure, we have adapted to the need for more power and cooling, and there have been incremental improvements. Energy efficiency has become important. But by and large, we have simply improved old designs incorporating time‐proven techniques used in other types of facilities such as industrial buildings, schools, hospitals, and offices.
Today, every data center is a unique, one‐off design to match an organization's unique requirements and local conditions. Tomorrow's data center will need to be designed with a holistic approach, one that marries advancements in IT technology and management with facilities design and life cycle economics.
One approach that we expect in the future will be a throwaway data center with minimal redundancy in the mechanical and electrical systems. A large appliance inexpensive enough to be deployed at sites around the world is selected based on business‐friendly governments and inexpensive power.
Equipped with self‐healing management systems, meshed into a seamless operation, and populated with IT systems prior to deployment, these systems would serve their useful life, and the data center would be decommissioned.
With sufficient redundancy and diversity of sites, any one site could be off‐line due to planned or unplanned maintenance. To take an existing data center off line and redeployed it can base on changing economic needs such as customer demand, short‐term energy shortage, or fiber transmission services. At the end of life, it could be refurbished in the field, shipped back to a central depot for refurbishment, or sold to a low‐end noncompeting organization.
The combination of the reduction in initial facility development cost, mass production to a consistent standard and performance level, speed of deployment, and the flexibility to meet changing economic environments is a very attractive package. The ability to implement this approach largely already exists for the most part, considering the following:
- Lights‐out operations already exist. Management systems have progressed to a point where most operations can be performed remotely. They allow applications to exist at multiple locations for production and backup purposes, for co‐production, and for load balancing or to meet performance objectives.
- IT platforms are getting smaller and smaller. Servers have progressed from several rack units (RUs) to one RU and blades. You can fit as many as 70 blades servers in a single rack. Processing capacity is almost doubling every 3 years, and virtualization will greatly increase the utilization of the processing capacity.
- While the amount of data we store today has increased tremendously, the density of storage systems has increased tremendously and will continue to do so. In 1956, IBM introduced the random access method of accounting and control (RAMAC) with a density of 2,000 bits/in2 Today's latest technology records at 1.34 Tbits/in2 (Seagate) more than1012 in that's in 60 years.
A containerized data system could be configured to provide the same level of computing that is provided in today's average data center of perhaps 10–20 times its size.
There are advantages from the facilities perspective as well. Data centers are built for a specific purpose. We struggle to increase the availability of the physical plant, and every improvement comes at an ever‐increasing expense (the law of diminishing returns). No matter how much redundancy we design into the facilities plant, we cannot make one 100% available. By comparison, inexpensive data center appliances with dynamic load balancing and redundant capacity deployed throughout the world would not need five nines of availability to be successful.
According to the Uptime Institute, the cost of a Tier III facility is generally believed to be $200/sf and $10,000/kW or higher. While the building does depreciate, it's the physical plant (the $10,000/kW) that becomes obsolete over time. In a manner of thinking, we are already building throwaway data centers, at a great expense and without a good exit strategy!
There has always been a mismatch between the investment term for the facility and IT systems that the facility houses. IT refresh cycles are typically 3 years (probably more likely 5 years for most organizations), yet facilities are expected to last 15 years or more. This mismatch between investment terms means that data centers have to be designed to accommodate an unknown future. This is an expensive approach.
While such a concept will not be suitable for companies that are the largest of centers, it could be very beneficial for smaller operators, the very clients that collocation centers are seeking to put in their centers today.
ACKNOWLEDGMENT
This chapter is adapted from the first edition of Data Center Handbook and has been updated by its Technical Advisory Board. Sincere thanks are extended to Mr. Ken Baudry and the members of the Technical Advisory Board who spent invaluable time in sharing their in‐depth knowledge and experience in preparing this updated version.
REFERENCE
- [1] Submarine cable map, TeleGeography. https://blog.telegeography.com/2020‐submarine‐cable‐map (accessed 10/22/2020)
FURTHER READING
- Best Practices Guide for Energy Efficient Data Center Design, Office of Energy Efficiency and Renewable Energy (EERE), Department of Energy. https://www.energy.gov/eere/about‐office‐energy‐efficiency‐and‐renewable‐energy. Accessed 10/22/2020.
- National Oceanic and Atmospheric Administration. Available at http://www.nhc.noaa.gov/. Accessed on October 22, 2020.
- National Weather Service. Available at http://www.weather.gov/. Accessed on October 22, 2020.
- Rath J. Data Center Site Selection, Rath Consulting. Available at http://rath‐family.com/rc/DC_Site_Selection.pdf. Accessed 10/22/2020.
- The Uptime Institute. Available at http://uptimeinstitute.com/. Accessed on October 22, 2020.
- U.S. Geological Survey. Available at http://www.usgs.gov/. Accessed on October 22, 2020.