22.1 Introduction

Energy efficiency in data centers is an important topic that is commonly discussed in the information technology (IT) and communications industry. Many of the recent data center surveys conducted by equipment manufacturers, academia, and consortia show that energy efficiency ranks high on the list of top priorities for data center operators and clients.

There are several forces driving improvement in data center energy efficiency. First and foremost, energy prices are increasing. Figure 22.1 shows data from the U.S. Energy Information Administration (EIA) [1]. The average retail price of electricity from 1960 to 2010 has grown at 0.17 nominal cents per year (not adjusted for inflation). With the demand for energy increasing to meet worldwide demand, costs are expected to continue to rise. The simplest way to reduce the utility bill and cost associated with it is to use less electricity.

Second, energy supply is increasingly at risk from short-term or long-term disruptions. These disruptions can be caused by things such as civil unrest, terrorism, politics, natural disasters (hurricanes, earthquakes), inadequate infrastructure, and accidents. Third, climate change is influenced by the emissions generated from energy use and the emissions of greenhouse gases, such as perfluorinated compounds (PFCs) and SF₆, in various manufacturing processes and systems operations. The greenhouse gas emissions trap heat in the atmosphere and cause increases in surface temperatures. Finally, governments are responding to all three of these issues by developing and implementing voluntary or regulatory-based programs for energy efficiency in a range of product types and system operations including information and communications technology (ICT) products and data centers. Governmental programs and policies such as the European Union (EU) Emissions Trading Directive, the EU Energy Efficiency Directive, the United States Environmental Protection Agency (USEPA) Mandatory Greenhouse Gas Reporting Rule, and the USEPA ENERGY STAR® program have established measures to reduce carbon emissions and reduce product and system energy use with the goal of reducing the quantity of Greenhouse Gases released into the atmosphere for each unit of GNP produced [2].

IT equipment manufacturers have traditionally focused design efforts on delivering equipment with greater computing, storage, and networking capability. Advancements in semiconductor and interconnect technologies have enabled manufacturers to significantly increase equipment performance per unit of power consumed with each generation of equipment [3]. In addition, further performance power improvements are delivered by reducing the energy consumption of various components through approaches such as Solid State Drives to replace or augment Hard Disk Drives, processors, memory, and I/O with low energy use states for periods when little or no workload is present, and the use of cache for storage systems. These innovations and system improvements have enabled the equipment to perform and manage ever more complex tasks and processes more quickly.

The demand for improvements in data center energy efficiency from data center operators, data center users, governments, and non-governmental organizations (NGOs) has accelerated the efforts of ICT product development teams to incorporate product features that improve the utilization of energy. The energy efficiency of IT equipment is assessed through three product capabilities: the amount of work that the equipment can deliver for each unit of energy supplied (performance/power profile), the ability to maximize the amount of work done (virtualizing workloads and maximize system utilization), and the intelligence to reduce power when workload is not present. Properly managed and balanced, these three capabilities can collectively contribute to the optimization of the IT equipment’s workload delivery for a given energy use.

IT equipment that is executing a single application or workload is often only performing meaningful work for a small percentage of time—often 10% or less. The remainder of the time, the equipment sits idle, consuming 30–90% of the power and generating 30–90% of the heat that occurs when the equipment is fully loaded. In order to improve system utilization and make better use of each piece of IT equipment, IT equipment companies have developed and deployed server and storage virtualization technologies. These technologies enable the equipment to partition its resources and concurrently support multiple workloads. This increases system utilization to 20–60% or beyond and allows a single server or storage device to do the work that previously had to be managed by multiple systems. In aggregate, the energy consumed and space required by the single server running multiple applications is typically in the range of 20–80% less than a group of servers running individual applications, depending on the application and the capabilities of the original group of servers.

Even virtualized Windows and UNIX servers are idle approximately 40–80% of the time. When minimal workloads are present, it is appropriate to idle all or parts of system components to minimize energy use while maintaining the equipment in a ready state. Processor, memory, and network technologies have been developed and deployed to allow systems to reduce their power use while maintaining specified performance levels or response times when the quantity of workload is reduced. These power management capabilities typically require software enablement, but when deployed, can reduce overall data center energy use by 10–20%.

A consensus has emerged in the ICT industry and the public arena regarding the need to improve data center energy efficiency and by extension, the efficiency of ICT equipment, which has resulted in the development or proposal of voluntary programs and regulations. One of the earliest laws was passed in Japan: the Law Concerning the Rational Use of Energy (Japan Energy Law (JEL)). Established in 1994 as a result of Japan’s commitment to the Kyoto Protocol, JEL established a power per performance metric for servers and watts per GB metric for storage. Each type of product has several categories that are established based on the product architecture. Each category has a weighted average target that must be met by individual products marketed in Japan. The target has been made progressively more stringent through periodic updates. The performance measurements for the best performer in a given category for a given period (the top runner) are used as the new target. While JEL measures one aspect of ICT equipment efficiency, the workload delivered per unit of energy consumed, it does not take into account power management capabilities or the ability to virtualize and achieve higher system utilizations.

In 2006, the U.S. EPA and Department of Energy (DOE) announced a partnership to improve the energy efficiency of servers and data centers. As part of this partnership, the EPA began collecting data on the energy use of data centers under the ENERGY STAR building program. The EPA published ENERGY STAR Version 1.0 Program Requirements for Computer Servers in 2009 and for Uninterruptable Power Systems (UPSs) in 2012. In 2013, EPA implemented Version 1 requirements for Storage systems and Version 2 for computer servers, as well as investigating requirements for Large Network Equipment. There has also been significant interest in this area by legislatures and governments around the world, including the EU, under the Energy-Related Products Directive, Korea, and China.

The remainder of this chapter will provide detail on the technical requirements and the evolution of the ENERGY STAR programs, as well as their impact and influence on the development of regulatory programs. The future landscape of programs and laws and their attendant requirements for workload and system utilization metrics will also be considered.

22.2 Computer Servers

In December of 2006, the USEPA announced that they were initiating a requirements development process for Computer Servers, launching a 3-year effort to develop and publish ENERGY STAR requirements for Computer Servers. Because of the complexity of server systems, driven by the wide range in number and type of component configurations that could be created within a single model type, establishing requirements was a challenge for the ENERGY STAR program. The only product of comparable complexity for which the program had previously undertaken was computer workstations, which had significantly fewer configuration permutations in a single model and a very different power profile from a server. Nonetheless, development of a computer workstation had taken several years, foreshadowing the challenges of establishing a computer server specification.

After releasing the Specification Framework Document and holding extensive stakeholder discussions, EPA evaluated how server systems should be categorized and what energy efficiency criteria should be established. Based on the analysis of the available data, it was determined that servers were best categorized by the number of processor sockets. It was also determined that manufacturers could qualify either individual product configurations or product family. A product family was defined as a range of configurations within a given product model or machine type. Under Version 1, the product family enables a manufacturer to provide the required data on the Power Performance Data Sheet for three representative machine types or models, based on processor socket power and number of cores, to qualify all the configurations for that machine type. The manufacturer is required to certify that all configurations covered by the qualification will meet the relevant requirements. Under Version 2, the product family definition has been broadened, requiring manufacturers to provide the power use, performance, and configuration data for five defined product configurations, thus allowing the qualification of a range of processor socket power and core count within the product family. This simplifies and reduces the testing regime while providing power use and efficiency data on the full range of configurations for the product family.

EPA identified the key server capabilities that should be considered in creating criteria to differentiate energy-efficient servers: power supply efficiency, idle power, the workload capability of the server, server utilization or virtualization capability, a performance power metric, and the ability of the server to report power use and server inlet air temperature to the network. These server capabilities were explored through a set of data requests and four drafts of the ENERGY STAR Computer Server requirements before a final specification document was released in May of 2009. Each of the aforementioned attributes was analyzed in some detail before the final requirements were published.

22.2.2 Idle Power

The server utilization data, with servers sitting idle for significant periods of time, spurred an interest in setting an idle power criterion for server systems. At the time, processor systems had power management functionality that could enter low power modes if no workload was present or that could adjust the voltage and frequency of the processor or individual cores to correspond to the level of workload present in the server. Figure 22.2 shows the different power management modes and their effect on processor frequency (and by association processor power use) for an IBM Power™ processor. The Power7 processor offers four power management modes, each with its own specific power profile. Static Power Saver (SPS) reduces power use, but also can impact system performance, Dynamic Power Saver–Favor Performance (DPS-FP) enhances performance when workload is present and reduces frequency when the processor is idle, and the DPS (power) varies frequency and voltage to deliver power proportional workload processing.

The nominal mode has a consistent frequency and power use no matter how much workload is present. The DPS-FP and DPS power management modes as well as similar power management modes on ×86-based processors can reduce power at idle by up to 60% or more when compared to the power use at the maximum workload. Power management capabilities are also available for memory systems and I/O.

While the server power use data collected by EPA suggested that an idle criterion made sense for server systems, implementation was complicated by the increasing complexity and range of configurations as systems expanded from one processor socket to four processor sockets. The data analysis in Chart 1 of the document “Idle Data Analysis and Charts for the Draft 3 ENERGY STAR Computer Server Specification”² showed reasonable distributions for one- and two-socket systems, but large idle and maximum power increases for four-socket systems as these became more complex. The power range for four-socket systems is illustrated by the range of maximum power (x-axis) for system catalogued in Figure 22.3, which shows the percentage of idle power to the full power for the maximum configuration of different machine types qualified to ENERGY STAR through August 2011.

Based on the available data, EPA set an idle criterion for one and two processor socket systems and required a qualified server system to ship with power management enabled when testing the idle power used. In recognition of the complexity of the four processor systems, EPA required that qualified systems ship with power management enabled, but did not set an idle criterion for four-socket systems. As illustrated by Figure 22.3 for four-socket servers, the ratio of idle power to maximum power for ENERGY STAR qualified systems is consistently above 50%, enabling server systems with power management enabled to reduce ongoing server power use by 30% over the life of the server, assuming that a server is idle 40–80% of its operating time. Similar results can be demonstrated for one or two processor socket systems.

22.3 Storage Systems

EPA released the Storage Specification Framework Document in June of 2009, shortly after publishing Version 1 of the Computer Server Requirements.⁶ Storage systems, with their broad selection of media types, the range of media types/configurations that can be offered with each machine type, and dependency on data placement as well as software functionality to improve performance and system utilization, represented an even greater challenge than servers in establishing ENERGY STAR requirements. EPA collaborated with the Storage Networking Industry Association (SNIA) Green Storage Initiative⁷ to adopt storage system categories and a metric test procedure. The requirements process took over 3 years and Version 1 was finalized in August 2013.

In order to define product categories for storage systems, EPA adopted the taxonomy from the SNIA Emerald™ Power Efficiency Measurement Specification.⁸ version 1 of the Storage System Requirements covers On-Line Categories 2 through 4, though additional categories may be specified in future versions of the Requirements (Table 22.2).

EPA has determined that there are four key criteria for assessing storage system efficiency: power supply efficiency, performance/power metrics, capacity optimization methods (COMs), and reporting of storage system power use. Idle power was determined to be of limited value in assessing system energy efficiency as 70–90% of the power use of storage systems resulted from the high-density drives (HDDs), continuously spinning disks for which currently available technologies have limited, if any, power management capability. COMs were identified as providing improved energy efficiency for the data center, as they can improve system utilization and reduce the number of storage media required to execute a given workload.

22.4 Uninterruptable Power Systems

In May of 2012, EPA released Version 1 ENERGY STAR Program Requirements for Uninterruptable Power Supplies (UPSs).⁹ The requirements identified four classes of UPS products covered by the requirements, one of which is Data Center UPSs intended to protect large installations of ICT equipment such as enterprise servers, networking equipment, and large storage arrays. The devices covered by the requirements include Static and Rotary UPSs with one of two output types: AC-output and DC-output UPS/rectifiers. The UPS Requirements set minimum average efficiency and power factor requirements for AC-output UPSs and DC-output UPS/rectifiers. A UPS system with capacities larger than 10,000 W, which includes metering and communication capability, receives a 1% efficiency incentive (reduced minimum efficiency) to encourage inclusion of the capability to report power use to a networked power monitoring system.

22.5 Networking Equipment

In October 2012, EPA announced its intent to develop ENERGY STAR requirements for large networking equipment. In the discussion document, EPA asserted that it would be possible to reduce the energy use of network equipment by 20–50% through the adoption of efficient technologies. Given the experience with Server and Storage systems, it is likely that the requirements development process will take over years to complete.

The ENERGY STAR program has published or is developing requirements for the key components of the individual pieces of the ICT infrastructure in the data center. The requirements are intended to recognize manufacturers whose products can deliver the highest quantity of workload per unit of energy consumed through continued improvements in performance, power management, and the functionality to increase hardware utilization. The initial work on the ENERGY STAR requirements for data center IT equipment has focused on defining product families and relevant performance/power metrics as well as establishing basic requirements for product energy use characteristics: power supply efficiency criteria, idle power requirements for one- and two-socket servers, the enablement of power management capabilities on shipped products, and the collection of performance/power data to inform the development of criteria in subsequent versions of the requirements.

Currently available enterprise ICT equipment offer data center operators significant functionality to reduce data center energy use if the functionality is enabled. Implementation of power management on a server that has utilization of 30% can reduce power use by 20–40% depending on the extent of idle power savings. Utilizing virtualization capabilities on server and storage systems can enable a single machine to do the work of 6–10 current or previous generation technology machines. Improvement in ICT equipment energy efficiency and functionality to manage more workload will continue as manufacturers innovate to deliver capabilities that both improve the performance and the efficiency of the ICT equipment.

22.6 Future Trends in Product Energy Efficiency Requirements

Data center energy efficiency, as measured by the amount of work the data center delivers for each unit of energy it consumes, is influenced by all the data center systems: the IT hardware, management of IT workload placement, the facilities hardware, the management of the data center thermal profile, and management systems, which integrate some or all of these activities. As the building blocks of the data center, IT and facilities equipment will continue to be a focus of voluntary and regulatory energy efficiency programs around the globe. ENERGY STAR, in collaboration with industry stakeholders, has begun the effort to define relevant metrics to assess the energy efficiency of data center equipment. These initial efforts have made it clear that establishing energy efficiency criteria for these complex systems is a difficult undertaking. The range of system configurations, functionality, and types of unique, distinct workloads supported requires a flexible approach to assessing the energy efficiency of enterprise-level ICT equipment. The pace of technological change and innovation in the industry introduces further complexity, as standards and metrics established today may be rendered obsolete or marginalized by technology changes. Finally, the lack of general metrics, as opposed to workload specific metrics, to assess the performance/power profile of the equipment and the lack of measured system data for those metrics that do exist require a measured, incremental approach to the development of workable ICT equipment energy efficiency standards. While progress has been made, much work remains to develop standards and metrics that provide a meaningful assessment of the energy efficiency of ICT equipment.

The initial steps to manage ICT equipment efficiency have focused on simple, easily measured requirements: power supply efficiency, enablement of power management functionality, and the measurement and reporting of the maximum and idle power use of a range of configurations of a given machine type or model. Some or all of these requirement types have been implemented in various jurisdictions around the globe: Japan (the Japan Energy Law), Mexico (Energy Use Reporting Requirements), the European Union (Power Supply Efficiencies), and the United States (ENERGY STAR requirements). These requirements have created the first tier of efficiency requirements and begun the process of more complete reporting of power use and performance/power metrics across the range of system configurations available in the market.

The next step, which is expected to unfold over the next several years, is the implementation of performance/power requirements and grading systems in various jurisdictions. The ENERGY STAR program has established testing and metrics protocols to serve as a starting point for energy efficiency requirements and has established a clear path to collect performance/power data for computer servers and storage systems. The collected data will be used to establish performance/power criteria, with a top runner or leadership focus, for these product types over the next 2–5 years.

Several government entities have also declared their intent to establish performance/power regulatory requirements in this same time frame, including, but not limited to, California, China, European Union, and Korea. These entities have launched or are completing studies or regulatory development efforts. China and Korea are most advanced in these efforts. In China, the Ministry of Environmental Protection released Technical Requirements for Environmental Labeling of Products—Server,¹⁰ which covers one to four processor socket rack and tower computer servers, blade servers, and storage servers. The program went into effect in April of 2011 and is voluntary, but will be used to inform government procurement decisions. The server requirements draw heavily from the ENERGY STAR requirements. The storage requirements include power supply efficiency, a watts/IOPs criteria, and a criteria for idle to maximum power draw. Korea has indicated its intent to establish a computer server energy efficiency grading system in 2014/2015 for one and two processor socket servers using some limited combination of SERT worklets. The European Union and the California Energy Commission (CEC) have indicated their intent to initiate a study on Computer Server energy efficiency requirements in 2013 and 2015 respectively with the intent of establishing requirements in 2015 or 2016. Ideally, these programs, and others like them in other jurisdictions, will build on the testing protocols, metrics, and measured data collected through the ENERGY STAR program. They will help standardize the testing as well as data generation and collection processes, while enabling individual governments to set metric criteria appropriate to the conditions and interests of their jurisdiction.