20.1 INTRODUCTION

Data centers occupy an increasingly valuable place in the lives of everyone. This is true because every request for data from a search engine, every piece of music played, and most all of the digital life that each person leads come to us via data centers.

The constant need to access digital information creates the requirement for data centers with the highest uptime possible (Fig. 20.1). Because of this, data centers are designed to have their Information Technology (IT) systems and their supporting Operational Technology (OT) systems operate at 99.999% or higher availability. This means that IT networks and OT electrical, mechanical, and security systems are designed with redundancy to ensure failover options are available in the event of the disruption of a major system. Large sums are spent to ensure redundant electrical and mechanical system paths are available in a data center and, yet, a single click of the mouse from a hacker can bring down an entire data center for hours.

In this light, network engineers have been particularly sensitive to create a cybersecure environment for the IT systems within the data center. Cyberattacks continue to rise and the need to detect and defend against such attacks is a clear and present need. Yet, while large sums of money are spent to protect the cybersecurity of IT systems, a loophole exists with the lack of cybersecurity within the OT networks of data centers. This creates the prospect that even the best electrical and mechanical designs can be compromised by even an inexperienced hacker, let alone a nation‐state bent on a destructive attack.

Because there are vast sums of written knowledge on the subject of IT cybersecurity, it is not the purpose of this chapter to try to add further to that body. The purpose of this chapter is to discuss the specifics of the much lesser‐understood and vitally important topic of OT cybersecurity. In this chapter, we shall look at the following topics concerning OT cybersecurity:

• Background of OT connectivity in data centers
• Vulnerabilities and threats to OT systems
• Legislation covering OT system security
• Cyber incidents involving data center OT systems
• Cyberattacks targeting data center OT systems
• Conclusions
• References
• Further Reading

20.2 BACKGROUND OF OT CONNECTIVITY IN DATA CENTERS

OT forms the base infrastructure in which all data is created, stored, and transported. All data is created from the electricity fed by critical power systems such as Uninterruptible Power Supplies (UPS's) and Power Distribution Units (PDU's). Data is no more than a series of electronic 0's and 1's created from the critical power systems. Once created, data is no different from any other perishable item. It must be stored and transported in a proper environment to ensure that the 0's and 1's are not destroyed due to a thermal breakdown of one of the systems. Finally, data have enormous value and they must be protected, both physically and electronically, in order to keep thieves from accessing, stealing, or destroying any of that data.

A data center can be seen as a hierarchical structure with OT systems at the base to create and sustain IT systems and their data. Any interruption in the OT systems will affect both the IT systems and their data. Any interruption in the IT systems will affect the data. As IT systems standardized on Ethernet network in the 1990s, manufacturers of both IT and OT systems began to work on standardized protocols in order to continuously monitor these systems.

The goal of every manufacturer of data center equipment was to detect faults in their systems as early as possible to avoid disruption of data center operations and the possible destruction of data. Three protocols emerged from this work:

1. Simple Network Management Protocol (SNMP)
2. Modbus/TCP (Transmission Control Protocol)
3. BACnet

These three protocols have become widely distributed throughout the equipment in a data center. SNMP is present in most every IT system and in most all power and cooling systems. Modbus is present in many power and most all cooling systems. BACnet, owing to its creator the American Society of Heating Refrigeration and Air‐Conditioning Engineers (ASHRAE), is present in most all cooling systems.

In their most basic iterations, these protocols provide the ability to monitor system health in IT and OT systems through central management servers. Network Management Systems (NMS) provide the bulk of IT management systems and are universally SNMP‐based. Building Management Systems (BMS) provide the bulk of OT management systems that are able to support SNMP, Modbus, and BACnet. In addition, it is common for other systems to be in place within data center management structures including Data Center Infrastructure Management (DCIM) systems. These systems query devices within a data center and are used by managers to ensure uptime of operations and the smooth processing of data.

NMS, BMS, and DCIM systems have become universal in data center operations. In fact, they are so pervasive within data centers that there has been considerable debate in the past few years as to whether the volume of SNMP, Modbus, and BACnet traffic is actually causing problems within data centers. Suffice it to say, these systems send and receive massive amounts of data within a data center in order to keep a highly granular view of the present conditions within the data center.

These three common protocols used for data center management were created in an era when data centers and, hence, the amount of OT and IT infrastructure to manage, were much smaller. Because of this, little attention was paid to the network overhead of these protocols at the time of their beginnings. In addition, cyber/physical security was not seen as a great challenge in a time of smaller data centers where physical eyes could see most everything that was taking place and cybersecurity had not become a global threat. In the next section of this chapter, we shall examine each of these three major protocols as to their cyber vulnerabilities and we will also take a look at other security threats that are present within major OT systems.

20.3 VULNERABILITIES AND THREATS TO OT SYSTEMS

The dominant communications protocols used for IT and OT systems were all developed with a great deal of input from vendors and customers. In the case of SNMP and BACnet, formal meetings were held among stakeholders over a period of many years to develop the protocols that are now in use. This section explores the development and implementation of these protocols and examines how cyber vulnerabilities came to exist in each.

20.3.2 Modbus/TCP

Modbus/TCP is sometimes called the grandfather of all present‐generation management protocols as its origins date to 1978 when it was a serial‐based protocol. The network version, known as Modbus/TCP, was developed in the 1980s and used the same data structure as the serial protocol to keep backward compatibility. As a serial‐based protocol, Modbus was developed with no security or encryption whatsoever. That is to say, its messages are passed over a wired or wireless connection in plain text so that anyone with a network sniffer program can read messages and can send messages to a Modbus‐based system. Figure 20.2 shows a Modbus transmission captured by a common network sniffer, Wireshark.

As can be seen in the screen shot of the sniffed Modbus/TCP information, the message and all of its contents (which are actually in Hexadecimal form and are translated by Wireshark) can be seen by anyone who is connected to that network.

This inherent insecurity in the Modbus protocol has led to many warnings from government and other organizations. The California Energy Commission specifically noted the danger of using Modbus devices within the State's Demand Response programs due to the potential for damaging the electrical grid and its customers.

The Modbus protocol has become the de facto industrial communications standard…The Modbus protocol lacks the ability to authenticate a user and hence middle man attacks can easily take place in Modbus. [2]

Clearly, any Modbus device will require an external security system to protect that device and all the systems that it supports. As with SNMP, this chapter will cover security strategies to enable the safe use of Modbus‐based systems in Section 20.7.

20.4 LEGISLATION COVERING OT SYSTEM SECURITY

There are a number of pieces of legislation that actively cover cybersecurity. Most are aware of the Health Insurance Portability and Accountability Act (HIPAA), which is a piece of national legislation that provides strict security requirements for healthcare and related facilities. Many are also aware of the General Data Protection Regulations (GDPR), which is the European Union's law that controls the protection of Personally Identifiable Information (PII) of those who live in an EU country. While awareness of these regulations is fairly high among data center professionals, the understanding of the provisions which apply to OT is scant. It's important for the engineering team that designs data centers as well as the facilities team which runs them to understand the key rules within these standards that apply to securing OT systems. This section will examine these two standards as they are related to OT systems and will also look at the recently passed California Law known as SB327 for securing connected devices. SB327 is becoming a template for other proposed local, state, and national legislation with regards to OT security and securing the Internet of Things.

20.5 CYBER INCIDENTS INVOLVING DATA CENTER OT SYSTEMS

While cyberattacks make headlines on a regular basis, the most common form of cyber disruption is known simply as a cyber incident. The U.S. National Institute of Standards and Technology (NIST) defines a cyber incident as follows:

An occurrence that actually or potentially jeopardizes the confidentiality, integrity, or availability of an information system or the information the system processes, stores, or transmits or that constitutes a violation or imminent threat of violation of security policies, security procedures, or acceptable use policies.

Note that a Cyber Incident may or may not be caused by an intentional act. It is an intentional action that causes an event to be classified as a Cyber Attack as opposed to simply a Cyber Incident. In some cases, as in the case which will be discussed in this section, it may not be determinable as to whether the incident was an intentional event or not. In such cases, the classification of cyber incident is appropriate.

A number of cyber incidents have affected systems inside of data centers. Some of these have become public knowledge and were published in the newspapers and in online publications. One such cyber incident that was widely publicized involved two British Air data center in the UK. On May 27, 2017, British Airways noted that their Boadicea House and Comet House data centers experienced major power outages due to an electrical grid surge. However, their power supplier, the National Grid, advised that there were no problems with its power systems within the vicinity of these data centers. Consequently, any loss of power had to originate from systems within the data center.

In the subsequent investigation, it was discovered that the UPS control at Boadicea House was altered into a configuration that resulted in a hard shutdown. The UPS is supposed to switch to battery in the event it sees power that is outside of a predefined tolerance window. In this case, rather than switch to battery in this alleged power surge, the UPS simply turned‐itself‐off. Further, because the UPS was in‐line with all power connections, the startup of the generator provided no immediate relief to the problem. The net result of the UPS shutting down and remaining down was a significant interruption to data operations that lasted for hours and cost the company enormous sums of money and operations disruptions. While the Comet House data center also failed at, or near the same time as Boadicea, no additional information on its failure has been made available.

How could a UPS fail to perform its normal functions and, rather than switch to battery, simply turn‐its‐systems‐off? The answer lies in the manner in which the SNMP communications card in this unit and in many units used throughout the world operate. One of the features of a UPS card that has been present since the early days of SNMP cards is the option to shut the system down if it experiences a power anomaly. There are a number of ways in which this can be programmed into the SNMP card to create a disaster including:

• Setting the amount of time that the UPS remains on battery before shutting down to 0 seconds
• Setting the battery percentage shutdown trigger to shut down the UPS at 100% battery charge remaining
• Setting the battery time remaining to 0

These settings are presented on many SNMP cards as a legacy feature from the 1990's. During that time, many smaller UPS units were connected to a single server and UPS manufacturers provided server‐resident software known as a “graceful shutdown” software to be triggered in the event of the UPS transferring to battery power. In this combination of UPS SNMP communications and server software, the user was allowed to choose to shut down the UPS once the server had gracefully shut down. The UPS SNMP card allowed the user to choose to shut down the UPS via simple timers or by battery time or charge percentage remaining.

Clearly, today's data centers operate with the philosophy that you should never shut down the server, which is the antithesis of the logic presented in these older UPS units. Unfortunately, many companies have failed to remove the shutdown options within their SNMP cards and this has led to a number of incidents such as the British Air disaster.

SNMP cards used in UPS systems are simply an example of features that can be exploited to create a cyber incident or to stage a cyberattack. It is incumbent upon the data center engineer and the facilities manager to understand all of the options that are present within the communication cards of all their mission critical systems. If shutdown features are exposed in these cards, the cards would need to be removed, reengineered, or have their access restricted to a single manager. No system located in a data center that is manned on a 7 × 24 basis needs to have an option to shut down one of their critical OT systems via a network connection.

20.6 CYBERATTACKS TARGETING OT SYSTEMS

Cyberattacks are events when there is clearly an attacker(s) that have attempted to violate security systems within a location. The NIST formally defines a cyberattack as:

An attempt to gain unauthorized access to system services, resources, or information, or an attempt to compromise system integrity.

Thus, a cyberattack is an intentional effort to obtain unauthorized access to systems and their data or to destroy the integrity of systems. A number of cyberattacks have taken place and are continuing to take place on OT systems inside data centers. In many cases, the attacks simply use these systems as a backdoor to steal data. In others, systems have been shut down. In most cases where data theft is involved by using OT systems as backdoors, the actions are not recognized until many months after the event. One such attack that used OT systems as a backdoor to steal data was discovered and publicized but, only because the hacker responsible for the attack posted the means which they used to steal the data in an online site.

In this specific instance, a hacker used a Rack Power Distribution Unit (PDU) as a means to enter a server within that same rack. Rack PDU's represent a pernicious threat both because of their proximity to data servers and because they are ubiquitous throughout data center cabinets and racks. The targeted company was Staminus Communications, ironically which is a provider of cybersecurity protection services. In this case, the Staminus data servers were penetrated and data was stolen, all of this using a path through the rack PDU. The data which was stolen were then published to embarrass certain companies and individuals as part of the taunting engaged‐in by the perpetrator.

In another case, an international incident evolved from a Russian attack on a Ukrainian power plant which involved attacking a UPS in their small data operations center. In this case, a UPS communications card was penetrated and malware was loaded onto that card. The malware was set to turn‐off the UPS at the exact same instant as another attack was shutting down the master breaker which fed power from the plant to the grid. By shutting down the UPS at the same time as grid power was interrupted, this nation‐state attack succeeded in creating a blackout in the control room of the plant and left the managers unable to communicate internally or externally.

Power systems are a target of cyber criminals because of their ability to be in proximity to data and because they can be used to shut‐off critical power. This makes them prime targets to use as backdoors by data thieves and to use as a means for sabotage for those so inclined.

Many cybersecurity experts look at large scale cyberattacks on infrastructure, such as the attack in the Ukraine, as test cases that are being perfected for larger attacks against government and businesses. Because of this, it should not be a surprise that the SolarWinds cyberattack used that company’s SNMP Network Management System (NMS) as the basis for its massive attack on the United States and other countries.

An NMS like the SolarWinds Orion is actually a perfect tool for a cyberattack because it is routinely used to monitor and control both IT systems, such as servers and switches, as well as OT systems like Rack PDUs and even larger UPSs. The key is that these systems both monitor devices using SNMP Get requests and they can also control devices with SNMP Set commands. In the SolarWinds attack, a piece of malicious code was inserted into the software update of the Orion system and this code gave its perpetrators unfettered use of the system’s ability to both monitor and control critical IT and OT systems at will.

Two things are clear from this attack. Firstly, that any system which uses SNMP can be used to gain access to IT systems to surveil and steal data from these systems. Second, that SNMP‐based systems can be used, much as they were in the Ukraine, to plant code in OT systems to surveil and accomplish other nefarious purposes. The full extent of the damage done in this attack will likely not be known for months as it is believed that many pieces of malware are now in place but are sitting, undetected, until programmed to act.

In order to understand the level of vulnerability that exists in presently installed power systems, the author undertook a study to view power systems located in data centers yet, were directly viewable on the Internet. It was discovered that a significant number of data centers have such systems that were directly connected to the Internet at the time of the viewing. This research turned up well more than 100,000 power systems in data centers that can be discovered. In addition, a large number of environmental control systems were discovered as well.

It is unfortunate that power devices have been so readily connected to the Internet but, by publishing this information it is hoped that the practice can be stopped and reversed for existing systems. In light of this, it should be the goal of the data center engineer and the facilities manager to ensure that all applicable security regulations and laws are being followed and, most importantly, that all basic security practices necessary to ensuring the safe and reliable use of OT equipment are followed. In the following section of this chapter, a roadmap to implementing a successful cybersecurity strategy for data center OT systems will be put forth.

20.7 PROTECTING OT SYSTEMS FROM CYBER COMPROMISE

In this chapter, it has been shown that OT systems within data centers can be compromised and that cyber incidents and attacks have resulted in large losses to those who had used that data center for processing. With a number of pieces of legislation now active and new laws being debated about cybersecurity for OT systems, it is the time for all data centers to work to mitigate the possibility of a cyberattack on their OT systems. In this section, we will explore sound and proven ways to greatly reduce the possibility of a cyberattack on these systems.

As we look to protect OT systems from cyberattacks as well as cyber events, the first thing that must be noted is that there are simply no known devices that will protect every system from all forms of cyberattacks. Because of this, it is necessary to provide a layered system of multiple types of defenses in order to best thwart attacks. Cyber events, whether accidental or not, can be reduced to near zero with proper layering of cyber defenses. Cyberattacks, while never having their possibility reduced to zero, can have their threat greatly reduced by a multilayer defense strategy.

The four layers of defense that every organization should use as a minimum for its OT cybersecurity strategy include the following:

1. Segment the OT network from the IT network
2. Create a perimeter firewall around the OT network
3. Provide Virtual Private Network security to each management console
4. Use 2‐Factor Authentication when available

These four defense strategies will be discussed as individual items in the following sections. By employing these four layers within an OT network, a system of speed bumps is created which greatly adds to the complexity that must be navigated in order for an unauthorized individual to gain access to a piece of equipment. While it is also true that these layers do slightly increase the time to login to systems for the local staff, there is always a trade‐off between convenience and security. In order to have a reasonable amount of security present in a system, there must be some speed bumps which reduce some measure of convenience.

This is no different from other types of security with which we have all become accustomed. For example, the use of airport screening systems is now a standard as is the use of two‐factor authentication to log into critical IT systems. It should be not different for the need to protect OT systems. These four layers of defense will be discussed in order in the following sections.

20.7.3 Install a Virtual Private Network for Each Management Console

Most data centers use a variety of management systems including a BMS, DCIM, Energy Monitoring System (EMS), and others. In general, users of these systems have access to them both locally and remotely. Whenever a remote connection to one of these systems is used, there is the possibility of eavesdropping and hacking. Because of this, it is vital that each individual system have a dedicated VPN system.

Through the use of segmentation and the use of a perimeter firewall, two important speed bumps are erected in a network system. By adding a VPN to each console system, as shown in Figure 20.3, you add an addition speed bump by creating a secure, dedicated login system for the individuals who may access one of those systems at any given time.

20.7.4 Use 2‐Factor Authentication

Many different types of systems now offer the use of a 2‐Factor Authentication (2FA) in order to secure logins and system changes. In some cases, these are offered directly from the vendor but, 2FA software can also be purchased from a number of organizations and then added to console systems such as BMS, DCIM, EMS, etc.

A 2FA system is an added level of security that adds a second “personal touch” layer of authentication to a system login or system change request. Typically, a piece of 2FA software is loaded on the host server and then connected to an external hardware device, such as a finger print scanner or it is virtually connected to a mobile app. Upon logging into that system, the system will either request that the user scan their finger or, will send a message to their mobile phone and await their approval of the login as shown in Figure 20.4 below. The same process can be configured in the event of a major system change request, where a second layer of authentication is needed before the change is implemented.

2FA systems are standard at most critical government facilities and are now entering many data center IT applications as well. Because the OT network must remain fully operational to support the IT applications, it is a certainty that 2FA must be adopted for all key console systems that monitor or manager OT systems within a data center.

20.8 CONCLUSION

Cyber threats are a real and growing problem for managers of IT data systems and now for managers of OT data center infrastructure as well. Intentional cyberattacks and unintentional cyber events can be equally destructive to the data that is being processed, transported, and stored in a data center. With a single click of a mouse, a UPS system or other mission critical OT system can be shutdown, offsetting the best electrical engineering and mechanical engineering design efforts.

Fortunately, a wealth of cybersecurity resources are available to the facilities engineer and manager. These resources include the latest generation isolation standards, firewalls, VPN's, and 2FA systems. The use of a layered approach employing all of these options creates a well‐fortified defense for the OT network and its systems. This creates the protection needed to ensure that the high uptime goals for the data center can be reached and that they will not be compromised by a cyber event or cyberattack with respect to an OT system.

REFERENCES

1. [1] The California Energy Commission. Best Practices for Handling Smart Grid Cybersecurity. Final Project Report; May 2014.
2. [2] BACnet, a Data Communication Protocol for Building Automation and Control Networks. Addendum g to ANSI/ASHRAE Standard 135‐204.
3. [3] Traynor P, Lawrence N. Under New Management, Practical Attacks on SNMP Version 3. Georgia Institute of Technology; 2012.