14.2.1 RINSE

The Scalable Simulation Framework (SSF) [20] is a framework that can be extended to support complex systems such as fluid dynamics, raytracing, and computer networks. Based on SSF, RINSE is a network simulator that serves as the basis for this research and has been through several iterations. RINSE is currently maintained by a team of developers working on the TCIP project at the University of Illinois. Developers are working on various topics including wireless communications, switch modeling, and intrusion prevention through game theory. RINSE has traditionally been used as a wireline network communications simulator to explore malicious behavior in the Internet—namely, worms, botnets, and denial-of-service attacks. The scalability that RINSE allows and the similarity of the Internet to the SCADA infrastructure make it a good candidate for simulating SCADA traffic.

RINSE has a number of properties that make it amenable to large-scale simulation. SSF enables highly parallelizable models by partitioning graphs into sub-models. These submodels, which compose the main model, are divided such that communication between them is kept to a minimum, which allows the maximal advantage of multicore systems. Also, RINSE supports multiple resolutions. That is, RINSE can calibrate the fidelity of a simulation to ensure that it runs in real time. It does this by providing a fluid model [21] for traffic and allowing both full-fidelity traffic and fluid models to exist within the same simulation. These fluid models exist for various components of RINSE such as transport protocols, routers, and links. These models also exist for modeling network topologies, and by utilizing these models, RINSE achieves a significant speedup over using a full-resolution model.

RINSE also supports simulation speeds that are faster than real time such that the simulator is not dependent on wall-clock time to advance the simulator time. This is important when interfacing with real devices through emulation. Instead of synchronizing with the wall clock, the simulator keeps track of its own timeline and, where possible, computes traffic ahead of when it will be needed. Combined with prioritizing emulated packets, RINSE can provide a large-scale simulation environment that is not slowed down by interacting with physical devices.

Additionally, RINSE is modular in such a way that new protocols and new models can be developed. By implementing new protocols through extending the base ProtocolMessage, RINSE can support the integration of existing or nascent protocols. Likewise, the ProtocolSession class allows us to develop new layers in the protocol stack. In the case of DNP3, for instance, the relays are derived from the ProtocolSession class and communicate with the data aggregators through messages derived from the ProtocolMessage class. The ProtocolSessions that comprise a host are indicated in a file by using the Domain Modeling Language (DML). DML describes a model as a tree of key-value pairs, and a file containing a DML model is passed to the simulation engine at runtime. These key-value pairs are then interpreted and used to derive the network topology, hosts, protocols, and traffic patterns. This allows RINSE to simulate any number of models without having to recompile, thus increasing the turnaround time between testing iterations of the same model.

To use RINSE to model the power grid, we must first define the protocol that virtual SCADA devices will use to communicate with one another. When simulating a new protocol or device, there always exists the question of how accurately we must model the proposed design. The option of tweaking parameters of a different model to estimate the new model rarely works because often there are fundamental features that do not lend themselves to be portrayed by a different model. For instance, we could hypothetically model the power grid as a purely Transmission Control Protocol/Internet Protocol (TCP/IP)-oriented network, with parameters such as poll interval, packet size, latency, and bandwidth set to portray the parameters present in the power grid. However, this would completely ignore the idiosyncrasies of the grid itself. For instance, if there is a vulnerability in the protocol specification or implementation, it would be agnostic to the network layout and configuration. In this manner, we can say that to capture the quirks and security assumptions of a protocol, we must model that protocol as accurately as the scale of the network permits. Here, we provide a brief overview of the three DNP3 layers.

14.2.2 DNP3 Overview

Designed to provide interoperability and an open standard to device manufacturers, DNP v3.0[22] has gained prominence in the U.S. electrical grid as the communication protocol among power grid equipment. Versions 1.0 and 2.0 were never released to the public. So as to not confuse this protocol with previous implementations, this chapter will refer to DNP v3.0 as DNP3. DNP3 is designed to operate in environments with a high electronics density. These environments can be fairly noisy, and therefore, DNP3 is designed to be as robust as possible with respect to detecting and recovering from error. Cyclic redundancy check (CRC) bytes are extensively used to detect when bits have been flipped and a small frame size (292 bytes) is used to localize errors and reduce the overhead imposed by resending frames.

The protocol was designed as a stack of three layers: the Data Link Layer, the Pseudo Transport Layer, and the Application Layer. The physical medium is generally either Ethernet or RS-485. Since these standards are common, DNP3 can be run over existing networks or networks can be built from the ground up to support DNP3 SCADA networks. The Data Link Layer can either be run directly on the physical medium or it can be encapsulated by other protocols as seen in Figure 14.1. This layer provides framing information and reliability. On top of the Data Link Layer is the Pseudo-Transport Layer. This simple layer is used to support fragmentation. Finally, the top layer is the Application Layer that acts on behalf of the user for requesting/confirming/sending/receiving requests and data.

14.2.3 Attacking DNP3

As one of the primary protocols used to transmit information in the power grid, it is important that we model DNP3 as accurately as possible. One reason for this is that there exist deficiencies in the protocol that allow it to be compromised. For instance, related work has developed a taxonomy to describe attacks against DNP3 that comprises 28 generic attacks (of which there are 91 specific instances) [26], thus showing how vulnerable the protocol is. Countermeasures for these holes could be tested in a simulation environment before becoming part of the standard. Other work has compiled a survey of SCADA-related attacks and discusses techniques such as attack trees, fault trees, and risk analysis as they pertain to CIs [27]. Indeed, much research has been done on both SCADA security gaps [12,28,29,30] and their countermeasures, including data set security [31], SCADA-specific intrusion prevention systems [32], and encapsulating DNP3 in another protocol such as SSL/ TLS (commonly used for secure internet communications, operating at layer 6 of the network stack) or IPSec (a less common protocol used for securing internet connections at network layer 3) [33]. In general, the power grid is susceptible to attack, and while the exact nature of these vulnerabilities is out of the scope of this chapter, this knowledge provides the impetus to model DNP3 accurately enough to reproduce the vulnerabilities in our virtual test bed.

14.2.4 Modeling DNP3

No matter the actual implementation of the protocol in the simulator, the protocol must be able to handle all three layers of the stack (Data Link, Transport, and Application). Inside the simulator, we treat the three layers as a combined payload to be transported by TCP/IP, and packets are routed using the IP header as opposed to the Data Link Layer header. However, when dealing with emulated packets, the Data Link source and destination fields are used to direct packets to and from the proper hosts. More information about this can be found in Section 14.2.3, which discusses how translation is done between the IDs of the virtual hosts and their emulated DNP3 addresses.

Dealing with emulated packets is an important concern for our use of this virtual DNP3 model. Being compliant with physical devices enables many potential use cases. Without external communications, the RINSE model would provide limited usefulness. It would provide background traffic, metrics regarding correctness and scaling of technologies, and insight into large-scale SCADA networks. However, by being interoperable with physical equipment, more use cases are available that involve a control station. This provides benefit by potentiating training with human-in-the-loop event analysis, incremental deployment analysis, and attack robustness analysis [34].

14.3.1 Virtual Relays

Relays are responsible for control of physical lines as well as gathering data pertaining to their operation. Relays must determine various characteristics of these lines, such as phase angle, voltage, real and reactive power, and other such values. Additionally, relays must provide information about their own operation such as status values, counters, and synchronization efforts. It is unrealistic to support all of these features in a virtual relay, as that would severely hamper scalability. As such, it is important to determine which subset of features must be supported to provide the largest functional coverage at the lowest computational cost. The functionality that covers a large majority of typical requests is as follows:

• Read Class X data—X can be 0, 1, 2, or 3. Different values of X represent different priorities, where 1 refers to data that changes the most often, and 3 the least. Class 0 data includes all data points. Since PowerWorld provides some of these values either directly or indirectly (through simple mathematical operations), this sort of data can be provided to the data aggregators at the substation level.

• Turn breaker on/off—A request sent to the relay to turn on or off can then be passed on to the State Server.

When not processing a request, the relays are sitting idle. During this time, they are waiting for a data aggregator to issue a request. Currently, the relays support a limited subset of what their physical analogues support. The relay will thus determine if it can answer the request, and if it is either a data read request or a command, it will perform the required actions. If the request is a read, then the relay will retrieve data from the State Server through its own designated shared memory. If the data aggregator request is a command, the relay takes a corresponding action by writing to shared memory. The State Server is then responsible for forwarding this command to PowerWorld. Once the appropriate action is taken, the relay prepares a response. The response contains either the data or a confirmation, depending on whether the request was a data request or a command, respectively. Once sent, the relay reenters the idle mode and waits for the next request. Some of the difficulties of detecting events and issuing unsolicited responses are discussed in Section 14.6.2.

14.3.2 Virtual Data Aggregators

The virtual data aggregators have a control flow similar to that of the virtual relays. However, since they require data from the relays instead of PowerWorld, their polling patterns are somewhat different. Instead of sending out one data request per window, an aggregator will send out n requests per window, where n is the number of relays reporting to it.

The control flow for the master thread, seen in Figure 14.3, describes how the virtual data aggregator polls the relays for which it is responsible. First, the data aggregator waits t seconds before beginning the round of requests. Starting with relay 0, the data aggregator prepares a request and waits the appropriate amount of time that its physical analogue would take to produce the request. Then it sends the request and waits for the reply. Once it receives the response, the data aggregator then begins processing the command. Processing the command depends on which data were requested and how the response is formatted. In the basic scenario, the data supplied by the response is simply entered as a corresponding entry into a table, which can be queried by the control station. Once the response is processed, the data aggregator moves on to the next relay and starts the request flow again. Once the data aggregator has polled all of the relays in its list, it waits until the next polling period and resets the current relay to 0.

The data aggregator also acts as a source of data for the control station. This responsibility is handled by the virtual data aggregator’s slave thread. The slave thread essentially acts as a server, remaining idle until it receives an incoming request from the control station. When it receives a request, it begins to process it. The data aggregator then makes a decision based on the function code in the request. If the function corresponds to a read, then the data aggregator will provide the requested data out of its table. If the function is a command, then the data aggregator passes the command to the correct relay and waits for a response. Once either the relay responds or the data is ready (depending on which function code was sent to the data aggregator), the data aggregator prepares a response to the control station. After this, the data aggregator waits for an appropriate delay, corresponding to the delay its physical analogue would take for preparing the packet. Finally, the data aggregator sends the response and goes idle.

14.3.3 State Server

The State Server acts as a single point of contact between the RINSE and the PowerWorld simulation environments. The same procedure that generates the DML file also generates a mapping between PowerWorld entities and the virtual relays that are assigned to monitoring them. Using this mapping, the State Server generates requests according to the PowerWorld Application Programming Interface (API). The requests are split into four different groups—lines, generators, loads, and shunts—which correspond to power grid components. After PowerWorld processes these commands, it sends back a response with the appropriate values.

PowerWorld was designed with a server thread, which can serve data to external applications, for which no DNP3 converter had been developed. Hence, we have created this State Server, which consists of four similar tasks being carried out in succession. Polling periods start by preparing a request for shunt values from PowerWorld. After waiting for a response, it processes the response and moves on to requesting generator values. The State Server then repeats the process for load and line values. Finally, the State Server waits t seconds before starting the process over again.

Once the data is transferred from PowerWorld to RINSE, it is the responsibility of the State Server to partition the data. Depending on which values were requested, the amount of data a relay will receive can vary from five to nine bytes total. Then, the State Server splits up the response into appropriately sized portions and shares them with the relays through shared memory.

This choice of using shared memory as opposed to explicitly passing a message with the data was made to better represent the practical circumstances. Its sole role is to deliver data from PowerWorld to the virtual relays, and since the State Server has no physical analogue, its latency should be no greater than the time it takes for a relay to measure the line it is connected to—essentially zero. Transmitting this information over the routing network would introduce latency into the system that has no real world analogue.

14.4.1 Lab Setup

To connect our complete virtual SCADA system to physical devices, we rely on the generosity of several donors, who have donated a number of SCADA devices, software, and technical support. VPST currently utilizes two SEL-421 relays, an SEL-3351 data aggregator, and a workstation running the OSI Monarch OpenView Energy Management System (EMS) software suite [36]. The relays are each attached to an Adaptive Multichannel Source (AMS) that supports lab testing of Schweitzer Engineering Laboratories (SEL) devices by providing sinusoidal waveforms that feed into the relays.

Our communications simulator, RINSE, provides the SCADA simulation for our local lab environment. Its components have been described in Sections 14.2.1 and 14.3. Using these components, we can generate any desired topology. Shown in Figure 14.4 is a simple topology that includes a virtual control station proxy, multiple substations with one data aggregator each and multiple relays per substation, and a State Server that communicates with a virtual PowerWorld proxy. The PowerWorld proxy is directed through emulation to a computer running PowerWorld, which runs a steady-state power simulation. PowerWorld has an API that can interface with external devices using its proprietary protocol. In this figure, we do not show PowerWorld interfacing with anything besides RINSE, although the capability is there, and may be beneficial to certain use cases.

To provide emulation support, we also utilize proxies hosted on physical machines. Although they are not shown in Figure 14.4, our setup would not function without them. Since RINSE operates on its own virtual private network, to direct traffic toward one of the virtual nodes, the traffic must originate from within that virtual private network. To allow external traffic passing, RINSE comes equipped with a gateway that supports OpenVPN connections. >From a workstation, we connect using an OpenVPN client to the OpenVPN server on RINSE’s host machine. This allows a physical machine to be in the same private address space as the virtual machines. Once this connection is complete, we can redirect this proxy machine’s incoming traffic to any destination inside RINSE. We also can redirect any traffic that arrives on the private network address to physical computers. In this manner, the State Server can communicate with PowerWorld in both directions. Also, by redirecting different ports to different addresses, we can allow the Control Station to poll any device in RINSE.

The lab is on a switched network, which provides two useful capabilities. The first is that we can use WireShark to examine traffic between all of these hosts. This allows us to troubleshoot communications between the devices, which is important since the relays only support maintenance through telnet. For instance, if we notice that the control station is no long updating, we can pinpoint the break in communication. In addition, it also allows us to inspect the packets themselves. This has proved helpful for meeting timing requirements as well as modeling both the DNP3 and the virtual hosts.

When building a system such as this test bed, fidelity is an important concern. That is, the system must perform in the same manner as the operational analogue. There are varying degrees of fidelity, and development of this system has gone through phases addressing different levels of fidelity. First, the system must interact with physical equipment in such a way that the physical equipment does not realize that it is operating inside a partially virtualized environment. This is termed interoperability. Second, the virtual hosts must accurately reflect characteristics such as response times and accuracy of data. This can be achieved by tuning the corresponding parameters. Third, verification must be run on the virtual system itself to ensure that no race conditions exist in the modeling code. Finally, since SCADA network information is closely guarded, an expert must be brought in to verify that the virtual network reflects a practical network. The following process should serve as a guideline for incorporating new virtual hosts into our framework.

14.4.2 Interoperability

The first goal in creating a virtual host is allowing it to communicate with an external device. Reaching this goal requires several steps. First, we create a new virtual host that receives a transmission and responds with a hard-coded message. This hard-coded message can be devised by examining traffic between physical devices. For instance, when devising responses for virtual data aggregators, we look at requests from the actual control station to the actual data aggregator and copy the response. With this model, it allows us to test the communication channel. The communication channel, which is made up of the virtual networking stack, a virtual proxy, a physical proxy, a physical routing network, and a physical end-host, can have some robustness issues that must be resolved before experimenting with a more dynamic response. Once the physical end-host receives and successfully decodes the hard-coded message, we consider this stage a success.

The next step is to generate responses dynamically. To do this, there are two steps. The first step is to create a virtual analogue of the physical protocol. In this case, the protocol is DNP3.0 encapsulated by TCP/IP. Within RINSE, this is represented as a ProtocolMessage with a corresponding ProtocolSession. The ProtocolMessage class contains member data to correspond to each of the fields specified in Section 14.2.2. When communicating with a physical host, the class structure must be converted to a byte stream and there are functions that perform this for both directions. Additionally, there is a helper function to compute and interpose the CRCs when converting to a byte stream.

After the protocol is implemented, the next step is to produce the dynamically generated message with static data. To do this, we extract the DNP3.0 request, convert it to a DNP3.0 ProtocolMessage, and process it. The control flow has been discussed in Section 14.3.2.

Once we ensure that the virtual data aggregator is able to correctly generate a response with static data, it is time to generate a response with dynamic data. For the control station to observe a changing environment, responses from the data aggregators must reflect a changing environment. Therefore, each of the relays must also derive its state from a changing environment. Each could poll PowerWorld independently, but this would not scale well. Instead, we chose to implement a single point of contact—namely, the state server. In this manner, we eliminate costly overhead by compressing multiple requests and responses into one.

14.4.3 Tuning Parameters

Once we have a generalized virtual model, it can be tuned to represent any number of physical analogues. By tuning parameters such as polling interval, polling pattern, and response time, we can model various types of data aggregator computers. New features could also be added by specifying them in the DML without designing brand new virtual data aggregators. Virtual relays do not have any polling interval (the data is already available upon receiving a request); however, they can be matched to their physical analogues by changing their response time. Were a virtual Modbus protocol created, another parameter in DML could be specified for protocol support. If a new physical device cannot be modeled by changing some of these parameters, then either altering the model or creating a new device class would provide the necessary functionality.

14.4.4 Expert Analysis

Because of the critical nature of the power grid operations, it is difficult to acquire network topologies. It is also difficult to acquire the designs for SCADA devices from their vendors. In addition to creating models based on observation of the lab devices, some power experts in TCIP have offered their expertise. According to them, aspects such as polling patterns and the parameterization of virtual models appear to be sufficient. Network topologies can be managed through DML files and do not require RINSE development access. The topology described in Figure 14.4 is simplified, but accurate enough to start security analysis. The two suggested topics to be tackled next are expanded functionality of the meters and extra virtual models for other devices. However, this ought to be done on a case-by-case basis as model development consumes time and resources. For current models, there exist enough parameters to allow an experimenter to model a device without delving into the RINSE source code, but future models may require extensive familiarity with the simulation engine.

14.5.1 Create the Basic Network

The first step in the process is to design a power network that exhibits traits that may be of interest. Someone who is knowledgeable of a typical power grid network will design this, as their knowledge should translate into a realistic representation. One such power network can be seen in Figure 14.5, which shows a power transmission network system for a large city, as displayed by PowerWorld. The long lines are transmissions lines that transmit power between buses, which are the short lines with dots in the figure. Connected to the buses are generators and loads, represented by the circled arrows and ground symbols, respectively. PowerWorld itself offers many tools for designing, interacting, and assessing power flow models. For the purpose of VPST, the important features that it offers are the abilities to manually create and export a design as well as support a real-time simulation.

From the PowerWorld simulation, key information will be extracted. Typically, these are the IDs of lines, buses, generators, shunts, and loads. From here, we generate the DML and EMS configuration as well as set up the proxies. The corresponding RINSE model is shown in Figure 14.6. From the PowerWorld view, 36 buses and 196 distinct connections were found. These distinct connections have been categorized into lines, loads, generators, and shunts and are then represented by relays in the DML. As can be seen in the zoomed-in box in Figure 14.6, the relays are then annotated to provide a reference point in the PowerWorld model. Likewise, the corresponding EMS configuration in Figure 14.7 shows a list of RTUs (i.e., data aggregators) that the EMS is aware of and is able to poll for data. Currently, configuring the EMS is done by hand, but efforts are under way to automate this process as well.

This portion of the workflow is set up to be as automatic as possible. There are scripts to extract the information from PowerWorld, to create the DML model and its corresponding routing information, to log in the proxy machines and set up the redirects, and to produce the control station configuration file. Future efforts are ongoing to reproduce the entire lab setup in another location, provided similar resources exist there as well.

14.5.2 Integrate New Technology

To integrate a new technology into the framework, there exist two options. The first option is to provide a stand-alone device or model that can be accessed through a network device. By doing this, the actual technology is tested, as opposed to a recreation. The way in which this is done is similar to how the State Server receives its values from PowerWorld (see Section 14.3.3).

The other option is to incorporate a model of the technology directly into RINSE. When using this option, the two parties will develop an accurate model of the new technology, whether that is a protocol or a bump-in-the-wire device or anything else. RINSE is written in C++ and as such can accept a C++ implementation of the new technology. The original implementation must be ported to C++, allowing some modifications to provide networking support inside RINSE. Further efforts are planned to make RINSE as modular as possible, to ease extensibility for cases such as this.

There are trade-offs to consider here, which must be evaluated on a case-by-case basis. For instance, in the C++ implementation case, there are issues of fidelity and time of development. It takes time to develop a model and ensure that it corresponds to the actual technology. There are also issues of ensuring that the C++ model is in lockstep with the current iteration of the product. In the emulation case, there may be issues of scalability, latency, and availability of measurement data. If time allows, implementing the technology directly inside of RINSE will usually allow for better results, as the full benefit of scalable simulation can be realized.

14.5.3 Evaluate Experiment

Once all of the connections are set up and the entire communication flow is shown to be working, it is time to test the technology in silico. During the course of the experiment, it is possible to interact with the model and test how it reacts to certain commands. For instance, the EMS software can provide a place for an operator to monitor the grid. Depending on the new technology, it may become easier or more difficult for the operator to understand what is happening in the system. For instance, new technology implemented in RINSE may simplify the job of the operator so that fewer errors occur. On the contrary, it may accidentally drop communication under certain conditions. PowerWorld can offer information about the performance of a new technology. For instance, if a command sent from the EMS never produces a change in PowerWorld, this can be seen quickly in PowerWorld and traced to its root cause in RINSE by looking at metrics such as bandwidth usage, dropped packets, and average latency, or by using tcpdump for postmortem evaluation. From there, RINSE supports quick turnaround time through network creation at runtime, easily configurable network topologies, and immediate feedback.

14.6.1 Conclusion

In this chapter, we have discussed the development of virtual models for DNP3.0, data aggregators, and relays within the RINSE framework. These models are important as they provide a realistic environment in which experimentation on emerging power technologies can take place. The smart grid and its related technology are being integrated into the existing grid at a rapid pace and with little validation. We must understand how this new technology affects existing technologies and infrastructure. By utilizing the models discussed above, we provide such an environment to test these technologies. To provide a realistic simulation, the virtual relays acquire data from PowerWorld, a power flow simulator, through a virtual host termed the State Server. This State Server captures snapshots of the electrical simulation and provides them to the virtual relays through shared memory. By enabling real-time information to be passed through our SCADA simulation, RINSE provides a platform for testing new technologies in a scalable, high-fidelity manner. A sample workflow that shows the steps that are required to verify the functionality of a new technology was presented.

14.6.2 Future Work

One area that must be modeled is report-by-exception polling. This can come in two flavors. The first is a polled report-by-exception, which is a poll that only asks for data that has changed. Another form is unsolicited report-by-exception. This allows a relay to send a message if something has changed state, without being asked to do so. This type of reporting is also currently unimplemented, but would require greater effort because of the nature in which the State Server is implemented. Without having access to the actual grid, but through the advice of experts, it seems as though relays support report-by-exception, but often do not use it as resources may be better spent on real-time monitoring. In any case, this seems like a natural feature to implement.

The DNP Users Group has published test procedures for Levels 1 and 2, the simplest implementations. Future work should ensure that the virtual models developed in RINSE pass these tests, or at least that subportions of these tests are deemed relevant to the demands of the experiment. Models may be introduced that conform to varying levels of these tests, so as to provide certain guarantees regarding both fidelity and performance.

One area of concern is that of clock skew among the relays. As implemented, the State Server receives values in chunks. All data within a given chunk has occurred during the same time slice. However, in the physical grid, data from separate relays are never guaranteed to occur during the same time slice. This is the type of problem that synchrophasor technology [37] is attempting to solve. However, the question of how, or even whether, to simulate this effect will require some examination. Again, this raises the question regarding fidelity versus performance. To simulate this clock skew, the state server must query each data point separately and provide randomization in the timing as well. This overhead may impact the performance of the simulation as well as the freshness of the data.