Chapter 11. Utilities

Utilities are a mainstay of the modern world. Be it electric power, gas, or water, utility companies provide the services that run our cities, businesses, and, indeed, entire economy. One power company executive was recently asked to explain his company’s core business function in one sentence. He replied: “We supply electric power for the enjoyment of our customers.” It may seem strange to think of “enjoying” electric power; it’s one of those things we tend to take for granted—but there is truth in the comment: Reliable electrical power allows us to enjoy many things in life. When the power goes off, we are suddenly without access to our treasured electronic devices and find ourselves desperately looking for candles and matches. It feels like we have been abruptly thrown back into the Dark Ages, trying to figure out how people used to entertain themselves before electric power. Some people joke that the function of the power company is simply to “keep the lights on.” For most of us, that is exactly what we expect because when the lights go out, our normal life screeches to an abrupt halt. We experience similar challenges when our gas and water services stop.

The importance of utilities to the basic function of society is evident as many governments categorize them as “critical infrastructure,” and most countries have strict laws to ensure their safety, protection, continued operation, and quality.

The size and scale of utility networks can be truly massive. Consider that each home connected to a utility company’s services requires a meter for each service. As each meter in the utility network becomes automated through IP and IoT, the scale of that network becomes huge. Imagine a midsized city with half a million homes and businesses that receive electric, water, and gas service from different utilities. If you assume one IP address for each type of meter, the network has already scaled to 1.5 million endpoints, not to mention the network infrastructure supporting these nodes. However, this example is actually very small compared to the size of some modern smart meter networks, which now scale into the tens of millions. From a pure IP perspective, very few organizations have dealt with the challenges involved in designing, deploying, and managing a network of this size.

The main focus of this chapter is on the digitization journey of electric power companies and, in particular, how IoT is being used to build smart grid networks. This is not to imply that gas or water utilities are less important, but rather reflects the fact that electric utilities have been the first to embrace the potential of grid automation and analytics through IoT. In addition, many of the technologies commonly found in electric power grids lend themselves very well to the automation benefits of IoT. That being said, many of the principles discussed in this chapter are also applicable to other types of utilities.

This chapter introduces the concept of the smart grid and explores some of the underlying IoT technologies that are transforming the way power is generated, transmitted, and delivered. It includes the following sections:

An Introduction to the Power Utility Industry: This section describes the power utility industry and provides a better understanding of its business models and technical challenges.

The GridBlocks Reference Model: GridBlocks is a foundational architecture for the smart grid. This section discusses this reference model and how it can be used to build a coherent smart grid strategy.

The Primary Substation GridBlock and Substation Automation: The substation is the place where power is transmitted and distributed. Assets in the substation are becoming highly connected and automated. This section explores automation solutions and the IoT building blocks that are now being deployed in substations.

System Control GridBlock: The Substation WAN: The utility WAN allows interconnection between substations and to the control center. The utility WAN is now transitioning from tradition TDM transport to IP packet-based networks. This section examines design considerations that allow the utility WAN to carry some of the most sensitive applications of any industry.

The Field Area Network (FAN) GridBlock: This section explores the FAN as a foundational element to connect intelligent devices, including smart meters and devices on the distribution grid, allowing utilities to harness the power of automation and data analytics.

Securing the Smart Grid: The electric grid is considered “critical infrastructure.” This section explores concepts such as SCADA security, NERC CIP, and security best practices for the distribution grid.

The Future of the Smart Grid: IoT has already had a profound impact on power utilities worldwide, resulting in new processes and business models. In the future, distributed energy generation, clean energy, and electric vehicles will further disrupt grid technology.

Generation: Power generation is where the electricity gets produced. Power production typically includes nuclear, hydroelectric, gas, and coal pants. Once generated, high-voltage (HV) electrical power is sent through high-voltage transmission lines into the transmission system. The generation company is also responsible for responding to the fluctuating power demands of the end customers.

Transmission: Power transmission takes the HV power over long distances— typically 115 kV and above over distances of 50 km and greater. Transmission lines include aerial lines and also submarine cables that transmit HV electrical power over long distances underwater. The transmission system is responsible for connecting HV lines from generation stations to substations throughout the service area. When you see large metal towers along the highway supporting long power cables, these are the transmission lines bringing power from the generation plant to the substations.

Distribution: Power distribution includes the part of the utility network from the substation to your home or business. This includes the medium-voltage (12.5 kV, for example) powerlines you see on poles around your neighborhood, including pad mount transformers. Note that power is stepped down to low-voltage at the transformers near your home and typically runs at a couple hundred volts toward the end customer. It is important to note that there are some differences between the North American and European distribution grid models. For example, in Europe, it is common to see secondary substations. Also, low-voltage is 240 V in Europe and 110 V in North America.

Figure 11-1 illustrates how generation, transmission, and distribution work together to bring power to end customers.

As inexpensive and reliable serial communications became widely available, more capabilities came, at lower prices, allowing for wide deployment. When Ethernet first became available, no one dreamed that it would become cheaper than serial or that computing would become so inexpensive and powerful that it could be used almost anywhere throughout the grid instead of being a centrally located behemoth-sized mainframe in a glass-walled control room.

Given the longevity of electrical control and monitoring systems, as well as the vast scale of utility networks, it is simply not economical to replace all legacy utility systems when new technologies become available. However, as new systems with new capabilities are installed, and as equipment and system prices decline, they are bringing greater reliability and cost benefits to utilities.

While OT networks are not as flexible as their IT counterparts, OT engineering departments have continually adapted to take advantage of newer technologies supporting the power grid. This has included developing ways to support many generations of legacy systems on new networks. OT engineers are always looking for better, more cost-effective ways to do things, and this often includes utilizing IT technology whenever possible. IT technology has the benefit of wide adoption in the industry, which means it is easy to find qualified people to design and support networks and application servers. The challenge was, and continues to be, understanding the OT physical systems and making sure that general-purpose IT, which is primarily based on IP networking technology, is up to the job. (IT and OT are introduced and defined in Chapter 1, “What Is IoT?”)

As the utility OT networks begin to migrate to IP communications and use IoT architectures, the sizes of the OT networks become orders of magnitude larger than those of IT counterparts. Take, for example, advanced metering infrastructure (AMI), where the electric meters become “smart” IP-enabled devices that are connected to a single network. The AMI network alone may have millions of nodes, all of which may become IPv6 routable endpoints. A question arises: Who designs, operates, and manages the AMI network, along with other IP-based OT networks? Is it the OT team, which has the skills to understand utility applications, or is it the IT team, which has experience in IP networking?

These challenges become even more evident as the IT and OT networks become interconnected. Concerns that need to be addressed include the following:

How can network resiliency and redundancy be supported for mission-critical OT applications that keep the lights on?

Who will support remote access to distributed systems on the grid that must transit both the IT and OT networks?

How will security be governed in both the OT networks and the interconnection points between the IT and OT networks? Is this the responsibility of the traditional IT security department, or is a new paradigm required?

Will change management be governed in the same way as it is for IT systems, or does the criticality of the OT applications require a new set of rules to ensure the continuity of business?

Different organizations respond in different ways to these challenges. In the past, IT and OT were completely separate groups—ships in the night that rarely needed to interface with each other. Today, as networks converge, OT and IT need to work closely together. Some OT engineers are learning the IP skills needed to build and support complex OT systems, and IT engineers are learning important aspects of the utility’s core OT system. However, the expertise and knowledge that each party has acquired over many years has been hard won; this knowledge is not easy to transfer between departments in a short period of time. How long would it take an IT engineer to learn the intricacies of electrical protection and control systems? Likewise, how long would it take for protection and control engineers to lean the intricacies of Ethernet resiliency and IP routing, which are today forming the network transport of the applications they are responsible for? These skills take many years to develop and mature.

These challenges have ushered in the age of the smart grid—the combination of the electric power grid and the information and communications technology (ICT) that operates the grid, with objectives of efficiently delivering sustainable, economic, and secure electricity supplies.

Utility companies are now grappling with the IT/OT convergence challenge, and this is perhaps the first major industry to be confronted with the rigors of integrating IT and OT at such a large scale into a converged network with cohesive governance. In addition, the utility industry is now faced with the challenge of developing new industry standards that allow the secure interconnection of millions of substation and distribution OT devices into the enterprise IT network. To accomplish this successfully, an architectural approach must be followed. The GridBlocks reference model provides such an architecture for utilities and is discussed next.

In response to this need, Cisco developed the GridBlocks reference model. While other reference models exist, GridBlocks offers an easy-to-understand model for both novice and advanced users working in the utility space. The GridBlocks reference model, shown in Figure 11-2, depicts the entire bulk electricity supply chain, from wide area bulk power entities through generation, control centers, transmission grids, substations, distribution grids, and integration of distributed energy resources at the edge of the grid. The model is forward-looking and is intended to be a generalized end-state reference framework that can help assist in deploying and designing end-to-end secure energy communications solutions for all aspects of the grid, thus facilitating a new and powerful foundation for utilities—the smart grid.

The GridBlocks reference architecture provides the following benefits to utility operators:

Details a flexible, tier-based model that supports incremental improvements to logical sections (tiers) of the grid

Helps enable secure integration of both new and legacy technologies, improving overall manageability and visibility of network elements

Builds on open standards, primarily IP, preventing vendor dependency and also supporting interoperability and thus promoting lower costs

Enables the consolidation and convergence of utility networks, which has the effect of streamlining operations and reducing operational and capital costs while creating new value through increased functionality

Provides a digitization roadmap for utilities, allowing them to modernize different parts of the grid in stages

While the GridBlocks tier-based model allows segmentation of the utility’s capabilities and functional areas into tiers, it also supports consolidation of network elements into a single converged architecture. The tiers, starting from the bottom tier shown in Figure 11-2, are as follows:

Prosumer tier: The prosumer tier combines the dual roles of energy producer and consumer and encompasses external elements that might impact the grid. These are devices that are neither owned by the utility nor part of its infrastructure, but that interface with it somehow. This includes distributed energy resources (DERs) that produce local power from solar or some other means. This could also include energy storage systems and responsive loads in electric vehicles or industrial facilities. This rapidly maturing part of smart grid technology promises to be a major disruptive element in the future, as discussed later in this chapter.

Distribution tiers: The distribution network is the last mile of the power delivery system. This part of the grid lies between the distribution substation and the end user. For simplicity, it is broken into two subtiers, as follows:

Distribution Level 2 tier: This lower-level distribution tier is the last mile, or neighborhood area network (NAN), of the power delivery system. This part of the smart grid network supports metering systems, demand response systems, electric vehicle (EV) recharging stations, remote terminal units that are part of the distribution automation system, and many other types of devices.

Distribution Level 1 tier: Level 1 of the distribution tier connects the Level 2 tier networks to the distribution substation and provides backhaul services to the utility control center via the system control tier.

Substation tier: This tier includes all substation networks, including those in both the transmission and distribution substations. Transmission substations connect multiple transmission lines and typically involve higher voltages (115 kV and above), and feed power toward distribution stations. Distribution substations receive an input of typically 115 kV and above (or whatever is common in the service area) and feed power at 25 kV or less toward the end customer. Networks at this tier have a wide variety of requirements, from basic secondary substations to complex primary substations that provide critical power delivery functions, such as teleprotection (discussed in detail later in this chapter). Inside the substation, there are often strict network requirements, including resiliency, performance, time synchronization, and security. These substation requirements have resulted in the separation of functions, with independent buses for each (for example, the station and process bus functions). Primary distribution substations may also include distribution aggregation.

System control tier: This tier includes the wide area networks (WANs) that connect substations with each other and with control centers. The WAN connections in this tier require some of the most stringent latency and resiliency performance metrics of any industry. The substation WANs require flexibility and scalability and may involve different media types, including fiber or microwave. The system control tier supports connectivity for remote SCADA (supervisory control and data acquisition, covered later in this chapter) devices to the control center, event messaging, and teleprotection services between the relays within the substations.

Intra-control center/intra-data center tier: This is the tier inside the utility data centers and control centers. Both data centers and control centers are at the same logical level, but they have very different requirements. A data center is very familiar to the IT engineers, as it contains enterprise-level applications and services. A control center contains real-time systems that operate and control the grid itself, including power distribution and transmission systems, monitoring, and demand response. This tier needs to be connected to the substation through the system control tier so that important data can be collected and run by both IT and OT systems in the substations.

Utility tier: This tier is home to the enterprise campus networks. (Although the name implies that there is some grid-related function here, this is an IT-focused tier.) The utility tier is the connection point between the control center and the enterprise network, and it utilizes firewalls with the appropriate security policies to ensure that only trusted traffic from the enterprise network enters into the control center. (Note that firewalls are used throughout this architecture and between tiers, and this is but one example.) It is also important to note that most utilities operate multiple control centers and have highly dispersed enterprise networks, meaning that these networks must be securely connected through either metro networks or WANs (possibly reusing a WAN network as the system control tier).

Balancing tier: This tier supports connections between third-party power-generation operators and balancing authorities (as well as connections to independent power producers [IPPs]). In an electric utility, demand from customers may not always meet the generation supply. To manage load and demand, most utilities are interconnected with other utilities and can buy and sell electrical energy from each other when necessary. At times, there may be an excess of power in one utility and a shortage of electricity in another. The balancing authority has the delicate responsibility of managing electrical demand versus supply on the grid. If electrical demand and supply fall out of balance, blackouts can occur. The sensitive nature of the balancing tier highlights the need for a communications network that enables different parties to collaborate effectively and securely.

Interchange tier: The network at this tier allows electricity to be bought and sold between utility operators. In the utility world, electricity is transacted in much the same way as other commodities, such as oil and gas. The sale or purchase of electricity needs to happen in real time. Networks at this tier allow the utility to not only buy electrical energy when needed but also make a profit by selling excess power to other utilities when there is an opportunity to do so.

Trans-regional/trans-national tier: Most utility grids are interconnected with much larger supergrids. For example, Figure 11-3 shows how the utilities in different countries and regions are interconnected with one another to form what is known as the Synchronous Grid of Continental Europe. In North America, this is known as the North American Interconnection, and it is composed of interconnection points between the Texas Grid, Western Interconnection, Eastern Interconnection, and Quebec Interconnection, as shown in Figure 11-4. At this tier are the network connections between synchronous grids for power interchange as well grid monitoring and power flow management.

Wide area measurement and control system (WAMCS) tier: This tier includes connections to a critical component of the power grid, power management units (PMUs), which are responsible for wide area power measurements across the grid. Due to the scope of this tier, it needs to connect to several of the other tiers and is thus depicted as a vertical tier in Figure 11-2.

The GridBlocks reference model is a useful tool and blueprint that can be used as a foundation to build network elements within the tiers and link them to other tiers. It also provides a fundamental grouping of network capabilities into “grid blocks” that can be expanded in much greater detail. The following are some examples that are discussed in subsequent sections of this chapter:

Primary substation GridBlock: This GridBlock delves into the subject of substation automation and the interconnection of process bus and station bus devices within the substation.

System control GridBlock: This GridBlock connects substations to one another and with the control center. One of the key focus areas of this GridBlock is supporting WAN architectures that can deliver teleprotection services.

Field area network (FAN) GridBlock: The FAN is a rapidly developing area of the utility IoT network that supports the connection and management of distant distribution elements, smart meters, distribution automation, demand response, and more.

If you were to give a modern smart phone to Alexander Graham Bell, he would likely look at it in amazement and wonder. The capabilities of the modern smart phone bear little resemblance to the simple telephone he invented in 1876 and would likely be considered something from another one of his contemporaries, the science fiction writer H.G. Wells. However, if you were to take Thomas Edison into a modern power generation or electrical substation, he would likely be able to tell you the exact function of nearly everything he could see. Indeed, the progress of technology in the electrical power industry has moved at a much slower rate than in the telecommunications industry. However, this is beginning to change—and we have the technical beginnings of Alexander Graham Bell to thank for it.

One of the greatest progressive leaps in the past few decades in the electrical power industry has been the ability to connect devices and control them through telecommunications networks, and IoT is now taking this leap to a whole new level.

SCADA had its beginnings back in the 1950s, long before computer networks existed. It was intended to be a system in which an operator could manage remote industrial devices from a central point (often a mainframe computer system). In these early days, SCADA systems were independent, with no connectivity to other systems, and they relied almost entirely on proprietary protocols. Over time, remote WAN networks allowed SCADA connectivity to extend to RTUs, but these connections were typically point-to-point serial links that utilized RS-232 or RS-485 interfaces and were transported over TDM circuits.

Over time, SCADA transport began to adopt standards-based protocols and an open network architecture. Instead of relying on dedicated serial links connecting every SCADA slave, the substation LAN began to be leveraged for transport, with a local SCADA master residing at each substation. As high-speed, resilient, and flexible IP WAN networks became available, SCADA services began to be dispersed throughout the network and could use a centralized SCADA master in the control center.

The most widely deployed legacy SCADA communication protocols are Modbus, IEC 60870-5, and Distributed Network Protocol (DNP3).

Figure 11-5 illustrates a legacy substation where the electrical relays are attached via serial (RS-232 or RS-485) connections to RTUs, which are in turn connected to a SCADA gateway device that is connected to the substation Ethernet network. A SCADA gateway device typically functions in one of two ways. The first way is protocol translation, such as translation of native serial to IP encapsulation. Examples of this include DNP3 to DNP3/IP or IEC 60870-5-101 (serial) to 60870-5-104 (TCP/IP). The second way a gateway device may work is to tunnel the serial traffic through the IP network (for example, with raw sockets). For a more in-depth discussion of Modbus, IEC 60870-5, and DNP3, along with the transport concepts of protocol translation and raw sockets, refer to Chapter 6, “Application Protocols for IoT.”

While we expect these legacy SCADA transport mechanisms to exist for many years to come, long term, traditional SCADA systems are being replaced by a new technology standard that natively takes advantage of Ethernet and TCP/IP: IEC 61850.

IEC 61850 overcomes some of the most challenging vendor and network interoperability challenges in the substation and beyond. With 61850, dedicated serial links are replaced with Ethernet and IP, which means the copper wiring plant in the substation can be greatly reduced. The inherent flexibility of Ethernet means that IEDs can easily communicate directly with one another and with other elements of the communications infrastructure. Another key advantage offered by the flexibility of Ethernet is that interfaces are cheap and are being added by equipment vendors to all modern assets, which means unsupervised gear in the substation is now becoming a thing of the past.

The bay level, shown in Figure 11-6, relates to high-voltage devices that make connections to power and current transformers, switching gear, and so on. These devices make connections into the measurement system for protection and control. Devices in the bay level typically have two different types of network interfaces: one for SCADA management connected to the station bus and another connected to the process bus.

While a primary focus of the station bus is on SCADA transport over Ethernet and IP, IEC 61850 goes far beyond. The IEC 61850 communications structure defines three main traffic classes:

Manufacturing Message Specification (MMS; IEC 61850-8-1): MMS supports client/server communications over IP and is used for SCADA. MMS traffic is typically found on the station bus.

Generic Object Oriented Substation Event (GOOSE; IEC 61850-8-1): GOOSE uses Ethernet-based multicast (one-to-many) communications in which IEDs can communicate with each other and between bays. GOOSE is often used for passing power measurements and between protection relays, as well as for tripping and interlocking circuits. GOOSE is typically used over the station bus.

Sampled Values (SVs; IEC 61850-9-2): SVs are typically used on the process bus to carry voltage and current samples. A common use for SVs is for bus-bar protection and synchrophasors.

In the world of substation automation, GOOSE is an extremely important tool, as it is the primary 61850 message type used between electrical protection and control systems. Protection and control systems are among the most important gear found in a substation, as they are used to continually monitor power being delivered by transmission lines and feeders. If power is disrupted for some reason, the measurement system detects it within a few milliseconds and passes GOOSE messages through the Ethernet network to a peer relay that switches power delivery to an alternate line or feeder. If the GOOSE messages are not delivered correctly or within the required timeframe, the electrical relays can become confused, and power can be incorrectly switched, causing blackouts or even worse.

IEC 61850-9 defines process bus communications in which critical process-level equipment may communicate messages over Ethernet. Any upstream metering, protection, or measurement devices may then use this data as necessary.

Figure 11-7 illustrates a possible IEC 61850 substation automation design. As shown in this illustration, two separate Ethernet segments are used: the station bus and process bus. The station bus allows inter-IED communication for things like GOOSE messages for protection and control as well as SCADA communications. According to 61850, the process bus uses an entirely different set of Ethernet switches for the critical substation automation functions. This area of the substation cannot simply use a separate VLAN from the same switches on the station bus; it must use distinct physical switches for each bus. One reason is that the network resiliency requirements of the process bus go far beyond what standard Ethernet is capable of and require a new generation of resiliency protocols, described later in this chapter.

IEC 61850 is still a relatively new standard for communications within the substation and beyond. Thus, an immediate migration from legacy systems and protocols is not likely. Utility assets often have 20- to 30-year replacement or upgrade cycles, and migration to newer equipment takes time. In many cases, you can expect a substation to have a mixture of legacy serial-connected RTUs alongside modern IEDs that can take advantage of the Ethernet framework offered by IEC 61850. In time, it is expected that the substation process bus will also begin to adopt the 61850 capabilities. Recent developments in standards also allow 61850 to be routed outside the substation, as defined in the 61850-90-5.

Figure 11-7 shows a hybrid substation where both legacy RTUs are used together with more modern 61850-capable devices. Over time, as the availability of serial and TDM parts becomes difficult, it is expected that IEC 61850 solutions will dominate substation OT networks in all parts of the substation.

The Ethernet frames originating from the IED are bridged to both network interfaces and are given a sequence number. The two frames then traverse the two parallel network paths until they arrive at the receiving IED, again on two separate NICs. The receiving IED selects a preferred (active) interface and discards the frame received on the nonpreferred (backup) interface. In the event of a failure in one of the parallel networks, this approach guarantees that at least one of the packets will always arrive at the destination IED.

The scenario just presented assumes that the IEDs themselves are PRP capable and are thus able to make and remove multiple copies of each frame. This may not always be feasible because it would require not only an upgrade of the IEDs themselves to support PRP but also the deployment of dual redundant Ethernet networks.

A similar but slightly different approach is to single-attach an existing IED to a PRP-capable access switch. In this case, the PRP access switch acts as the redundancy box (or RedBox), making dual copies of the Ethernet frame and sending the copies over different VLANs on opposing sides of the network. The receiving PRP switch then forwards a single copy of the Ethernet frame to the relay and removes the duplicate copy. Note that one of the key advantages of PRP is that the intermediary switches do not need to be PRP capable. In this scenario, only the sending and receiving RedBoxes actually participate in the PRP redundancy, as detailed in Figure 11-8.

Much as in the preceding PRP RedBox example, with HSR, the IED has only a single attachment to the HSR RedBox Ethernet switch. With HSR, instead of making duplicate copies of the Ethernet frame and sending them over different VLANs, the HSR RedBox sends out duplicate copies on the same VLAN but on opposing sides of the ring. One key constraint of HSR is that all intermediary switches in the ring must be capable of understanding HSR to remove the duplicate copy after the primary frame is switched on toward its destination.

Protection, according to IEC 60384, is defined as “the provision for detecting faults or other abnormal conditions in a power system, for enabling fault clearance, for terminating abnormal conditions, and for initiating signals or indications.” Teleprotection is the mechanism by which this information is transported over a network.

Teleprotection is used by almost every utility in the world between transmission substations and between primary distribution substations. Teleprotection is used by utilities to signal between protection relays and ensure that power is continually delivered, even when part of the electrical grid is out. In the context of IoT, the protection relays are the endpoints that digitize important data which is then transported by the IP transport network.

Distance protection uses the concept that the impedance of an electric circuit is proportional to its length (the distance of the line). Thus, for a known line distance, the relay simply needs to measure the impedance of the line at key points, and then a calculation can show where the break is. If the measured impedance is different from what is expected, the relay can signal to the switch to either enable or disable a feeder line. Because line protection uses simple impedance measurements, latency or jitter between the communication relays is not a major concern.

Figure 11-9 illustrates a simple distance protection scheme with multiple zones. The relays measure impedance in the different zones and use this to isolate the location of the fault. Zones may overlap and extend beyond the zone line length to provide 100% primary trip protection and also to provide backup trip protection for adjacent lines. For example, in Figure 11-9, Zones B1 and B2 overlap to provide redundant protection.

Of course, with alternating current systems, current measurements vary over time, so current differential protection requires that timing be synchronized between substations. If the timing is not synchronized, current measurements between relays may be different at a given point in time, falsely indicating either a loss of current or overcurrent, thus causing the relay to signal a change to the switch that results in a power outage.

Two mechanisms are commonly used to synchronize relays to ensure that current samples are aligned. The first option is to use GPS-based synchronization. The second option, called channel-based synchronization, is based on two-way time transfer and utilizes the communication channel to exchange timestamped messages between relays. The channel-based synchronization technique is typically proprietary to the relay manufacturer. Common methods of timing synchronization include SyncE and IEEE 1588 Precision Timing Protocol (PTP). Figure 11-10 illustrates a current differential protection scheme that measures current vectors.

The need for synchronization between relays also implies that the communications path between the relays has to be deterministic and predictable. Due to the timing sync requirement, current differential protection has very strict telecommunications requirements related to packet delay and jitter, which means that all such schemes require symmetric forward and return path communication between the relays.

In the days before IEC 61850, relays were connected back-to-back between substations using TDM circuits. The amount of data communicated between the relays is actually very small, and typically a DS0 (or 56/64 kbps link) was all that was needed.

Various standards for interfaces have been developed for teleprotection relays over the years. These include ITU-T G.703 for copper connections and IEEE C37.94 for optical. These legacy interface types are unique and customized to the teleprotection application. In addition, ITU-T X.21 and E&M interfaces are also used for some legacy teleprotection relays. Often referred to as “ear and mouth,” E&M is a supervisory line signaling method that you may be familiar with from its use with analog voice trunks. In recent years, companies have started to deploy modern IEC 61850-90-12-based protection systems that take advantage of Ethernet interfaces.

The time synchronization requirement of current differential protection imposes an enormous requirement on the network. IEC 61850-90-12 states that end-to-end latency between relays should be no more than 10 ms. This includes the interface processing latency within the relay, the processing at the router, and the speed of light time across the link. This form of teleprotection includes another challenge: managing path symmetry. Just as it is important to manage one-way latency, the difference in bidirectional latency is even more sensitive. Typical relays can tolerate forward and reverse differential communications latency of no more than 500 µs–1 ms. If a protection circuit were to have different forward and reverse paths due to optimal IP routing issues, the relays could misinterpret the communications sync issue and trip the breakers, thus causing a loss of power. Truly, managing the end-to-end teleprotection latency budget is one of the most challenging aspects of a protection and control engineer’s job.

While IP-based WANs are a mostly positive development in the utility world, they do have one downside when it comes to teleprotection: While they use IP routing mechanisms to inherently find the shortest path to a destination, they by default do not use a predictable path with a known latency. If an MPLS network is able to find a better path to a destination, it will take it, without regard for the latency sensitivities of the underlying application it is carrying. There is a delicate balance here: While end-to-end latency must be minimized, it must also be bidirectionally consistent.

In response to this need, in 2008 the IETF and ITU jointly began working on a variation of MPLS that would be able to take advantage of all the benefits of traditional label switching but at the same time incorporate key elements of carrier switching and operations, administration, and management (OAM) that would allow applications such as teleprotection to be transported over MPLS. The result was MPLS–Transport Profile (MPLS-TP), which brings capabilities for traffic engineering, automatic protection switching (APS), and OAM.

MPLS-TP transports a point-to-point pseudo-wire (a virtual circuit transported over MPLS) over a prescriptive label switch path (LSP). The hop-by-hop LSP is programmed by a protection and control engineer such that the exact forward and reverse LSPs are the same (see Figure 11-11). This has the benefit of making latency predictable and symmetrical, and it also keeps jitter to a minimum. The pseudo-wire endpoints terminate at the teleprotection relays.

MPLS-TP also supports APS by identifying a known backup LPS path in case of a primary LSP failure. In this case, the backup LSP is deployed such that it also has predictable latency and path symmetry in case of failure.

One of the key benefits of MPLS-TP is that it supports end-to-end OAM. OAM allows for fault detection of the pseudo-wire at any point and is used as the trigger mechanism to fail over to a backup LSP. MPLS-TP implements in-band OAM capabilities using a generic associated channel (G-ACh) based on RFC 5085 (Virtual Circuit Connectivity Verification [VCCV]). The in-band OAM channel is like a point-to-point management/control circuit that can detect link or node failures and can signal backup LSP failover on the order of 50 ms or less. Figure 11-12 illustrates the G-ACh within the MPLS-TP pseudo-wire.

MPLS-TP is able to meet the requirements of teleprotection, but what about other similar MPLS modalities, such as MPLS–Traffic Engineering (MPLS-TE)? MPLS-TE was developed many years ago to explicitly and dynamically define a label switch path (LSP) through an MPLS network. As such, it has many similarities with MPLS-TP. However, although MPLS-TE can be used to meet the predictable latency and path engineering requirements of teleprotection, there is one downside: MPLS-TE does not have OAM capabilities. With MPLS-TE, it is still possible to create deterministic and symmetrical paths, as well as provide support for APS, but the implementation with MPLS-TP tends to be much simpler and has more similarities to carrier Ethernet switching.

MPLS-TE does has one key advantage over MPLS-TP: Its ability for call admission control (CAC). With CAC, the edge router is able to determine whether enough bandwidth exists along the path to support the requested circuit. In most cases, this capability is not critical for teleprotection traffic because the bandwidth requirements are minimal, but in practice it is a useful capability.

A new MPLS variant called Flex-LSP combines the best of both of these. Flex-LSP supports all the benefits of MPLS-TP, such as APS and OAM for pseudo-wires, while also supporting CAC and Layer 3 traffic engineering, much like MPLS-TE. As technology continues to improve, other MPLS modalities, such as segment routing, may also be appropriate for teleprotection in the future.

The FAN is designed to enable pervasive monitoring and control of all utility elements between the distribution substation and the end customer. This section of the grid includes metering applications for both customers and the distribution network system itself, and it also includes management of the electrical distribution network devices that help enhance energy delivery and build a low-carbon society.

The FAN GridBlock is built to be multiservice, meaning that it is not based on any vendor-specific, proprietary technologies that would limit its use to a single purpose, like so many legacy OT systems. In the past, Internet standards simply did not exist to build metering or distribution automation (DA) networks based on open standards. It was necessary to build a dedicated and independent network for each application. However, modern open standards and network compliance alliances (such as the Wi-UN and HomePlug Alliances) have helped establish interoperability standards that allow a single multiservice network to be deployed, supporting a wide array of applications and vendors. In the same way that the Wi-Fi Alliance has helped establish interoperability among Wi-Fi access points and end clients, these alliances are also establishing interoperability standards. It will soon be possible to have a fully functioning FAN network with various components supplied by different vendors, all using the same standards.

The FAN GridBlock leverages many of the standards discussed in Chapters 4, 5, and 6, including IPv6, IEEE 802.15.4 mesh, CoAP, and LTE. This flexible and open standards approach promotes multivendor plug-and-play capabilities with a well-understood framework for security, quality of service, resilience, and network management services. The result is a wide array of capabilities that go far beyond trivial metering use cases.

Figure 11-13 demonstrates a multiservice grid FAN supporting applications such as EV recharging stations, connected street lights, demand response endpoints, smart meters, and connections to remote SCADA RTUs in the distribution network.

To summarize, the key advantages of the modern FAN that make it attractive for utilities include the following:

Open and standards based: Core components of the network, transport, and application layers have been standardized by organizations such as the IETF and the IEEE and are interoperable with other compliant devices.

Versatile endpoint support: IPv6-based IoT endpoints are flexible and can be used in a wide variety of locations, including AMI (meters), street lighting modules, demand response devices, and distribution automation endpoints, such as SCADA RTUs.

Flexible headend deployment options: Because the FAN uses IPv6 transport, the headend aggregation points and security system can either be deployed on-premises or hosted in the cloud.

Flexible backhaul options: The FAN typically requires a field area router (FAR) that is mounted on the utility pole or in some other convenient location. The FAR is the termination point of the mesh network. A wide variety of backhaul options are typically available, including LTE, 3G, WiMAX, fiber optics, and even satellite backhaul in very remote communities.

Support for legacy applications: Through the use of a gateway, legacy devices (such as serial RTUs) can be connected to the IPv6 FAN at scale.

Scalable: IPv6 is capable of scaling to tens of millions of endpoints, easily managing the meters and street lights in a large utility network.

Highly secure: The FAN GridBlock incorporates multiple layers of security, including application and network layer encryption as well as endpoint authentication.

Stable and resilient: Thanks to the flexibility of IPv6, a well-designed FAN is able to offer strong network availability and resiliency. For example, if a FAR has its primary backhaul through Wi-Fi, LTE can be used as secondary backup, and IP routing protocols can be used to figure out the optimal path. In addition, using IP routing, the FAR can form redundant connections to both primary and secondary headend sites.

The following sections examine the application of the FAN in two key areas: advanced metering infrastructure (AMI) and distribution automation (DA).

With the advent of smart meters, it is now possible to read meters several times per day. In the case of commercial and industrial (C&I) meters, readings can be done every few minutes to provide up-to-the-minute visibility into power consumption. This has been extremely valuable for customers as they are now able to get highly accurate, per-month billing reports. Customers can also view their power consumption on an hourly basis through a web portal. Some utilities have implemented time-of-day billing, in which the cost of power is higher during peak periods. Having near-instantaneous feedback helps families understand their consumption patterns and save money on their electric bill. This demonstrates the power of IoT.

Figure 11-14 illustrates a smart meter web portal launched by a Canadian utility that has fully deployed IPv6 smart meters in its FAN.

Smart meters have several other unique benefits. For example, it’s now possible to remotely shut off a meter at will through a remote disconnect switch. While this could be viewed as a security concern, the benefit to the utility is that customers who haven’t paid their bills or who are stealing power can be shut off or restored without even dispatching a crew. This reduction in truck rolls saves an incredible amount of money and many labor hours.

In addition, most smart meters also come with an internal home area network (HAN) radio that is able to communicate with electrical devices inside the home, often through ZigBee. (For more information on ZigBee, refer to Chapter 4.) This allows the home or business owner to track power consumption on a per-appliance or per-device basis.

Figure 11-15 illustrates the anatomy of a modern smart meter.

In an IEEE 802.15.4 network utilizing an RPL mesh, the meters are mesh nodes and are thus repeaters. For more information on 802.15.4 and RPL, refer to Chapters 4 and 5. Each meter runs the IPv6 protocol stack and endeavors to find its place in the mesh through RPL. In a large mesh, only a handful of meters link directly to the FAR. Most are deeper in the mesh and have links between them. In this case, unlike most Layer 2 networks, where you try to limit the size of the broadcast domain, a large mesh is actually a good thing as it strengthens the mesh connectivity. The larger and denser the mesh, the further you are able to push it out into the neighborhood.

Consider the example of a large apartment building with an underground vault containing all the meters for that building (called a meter farm). While there may be hundreds of meters in the farm, representing each apartment or unit, only one meter needs to have an upstream RPL link to a parent node leading to the FAR. The rest are children of that meter. Figure 11-16 illustrates such an underground meter farm in a concrete vault. These underground meters simply form a branch off the main mesh.

Beyond smart metering, there are countless further use cases for FANs, as indicated in Figure 11-13. Two interesting use cases are discussed in more detail in the following sections: demand response and distribution automation.

Over time, utility engineers have addressed this problem by controlling electrical usage on less critical systems during peak periods so that electricity can still be available to customers throughout the grid. This has been accomplished through a mechanism called demand response (DR), which involves deploying remotely controlled devices that turn off the flow of electricity to certain devices on the grid during peak power use periods. For example, during peak usage periods, the utility can send out a broadcast message to customers using certain types of electric devices (such as electric water heaters) to automatically reduce the power consumption on those devices, thus making more power available for the rest of the grid.

In the past, DR controllers were nothing more than simple wireless pagers that would receive signals from the DR management system and then automatically shut off devices when instructed. They would then receive other signals when power became available and turn the devices back on (using a binary on/off type of operation).

Although there are many ways to reach a DR controller that is attached to an electric appliance (including cellular and 1901.2 PLC), a FAN can also be used for this application. In this case, the utility uses the FAN’s IPv6 network to communicate to DR controllers in specific parts of the grid where demand is reaching peak usage, and can centrally control their power consumption as necessary. Figure 11-17 shows an electric water heater fitted with a FAN mesh DR controller.

Due to the challenge of connecting distribution control and automation devices to a central network, they have, by and large, been designed to work as autonomous devices, in many cases with enough intelligence to operate without any supervisory control. However, as wireless network technology and availability have improved, it has become possible to connect distribution devices that sense the operating conditions of the grid to a communication network, thus greatly improving visibility into conditions of the distribution grid. This has also helped significantly improve the reliability and quality of electrical power in the distribution grid and has ushered in the age of distribution automation (DA).

DA seeks to improve the conditions, reliability, and power quality of the grid and is thus able to reduce costs and improve customer uptime and satisfaction. DA devices perform many different functions, from measuring the quality of electrical power to clearing temporary faults in lines. An example of a temporary fault would be a tree branch falling on a line, causing a temporary short before the branch finally falls to the ground. Clearing this kind of fault is the function of a recloser.

Layering these devices on a communications network causes the level of automation to increase dramatically. Not only does the utility gain the ability to determine the conditions of distribution grid devices through SCADA, but it can begin to collate and analyze the data generated by the thousands of DA devices to gain a better picture of the conditions of the overall grid.

The following are some examples of how FAN-based DA is being used:

Distribution SCADA systems: Earlier in this chapter, you learned that SCADA within the substation is enabling automation of the electrical grid. Through the use of FANs, the same level of management is now possible on the distribution grid for devices outside the substation. While several technologies are suitable for connecting to these IoT devices (including LTE and Wi-Fi), the scale capabilities of the FAN, along with the low bandwidth requirements of SCADA, make the FAN an ideal network platform. When the SCADA endpoints are remote, the communications can be either aggregated at the substation and then sent back to the control center or sent directly to the control center, bypassing the substation altogether.

Fault location, isolation, and service restoration (FLISR): In the past, power outages were discovered when someone called the power company to say that his or her power was out. Not only did this waste valuable restoration time, it didn’t help much in finding out where the system fault actually occurred. FLISR systems are designed to identify, locate, and diagnose problems so the utility knows instantly when an outage has occurred, and in some cases they even allow the circuits to self-heal. Circuit breakers, smart meters, and switches for fault clearing are all part of a FLISR system. Since many of these “things” are remote and require only minimal bandwidth, FANs make an ideal choice for transporting FLISR communications.

Integrated volt/VAR control (IVVC): Volt/VAR systems are used in the distribution grid to monitor and control voltage levels during peak periods and help conserve electrical usage. In the past, due to communications challenges outside the substation as well as availability of voltage sensors in the grid, volt/VAR optimization (VVO) deployments were limited. In recent years, communication systems such as cellular and FAN mesh networks have made it possible to collect information from voltage sensors and use that information to adjust voltage-regulating equipment such as capacitor banks in real time.

Figure 11-18 illustrates these various use cases, connected to a single multiservice field area network. In this illustration, many different applications are using a single FAN. However, the application servers that control functions for SCADA, FLISR, and IVVC all reside in the distribution management system located in the data center or in the control center.

The 2015 Ukrainian power attack discussed in Chapter 2, “IoT Network Architecture and Design,” that cut power to 103 cities and towns (and affected 186 more) involved a sophisticated simultaneous attack on six power companies. This attack, which affected the power grid’s SCADA network, began as malware on company computers and spread to the OT system. Today, utility companies are left with this decades-old management protocol that was not designed with security in mind.

To say that the Ukrainian attack left a deep impression on security teams in utility companies around the world would not be an overstatement. Due to this attack, and other less-well-known ones, utility companies are rushing to secure their newly converged and legacy systems as fast as possible.

According to a Cisco Security Capabilities benchmark study, 73% of utility IT security professionals say they’ve suffered a security breach, compared with an average of 55% in other industries. Certainly, utilities are a high-value target for cybercriminals. In 2015 Lloyds of London modeled the economic impact of a large-scale coordinated cyberattack on northeastern US utilities. The impact was predicted to be $243 million to $1 trillion. While such a widespread attack may seem unlikely, recent cyber attacks, such as the one on the Ukrainian power grid, show that such attacks are technically feasible and should be of concern to both utility operators and their customers.²

Different utility-based security architectures have been proposed. One such effort is IEC 62351, which was developed by IEC TC57 to support the security needs of IEC 60870 and 61850, and which encompasses a fairly wide scope. The North American Electric Reliability Corporation’s (NERC’s) Critical Infrastructure Protection (CIP) is a security model that was developed to protect bulk systems, and it continues to be one of the most important security subjects for North American utilities.

NERC CIP uses a risk-assessment security approach. Instead of using an exhaustive list of prescriptive recommendations and enforcing them through audits, NERC provides a clear vision of the security end state. This is a powerful methodology as it removes attention from just passing the audit by checking all the right boxes without truly trying to actually protect the networks. Rather, NERC CIP v6 helps utilities focus on what is actually important: securing their networks against attack, from both the inside and the outside. For example, instead of mandating a certain type and level of antivirus, NERC CIP v6 is more principle driven, requiring “malware protection.”

NERC CIP is primarily focused on establishing security policies, programs, and procedures. A key concept in this model is the assessment of the impact level that a security breach may have on assets in the utility. Utilities need to properly identify what impact level each asset fits into, with levels defined as high, medium, low, or no impact at all. Assets in scope are defined as ones that “If rendered unavailable, degraded, or misused, would adversely impact the reliable operation of the Bulk Electric System (BES) within 15 minutes of the activation or exercise of the compromise.”

NERC CIP v6 also requires intrusion detection/prevention systems (IDS/IPS) or some form of deep packet inspection (DPI). The standard also mandates that an electronic security perimeter (ESP) be defined where assets within the EPS are protected by two distinct security measures, such as a firewall and an IPS. In addition, a physical security perimeter (PSP) is defined, which includes other aspects, such as video surveillance and building access systems, and aims to protect the station against physical attack.

A key aspect of NERC CIP is that an ESP must be established for all high- and medium-impact BES cyber systems connected to a routable network, regardless of whether the segment containing the BES cyber system has external connectivity to any other network. Figure 11-19 illustrates a primary substation network, highlighting the ESP and PSP components.

Compliance with a standard is no guarantee of security, but it certainly goes a long way in raising awareness and enforcing accountability for a utility’s security posture. NERC CIP v6 is a large and complex subject, and its details are beyond the scope of this book. NERC CIP touches on areas of malicious code prevention, configuration and change management, vulnerability assessments, and security event monitoring. For further details on the current state of NERC CIP, see www.nerc.com.

FAN security is aligned to the following principles:

Access control: FAN devices reside in generally insecure locations, so the devices themselves need to have highly secure access control. If a grid IoT endpoint were maliciously added to a FAN, it could be a backdoor to the network. To this end, FAN endpoints and routers are recommended to support X.509 certificates, with both a factory-level certificate and a utility-specific certificate once the device is enrolled in the network. The ITU-T X.509 standard defines a structure for handling secure certificates and keys, and you may recognize it because it is commonly used to secure web and email communications.

Data integrity and confidentiality: FAN devices need encryption. Last-mile FANs often use unlicensed wireless technologies that could be easily sniffed. Encryption at each layer of the stack is strongly recommended. In addition, configuration files in FAN devices, such as the FAR, should be encrypted to prevent a hacker from accessing information from a stolen device.

Threat detection and mitigation: One way threat detection and mitigation are accomplished is through the logical separation of the FAN headend components and systems from other critical systems in either the substation or the control center. Much as with the NERC CIP v6 requirements mentioned earlier, it is a good idea to follow a defense-in-depth model and use more than one layer of deep packet inspection, such as a firewall and an IPS that understand industrial protocols, like SCADA. (Note that if FAN endpoints encrypt at the application layer, this limits visibility for deep packet inspection.)

Device and platform physical integrity: The field area assets, such as the FAR, need to be physically secured as much as possible. The routers should be tamper proof and have door alarms. In addition, IEEE 802.1AR (Secure Unique Device Identifier) is becoming standard on remote routers to not only speed deployment but also ensure that the device on the grid network is trusted.

Securing the smart grid is a complex and ever-evolving task, especially in highly distributed and public environments such as electrical distribution networks. While NERC CIP is primarily focused on securing generation and substation assets, security for the utility ultimately needs a wider lens. This must be addressed at each tier of the GridBlocks architecture, with a special focus on utility-specific protocols, such as SCADA, that were not designed with security in mind but today are transported over highly interconnected networks.

Now, more than ever before, the industry is being challenged on multiple fronts. Some of these challenges are disruptive and threaten the future of the industry. These challenges include requirements to incorporate electric power generated by inherently variable renewable resources, such as wind and solar, as well as integrated distributed energy resources (DERs), such as solar photovoltaic (PV) cells that are installed and owned by the customer rather than the utility but sell power back to the utility grid.

An interesting example of this is in Hawaii, where the average electric bill is more than three times greater than anywhere else in North America. The high cost of electricity has driven many Hawaiians to deploy solar PV panels to take advantage of the abundant sunshine. In fact, by 2016, more than 12% of Hawaiians had solar panels on their homes. While solar is a great way to generate clean energy, the challenge of integrating power produced by these homes into an island power grid that is isolated from any other power grid is extremely difficult. In addition, as more people add solar power to their homes, the power company has fewer paying customers; solar-powered homes connected to the grid become distributed generation nodes, and the power company has to pay customers for the use of their electricity.³ With fewer paying customers, there is less revenue to maintain the system. In 2016 these economic realities resulted in Hawaii putting a halt to further solar DER deployments on the island.

Some analysts project that over time, the growth of customer-owned generation could undermine the economic basis of utilities to a degree that they would be disrupted—much as we have seen happen with the Internet and music, retailing, and other industries. Some regard the potential for disruption by DERs not owned by the utility as threatening. Others see it as the transformation the grid needs to herald in the age of higher reliability, lower costs, and lower carbon emissions.

The age of distributed generation and renewable energy builds a very strong case for the smart grid. You can’t introduce renewable energies, particularly at the medium-/low-voltage layers, if you don’t control and monitor them. For example, an interesting challenge that utilities are facing with the rise of DERs is how power will be balanced and controlled on a grid where power generation is highly dispersed. For example, DERs such as PV cells generate DC power. However, the electrical grid runs on AC, where both current frequency and voltage/current phase are key elements in the delivery of high-quality power. For a DER to provide energy back to the grid, DC power needs to be converted to AC through a power inverter. While this seems simple enough, there are challenges to consider, such as how power will be balanced throughout the grid with so much distributed power generation by third parties. This underscores the need to have a reliable network system that is able to communicate between elements in the utility’s grid and IoT devices at the DER, such as the inverter or the smart meter.

Another disruptive change we are seeing is the rise of EVs. As more and more electric cars are introduced, they will require more power from the grid, and there is also the potential to use these fully-charged car batteries as remote power storage units. Engineers are looking for ways to use these EV batteries as a DR solution that could support the grid during peak power periods. This completely changes the concept of demand response and how power can be selectively used. Again, car batteries are DC powered, so the power needs to be converted to AC, and such a system would require both inverters and system metering to track the flow of power, both in and out of the utility’s grid. All this highlights the criticality of a reliable IoT communications network in the smart grid.

Regardless of how disruption and transformation play out around the world, the electric power industry will undergo more change in the next 10 to 20 years than it has seen in the past century.

IoT technologies are driving digital transformation in all aspects of the electrical grid, from generation to transmission to distribution, and are bringing in the era of the smart grid. A vendor-neutral holistic reference model for networking OT elements of the electrical grid into a single architecture is GridBlocks. GridBlocks divides various functions of the electrical grid into 11 tiers, allowing utilities to digitize in a systematic and methodical way.

This chapter discusses several elements of the GridBlocks architecture, including the primary substation GridBlock, and includes a discussion of substation automation techniques. This discussion focuses on the use of SCADA and the drive toward standardization through the IEC 61850 standard. This standardization focuses on supporting the station and process buses within the substation and various Ethernet switching designs that can be used to meet the rigorous requirements of IEC 61850.

This chapter also examines the system control GridBlock, with a particular focus on teleprotection systems over an MPLS WAN. Teleprotection relays have some of the most sensitive application-layer latency and jitter requirements in the world. This chapter discusses different design recommendations to meet these requirements.

This chapter also covers the field area network GridBlock, including how multipurpose FANs are driving a multiservice distribution grid network. Use cases such as AMI, DA, and DR are examined.

Smart grid security is a top-of-mind subject for many in the utility industry, especially as grid devices are being connected through IP. This chapter introduces key concepts of NERC CIP v6, as well as strategies for securing elements outside NERC’s scope, such as the distribution FAN network.

This chapter provides a glimpse into the future of the utility industry. Disruptive technologies such as distributed energy generation, microgrids, and electrically powered cars are not only challenging existing power grids in new ways but are major forces for digital disruption that will create new opportunities for innovation in the twenty-first-century smart grid.

2. Intel and Cisco, Utility Security: Exceeding Mandates to Mitigate Risk, 2016, www.cisco.com/c/dam/en_us/solutions/industries/energy/docs/greentech-white-paper.pdf.

3. Robert Fares, “3 reasons Hawaii put the brakes on solar—And why the same won’t happen in your state,” Scientific American, December 15, 2015, https://blogs.scientificamerican.com/plugged-in/3-reasons-hawaii-put-the-brakes-on-solar-and-why-the-same-won-t-happen-in-your-state/.