Luis Ayala

Cyber-Physical Attack Recovery Procedures

A Step-by-Step Preparation and Response Guide

Luis Ayala

Fredericksburg, Virginia, USA

Any source code or other supplementary materials referenced by the author in this text is available to readers at www.apress.com . For detailed information about how to locate your book’s source code, go to www.apress.com/source-code/ .

ISBN 978-1-4842-2064-1

e-ISBN 978-1-4842-2065-8

DOI 10.1007/978-1-4842-2065-8

Library of Congress Control Number: 2016943087

© Luis Ayala 2016

Cyber-Physical Attack Recovery Procedures: A Step-by-Step Preparation and Response Guide

The information and opinions contained in this document are provided “as is” and without any warranties or guarantees. Prior to acting on recommendations herein, the user should consult with licensed engineers to determine if the actions are safe for the specific equipment in your building. Reference herein to any specific commercial products, process, or service by trade name, trademark, manufacturer, or otherwise does not constitute or imply its endorsement, recommendation, or favoring by the United States Government, and this guidance shall not be used for advertising or product endorsement purposes.

The statements of fact, opinion, or analysis expressed in this manuscript are those of the author and do not reflect the official policy or position of the Defense Intelligence Agency, the Department of Defense, or the U.S. Government. Review of the material does not imply DIA, DoD, or the U.S. Government endorsement of factual accuracy or opinion.

Managing Director: Welmoed Spahr

Acquisitions Editor: Susan McDermott

Developmental Editor: Douglas Pundick

Technical Reviewer: William T, Beck

Editorial Board: Steve Anglin, Pramila Balen, Louise Corrigan, James DeWolf, Jonathan Gennick, Robert Hutchinson, Celestin Suresh John, Nikhil Karkal, James Markham, Susan McDermott, Matthew Moodie, Douglas Pundick, Ben Renow-Clarke, Gwenan Spearing

Coordinating Editor: Rita Fernando

Copy Editor: Kim Burton-Weisman

Compositor: SPi Global

Indexer: SPi Global

Cover image designed by Freepik.com

For information on translations, please e-mail [email protected] , or visit www.apress.com .

Apress and friends of ED books may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Special Bulk Sales–eBook Licensing web page at www.apress.com/bulk-sales .

uracy or opinion.

Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.

While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.

Printed on acid-free paper

Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail [email protected], or visit www.springer.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.

I want to thank my wife, Paula, who has been with me through thick and thin for the last 35 years.

I also thank our son, Christopher.

Introduction

You need to accept as fact that eventually your facility will be the target of a cyber-physical attack. The only choice you have in this matter is how your building will be attacked by reducing the number of possible attack vectors. The best you can hope for is a situation where a cyber-physical attack is detected quickly, building equipment automatically shuts down gracefully (before any damage is done or anyone is injured), and building systems can be rapidly restored to normal operation.

This book does not focus on how hackers can get in to your BCS. I don’t explain how hackers can overcome firewalls or defeat sophisticated security software. I leave that to others to explain. I don’t spend a lot of time discussing how to tell if a cyber-physical attack is underway (when everything shuts down—especially unrelated building systems, you know something’s wrong). The objective of this book is to plan how best to respond to a cyber-physical attack so you can make decisions quickly and take proper action to mitigate the impact of the attack. When a cyber-physical attack occurs, the last thing you want to do is make things up as you go. This book advocates a dynamic attack surface —automated on-the-fly changes of a building control system’s (BCS) characteristics and defensive counter-cyber operations to thwart actions of an adversary.

You can spend millions of dollars beefing up the secure perimeter of a building against a physical attack, but that can all be defeated by a hacker causing equipment already inside the facility to destroy itself or even explode simply by executing a well-planned cyber-physical attack. Having an active and effective malware detection program in place to protect your facilities is great, but it would be meaningless without a well-designed plan that tells building maintenance personnel what to do when a cyber-physical attack is underway.

Cyber-physical attack

A classic cyber-physical attack would be when a hacker is able to damage building equipment by sending destructive commands over the BCS to the equipment that changes the configuration setpoints above dangerous levels for which the equipment has not been designed, such as too high pressure or dangerously high temperature. A second type of cyber-physical attack is an attack that does actual physical damage or a coordinated cyber and physical attacks on vulnerable physical systems. For example, in 2013, a sniper shot at and damaged 17 electrical transformers in California, causing them to leak coolant, overheat, and shut down (physical component). 1 The cyber component to this attack was the fact that the attacker also disabled the utility controls system ability to signal an alarm.

Of course, building engineers assume a cyber-physical attack can’t happen because of safety devices installed to prevent catastrophic events. It’s true that many building control systems have hard-wired safeties designed to shut down equipment and these hard-wired safeties typically are not controlled by the BCS. However, a hacker can use these safety devices as part of the attack—in fact, the hacker is counting on safety devices turning things off. Keep in mind that the Chernobyl nuclear plant also had safety devices that were turned off—by insiders. 2 Sad to say, 40 percent of cyber-attacks are carried out by insiders. In addition, it is not uncommon for insiders to turn off safety features that tend to set off frequent false alarms. The first thing a hacker does after he installs a backdoor is disable safety devices. A second factor to consider is that multiple pieces of equipment will be attacked simultaneously in a well-planned cyber-physical attack. A hacker won’t just disable your boilers—he’ll disable everything that he can.

Facility engineers rely on equipment alarms to warn when equipment and processes approach dangerous situations and to (hopefully) allow equipment to power down gracefully. Generally, that is the case—unless of course if a hacker changed the configuration setpoints so the building controls “thinks” that a boiler’s 322°F water temperature and 1,500 psi pressure in a gas line designed for 400 psi are “normal” setpoints.

Of course, not every equipment failure or power outage is a cyber-physical attack. When your building is attacked, you probably won’t suspect it was a hacker—until you see a pattern. Lots of equipment will act “squirrelly” and you’ll know—it’s a cyber-physical attack !

When a cyber-physical attack occurs, it can mean years of court cases, job losses, higher insurance rates, and criminal litigation. Organizations with a high profile in the community have a responsibility to assess the vulnerability of their facilities to disruption by hackers. For example, it is not enough for a hospital to say that they took the normal standard of care in their industry—they will be called upon to show that they took every reasonable precaution. It also takes years to overcome the loss of safety credibility to employees and the local community.

Taking the right steps ahead of time, and equipping your facility and employees with the training, knowledge, and tools they need to prevent an attack and deal with one when it occurs may save lives. If you think I’m kidding, read about how a hacker can degrade or destroy your building equipment in the following chapters. If your building is connected to the Internet (directly or indirectly), a hacker may be able to install ransomware and take complete control of your building.

Take my advice and do the following:

  • Completely disconnect your building controls system from the Internet immediately.

  • Segregate the BCS from all other computer networks.

  • Remove all modems and wireless cards from all computers, printers, and servers.

  • Stop vendors from connecting remotely to your BCS. If they must connect to the BCS for maintenance, allow vendors to only use your laptops. These are laptops that stay in the building under lock and key and that are scanned for malware before and after they are used.

Have a professional perform a vulnerability assessment and report the results directly to top management. If you do these things, you will be able to sleep at night. Don’t be “that guy.”

Hand-Off-Auto Switch (H-O-A)

A device that has switches that maintain their position. Start and Stop buttons have momentary actions. The Off position prevents any operation. Used in a situation that has a single point of manual control to allow the motor (or other device) to (A) operate from an automated building control system, (O) not operate, or (H) operate with no safeguards or automated control. The Hand position is used to bump the motor or to operate for shorts times while observed by operating personnel.

I wrote this book because I searched unsuccessfully for cyber-physical attack recovery procedures specifically written for building controls or SCADA. I only found recovery plans when a hacker steals information or defaces websites. I did not find any that specifically address how to stop a cyber-physical attack intended to damage building equipment or how to react after a building has been attacked. This book should give you an idea how bad things can get, and how serious a problem a well-planned cyber-physical attack can be. Throughout this book, I use jargon taken from electrical and mechanical engineering fields as well as information technology and physical security environments. I include a definition of some terms that readers may not be familiar with in a text box or footnote. Otherwise, most definitions can be found in another book I wrote, the Cybersecurity Lexicon (Apress, 2016).

This book contains step-by-step instructions, checklists, and forms that you can use to develop your own Cyber-Physical Attack Recovery Procedures. Just as no two buildings are alike, no two Recovery Procedures are alike. Only a Recovery Procedures document that has been prepared by qualified engineers for your specific facility should be implemented. The appendix is merely a suggested template to be customized for your specific facility. A soft copy of the template is available for download (with fillable fields) on the Apress web page for this book ( www.apress.com/9781484220641 ).

This book is not a crisis management plan. It is not a COOP plan (Continuity of Operations Plan). It is not a business continuity plan, IT application response plan, or an IT infrastructure response plan. Those are designed to prepare for a cyber-attack on an enterprise IT system (database) or organize a response team prior to a natural disaster. Other folks have already written those.

This book assumes hackers have already gained entry to your building controls. I describe the damage hackers can do, and suggest how maintenance personnel should respond to avoid prolonging the attack, and how to recover from the attack. The Recovery Procedures document is a template for you to modify based on the specific equipment in your facility with instructions on how to restore building equipment to normal operation when systems begin to act erratically, or fail completely. When hackers shut off the building water, turn off the power, disable the sewage effluent pumps, and activate the fire alarm, you have to do something quick. You won’t have time to figure out the proper Sequence of Operations for your boilers and chillers. You need to quickly turn things off before a hacker can do any more damage. And, hackers can damage multiple systems in multiple buildings at the same time—from the other side of the planet. This book will help you create custom checklists for your equipment.

Normally, the response to a physical attack and the speed with which security staff reacts is a function of the type of attack. For example, a well-trained security force is able to respond to an active shooter in mere seconds. Unfortunately, a professional hacker can disable multiple pieces of critical building equipment in milliseconds, so monitoring the BCS for precursors of a cyber-physical attack, and an automated response based on intrusion detection is essential to prevent damage to expensive building equipment.

To make matters worse, unlike an active shooter who is no longer a threat once he has been dealt with, a hacker’s handiwork persists long after you’ve cut the cord, and a hacker can continue to damage your equipment even if he no longer is in direct communication with the building controls system. A “sneaky” hacker (as Jack Nicholson says, “Is there another kind?”) will install cyber booby-traps before making his presence known. Before maintenance personnel sound the all-clear, they will have to isolate each and every piece of equipment from the BCS and be prepared to operate all systems manually—just like the old days. Maintenance personnel will have to go to the equipment to read dials, turn valves and open dampers to bring the building back on line—safely.

Lastly, I want to mention the business case for putting Recovery Procedures in place for when a cyber-physical attack occurs. The cost to replace a 2 MW diesel backup generator is at least $2 million. An average chiller costs $200,000; a medium-size boiler probably $600,000. Normally, replacing any of these would take six months because these are long lead items. You may be able to repair some damaged equipment—provided you can get parts. Another thing to consider is that you will likely not be the only one attacked. Once the floodgates open and all-out cyber warfare begins, you will be ordering equipment at the same time as other victims of cyber-physical attacks.

Blamestorming

Figuring out who to blame when something goes wrong.

Oh, one more thing that I forgot to mention. An adversary can cause physical damage to your facilities, but there is another aspect to their activities that needs attention. The fact is, hackers are monitoring your company activities using your building controls and other building-related systems that your IT and security staff are probably unaware even exist.

Contents

  1. Chapter 1:​ Cyber-Physical Attack Recovery Procedures
    1. Purpose of the Recovery Procedures
    2. Cyber-Physical Attack Timetable
    3. Recovery Procedures Information
    4. Applicable Directives
    5. Objectives for a Plan
    6. Incident Response Teams
    7. Recovery Management Team (MGMT)
      1. General Activities
      2. Procedures by Phase
    8. Recovery Facilities Team (FAC)
      1. Procedures by Phase
    9. Recovery Tech Support Team (TECH)
      1. Procedures by Phase
    10. Recovery Security Team (SEC)
      1. Procedures by Phase
    11. Recovery Phases
      1. Phase 1:​ Detection
      2. Phase 2:​ Mitigation
      3. Phase 3:​ Recovery
    12. Assumptions
    13. Critical Success Factors
    14. Mission Critical Systems
  2. Chapter 2:​ Threats and Attack Detection
    1. Probable Threats
      1. Cyber-Physical Attack Detection
    2. Incident Response Tools
      1. Incident Categorization
    3. Mitigation
    4. Troubleshooting
      1. Step 1:​ Confirm that the “event” is a Cyber-physical Attack
      2. Step 2:​ Slow Down the Attack.​ Assume that the BCS is no Longer Under your Command
      3. Step 3:​ Stop the Attack.​ Shut Down the BCS
      4. Step 4:​ Assess the Damage to Whatever Failed, but Assume that all Equipment was Hacked
      5. Step 5:​ Replace Infected Servers and Repair Damaged Building Equipment
      6. Step 6:​ Reload the BCS and Restart Automatic Operation
  3. Chapter 3:​ Prevent Hackers from Destroying a Boiler
    1. Boiler Explosion
    2. Hot Water Heater Explosion
    3. Low Water Conditions
    4. Preventive Measures
    5. Improper Warm-up
    6. Start-up Procedures
    7. Start-up Checks
  4. Chapter 4:​ Prevent Hackers from Destroying a Pressure Vessel
  5. Chapter 5:​ Prevent Hackers from Destroying Chillers
    1. Troubleshooting Chillers
    2. Chiller Cyber-Attack Tree
  6. Chapter 6:​ Prevent Hackers from Destroying a Gas Fuel Train
  7. Chapter 7:​ Prevent Hackers from Destroying a Cooling Tower
    1. Troubleshooting Cooling Towers
  8. Chapter 8:​ Prevent Hackers from Destroying a Backup Generator
  9. Chapter 9:​ Prevent Hackers from Destroying Switchgear
  10. Chapter 10:​ Eight Steps to Defending Building Control Systems
    1. Discontinue Remote Connections to the BCS
    2. Implement Application Whitelisting
    3. Systematic Patch Management Regimen
    4. Reduce the Attack Surface
    5. Build a Defendable Network Environment
    6. Manage Authentication
    7. Monitor and Respond
    8. Do Not Use BCS Workstations for Anything Else
  11. Chapter 11:​ Block Hacker Surveillance of Your Buildings
  12. Chapter 12:​ Cyber-Physical Attack Recovery Procedures Template
    1. START HERE
      1. EVENT TICKET
      2. INCIDENT RESPONSE TEAM
      3. INCIDENT RESPONSE TEAM MEMBER LISTING
      4. EQUIPMENT RECOVERY PRIORITY LIST
      5. BCS NOTIFICATIONS REPORT
      6. PATCH MANAGEMENT REPORT
      7. VULNERABILITY ASSESSMENTS HISTORY
    2. PHASE 1:​ DETECTION
      1. Observables
    3. Real-World Alerts
    4. Virtual-World Alerts
    5. Intrusion Detection System Alerts
      1. EVENT DIAGNOSTICS TABLE
      2. MASTER LIST OF DEFICIENCIES
      3. DECISION TIME
    6. PHASE 2:​ MITIGATION
      1. Mitigation and Segmentation
      2. Building Equipment Damage Assessments
      3. BCS Network Integrity Checks
      4. Server/​Workstation Process Checks
      5. BCS INTEGRITY CHECKS
      6. BUILDING CONTROLS SYSTEM INFORMATION
      7. BCS SERVER DETAILS
      8. BUILDING INSPECTION
      9. EQUIPMENT OR SERVICE FAILURE REPORT
      10. DAMAGE ASSESSMENT ACTIVITIES
      11. RAPID INSPECTION CHECKLISTS
      12. COOLING TOWER
      13. BOILER OR PRESSURE VESSEL
      14. BOILER VISUAL INSPECTION
      15. BOILER OR PRESSURE VESSEL
      16. CHILLED WATER SYSTEM
      17. CHILLER VISUAL INSPECTION
      18. CHILLED WATER SYSTEM CHECKLIST
      19. AIR HANDLING UNIT CHECKLIST
      20. AIR HANDLING UNIT VISUAL INSPECTION
      21. COMPUTER ROOM AIR CONDITIONER
      22. PUMP:​ PRIMARY CHILLED WATER
      23. PUMP:​ SECONDARY CHILLED WATER
      24. PUMP:​ PRIMARY DOMESTIC HOT WATER
      25. PUMP:​ SECONDARY DOMESTIC HOT WATER
      26. FAN COIL UNIT
      27. DIESEL BACKUP GENERATOR
      28. DIESEL SYSTEM CHECKLIST
      29. LUBE OIL SYSTEM
      30. LUBE OIL SYSTEM CHECKLIST
      31. VENTILATION SYSTEM CHECKLIST
      32. VENTILATION SYSTEM
      33. ELECTRICAL SWITCHGEAR INSPECTION
      34. ELECTRICAL SWITCHGEAR CHECKLIST
      35. RESTART OPERATING CONDITIONS
    7. PHASE 3:​ RECOVERY
      1. RECOVERY EVENT RECORDING FORM
      2. BUILDING CONTROLS SYSTEM
      3. BUILDING CONTROLS SYSTEM FIELD TEST
      4. BOILER OR PRESSURE VESSEL
      5. CHILLED WATER SYSTEM
      6. AIR HANDLING UNIT
      7. COMPUTER ROOM AIR CONDITIONER
      8. COOLING TOWER
      9. PUMP:​ PRIMARY CHILLED WATER
      10. PUMP:​ SECONDARY CHILLED WATER
      11. PUMP:​ PRIMARY DOMESTIC HOT WATER
      12. PUMP:​ SECONDARY DOMESTIC HOT WATER
      13. FAN COIL UNIT
      14. DIESEL BACKUP GENERATOR
      15. VENTILATION UNIT
      16. ELECTRICAL SWITCHGEAR CHECKLIST
      17. INTEGRATED SYSTEMS TEST:​ EMERGENCY POWER
      18. INTEGRATED SYSTEMS TEST:​ EMERGENCY POWER
      19. INTEGRATED SYSTEMS TEST:​ EMERGENCY POWER
      20. OPERATIONAL SECURITY LOG
      21. HVAC CONTROL SYSTEM DRAWINGS
      22. VALVE SCHEDULE
      23. DAMPER SCHEDULE
      24. THERMOSTAT AND OCCUPANCY SENSOR SCHEDULE
      25. BUILDING EQUIPMENT SCHEDULE
      26. BUILDING OCCUPANCY SCHEDULE
    8. POINTS SCHEDULE
      1. BUILDING CONTROL NETWORK RISER DIAGRAM
      2. SEQUENCE OF OPERATION
      3. PERFORMANCE VERIFICATION TEST RESULTS
      4. COMMISSIONING REPORT
      5. EQUIPMENT MAINTENANCE MANUALS
      6. LIST OF AUTHORIZED MAINTENANCE LAPTOPS
      7. VENDOR CONTACT LIST
  13. Index

About the Author and About the Technical Reviewer

About the Author

Luis Ayala worked for the US Department of Defense for more than 25 years, with the past 11 years at the Defense Intelligence Agency. Prior to his appointment as a defense intelligence senior leader in 2008, he held several leadership positions at the branch and division levels.

His tenure culminated with the position as senior technical expert (facilities/construction). Mr. Ayala earned his Bachelor of Architecture degree from Pratt Institute and he received his Master of Science and Technology Intelligence from the National Intelligence University. NIU is the intelligence community’s sole accredited, federal degree–granting institution. His master’s thesis, titled “Cybersecure Facilities for the Intelligence Community,” is classified. Mr. Ayala was awarded the DIA Civilian Expeditionary Medal and the Civilian Combat Support Medal.

About the Technical Reviewer

William Beck has a BS in engineering from the University of South Carolina, majoring in structures and mechanics, and NROTC. He served six years in the US Navy as a lieutenant and naval aviator with a multi- and single engine, land, instrument, and commercial license. After earning an MBA from Webster University in 1980, Mr. Beck was hired by Exxon Co. USA as a subsurface engineer, where he worked his way up to senior engineer and also earned his professional engineer’s license in 1985. For 31 years, he worked for the federal government as an engineering branch chief for the General Services Administration’s Public Building Service Property Development in Fort Worth, TX.

Mr. Beck is married with two sons and two grandsons. He is an active pilot, scuba diver, traveler, and golfer.

Footnotes

1 Smith, Rebecca. “Assault on California Power Station Raises Alarm on Potential for Terrorism” Wall Street Journal , February 5, 2014.

2 Shanker, Thom. “Test Errors Caused Chernobyl, Soviets Say” Chicago Tribune , August 22, 1986.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.139.72.78