Introduction

If you want a good measurement problem, watch the movie Die Hard with a Vengeance. In the movie, the characters played by Bruce Willis and Samuel L. Jackson are trying to stop the bad guys and find themselves in a crowded park with a five-gallon plastic jug, a three-gallon plastic jug, a water fountain, and a big bomb attached to a scale. To defuse the bomb, they must place four gallons of water (with no more than a few ounces error) on the scale within a certain amount of time; otherwise, everyone dies. They solve the problem, of course, but only after realizing that the jugs and scale are not enough and that they need a precise, logical process to arrive at the necessary measurement. The scene is great. It has a measurement challenge, an acceptable margin of error, and unacceptable consequences should the measurement fail. And in the end, the problem is much less about metrics (volume and weight in this case) and much more about the process of measuring in support of a decision (whether or not to put the jug on the scale and risk going boom).

Measuring IT Security

This book is also about the process of measurement as much as it is about metrics themselves. IT security practitioners, from the CISO down, are increasingly being directed to measure security in their organizations and improve the effectiveness of their data protection activities. From regulatory and industry compliance for Sarbanes-Oxley or PCI DSS to discussions of “Advanced Persistent Threats” posed by nation states and transnational criminal or terrorist organizations, IT security has experienced a dramatic bump in visibility. No less an authority than the President of the United States has weighed in, with a 2009 review of America’s cyberspace policy that concluded that the digital infrastructure of the United States was neither secure nor resilient to ongoing attacks. At the top of the report’s list of recommendations for improving the security of our infrastructure was the requirement to implement better security measurement and metrics.

This brings us to an important and fundamental question: What is this thing we call security that we are so keen to measure? Our industry often uses words like security, risk, and vulnerability haphazardly, without first even bothering to define our understanding of what the terms mean. We often hear the mantra, you cannot manage what you do not measure, and I agree with this. But if you lack definition or consensus regarding the phenomena that you hope to manage (system performance versus human behavior, for example), then jumping straight into metrics is a recipe for frustration and failure. Your understanding of what you are measuring must be specific and agreed upon if your data is to be specific and accepted by everyone. Thus, a corollary to the mantra can be stated like so: You cannot measure what you do not understand.

A Rocky Understanding

Some of the most difficult IT security metrics work comes from trying to figure out what you are trying to figure out. After all, security isn’t a tangible thing. But forget security for a moment, and let’s look at measuring something “easier” like, for instance, a rock. Rock metrics seem pretty straightforward. Rocks have height, width, and depth that you can easily measure with a ruler. Rocks have weight that you can measure by putting a rock on a scale. It would be great if measuring security were this easy, and the way some security pros measure it, you’d think that it was. But even rocks have characteristics that complicate measurement. Rocks have mass, which is different from weight. How do you measure that? Rocks have chemical composition and mineralogy. Rocks have special metrics such as clast size, which is a measure of the size of the rock’s individual grains. And there are even more challenging metrics for rocks. Many rocks have social value and financial value that can be measured, although these metrics are far from intrinsic to the properties of the rock.

So it turns out that even measuring something that appears simple and tangible is not a straightforward proposition. If you do not understand what aspects of a rock you are interested in, you’ll have a much more difficult time assessing which metrics will increase your knowledge or improve your decisions regarding the rock. Would this rock be better to throw at my enemies or to polish and put into a ring? You might find yourself regretting that you hurled 24-karat diamonds at your adversaries in defense of your stash of iron pyrite. If we can’t even measure a rock without agreeing on our process and criteria of inquiry—how much more difficult will it be to measure IT security?

Security experts often fall into a trap of trying to measure security without first understanding what we really want to know. We may think we know, but too often our line of inquiry is simplistic and relates only to our immediate experiences and perceived priorities. How many of us have taken part in discussions about the security of our organization only to discover later (usually when it comes time to implement something) that everyone involved in the discussion had very different ideas about what security meant? This is especially common when business-side security managers are talking to security technologists. Business definitions of security differ from technical definitions, because the things a financial analyst is familiar with and cares about are often very different from what a firewall administrator is familiar with and cares about.

Improving Security with IT Security Metrics

As the security industry (and profession) matures, and as security is recognized as a core business process, the need for effective measurement of that process is growing. The IT security metrics movement is growing as well, in response to this need. This book is intended to contribute to the ongoing conversation about security measurement and to help you understand how to put metrics to effective use within your own organization. To this end, I have proposed a framework that helps situate security and security metrics within the context of business process improvement, and I hope to provide you with some ways of looking at measuring IT security that are new, and perhaps different from, what you might see in other metrics books.

How This Book Is Organized

I’ve divided this book into four parts, which reflect the general content and purpose of the individual chapters. I did not write the parts or chapters as independent modules, but rather as an interconnected narrative that starts at something like a beginning and closes with something like an end. (Of course, you do not have to read it linearly, but that is the way that I laid out the book.) I also constructed the book around the Security Process Management (SPM) Framework, a general methodology for creating a cohesive IT security metrics program that considers both tactical and strategic elements of a measurement program. So all things being equal, I suggest you read the book start to finish and feel free to skip those chapters covering concepts with which you are already proficient.

I have also invited several industry practitioners with experience in one or more aspects of metrics to contribute case studies to the book. Each part closes with one of these contributed case studies, more or less tied to the content of that particular set of chapters. The case studies serve to show how what I discuss may play out in different contexts and environments, and I hope you will find them useful alternative perspectives on measuring security.

Parts

The book has four parts.

Part One: Introducing Security Metrics

Part One discusses the state of IT security metrics today, critiques several existing security metrics and preconceptions regarding how security should be measured, and offers alternative ways of thinking about security metrics. The part also introduces concepts of data that are important in understanding how to measure security.

Part Two: Implementing Security Metrics

Part Two introduces the Security Process Management (SPM) Framework and discusses analytical strategies for security metrics data. This part also explores the concept of the security measurement project (SMP), a bounded metrics exercise that is a key component of the framework.

Part Three: Exploring Security Measurement Projects

Part Three discusses specific, practical examples of SMPs from goals, to data, to analysis. These project examples give readers a concrete introduction to the concepts referred to in earlier chapters, and shows how they can be implemented.

Part Four: Beyond Security Metrics

Part Four explores how to take a security metrics program and adapt it strategically to a variety of organizational contexts and environments, the goal being the continuous improvement of security over time.

Chapters

Each chapter in the book covers specific material germane to the understanding and development of IT security metrics and to the SPM Framework. I have made every effort to make the content of these chapters practical: Instead of just describing concepts, I strive to provide concrete, operational examples of what I am talking about. My goal is for readers to be able to form ideas about how they might operationalize those concepts within their own practices and organizations. To this end, chapters include methods, use cases, and tool descriptions that relate to security metrics and can describe templates and organizational considerations as well. Each chapter also includes a summary and recommendations for further, more in-depth, reading on the chapter concepts and topics discussed.

Final Thoughts

This book was born in an ending. As I finished my Ph.D. program, it became increasingly obvious to me that my industry colleagues could benefit from many of the social science research methodologies and techniques that I had been exploring for several years. My dissertation topic itself was less important. Writing a dissertation in the social sciences can be an exercise in taking an interesting, relevant idea and drilling down into it so deeply that it no longer applies to anything except itself. But the dissertation process is about practice more than inspiration. As I came up for air in the wake of my research, I realized that, while my specific topic wasn’t going to change security practices, the techniques and tools I had learned very well might do so. I was reading others’ ideas on security metrics and realizing that the security field was at the beginning of a journey that has been made by industries and research fields since the beginning of scientific exploration. We’re new at it, and we have a lot to learn. But measurement is not new by any means, and neither are the methodologies of inquiry and empirical observation by which measurement is accomplished. I hope to share some of these methods with you in this book. If I’ve done my research correctly, you will be unfamiliar with some of them. If I’ve done my job as an author well, you will find that you can use them to understand and improve your security operations. I hope that I’ve accomplished both.