Just a Little More Decomposition for Impact

A simple decomposition strategy for impact that many in cybersecurity are already familiar with is confidentiality, integrity, and availability, or “C, I, and A.” As you probably know, “confidentiality” refers to the improper disclosure of information. This could be a breach of millions of records, or it could mean stealing corporate secrets and intellectual property. “Integrity” means modifying the data or behavior of a system, which could result in improper financial transactions, damaging equipment, misrouting logistics, and so on. Lastly, “availability” refers to some sort of system outage resulting in a loss of productivity, sales, or other costs of interference with business processes. We aren't necessarily endorsing this approach for everyone, but many analysts in cybersecurity find that these decompositions make good use of how they think about the problem.

Let's simplify it even further in the way we've seen one company do it by combining confidentiality and integrity. Perhaps we believe availability losses were fairly common compared to others and that estimating availability lends itself to using other information we know about the system, such as the types of business processes the system supports, how many users it has, how it might affect productivity, and whether it could impact sales while it is unavailable. In Table 6.1, we show how this small amount of additional decomposition could look if we added it to the spreadsheet shown in the one‐for‐one substitution model shown in Chapter 3.

To save room, we are only showing here the impact decomposition for two types of impact. Each of these impact types would have its own annual probability and there would be formulas for combining all of these inputs. To see the entire spreadsheet, including how we determined which of these impacts occurred for how the costs are added up, just go to the full spreadsheet example at www.howtomeasureanything.com/cybersecurity. In addition to the original model shown in Chapter 3, you will also see this one and some other variations on decomposition.

The next steps in this decomposition are to determine whether confidentiality and integrity losses occurred, whether availability losses occurred, or both. When availability losses are experienced, that loss is computed by multiplying the hours of outage duration times the cost per hour of the outage. Any confidentiality losses are added in.

Just as we did in the simpler model in Chapter 3, we generate thousands of values for each row. In each random trial we determine the type of event and its cost. The entire list of event costs are totaled for each of the thousands of trials, and a loss exceedance curve is generated. As before, each row could have a proposed control, which would reduce the likelihood and perhaps impact of the event (these reductions can also be randomly selected from defined ranges).

If an event is an attack on a given system, we would know something about how that system is used in the organization. For example, we would usually have a rough idea of how many users a system has—whether they would be completely unproductive without a system or whether they could work around it for a while—and whether the system impacts sales or other operations. And many organizations have had some experience with system outages that would give them a basis for estimating something about how long the outage could last. We could decompose confidentiality further by estimating the number of records compromised and then estimate the costs as a function of records lost. We could also separate the costs of internal response, lost revenue as a function of duration of the outage, and even human safety where appropriate. You should choose the decomposition that you find more convenient and realistic to assess. There are many more ways to decompose impacts, and some decompositions of probabilities will be addressed in later chapters.

Now that you can see how decomposition works in general, let's discuss a few other strategies we could have used. If you want to decompose your model using a wider variety of probability distributions, see details on a list of distributions in Appendix A. And, of course, you can download an entire spreadsheet from www.howtomeasureanything.com/cybersecurity, which contains all of these random probability distributions written in native (i.e., without VBA macros or add‐ins) MS Excel.

A Few Decomposition Strategies to Consider

We labeled each row in the simulation shown in Table 6.1 as an event. However, in practice you will need to define an event more specifically and, for that, there are multiple choices. Think of the things you would normally have plotted on a risk matrix. If you had 20 things plotted on a risk matrix, were they 20 applications? Were they 20 categories of threats? Were they business units of the organization or types of users?

It appears that most users of the risk matrix method start with an application‐oriented decomposition. That is, when they plotted something on a risk matrix, they were thinking of an application. This is a perfectly legitimate place to start. Again, we have no particular position on the method of decomposition until we have evidence saying that some are better and others are worse. But out of convenience and familiarity, our simple model in Chapter 3 starts with the application‐based approach to decomposing the problem. If you prefer to think of the list of risks as being individual threat sources, vulnerabilities, or something else, then you should have no problem extrapolating the approach we describe here to your preferred model.

Once you have defined what your rows in the table represent, then your next question is how detailed you want the decomposition in each row to be. In each decomposition, you should try to leverage things you know—we can call them observables. Table 6.2 has a few more examples.

If we modeled even some of these details, we may still have a wide range, but we can at least say something about the relative likelihood of various outcomes. The cybersecurity professional who thought that a range for a loss was zero to a half‐billion dollars was simply not considering what can be inferred from what is known rather than dwelling on all that isn't known. A little bit of decomposition would indicate that not all the values in that huge range are equally likely. You will probably be able to go to the board with at least a bit more information about a potential loss than a flat uniform distribution of $0 to $500 million or more.

Remember, the reason you do this is to evaluate alternatives. You need to be able to discriminate among different risk mitigation strategies. Even if your range was that wide and everything in the range were equally likely, it is certainly not the case that every system in the list has a range that wide, and knowing which do would be helpful. You know that some systems have more users than others, some systems handle personal health information (PHI) or payment card industry (PCI) data, and some do not, some systems are accessed by vendors, and so on. All this is useful information in prioritizing action even though you will never remove all uncertainty.

This list just gives you a few more ideas of elements into which you could decompose your model. So far we've focused on decomposing impact more than likelihood because impact seems a bit more concrete for most people. But we can also decompose likelihood. Chapters 8 and 9 will focus on how that can be done. We will also discuss how to tackle one of the more difficult cost estimations—reputation loss—later in this chapter.

Decision Analysis: An Overview of How to Think about a Problem

A good background for thinking about decomposition strategies is the work of Ron Howard, who is generally credited for coining of the term “decision analysis” in 1966.³ Howard and others inspired by his work were applying the somewhat abstract areas of decision theory and probability theory to practical decision problems dealing with uncertainties. They also realized that many of the challenges in real decisions were not purely mathematical. Indeed, they saw that decision makers often failed to even adequately define what the problem was. As Ron Howard put it, we need to “transform confusion into clarity of thought and action.”⁴

Howard prescribes three prerequisites for doing the math in decision analysis. He stipulates that the decision and the factors we identify to inform the decision must be clear, observable, and useful.

• Clear: Everybody knows what you mean. You know what you mean.
• Observable: What do you see when you see more of it? This doesn't mean you will necessarily have already observed it but it is at least possible to observe and you will know it when you see it.
• Useful: It has to matter to some decision. What would you do differently if you knew this? Many things we choose to measure in security seem to have no bearing on the decision we actually need to make.

All of these conditions are often taken for granted, but if we start systematically considering each of these points on every decomposition, we may choose some different strategies. Suppose, for example, you wanted to decompose your risks in such a way that you had to evaluate a threat actor's “skill level.” This is one of the “threat factors” in the OWASP standard, and we have seen homegrown variations on this approach. We will assume you have already accepted the arguments in previous chapters and decided to abandon the ordinal scale proposed by OWASP and others and that you are looking for a quantitative decomposition about a threat actor's skill level. So now apply Howard's tests to this factor.

The Clear, Observable, and Useful Test Applied to “Threat Actor Skill Level”

• Clear: Can you define what you mean by “skill level”? Is this really an unambiguous unit of measure or even a clearly defined discrete state? Does saying, “We define ‘average’ threat as being better than an amateur but worse than a well‐funded nation state actor” really help?
• Observable: How would you even detect this? What basis do you have to say that skill levels of some threats are higher or lower than others?
• Useful: Even if you had unambiguous definitions for this, and even if you could observe it in some way, how would the information have bearing on some action in your firm?

We aren't saying threat skill level is necessarily a bad part of a strategy for decomposing risk. Perhaps you have defined skill level unambiguously by specifying the types of methods employed. Perhaps you can observe the frequency of these types of attacks, and perhaps you have access to threat intelligence that tells you about the existence of new attacks you haven't seen yet. Perhaps knowing this information causes you to change your estimates of the likelihood of a particular system being breached, which might inform what controls should be implemented or even the overall cybersecurity budget. If this is the case, then you have met the conditions of clear, observable, and useful. But when this is not the case—which seems to be very often—evaluations of skill level are pure speculation and add no value to the decision‐making process.

Avoiding Over‐Decomposition

The threat skill level example just mentioned may or may not be a good decomposition depending on your situation. If it meets Howard's criteria and it actually reduces your uncertainty, then we call it an informative decomposition. If not, then the decomposition is uninformative, and you are better off sticking with a simpler model.

Imagine someone standing in front of you holding a crate. The crate is about 2 feet wide and a foot high and deep. They ask you to provide a 90% CI on the weight of the crate simply by looking at it. You can tell they're not a professional weightlifter, so you can see this crate can't weigh, say, 350 pounds. You also see that they lean a bit backward to balance their weight as they hold it. And you see that they're shifting uncomfortably. In the end, you say your 90% CI is 20 to 100 pounds. This strikes you as a wide range, so you attempt to decompose this problem by estimating the number of items in the crate and the weight per item. Or perhaps there are different categories of items in the crate, so you estimate the number of categories of items, the number in each category, and the weight per item in that category. Would your estimate be better? Actually, it could easily be worse. What you have done is decomposed the problem into multiple purely speculative estimates that you then use to try to do some math. This would be an example of an uninformative decomposition.

The difference between this and an informative decomposition is whether or not you are describing the problem in terms of quantities you are more familiar with than the original problem. An informative decomposition would be decompositions that use specific knowledge that the cybersecurity expert has about their environment. For example, the cybersecurity expert can get detailed knowledge about the types of systems in their organization and the types of records stored on them. They would have or could acquire details about internal business processes so they could estimate the impacts of denial of service attacks. They understand what types of controls they currently have in place. Decompositions of cybersecurity risks that leverage this specific knowledge are more likely to be helpful.

However, suppose a cybersecurity expert attempts to build a model where they find themselves estimating the number and skill level of state‐sponsored attackers or even the hacker group Anonymous (about which, as the name implies, it would be very hard to estimate any details). Would this actually constitute a reduction in uncertainty relative to where they started?

Decompositions should be less abstract to the expert than the aggregated amount. If you find yourself decomposing a dollar impact into factors such as threat skill level, then you should have less uncertainty about the new factors than you did about the original, direct estimate of monetary loss.

If decomposition causes you to widen a range, that might be informative if it makes you question the assumptions of your previous range. For example, suppose we need to estimate the impact of a system availability risk where an application used in some key process—let's say order taking—would be unavailable for some period of time. And suppose that we initially estimated this impact to be $150,000 to $7 million. Perhaps we consider that to be too uncertain for our needs, so we decide to decompose this further into the duration of an outage and the cost per hour of an outage. Suppose further that we estimated the duration of the outage to be 15 minutes to 4 hours and the cost per hour of the outage to be $200,000 to $5 million. Let's also state that these are lognormal distributions for each (as discussed in Chapter 3, this often applies where the value can't be less than zero but could be very large). Have we reduced uncertainty? Surprisingly, no—not if what we mean by “uncertainty reduction” is a narrower range. The 90% CI for the product of these two lognormal distributions is about $100,000 to $8 million—wider than the initial 90% CI of $150,000 to $7,000,000. But even though the range isn't strictly narrower, you might think it was useful because you realize it is probably more realistic than the initial range.

Now, just a note in case you thought that to get the range of the product you multiply the lower bounds together and then multiply the upper bounds together: That's not how the math works when you are generating two independent random variables. Doing it that way would produce a range of $50,000 to $20 million (0.25 hours times $200,000 per hour for the lower bound and 4 hours times $5 million per hour for the upper bound). This answer could only be correct if the two variables are perfectly correlated—which they obviously would not be.

So decomposition might be useful just as a reality check against your initial range. This can also come up when you start running simulations on lots of events that are added up into a portfolio‐level risk, as the spreadsheet shown in Table 6.1 does. When analysts are estimating a large number of individual events, it may not be apparent to them what the consequences for their individual estimates are at a portfolio level. In one case we observed that subject matter experts were estimating individual event probabilities at somewhere between 2% and 35% for about a hundred individual events. When this was done they realized that the simulation indicated they were having a dozen or so significant events per year. A manager pointed out that the risks didn't seem realistic because none of those events had been observed even once in the last five years. The estimates would only make sense if the subject matter experts had reason to believe there would be a huge uptick in these event frequencies (it was over a couple of years ago, and we can confirm that this is not what happened). But, instead, the estimators decided to rethink what the probabilities should be so that they didn't contrast so sharply with observed reality.

A Summary of Some Decomposition Rules

The lessons in these examples can be summarized in two fundamental decomposition rules:

• Decomposition Rule #1: Decompositions should leverage what you are better at estimating or data you can obtain (i.e., don't decompose into quantities that are even more speculative than the first).
• Decomposition Rule #2: Check your decomposition against a directly estimated range with a simulation, as we just did in the outage example. You might decide to toss the decomposition if it produces results you think are absurd, or you might decide your original range is the one that needs updating.

These two simple rules have some specific mathematical consequences. In order to ensure that your decomposition strategy is informative—that is, results in less uncertainty than you had before—consider the following:

• If you are expecting to reduce uncertainty by multiplying together two decomposed variables, then the decomposed variables need to not only have less uncertainty than the initial range but often a lot less. As a rule of thumb, the ratios of the upper and lower bounds for the decomposed variables should be a lot less than a third the width of the ratio of upper and lower bounds of the original range. For the case in the previous section, the ratio of bounds of the original range was about 47 ($7 million/$150,000), while the other two ranges had ratios of bounds of 16 and 25, respectively.
• If most of the uncertainty is in one variable, then the ratio of the upper and lower bounds of the decomposed variable must be less than that of the original variable. For example, suppose you initially estimated that the cost of an outage for one system was $1 million to $5 million. If the major source of uncertainty about this cost is the duration of an outage, the upper/lower bound ratio of the duration must be less than the upper/lower bound ratio of the original estimate (5 to 1). If the range of the duration doesn't have a lower ratio of upper/lower bounds, then you haven't added information with the decomposition. If you have reason to believe your original range, then just use that. Otherwise, perhaps your original range was just too narrow, and you should go with the decomposition.
• In some cases the variables you multiply together are related in a way that eliminates the value of decomposition unless you also make a model of the relationship. For example, suppose you need to multiply A and B to get C. Suppose that when A is large, B is small, and when B is large, A is small. If we estimate separate, independent ranges for A and B, the range for the product C can be greatly overstated. This might be the case for the duration and cost per hour of an outage. That is, the more critical a system, the faster you would work to get them back online. If you decompose these, you should also model the inverse relationship. Otherwise, just provide a single overall range for the cost of the impact instead of decomposing it.
• If you have enough empirical data to estimate a distribution, then you probably won't get much benefit from further decomposition.

Another Way to Model Reputation Damage: Penance Projects

We aren't saying major data breaches are without costs. There are real costs associated with them, but we need to think about how to model them differently than with vague references to reputation. If you think your risks are more like Adobe's, Merck's, or Marriot's and less like SolarWinds’ and Equifax's, then you need a different approach than just a drop in stock price to model impact on reputation. For example, instead of stock price drops or loss of revenue, management at Home Depot reported other losses in the form of expenses dealing with the event, including “legal help, credit card fraud, and card re‐issuance costs going forward.”¹⁶

Note that legal liabilities like what Home Depot faced are usually considered separately from reputation damage and there is now quite a lot of data on that point. Regulatory fines and Advisen data have been collected now for years. Even the European Union's General Data Protection Regulation (GDPR)—which was enacted just as the first edition of this book was released—now has thousands of fines in publicly available databases.

Costs, other than stock price drops and legal liabilities may be more realistically modeled as “penance projects.” Penance projects are expenses incurred to limit the long‐term impact of loss of reputation. In other words, companies appear to engage in efforts to control damage to reputation instead of bearing what could otherwise be much greater damage. The effect these efforts have on reducing the real loss to reputation seem to be enough that the impact seems hard to detect in sales or stock prices. These efforts include the following:

• Major new investments in cybersecurity systems and policies to correct cybersecurity weaknesses.
• Replacing a lot of upper management responsible for cybersecurity. (It may be scapegoating, but it may be necessary for the purpose of addressing reputation.)
• A major public relations push to convince customers and shareholders the problem is being addressed. (This helps get the message out that the efforts of the preceding bullet points will solve the problem.)
• Marketing and advertising campaigns (separate from getting the word out about how the problem has been confidently addressed) to offset potential losses in business.

For most organizations that aren't in the types of services Equifax or SolarWinds provide, these damage‐control efforts to limit reputation effects appear to be the real costs here—not so much reputation damage itself. Each of these are conveniently concrete measures for which we have multiple historical examples. Of course, if you really do believe that there are other costs to reputation damage, you should model them. But what does reputation damage really mean to a business other than legal liabilities if you can't detect impacts on either sales or stock prices? For example, employee turnover might be an arguable loss. Just be sure you have an empirical basis for your claim. Otherwise, it might be simpler to stick with the penance project cost strategy.

So if we spend a little more time and effort in analysis, it should be possible to tease out reasonable estimates even for something that seems as intangible as reputation loss.