Overview
- People often measure things because somebody—some edict or some policy—stipulates that things should be measured.
- Unless measurement’s role is thought through, it can degenerate into a meaningless exercise.
- Measuring lengths down to the nearest sixteenth of an inch with a ruler that contains only quarter-inch marks.
- Measuring things without measurement context; measurements are not taken for the purpose of answering specific questions.
- Expressing measurements in a language the intended audience cannot understand, in effect, a foreign language.
- Measurement for measurement’s sake is a waste of time and money.
- This chapter
- describes how to measure the effectiveness of ongoing enterprise risk mitigation and security operations;
- offers guidance on how to measure the effectiveness of cybersecurity functional areas and their associated capabilities in everyday terms familiar—and therefore meaningful—to the enterprise; and
- describes expert judgment and observed data measurement approaches.
- Regardless of the measurement approach, an enterprise measurement program needs to produce results that support informed business decisions.
Topics
- Cybersecurity Measurement
- Cybersecurity Program Measurement
- Visualizing Cybersecurity Assessment Scores
- Cybersecurity Measurement Summary
Cybersecurity Measurement
- How does an enterprise measure cybersecurity?
- There is no shortage of assessment frameworks that
- focus on business processes or security controls;
- provide guidance on how to judge process, or control presence or compliance; and
- generally do not include guidance on how to score or measure the effectiveness of the security controls.
- For example, an enterprise evaluates its cybersecurity program against a control framework.
- 80% of controls are present and functioning.
- Is the cybersecurity good?
- Or are the 20% of the controls that are missing the ones that the attackers are exploiting to steal the enterprise’s data?
- Control frameworks used to measure cybersecurity programs pose challenges because they
- are often designed around programs achieving 100% compliance, else they are deficient;
- do not provide much guidance on how to prioritize remediation for noncompliance; and
- do not provide much guidance on how to prioritize maintenance of security controls to ensure the most important controls stay operational.
- Another framework challenge has to do with quantifying cybersecurity measurements.
- How does an enterprise score itself when it is trying to determine the effectiveness of its security controls ?
- When using frameworks with lots of controls,
- the enterprise can count how many controls are effective, ineffective, or absent, and they can score the program based on various ratios.
- When there are fewer controls,
- the enterprise may need to consider some shades of gray and give partial control scores based on how well they are implemented or how effectively they are used.
- When some controls are more important than others,
- the enterprise may need to weight scores to account for more important controls carrying more weight in the overall evaluation.
- Measurement challenges are non-trivial and contribute to the uncertainty in trying to get useful, actionable results from a cybersecurity program evaluation.
Cybersecurity Program Measurement
- The graphic depicts a cybersecurity program measurement approach that leverages (1) risk mitigations, (2) functional areas, and (3) ongoing security operations.
- This approach also encompasses the corresponding (1) security controls, (2) security capabilities, and (3) technologies and processes.
Object Measurement Example
Context
- Object Measurement (OM) is a methodology that can be used to measure an enterprise cybersecurity program and produce actionable results.
- Generally, OM consists of six steps.
- Step 1: Define the questions to be asked
- Step 2: Select appropriate objects to measure
- Step 3: For each object, define the object characteristics to measure
- Step 4: For each characteristic, create a value scale
- Step 5: Measure each object characteristic using the value scales
- Step 6: Calculate the overall index (e.g., Cybersecurity Program Assessment Index)
- The following example assessment illustrates how OM is used to measure enterprise cybersecurity program effectiveness.
OM Step 1: Define the question(s) to be answered
- For this example assessment, the question is:For the selected security scope, how effective is the enterprise’s cybersecurity program against cyberattacks?
- An enterprise can use various measurement approaches to measure cybersecurity.
- Expert judgment assessment is frequently used in real-world assessments, and will be demonstrated in this example along with techniques for managing its limitations and challenges.
- The challenge is that different evaluators can produce widely different results.
- Results are not always reproducible across different environments or at different times.
- Observed data associated with the cybersecurity processes or security controls can also be used.
- Results are generally more objective, more reproducible, and less subject to individual opinions, knowledge, experience, or judgment.
- For this measurement example, an expert judgment assessment is described and demonstrates how to reduce its intrinsic challenges.
OM Step 2: Select appropriate objects to measure
- Measurements can be done with different levels of detail and fidelity (in other words, various assessment scopes ), allowing for quick, high-level assessments or thorough, detailed assessments.
- Level 1—Focus on Risk Mitigations: Measure Risk Mitigations
- Level 2—Focus on Functional Areas: Measure Risk Mitigations + Functional Areas + Security Operations
- Level 3—Focus on Security Capabilities: Measure Risk Mitigations + Security Capabilities + Security Operations
- Level 4—Focus on Controls, Technologies, and Process: Measure Security Controls + Technologies + Processes
- The assessment scope is a portion of the enterprise’s environment where a security compromise will have a business impact.
- Within well-defined scopes , systems work together to maintain a particular security posture to defend against a business impact.
- Scope is frequently defined based on regulatory, statutory, or liability requirements.
- This example expert judgment assessment is a Level 2 Assessment.
- Appropriate objects to measure are
- risk mitigations,
- functional areas, and
- security operations.
OM Step 3: For each object, define the object characteristics to measure
- For this example assessment, the object characteristics to be measured are shown in the right-hand column of the figure.
- Risk Mitigations (Attack Sequence)
- Establish Foothold
- Command and Control
- Etc.
- Functional Areas
- Systems Administration
- Network Security
- Etc.
- Security Operations
- Policies and Policy Exception Management
- Project and Change Security Reviews
- Etc.
OM Step 4: For each characteristic, create a value scale
- Value scales help associate an enterprise vocabulary (that is, language) with measurement.
- The challenge is to establish value scales to make meaningful measurements.
- Meaningful means “the enterprise uses the measurements to determine whether and where cybersecurity needs to be improved.”
- OM value scale types include discrete, binary, and sliding.
- Value scale s have a minimum and maximum numeric value, along with a plain-language description on the tick-mark labels.
- The numeric range of values is not restricted to zero (0.00) to one (1.00) and can accommodate any numeric range.
- Discrete value scales allow for distinct interim numeric values and corresponding tick-mark labels (for example, 0.00 = Absent; 0.25 = Weak, and so on).
- Binary value scales are often used to measure on/off or yes/no or desired behavior/lack of desired behavior.
- Sliding value scales measure a minimum numeric value, a partial numeric value based on a ratio, and a maximum numeric value.
- Value scale tick-mark labels need to be defined in everyday enterprise language to aid in communicating measurement results.
- Note: There is no one set of terms (that is, numeric values and tick-mark labels) that defines value scales.
- The enterprise decides what terms define its value scales.
- For this example assessment, the expert judgment value scales to be used are listed in the right-hand column of the figure and subsequently explained in detail.
- Mitigations
- Risk Probability Reduction
- Risk Impact Reduction
- Functional Areas
- Effectiveness
- Comprehensiveness
- Security Operations
- Maturity
- Utilization
- An enterprise can gauge its risk mitigation efforts based on two dimensions.
- Risk Probability Reduction
- 1.00—High probability reduction
- 0.50—Medium probability reduction
- 0.00—Low probability reduction
- Risk Impact Reduction
- 1.00—High impact reduction
- 0.50—Medium impact reduction
- 0.00—Low impact reduction
- If an enterprise is significantly reducing the probability and the impact of the risk, then the enterprise’s security can be considered very good or excellent.
- An enterprise can gauge its functional areas based on two dimensions.
- Effectiveness measures how effective the functional area is in protecting the enterprise and mitigating cybersecurity risks.
- 1.00—High effectiveness
- 0.50—Medium effectiveness
- 0.00—Low effectiveness
- Comprehensiveness measures how comprehensively the functional area is used to protect the security scope.
- 1.00—High comprehensiveness
- 0.50—Medium comprehensiveness
- 0.00—Low comprehensiveness
- An enterprise can gauge its security operation s based on two dimensions.
- Maturity measures how well implemented the security operations element is.
- 1.00—High maturity
- 0.50—Medium maturity
- 0.00—Low maturity
- Utilization measures how much the security operations element is used in the enterprise.
- 1.00—High utilization
- 0.50—Medium utilization
- 0.00—Low utilization
OM Step 5: Measure each characteristic using the value scales
- Assess the characteristics using the value scales and record the resulting raw measurement data.
- The graphic shows this assessment’s example raw measurement data that represents an expert’s judgment as expressed by the defined value scales.
OM Step 6: Calculate the overall cybersecurity program assessment index using object measurement index equation
- How effective is the enterprise cybersecurity in protecting against cyberattacks ?
- The graphic shows how the raw measurement data is input to an appropriate OM index equation to calculate the example expert judgment indices.
- RiskMitigations Index = 0.55
- FunctionalArea Index = 0.70
- SecurityOps Index = 0.57
- Also, the three indices above can be aggregated into an overall summary index.
- CybersecurityProgramAssessment Index = 0.64
- Appropriate people in the enterprise should meet to examine the measurements and discuss how to address the corresponding results.
- To support the discussion, the measurements need to be expressed in everyday terms that make sense to the enterprise so that leadership can make informed decisions.
Visualizing Cybersecurity Assessment Scores
Cybersecurity assessment measurements can be visualized in a number of ways, but enterprises need to decide what makes the best sense for their organizational culture.
- The graphic below shows how enterprise risk mitigations can be considered in terms of two expert adjustment value scales regarding risk probability and risk impact reduction.
- The graphic below shows how enterprise risk mitigations can be visualized numerically when the value scales for probability and impact reduction are combined.
- The graphic below shows what expert value judgments were recorded in terms of risk probability and risk impact reductions (ellipses) and the resulting metrics for the attack sequence steps (solid circles with numbers).
- The graphic below shows the overall combined risk mitigation metric (also known as RiskMitigations Index) with a numeric value of 0.55, which was calculated with the Object Measurement. Index Equation
Cybersecurity Measurement Summary
- Presented fundamental measurement principleMeasurements need to be expressed in everyday terms that are familiar to the enterprise; otherwise, the measurements may, at best, be of little value.
- Demonstrated how to quantify the extent to which an enterprise is defending itself against cyberattacks
- Focused on showing how cybersecurity experts, using their expert judgment, can assess an enterprise’s cybersecurity posture
- Presented example expert judgment value scales for example assessment
- Introduced Measurement Map concept to visualize how value scales can be combined to help communicate how the assessment is defined
- Used example calculations to show how a collection of expert judgments can be combined into a single number called an index that gives enterprise management the means to chart a corrective action course to improve its cybersecurity posture
- Presented alternative visualizations of raw measurement data, combined expert measurement data, and overall combined metric