Overview
- The purpose of this appendix is to bring together a previously introduced hierarchy of cybersecurity concepts into three worked-out numerical examples and address the following questions:
- What is the enterprise quantitative effectiveness in defending itself against cyberattacks?
- How does the enterprise quantitatively improve this effectiveness?
- A cybersecurity program lends itself well to performing top-down security assessments at progressively increasing levels of detail.
- Each assessment level can be performed independently.
- Assessments can be done in progressive passes to get increasing detail focusing on the areas of greatest interest.
- Multiple passes help to gain a greater understanding of the enterprise security posture and areas for improvement.
- The graphic depicts a top-down Cybersecurity Program Assessment framework with four levels of focus and detail.
- A Cybersecurity Program Assessment can consist of one or more assessments.
- Level 1 Assessment
- Very high-level assessment that focuses on risk mitigations using expert judgment
- Level 2 Assessment
- High-level assessment that focuses on functional areas using expert judgment
- Extends Level 1 results to consider functional areas and security operations
- Level 3 Assessment
- Detailed assessment that focuses on capabilities using observed data
- Extends Level 1 results to consider functional areas at the capabilities level and security operations considering individual operational processes using expert judgment
- Level 4 Assessment
- More detailed assessment that considers controls, technologies, and/or processes using expert judgment or observed data
Note: Level 4 Assessment is not addressed in this appendix.
- Worked-out example numerical assessments illustrate how an enterprise can obtain answers to questions posed about its cybersecurity effectiveness and its evolution.
Topics
- Sample Assessment Scope and Methodology
- Level 1 Assessment: Focus on Risk Mitigations
- Level 2 Assessment: Focus on Functional Areas
- Level 3 Assessment: Focus on Capabilities
Sample Assessment Scope and Methodology
Overview
- This appendix presents three sample assessments for a hypothetical enterprise.
- Level 1 Assessment focuses on risk mitigations against all steps of the attack sequence.
- Level 2 Assessment focuses on functional areas.
- Level 3 Assessment focuses on security capabilities.
- These three sample assessments
- analyze a single security scope consisting of the enterprise’s general-purpose IT environment
- use the Object Measurement Methodology to identify the attributes for evaluation and score those attributes
- combine the scores into a single Cybersecurity Program Assessment Index that represents the enterprise’s security posture
Level 1 Assessment: Focus on Risk Mitigations
Context
- Level 1 assessment involves analyzing the enterprise cybersecurity attack sequence.
- Assessment security scope
- is enterprise’s general-purpose IT environment; and
- considers how well the cybersecurity controls are reducing the probability and the impact of attacks making it through each step of the attack sequence.
- This assessment is an exploratory assessment providing a high-level evaluation of enterprise’s overall cybersecurity posture.
- Results used to help direct further cybersecurity assessments.
- Since assessment is only looking at risk mitigations, it does not account for other cybersecurity details that could be critically important.
- The primary advantage of this Level 1 Assessment is the enterprise does not have to go into too much detail or spend too much time in analysis to obtain executive-level results.
Object Measurement Steps
- OM Step 1
- What is the strength of the enterprise’s cybersecurity when only considering risk mitigations to reduce the probability or impact of targeted cyberattacks?
- OM Step 2
- Risk mitigations against attack sequence steps
- 1.Establish foothold
- 2.Command and control
- 3.Move laterally
- 4.Escalate privileges
- 5.Complete the mission
- 1.
- OM Step 3
- For each attack sequence step, the object characteristics will be measured using expert judgment as follows:
- Risk probability reduction that represents the amount that attacks are less likely to succeed at the specific step of the attack sequence due to the presence of controls that disrupt, detect, delay, or defeat the attack
- Risk impact reduction that represents the amount that the impact of an attack is reduced at the specific step of the attack sequence due to the presence of controls that disrupt, detect, delay, or defeat the attack
- A cybersecurity expert will use the above two characteristics and use expert judgment to define corresponding value scales and then evaluate the enterprise’s cybersecurity controls’ effectiveness.
- OM Step 4
- The expert defines the value scales for (1) risk probability reduction and (2) risk impact reduction, as shown below.
- These expert judgment value scales provide definitions and values for high, medium, and low with regard to probability reduction and impact reduction
- It is possible for the risk mitigations of a step of the attack sequence to
- reduce the probability of attack but not the impact; and
- reduce the impact but not the probability of attack.
- Once the expert judgment value scales are defined, their values can be combined using an appropriate OM equation.
- However, the graphic shows an alternative way to combine their values via an OM Scoring Matrix .
- The matrix combines the value scales in a way that is simpler than using the full OM equation, but produces similar results.
- The matrix defines a combined expert judgment value scale for an assessment.
- Inputs to the OM Scoring Matrix are the risk probability reduction and risk impact reduction expert measurements.
- Matrix converts the expert judgment measurements into a single combined measurement.
- An OM Measurement Pane can be created to help record assessment measurements.
- Level 1 expert judgment value scales are simplified here for display purposes only.
- The graphic shows how the two expert judgment value scales are networked to the attack sequence steps that are related to risk mitigations.
- Using the OM Scoring Matrix, the graphic shows possible values for the (1) two expert judgment value scales and (2) individual attack sequence steps when the two value scales are combined.
- OM Step 5: Measure each object characteristic using a value scale .
- Each attack sequence step value is tied to a definition expressed in enterprise cybersecurity language that helps explain the underlying meaning of the numbers and preserves, in part, the context of this assessment.
- The graphic shows Level 1 expert judgment value scales defined and used for assessment.
- Expert judgment risk mitigation values for each of the five attack sequence steps and their combined measurement based on OM Scoring Matrix are shown in the graphic below.
- OM Step 6: Calculate the overall Cybersecurity Program Assessment Index.
- The graphic to the right shows
- recorded expert value judgments in terms of risk probability and impact reductions (see ellipses);
- the resulting attack sequence step metrics based on the OM Scoring Matrix (see solid circles with numbers); and
- how the overall Level1 _Index is calculated with an appropriate OM index equation as shown in the graphic below.
What does a Level1_Index = 0.47 or 47% really mean?
- The result represents the enterprise’s cybersecurity effectiveness as determined from the expert judgment terms used to define the expert judgment value scales.
- By navigating the measurement map from left to right, the Level1_Index score can be unfolded and traced back to the expert judgment value scales.
- For example, Escalate Privileges = 0.50 means
- Risk Probability Reduction = 0.50
- Medium probability reduction = For an attack to succeed, it must overcome two or more security protections, and breaches of at least one protection will be detected before the attack is complete.
- Risk Impact Reduction = 0.50
- Medium impact reduction = Successful attacks are detected and the impact and cost of the risk are reduced some, but not significantly.
Level 2 Assessment: Focus on Functional Areas
Context
- Level 2 assessment involves analyzing
- Risk Mitigations,
- Cybersecurity Functional Areas, and
- Security Operations.
- Assessment security scope
- Consists of enterprise’s general-purpose IT environment
- Assessment considers
- how well cybersecurity controls are reducing the probability and impact of cybersecurity attacks;
- how effectively the functional areas are reducing potential security gaps and security issues; and
- how comprehensively the security operations are utilized.
- Assessment
- reflects expert judgment evaluating each functional area in its entirety;
- does not account for other cybersecurity details that could be critically important:
- cybersecurity capabilities,
- processes, and
- technologies;
- Might be performed via a telephone interview or at a leadership whiteboard session; and
- has the purpose of getting quickly to a high-level impression of the enterprise’s cybersecurity posture.
- Results can be used to support or validate strategic decisions about which cybersecurity areas need focus and which areas do not.
Object Measurement Steps
- OM Step 1
- What is the strength of the enterprise’s cybersecurity when considering a high-level evaluation of risk mitigations, cybersecurity functional areas, and a high-level assessment of security operations?
- OM Step 2
- Select the appropriate objects to measure.
- 1.Risk mitigations against attack sequence steps with regard the Level 1 Assessment
- 2.Cybersecurity functional areas
- 3.Security operations
- 1.
- OM Step 3
- For risk mitigations, the Level1_Index is to be simply carried over from the previous example assessment.
- Cybersecurity functional areas and security operations will be measured using expert judgment as follows:
- Effectiveness represents the amount the functional area capabilities or operational processes are present and properly configured to support the applicable cybersecurity scope. The functional area is effective if the capabilities or operational processes it contains are mature and working properly, even if there are only a few of them.
- Comprehensiveness represents the amount the functional area capabilities or operational processes are being actively used to support the applicable cybersecurity scope. The functional area is comprehensive if it contains many of the capabilities and operational processes, even if they are not mature or working properly.
- OM Step 4
- The expert defines the value scales for (1) effectiveness and (2) comprehensiveness, as shown below.
- These expert judgment value scales provide definitions and values for high, medium, and low with regard to effectiveness and comprehensiveness.
- It is possible for a functional area to be
- effective but not comprehensive; and
- comprehensive but not effective.
- Once the expert judgment value scales are defined, their values can be combined using an appropriate OM equation.
- However, the graphic shows an alternative way to combine their values via an OM Scoring Matrix .
- The matrix combines the value scales in a way that is simpler than using the full OM equation, but produces similar results.
- The matrix defines a combined expert judgment value scale for an assessment.
- Inputs to the OM Scoring Matrix are the comprehensiveness and effectiveness expert measurements.
- The matrix converts the expert judgment measurements into a single combined measurement .
- An OM Measurement Pane can be created to help record assessment measurements.
- The graphic shows measurement panes for 11 functional areas, and one for security operations (lower-right).
- Level 2 expert judgment value scales are simplified here for display purposes only.
- The graphic shows how the two expert judgment value scales are networked to individual cybersecurity functional areas and overall security operations.
- Using the OM Scoring Matrix, the graphic shows possible values for the two expert judgment value scales. Also shown are possible combined values for (1) individual cybersecurity functional areas and (2) overall security operations.
- OM Step 5: Measure each object characteristic using a value scale.
- Each value is tied to a definition expressed in enterprise cybersecurity language that helps explain the underlying meaning of the numbers and preserves, in part, the context of the assessment.
- The top graphic shows Level 2 expert judgment value scales defined and used for this assessment.
- The bottom graphic shows expert judgment values for risk mitigations, functional areas, and overall security operations.
- The Risk Mitigations value is carried over from Level 1 Assessment.
- Functional Area Values are expressed in terms of effectiveness, comprehensiveness, and combined measurements.
- Security Operations value is expressed in terms of effectiveness, comprehensiveness, and combined measurements.
- OM Step 6: Calculate the overall Cybersecurity Program Assessment Index.
- The graphic shows
- recorded expert value judgments in terms of effectiveness and comprehensiveness (see ellipses);
- resulting functional area metrics and overall security operations metric based on the OM Scoring Matrix (see solid circles with numbers); and
- how the overall Level2 _Index is calculated with an appropriate OMIndex equation, as shown below.
What does a Level2_Index = 0.53 or 53% really mean?
- The result represents the enterprise’s cybersecurity effectiveness as determined from the expert judgment terms used to define the expert judgment value scales.
- By navigating the measurement map from left to right, the Level2_Index score can be unfolded and traced back to the expert judgment value scales.
- For example, Incident Response = 0.75 means the following:
- Effectiveness = 1.0
- High Effectiveness = The functional area is effective and has few capability gaps.
- Comprehensiveness = 0.50
- Medium Effectiveness = The functional area is used in many of the places where it could be used (in other words, 25% ≤ potential utilization ≤ 75%).
Results Visualization and Analysis
- The upper graphic visualizes assessment results using a vertical bar (column) chart.
- Functional areas can be quickly compared to one another.
- Weak functional areas can be identified for subsequent improvement.
- Values are shown on a scale from 0% to 100% while previous values were shown on a scale from 0 to 1.0; values are the same.
- The bottom graphic visualizes the results using a spider chart (also known as a radar chart).
- Useful for emphasizing that all the functional areas should be of approximately equal effectiveness (that is, the target value) for overall enterprise cybersecurity to be effective
- Three functional areas scoring the lowest stand out compared to the rest.
- Systems Administration
- Data Protection and Cryptography
- Monitoring, Vulnerability, and Patch Management
- Weak functional areas may be exploitable and should be prioritized for investment and improvement.
- Both graphics show a target value at 75% (0.75).
- Represent the enterprise’s cybersecurity goal for the example security scope
- Make it easy to see at a glance where deficiencies lie
- Help leadership to understand the magnitude of the gap between what is present and what is desired
- Target value does not necessarily need to be 100% (1.0).
- Business leadership sets the target bar to an acceptable level to balance
- competing interests of business expediency and cybersecurity protection; and
- controlling security costs and the impacts security can have on productivity.
Level 3 Assessment: Focus on Capabilities
Context
- Level 3 assessment involves analyzing
- risk mitigations,
- functional areas, and
- security operations.
- Assessment security scope
- consists of enterprise’s general-purpose IT environment.
- Assessment
- incorporates the expert judgment results of the Level 1 Risk Mitigations Assessment of the five-step enterprise attack sequence;
- uses observed data to assess the 113 enterprise cybersecurity functional area capabilities;
- uses expert judgment to evaluate the 17 operational processes; and
- uses object measurement equation to calculate aggregate scores for each functional area and security operations.
- The scores for Risk Mitigation, Functional Areas, and Security Operations are combined together to get a single Level3_Index value for the entire enterprise security scope.
- includes a detailed analysis of the enterprise’s cybersecurity capabilities and operational processes;
- uses observed data rather than expert data for most of its data gathering;
- is significantly more detailed than Level 1 and Level 2 Assessments; and
- requires a commensurately greater level of effort to perform.
- When performed by an external assessor, it generally requires onsite visits and interviews with IT leadership involved with each of the functional areas and their capabilities.
- Once completed, it delivers a detailed view of the enterprise’s cybersecurity posture that can then be used to support detailed strategic, programmatic, and tactical decision-making.
Object Measurement Steps
- OM Step 1
- What is the strength of the enterprise’s cybersecurity when considering a detailed evaluation of risk mitigations, 113 functional area cybersecurity capabilities, and 17 security operational processes?
- OM Step 2
- Select the appropriate objects to measure.
- Risk mitigations : 5 attack sequence steps
- Functional area capabilities : 113 capabilities in the 11 functional areas
- Operational processes : 17 processes and supporting information systems
- OM Step 3
- Risk mitigations will be evaluated using expert judgment based on their effectiveness in reducing the probability and the impact of the attack sequence.
- The 113 functional area capabilities will be objectively evaluated using observed data .
- The 17 operational processes will be evaluated using expert judgment as follows:
- Maturity represents how mature the operational process is in terms of how consistently it is performed, the quality of the data it maintains, and its ability to handle exceptions and special cases. A mature operational process is robust and reliable, uses appropriate supporting tools and technologies, and is consistently performed using documented procedures.
- Utilization represents how comprehensively the enterprise utilizes the operational process across the assessed scope. A well-utilized process is used everywhere it should be used with safeguards in place to ensure it cannot be bypassed, ignored, or arbitrarily exempted.
- It is possible for an operational process to be mature but not utilized, or well utilized but immature.
- OM Step 4: For each characteristic, create a value scale.
- Risk mitigations values will be carried over from the Level 1 Assessment along with their associated expert judgment value scales.
- The 113 capabilities of the 11 functional areas will be assessed using the observed data value scales that are defined in Appendix E of this study guide.
- The 17 operational processes will be evaluated using expert judgment maturity and utilization value scales.
- The expert defines the value scales for (1) maturity and (2) utilization in terms of high, medium, and low values, as shown below.
- Once the expert judgment value scales are defined, their values can be combined using an appropriate OM equation.
- However, the graphic shows an alternative way to combine expert measurement values via an OM Scoring Matrix .
- The matrix combines the value scales in a way that is simpler than using the full OM equation, but produces similar results.
- The matrix defines a combined expert judgment value scale for an assessment.
- Inputs to the OM Scoring Matrix are the utilization and maturity expert measurements.
- The matrix converts the expert judgment measurements into a single combined measurement .
- An OM Measurement Pane can be created to help record assessment measurements.
- The graphic presents the Systems Administration measurement pane.
- Similar measurement panes can be created for the other functional areas, but they are not shown here.
- As assessment complexity increases, it is helpful to have measurement panes.
- Help explain the underlying meaning of the assessment because each “tick mark” is defined by enterprise language
- Help organize possible measurements
- Help with recording measurement
- Help preserve, in part, the context of the assessment
- The graphic presents the Security Operations measurement pane.
- The graphic shows the overall Level 3 Assessment: Focus on Capabilities Measurement Map .
- Using the OM Scoring Matrix, the graphic shows possible values for the two expert judgment value scales regarding Security Operations. Also shown are possible combined values for individual security operations processes.
- OM Step 5: Measure each object characteristic using a value scale.
- Involves assessing the 113 capabilities by functional area, along with the 17 operational processes
- Data collection for the capabilities uses observed data based on the value scales in Appendix E of this study guide.
- Data collection for the security operational processes uses expert judgment based on the utilization and maturity values scales.
- The graphic shows the systems administration measurement pane with recorded observed data measurement indicated by circled values.
- Similar measurement panes can be used to record observed data measurements for the other functional areas and capabilities, but such measurement panes will not be shown here.
- However, the following slides present the observed data values for the 113 capabilities of the 11 functional areas in a tabular form.
OM Step 5: Measure each object characteristic using a value scale
OM Step 5: Measure each object characteristic using a value scales
- OM Step 5: Measure each object characteristic using a value scales
- The graphic presents sample measurements for the Security Operations 17 operational processes using expert judgment value scales.
- The graphic presents summaries of the Security Operations compiled from the maturity and utilization expert judgment value scales.
- OM Step 6: Calculate the overall cybersecurity Program Assessment Index .
- The graphic presents summaries of the program assessment measurements for Risk Mitigations, 11 Functional Areas, and Security Operations.
What does a Level3_Index = 0.46 or 46% really mean?
- The result represents the enterprise’s cybersecurity effectiveness determined from
- values assigned to the observations; and
- expert judgments that were recorded.
- By navigating the measurement map from left to right, the Level3_Index score can be unfolded and traced back to those original observations and expert judgments.
- Risk mitigations can be traced back to the Level 1 Assessment expert judgment value scales regarding risk probability reduction and risk impact reduction.
- Functional areas can be traced back to the cybersecurity capability observed data value scales (for example, Systems Administration: Bastion hosts, Out-of-Band (OOB) management , Network isolation , and so on).
- Security operations can be traced back to expert judgment value scales regarding maturity and utilization .
Results Visualization and Analysis
- The upper graphic visualizes assessment results using a vertical bar (column) chart from 0% to 100%.
- Functional areas can be quickly compared to one another.
- Weak functional areas can be identified for subsequent improvement.
- The bottom graphic visualizes the results using a spider chart (also known as a radar chart).
- Useful for emphasizing that all the functional areas should be of approximately equal effectiveness (that is, the target value) for overall enterprise cybersecurity to be effective
- Level 3 graphics
- Capture security scope’s overall cybersecurity effectiveness to a higher level of precision than Level 2 Assessment due to more detailed underlying observations
- Level 3 Assessment
- Captures the status of the functional areas in greater granularity
- Provides clarity on which functional areas are the weakest when compared to the others, or to the scope’s cybersecurity target value
Comparing Cybersecurity Assessment Results
- OMIndex is a powerful tool.
- Assesses and measures an enterprise’s cybersecurity posture
- Models potential effects of cybersecurity projects or improvements
- Different assessment levels.
- use different values scales to rate the characteristics of the enterprise’s cybersecurity;
- apply those scales using the level of granularity appropriate to the assessment; and
- produce results that are not entirely analogous, but use a common rating scale.
- The lower graphic shows the OMIndices calculated from the three sample assessments.
- Scores are similar but not identical.
- All scores represent the evaluated scope’s overall security posture based on different sets of source data.
- Assessments that are to be compared to one another should be performed at the same level of assessment so the evaluated criteria and value scales are all the same.
Using Cybersecurity Assessment Results
- Enterprise value scale language
- helps explain the underlying meaning of the OM numbers; and
- preserves, in part, the context of each assessment.
- Future assessments may not involve the same cybersecurity expert(s).
- New experts can use the same value scales for their assessments.
- This helps ensure past and future results can be objectively compared to one another.
- It allows an enterprise to continuously track its cybersecurity posture over time.
- Simply stated, the OM Index is much more than a number … it quantitatively represents the entire assessment and its objective.
- Once the enterprise has an assessed security scope,
- cybersecurity improvements can be modeled; and
- what-if analyses and comparisons can be performed.
- Modeling involves
- calculating the OMIndex for the baseline security scope;
- adjusting the input values to reflect desired changes to the risk mitigation measures, cybersecurity capabilities, or operational processes; and
- the OMIndex then being recalculated as if the improvements are already completed and the effect on the index being observed.
- Bang for the Buck modeling
- estimates the project’s value vs. its cost;
- calculates the OMIndex before a cybersecurity improvement project is begun and after it is completed; and
- reflects quantitively the amount the enterprise’s cybersecurity is expected to improve when the project is completed vs. the project’s cost or effort.
- By modeling this calculation for different proposed projects, the enterprise can get an idea of which projects might be the most effective or cost-effective at improving the cybersecurity posture.
- This, in turn, helps business leaders
- focus on those project that deliver the greatest potential results at the lowest costs; and
- make well-informed decisions regarding cybersecurity improvements compared with other business priorities.
