Scott E. Donaldson1 , Stanley G. Siegel2, Chris K. Williams3 and Abdul Aslam3
(1)
Falls Church, Virginia, USA
(2)
Potomac, Maryland, USA
(3)
San Diego, California, USA
Fundamental Principles
An enterprise wants to protect itself from cybersecurity attacks that are constantly morphing.
Consequently, successfulenterprise cybersecurity is a continual improvement exercise designed to address the evolving cyberthreats.
Measurement is a means for effecting enterprise cybersecurity improvement.
People often think of cybersecurity
from a single perspective such as a manager, technologist, or cybersecurity expert;
in terms of a function such as systems administration, network security, or data protection and cryptography; or
in terms of a capability such as network isolation, network traffic analysis, or digital certificates.
However, measuring enterprise cybersecurity effectiveness involves multiple dimensions.
Mathematical and scientific disciplines often handle multidimensional quantities with entities known as vectors.
Physics uses vectors to describe many quantities such as displacement, velocity, and acceleration.
To illustrate from the list of physics quantities, the change in position of a particle is called a displacement.
When we go to work in the morning, we displace ourselves from our home to our place of work.
This displacement can be represented as an arrow on a map drawn from home to work.
The graphic depicts this displacement concept in one, two, and n dimensions.
Simply stated, the calculated vector length combines multiple dimensions into a single quantity or index.
Object Measurement (OM) uses this notion of an index to measure enterprise cybersecurity effectiveness.
The left-hand side of the graphic depicts how OM uses the notion of a vector to derive an overall index.
The right-hand side depicts a corresponding example, cybersecurity effectiveness index (CSEIndex)
, based on three cybersecurity functional areas.
It is acknowledged the dimensions chosen to fold into this example cybersecurity effectiveness index are not necessarily the same dimensions an enterprise may use.
There is no one way to measure cybersecurity effectiveness, but there are fundamental principles whose application can increase the likelihood that enterprise cybersecurity programs will be successful.
As described in the rest of this appendix, OM quantifies almost any object (such as enterprise cybersecurity functional area or capability) in terms of value scales that help tie measurement to familiar enterprise language.
The graphic notionally depicts OM combining multiple value scale measurements into an overall OMIndex score (in other words, an overall index).
Although the OMIndex, like the Consumer Price Index, folds a number of individual measurements into a single quantity, the index can be “unfolded” to gain insight into the underlying measurements.
Even though OM can measure almost anything, OM is not a measurement silver bullet.
Topics
OMIndex Equation
OM Steps
OM Value Scales
OM Measurement Map
Expert Judgment OM Example
Observed Data OM Example
Other Cybersecurity-Related Measurements
OMIndexEquation
General Equation
The graphic depicts the general OMIndex Equation
, where vector dimensions are expressed in terms of object attributes.
Each attribute can be weighted, and there are no mathematical limits to the number of attributes.
However, keep in mind that enterprise measurement programs will fail if they are too onerous.
Note that the denominator is set up
to normalize the OMIndex.
In other words, restrict the OMIndex range from zero to one.
Removing the denominator eliminates this normalization.
Example Equations
The graphic depicts three examples of
how the OMIndex equation can be used.
Example 1 represents the case in which an object is characterized by five attributes.
Example 2 represents the case in which the first attribute is considered twice as important as the other attributes.
Example 3 represents the case in which the second and third attributes are suppressed.
The OMIndex Equation
provides an enterprise with a generalized measurement methodology that can be tailored to specific enterprise measurement requirements.
OM Steps
OM includes the following steps:
Step 1: Define the questions the enterprise wants to answer.
Step 2: Select appropriate object(s) to measure for collecting relevant data to answer the defined enterprise questions.
Step 3: For each object, define the object characteristics to measure.
Step 4: For each characteristic, create a value scale with tick marks and corresponding tick-mark descriptions in plain, unambiguous language.
Step 5: Measure each characteristic by (a) using expert judgment to form an opinion and matching the opinion with the appropriate value scale(s) and tick-mark value(s), or (b) matching the observed data with the appropriate value scale(s) and tick-mark value(s).
Step 6: Substitute the selected tick-mark numeric values into an appropriate OM equation to calculate the overall index.
OM Value Scales
Fundamental Principles
Value scales help associate an enterprise vocabulary (that is, language) with measurement.
The challenge is to establish value scales to make meaningful measurements.
For this study guide, meaningful means “the enterprise uses the measurements to determine whether and where cybersecurity needs to be improved.”
OM value scale types include the following:
Discrete
Binary
Sliding
Value scales have minimum and maximum numeric values, along with plain language description on the tick-mark labels.
The numeric range of values is not restricted to zero (0.00) to one (1.00) and can accommodate any numeric range.
The enterprise decides what terms define its value scales.
Discretevaluescales allow for distinct interim numeric values and corresponding tick-mark labels (for example, 0.00 = Absent; 0.25 = Weak, and so on).
Binaryvaluescales are often used to measure on/off or yes/no or desired behavior/lack of desired behavior.
Slidingvaluescales measure a minimum numeric value, a partial numeric value based on a ratio, and a maximum numeric value.
Value scale tick-mark labels need to be defined in everyday enterprise language to aid in communicating measurement results.
There is no one set of terms (that is, numeric values and tick-mark labels) that define value scales.
Example Expert Judgment
Value Scales
Experts have their own experience-based language to describe their area of expertise to non-experts.
Such language often embodies their educated guesses or intuitive judgment.
The graphic depicts a value scale defined in expert judgment language for any cybersecurity functional area.
Expert judgment language may be somewhat “squishy,” but it enables the expert to designate a particular value scale tick mark and its corresponding value as appropriate for the situation.
For example:
If an expert thinks a functional area is poorly supported and has a very low level of maturity, then the expert would designate 0.25 as the appropriate value for the situation.
Similarly, the graphic depicts an example of an expert measuring any cybersecurity functional area capability.
For example:
If an expert thinks a functional area capability is present, but with only limited utilization or major issues with its design or operation that sharply limit its effectiveness, then the expert would designate 0.50 as the appropriate value for the situation.
The expert judgment scales described here are not set in stone and may be somewhat squishy.
Example expert value scales are provided as starting points for consideration.
Each enterprise needs to create its own meaningfulvalue scales.
Example Observed Data Value Scales
Observed data
value scales are similar in structure to expert judgment value scales.
Minimum value
Maximum value
Tick mark labels
However, the tick-mark labels represent observable events, also known as “measurement triggers.”
Each tick-mark label can be observed as opposed to the expert judgment value scale tick-mark labels.
The graphic depicts an observed data value scale for a specific capability, Virtualization and Storage Area Network Management.
As with expert judgment value scales, there is no one set of terms that defines observed data value scales.
OM Measurement Map
Basic Structure
To help define and organize value scales for an object to be measured, it is convenient to create an OM measurement map.
The graphic depicts two generic measurement maps that can be used in concert with the OM Six-Step Methodology.
The upper half of the graphic depicts Object A in terms of a number of characteristics and value scales.
The lower half of the graphic depicts Object B in terms of a number of characteristics, sub-characteristics, and value scales.
Measurement maps define value scales at the lowest level (far right-hand side of map).
A measurement map helps define objects in unambiguous terms and represents, in part, the
scope of what is to be measured.
The defined objects, via value scales, provide a consistent measurement vocabulary.
Example Enterprise Cybersecurity Program Assessment
Measurement Map
The graphic depicts an example measurement map established for conducting an enterprise cybersecurity program assessment.
Assessment is structured and scoped, in part, by the following components:
Risk mitigations associated with a cyberattack sequence
Functional areas defined in terms of enterprise cybersecurity capabilities
Security operations associated with enterprise day-to-day security activities
Expert Judgment OM Example
OM Six-Step Methodology
Step 1: Define the questions the enterprise wants to answer.
How effective is the current enterprise security posture?
Step 2: Select appropriate object(s) to measure for collecting relevant data to answer the defined enterprise questions.
Enterprise Cybersecurity Effectiveness depends on the selection of appropriate objects to defend the enterprise against cyberattacks.
Step 3: For each object, define the object characteristics to measure.
Enterprise cybersecurityeffectivenesshas the following 11 functional areas that will be the characteristics for measurement.
Step 4: For each characteristic, create
a value scale with tick marks and corresponding tick-mark descriptions in plain, unambiguous language.
Use the cybersecurity functional area value scale below to define expert judgment value scales for the 11 cybersecurity functional areas.
The graphic shows expert judgment value scales.
Excellent: FA1 has most of the capabilities that would be effective against anticipated threats, and those capabilities are very mature and operating properly.
Very Good: FA has numerous
capabilities, and those capabilities have relatively few major issues.
Good: FA is supported, with important capabilities present, but also with issues that hinder the effectiveness of those capabilities.
Poor: FA is poorly supported, and has a very low level of maturity.
Absent: FA has no or very few capabilities present and is ineffective at providing enterprise protection.
Step 5: Measure each characteristic (in other words, 11 enterprise cybersecurity functional areas) by using expert judgment to form an opinion and matching the opinion with the appropriate value scale and tick-mark values.
The graphic showsvaluescales with example expert judgment measurements indicated by circled values.
Step 6: Substitute the selected tick-mark numeric values into an appropriate OM equation to calculate
an overall index.
Once the functional areas have been scored, the measurements can be aggregated together into an Object Measurement Index.
For this example, the upper graphic shows the expert judgment Cybersecurity Effectiveness Index, CSEIndex.
The resulting CSE Index = 0.52, 2which is greater than “good” and less than “very good.”
The lower graphic illustrates one way to visualize CSEIndex = 0.52.
Value Scales can be used to visualize changes in cybersecurity functional area effectiveness over time.
Since measurement is used, in part, to increase cybersecurity effectiveness, assume
the enterprise implemented an improvement program based on the measurement results; and
some time has elapsed
after the original measurements were taken.
New expert judgment measurements (green ellipses and dashed arrows) were recorded after the cybersecurity improvements were implemented for 9 of 11 functional areas.
The OM equation can then be used to calculate an updated overall index to reflect changes in cybersecurity functional area effectiveness over time.
Repeat Step 6: Substitute the
updated selected tick-mark numeric values into an appropriate OM equation to calculate an overall index.
For this example, the upper graphic shows the expert judgment Cybersecurity Effectiveness Index, CSEIndex.
The resulting CSEIndex = 0.79, which is greater than “very good” and less than “excellent.”
The lower graphic illustrates one way to visualize CSEIndex = 0.79.
The top graphic depicts the expert judgment measurement results.
Original (or baseline) measurements, where CSEIndex = 0.52
Updated measurements after improvements, where CSEIndex = 0.79
The bottom graphic depicts cybersecurity assessment results tracked over time and helps to communicate results of specific infrastructure investments.
The OMIndex Equation provides direct linkage between the defined functional area value scales and an expert’s judgment.
This expressed linkage is tied to enterprise cybersecurity improvement activities.
By tracking CSEIndex over time, the enterprise has a means for using expert judgment to guide its ongoing cyberdefense improvement activities.
Observed Data OM Example
OM Six-Step Methodology
The next set of pages describe an observed data example, examining a single functional area and its capabilities
Step 1: Define the questions the enterprise wants to answer.
How effective is the current systems administration functional area?
Step 2: Select appropriate object(s) to measure for collecting relevant data to answer the defined enterprise questions.
Systems Administration effectiveness in defending the enterprise against cyberattacks.
Step 3: For each object, define the object’s characteristics.
This step is different from the expert judgment measurement example as nine individual Systems Administration capabilities are to bemeasuredvs. the eleven enterprise cybersecurity functional areas.
Step 4: For each characteristic, create a value scale with tick marks and corresponding tick-mark
descriptions in plain, unambiguous language.
The graphic shows observed data cybersecurity capability value scales for Systems Administration.
Step 5: Measure each characteristic (in other words, nine Systems Administration capabilities) by matching the observed data with the appropriate
systems administration capability value scales and tick-mark values.
The graphic shows value scales with example observed data measurements indicated by circled values.
Step 6: Substitute the selected tick-mark numeric values into an appropriate OM equation to calculate
an overall index.
Once the Systems Administration capabilities have been scored, the measurements can be aggregated together into an Object Measurement Index.
For this example, the upper graphic shows the observed data Systems AdministrationEffectivenessIndex, SACSEIndex.
The resulting SACSEIndex = 0.81.
The lower graphic illustrates one way to visualize SACSEIndex = 0.81.
What does 0.81 mean?
Systems Administration Cybersecurity Effectiveness is exactly what was observed (in other words, the observed data) as follows:
Enterprise uses Bastion Hostcomputersalong with other protection methods such as strong authentication.
Enterprise uses a secure, dedicated channel to manage critical systems during an outage.
Enterprise conducts systems administration on networks isolated from business traffic.
Enterprise manages some enterprise servers using centralized KVM, ILO, and Power Controls.
Enterprise isolates and protects either Virtualization or SAN Management.
Enterprise uses a separate administrative interface to administer the enterprise IT assets.
Some enterprise Systems Administrators (SAs) use multifactor authentication to access enterprise resources.
SAs activities are logged and
audited periodically, and logs are under SA control.
Enterprise logs commands/keystrokes and analyzes the logs periodically.
Value Scales can be used to visualize changes in cybersecurity capability effectiveness over time.
Since measurement is used, in part, to increase cybersecurity effectiveness, assume
the enterprise implemented an improvement program based on the measurement results; and
some time has elapsed
after the original measurements were taken.
New observed data measurements (green ellipses and dashed arrows) were recorded after the cybersecurity improvements were implemented for 2 of 9 capabilities.
The OM equation can then be used to calculate an updated overall Systems Administration index to reflect changes in capability effectiveness over time.
Repeat Step 6: Substitute the updated
selected tick-mark numeric values into an appropriate OM equation to calculate an overall index.
For this example, the upper graphic shows the observed data Systems Administration Cybersecurity Effectiveness Index, SACSEIndex.
The resulting SACSEIndex = 0.91.
The lower Graphic illustrates one way to visualize SACSEIndex = 0.91.
The top graphic depicts the observed
data measurement results.
Original (or baseline) measurements, where SACSEIndex = 0.81
Updated measurements after improvements, where SACSEIndex = 0.91
The bottom graphic depicts cybersecurity assessment results tracked over time and helps to communicate results of specific infrastructure investments.
The OMIndex Equation provides direct linkage between the defined cybersecurity capability value scales and observed data.
This expressed linkage is tied to enterprise cybersecurity improvement activities.
By tracking SACSEIndex over time, the
enterprise has a means for using observed data to guide its ongoing cyberdefense improvement activities.
Other Cybersecurity-Related Measurements
Two-Step Measurement Approach
In addition to OM-based cybersecurity measurement,
it may be useful for an enterprise to establish other cybersecurity program measurements.
What attributes of the cybersecurity program are of interest to measure?
Which activities contribute to successfully securing the enterprise from cyberattacks?
An effective enterprise cybersecurity program protects the enterprise in a cost-effective manner that balances
technology, process, people, budgets, and external compliance requirements, while supporting the business mission as much as possible.
This section presents a high-level, two-step measurement approach that can be used to effect cyberdefense improvement.
Step 1: The application of metrics to cyberdefense activities to provide insight into the extent to which these activities are, or are not, contributing to effective cyberdefense.
Step 2: Those activities that are not contributing to effective cyberdefense will be modified (or eliminated) until they do. These modification are what cyberdefense improvement means.
Improvements are measured individually and then averaged to provide insight into what cyberdefense areas have improved or not improved.
Measurement, in part, involves collecting data and putting it into a meaningful form for cyberdefense improvement purposes.
Such activities should not be onerous because they will get in the way of the cyberdefense program.
Metrics need to be simple to collect and analyze.
However, simplicity can cause the metrics to be limited regarding the insight they provide into cyberdefense workings.
For the near term, the enterprise should collect some simple metrics to see if they help highlight activities that should be changed to effect cyberdefense improvement.
More sophisticated cybersecurity measurements can be added if needed.
This section presents example cybersecurity measurements for the first three security operational processes in the graphic below.
Policies and Policy Exception Management
The top graphic depicts the Policies and Policy Exception Management process that maintains enterprise policies, as well as exceptions to those policies.
Enterprises may be good at establishing cybersecurity policies and maintaining them, but managing exceptions tends to be more problematic.
Policy exceptions need to be formally approved and then re-certified on a regular basis.
Security needs to observe policy exceptions to watch out for cases where the “exception becomes the rule.”
The bottom graphic depicts two example metrics that provide a quantitative means for assessing the extent to which enterprise cybersecurity policies are integrated into the enterprise business culture.
Project and Change Security Reviews
The top graphic depicts the Project and Change Security Reviews process that ensures, in part, that IT systems are designed and deployed with cybersecurity capabilities “baked in” and practical.
Should be integrated into the larger systems development life cycle
Can also be integrated into the management gates of the enterprise IT project and change process
Cybersecurity needs to be considered on major IT initiatives as well as associated initiative risks and mitigations.
The bottom graphic depicts two example metrics that provide a quantitative means for assessing the extent to which cybersecurity is designed and deployed with “baked-in” cybersecurity capabilities.
These measurements provide a visible “yardstick” for
portraying the enterprise’s security posture with respect to project and change security reviews.
Risk Management
The top graphic depicts the Risk Management process that helps the CISO track risks in terms of the risks’ business impact, not their technological impact.
Risks should be technology-agnostic.
Technology factors into the risk process as vulnerabilities are identified and exploited by attackers.
Consequently, associated business risks can increase and possibly require additional mitigations.
The bottom graphic depicts three example metrics that provide a quantitative means for assessing the extent to which identified, analyzed, and tracked risks are mitigated over time.
These Risk Management measurements offer the enterprise a means for improving cybersecurity functional areas that may be falling short in dealing effectively with risks.
Enterprise may find for a given reporting period there have been successful cyberattack incidents linked to the e-mails of one or more enterprise users.
Possible starting point for mitigating the risk of e-mail security breaches may point to a shortfall in the E-mail Security Capability of the Application Security Functional Area.
The mitigation process may trigger an upgrade to the enterprise risk management process for the E-mail Security Capability.
Security team designs better controls for the E-mail Security Capability.
Business leadership approves the risk mitigation plan for the improved controls.
Engineering team implements the improved controls.
Operations team maintains the operation of the improved controls.
Security team tracks the extent to which the E-mail Security breaches may have been mitigated.
This tracking would show up in the updates to the risk management metrics in subsequent reporting periods.
Due to a rounding error in the companion Enterprise Cybersecurity book, this CSEIndex value of 0.52 is different than the CSEIndex value of 0.48 found on page 394 (Figure F-11) and page 396 (Figure F-14).
