CHAPTER 12
Learning Security: Different Contexts for Security Process Management

I have come a long way from my initial descriptions of how we measure IT security today and why we should try to do it better. The Security Process Management (SPM) Framework is one way of structuring your security metrics efforts, and, if implemented correctly and conscientiously, the framework can seriously improve your ability to understand and protect information assets. But this can also be said of many other frameworks and models for security. The secret is not in the strategy, but in the correct and conscientious implementation of that strategy and then living and tweaking the strategy day in and day out over time. The SPM Framework is my take on how to measure IT security effectively, based on my years of experience, research, and interpretation.

Even if you accept some or all of what I’ve proposed and you decide to employ those elements of IT security metrics within your own organization and environment, your experiences, knowledge, and interpretation will be unique. Your organization will be unique, as will the culture in which you measure security and the resources that you have available to institute a metrics program.

Since the SPM framework requires that you not only embrace metrics and data, but that your organization embraces learning from those metrics and data, you will need to decide how best to adapt measurement and metrics to your unique challenges. Everyone has his or her own way of learning. To make your security metrics powerful and successful, you must determine how to articulate the true value of your data and your findings. It is not enough to describe your security—you have to convince others in the organization to make decisions based on those descriptions and analyses and to incorporate your insights into their own operations.

Organizational Learning

Much academic and industry research has examined the ways that organizations learn and adapt to their changing environments. Some of this research has been conducted in the fields of knowledge management and enterprise collaboration, areas I discussed in the preceding chapter in the context of the Security Improvement Program (SIP). But sometimes research takes these ideas a bit further and looks at how organizations create, share, and use knowledge in novel or innovative ways.

I am always interested in research that moves from mechanics and technologies into the ways that organizations function as systems and even begin to look less like organizations and more like organisms in the way they operate. When you dissect something, you lose some perspectives in order to gain others. If you ever dissected a frog in biology class, you know that identifying internal organs is very different from experiencing the hopping, swimming, croaking animal in a pond. But which is the real frog? A similar question can be asked of an e-mail or of an organization: Is an e-mail just the bits and packets involved, or does it include the words, meaning, and intent? Is the company the collection of machines, people, and buildings that make up the individual parts, or is it the entity that grows, competes, and succeeds? When a security incident occurs, is it just a machine that was compromised or an individual who was responsible, or did the company itself get hacked?

Attempts to build a learning organization are often concerned with moving knowledge and awareness from the individual to the group and back, and using the results to support better decisions. When a disconnect occurs between individual and group understanding, all sorts of problems can crop up. We are all aware of situations in which common sense for the company is completely at odds with the common sense of individual employees. (These are the sorts of conflicts that have made Scott Adams, the creator of the “Dilbert” comic strip, a wealthy man.) Overcoming these tensions and building an adaptable balance between listening to the individual and dictating to the individual is an important characteristic of a true learning organization, however, it may be accomplished. Organizational learning can be viewed along this continuum, moving from a focus on how people gain new knowledge and putting it to use as individuals doing a job, up through the ways that an enterprise gains and uses knowledge and makes itself into more than just a collection of individual skills and experiences.

There is no one sure way that an enterprise can make itself into a learning organization. This chapter offers a few different perspectives on how companies learn and make sense of the world and how this can apply to IT security measurement and managing the security process for continuous improvement. As I said earlier, we all learn differently. Organizations do, too, and each organization faces its own internal and external contexts in which it must make sense of security-related information. Thinking about how your organization learns and adapts, and implementing the SPM Framework within an appropriate context, can mean the difference between successful measurement and failed metrics.

Three Learning Styles for IT Security Metrics

The following examples offer three views of organizational learning, built around existing tools and concepts that can support IT security. These examples are deliberately general, and most organizations would be able to use a combination of styles and approaches to meet their needs. But they do serve to illustrate how differences in a company culture might need to be considered to achieve the sort of continuous process improvement of the SPM Framework.

As the framework is implemented, security measurement projects (SMPs) conducted, and the results analyzed and interpreted, you will need to understand how those results will be put to work in the larger context of enterprise-wide security. Will it be more important to assign and tightly control the metrics, perhaps because your company is heavily regulated or exists within a very conservative or competitive environment? Or does your business operate in a world that values rapid adaptability and pushes more authority and autonomy down the organizational chart to ensure maximum agility? These, too, are important metrics to explore and questions to answer as you decide how you plan to measure and improve your security.

Standardized Testing: Measurement in ISO/IEC 27004

In late 2009, the International Standards Organization and the International Electro-technical Commission (IEC) published a new international standard for building a security metrics program. ISO/IEC 27004, “Information technology—Security techniques—Information security management—Measurement,” is designed to complement ISO/IEC 27001, the standard for setting up an information security management system (ISMS) within an organization. ISO/IEC 27004 describes a set of best practices for measuring the results of the ISMS, a requirement under ISO/IEC 27001. Since 27001 is the only certifiable standard in the family, meaning the only one that you can actually audit against, the rest of the 27000 standards are closely integrated into the certification requirements, although they can also be used for general best-practices guidance.

The thing about standards is that, by definition, they have to apply in the same way to everyone. So they tend to be very structured approaches to achieving outcomes. In a standards-based learning organization, progress is defined by measuring the same things over and over again and seeing if they improve. You might call it “no company left behind,” and it inherits all the benefits and the baggage of standardized testing in public schools. On the positive side, results of standardized testing (audits, in the case of industry standards) provide a set of reliable, repeatable data against a clearly defined baseline of performance. You either pass or you don’t. Success is meant to be unambiguous. Of course, on the negative side, you have all the problems that come with the pressure to conform to something that may be seen as a least common denominator or that may poorly reflect reality. You also encounter the business equivalent of “teaching to the test” as organizations worry more about passing the audit and less about actual quality or improvement.

In the case of the ISO/IEC 27000 standards, the standards bodies recognized that every organization was unique and that the standard could not dictate every detail of IT security to those adopting ISO/IEC 27001. So the 27000 standards don’t tell you how to do everything, but what they do tell you to do must be done in a very specific way. Certain activities are required, such as conducting risk assessments and periodic formal management reviews of the security program, as are certain key documents regarding the ISMS. The standard also requires that a compliant organization formally define metrics and a measurement process for the ISMS, although 27001 does not specify how to do this.

ISO 27004 does specify how to set up a measurement program for the ISMS, including the objectives, models, and criteria for success that such a program should contain. The standard defines how measurements should be constructed, how data should be collected and analyzed, and how the measurement program should be documented and integrated into the ISMS. The standard is very structured and quantitatively focused, and the measurement criteria that it recommends is designed to keep metrics and measurement results simple, easy to obtain, and easy to understand. This diagnostic approach to security is good at answering the daily questions of who, what, when, and where, but it is unlikely to provide much insight into the how’s and why’s of your security program.

ISO/IEC 27004 typifies an organizational learning style that prioritizes data over knowledge and defined, repeatable metrics over innovation or exploration. This does not mean that 27004 reflects a poor learning style. Standards are used to put structure around a set of operations such as security or quality, and they accomplish this by normalizing those operational processes against predefined criteria. In this context, measurement is about reinforcing the baseline. Improvement is part of the process, but improvements under these standards tend to be conservative and incremental.

ISO/IEC 27000 encourages control above everything else. An organizational environment in which a 27004-based metrics program is likely to be the most valuable will often be one that realizes the need for strong centralized control and authority. Companies looking to establish structured security operations to improve existing ad hoc or chaotic operations, or companies that function in highly regulated or low-margin industries where a security incident can mean the real difference between success and failure, will care more about making sure things work than experimenting with new ideas.

Perhaps because they reflect the current state of many security organizations today, these types of metrics programs are currently top of mind in industry. Security is viewed as an increasing problem, and the general perception is that security isn’t done very well, even in large and sophisticated enterprise environments where protecting information assets should be a critical operation. Even if their recommendations are not as structured and mechanical as ISO/IEC 27004, most security metrics experts today recommend implementing measurements that deliver easy, repeatable data rather than answer deeper security questions or create theories about why security is the way it is.

Implementing standardized measurement requires top-down management commitment and the ability to implement and maintain controls and processes across the entire corporate structure. For a company that has little or no security measurement capability, simply employing a sustainable metrics program can be innovative and revolutionary in itself.

Finally, there is nothing wrong with taking metrics a step at a time. Too often, when an organization tries to bite off more than it can chew with any project or initiative, it turns out in hindsight that a more modest and achievable approach would not only have been easier but also more productive and valuable.

The School of Life: Basili’s Experience Factory

You might recall that Victor Basili was the creator of the Goal-Question-Metric (GQM) methodology that I adapted to IT security as part of the SPM Framework. While GQM was designed for use in creating more effective metrics, Basili also developed a model for organizational learning called the experience factory. Like GQM, the experience factory concept was first developed to support software quality engineering, but Basili and his research colleagues expanded the concept to apply to a variety of institutional settings.

The purpose of an experience factory is to collect, store, and disseminate all of an organization’s experience, usually regarding a particular topic or activity, as a formal and structured operation. An experience factory exists as a dedicated organization within a larger organization, comprising specialists whose task is to provide a learning infrastructure for everyone else. The experience factory metaphor comes from the idea that the factory takes inputs from all areas of the organization, including the results of measurement projects, information about the company or industry environment, and data from a variety of company performance indicators, as raw materials. These materials are then processed, value is added to them, and they are used to create experience products that can be disseminated and reused throughout the rest of the organization to support strategy. The result is the creation of an enterprise-wide feedback loop that facilitates the development of data that may be specific to an individual or specialized function into knowledge that is usable and provides value to all decision-makers. Experience factory products may include regular reports, information-on-demand capabilities, and internal consulting services to help business units and departments meet their needs.

The experience factory concept is about building capabilities within an organization that are similar to what you might find in vendors such as Forrester, Gartner, or IDC that provide industry analysis and market intelligence. Less security-specific than ISO/IEC 27004 and certainly less prescriptive, the experience factory concept does not rely on a highly structured metrics baseline that is pushed out to the company and audited. The experience factory is about collecting metrics data and insights from many different sources and many different perspectives, including audits against standards, to support decisions and strategies. Measurement in this environment is less about top-down control than it is cross-functional collaboration. That being said, the experience factory is also an actual infrastructure that requires resources and commitment from the organization. There will still be a need to mandate the creation and management of the factory and to ensure that others use it. But a successful experience factory will, through the products and feedback that it provides to the organization, require less and less direct intervention by the powers that be. As internal stakeholders make use (and reuse) of collective organizational experience, they will begin to depend on the products of the factory and find their decision-making abilities hindered without these products. This self-perpetuation is quite different from the concept of a standard that must be continually reinforced by the organization, because the perceived value of the standard itself is known only to a few stakeholders while the rest of the organization perceives value primarily from passing the audit.

The experience factory reflects an organizational learning style that puts more emphasis on building connections between existing baselines than on building the baselines themselves. To extend the factory metaphor a bit, if the organization has no ready source of the raw materials it needs to operate (data and experiences), then it has nothing to which it can add value and cannot produce anything, just as a manufacturer cannot produce widgets without raw physical materials. Organizations building an experience factory must produce a chain of suppliers for their data first. In the case of IT security, the components of the SPM Framework can provide such materials. In fact, the experience factory builds upon the concept of the SIP, creating a formal capability for taking metrics data and forming it into security experience products for the entire organization.

Mindfulness: Karl Weick and the High-Reliability Organization

This last example of organizational learning styles is perhaps the most unconventional for security professionals who may be accustomed to and comfortable with technology, defined baselines, and quantitative metrics. But those same professionals may be surprised at how applicable this style of learning is to the situations and environments that CISOs and security managers must deal with every day. This example is drawn from the work of Karl Weick, a scholar and organizational theorist at the University of Michigan. Weick has spent decades researching how organizations use and share information in order to function, learn, and grow as systems and social entities, and how they make sense of their environments and business processes. His book The Social Psychology of Organizing has been selected as one of the top ten business books ever written. One of the most interesting ideas Weick developed, and one very suited to IT security, is the concept of mindfulness and its role in high-reliability organizations, or HROs.

Weick and his colleagues studied organizations such as aircraft carriers, nuclear power plants, and firefighting crews, all of which operate in complex and extremely dynamic environments where unexpected events and opportunities for failure are high. When failure does occur in these environments, it can result in exceptional risk and disastrous consequences for the organization, including loss of life and physical destruction. Simple math would seem to indicate that if an organization’s chances for failure were greater than average and the potential damage from those failures was also greater than average, the organization would experience more than its share of failure-related loss. But counterintuitively, Weick found that such organizations actually experienced failure less often than other enterprises and were more reliable than the average, leading them to be designated as an HROs. Weick wanted to know why HROs failed less often in environments where opportunities for failure were much greater, and the results of his research reveals a lot about organizational learning styles.

Put simply, HROs fail less often than other organizations because when they do fail, the results are often catastrophic. Failure for these organizations might very well mean that some or all of the organization ceases to exist physically. Weick’s research proposed that HROs were forced to operate differently as a result, with different processes and business structures that increased their reliability and performance. Weick found that HROs were structurally different from other organizations in some ways, but that the real reason they were more successful had less to do with operational processes and more to do with how the organization viewed itself and the world, and how it made decisions based on this different model of thought. At the core of this difference was a learning style that Weick calls “mindfulness,” in which the organization is constantly and continuously maintaining awareness of what is happening. Of particular interest to mindful organizations is the near real-time awareness of small events that, over time, can cascade and grow into a serious crisis.

Weick identified several central traits that could be observed in a mindful organization:

HROs thrive on failure, seeing mistakes not as something that should not be allowed to happen but as things that are bound to happen and should be identified and corrected while they are still small.

HROs accept complexity and are much less likely than other organizations to oversimplify their activities or their environments.

HROs focus on operational resiliency, meaning that they take a detailed interest in the daily, mundane activities that keep the organization functional, and they are good at brainstorming how things can go wrong with those activities.

HROs allow authority and decisions to flow up and down the organizational chart as they follow experience and expertise, meaning that hierarchy and position are less important than who has the best answer to the question at hand.

Weick’s research is very applicable to IT security, where many security groups seem to be the opposite of an HRO. I often find that security managers will stress over the threat of super-hackers and zero-day attacks, while neglecting to understand mundane operational issues such as passwords and principles of least privilege that are far more likely to result in failure. Problems are simplified (users, compliance, technology) as are solutions (policies, checklists, more technology). And when failure does occur, it is often followed closely by blame and recrimination about who is at fault. Security can often look less like an aircraft carrier and more like a dysfunctional family.

Weick’s prescription for success is for organizations to study the operations of HROs and model themselves after them. The result is less about changing enterprise structure or mandating certain controls and much more about changing the psychology and culture of the organization, which is much more difficult. Adopting a mindfulness style of organizational learning in the context of IT security metrics is probably best suited to companies that already have established metrics programs and the means to share experience. In these cases, the SPM Framework can provide defined measurements and operational baselines to address the empirical assessment of security that would need to accompany the transition into a high-reliability organization for IT security.

Final Thoughts

Thinking about your organization’s learning style and psychology may seem a bit beyond your goal of setting up an IT security metrics program, but that is simply not the case. Everything in this book is about organizational learning in one way or another. I am a firm believer that measuring security activities is one of the single most important efforts that an organization can undertake as we move into the twenty-first century’s digital infrastructure. But simply handing a student a ruler doesn’t guarantee that he will become a successful scientist and not just someone who can tell you how long things are but not give you any other answers. I began this book by discussing Lord Kelvin and the belief that things that cannot be expressed in numbers cannot be understood, as well as the play that this idea has received in the security metrics field. I’ll tell you now that I think things that can be completely explained just using numbers are not really worth understanding and require so little information to be associated with them that they are unlikely to improve anything.

Measurement is about learning and understanding, and metrics are the building blocks of measurement, but they are not the totality of it. Measurement is about priorities and consensus, symbolism, and meaning. I have tried to build a metrics framework in these chapters that remains practical and grounded but never loses sight of the fact that human collaboration and interpretation must be included in any measurement attempt that you undertake. Indeed, they will be there lurking beneath your numbers and graphs even if you choose to pretend they are not.

Summary

Security metrics should be considered in the context of organizational learning and the capabilities of your enterprise to use, reuse, and benefit from the information that is generated by your measurement efforts. Organizational styles of learning can be estimated and applied to security metrics to try to match your organization’s culture and environment to how you measure security and share results. Three styles of organizational learning were discussed here, including standards such as ISO/IEC 27004, which is part of the ISO/IEC 27000 family of international security standards; the experience factory developed by Basili, who also created the GQM methodology; and the concept of mindfulness in high-reliability organizations, developed by organizational theorist Karl Weick. Each style has its own characteristics, and a successful security metrics program will consider how metrics data and findings can be best applied to organizational strategy and decision support based on the learning style of your particular company or enterprise.

Further Reading

Basili, V., et al. Implementing the Experience Factory Concepts as a Set of Experience Bases. www.cs.umd.edu/~basili/publications/proceedings/p90.pdf

Basili, V., et al. The Experience Factory.www.cs.umd.edu/projects/softeng/eseg/papers/fact.pdf

Weick, K. The Social Psychology of Organizing. Addison-Wesley, 1979.

Weick, K., and K. Sutcliffe. Managing the Unexpected: Assuring High Performance in an Age of Complexity. Jossey-Bass, 2001.

Senge, P. The Fifth Discipline: The Art & Practice of the Learning Organization. Broadway Business, 2006.