CHAPTER 6
Designing the Security Measurement Project

Metrics are the engine of security measurement, as I described in Chapter 4, but engines are not usually capable of independent motion. Instead, engines are used to power other things—and security metrics are no different in this regard. You need a vehicle for your metrics, a way to harness their power and benefits toward a larger goal. Security measurement projects (SMPs) are the organizing structures that contain and channel the process of collecting security metrics. They allow you to modularize metrics activities and create more easily manageable building blocks for long-term security improvement. Like any IT project, successful SMPs benefit from forethought and planning as well as organized and effective management throughout the project lifecycle.

Before the Project Begins

The success or failure of many projects are often determined before the kick-off meeting even takes place. Poor planning and inadequate understanding of what a project is supposed to accomplish has killed the potential of many otherwise well-intentioned efforts to improve IT security. Too often, particularly in reactive IT security organizations, a project is synonymous with a firefighting exercise designed to complete an otherwise neglected task in a short amount of time before the auditors or some other authority figure demands accountability. As a result, the implicit purpose of some projects is not much more elaborate than showing that the project (for instance a risk assessment or a policy review) has been completed. If risks are accurately identified, security vulnerabilities are really mitigated, or policies are actually made more robust and usable, this is icing on the cake. The main objective is to cross that task off the team’s to-do list.

In an environment of tight budgets, overworked staff, and increased regulatory scrutiny, we can understand these “do what we can” strategies, but security staff and company leadership should not fool themselves into thinking that sustainable security improvement is a result of the effort. More likely, the organization ends up with check-the-box compliance management and the same false sense of security that plagues other aspects of data protection.

An effective way to avoid these project pitfalls is to adopt an approach to security management projects that does the following:

Emphasizes manageable, measureable projects over vague initiatives. Successful SMPs should be tightly bounded (even exploratory projects) and clearly understood by all involved.

Treats projects as individual links in a chain rather than self-contained activities. A series of smaller, focused projects conducted regularly and coordinated over the course of a year has a better chance of success than a large project that tries to accomplish everything at once and is then forgotten for the rest of the year.

Seeks to expand the project beyond the project team or even the sponsoring organization. Security metrics projects impact the entire organization; therefore, the security project team should actively seek ways to evangelize the results of the project to other areas of the organization. This may involve the project team actively engaging nontraditional stakeholders to determine what the project can do for them.

Project Prerequisites

Before project kickoff, you should have already gathered certain information that will be useful, if not critical, to the success of your SMP. This is the point in the project at which the needs and requirements of the CISO or security organization are at the fore, although these requirements may be dictated from elsewhere in the enterprise (compliance officers, the CFO, manufacturing, and so on).

Goal-Question-Metric Analysis

The pre-project stage is the perfect place to conduct your GQM analysis, if you have not already done so. You likely will have high-level goals in mind, or you probably would not be considering a project, but GQM is the means by which these broad goals are narrowed and contextualized, and the supporting information and measurements needed to meet the goal are identified. The GQM analysis should be formally documented and included as a foundational document in a project-specific repository.

Review of Previous Efforts

In academia, when you write a thesis, dissertation, or other long research study, you are usually required to conduct what is known as a literature review. The lit review, as it is colloquially known, is a thorough examination of (ideally) everything else that has been written on the topic of your research. The purpose of the lit review is to demonstrate that you understand the background of your topic and to ensure that you are not wasting your readers’ time by rehashing existing work or misleading them by taking credit for ideas that are not new. It isn’t a perfect system (and the more difficult the subject, the more literature there is to review), but it is a time-tested means of moving knowledge forward. This concept also has a lot to offer security analysts and project managers.

As you prepare for your project, you should attempt to learn about everything that has already been done relative to the project goals and metrics. If you are assessing some aspect of security, find out whether it has been assessed previously. If you are working on a compliance-related issue, try to understand who else in the company has worked on that particular compliance goal. You may find that the goals, questions, and metrics that you have identified for your project have already been identified in whole or in part elsewhere, even if for a different purpose or organizational unit. You may even find that your own group has already worked on them, but the report has been sitting for several years on the shelves of several successive employees.

By finding and reviewing this data, you can save time and get valuable insights into where to put the effort of the current project. The data you collect may also let you quickly transform your metrics analysis into something more sophisticated, by adding baselines, longitudinal aspects, or other advanced analyses given that you now have existing data to compare with the data that you collect as part of the project. Importantly, understanding what has or has not already been undertaken can help you respond more effectively to concerns or critiques on the part of project stakeholders and sponsors.

Data and Analysis

Since you have already developed GQM criteria for the project, you should give some thought to data sources and analysis that will be necessary for the measurement project. You may not have all the answers at this stage, but some thoughts on how you will develop your metrics data collection strategies and what analysis techniques you think can be useful for the metrics you have selected can help you as you prepare the project plan and begin to assign resource requirements.

When considering data, remember that in many cases you will not own or control access to the repositories or other sources of data that you need to collect. Your planning process should include consideration of stakeholders you will need to work with to get data in the first place, whether that means an administrator giving you access to the systems she controls or a manager giving you access to staff members for purposes of interviews and discussions.

Deciding on a Project Type

Another way that you can begin getting specific and anticipating how the project will progress is to think early about what kind of measurement project you will actually conduct. We talk about projects in a generic sense all the time in IT, but there can be serious differences between one project and the next and one type of project and another. Some of these considerations will emerge from your goals and questions, but it can be helpful to consider the structural limitations and necessities that are involved in different project types, which might include the following examples.

Descriptive Projects

The most common projects we deal with in IT security are those that describe a current state in some aspect of security, and then perhaps we use the results to make an effort to improve security in a future state. If you gather data regarding event and incident statistics for a management meeting, you have completed a descriptive SMP. Measurement projects of this type require you to think about where you will get your data and what descriptions will be of the most use to you and to your audience, and they may involve analysis and recommendations for future improvements, particularly if the description is not favorable to the goals of the project stakeholders or sponsors.

Experimental Projects

Experiments are defined as tests or procedures that are carried out to further knowledge, expand capabilities, or analyze preexisting information. We do not usually think of ourselves as conducting experiments in security operations, and in fact we may specifically deny that we do so, because experimentation carries the implication of unknown results and possible wasted effort. (For instance, we might not call implementing a new secure e-mail system an experiment.) But most scientific experiments are not blind attempts to do something new, but rather very detailed and sophisticated tests of what is expected in a process—just as most IT security implementations have a chance of failure after they go into production, despite our best efforts and intentions. Pilot projects can be a kind of experiment in IT environments, but pilots tend to be limited to small implementations of a new system or technology to see how it functions (it would be quite common to have a pilot project for the new e-mail system I mentioned previously). Real experiments are a bit different in their purpose and methodology.

From a security metrics perspective, experimental projects can be any project in which comparing observations leads to conclusions about some state of affairs. Just because a project is experimental does not mean that it is a research project instead of an operations project. The manufacturing industry, for example, regularly uses statistical quality control experiments to determine whether production is uniform and efficient, and to shed light on causes when this is not the case. Security teams can and should use experimental designs to measure operational activities as well. This can include using inferential statistics to gain insight into a population, or fielding new configurations or technologies to effect security changes.

At the end of the project, you will have knowledge of how things may be or actually are different between your control groups and your experimental groups, and you can test null and alternative hypotheses through observation. One of the objectives of successful experimental projects is to manage your analysis and findings adequately so that you have some idea of why differences exist between those states, so you can intelligently articulate those results to your project stakeholders.

Compliance Projects

Compliance projects demand that the security program adhere to criteria or specifications developed by authorities usually external to the program. These projects involve meeting legal and regulatory requirements, aligning the program to industry standards, or fulfilling contractual or other business obligations. The interesting aspect of compliance projects is that you will usually not be able to self-assign criteria for a successful project, other than whether or not compliance was achieved. The details and specifics of what defines that success are mandated upon the security group from outside. This means that in order to be successful in these projects, you will need to understand in detail what someone else cares about, and what they care about may be documented in formats or languages that you are not accustomed to or experienced with. (Reading government regulations or legal contracts for comprehension is a discipline all its own.) So when considering compliance projects, you should immediately begin deciding which outside stakeholders you need to include to improve your chances for success. Your data and analyses may demand special insights and skills that exist outside of the security program.

Of course, these are not the only types of projects that you will encounter or develop. I have described each of these in quite general terms. Different project types will overlap, and others will not fall into any of the preceding categories. The main take-away for this preparatory stage is to think about the structure of your measurement project and the unique aspects that any given structure may carry with it. This will help you anticipate challenges and potential problems and help you understand where the most value can emerge from any given activity.

Tying Projects Together

SMPs are one more intermediate component of the Security Process Management (SPM) Framework. As measurement projects and the metrics and data that they encompass are completed and incorporated into the organization’s experience and knowledge, they begin to form the next level structure, the Security Improvement Program (SIP), which is described in detail later in the book. But the SIP cannot spontaneously emerge from measurement projects any more than measurement projects spontaneously emerge from goals, questions, and metrics. Projects must be designed so that they link with other projects, providing input to some projects and receiving outputs from others. These inputs and outputs may be direct or indirect, and they may be limited to historical context only. But even historical context would be an improvement in many security programs, where it seems that the ravages of time and reorganization can make it difficult to understand what transpired one or two years ago, much less over the life of the security program.

You can build cross-project functionality into your metrics program in a number of ways, but all of them require that the owners and stakeholders of the projects first make a commitment to ensure that the projects remain linked and cross-referenced. This commitment need not come from senior management, although it certainly helps if it does—and senior management commitment is necessary when the scope of the project crosses team or functional boundaries. But any security manager or analyst working on their own projects can take the initiative and build continuity into their projects just by demanding (from themselves and from others as they are able) that projects be documented and that documentation be maintained for whomever wants to review it.

Building a project catalog can help significantly in such cases, and the catalog does not need to be fancy, although it must be usable. (I always find spreadsheet-based catalogs difficult to use. I prefer narrative documents in which more information can be captured, with tables for more structured data as needed.) The catalog should be as complete as possible and as available as necessary. This could mean assigning a project catalog owner who is tasked with passing on the responsibility if he or she moves on to other jobs.

Getting Buy-in and Resources

The adage that “you can’t get something for nothing” is a cornerstone of security, although the industry does not always remember it. Perhaps more than other aspects of IT, security is almost all about tradeoffs and compromise (in both senses of the word), and this applies to SMP management, too. Security professionals know a lot about what needs to be done to improve the posture of their programs and their infrastructure, and we know firsthand the consequences that can result from not having protections and controls in place. Where we have less success is in understanding that everyone else may not understand or share our experiences and insights. Nothing is more frustrating than watching people behave in ways you know are self-defeating—nothing except, perhaps, trying to convince them to change.

So when it comes time to get the support and resources for SMPs, it will not be enough for you to make appeals based on what you know to be correct or valuable as a security specialist. At the time of this writing, the economic downturn has exerted pressure on businesses that make it challenging to get the resources necessary simply to do what they have always done, and budgets even for daily operations have been drastically cut. But even in the wake of a recovery, there will always be competition for limited resources within organizations. To ask for more money, people, or tools means you’re going to have to up your game, and that means you need to ask yourself, to paraphrase the famous line, “not what your organization can do for you, but what you can do for your organization.”

Identifying Stakeholders and Sponsors

The success of most projects is directly proportional to the number of people who believe the project needs to be done and done correctly. It is a given (in fact, a cliché) in IT security that you must have management support to have success. Management support ideally refers to senior leadership support at the CXO or board level, but in practice, such support is more of a formality unless mandated by a compliance requirement such as Sarbanes-Oxley or ISO 27001 (and sometimes even then it can be difficult).

My philosophy on management support is that depending upon senior management buy-in as a prerequisite for action is the wrong way to approach the challenge. Instead, I advocate a broad approach by which you attempt to influence operational management and the front-line and mid-management levels, where value can still be tangibly measured and expressed. If you can convince peers, particularly peers outside the security realm, that a project will add value to their bottom-line management needs, this support will begin to be expressed upward. Eventually, senior leadership in the enterprise will find they are fielding security project requests not from the security team, but from the managers and stakeholders in their own areas. As security becomes a priority for more than just security people, it will get the attention of leaders who are more attuned to detecting trends and generalizing across the enterprise than to evaluating and comparing individual cross-functional needs and requests.

Approaching projects in this way will require a bit of a change on the part of security teams as well. We can be an insular and suspicious lot, not accustomed to or comfortable with diplomacy and putting others’ priorities ahead of our own. But the security world needs to get better at helping others understand what we do and, more important, why we do it, and we need to express these things in terms of the language and requirements of other groups and functions. Security pros who are good at this will find opportunities for expanding their influence and prestige across the enterprise.

Estimating Resources

Few things will kill the buzz of a good security metrics project faster than going over budget or coming in late due to a lack of effective planning. In the case of compliance, the results can be worse, particularly if the auditors are ready to walk in the door and you are not adequately prepared. So measurement project managers will do well to consider and analyze project resources seriously before you begin. One of the benefits of taking a framework-based approached to security metrics, one that recognizes that security is being assessed continually rather than periodically, is that you can afford to be more conservative with your projects. It is better to develop a project of limited scope, which is manageable and which can provide incremental security value, than to attempt to take on too much and fail in execution, follow-up on the results, or both. Small, well-coordinated projects allow for much more granular control over the security program and have the benefit of being easier to scope and easier to complete.

When estimating measurement project resources, you need to consider questions of data collection and analysis. As I discussed in previous chapters, preparing data for analysis can be very time consuming, and if you are choosing new analytical techniques, unforeseen learning curves can be associated with new tools and practices. If you are partnering with other stakeholders, especially those outside the security group, you should also consider that it may be necessary to explain your progress and to ensure that their goals continue to be aligned with your own. And always consider the impact of other duties and daily operations on the measurement project. Your plan should include an implicit recognition that nothing ever goes exactly as planned.

Borrowing from project management methodology, it is advisable to conduct a risk analysis on your measurement project that can help you identify areas of uncertainty and potential problems that could arise over the course of the project. Interestingly enough, risk analyses in project management often look a lot like risk analyses in IT security and usually involve the project team qualitatively discussing and attempting to categorize and prioritize subjective understandings of risk. If your organization does not have defined project management methodologies, it may be necessary to guess a bit in the beginning, but in security metrics everything has the potential to become data, and you should be documenting project progress, including problems, overruns, and delays, so that the next project risk analysis has more than just opinion on which to operate. Specific resource issues to consider as part of the risk analysis include the following:

People What risks are presented by the project stakeholders themselves? What happens to the project if a stakeholder withdraws support? What happens if you lose a resource due to unforeseen circumstances?

Material and operational resources Which resources are critical to the project’s success? Could certain data sources, tools, locations, or monetary resources significantly impact the measurement project if they were altered or became unavailable?

Technical and analytical resources What risks are imposed by the techniques and tools that you have selected? Are you choosing commercial or open source tools to complete the project? What happens if a new tool is needed during the project?

Contingency planning For all the risks associated with the project, what are the contingency plans for dealing with any particular risk? Are workarounds available, or will certain risks threaten the completion or success of the project? Have all risks and contingencies been communicated to project stakeholders?

Managing projects is a discipline and craft unto itself, and as you consider setting up a formal security management program, you should also look at setting up formal project management programs to facilitate your metrics. Not only will this help with individual projects, but it will facilitate and improve the collaboration and coordination of SMPs that takes place as part of the SIP, described later in the book.

Presenting a Business Case for Metrics

After the project has been defined, the security metrics team should develop a formal business case around the measurement project for several reasons: A business case is a good method by which to document the project and archive it for future use. But equally important, documenting a business case allows you to articulate to all stakeholders and sponsors exactly what is to be accomplished through the measurement project, and what each of them can hope to get out of it. There is no set template or best practice for the project business case, but it should be readable and as brief as possible while still being adequately descriptive. Here are some things to include in the business case:

Stakeholders and sponsors The business case should describe everyone who has a stake in the project and what that stake is. It is important that participants feel included in the process, and it is also important that they see others’ involvement. A business case that includes several sponsors and offers cross-functional support can add immediate credibility to a project.

Goals, questions, and metrics The business case should clearly articulate the results of the GQM analysis and should tie the results to the goals and requirements of specific stakeholders.

Project cost and project benefits The business case should tell each reader why establishing and analyzing these security metrics are important and what it will take to realize the value that they provide. It may not be possible to forecast financial benefits of the project immediately (that may be exactly what the metrics are designed to reveal), and in these cases the business case should explain this.

Risk analysis results The measurement project team should be up front about risks and contingencies identified during the project risk analysis. There should be few surprises over the course of the project, even if something goes wrong.

Formal acceptance At the conclusion of the business case, a process for acceptance of the SMP by all associated stakeholders should be defined. It is best if this includes formal sign-off by sponsors and those providing project resources.

Having set the stage and done your best to consider the criteria for success of your measurement project, the operational phases of your metrics activities can begin.

Phase One: Build a Project Plan and Assemble the Team

The business case documented the project for sponsors and stakeholders. The project plan is the formal documentation of the project for those operationally involved in its execution. It guides the project team members in their efforts to complete the project.

The Project Plan

A project plan is a documented operational map of the entire project that is designed to record all pertinent details in one place. Many resources are available for project managers, including a variety of templates for project plans, so I will not attempt to reinvent the wheel for this chapter. But at a minimum, the project plan should capture the project goals, deliverables, and milestones at a level of detail that exceeds the project business case and allows the project to be effectively managed. The project plan should also be included in the project catalog developed in support of the SIP. The plan should also be reviewed and consulted regularly during the operational life of the project to ensure that milestones are met and deliverables meet project stakeholders’ expectations.

Project Goals

The description of the project goals in the project plan may be derived from the project business case, and the need for more detail is perhaps less imperative than the need for milestones and deliverables. But the project goals should include descriptions of stakeholders and the associated stakeholder priorities that were reflected in the business case. Documenting these goals in the project plan enables baseline development and goal tracking over time when projects are linked and cross-referenced, and the inclusion of the goals in the operational details of the project serve as a guidepost to the project team as the work effort progresses.

Project Deliverables

The associated project deliverables should be directly mapped to the goals identified in the project plan. Deliverables can include descriptive reports, findings from experiments or inferential analyses, readiness to pass an audit, or the establishment of other projects as part of the improvement program. Whatever the deliverables are, they should be documented and explicitly aligned with the goals they meet and support. The project plan should specify the expected format and approximate structure of each deliverable and should identify specific stakeholder requirements for deliverables. For instance, in vulnerability assessment projects, there may be requirements to deliver a higher-level report to a project business sponsor, but the technical stakeholders in a project may be more interested in the raw metrics data. The project team should understand different needs and develop customized deliverables accordingly.

Project Milestones

Milestones should be established for all project deliverables, taking into account the resources available to the project and the complexity of the deliverable product. Milestones should be developed on an individual basis for each task and subtask of the measurement project, and these tasks should be assigned to owners within the team. Project timelines should also be established and developed in conjunction with the milestones. Where dependencies exist between deliverables or related activities, these should be noted within the project plan.

Project milestone development can be a manual process, but the evolution of project management software has removed much of the heavy lifting involved with planning and executing on project schedules. Milestones and timelines are important not only for the project goals, but also as data sources for empirically assessing the project’s effectiveness. Like any other data, knowing where you succeeded and where you failed to achieve a milestone within a set time period can generate new questions and insights about your security operations. Many organizations will have access to dedicated project management tools and resources, and project teams should take advantage of these tools, a few of which I discuss at the end of the chapter.

Project Details

In addition to pre-established details, the project plan should give team members the ability to add details and track the project as it proceeds. Records of decisions, activities, and problems that occur during the course of the project should be noted and be included as working notes within the project plan. If regular project meetings occur, the minutes or meeting notes from these sessions should also be included, as should descriptions of metrics activities including data collection and analysis.

Documenting project details can often seem like extra work for little gain, but the effective recording of a project journal can prove invaluable when it comes time to analyze data and articulate findings to stakeholder audiences and sponsors. Project details also serve as supporting data in the project catalog, providing project managers and security analysts the benefit of the team’s experience even after the details of the project are lost from memory. This movement from tacit project team knowledge (that which is informal and undocumented) to explicit knowledge (that which is documented and preserved) helps the project to achieve an impact on organizational knowledge management and not just the security issue immediately at hand.

The Project Team

In most cases, the staffing of the project team will not be very flexible. Security staff will be assigned to projects based on roles and ownership of the resources that the project is designed to measure. Outside resources, when included, will be contingent on the availability of people and perhaps on skills and expertise (usually the former will trump the latter, unless a sponsor is truly invested in the results of the project). So when the SMP manager assembles the project team, often the best that he or she can do is to try to ensure that the available resources are appropriately tasked.

Skills

The first thing to consider when assigning project resources is the mapping of team members’ skills to the tasks associated with the project. These assignments become more important as particular data collection and analysis techniques are selected and implemented. Asking project team members to perform tasks that are difficult or uncomfortable for them can threaten both the team dynamic and the project results. If some team members are shy or reserved, it may not be the best idea to send them out to interview managers in other business units. Similarly, asking a very gregarious and social team member to sit in a cube and learn to crunch statistics may not be the best use of that individual’s unique skills.

At minimum, make an effort to map people to those project tasks to which they are best suited. This may seem like common sense, but I’ve been involved in a lot of projects where it seems that tasks were randomly assigned to project participants with no real thought of whether that assignment was smart. Naturally, there may not always be the luxury of choice on a security project, but at the very least the project lead should spend some time developing a skills matrix for the team so that people believe that an attempt was made to make the best use of each individual’s talents and strengths. Even if there is no way to assign each member of the team that one task that they are most capable of doing or are most interested in, taking an inclusive and sympathetic approach to assigning project duties can have a positive effect on morale and the project working environment.

Commitment

Along with creating a skills matrix, my experiences have taught me that it pays to recognize up front that not all project members are equally committed to the task at hand. This doesn’t mean that some of your team will be slackers, although they might very well be, but reflects the fact that in any dynamic environment, some people will be struggling with conflicting schedules and requirements that mean they will not always be able to dedicate themselves to the SMP. You can prevent a lot of animosity and wasted effort by recognizing this fact up front, not taking it personally, and simply dealing with it. Asking the team up front to provide estimates of their ability to commit their time over the course of the project can identify problems before they grow acute. If a project team member knows, for instance, that he will be on vacation for the last quarter of the project or that he is currently finishing up a different project and won’t be able to engage fully yet, then recognizing such facts can go a long way toward making sure these issues don’t result in a delay.

Collaboration

Another aspect of the project that should be decided up front is how the team will collaborate. Today’s work environments allow for many more options in this regard, as there may be less need for physical meetings or co-location of the team members over the course of the project. Communication and collaboration mechanisms should be discussed and decided upon at the beginning of the project, preferably during the project kickoff meeting at the latest, and should be documented in the project plan. Collaboration tools and processes should take into account the need to share information and project data, as well as any differences in location or time zones (in today’s global environment this can be especially important).

One important aspect of collaboration is making sure that important project information is documented as part of the project catalog. Commonly used collaboration mechanisms such as e-mail and instant messaging can make it difficult to archive and share project interactions. The measurement project lead should put some thought into the types of project information that need to be recorded, the level of detail necessary for this information, and how to ensure that any interactions by team members are properly documented and included in the project working papers.

Phase Two: Gather the Metrics Data

Once the project plan and team members are in place, the project can move forward with answering the questions and gathering the metrics data necessary to support the project goals. Several important considerations are required in this phase of the project, most of them concerning the appropriate ways in which data is collected, stored, and protected.

Collecting Metrics Data

The data that you collect will vary, perhaps widely, according to the goals and metrics that you have developed. Some data collection, particularly that in support of descriptive measurement projects, will not require changes to existing practices, and you will use the same tools and sources you used previously, even if you end up conducting more advanced analysis on that data. But if you are incorporating other goals, such as prediction, longitudinal study, or qualitative approaches, you may have to develop new means of collecting as well as analyzing your security metrics data.

The first thing to consider is whether or not the data you need is immediately available through existing systems and resources. The more your project draws from different groups within the enterprise, the less likely it is you will be able to gather the data you need centrally. The same holds true for metrics that do not rely only on system-generated information. Even with system data, you may need to go through archives and historical data repositories to find what you need. You will need to identify and get authorization to use data from any sources not under your immediate administrative control, and your project business case and project plan can help you justify these requests. If your data depends on interactions with people, whether through surveys, interviews, or other interactions, you will need to identify who you must talk with and get the appropriate approvals as well.

Herding all these cats can be a big challenge and time-consuming in and of itself, taking away from the time you actually have to collect and analyze the data that is core to your measurement project. You may track down the data only to realize that you now have to devote significant time to cleaning it up to get it into a usable form. Or if your data has been generated by some customized or home-brewed system, you may need to go back and forth with the owner to translate what exactly the data points or outputs represent. Sometimes you may even discover that the data you’re looking for doesn’t exist and you are forced to look for a different repository or change requirements and goals based on data sources that you actually can find.

When it comes to interpersonal data collection such as interviews and ethnographic analysis in which you are interacting with a colleague or a group within the organization, there are important concerns. In most research using these techniques, it is common for these observations to be recorded, including interview conversations and even visual recording of the groups under analysis. In industry settings, this can be difficult to do. People are naturally nervous about being recorded in the workplace, and while the data is much more complete when fully recorded, it can be offset by the tendency for people to be less honest or forthcoming. If you cannot record the data you collect, and most of the time you will not be able to, then you must fall back on detailed note-taking as your primary means of collection. In interviews, it often helps if two analysts work together—one conducting the interview and taking some notes while the other is responsible for collecting as much data as possible.

Storing and Protecting Metrics Data

After data is collected, it is important that you give thought to how it will be stored and accessed. You want to make sure that the data you will be using for your project remain in the same state they were when you observed them, and you want to ensure that they are properly controlled and secured, particularly when they involve sensitive data such as information about security operations or personally identifiable data about interview or survey participants. It is best to have a dedicated, secure location (physical or electronic) in which to store the collected data and to limit access to the data only to the project team. If data cleaning or normalization takes place, or if different versions of the data are being used as the measurement project progresses, it is important that someone keep track of these changes. Nothing is worse than getting halfway through an analysis only to realize that you are using a different data set than the one you intended. Even worse is never to catch the mistake and have it influence your findings and conclusions. Security metrics are all about the data, and ensuring that you have access to the correct data and that you can easily document and justify your analysis process at the data level represents an important level of project governance.

Business, legal, and even ethical concerns may also be associated with the data that you have collected. Recall previous statements I have made about data retention and the need to take action on findings. Collecting metrics data often means that you are creating new knowledge and new corporate records. If these records involve particular systems, groups, or individuals, they should be assessed as part of the company’s records retention schedule and included as official company documents. At the end of the measurement project, a decision should be made, in accordance with the retention schedule, regarding which project documents should be kept and which should be archived or destroyed. Project business cases, plans, and final deliverables should always be retained as part of the SIP (again, within the guidance of the company’s retention schedule), but the data collected as part of the process should be considered on a case-by-case basis and kept as necessary to support the security program.

Phase Three: Analyze the Metrics Data and Build Conclusions

Chapter 5 described security metrics analysis techniques in detail. After you have successfully collected your data, it is time to put one or more of these techniques to use. Once again, if your analysis is primarily descriptive, you may not need to change much in terms of how you undertake this phase of the project, other than perhaps approaching your analysis with a broader understanding of the roles that metrics, data, and analysis play in your security program. If, however, you have collected data for predictive analysis, experimentation, or hypothesis testing, you will have to deal with additional tasks and requirements in analysis. The most important of these, particularly in cases of using data to generalize or compare and test competing explanations of aspects of security, is that your analysis plan should be developed ahead of time and explicitly included in your project plan. The reasons for this pre-determination are worth revisiting.

Central to the concept of inferential statistics is that you develop criteria and thresholds for acceptance regarding explanations and generalizations that answer your questions objectively. In other words, you want to avoid any temptation to cheat by altering the conclusions based on what you wanted to find. It is much easier to avoid getting into these situations if you have decided those criteria and thresholds, and documented them, before you begin collecting and analyzing your evidence. If these parameters of the analysis are part of the project plan, just like the deliverables and milestones, then any changes become obvious and must be discussed with the team and possibly with stakeholders and sponsors. Conversely, if you have developed these criteria and thresholds as part of an approved and accepted project plan, then you can more easily defend any surprising or unpopular findings or conclusions to your project stakeholders. A well-defined analysis plan is like a contract between analysts and audience. It may not always protect you from requests to change your conclusions based on politics or personal feelings, but it puts you in a better position to defend your case should such requests be made.

Another consideration for analysis that should be included in your project plan is to ensure that you have included adequate time to explore the data and develop your conclusions. Analysis takes time, and stakeholders often will be looking for your findings within days of your completing data collection. One researcher I know, an anthropologist who conducts qualitative research for a major technology company, described how every time she came back from fieldwork, her product teams would begin pressuring her for results. And every time she had to explain that they could get the raw data, which would be useless to them, quickly, or they could allow her to complete her analysis and actually get something that would add value to their efforts. You can ward off some of this impatience by realistically building the analysis into your project schedule, but you should also consider the actual resources it will require for your analysis. It may not be necessary for the entire project team to be involved in the analysis, especially when the skills and tools for specialized analysis are in the hands of only a few members. In these situations, you should consider releasing team members back to normal duties and continuing the project with a core analytical team. If you choose this option, I recommend continuing to keep the larger team in the loop, and bringing the entire team together when it is time to present your results to sponsors and stakeholders. This way, everyone is still able to participate in and take deserved credit for their roles in the measurement project.

Phase Four: Present the Results

While collecting and analyzing security metrics data carry unique challenges and obstacles that must be overcome, presenting the results of your metrics analysis presents its own challenges. When you have put a great deal of effort into developing information that is valuable and can contribute to the improvement and success of the organization, you obviously will want everyone to take that information as seriously as you do. But you cannot assume that this will happen simply on the merits of the results. The presentation of metrics findings always has elements of marketing and sales to it, and the wise security metrics professional will realize that even the best data analysis in the world is less useful if you can’t get anyone to read it. Sometimes a slide deck is just not enough, and nothing is worse than watching excellent measurement and analysis fail to impact because the results were not presented correctly. For very important projects you may even consider hiring outside communications or marketing specialists to assist your security metrics efforts by enhancing presentation and dissemination of results. This may be a particularly attractive option if these skills are lacking within the existing security organization.

If you have worked closely to get buy-in and support from your stakeholders, and you’ve done a decent job of showing those stakeholders how your metrics benefit them directly, it will probably less difficult to keep their interest in your results. The goals, questions, and metrics that you have developed prior to beginning the project will go a long way in guiding how you present results. Nevertheless, you should not assume that every audience has the same interests or needs regarding the analyses you have conducted and the conclusions that you have made. It helps to perform a bit of market segmentation work on your larger audience to ensure that you are meeting these different needs.

Some of the groups to which you will likely be presenting information include the following:

Nontechnical management If you have developed stakeholders outside of the security group, or if your conclusions are being presented up the leadership chain, it is likely that your audience will include nontechnical people who have little interest in technical details or even security, except as these things impact issues such as dollars, productivity, or compliance.

Technical management In many cases, you will be working with people who do have technical skills but are also concerned with how to translate technical details into business value and articulate that value to nontechnical peers and supervisors.

Operational staff When your conclusions involve actions such as remediation or system configuration changes, the technical personnel responsible for implementing these recommendations will often be interested in detailed technical and analysis data from your project.

Users In some cases, your data will drive changes in organizational behavior, including the development of new policies or training and awareness programs. You should be able to present your findings to these groups in ways that are understandable and that explain why these changes are necessary.

Outside entities If SMPs have been conducted to support audit or compliance objectives, it may be necessary for you to translate the results into the specific language of the auditors, regulators, and consultants that you will work with to meet your larger organizational goals.

Textual Presentations

Written reports are a mainstay of all research, whether in business or academia. Unless you are dealing with very specific goals and analysis, you will almost certainly develop some sort of written report for your project, even if it serves mainly as background information. You have already developed some documents to this end, including your project business case and project plan. Although it’s common, I recommend against shoving all your results into PowerPoint, which is unsuitable for presenting large chunks of text. Instead, take the time to develop at least a written summary of the results of your project. This document does not have to be long, but it should be detailed and in narrative form so that someone down the line can read it and get a richer understanding of the project results. You may disseminate this overview before the presentation to add context to the shorter slide summations, or make it available afterward to add more depth. This becomes especially important in the context of the SIP and the project catalog, when the goal is to build connectivity and context between projects over time.

As you build project documents, you should strongly consider using a standardized style guide and to take issues of readability into consideration. A style guide is a reference document (often a book) that defines standard and accepted ways of producing written communication. The MLA Style Guide is a well-known example of such a reference that provides advice on grammar, structure, citations, and other necessities of effective writing. Numerous useful style guides are available for business writing, a few of which I list at the end of this chapter. The sad fact is that a lot of business writing today puts little or no effort into ensuring that the writing is consistent, correct, and readable, an avoidable mistake that can severely limit the usefulness of your metrics reporting.

Visual Presentations

We are all taught that a picture is worth a thousand words, and, whether or not this is true, you will certainly benefit from visual presentations of your data and findings as you proceed with your measurement projects and your security metrics program. You probably already have experience building charts and graphs in spreadsheets and presentations and tables in word processing documents. These are all useful tools for presenting your metrics analysis results. If you are using advanced statistical or qualitative analysis software, you will want to explore the capabilities that these tools offer for visual representation of your results as well.

I will explore examples of visual data presentation techniques in subsequent chapters and case studies, but for now consider some basic visualization techniques:

Charts and graphs The workhorses of visual presentation, these can include histograms and other bar charts, pie charts, line graphs, and a variety of other visual aids. Even simple red/yellow/green matrices can be very useful in conveying data visually and intuitively, so long as you can adequately explain the complexities and nuances that may lurk behind the colors.

Maps A map is a representation of just about anything, including geographic areas, technologies, people, or concepts, built with some navigational purpose in mind. Maps can help you visually describe processes, social networks, and the relationships between your data sources and results. Maps can even be used to represent themes, stories, and histories that emerge from qualitative data analysis.

Scorecards and matrices Designed as ways to summarize and visualize disparate concepts and reveal relationships, these visual tools include balanced scorecards for presenting performance indicators as well as diagrams such as SWOT (strengths, weaknesses, opportunities, and threats), force field diagrams, and positioning matrices.

Disseminating the Results

An important question that the measurement project team must answer is how the results of the project will be disseminated to the various stakeholders and sponsors involved. It is preferable, when possible, to have some face-to-face interaction with all the stakeholder groups involved in the project. Sending results over e-mail or posting to a server can eliminate a great deal of useful interaction and runs the risk that the results will be reviewed in a cursory fashion, if at all. You want to try to get in front of the people you have sold on the project so that you can explain to them how you met their needs, understanding that this may not always be an option.

Group presentations can be useful, and are often conducted at the close of a measurement project. If you are presenting to a group, you need to understand who is represented and adjust your content accordingly. If you have limited time and results that include both technical and nontechnical conclusions, you may want to consider having more than one results meeting, perhaps hosting several meetings with individual stake-holder audiences. This can have some limiting effects, as there is benefit to getting all the stakeholders into the same room, but it may be unavoidable.

Whatever dissemination mechanism you choose, you should also build into the project plan a capability for following up with project stakeholders and sponsors over time, both to elicit their feedback on how they used the results of the project, and to maintain a network of potential supporters of ongoing projects and initiatives around the security metrics program.

Phase Five: Reuse the Results

Security metrics are most beneficial when they are developed and maintained over time within the context of continual improvement such as the SPM Framework. The most common mistake I see in security programs throughout the industry is the lack of continuity and reuse of security data across projects and throughout the life of the organization. The idea of reusable and consistent measurement of security processes over time is embedded into the idea and implementation of security capabilities maturity, but many organizations remain at the low, ad hoc end of the maturity scale.

I will cover the reuse of security metrics, measurement project results, and the development of structures to facilitate continuous organizational learning later in the book, but building the hooks for reuse into your SMPs is an important prerequisite to realizing a long-term vision for your security program. In every project you develop, explicit questions and follow-up actions should extend beyond the immediate life of the project. These can be as simple as periodic follow-ups with the project team members and key stakeholders to review how the results of the project were incorporated into the organization’s activities, or they can be more formal reviews conducted as part of compliance or management initiatives. But at the end of the day, it will be the security team and the CISO that must take primary ownership for ensuring that the efforts made to measure security are not neglected or eclipsed in favor of the daily grind of security operations. The need to move from tactical to strategic thinking in security begins with those tactics themselves, in the form of the security projects we conduct every day.

Project Management Tools

Project management is an enormous discipline and a thorough discussion is outside the scope of this book. Many resources are available for guidance on how to manage SMPs, and your own organization may already have resources for effective, standardized project management. If not, there is no shortage of good places to look to improve your project management skills and capabilities, none of which are necessarily specific to IT security:

Project management software Many vendors, from Microsoft to cloud startups, offer advanced project management tools that include features such as scheduling and resource allocation, milestone tracking, and project risk analysis. Some are expensive, but several open source project management tools are available as well, including Open Workbench, Project.net, and Project Open.

Project management organizations Professional associations dedicated to project management exist globally, including the Project Management Institute, which provides international certification for project management professionals.

Project management training and skill building There are many books, courses, and classes that can help you or your team improve project management skills. A quick web search on “project management resources” is a good place to start if you are interesting in building these skills personally or within your team.

Summary

The SMP is the primary vehicle for operational analysis of the security metrics you develop within the SPM Framework. Measurement projects allow you to create a modular metrics program around tightly bounded goals that are linked and reused over time to facilitate continual security improvement for the organization.

Your measurement project work begins before the project itself ever kicks off, and includes aligning the project with GQM analysis, reviewing what has been done before in regard to the work being conducted, and developing and identifying stakeholders and sponsors for the project. Supporting individuals and groups will all have different goals and requirements for security, and for stakeholders outside of the security group these goals may not even be described in terms of security.

If the program is to be truly successful, it is incumbent upon the security team to promote and champion security metrics on the basis of more than just the needs of the security organization. To accomplish this, the team should build a formal project business case that can be used to communicate and promote the project activities and goals.

Once a SMP begins, it consists of five basic stages:

1. Building the project plan and assembling the project team.

2. Gathering the metrics data.

3. Analyzing the metrics data and building conclusions.

4. Presenting the results.

5. Reusing the results.

Different projects will have different approaches, data sources, analytical techniques, and results. Wherever possible, the project team should use existing organizational resources to keep the projects standardized. If standards for project management or results presentation do not exist, the security metrics team should consider developing standards, including style manuals and project management tools and skills to ensure that the value of the measurement projects are maximized and utilized by the widest possible audience.

Further Reading

Alred, G., et al. The Business Writer’s Handbook, 9th Ed. St. Martin’s Press, 2008.

Modern Language Association of America. MLA Handbook for Writers of Research Papers, 7th Ed. 2009.

Project Management Institute. A Guide to the Project Management Body of Knowledge (PMBOK Guide), 4th Ed. 2008. Available from www.pmi.org.