21 Stage 3: Prevention
If you do not actively attack the risks, they will actively attack you.
—Tom Gilb
In this chapter, I describe the practices that characterize the prevention stage of risk management evolution. Prevention is a transitional stage, when the risk management approach changes from reactive avoidance of risk symptoms to proactive elimination of the root cause of risk. Individuals on the project team are aware of basic risk concepts, and most of them have experience in risk assessment. They have already learned a valuable lesson: assessed risks will be realized unless action is taken to prevent them. Through this insight comes the realization that what we do not know will hurt us. Most people are comfortable identifying risks but are unsure how to quantify them. They define a process to help them follow through for risk control. Managers acknowledge the paradox that control is best achieved by first accepting the lack of it. They recognize that only the technical staff can control technical risk, which is often the source of cost and schedule risk. To achieve risk control, managers know they must delegate the responsibility of assessing risk to the team. There is a shift from risk management viewed as a manager’s activity to a team activity. This is what we observe in the prevention stage project that I discuss in this chapter. The people and practices of the project are authentic, but the project name remains anonymous.
This chapter answers the following questions:
What are the practices that characterize the prevention stage?
What is the primary project activity to achieve the prevention stage?
What can we observe at the prevention stage?
21.1 Prevention Project Overview
The project is a $100 million command-and-control center upgrade. The prime contractor, subcontractor, and the customer are colocated at the development site. The system concept of operation is a complex turnkey system incorporating communications, telemetry processing, timing, command and control, and data processing. There are special constraints for safety-critical equipment with requirements for no single point of failure. Software is being developed in Ada 95 using an object-oriented tool set. There are approximately 2,000 classes in the object model. Reuse on the 220K total lines of code is estimated at 15 to 20 percent.
The prime and subcontractor formed an integrated team to leverage organizational resources, which include a quality philosophy and software process assets. The team also inherited relevant experience in risk management from previous projects. The foundation of the project’s risk management practice has the following elements:
A risk-driven development model.
Quality management techniques.
Organizational process assets.
Previous project experience in risk management.
21.1.1 Use a Risk-Driven Life Cycle Model
The project required the use of a spiral development model [Boehm88]. In response to this requirement, the project software standards manual described a rigorous risk management process that is a key element in the successful use of the spiral life cycle model. Before the start of each product development cycle, spiral model management activities were used to permit a more opportunistic approach to be developed based on the current project situation and knowledge gained from previous product development cycles and risk mitigation activities. The management activities included identifying the objectives, constraints, and alternatives using the worksheet shown in Table 21.1. (This worksheet is a modified version of that presented in [Boehm88].) Objectives provide focus on specific requirements, constraints provide a basis for scoping the work to be performed, and alternatives for achieving the objectives and the tasks necessary for each strategy are outlined. The approaches are evaluated, and the best alternative is selected.
Table 21.1 Objectives, Constraints, And Alternatives Worksheet
21.1.2 Reuse Quality Management Techniques
The team believed that a major factor contributing to the success of prior projects was corporate commitment to the total quality management (TQM) initiative. Total quality management is the vision, guiding principles, and philosophy that form the foundation of organizational improvement with respect to people, processes, and products. Both the prime and subcontractor organizations were industry leaders in TQM and were registered ISO 9001. Each organization had received several quality awards for leadership, performance, and excellence on projects. The team was built on a TQM philosophy, with 100 percent of the team trained a minimum of eight hours in quality management. The team reused the following quality techniques in risk management:
Brainstorming.
Causal analysis.
Consensus process.
Delphi process.
Nominal group technique.
Problem-solving process.
21.1.3 Leverage the Software Process Assets
Management procedures for software development were derived from the prime contractor standard software management process. Both prime and subcontractor organizations held an SEI Level 3 rating.1 As the project team found out, some of the assets of an SEI Level 3 organization enable development of risk management practices:
1 There is not a correlation between SEI software process maturity levels and risk management evolutionary stages.
Engineering process group. A separate process group that provides a focus for process definition and process improvement.
Process asset library. A repository that maintains hard copy and soft copy resources such as a template for a risk management plan, risk assessment handbook, and historical project risk databases.
Training material. A course that teaches managers and developers a common vocabulary and a standard process.
Several key process areas in the SEI CMM require risk management practices [Paulk93]. The prime contractor organization had enhanced division risk management practices through a software risk management action team in order to comply with SEI Level 3. The standard risk management process was subsequently trained in a software project management course.
21.1.4 Build on Previous Risk Management Experience
The prime and the subcontractor organizations had previous experience using risk management techniques. Projects that were relevant to this one showed prior success through the use of various risk management strategies. The past performance of these projects provided documented evidence of an assortment of experiences using risk management.2 Table 21.2 shows the organizations’ previous risk management experience. The results from use of risk management on these ten projects were favorable. Some of the risk management practices were contingencies and workarounds using common sense. Others show the implementation of more progressive strategies, which achieve team goals through collaboration of people.
2 Of the prior projects, 80 percent were performed by the prime contractor.
Table 21.2 Previous Risk Management Experience
21.2 Risk Assessment Results
A risk assessment was performed early in the project by an assessment team consisting of three members of the organization SEPG. At the participants’ briefing, risk was explained as an uncertainty with defined probability and impact. The benefits to cost and schedule through risk management would be potential problem avoidance and opportunity awareness. Project participants were informed of their SEPG point of contact and that there would be a risk champion on every project. The policies, procedures, methods, and training available for risk management were explained. The diagrams for both the risk assessment and risk management processes were reviewed. The SEI risk taxonomy [Carr93], risk evaluation criteria, and risk assessment schedule were given as part of the briefing handouts to the participants. Risk assessment ground rules were discussed as follows:
Be on time. You cannot participate in the group if you arrive late. Disruptions affect the systematic approach to risk discovery and are unfair to those who are on time. Ask the assessment coordinator to replace you if you cannot be on time.
Stay in assigned group. Do not switch groups or send alternates. If you cannot attend, you should notify the assessment coordinator as soon as possible.
Do not attribute risk to people. Do not discuss or attribute risk items surfaced in the meetings with others.
Submit risk appraisal. Understand that the risk groups are a representative sample of the project personnel. The remaining individuals are not being ignored. Everyone should submit risk appraisals now and throughout the life of the project.
21.2.1 Involve the Project Team
Forty-five participants were interviewed in ten peer groups, each with three to five people. Approximately 50 percent of the entire team participated, including customer and subcontractor representatives. The original risk assessment schedule is shown in Table 21.3.
21.2.2 Analyze the Distribution of Risk
Each individual identified six or more risks. The number of risks identified in each peer group ranged from 16 to 41, with an average of 28 risks per interview session. The project participants scored the risks using the risk evaluation criteria shown in Table 21.4, and the risk assessment team classified the risks according to the SEI software risk taxonomy. The risk frequency distribution by taxonomy element is shown in Figure 21.1.
Table 21.3 Risk Assessment Schedule
21.2.3 Report the Sources of Risk
In response to questions raised by the project participants during the interview sessions, the term risk was further defined in the assessment debriefing:
A risk is a potential problem.
Table 21.4 Evaluation Criteria for Risk Scoring
A risk is an uncertainty with a probability and impact.
A risk may be an existing problem that is allowed to continue.
A risk may be failure to take advantage of an opportunity or improvement.
Eleven viewgraphs captured the major risk categories with specific risks and associated database log numbers. Significant sources of risk were reported in the following areas:
Schedule. Waterfall management and spiral development.
Process. Inconsistent awareness and understanding, follow-through, and commitment.
Tools. New, unfamiliar, and unproved.
Meetings. Too many, conflicting and overlapping, agendas, interruptions, action items not tracked.
An experienced risk assessment team knows to present both strengths and weaknesses. By observation and recording participant comments, the assessment team was able to report more strengths than sources of risk. Sources of strength were noted as follows:
Team effort in risk assessment interviews. Good communication and support for each other.
Commitment to ensure quality products.
Figure 21.1 Frequency distribution by taxonomy element. The total number of risks identified in each of 12 elements of the SEI software risk taxonomy highlights the need for risk management in these areas.
Enthusiasm to apply new process and methods.
Innovative process and technologies expanding customer and contractor capabilities.
Commitment to provide the team with tools and training.
Abundant system and software engineering experience.
Existing risk management, as evidenced by a documented process, commitment, and implementation (management review board and project tracking).
21.3 Risk Manager
A risk manager has responsibility for coordinating risk management activities. The part-time position of risk manager (about 25 percent of a full-time position) was assigned to a senior technical person on the project. Over a three-year period, the same risk manager championed the risk management process. According to the risk manager, “I have seen it go from a project manager who did not understand what this whole process was about, to a project manager who uses it as a central element of how he manages each of his task leaders.” The key ingredient to the project’s success over the three years was the central focus provided by assigning the role of risk manager.
The view of the risk manager is that risk management is a systems engineering process required by the customer for project control. In the risk manager’s opinion, the role of a risk manager is needed to achieve objectives in the following areas:
Methods for risk management. Provide methods for systematic identification, mitigation, and tracking of long-term issues. The idea behind risk management is to look into a project and see where the sensitive areas are—likely places for problems—and to do something about them.
Focus for management. Provide the project management team with a tool to focus on immediate project problem areas. The engineering staff generates risk information largely by their work in designing the system. Risk management flushes this information up to the management team, where the process is addressed.
Insight for the customer. Provide customer insight into problem resolution. Management and the customer are always after insight as to how the project is doing. Risk management offers insight into technical performance.
Trend information. Look at where your estimate at completion (EAC) trend is going. Very seldom do we have enough knowledge to say we know what the end item cost of the project is going to be. The cost climbs because of problems that need to be solved, and that costs money. This process helps provide trend information.
In general, the risk manager must be able to command respect from senior-level management and technical individuals for them to listen to the judgment of the risk manager. The risk manager should be a senior-level technical person with a broad enough background to understand all the variation of risks that come up and have the ability to discuss these with the people involved. The risk manager does not need to be an expert, because he or she is not responsible for mitigating the risk.
The risk manager on the project worked for the chief systems engineer, who did not have the time for the role of risk manager. The risk manager was a conduit back to management on the 20 to 30 risk issues that management had neither the attention nor the time to chase individually. The risk manager worked with the subcontractor and customer as much as anyone else on the project. The risk manager was given the responsibility and authority for administering the risk management process. This responsibility included ensuring:
Complete documentation. Document the definition of risk coherently on a risk management form.
Consistent evaluation. Evaluate each risk consistently to enable comparison with other risks.
Correct mitigation plan. Develop a mitigation plan to reduce the probability and consequence of a risk that is accurate and makes sense technically. Back up the numbers presented by supporting data or rationale.
Coordinate responsibility. Tag the risk to someone to ensure that none of it becomes lost. Hold those who are responsible accountable for risk management.
From the risk manager’s perspective, the implementation of risk management on the project posed a significant number of problems, most having to do with how the engineering teams themselves viewed risk management and went about accomplishing it. In this section, the risk manager describes antidotes that provide an understanding of the practices and problems that were encountered.
21.3.1 Focus the Integrated Product Team
The project was organized as an integrated product team, with risk identification a responsibility of the system engineers, project leaders, and project management as a group. The risk management responsibility for each job category was as follows:
Management review board. The management review board reviewed all the risks on a monthly basis.
Customer. The customer was part of the review process, not separate from anyone else in terms of looking at the data. The customer saw all of the assessments and understood all of the risks. Some risks were under the customer’s domain (e.g., providing the support services needed for test).
Project manager. The project manager provided direction back to the team as to what to do about the risk issues. The project manager used the process to focus resource allocation.
Technical leaders. Technical leaders provided the basic information to the management review board, including where the team was going with the risks and how they were doing with risk management.
Project engineers. The responsibility of resolving risk rested with the project engineers assigned to a technical task and the project manager who controlled the entire project.
21.3.2 Describe Risk Management Practices
The project risk management process can be described as a set of activities to identify and resolve risks. The risk manager described the risk management process in five steps:
1. Risk identification. We think of risk identification as ongoing. You identify risks whenever you can. As you learn more and more about the project over time, you certainly identify the risks. Any person, at any time, can submit a risk item to the system engineer by completing the risk management form. Project leaders also maintain a list of watch items. Watch items are issues that tend to be long term and may be resolved by the normal engineering process. The other thing you should do is a survey. Most projects ask you to do a survey at the start. We found that periodic surveys are required. We conduct a survey every six months or upon major engineering change proposal or redirection.
2. Risk analysis. We ask project leaders to assess the impact of risk and put it down on paper using the risk management form. We evaluate watch items monthly to determine if they have diminished, or if they should be escalated to a project risk level. The management team reviews project risk on a monthly basis.
3. Development of a mitigation plan. The project management and system engineer develop an action plan. The action plan identifies alternatives and selection criteria if appropriate. The action plan identifies the person responsible for implementing the action plan. The action plan itemizes additional resources necessary to perform the plan. The management team reviews action plans that are documented and summarized on a risk management form.
4. Mitigation plan execution and tracking. Once you have a risk that you understand, you have to look at it on a monthly basis. This is a conversation between the project manager and the task manager that owns the risk—not a conversation between the engineer and engineer, because they should be doing what they are told. The project manager needs to know exactly how the mitigation plan is turning out, what is happening to that risk. Is it getting bigger or getting smaller? Then you have the plan adjustments, because the risk goes up and down as you look at it over time. We track about 15 to 25 risks on the watch list, but the focus is on the top ten. The rest migrate up or down over time.
5. Risk closure. You close a risk when it is mitigated or when circumstances are modified so the risk is redefined or eliminated. When the risk is mitigated as far as possible but some residual impact exists, project management determines the impact to project EAC. Unmitigated risk impacts EAC. Of course, there are always instances in which EAC does increase as a result of closing out a risk. Then it is there—that is why you have overspent. We try to resolve small risk in two months, medium risk in four months, and huge risk in about six months. Any longer than that, there is something major wrong, and you are probably in a situation where your whole project plan is in jeopardy.
21.3.3 Describe Risk Management Pitfalls
Early on, the project management did not understand how to use the data and how to make the risk be a central part of the way they managed. As time went by, they found problems associated with the risks they had identified and started seeing that their management efforts were focused around resolving these risks. The risk manager observed, “Within the last year and a half, project management has been very committed to holding cost and schedule, and the only way they are going to do that is risk management.” Some of the problems and how the project addressed them are described below.
Scaling the Spiral Model. Software development began with an object-oriented analysis that took about six months. The engineers wrote a preliminary set of requirements based on the results of that analysis. The requirements for the software and hardware components took almost two years to negotiate. The whole spiral was not implemented because they never completed the requirements phase. The project threw out their adaptation of the spiral model3but maintained the orientation of risk as a driving factor in system development.
3 The spiral model was originally developed for an internal research and development project [Boehm96]. Difficulties with the original spiral model led to the WinWin Spiral Model, which provides for identification of stakeholders and negotiation of win conditions [Boehm97].
Believing in the Process. If a group in the project believes risk management is just a bunch of paper, they are only going to pay lip-service to the process. Fundamentally, three things happen: the discipline goes away, the organization is uncovered for major problems, and insight into that portion of the project is lost. The project manager must use this whole process to focus on how to apply project resources. If the project manager does not believe in risk management, everyone is wasting time. You get out of it what you put into it.
Overreacting to Risk. Risk management offers insight into technical performance and provides a sense that you are telling about problems that you have. The reaction to risk can often exacerbate the problem rather than help to solve the problem. Without sensitivity in discussing risk, you can get a project in trouble, get your customer in trouble, and get yourself in trouble. The way to discuss risk with the customer has to be in terms of a forum where everybody understands the definition of all of the terms.
Lacking Control of Risk. Risks do not include items that go beyond the contract. If something outside your control is affecting the project, one response is to tell the customer that the project does not have the resources to go beyond the contract. Let the customer decide whether to attack these risks.
Tracking Long-Term Issues. One of the major problems with the risk management process is how long-term issues are tracked. Early assessment of risk means not knowing whether that risk is going to take place. The project used a watch list. People were wasting so much time looking at things that were a low probability of occurring that the management team’s confidence began to drop. The watch list effectively tracked issues over the long term.
21.4 Risk Practice Survey
Every new process should be piloted and evaluated for improvement, and risk management is no exception. A risk management survey was given to those who participated in the baseline project risk assessment. The purpose was to improve risk management practices through feedback from individuals using the risk management process. The project manager was interviewed to describe the use of risk management and to approve the survey for distribution and collection. To maintain anonymity, distribution and collection of the surveys was performed by the project manager’s secretary.
21.4.1 Categorize Survey Participants
The risk management survey was designed to assess the state of the practice in risk assessment and risk management by asking survey participants to identify their perceptions of the performance and importance of risk practices on the project. Only 2 percent of the surveys were returned initially. After the project’s engineering director sent an e-mail message requesting project support, the return climbed to 42.8 percent (18 survey respondents). On the average, the respondents had over 11 years of experience on software projects. Figure 21.2 illustrates the survey participants’ roles as a percentage of the total surveyed.
21.4.2 Summarize Risk Practice Performance
Figure 21.3 charts the team’s perception of the importance and performance of risk management practices. Importance was rated significantly higher than project performance, indicating the perceived need for risk management in all process elements.
Figure 21.2 Risk management survey participants.
Figure 21.2 Risk management practice. On a five-point scale, the importance of risk management practices is rated higher than the corresponding practice performance.
21.5 Risk Practice Observations
Risk management survey results included responses to five open-ended statements. Following are some of the comments:
1. Some effective risk management practices are:
“Identify risks consistently and constantly. Just having a risk program in place is a major step in the correct direction.”
“Identification and documentation of well-known risks.”
“Management review board.”
“Regular activities planned to evaluate risks and identify new ones. Includes risk management in project plans, test plans, and transition plans.”
“Tracking the top 10–20 project risks.”
2. Some ineffective risk management practices are:
“Identification and documentation of new risk—some have not been brought to the project level.”
“Risks tend to be identified bottom up. Systems-level architectural risks are not visible.”
“Leaving it up to only a few assigned ‘experts’ on the project.”
“Since the IPTs do not see a direct benefit (and they are pressed with other duties) the risk management is seen as a hoop to jump through. This has resulted in risks not being presented because the process levies what is seen as extra work.”
“Focus on development almost exclusively for nominal flight conditions. Worst-case scenarios are important to the development of robust software and are effective at revealing faulty and/or inadequate designs.”
3. One thing needed in my work group is:
“Some better ways of quantifying the real risks identified.”
“For the integrated product teams to address and restructure risks.”
“More visibility into project risks. More input into risk identification and quantification, especially software.”
4. The risk management lessons learned on the project are:
“Must be more proactive in seeking out risks.”
“Just having a risk management process is not enough. It has to be used and dynamic.”
“Be careful about tightly coupling project contractual dollars to risk assessments. Identification of ‘potential’ overruns makes everyone uneasy.”
5. Other comments regarding software risk management are:
“It [risk management] needs to be generated and used below the systems level.”
“Overall good job identifying risks, not so good doing something about them.”
“Risk management statements apply to the project and system level now. Software-specific definitions will not start for several months.”
“Risk management is much more effective than management by objective, and a definite step in the right direction. It focuses all on the important problems at hand much earlier in the life cycle than other techniques. More efficient means are needed to collect and evaluate metrics.”
21.6 Summary and Conclusions
In this chapter, I described the practices that characterize the prevention stage of risk management evolution. These practices include a documented project risk management plan, written to satisfy customer requirements for risk management. Risk identification occurs at any time using a risk management form. Risk assessment is performed monthly at project management reviews, at six-month intervals using a survey method, and occasionally at an all-hands meeting off-site. Risk is defined in terms of low, medium, and high risk. Risk mitigation plans are documented and assigned to a responsible person. A watch list is maintained that tracks long-term risk. Risks are assigned to the technical leaders, and a risk manager coordinates action items among the various functional groups within the integrated product team. Quality management techniques are reused in risk management to determine the root cause of risk. The organization provides risk management process training as part of a software project management course. A risk practices survey is used to capture feedback for process improvement.
The primary project activity to achieve the prevention stage is the systematic application of a risk management process. Involvement of the people and root cause analysis are two quality principles that are used to prevent risk. Understanding the source of risk is the key to risk prevention. The involvement of the entire team is necessary to identify risk in all project areas.
At the prevention stage, we observed the following:
A risk management champion. A senior technical person fills the part-time position of risk manager to coordinate the risk management plan implementation and act as a catalyst for risk management activities. The risk manager has the authority and responsibility to ensure the integrity of the risk management process.
Process discipline. The risk management process is a tool for insight into implementation problems that may affect the estimated project cost at completion. To achieve insight, all project elements must participate. Management supervision is important to force process consistency.
Problem prevention. The project manager knows that risk management can be used to avoid problems. Reflecting on this fact, the risk manager said, “The project manager believes in it. There is no guarantee, except that you know where most of your problems are.”
21.7 Questions for Discussion
1. List five techniques of a quality philosophy that can be reused in risk management. Describe the advantages of each technique.
2. In what ways does experience in risk management increase the likelihood of success in managing risk? How likely do you think it is to expect success in managing risk the first time you try?
3. In your own words, define the term risk.
4. Summarize three methods for risk identification that would be cheaper than a formal risk assessment interview session. Include the percentage of team involvement using each method.
5. Improvement means better, faster, cheaper—among other qualities. How would you improve the formal risk assessment method in each of these three areas? List the other ways you could improve the method.
6. Discuss the advantages and disadvantages of a project risk manager position.
7. As an individual contributor and team member, you have a risk that is outside your control. Describe the risk, and discuss how you will handle the situation.
8. Give five reasons that it is important to obtain feedback from people. How could you increase the percentage response using a survey method to obtain feedback?
9. Why is risk prevention cheaper than problem detection? Discuss the consequence of risk prevention. Discuss the consequence of problem detection.
10. Do you agree that you should actively attack risk? Discuss why you do or do not agree.
21.8 References
[Boehm88] Boehm B. A spiral model of software development and enhancement. IEEE Computer, 21(5): 61–72, 1988.
[Boehm96] Boehm B. Software Management & Economics. Computer Science 510 Video lecture. Los Angeles, CA: University of Southern California, 1996.
[Boehm97] Boehm B, Egyed A, Kwan J, and Madachy R. Developing Multimedia Applications with the WinWin Spiral Model. Proc. Sixth European Software Engineering Conference and Fifth ACM SIGSOFT Symposium on the Foundations of Software Engineering, Zürich, Switzerland, September 1997.
[Carr93] Carr M, Konda S, Monarch I, Ulrich F, Walker C. Taxonomy based risk identification. Technical report CMU/SEl-93-TR-6. Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon University, 1993.
[Gilb88] Gilb T. Principles of Software Engineering Management. Reading, MA: Addison-Wesley, 1988.
[Paulk93] Paulk M, et al. Capability Maturity Model for Software. Version 1.1. Technical report CMU/SEI-93-TR-24. Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon University, 1993.