13 Improve Practice

The majority of men meet with failure because of their lack of persistence in creating new plans to take the place of those which fail.

—Napoleon Hill

We often rely on a system of checks and balances to maintain satisfactory process results. For example, we rely on the president, the Congress, and the Supreme Court to maintain democracy in our country. Laws are amended through a legal process designed to ensure liberty and justice. The U.S. Constitution has endured due to this system of checks and balances and the flexibility to create new laws to take the place of those that are outmoded. Similarly, a system for managing risk requires checks and balances to sustain the practice over time. We must periodically check our risk management practice for potential improvement. To obtain feedback for improvement, you can survey people’s perception of their own practice. Collectively, the people should decide what needs improvement.

In this chapter, I provide an appraisal method to improve your risk management practice. I describe how to use a risk practices survey to measure your progress in managing risk quantitatively.

This chapter answers the following questions:

What are the steps to improve your risk management practice?

Why is importance the key to performance?

Why is it necessary to improve quantitatively?

13.1 Develop an Appraisal Method

There are various reasons for assessing risk management practices. A structured appraisal method is used for the following reasons:

To report risk management capability.

To establish a baseline for improvement.

To develop a plan for improvement.

To measure progress against an improvement plan.

To select a contractor or subcontractor.

To monitor practice performance.

13.1.1 Design a Risk Practices Survey

The risk practices survey is an appraisal method to obtain perceptions of risk management practices. I developed it to provide a baseline for quantitative process improvement [Hall95b]. Quantitative process improvement (QPI) yields objective measures through statistical analysis of subjective perceptions. The risk practices survey shown in Figure 13.1 requires survey participants to identify their perceptions of the performance and importance of risk management practices. Performance is how well the practice is completed. Importance is how significant the practice is. A five-point scale is used to rate responses to each statement. Responses to five open-ended questions provide the supporting evidence for project strengths and weaknesses. The risk practices survey is inexpensive. The time to complete the survey is approximately 15 minutes. You can conduct the survey on a quarterly basis. It is also automated. After you enter survey responses into a spreadsheet, you can automatically generate a graph of strengths, weaknesses, and areas for improvement. And it is quantitative: people’s perceptions translate into objective numbers that measure risk management practices. You can easily track progress by applying the survey periodically.

13.1.2 Categorize the Survey Participants

The survey is given to all those with responsibilities for risk management. Anyone who has used a project charge number can be considered a candidate survey participant. The survey includes categories for all possible roles, to ensure adequate project representation. As shown in Figure 13.2, risk practices survey participants are a mix of management and technical personnel. Survey results are the collective knowledge and experience of the project team, organization management, and the customer. Responses are categorized by project and organization role, so that they may be compared. This particular mix of survey participants comes from data from three software-intensive projects in the early phase of development [Hall95a].

Figure 13.1 Risk practices survey.

Figure 13.2 Risk practices survey participants. The survey participants’ roles are shown as a percentage of the total surveyed.

13.2 Assess Risk Practices

There are three steps to assess risk management practices using the risk practices survey: (1) Administer the survey to the participants. (2) Analyze the quantitative survey results using a spreadsheet. (3) Use the subjective survey feedback for the reasons discussed in section 13.1, such as “to establish a baseline for improvement.”

13.2.1 Administer the Risk Practices Survey

Before administering the appraisal method, obtain permission from the responsible manager. The manager can help you obtain timely data and can characterize the project in terms of size, structure, and application domain.

13.2.2 Analyze the Risk Practices Survey Results

To analyze the risk practices survey results, enter responses into a spreadsheet that you can use to graph the survey data. Survey responses from 0 to 4 provide an ordinal ranking, but this does not measure the distance between the data points. To normalize the data, scale the data to fit a normal distribution. Map the ordinal scale 0 to 4 to fit a standard normal curve with a mean of 0 and a standard deviation of1. Count the total number of responses for each response (from 0 to 4) and divide that by the total number of responses to determine the percentage for each slice. Find the scaled value corresponding to each score from 0 to 4 by determining the mean for each slice. Table 13.1 shows the scaled values after the data were transformed. Substitute these adjusted scores for the raw scores in subsequent analysis. By normalizing the data, you enable metrics and statistical comparisons.

13.2.3 Establish a Baseline for Improvement

Plotting relative importance versus performance and the mean importance and performance provides four quadrants that categorize risk management practices. The quadrants show relative strengths and weaknesses and may be used to identify areas for improvement. In Figure 13.3, Identify Risk falls in the quadrant describing high importance and high performance, while Resolve Risk falls in the quadrant describing low importance and low performance. Importance is the key to performance because we prioritize activity based on importance. The consequence of improperly valuing importance is that performance will likely suffer in proportion. The gap between performance and importance shows the need for risk management practices.

13.3 Develop an Improvement Plan

An improvement plan should define specific areas to be improved. To develop a realistic improvement plan, you must understand the difference between “as is” and “should be.” Determine your current “as is” practice by substantiating the objective¹ baseline using specific examples from the subjective² survey responses. The practice “should be” is detailed in the Risk Management Map (see Chapter 3). You can develop an improvement plan based on the difference between “as is” and “should be.” Be sure to adjust the plan for your set of constraints.

¹ A statistical average is objective, assuming that the sample surveyed is representative of the project.

² Individual observations from the risk practices survey are subjective.

Table 13.1 Data Transformation For Statistical Analysis

Figure 13.3 Perceived strengths and weaknesses of a risk management process.

13.4 Implement the Improvement Plan

Management should assign responsibility to implement the improvement plan. To execute the plan, involve people on projects as required to promote buy-in from the organization. Improvement plans should focus on the evolution of risk management technology that will be leveraged to satisfy the project’s risk management needs.

13.5 Summary

In this chapter, I described the steps to improve risk management practice:

1. Develop an appraisal method.

2. Assess risk practices.

3. Develop an improvement plan.

4. Implement the improvement plan.

Importance is the key to performance because we prioritize activity based on importance. To improve the performance of risk management practices, we must first understand the value of the practice.

It is necessary to improve quantitatively, because only then can we use metrics and statistical comparisons. Over time, quantitative results characterize progress and trends in performing risk management.

13.6 Questions for Discussion

1. Do you agree that the people who use a process should decide what needs improvement? Discuss why you do or do not agree.

2. What steps would you take to improve the risk management practice on your project?

3. List three reasons for using a structured appraisal method to assess risk management practices.

4. How could a person’s job category be used to analyze a risk practices survey? In what way would comparisons of results by job category serve as a check and balance on the findings?

5. You are a process consultant hired to assess an organization’s risk management capability. What could you do to ensure the integrity of the data from a risk practices survey?

6. Discuss the value of normalizing risk practices survey data.

7. You are a process group member responsible for improving risk management. The quantitative baseline of risk management practices shows that performance lags importance by at least a point. What does this tell you about the need for improvement?

8. You are a member of a process action team responsible for developing an improvement plan for risk management practices. Both performance and importance of risk management practices are consistently low. What is your strategy for improvement?

9. Explain how you would develop a realistic improvement plan. What constraints could cause you to adjust your plan?

10. How can you determine if a plan is failing? What would you do if your plan failed?

13.7 References

[Hall95a] Hall E. Proactive Risk Management Methods for Software Engineering Excellence. Doctoral dissertation, Computer Science Department, Florida Institute of Technology, Melbourne, FL, April 1995.

[Hall95b] Hall E, Natwick G, Ulrich F, Engle C. Streamlining the risk assessment process. Proc. Seventh Software Technology Conference, Salt Lake City, UT, April 1995.

[Hill96] Hill N. Think and Grow Rich. New York: Fawcett Crest, 1996.