12 Verify Compliance
The goal of competitive industry is to provide a product and service into which quality is designed, built, marketed, and maintained at the most economical costs which allow for full customer satisfaction.
—A. V. Feigenbaum
W. Edwards Deming proved that improvement in quality starts a chain reaction that yields an increase in productivity. But as scientist Jonah told plant manager Alex Rogo, “Productivity is meaningless unless you know what your goal is” [Goldratt86]. The goal of managing risk has an intermediate objective of verifying compliance of project practices to the risk management plan, a way to engineer quality results. This intermediate objective is necessary to overcome the obstacle of a faulty plan or deficient practices. The objective of verifying compliance is to determine improvement potential of the plan and of the practice. The distinction between verifying compliance and improving process is the difference between short-term and long-term advantage. In the beginning of a project, “verify compliance” precedes “improve process.” We cannot improve a process that has not been adequately planned or implemented correctly. In this chapter, we discuss improvement of the plan and the work, and not the process. Deming said, “It is not sufficient to improve processes. There must also be constant improvement of design of product and service” [Deming86].
Quality provides customers with products and services that fully satisfy their requirements. Quality assurance (QA) is the practice of ensuring that quality standards are met through quality control, which consists of the methods by which quality is measured, reported, and improved. The majority of quality assurance practices today are reactive, aimed at detecting and correcting problems that already exist [Kolarik95]. There is, however, a new quality philosophy, directed at problem prevention, which I call proactive quality assurance. Proactive QA takes a broad perspective and is not limited to reducing the number of software defects through inspection of source code. This strategy requires an emphasis on cause-effect knowledge, risk analysis, experience, and judgment to justify action. Proactive QA can lead to accelerated development cycles and avoidance of losses, advantages that contribute toward quality for the customer and thus yield a more productive environment. An important customer of proactive QA practices is project management.
In this chapter, I describe how to verify compliance of risk management activities through an independent audit. I describe how to ensure that project practices adhere to a documented risk management plan.
This chapter answers the following questions:
What are the steps to verify compliance to a risk management plan?
What are three major goals of quality assurance?
Which standards provide guidance for quality assurance?
12.1 Review the Risk Management Plan
There is a cause-and-effect relationship between the quality of a plan and the quality of the results. For this reason, we begin an investigation into results by reviewing the plan of activities. The first step in verifying compliance to risk management practices is to review the risk management plan in order to understand the activities, agents, and artifacts of the plan to prepare for a compliance audit. Activities are the risk management practices expected to be performed by the project personnel. Agents are the project roles with responsibility for risk management activities. Artifacts are the expected outputs produced by performing risk management.
The plan should satisfy the following elements, established with the help of QA personnel through participation on process action teams:
Completeness. Do the contents consider all aspects of risk management? Use an outline of a risk management plan as a checklist. Initial the checklist when the plan is complete with respect to the major sections of the outline.
Understandability. Is the plan easy to read and comprehend? Perhaps a glossary is necessary so that new employees or subcontractors can interpret the plan as intended.
Level of detail. Is the level of detail sufficient to execute the plan? A detailed plan specifies what will be done, when, by whom, and how much it will cost. If these aspects of the plan are not clear, the plan needs additional detail.
Consistency. Is the plan ambiguous? Look for any contradictions that would confuse the implementation of the plan. For example, inconsistent terminology in the plan can cause people to have difficulty communicating about risks.
Realistic. Is the perspective of the plan practical? Any plan that claims, “Everyone on the project will continuously perform risk management,” is not realistic. Check for altruistic statements that lack common sense.
12.2 Audit Agents and Artifacts
When quality is vital, some independent checks are necessary—not because people are untrustworthy but because they are human [Humphrey89]. Quality assurance can be effective when competent professionals report through an independent chain of command and support the development of product quality. On large projects, managers need help performing the task of quality assurance. On projects that cannot afford to staff a quality organization, people can monitor each other’s work. On small projects, managers can perform the role of quality assurance. On really small projects, quality assurance can be a part-time role. Quality assurance monitors its own organization to ensure that established standards and procedures are followed. Its prime benefit to management is the assurance it provides them that directions are actually implemented.
If you want a high-quality software system, you must ensure that each of its parts is of high quality [Humphrey95]. Auditing agents and artifacts will help to uncover potential problems. Quality assurance is responsible for auditing the quality actions of agents (e.g., project personnel) and alerting management to any deviations. Quality assurance audits the quality of artifacts (e.g., process evidence) to ensure management that the work is performed the way it is supposed to be. Table 12.1 contains a set of audit questions for agents, artifacts, and activities to investigate risk management practices, developed by twenty QA professionals from the Defense Logistics Agency (DLA) during software risk management training. (These questions are worth sharing, because they may help prepare you for a government audit of your risk practices.)
Three industry and government standards require quality audits: ISO 9001, MIL-STD-498, and SEI CMM.
The purpose of ISO 9001 is for external quality assurance [ASQC94]. Guidelines for the application of ISO 9001 to the development, supply, and maintenance of software are detailed in ISO 9000–3 [ISO91]. This standard is for use when you must ensure conformance to specified requirements during design, development, production, installation, or servicing. ISO 9001 requires a quality plan that implements a quality management system for a project. The quality plan is the basis for project monitoring by reviews and audits. ISO 9001 emphasizes process management [Jenner95]. ISO 9001 describes Internal Quality Audits (clause 4.17) as an integral part of the input to management review activities. This clause requires an organization to establish and maintain documented procedures for planning and implementing internal quality audits. The procedures help to verify whether quality activities and related results comply with planned arrangements and determine the effectiveness of the quality system. Where you identify nonconformances to the quality management system, you should recommend corrective action. The results of audits are communicated to management, and any deficiencies found are corrected in a timely manner. Followup audit activities verify and record the implementation and effectiveness of the corrective action taken.
Table 12.1 Dla Audit Questions for Agents, Artifacts, and Activities
The purpose of DoD MIL-STD-498 is to establish uniform requirements for software development and documentation [DoD94]. This standard implements the development and documentation processes of ISO/IEC DIS 12207. It interprets all applicable clauses in MIL-Q-9858A (Quality Program Requirements) and ISO 9001 (Quality Systems) for software. MIL-STD-498 is the military standard for software development and documentation that supersedes DoD-STD2167A, DoD-STD-7935A, and DoD-STD-1703. This standard requires software quality assurance (SQA) as ongoing evaluations of activities and resulting products to ensure that each activity is being performed according to the plan. It requires that the persons responsible for ensuring compliance with the contract shall have the resources, responsibility, authority, and organizational freedom to permit objective SQA evaluations and to initiate and verify corrective actions.
The purpose of SEI CMM for software is to describe the key elements of an effective software process [Paulk93]. The CMM describes an evolutionary improvement path from an ad hoc, immature process to a mature, disciplined process. The SEI CMM describes the auditing process of software quality assurance at Level 2. Software quality assurance involves reviewing and auditing the software products and activities. It verifies compliance with applicable procedures and standards and provides the software project and other appropriate managers with the results. The verifying implementation common feature in each key process area (KPA) describes the specified auditing practices to ensure compliance for that KPA. The software quality assurance KPA goals are as follows:
1. Software quality assurance activities are planned.
2. Adherence of software products and activities to the applicable standards, procedures, and requirements is verified objectively.
3. Affected groups and individuals are informed of quality assurance activities and results.
4. Noncompliance issues that cannot be resolved within the software project are addressed by senior management.
12.3 Generate an Audit Report
An audit report provides visibility into project risk management performance. It is generated to document the review and audit findings. The project audit findings summarize implementation performance and detail any discrepancies against the risk management plan. The report should show if requirements have been achieved and the nature of any nonconformance. The quality standards discussed in the previous section provide different views on the content of an audit report. The differences between the ISO and SEI standards are sufficient to preclude a rigid mapping, but the similarities provide a high degree of overlap [Paulk95]. Each standard supports a proactive quality assurance role on the project.
ISO 9001 recommends preventive action. In clause 4.14, Corrective and Preventive Action, ISO 9001 requires an organization to identify the causes of a nonconforming product. Corrective action attempts to eliminate the causes of actual deviations. British training literature interprets corrective action as addressing the noncompliance issues identified in an audit [UKBCS92]. Preventive action attempts to eliminate the causes of potential nonconformity’s.
MIL-STD-498 requires analysis to detect trends in reported problems. MILSTD-498 addresses the evaluation of corrective actions to determine whether problems have been resolved, adverse trends have been reversed, and changes have been correctly implemented without introducing additional problems.
SEI CMM requires SQA to review and audit activities and work products for defect prevention and report the results. The CMM’s Software Quality Assurance KPA notes that compliance issues are first addressed within the software project and resolved there if possible. For issues not resolved within the project, the SQA group escalates the issue to an appropriate level of management for resolution.
12.4 Track Action Items
Quality assurance is responsible for tracking audit action items until closure. The quality system should require a timely response to action items. All quality standards require maintenance of records of each activity to verify compliance. Records may be in the form of any type of media, such as hard copy or electronic soft copy. In clause 4.16, Control of Quality Records, ISO 9001 requires identifying, collecting, cataloging, filing, and maintaining all records relating to the quality management system. These records need to be managed so that they can be easily retrieved to provide evidence that the quality management system is being used and that all its requirements are being satisfied. MIL-STD-498 requires preparation and maintenance of records for each SQA activity. These records are maintained for the life of the contract. In the SEI CMM, the practices defining the maintenance of quality records are distributed throughout several key process areas. The Software Project Tracking and Oversight KPA requires action items to be assigned, reviewed, and tracked to closure. This activity is reviewed with senior management on a periodic basis.
12.5 Summary
In this chapter, I described how to verify compliance of risk management activities through an independent audit. The steps to verify compliance to a risk management plan are as follows:
1. Review the risk management plan.
2. Audit agents and artifacts.
3. Generate an audit report.
4. Track action items.
I described three major goals of quality assurance:
Ensure compliance. Conduct independent reviews and audits. Check the plans and work against established standards by auditing the evidence.
Report discrepancies. Alert management to deviations from standards by reporting audit findings. Expose deviations from standards and procedures as soon as possible.
Monitor quality. Improve quality by making recommendations to prevent problems and tracking action items to closure.
I described three standards that provide guidance for software quality assurance:
ANSI/ISO/ASQC Q9001-1994, Quality Systems—Model for Quality Assurance in Design, Development, Production, Installation, and Servicing.
DoD Military Standard MIL-STD-498.
SEI Capability Maturity Model for Software.
12.6 Questions for Discussion
1. Compare and contrast reactive and proactive quality assurance.
2. Discuss why productivity is meaningless unless you know what your goal is.
3. Explain how verifying compliance of practices to plans is a way to engineer quality results.
4. Explain why you must verify risk management implementation before you improve the risk management process.
5. List five artifacts of performing risk management.
6. In your opinion, what are the attributes of a high-quality risk management plan?
7. Do you think that quality assurance professionals can be effective when they do not report through an independent chain of command? Discuss why you do or do not think so.
8. List five responsibilities for the project role of quality assurance.
9. Discuss how to ensure the compliance of quality assurance practices.
10. Do you agree that the goal of competitive industry is to provide quality products and services at the most economical costs? Discuss why you do or do not agree.
12.7 References
[ASQC94] American Society for Quality Control. Quality Systems—Model for Quality Assurance in Design, Development, Production, Installation, and Servicing. Milwaukee, WI: ANSI/ISO/ASQC Q9001–1994.
[Deming86] Deming W. Out of the Crisis. Cambridge, MA: MIT Center for Advanced Engineering Study, 1986.
[DoD94] Department of Defense (DoD). Software Development and Documentation. DoD Military Standard MIL-STD-498, AMSC NO. N7069, December 1994.
[Feigenbaum83] Feigenbaum A. Total Quality Control. Third ed., New York: McGrawHill, 1983.
[Goldratt86] Goldratt E. The Goal: A Process of Ongoing Improvement. Sarasota, FL: SMS, 1986.
[Humphrey95] Humphrey W. A Discipline for Software Engineering. Reading, MA: Addison-Wesley, 1995.
[Humphrey89] Humphrey W. Managing the Software Process. Reading, MA: Addison Wesley, 1989.
[ISO91] ISO 9000–3. Quality Management and Quality Assurance Standards—Part 3: Guidelines for the Application of ISO 9001 to the Development, Supply and Maintenance of Software, 1991.
[Jenner95] Jenner M. Software Quality Management and ISO 9001. New York: Wiley, 1995.
[Kolarik95] Kolarik W. Creating Quality: Concepts, Systems, Strategies, and Tools. New York: McGraw-Hill, 1995.
[Paulk95] Paulk M. How ISO 9001 compares with the CMM. IEEE Software, January, pp. 74–83, 1995.
[Paulk93] Paulk M, et al. Key practices of the Capability Maturity Model, Version 1.1. Technical report CMU/SEI-93-TR-25. Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon University, 1993.
[UKBCS92] United Kingdom. TickIt: A Guide to Software Quality Management System Construction and Certification Using EN29001. Issue 2.0. London: UK Department of Trade and Industry and the British Computer Society, 1992.