11 Software Quality Assurance
Acronym
| CMMI | Capability Maturity Model Integration |
| IEC | International Electrical Technical Commission |
| IEEE | Institute of Electrical and Electronic Engineers |
| ISO | International Standards Organization |
| PDCA | plan, do, check, act |
| PQA | product quality assurance |
| PQE | product quality engineer |
| SAS | Software Accomplishment Summary |
| SCI | Software Configuration Index |
| SCM | software configuration management |
| SEI | Software Engineering Institute |
| SQA | software quality assurance |
| SQE | software quality engineer |
11.1 Introduction: Software Quality and Software Quality Assurance (SQA)
11.1.1 Defining Software Quality
To meet the needs of the safety-critical domain, software developers must be committed to quality. High-quality products don’t just happen. They are the result of well-managed organizations with a commitment to quality, as well as talented, conscientious, and disciplined engineers.
The American Heritage Dictionary includes the following in the definition of quality: “an inherent or distinguishing character,” an “essential character,” and a “degree or grade of excellence” [1]. In software engineering, there are multiple views of what quality is—two tend to be pervasive. First is the developer’s perspective: it is a quality product if the software meets their defined requirements. The second is the customer’s perspective: it is a quality product if the software meets their needs. A product that meets its defined requirements but not the needs of the customer isn’t considered high quality by the customer. The requirements are crucial to bridging the gap between the developer’s and customer’s view of quality. The requirements must be developed to meet the customer’s needs, so that the developers can produce and verify software that meets those requirements. As noted in Chapter 6, it is important to get the customer involved in the requirements definition process. Without close customer and developer coordination during the requirements definition phase, quality is an elusive goal.
Roger Pressman writes: “You can do it right, or you can do it over again. If a software team stresses quality in all software engineering activities, it reduces the amount of rework that it must do. That results in lower costs, and more importantly, improved time-to-market” [2]. “The Quality Program is a framework for building quality into a product, doing the evaluations necessary to determine if the framework is working, and evaluating the quality actually achieved in the product” [3]. Quality is affected by activities throughout the life cycle, including requirements definition, design, coding, verification, and maintenance.
11.1.2 Characteristics of High-Quality Software
Quality attributes are often used to identify the goals of high-quality software. Such attributes include correctness, efficiency, flexibility, functionality, integrity, interoperability, maintainability, portability, reusability, testability, and usability. The International Standards Organization (ISO) and International Electrical Technical Commission (IEC) Standard 9126 (ISO/IEC 9126) define a set of quality attributes called characteristics and subcharacteristics. These characteristics and subcharacteristics are summarized in the following [4,5]:
Functionality: The capability of the software product to provide functions which meet stated and implied needs when the software is used under specified conditions. Subcharacteristics of functionality include suitability, accuracy, interoperability, security, and functionality compliance.
Reliability: The capability of the software product to maintain a specified level of performance when used under specified conditions. Subcharacteristics of reliability include maturity, fault tolerance, recoverability, and reliability compliance.
Usability: The capability of the software product to be understood, learned, used, and attractive to the user when used under specified conditions. Subcharacteristics of usability include understandability, learnability, operability, attractiveness, and usability compliance.
Efficiency: The capability of the software product to provide appropriate performance, relative to the amount of resources used, under stated conditions. Subcharacteristics of efficiency include time behavior, resource utilization, and efficiency compliance.
Maintainability: The capability of the software product to be modified. Modifications may include corrections, improvements, or adaptation of the software to changes in the environment and in requirements and functional specifications. Subcharacteristics of maintainability include analyzability, changeability, stability, testability, and maintainability compliance.
Portability: The capability of the software product to be transferred from one environment to another. Subcharacteristics of portability include adaptability, installability, coexistence, replaceability, and portability compliance.
Companies often implement metrics to measure a subset of these attributes, although many of them are actually quite difficult to quantify.
11.1.3 Software Quality Assurance
Most companies implement an SQA process to help ensure that the required quality attributes are satisfied and that the software meets its requirements. “A formal definition of Software Quality Assurance is that it is the systematic activities providing evidence of the fitness for use of the total software product” [6]. Pressman writes: “Quality assurance establishes the infrastructure that supports solid software engineering methods, rational project management, and quality control actions …In addition, quality assurance consists of a set of auditing and reporting functions that assess the effectiveness and completeness of quality control actions” [2]. SQA activities provide confidence that adequate processes are established and followed in order to produce products that meet the requirements and the customer’s needs.
The DO-178C SQA process is an integral process that runs continuously throughout the software planning, development, verification, and final compliance efforts. One or more software quality engineers (SQEs) ensure that the plans and standards are developed and followed throughout the implementation. The SQE(s) also performs a conformity review to confirm completeness and compliance.
Although DO-178C requires an SQA process, it should be noted that quality is not solely the responsibility of the SQA personnel. Additionally, software quality cannot just be assessed at the end of the project, as it is for some engineering disciplines. As William Lewis notes: “Quality cannot be achieved by assessing an already completed product. The aim, therefore, is to prevent quality defects or deficiencies in the first place, and to make the products assessable by quality assurance measures …In addition to product assessments, process assessments are essential to a quality management program” [6]. Emanuel Baker and Matthew Fisher echo the sentiment: “While evaluation activities are essential activities, they alone will not achieve the specified quality. That is, product quality cannot be evaluated (tested, audited, analyzed, measured, or inspected) into the product. Quality can only be ‘built in’ during the development process” [3]. Quality is an ongoing process and is the responsibil ity of the entire software team. DO-178C encourages quality through the use of standards and verification activities throughout the software life cycle. Quality must be built in—it cannot be audited in, policed in, or tested in.
DO-178C requires that all SQA objectives be satisfied with independence. DO-178C provides a slightly different definition for independence for SQA than it does for verification. DO-178C defines independence as:
Separation of responsibilities which ensures the accomplishment of objective evaluation. (1) For software verification process activities, independence is achieved when the verification activity is performed by a person(s) other than the developer of the item being verified, and a tool(s) may be used to achieve equivalence to the human verification activity. (2) For the software quality assurance process, independence also includes the authority to ensure corrective action [7].
The first half of the definition relates to verification independence which was discussed in Chapter 9. The second half applies to the SQA process and where it exists in the organizational structure. Although it isn’t required, most companies meet the independence requirement by having a separate SQA organization which does not report to engineering. This section uses the term SQA to refer to the group that is responsible for performing the SQA activities. SQA may be a separate organization (which is preferred) containing one or more SQEs or just a separate function within the engineering organization (this approach tends to be limited to very small companies).
DO-178C encourages a process-oriented SQA approach. SQA is primarily responsible for ensuring that the processes in the approved plans and standards are followed. However, some organizations are beginning to real ize that process alone is inadequate. Some companies are embracing the Software Engineering Institute (SEI) concept of product quality assurance (PQA), where an engineer is responsible for ensuring the quality of the product and not just the process. With the PQA approach, the product quality engineer (PQE) and SQE work closely together to ensure both product and process quality. Process evaluations demonstrate that the processes were followed; whereas product evaluations demonstrate the correctness of the process outputs. It should be noted that this has always been the concept of verification in DO-178B and now DO-178C; however, PQEs can help bridge the gap between engineering and SQA. The use of the PQE does require a PQE with domain knowledge and experience for the product being developed.
ISO has developed a number of documents to define good SQA practices, including a family of documents referred to as ISO 9000* (which includes ISO 9000, 9001, and 9004). ISO 9000 gives the vocabulary and basics of the series of standards on quality systems. ISO 9001 integrates earlier standards known as ISO 9001, 9002, and 9003. ISO 9001 is general and can be applied to any product. ISO 9001 includes the following basic elements [2]:
Establish the elements of a quality management system.
Document the quality system.
Support quality control and assurance.
Establish review mechanisms for the quality management system.
Identify quality resources, including personnel, training, and infrastructure elements.
Define methods for remediation.
Most developers of safety-critical software pursue and maintain an ISO 9000 registration. ISO 9000 registration is granted by an accredited third-party body. Surveillance occurs every 6 months. Reregistration is required every 3 years [5]. Other industry-wide organizations and standards, such as SEI’s Capability Maturity Model Integration® (CMMI®) and the Six Sigma strategy, which was originally popularized by Motorola in the 1980s, also provide a framework for SQA.
11.1.4 Examples of Common Quality Process and Product Issues
There are a variety of quality issues that might be present in a software development effort. Table 11.1 summarizes some of the common process and product issues. Obviously, most of the process issues can lead to product issues if not proactively resolved. Therefore, both process quality and product quality are important to consider.
11.2 Characteristics of Effective and Ineffective SQA
11.2.1 Effective SQA
The purpose of SQA has been explained and SQA’s activities will be explored shortly. Before examining what SQA does, let’s consider some of the prerequisites for effective SQA. “Quality is never an accident; it is always the result of intelligent effort” [8]. The following characteristics are essential to effective SQA implementation.
Characteristic 1: Top-level management support. In order for SQA to be effective, they must have the buy-in, commitment, and support of top-level management. This needs to be a sincere commitment, not just lip service.
Table 11.1 Common Process and Product Issues
| Common Process Issues | Common Product Issues |
|
|
Top-level management must be committed to the overall quality of their products and to providing the resources required to staff and train the SQA organization. This concept is strongly enforced by quality experts such as Kaoru Ishikawa, Joseph M. Juran, W. Edward Deming, and Watts S. Humphrey.
Characteristic 2: Independence. Effective SQA is independent of the development and verification organizations. Independence helps to reduce errors from extensive familiarity with the day-to-day process or product being evaluated. Additionally, independence helps to enforce corrective action when needed. Without independence, there is pressure (sometimes intentional and sometimes unintentional) to compromise when the budget and schedule demands arise.
Characteristic 3: Technical competence. In order for the SQA organization to be effective and to earn the respect of the development and verification teams, it must be staffed with well-qualified and experienced technical engineers. Without technical strength, SQA will not identify the real issues in a timely manner, and they will not be respected by the engineering organization.
Characteristic 4: Training. SQA personnel, as well as the overall engineering organization, must be properly trained regarding software quality expectations. Training should be ongoing to ensure that any improper mindset or behaviors are erased or at least minimized and that new personnel are properly indoctrinated. SQA should be trained in the following areas: software development and verification, auditing or assessment skills, and DO-178C. In teaching DO-178B (and now DO-178C) classes around the world for over 15 years, I’ve had no more than 10 SQA personnel attend. Unfortunately, most companies are not investing in training their SQA engineers, and their overall software quality reflects the lack of investment.
Characteristic 5: Continuous improvement. Organizations with good quality are always aiming to improve. Deming recommends the PDCA (plan, do, check, act) process, which is commonly referred to as the Deming Cycle or the PDCA Circle because it is a circular process. The concept is to develop a plan and then execute the plan. The activity is monitored (checked) and action is taken to improve the process. The PDCA process is ongoing in order to continuously improve the overall effectiveness of the process and the quality of the products produced.
11.2.2 Ineffective SQA
Over the years, I’ve been frustrated by the lack of good SQA processes in many organizations. In evaluating why SQA is not effective, I find it is nearly always related to a failure in one of the five areas noted earlier (i.e., toplevel management support, independence, technical competence, training, or continuous improvement). Failure in one or more of these areas leads to the following issues:
Understaffed SQA organization. If management doesn’t support SQA, they are often spread too thin to carry out the job.
Unqualified SQEs. It is challenging to attract good engineers to SQA. Many of the top engineers want to design software, rather than look at other people’s design. Therefore, SQA managers need to get creative to attract good talent. Some companies use a rotation process (using 1 or 2 year terms) as a stepping stone to project management. Some companies also use an empowering process (using the SQA personnel to train and lead the company’s commitment to quality). When people understand some of the benefits of being in SQA, they are more likely to join the team. Table 11.2 summarizes the qualifications and characteristics to look for when hiring an SQE.
Disregarded and hence ineffective SQA organization. When SQA has unqualified or ineffective personnel, engineering doesn’t respect or take action on SQA’s input.
Table 11.2 Qualifications and Characteristics of Effective SQEs
| SQE Background and Skill Set |
|
11.3 SQA Activities
This section considers the typical tasks that the SQA organization (using one or more SQEs) performs in order to ensure that plans, standards, and company processes are followed.
Task 1: Review plans. SQA reviews the PSAC, the software plans, and the development standards, considering the compliance with DO-178C objectives and the consistency between the plans [7].* (Chapter 5 provides information on the plans and standards.) Most companies hold a peer review of the plans and SQA participates in that peer review. Checklists are normally completed by both the project and SQA as part of the review effort. In addition to the software plans, SQA should be knowledgeable of the system-level and safety plans since they may impact the software processes.
Task 2: Write the SQA Plan. The SQA team is responsible for authoring the SQA Plan and keeping it updated. Most companies have a company-wide SQA Plan. However, each project may have some unique considerations which are typically detailed in a project-specific SQA Plan or in the PSAC.
Chapter 5 provided a summary of what is typically included in the SQA Plan. Additionally, other resources may be used to help identify the components of the SQA Plan. For example, the Institute of Electrical and Electronic Engineers (IEEE) Standard 730 identifies typical components for inclusion in the SQA Plan [9]. Table 11.3 summarizes the IEEE Standard 730 proposed sections.
Task 3: Approve key data items. SQA generally approves key data items, such as the plans, standards, requirements, design, verification cases and procedures document, verification report, Software Life Cycle Environment Configuration Index (SLECI), Software Configuration Index (SCI), tool qualification data, and Software Accomplishment Summary (SAS). Oftentimes, key data items cannot be released without SQA’s approval.
Table 11.3 Summary of IEEE Standard 730
Task 4: Audit life cycle data. SQA audits the software life cycle data to ensure that the processes are followed, standards are used, checklists are properly completed, and DO-178C objectives are satisfied.* Sometimes audits are performed during the peer reviews or in conjunction with the certification liaison reviews. However, SQA audits may be performed independent of other activities. Some SQA organizations use the plans and standards to develop a software audit kit; SQA then audits the processes and data using the audit kit. This approach provides detailed evidence that the project is complying with the approved plans and standards.
Task 5: Participate in reviews. At their discretion,† SQA participates in peer reviews throughout the software development and verification processes. SQA looks for technical issues, as well as the engineering team’s compliance with the peer review process, completion of checklists, satisfaction of entry and exit criteria, etc. Participation in peer reviews is a good way for SQA to stay in tune with the project’s activities and issues.
Task 6: Transition criteria assessment. DO-178C Table A-9 objective 4 states: “Assurance is obtained that transition criteria for the software life cycle processes are satisfied” [7]. This SQA objective is often addressed by SQA’s participation in the peer review process. However, SQA may have a separate assessment activity to evaluate transition criteria.
Task 7: Witness tests. SQA often plays a key role in witnessing software tests. Some organizations require a certain percentage of SQA witnessing during test execution. The expectations should be identified in the SQA Plan and clearly communicated to the project.
Task 8: Audit the environment. SQA typically audits the development and verification environments to ensure that they are consistent with the approved configuration (usually documented in the SLECI). Prior to building the software for release, SQA may witness the build machine setup (as explained in Section 8.4.1) to ensure the approved procedures are followed and that the build machine complies with the approved configuration. Prior to formal testing (i.e., testing for certification credit), SQA normally inspects or witnesses the test station setup to ensure that it complies with the approved configuration.
Task 9: Witness build and load. Using the documented build and load procedures, SQA may perform the build and load themselves or witness someone else implementing the build and load procedures in order to confirm repeatability. As explained in Chapter 8, it is best to have someone who doesn’t normally do the build and load (e.g., an SQE or another engineer) execute the procedures.
Task 10: Participate in change control board. SQA is often a key player on the change control board as they evaluate proposed changes and assess the implementation of the changes. SQA’s approval is normally required for the closure of a problem report or change request.
Task 11: Track and evaluate problem reports. SQA tracks the problem reports throughout the software development and verification effort to make sure that the problem reporting processes defined in the Software Configuration Management (SCM) Plan are followed. The problem reports are also evaluated for completeness, accuracy, and adequacy. SQA also ensures that all changes driven by problem reports (or change requests) are verified prior to closure.
Task 12: Evaluate SCM processes and records. SQA audits SCM records and processes to ensure that they comply with the SCM Plan. SQA evaluates SCM during development and verification activities. SQA ensures that the data in the configuration management system align with the data identified in the SCI; that is, they verify that the configuration management library and the SCI are complete and accurate. SQA also audits the change control process throughout the software development and verification efforts to ensure that problems are properly documented, changes are agreed upon, changes are correctly implemented and verified, and all documentation is appropriately updated.
Task 13: Audit suppliers. If the project uses suppliers or subcontractors, SQA audits the suppliers’ SQA processes, as well as their overall development and verification effort.
Task 14: Identify corrective actions. As SQA performs their tasks, they may identify required corrective actions. The corrective actions may be documented in a problem report or in a separate SQA record known as a corrective action report or something similar. In addition to identifying corrective actions, SQA needs to follow up to ensure that the actions are properly carried out. Corrective actions should be addressed prior to release of the SAS.
Task 15: Review and approve deviations and waivers. SQA reviews any deviations or waivers to approved processes to ensure DO-178C compliance and safety are not impacted. SQA also ensures that all such deviations or waivers are documented per the approved process (such as a problem report). Similarly, SQA reviews and approves any redlines to approved test procedures.
Task 16: Perform conformity review. Typically, the last step of the DO-178C compliance effort is the conformity review. Per DO-178C section 8.3: “The purpose of the software conformity review is to obtain assurances, for a software product submitted as part of a certification application, that the software life cycle processes are complete, software life cycle data is complete, and the Executable Object Code is controlled and can be regenerated” [7]. The review is typically documented in a conformity review report and ensures the following [7]:
The planned software life cycle processes have been completed and software life cycle data have been generated.
The life cycle data are traceable to the system and safety requirements that they came from.
There is evidence that the software life cycle data were produced and controlled in accordance with the plans and standards.
There is evidence that each problem report has been evaluated and has a recorded status.
Any requirements deviations have been documented and approved (typically using the problem reporting process).
The executable object code can be generated from the source code documented in the SCI, using the identified build instructions.
The approved software can be loaded using the released load instructions.
Any problem reports deferred from previous conformity reviews are reevaluated.
If any previously developed software is used for certification credit, it is confirmed that the current software baseline is traceable to the previous baseline and all changes have been approved.
Task 17: Document activities in SQA records. SQA activities are documented in SQA records to provide evidence of SQA involvement throughout the software development and verification effort. SQA records typically include the following information: what was evaluated, date, name of evaluator, evaluation criteria, evaluation status (compliance or noncompliance), findings, person(s) required to respond, due date for response, severity of the finding(s), and updates relative to the response acceptance [10]. Because SQA records are considered control category #2 (CC2) data, they need unique configuration identification. (See Chapter 10 for information on CC2.) Configuration identification is typically accomplished by including the date and the subject in the title of the SQA file.
References
1. The American Heritage Dictionary of the English Language, 4th edn. (Boston, MA: Houghton Mifflin Company, 2009).
2. R. S. Pressman, Software Engineering: A Practitioner’s Approach, 7th edn. (New York: McGraw-Hill, 2010).
E. R. Baker and M. J. Fisher, Chapter 1—Organizing for quality management, in Software Quality Assurance Handbook, 4th edn., G. Gordon Schulmeyer, Ed. (Norwood, MA: Artech House, 2008), pp. 1–34.
4. ISO/IEC 9126, Software Engineering—Product Quality Int’l Standard (International Standard, 2003).
5. H. Van Vliet, Software Engineering: Principles and Practice, 3rd edn. (Chichester, U.K.: Wiley, 2008).
6. W. E. Lewis, Software Testing and Continuous Quality Improvement, 2nd edn. (Boca Raton, FL: CRC Press, 2005).
7. DO-178C, Software Considerations in Airborne Systems and Equipment Certification (Washington, DC: RTCA, Inc., December 2011).
8. G. Gordon Schulmeyer, Chapter 2—Software quality lessons learned from the quality experts, in Software Quality Assurance Handbook, 4th edn., G. Gordon Schulmeyer, Ed. (Norwood, MA: Artech House, 2008), pp. 35–62.
9. Software Engineering Standards Committee of the IEEE Computer Society, IEEE Standard for Software Quality Assurance Plans, IEEE Std 730-2002 (New York: IEEE, 2002).
10. J. Meagher and G. Gordon Schulmeyer, Chapter 13—Development quality assurance, in Software Quality Assurance Handbook, 4th edn., G. Gordon Schulmeyer, Ed. (Norwood, MA: Artech House, 2008), pp. 311–330.
*ISO-9000 is entitled Quality Management Systems – Fundamentals and Vocabulary; ISO-9001 is Quality Management Systems – Requirements; and ISO-9004 is Managing for the Sustained Success of an Organization – A Quality Management Approach.
*This is identified in DO-178C Table A-9 objective 1. This objective was part of DO-178B Table A-1, but was moved to Table A-9 for DO-178C. It has always been the responsibility of SQA to review plans for consistency and compliance with DO-178B (and now DO-178C) objectives.
*Per DO-178C Table A-9 objectives 2 and 3, one of SQA’s objectives is to ensure compliance with the plans and standards. Auditing is one of the ways that this is carried out.
†Frequently, the SQA Plan identifies the target percentage of peer reviews the SQA will participate in. The percentage may be driven by the overall risk of the project.