After completing this chapter, you will be able to understand the:

• – utility of software audits;
• – audit of a management system;
• – software audit and problem resolution according to the ISO 12207 standard;
• – software audit process recommended by the IEEE 1028 standard;
• – assessment of type audit recommended by the CMMI^® model;
• – corrective and preventive process;
• – audit section of the SQA plan recommended by the IEEE 730 standard.

6.1 Introduction

Different types of reviews were presented in the previous chapter. This chapter is dedicated to the audit, which is one of the most formal types of reviews. We begin by providing definitions that are presented in some standards.

Different types of conformity certificates (e.g., audits) respond to different needs, such as the needs of an organization that develops software products or those of a client of a software product supplier. The independence level of the auditor as well as the cost varies depending on the audit type. The notes of the definition in the following text box indicate that there are internal audits and external audits performed by second and third parties.

There are different types of audit:

• – audits to verify the compliance to a standard, such as the audits described in International Organization for Standardization (ISO) standards such as ISO 9001 and IEEE 1028;
• – compliance audits for a model such as the Capability Maturity Model Integration (CMMI) that are used to select a supplier before awarding a contract or assess a supplier during a contract;
• – audits ordered by the management team of the organization to verify the progress of a project against its approved plan.

A mandate can be assigned to an external consultant or to members of the personnel of the organization that are not involved with the project, specifying the questions that the audit should answer, the audit schedule, the audit participants, and the format of the audit report (e.g., findings and recommendations).

We end this section with the definition of an audit as presented by the Project Management Institute (PMI^®) given that audits are often ordered by project managers themselves.

In addition to verifying compliance, it should be noted that this audit definition underlines the fact that the audit can also be done to improve organizational performance.

Remember that in the cost of quality model, audit is an evaluation practice also referred to as a detection activity: the verification or evaluation costs of a product or of a service during the different phases of the development life cycle.

Table 6.1 presents the components of detection costs.

In Chapter 1, the main software business models were presented. Audits allow for detecting defects and therefore diminish the software product development risks. Custom systems written on contract, mass market software, commercial, and mass-market firmware use audits extensively.

The software audits can be performed from several perspectives:

• – an external registrar who audits the quality system to assess its compliance for ISO 9001 certification;
• – an external auditor that audits for compliance to government standards, such as for the medical industry (e.g., standards of the Food and Drug Administration), or to certify the achievement of a CMMI maturity level;
• – an internal auditor that audits a project or software process to ensure that internal controls are adequate. This perspective, for example, can focus on security and fraud detection or simply the identification of inefficiencies. It is a way to ensure that the mandatory processes of the information system are effectively applied. These audits may also prepare an organization to comply with a law or an external regulation such as the Sarbanes-Oxley law of 2002 [SAR 02];
• – SQA that audits projects and processes on behalf of management to ensure that teams follow the mandated life cycle processes. This perspective is mainly concerned with the effectiveness and improvement of processes;
• – the management team or a project manager may also request that an audit be conducted to determine if a specific internal activity or project assigned to an external provider is complying with the contract specifications and agreed/prescribed clauses. This audit perspective is performed at a specific time or milestone to verify if the work is progressing according to plans;
• – a professional designation, such as the board of professional engineers of California, may decide to audit a professional to determine if his work meets its commitments to the code of ethics and that the engineer correctly applies, through his services, the concerns for safeguarding life, health, property, economic interests, the public interest, and the environment.

Why audit? Whatever the business model of the organization, that is, contracted or in-house software development, commercial software development, mass-market software, or embedded software, all organizations, even public organizations, want to meet their objectives. One way to ensure meeting their objectives is to ensure the constant compliance and improvement of processes. Audit activities are usually targeted to the most important projects of the organization and its external supplier's activities. For example, Professor April [APR 97] describes the results of one year of SQA audits, at Bell Canada, and the findings. In internal audits, SQA promotes defect prevention and encourages teams to always meet their customer commitments while respecting internal rules. Software project audits are usually requested by management to ensure that the software team and contracted suppliers:

• – know their duties and obligations toward the public, their employers, their customers, and their colleagues;
• – use the processes, practices, techniques, and normalized methods suggested by the company;
• – reveal any deficiencies and shortcomings in daily operations and try to identify required corrective actions (CAs);
• – are encouraged to develop a personal training plan for their professional skills;
• – are monitored in the course of their work on high profile projects of the company.

Any project manager or software developer can expect to be audited at some point in their career. It is therefore important to understand this formal review process and ensure one is always ready to be audited. It is also quite possible that you will have to participate in an audit as a member of the auditing team. Software project audits are described in ISO 12207, ISO 9001, CobiT [COB 12], and the CMMI.

6.2 Types of Audits

6.2.1 Internal Audit

An internal audit, also called a first-party audit, can be useful for a software supplier wanting to obtain an ISO 9001 certification. It is the least expensive approach to prepare for conformity to an international standard.

ISO/IEC 17050-1 [ISO 04a] describes the requirements for a supplier conformity statement that indicates that a product (including a service), process, management system, individual, or organization meet the specified requirements originating from a standard, a process model, a law, or regulation. The supplier conformity declaration form shall identify: the issuer of the statement, the subject of the statement, standards or specified requirements, and the person who signed the statement. Figure 6.1, taken from ISO 17050-1, shows an example of a supplier conformity declaration form.

ISO/IEC 17050-2 [ISO 04b] indicates that the organization that declares conformity must make supporting documentation available with this form.

A copy of this declaration can be displayed on the website of the organization. The organization should also have put in place a procedure allowing for the re-evaluation of the validity of the declaration, when necessary, should there be any modifications to the development processes or to the standards used.

6.3 Audit and Software Problem Resolution According to ISO/IEC/IEEE 12207

ISO 12207 defines audit requirements and a decision management process.

6.4 Audit According to the IEEE 1028 Standard

In the previous chapter, we have seen that the IEEE 1028 describes the many different types of reviews and audits as well as the procedures for the execution of these reviews. Note that this standard explains how to conduct audits and not on their need or how to use the audit reports.

Table 6.2 summarizes the audit characteristics according to the IEEE 1028 standard. These characteristics will be explained in more detail in this section.

The purpose of the IEEE 1028 standard is to define systematic reviews and audits that apply to user's software acquisition, supplier, development, operation, and maintenance processes. The standard describes how to conduct an audit. It also describes the minimum acceptable requirements for software audits, explaining:

• – the audited team participation;
• – the audit documented results;
• – the documented procedure to conduct the audit.

To perform an audit, the auditor will have to read a set of documents (e.g., software process, products) that will need to be available at audit time. The IEEE 1028 standard lists software products that are most likely to be audited. Table 6.3 shows a partial list of these software products.

The standard also addresses the topic of auditing software project processes. The organization should prepare checklists that are specific to each process or product that is audited.

6.4.3 Audit Conducted According to IEEE 1028

The IEEE 1028 standard insists that before an audit can start, the initiator has appointed an auditor and clarified the scope of the audit. The audited organization should have communicated the audit procedure and the rules against which each project team is audited. When the auditor has confirmed this situation, the auditor will have to demonstrate that he is experienced, trained, and certified for this audit scope.

The audit process, described in Figure 6.2, shows the recommended audit process activities (adapted from IEEE 1028 [IEE 08b]):

• – plan the audit: an audit plan is prepared and approved by the initiator;
• – prepare and lead an opening meeting: an opening meeting, involving the auditors and the audited organization, has the objective of explaining the scope, audit process, and the targeted software process and products that will be audited, the audit schedule, the expected contributions from individuals that will be interviewed, resources involved (e.g., the meeting rooms, access to quality records) as well as the information and documents that will be audited;
• – prepare the audit: this activity reviews the audit plan to be completed, the audited organization readiness, the access and preliminary review of the processes and products that will be audited, the organization life cycle and standards, and the evaluation criteria before the audit start;
• – collect the objective evidence: this is the heart of the audit and requires the collection and analysis of evidence. The key findings will be presented to the audited organization. Then a final report is produced and communicated to the audit sponsor;
• – audit follow-up: the audit sponsor communicates with the audited organization to determine the required CAs.

In this figure, one activity is added to the IEEE 1028 current activity list: “Archive documents of audit” to ensure that all the documentation associated with the audit is kept. In fact, the audit documentation is useful as evidence that the organization is conducting ISO 9001 audits and improvements, or for a future external audit for example. Archived documents can include:

• – the approved audit mandate;
• – the audit plan;
• – identification of participants;
• – the minutes, opening and closing document;
• – the auditor's checklist;
• – the audit report;
• – the list of improvements needed;
• – the CAs and their follow-up to closure.

Finally, note that audit activities, like all other software life cycle activities, can be audited as well. In large organizations, many auditors conduct software project audits according to a yearly audit plan.

The following text box presents an example of an extract from an audit report.

6.5 Audit Process and the ISO 9001 Standard

The ISO 9001 standard has made quality system audits popular. Never before has a standard had so much attention and acceptance worldwide. Production plants were the early adopters of the ISO 9001 standard. They were aware of the benefit of having an independent quality certification as a competitive advantage. This popularity grew when suppliers started asking for the certification before awarding contracts.

Although there is a growing popularity of the use of ISO 9001 for production organizations, it is not the case for service organizations such as software developers. These organizations have a more difficult time separating the final product from the development life cycle processes [MAY 02]. IT requires an additional effort to interpret ISO 9001 clauses in this context.

To ease the use of ISO 9001 for the service industry, including the software industry, interpretation guides have been produced. For software, two sources are popular for the interpretation of ISO 9001 [ISO 15]:

• – ISO/IEC 90003 is the interpretation guide for software for the ISO 9001 standard [ISO 14]; with respect to internal audits, ISO 90003 provides the following information:

• When software development organizations plan their development program, it usually coordinates with the audit planning when projects and quality management systems are selected. Audit planning tries to select projects in order to cover all the development life cycle processes for each phase;
• This could require an audit of different projects, at different development life cycle phases, or the audit of only one project throughout its evolution across life cycle phases. When the targeted project modifies its schedule, the internal audit calendar should also be reviewed to adjust audit dates or to choose another project.

• – the interpretive material (TickIT guide version 5.5 [TIK 07]) developed to train and certify auditors of the ISO 9001 software quality system.

These ISO 9001 interpretation guides clarify how each of the ISO 9001 clauses apply to an organization that develops, maintains, and operates software products. It is interesting to note that the ISO 90003 [ISO 14] interpretation guide is aligned with ISO 12207.

We have discussed that improvement and conformity assessments will always consider the assessment of the current processes used by an organization. The software processes used by the organization will be compared to the standard clauses or to process model practices like those of the CMMI. Capers Jones compares assessment benchmarking to a “medical checkup” of the software organization; this is unavoidable for the identification of what is going right and what has to change. This “medical checkup” is done using a list of the best practices of the industry. In the software industry, this examination is used for benchmarking or to kick start a process improvement program.

We have seen that in order to be ISO 9001 certified, an independent audit will need to be performed. The certifier will need to use an audit process for this.

6.5.1 Steps of a Software Audit

Figure 6.3 presents a model of the key steps that have to be successfully executed in order to obtain an objective assessment of the state of the software processes of an organization.

The first step consists of identifying and interviewing the individuals that will be part of the audit team. These individuals may need training and certifications to join the assessment team.

The second step, once the team is in place, is to have an initial contact with the audited organization either with a meeting or through a preliminary questionnaire. This is followed by a meeting to agree to the scope of the audit, the processes that will be reviewed, a proposed schedule, and the number of individuals involved. The main objective of this step is to prepare everyone for the audit so it is not a surprise. The project manager is then asked to send back the preliminary conformity assessment and provide the security clearance to access the project documentation drives. Questionnaire answers and a preliminary review of the project documents (e.g., project management and technical documents) will allow the auditor to tailor the choice of interviewees’ and solidify the audit schedule.

Next, we need to prepare for the audit that typically lasts 1 or 2 days. This fourth step consists of visiting the site, making presentations, interviewing individuals, and reviewing artifacts. During these 2 days, the inventory of interviews and documents is kept up to date and findings are captured and rated.

Using the information collected, initial findings are made and validated with the participants. Professional judgment is required to determine if the execution of the practices satisfies the reference standards or model. Once findings are validated and ranked, the final report is produced. This takes approximately 10 days. It is sent to the distribution list identified earlier in the audit plan. A last and optional step is to help the team achieve compliance, if asked.

Finally, all the audit information is packaged and archived for future reference. All the compliance data are also analyzed and saved.

All types of audits should publish their process and make it available.

Process audits should be conducted on a regular basis for two main reasons: first, to ensure that practitioners use the processes of the organization and secondly to discover errors, omissions, or misunderstandings in the application of a process. Process audits are also used to assess the degree of use and understanding of practitioners. For example, a new document management process was introduced and practitioners were invited to produce and update documents using this new process. It is well known that engineers are not very inclined to document their work. They often see documentation as a “necessary evil.” Following is an example of an audit conducted to evaluate the conformity of developers to the documentation management process of an organization.

6.6 Audit According to the CMMI

The Software Engineering Institute (SEI) has developed assessment methods to be used in conjunction with its process models. A Software Process Assessment and a Software Capability Evaluation (SCE) [BYR 96] are available for use. The SCE is used for supplier selection to verify that contractual clauses are met.

When the SEI updated its model to launch the CMMI for Development in early 2000, the SCE audits became less used. In the CMMI, audits are activities described as part of the “process and product quality assurance.” This process area aims at providing management with an objective state of the project's software processes and products. For the CMMI, audits are one of the many techniques used, similar to inspections and walk-throughs, to conduct objective assessments. The CMMI states what an objective assessment is in the Process and Product Quality Assurance process area. To ensure objectivity, the following issues must be addressed [SEI 10a]:

• – description of the reporting structure of the SQA to show its independence;
• – establish and maintain clearly formulated assessment criteria:

• what is going to be assessed?
• when, or, how will the process be assessed?
• how will the assessment be conducted?
• who must be involved in the assessment?
• what product of an activity will be assessed?
• when and how will the product of an activity be assessed?

• – use the formulated criteria to assess the conformity of the process descriptions, standards, and procedures executed and for the product of an activity assessment.

6.7 Corrective Actions

After an internal or external audit, an organization must perform CAs to correct the observed deficiencies. It is also possible to treat the preventive actions, an incident report, and customer complaints using the CA process.

A CA aims at eliminating potential causes of non-conformity, a defect, or any other adverse event to prevent their recurrence. Although rectifying a problem focuses on the correction of a very specific case, a CA eliminates the root cause of a problem. CMMI has a process area named “Causal analysis and resolution” to identify and eliminate root causes. The purpose of this process area is to identify causes of selected outcomes and take action to improve process performance [SEI 10a].

6.7.1 Corrective Actions Process

The problems encountered when developing systems that include software, or that occur during their operation, can come from defects in the software, in the development process itself, or in the hardware of the system.

To facilitate the identification of problem sources and apply appropriate CAs, it is desirable that a centralized system is developed to track issues through to resolution and to determine their root cause.

Since the validation, monitoring, and problem resolution may require coordination of different groups within an organization, the SQAP should specify the groups authorized to report/raise incident reports and CAs as well as for the submission of unresolved issues to management.

The following text box is an excerpt from the IEEE 730 standard that describes the requirements regarding this process.

An organization must implement a CA process following the conduct of internal or external audits. This process should cover software products, agreements and software development plans.

A CA process, in a closed loop, may include the following elements:

• – inputs: for example, an audit report, a non-compliance, or a problem report;
• – activities:

• register non-conformities in the issue tracking tool used by the organization,
• analyze and validate the problem to ensure that the organization's resources are not wasted,
• classify and prioritize the issue/problem,
• analyze the problems to conduct trend analysis that can be identified and addressed,
• propose a solution to the problem,
• solve the problem and ensure that the resolution does not cause other problems, or if the non-compliance issue cannot be resolved within the project, refer it to the appropriate management level,
• verify the problem resolution,
• inform stakeholders of the problem resolution,
• archive the problem documentation,
• update the information in the issue tracking tool;

• – outputs: for example, the resolution file, a corrected version of the software.

Figure 6.4 describes a problem resolution process.

Some organizations include a section in their problem report template where the person responsible for the service, as the head of the development organization, must propose a possible solution to correct the problem as well as a planned resolution date (see Figure 6.5).

Once this information is stored in the tool, the SQA can track every issue through to resolution and may sometimes have to intervene to remind the person responsible that he has unresolved issues that have exceeded the registered deadline.

As an independent body, the SQA has an escalation mechanism in the organization for raising the issue to a higher level in the event that the problem is not resolved. Here is a three-level escalation mechanism:

• – escalate to the first level: if CAs are not carried out in accordance with the commitments, the SQA representative meets with the head of the software project to examine the plan to be implemented to ensure CAs, status of CAs and the risk of not completing CAs. The parties negotiate and agree on CAs and new deadlines for the implementation of CAs. The SQA representative documents this agreement and obtains the signature of the head of the software project;
• – escalate to the second level: if the head of the software project is not receptive (to CA or the time limit), the SQA manager meets with the software project manager to review the plan to implement CA, the status of CAs and the risk of not completing CAs. The SQA manager documents decisions and obtains the signature of the software project manager;
• – escalate to third level: if the software project manager is not receptive (to CA or the time limit), the SQA manager meets with senior management to discuss a plan to initiate CAs. The SQA manager documents decisions and obtains the signature of senior management.

6.8 Audits for Very Small Entities

Very small entities (VSEs) have modest means and little time to devote to an audit. However, many VSEs wish to be audited or assessed either to meet the requirements of a client or to increase their brand both nationally and internationally. Thus, a VSE could stand out from other VSEs and become an organization with which a client could develop a business relationship.

When the ISO workgroup was mandated to develop the ISO 29110, a survey was addressed to VSEs located in more than 32 countries. A large number (74%) of VSEs that responded to the survey said that it was very important to them to obtain a certification to improve their profile. A formal certification was requested by 40% of VSEs that answered this survey [LAP 08].

In the chapter on standards, we introduced the ISO 29110 standard. The VSE can, as described in this chapter, conduct an internal audit in accordance with ISO 17050. If the organization complies with ISO 29110, it may therefore declare itself as conforming to this standard, as long as the auditor's independence can be demonstrated by the absence of a parallel activity to be audited, divergence or conflict of interest.

VSEs also have access to external audits of second and third parties. An audit by a second party can be performed by external auditors. In this way, a VSE can demonstrate that an external auditor confirmed its conformity to ISO 29110. This type of audit can be performed at a low cost to a VSE.

Recall that third-party audits are conducted by independent auditing organizations, such as regulatory authorities or bodies granting registration or certification. The certification process is illustrated in Figure 6.6. It begins when a company contacts a certification body to start the certification process. Once the auditor has determined that the VSE is willing to be audited, the auditor launches the audit: the preparation of the audit (e.g., document review, planning, and preparing the audit), the implementation of the audit (e.g., conducting the opening meeting, reviewing documents, collecting and verifying information, producing findings and conclusions, and conducting the closing meeting), the preparation and distribution of the audit report and the end of the audit. Following the issuance of the audit certificate, the certification body will perform, typically on an annual basis, a surveillance audit to ensure continued conformance. An audit for the renewal of the certification is conducted to confirm continued conformance.

ISO 29110 was implemented in a Peruvian start-up of four people. The VSE was created in 2012. After implementing the Basic profile, two processes were executed in a 900-hour customer project [GAR 15]. Only 18% of the total effort consisted of rework. The following text box describes the ISO 29110 third-party audit. The VSE had more than 10 employees at the time of certification in 2014.

After the successful audit of the Peruvian VSE, a local paper reported on this event. Following this article, Peruvian companies contacted the VSE to know more about the process and discuss business opportunities. Later, a large Peruvian insurance company attributed a software development contract to this VSE. In 2017, this VSE had 23 employees.

6.9 Audit and the SQA Plan

The IEEE 730 standard [IEE 14] demands that the SQA activities of a project need to be coordinated with audits and other life cycle processes required to ensure the conformity and quality of the process and the product. It requires SQA to ensure that the audit concerns have been well explained to the project teams and that they audit the software development activities periodically to determine consistency with defined software life cycle processes. For this to happen, the organizational processes must be published to the organization beforehand.

It also imposes that SQA, independent from the project teams, audit projects periodically to determine conformance to defined project plans and assess the skill and knowledge needs of the project and compare them to the skill and knowledge of the organization's staff to identify any gaps. Where projects involve external suppliers, it is a good practice to conduct at least one compliance audit and include it in the contract terms.

For the audit activities of the project, the IEEE 730 standard asks that the project team be ready to answer the following questions:

• – is a subcontractor or external supplier required by the contract? If yes, have periodic reviews and audits been performed to determine if software products fully satisfy contractual requirements?
• – have any issues raised as part of these supplier audits been reviewed and assessed for impact?
• – have corrective/preventive action plans been developed for any non-conformities identified during the supplier audit?
• – have project non-conformances been recorded and appropriately resolved?
• – have project CAs plans been established for items that did not meet the system requirements?

For SQA, the following questions should be answered for each project identified in the audit plan of the organization:

• – has an appropriate and effective audit strategy been developed for the project?
• – has an appropriate and effective audit strategy been implemented for the project?
• – has compliance of selected software work products, services or processes with requirements, plans, and agreements been determined according to the audit strategy?
• – are audits conducted by an appropriate independent party?
• – have the audit results been documented?
• – have all issues detected during an audit been documented as non-conformances?
• – have all non-conformances been considered for CA?
• – have all CAs that were implemented proven to be effective as determined by effectiveness measures?
• – has an appropriate justification been provided for each non-conformance not requiring CA?

6.10 Presentation of an Audit Case Study

In this section, the result of project conformity audits completed at a US system development location of Bombardier Transport, where Professor Laporte was involved, is presented.

6.11 Success Factors

The following text boxes list factors that affect quality with regards to the software audit.