5.2.1 Personal Review

A personal review is done by the person reviewing his own software product in order to find and fix the most defects possible. A personal review should precede any activity that uses the software product under review.

The principles of a personal review are [POM 09]:

• – find and correct all defects in the software product;
• – use a checklist produced from your personal data, if possible, using the type of defects that you are already aware of;
• – follow a structured review process;
• – use measures in your review;
• – use data to improve your review;
• – use data to determine where and why defects were introduced and then change your process to prevent similar defects in the future.

The following practices should be followed to develop an effective and efficient personal review [POM 09]:

• – pause between the development of a software product and its review;
• – examine products in hard copy rather than electronically;
• – check each item on the checklist once completed;
• – update the checklists periodically to adjust to your personal data;
• – build and use a different checklist for each software product;
• – verify complex or critical elements with an in depth analysis.

Figure 5.6 outlines the process of a personal review.

As we can see, personal reviews are very simple to understand and perform. Since the errors made are often different for each software developer, it is much more efficient to update a personal checklist based on errors noted in previous reviews.

5.2.2 Desk-Check Reviews

A type of peer review that is not described in standards is the desk-check review [WAL 96], sometimes called the Pass around [WIE 02]. It is important to explain this type of peer review because it is inexpensive and easy to implement. It can be used to detect anomalies, omissions, improve a product, or present alternatives. This review is used for low-risk software products, or if the project plan does not allow for more formal reviews. According to Wiegers, this review is less intimidating than a group review such as a walk-through or inspection. Figure 5.7 describes the process for this type of review.

As shown in Figure 5.7, there are six steps. Initially, the author plans the review by identifying the reviewer(s) and a checklist. A checklist is an important element of a review as it enables the reviewer to focus on only one criterion at a time. A checklist is a reflection of the experience of the organization. Then, individuals review the software product document and note comments on the review form provided by the author. When completed, the review form can be used as “evidence” during an audit.

In this book, several checklists are presented. Here is a list of some important features of checklists:

• – each checklist is designed for a specific type of document (e.g., project plan, specification document);
• – each item of a checklist targets a single verification criteria;
• – each item of a checklist is designed to detect major errors. Minor errors, such as misspellings, should not be part of a checklist;
• – each checklist should not exceed one page, otherwise it will be more difficult to use by the reviewers;
• – each checklist should be updated to increase efficiency;
• – each checklist includes a version number and a revision date.

The following text box presents a generic checklist, that is, a checklist that can be used for almost any type of document to be reviewed (e.g., project plan, architecture). For each type of software product (e.g., requirements or design), a specific checklist will be used. For a list designed to facilitate the detection of errors in requirements, we could add the EX identifier and include the following element: EX 1 (testable)—the requirement must be testable. For a list of verifications for a test plan, one might use the TP identifier.

In the third step of the desk-check process, the reviewers verify the document and record their comments on the review form. The author reviews the comments as part of step 4. If the author agrees with all the comments, he incorporates them into his document. However, if the author does not agree, or if he believes the comments have a major impact, then he should convene a meeting with the reviewers to discuss the comments. After this meeting, one of three options should be considered: the comment is incorporated as is, the comment is ignored, or it is incorporated with modifications. For the next step, the author can make the corrections and note the effort spent reviewing and correcting the document, that is, the time spent by the reviewers as well as the time spent by the author to correct the document and conduct the meeting if this is the case. The activities of the desk-check (DC) review are described in Figure 5.8. In the final step, the author completes the review form illustrated in Figure 5.9.

Figure 5.9 illustrates a standard form used by reviewers to record their comments and the time they devoted to the revision of the document. The author of the document collects these data and adds the time it took him to correct the document. The forms will be retained by the author as “evidence” for an audit by the SQA of the organization the author belongs to, or by the SQA of the customer.

As an alternative to the distribution of hard copies to reviewers, one can place an electronic copy of the document, the review form and the checklist in a shared folder on the Intranet. Reviewers are invited to provide comments as annotations to documents over a defined period of time. The author can then view the annotated document, review the comments, and continue the Desk-Check review as described above.

In the next sections, we describe more formal reviews.

5.3.1 ISO/IEC 20246 Software and Systems Engineering: Work Product Reviews

The purpose of ISO/IEC 20246 Work Product Reviews is [ISO 17d]: “to provide an International Standard that defines work product reviews, such as inspections, reviews and walk-throughs, that can be used at any stage of the software and systems life cycle. It can be used to review any system and software work product. ISO/IEC 20246 defines a generic process for work product reviews that can be configured based on the purpose of the review and the constraints of the reviewing organization. The intent is to describe a generic process that can be applied both efficiently and effectively by any organization to any work product. The main objectives of reviews are to detect issues, to evaluate alternatives, to improve organizational and personal processes, and to improve work products. When applied early in the life cycle, reviews are typically shown to reduce the amount of unnecessary rework on a project. The work product review techniques presented in ISO/IEC 20246 can be used at various stages of the generic review process to identify defects and evaluate the quality of the work product.”

ISO 20246 includes an annex that describes the alignment of the activities of the ISO 20246 standard and the procedures of the IEEE 1028 standard presented below.

5.3.2 Capability Maturity Model Integration

The CMMI^® for Development (CMMI-DEV) [SEI 10a] is widely used by many industries. This model describes proven practices in engineering. In this model, a part of the “Verification” process area is devoted to peer reviews. Other verification activities will be considered in more detail in a later chapter. Figure 5.10 is an extract of the staged representation of the CMMI-DEV which describes peer reviews.

The process and product quality assurance process areas provide the following list of issues to be addressed when implementing peer reviews [SEI 10a]:

• – Members are trained and roles are assigned for people attending the peer reviews.
• – A member of the peer review who did not produce this work product is assigned to perform the quality assurance role.
• – Checklists based on process descriptions, standards, and procedures are available to support the quality assurance activity.
• – Non-compliance issues are recorded as part of the peer review report and are tracked and escalated outside the project when necessary.

According to the CMMI-DEV, these reviews are performed on selected work products to identify defects and to recommend other changes required. The peer review is an important and effective software engineering method, applied through inspections, walk-throughs or a number of other review procedures.

Reviews that meet the CMMI requirements listed in Figure 5.10 are described in the following sections.

5.3.3 The IEEE 1028 Standard

The IEEE 1028-2008 Standard for Software Reviews and Audits [IEE 08b] describes five types of reviews and audits and the procedures required for the completion of each type of review and audit. Audits will be presented in the next chapter. The introductory text of the standard indicates that the use of these reviews is voluntary. Although the use of this standard is not mandatory, it can be imposed by a client contractually.

The purpose of this standard is to define reviews and systematic audits for the acquisition, supply, development, operation and maintenance of software. This standard describes not only “what to do” but also how to perform a review. Other standards define the context in which a review is performed and how the results of the review are to be used. Examples of such standards are provided in Table 5.1.

The IEEE 1028 standard provides minimum acceptable conditions for systematic reviews and software audits including the following attributes:

• – team participation;
• – documented results of the review;
• – documented procedures for conducting the review.

Conformance to the IEEE 1028 standard for a specific review, such as an inspection, can be claimed when all mandatory actions (indicated by “shall”) are carried out as defined in this standard for the review type used.

This standard provides descriptions of the particular types of reviews and audits included in the standard as well as tips. Each type of review is described with clauses that contain the following information [IEE 08b]:

1. Introduction to review: describes the objectives of the systematic review and provides an overview of the systematic review procedures;
2. Responsibilities: defines the roles and responsibilities needed for the systematic review.
3. Input: describes the requirements for input needed by the systematic review;
4. Entry criteria: describes the criteria to be met before the systematic review can begin, including the following:
   1. Authorization,
   2. Initiating event;
5. Procedures: details the procedures for the systematic review, including the following:
   1. Planning the review;
   2. Overview of procedures;
   3. Preparation;
   4. Examination/evaluation/recording of results;
   5. Rework/follow-up;
6. Exit criteria: describe the criteria to be met before the systematic review can be considered complete;
7. Output: describes the minimum set of deliverables to be produced by the systematic review.

5.4.1 Usefulness of a Walk-Through

There are several reasons for the implementation of a walk-through process:

• – identify errors to reduce their impact and the cost of correction;
• – improve the development process;
• – improve the quality of the software product;
• – reduce development costs;
• – reduce maintenance costs.

5.4.2 Identification of Roles and Responsibilities

Four roles are described in the IEEE 1028: leader, recorder, author, and team member. Roles can be shared among team members. For example, the leader or author may play the role of recorder and the author could also be the leader. But, a walk-through shall include at least two members.

The standard defines the roles as follow (adapted from IEEE 1028 [IEE 08b]):

• – Walk-through leader

• conduct the walk-through;
• handle the administrative tasks pertaining to the walk-through (such as distributing documents and arranging the meeting);
• prepare the statement of objectives to guide the team through the walk-through;
• ensure that the team arrives at a decision or identified action for each discussion item;
• issue the walk-through output.

• – Recorder

• note all decisions and identified actions arising during the walk-through meeting;
• note all comments made during the walk-through that pertain to anomalies found, questions of style, omissions, contradictions, suggestions for improvement, or alternative approaches.

• – Author

• present the software product in the walk-through.

• – Team member

• adequately prepare for and actively participate in the walk-through;
• identify and describe anomalies in the software product.

The IEEE 1028 standard lists improvement activities using data collected from the walk-throughs. These data should [IEE 08b]:

• – be analyzed regularly to improve the walk-through process;
• – be used to improve operations that produce software products;
• – present the most frequently encountered anomalies;
• – be included in the checklists or in assigning roles;
• – be used regularly to assess the checklists for superfluous or misleading questions;
• – include preparation time and meetings; the number of participants should be considered to determine the relationship between the preparation time and meeting and the number and severity of anomalies detected.

To maintain the efficiency of walk-throughs, the data should not be used to evaluate the performance of individuals.

IEEE 1028 also describes the procedures of walk-throughs.

5.6.1 Project Launch Review

The project launch review is a management review of: the milestone dates, requirements, schedule, budget constraints, deliverables, members of the development team, suppliers, etc. Some organizations also conduct kick-off reviews at the beginning of each of the major phases of the project when projects are spread over a long period of time (as in several years).

Before the start of a project, team members ask themselves the following questions: who will the members of my team be? Who will be the team leader? What will my role and responsibilities be? What are the roles of the other team members and their responsibilities? Do the members of my team have all the skills and knowledge to work on this project?

The following text box describes the kick-off review meeting used for software projects at Bombardier Transport.

5.6.2 Project Retrospectives

If the poor cousin of software engineering is quality assurance, the poor cousin of quality assurance reviews is the project retrospective. It is ironic that a discipline, such as software engineering, which depends as much as it does on the knowledge of the people involved, dismisses the opportunity to learn and enrich the knowledge of an organization's members. The project retrospective review is normally carried out at the end of a project or at the end of a phase of a large project. Essentially, we want to know what has been done well in this project, what has gone less well and what could be improved for the next project. The following terms are synonymous: lessons learned, post mortem, after-action-review.

Basili et al. (1996) [BAS 96] published the first controlled experiments that captured experience. This approach, called Experience Factory, where experience is gathered from software development projects, is packaged and stored in a database of experience. The packaging refers to the generalization, adaptation, and formalization of the experience until it is easy to reuse. In this approach, experience is separate from the organization that is responsible for capturing the experience.

A post mortem review, conducted at the end of a phase of a project or at the end of a project, provides valuable information such as [POM 09]:

• – updating project data such as length, size, defects, and schedule;
• – updating quality or performance data;
• – a review of performance against plan;
• – updating databases for size and productivity;
• – adjustment of processes (e.g., checklist), if necessary, based on the data (notes taken on the proposal process improvement (PIP) forms, changes in design or code, lists of default controls indicated and so on).

There are several ways to conduct project retrospectives; Kerth (2001) lists 19 techniques in his book [KER 01].

Some techniques focus on creating an atmosphere of discussion in the project, others consider past projects, still others are designed to help a project team to identify and adopt new techniques for their next project, and some address the consequences of a failed project. Kerth recommends holding a 3-day session to make a lasting change in an organization [KER 01]. This section presents a less stringent and less costly approach to capturing the experience of project members.

Since a retrospective session may create some tension, especially if the project discussed has not been a total success, we propose rules of behavior so that the session is effective. The rules of behavior at these sessions are:

• – respect the ideas of the participants;
• – maintain confidentiality;
• – not to blame;
• – not to make any verbal comment or gesture during brainstorming;
• – not to comment when ideas are retained;
• – request more details regarding a particular idea.

The following quote outlines the basis of a successful assessment session.

The main items on the agenda during a project retrospective review are:

• – list the major incidents and identify the main causes;
• – list the actual costs and the actual time required for the project, and analyze variances from estimates;
• – review the quality of processes, methods, and tools used for the project;
• – make proposals for future projects (e.g., indicate what to repeat or reuse (methodology, tools, etc.), what to improve, and what to give up for future projects).

A retrospective session typically consists of three steps: first, the facilitator explains, along with the sponsor, the objectives of the meeting; second, he explains what a retrospective session is, the agenda and the rules of behavior; lastly, he conducts the session.

A retrospective session takes place as follows:

Step One

• – presentation of the facilitators by the sponsor;
• – introduction of participants;
• – presentation of the assumption:

• Regardless of what we discover, we truly believe that everyone did the best job, given his qualifications and abilities, resources, and project context.

• – Presentation of the agenda of a typical retrospective session lasting approximately three hours:

• introduction;
• brainstorm to identify what went well and what could improve;
• prioritize items;
• identify the causes;
• write a mini action plan.

Step Two—Introduction to the retrospective session

• – what is a retrospective session?
• – when is a lesson really learned?

• what is individual learning, team learning?
• what is learning in an organization?

• – why have a retrospective session?
• – potential difficulties of a retrospective session;
• – session rules;
• – what is brainstorming? The rules of brainstorming are:

• no verbal comments or gestures;
• no discussion when ideas are retained.

Step Three—Conducting the retrospective session

• – chart a history (timeline) of the project (15–30 minutes);
• – conduct brainstorming (30 minutes);

• individually, identify on post-it notes:
  • what went well during the project (e.g., what to keep)?
  • what could be improved?
  • were there any surprises?
• collect ideas and post them on the project history chart

• – clarify ideas (if necessary);
• – group similar ideas;
• – prioritize ideas;
• – find the causes using the “Five Why” technique:

• what went well during the project?
• what could be improved?

• – final questions;

• for this project, name what you would have liked to change?
• for this project, name what you wish to keep.

• – write a mini action plan;

• what, who, when?

• – end the session;

• ensure the commitment to implement the action plan;
• thanks to the sponsor and the participants.

Even if logic dictates that conducting project retrospective or lessons learned sessions are beneficial for the organization, there are still some factors that affect these types of sessions:

• – leading lessons learned sessions takes time and often management wants to reduce project costs;
• – lessons learned benefit future projects;
• – a culture of blame (finger pointing) can significantly reduce the benefits of these sessions;
• – participants may feel embarrassed or have a cynical attitude;
• – the maintenance of social relationships between employees is sometimes more important than the diagnosis of events;
• – people may be reluctant to engage in activities that could lead to complaints, to criticism, or blame;
• – some people have beliefs that predispose them to the acceptance of lessons learned; beliefs such as “Experience is enough to learn” or “If you do not have experience, you will not learn anything”;
• – certain organizational cultures do not seem able or willing to learn.

5.14 Further Reading

1. WIEGERS K. The seven deadly sins of software reviews. Software Development, vol. 6, issue 3, 1998, pp. 44–47.
2. WIEGERS K. A little help from your friends. Peer Reviews in Software, Pearson Education, Boston, MA, 2002, Chapter 2.
3. WIEGERS K. Peer review formality spectrum. Peer Reviews in Software, Pearson Education, Boston, MA, 2002, Chapter 3.