5 QUALITY

LEARNING OUTCOMES

When you have completed this chapter you should be able to demonstrate an understanding of the following:

5.1 INTRODUCTION

The key quality concern for IT projects is providing customers with the systems they need and which meet their requirements at a price they can afford. In order to achieve this, quality needs to be built into a product. It cannot be added to a product after it has been created – except with great difficulty and cost. It is like trying to alter the foundations of a building once it is complete. To build in quality requires a commitment from all parties to the project, from the project sponsor through all the levels of management to the technical staff, customers, users and the clerical support staff. In 1979, Philip Crosby wrote a book which opened:

Crosby was not arguing that increasing the quality of a product did not cost money; rather, that the costs of remedying lapses in quality would be even greater. There is, as explained in Chapter 1, a relationship between quality, cost and time. The level of quality required will have an impact on both time and cost, but the more effort goes into quality at the beginning of a project, the less expenditure is needed on correcting faults at the end of the project.

5.2 DEFINITIONS OF QUALITY

A dictionary definition of quality is ‘a degree or level of excellence’, as in the phrase ‘high-quality goods’. We hope all the products of a project are of a high quality. But the definition is subjective: for example, when comparing cars, people do not agree on the quality of different makes. Another definition is ‘conformance to standard’. Within a project process there will be certain standards to which those developing the system ought to conform. However, the generally accepted definition which should apply to all projects is that the deliverables should be ‘fit for purpose’. Again referring to cars, a Rolls Royce may not be the best vehicle if you need to get to work through heavily congested traffic on time: a scooter might be more effective. The original international standard on quality, ISO 8402:1994, formally defined quality as ‘the totality of features and characteristics of a product or service which bear on its ability to satisfy stated or implied needs’.

One aspect of this ‘fit for purpose’ definition – reliability – shows how the concept applies. If the Canal Dreams ebooking enhancement fails after delivery it would be annoying but not life threatening. However, if the control systems in an aircraft in flight fail, that would be disastrous. The effort, and hence the cost, of making sure that the aircraft system does not fail would be considerably greater than that required for the Canal Dreams system. The costs of a failure in required quality would also be higher. Hence the required quality varies depending on the type of system under development and the money the customer is prepared to pay. Largely, it is the customer who decides the level of quality to be built into a system. Those responsible technically for a project must advise the customer on the benefits of a well-engineered system, but finally it is the person paying who should call the tune. However, suppliers have a professional and legal commitment to the general public to ensure that the systems they produce are safe.

As software systems become more complex, it is impossible to ensure that the software will never fail. Hardware and infrastructure might also be subject to failure. Thus it is important to examine the ways in which the systems under development will behave in the event of various types of failure. This will directly influence the development process and how quality is measured. For example, a control system for a nuclear power plant will be designed to ‘fail safe’ – that is, in the event of a systems failure it will revert to a safe state, for instance by closing down. Obviously such a requirement must be specified in the quality criteria for the system. Where something is inherently dangerous, as in this example of the nuclear power plant, the developers would have a duty of care, not just to the project sponsor, but to the world at large.

5.3 QUALITY CHARACTERISTICS

The definition of ‘fitness for purpose’ needs elaborating so that it can be applied practically. Another international standard, ISO 9126-1:2001, seeks to define a set of standard characteristics by which software quality can be measured. It specifies six high-level quality characteristics:

The standard itself goes into a lot more detail for each heading, but these top-level qualities are useful guides, particularly when trying to establish measurable quality criteria. The qualities can be subdivided into sub-qualities which are then measured. For example, in the Canal Dreams ebooking enhancement, response time in answering queries about the availability of boats is an important part of usability. For the application, engineering measurements have to be mapped to a value on a scale reflecting user satisfaction – for example, a response time under five seconds might correspond to ‘acceptable’.

5.4 QUALITY CRITERIA

To ensure that quality is built into a product the level of quality required has to be specified at the beginning, before the product is developed. In Chapter 2, the concept of a product definition was introduced. Each product definition includes a section headed quality criteria. The accurate completion of this section enables project team staff to check the product is fit for purpose later on in the development cycle, by seeing whether it meets the quality criteria specified.

In order to achieve this, the criteria themselves have to have three qualities:

As an example, the aircraft control system could have a specific requirement that the product must never fail. That is quite clear. It is also measurable: by testing the product for an exhaustive amount of time until the product fails, it can be demonstrated that the requirement has not been met. In this case, we can prove when the quality criterion has not been met, but we cannot prove that it will not fail at some point in the future. Hence the ‘never’ requirement is not achievable. We can, however, provide an estimate of the probability of failure which will get smaller as testing continues. As noted with the nuclear power station, we can also plan for the possibility of failure, for example, by allowing a reversion to manual control.

An organisation should develop standards for product definitions or other documents which contain quality criteria as the basis for its quality procedures.

5.5 QUALITY CONTROL VERSUS QUALITY ASSURANCE

Having introduced the concept of quality criteria and how they should be specified, we now discuss how the quality criteria of a product created by a project are checked. In an ideal environment the project will take place in an organisation committed to quality and with standards already in place for certain activities. If this is not so, part of project set-up will be the creation of the framework for managing the quality of the project.

The quality framework is called the quality management system (QMS). It may be based on the ISO 9001 series of standards (see Section 5.10). Within the QMS, there will be a quality strategy and quality assurance processes. They have to be reviewed and, if necessary, modified to meet the specific requirements of each new project.

A quality strategy defines the QMS and includes:

The methods for exercising quality control are discussed in detail later, but generally these tend to take the form of a review. Quality assurance stands alongside reviews but is external to them. Quality assurance is like an audit. It is designed to confirm that proper procedures are in place and have been applied correctly. The difference between the two can be illustrated by the following example. A metal rod must be 15mm in diameter (± 0.1mm). Control is achieved by passing each rod through a measuring device which triggers rejection if the rod is out of tolerance. Assurance is exercised by checking that the measuring device is accurate and that all rods go through it.

For each component, quality criteria are specified. As we have seen above, this is in effect the first part of quality control, for without the criteria no control can take place. A control process is then undertaken to ensure that the criteria have been met. This is followed, at an appropriate time, by an assurance process which confirms that the agreed procedures have been followed and that all products have undergone the necessary checks.

You will recall from Chapter 1 that the systems development life cycle has a number of stages – for example, requirements analysis is followed by systems design, followed by construction and testing. Each stage contains several quality control processes, in the form of some sort of review, and a quality assurance process takes place at the end of each stage. If proper quality control is lacking, corrective action may be mandated. The quality control and, if necessary, the quality assurance processes are repeated until the quality criteria are met.

5.6 QUALITY PLAN

The production of a quality plan is vital to the success of a project. It specifies the particular standards that apply to the project. They could be existing standards – for example, laying down the content and layout of product definitions. Modifications to existing standards may be needed because of the special characteristics of a new project. In some cases they could be completely new, as a project goes into territory not previously explored. The standards may derive from industry standards or be developed internally. If there are any gaps, then the project manager must ensure that the appropriate standards are specified.

The plan also specifies how, when and by whom the quality control activities should be undertaken, the quality assurance processes to be followed and who will carry them out. It may also include configuration management and change control procedures (see Chapter 4).

The quality plan itself is subject to quality control and quality assurance processes. Quality control would normally be an integral part of the project team’s work. The quality assurance activities, on the other hand, are usually carried out by people outside the project team who report directly to the steering committee/project board. This separation of responsibilities helps to ensure that the process is transparent and reduces possible conflicts of interest.

5.7 DETECTING DEFECTS

5.7.1 The V Model

The V Model is a useful model of the systems development process (see Figure 5.1) in which the solid lines represent the forward progress of the project and the dotted lines represent the way in which quality control is exercised.

There are two quality control processes at work: one between stages and the other across the V. For example, the requirements specification describes all the functions and quality attributes that the customer requires in the system. This should include an acceptance test plan showing how the requirements are going to be assessed in the final system. Using the acceptance test plan, user acceptance testing can be undertaken to demonstrate that the final system to be delivered does indeed meet the requirements identified. This link between the requirements specification and user acceptance testing is shown by the dotted arrow between the two, identified as the ‘acceptance test plan’.

Systems design follows requirements specification and is joined to it by an arrow. As part of the quality control process for systems design it is necessary to check that everything specified in the requirements specification has been incorporated into the design. If not, then the design is incomplete. It is also important to ensure that no requirements have been introduced into the design that were not present in the original requirements. Systems design has two dotted arrows: one across to systems testing and the other back to requirements specification. The horizontal arrow shows the systems testing that needs to be carried out to validate the design after it has been implemented. The arrow to the requirements specification indicates that the design process may discover errors in the systems requirements. For example, gaps may be found in the definition of the requirements, or two requirements may be found to be inconsistent. The same processes are applied to each stage of development.

5.7.2 Process requirements

The importance of quality criteria for each product was stressed earlier. Within a project, as we saw in Chapter 2, a product is the output from a process or activity. The qualities that are required in a product are captured by following the correct processes that create the product. This makes it necessary to specify for each stage, and within each stage for each activity, the process requirements:

Entry requirements state what must exist before the stage or activity can begin. For example, before design can begin, the requirements documentation must be agreed and signed off and any design techniques to be used must be specified. If new techniques are being introduced, then the necessary training should have been given. If design begins before requirements are agreed, this will lead to problems if the requirements are changed while the design is underway. Either the design will have to be reworked, at additional cost, or the need to amend the design may be forgotten and only picked up much later, when making changes will be more expensive.

Implementation requirements define how the process should be done. For example, in design, implementation requirements may specify the use of techniques such as entity modelling or logical process modelling. Implementation requirements also specify the software tools that are to be used.

Exit requirements indicate what should be in place for a successful sign-off of the activity. The exit requirements for design documentation are that it:

5.7.3 Defect removal process

Before a defect can be removed, it must be identified. This is relatively straightforward – though more costly – in the later stages of development, when test cases can be run though the system to see if they give the expected results (dynamic testing). In the earlier stages of a project, different techniques have to be applied, such as:

When specifying quality criteria that apply to, say, a software specification, the technique for testing if those criteria are met should be specified, along with the type of staff who will implement the technique and the tools they will need.

Desk checking

Desk checking is the most basic of the review activities. Authors, or creators, of products review what they have produced. This may mean reading it through and checking for errors in the case of a document, or working through the logic of a program to identify mistakes. It will, hopefully, remove many trivial errors before the product is subjected to more vigorous scrutiny.

Document review

In a document review, one or more people other than the author read a document to ensure that it meets the specified quality criteria. For example, is it complete, unambiguous, self-consistent and consistent with related documents? Is it clear? Are all technical terms properly used and understood? Does it conform to the agreed layout? If the document is a requirement, then such a review could well be carried out by the potential user of the system.

Peer review

A peer review is a particular type of document review that is often carried out on a design document or actual code. The author’s co-workers (or peers) examine copies of the document and make comments about it. The issues considered include:

The peer reviewers of software often dry-run the code – that is, take some test input data and manipulate it on paper according to the instructions in the code.

Peer reviews can be done relatively informally within the project team. However, the time needed by the reviewers needs to be officially scheduled – after all, they have their own software on which to work as well.

Inspection

An inspection is a more formalised way of reviewing a product. Its purpose is to review the product in order to identify defects in a planned, independent, controlled and documented manner. It is a process with the following structure:

There are four roles within this process:

(1) The moderator sets the agenda and controls the review process, ensures actions and required results are agreed and, once the process is complete, signs off the review.

(2) The author provides the reviewers with relevant product documentation, answers questions about the product during the review and agrees actions to resolve defects.

(3) Reviewers undertake the review, assessing the product against the quality criteria, identifying potential defects and ensuring that the nature of any defect is clearly understood by the author.

(4) The scribe takes notes of the agreed actions, who is to carry them out and who is to check corrections. He or she confirms these details at the end of the meeting.

Walkthrough

A walkthrough is a particular technique which can be used during an inspection or on its own. The author takes the audience through the documents and they feed back their comments on it. The audience may be drawn from both technical and user groups, each of which would have a different view on the documents. For example, IT infrastructure management staff may be concerned about the proposed system’s impact on what they already manage. Again there would be a scribe to record any actions agreed.

With peer reviews, inspections and walkthroughs, no attempt should be made during the meeting to solve the problems identified. Problems should be recorded; the author will then go away and seek to come up with a solution. The review can then be repeated if it is felt that the changes required were of sufficient significance. An alternative is for one person to be instructed to ensure that the necessary alterations have been made.

Pair programming

The review techniques described above depend on copies of documents, including code, being printed and additional reviewing activities being scheduled. To avoid this, in agile development environments, code developers sometimes work in pairs. The pair take turns to type in code at the workstation while the other advises and checks on what is being entered. This is rather like a real-time version of a peer review.

Static testing

Some software tools (programs) carry out static testing by analysing the structure of the code. Such tools look at the branches and loops in a program and calculate a measure of complexity. The more complex a software component is, the more difficult it will be to maintain. In addition, dynamic analysis can identify code which is not executed by the test data used. This is useful as it may show up a shortcoming in the test data or unneeded code in the software.

All the above processes take place during the activities on the left-hand side of the V model. The quality control processes on the right-hand side are dominated by dynamic testing.

5.8 DYNAMIC TESTING

Dynamic testing is divided into various levels:

For each type of testing, a set of test data and a set of expected results must be produced. Referring back to the V model, a test plan should have been produced at the appropriate stage in the development process on the left-hand side of the V. The plan should contain the necessary guidance for producing the test data, if not the test data itself.

Unit testing

Unit testing is the very basic testing to be carried out on a piece of software. Test data for unit testing should be designed to cover the usual range of input expected for the system. Each function that the software is expected to handle should be tested. The test should take place not just once but in various combinations and different sequences. Often problems lie in the combinations of data and transactions. Testing is then extended to cover, for example, numbers just inside and just outside any specified limits. Alphabetic fields are tested with entries longer than that which the system should permit. Mandatory fields are left blank. This is not an exhaustive list but is indicative of what is necessary. All tests are designed to ensure that this particular unit will not fail because of bad data or unusual combinations but will handle them in a predefined way.

Each set of test results should be carefully checked against the expected results to ensure agreement. Any discrepancies should be noted and reported so that the software can be amended and retested. Sometimes, however, the expected results are wrong! It is important to keep records of all faults found and resulting changes made, so that if problems arise at a later stage there is a trail which can be followed to establish how they were introduced.

There are tools – commonly called test harnesses – which can simulate programs or subroutines that supply data to or use data from the module under test. The test harness can record data input and output and the routes through the program which have been exercised. Automated tools can also simulate keyboard input and capture and compare screen output with expected results.

Once unit testing has been successfully completed the unit can be signed off and registered as a configuration item (see Chapter 4).

Integration testing

Integration testing links a number of system components and runs them as a whole. This checks that the units communicate properly with each other. The sorts of things that come to light at this stage include:

Some of these can be avoided by the use of shared databases and database management systems (DBMS), but transaction files which link different sub-systems can still be mistakenly defined to have different formats in different places.

As errors are found they will be recorded and corrected. The integration test will then need to be repeated.

Systems testing

Systems testing is the final stage in testing by the development professionals. It involves running the whole system on the infrastructure that will be used when the system is operational. It may sound just like a step up from integration testing but there are many additional issues which must be addressed, for example:

Again, there are tools to help. They can be used to simulate large numbers of users and high volumes of data.

User acceptance testing

User acceptance testing is the crucial test. Can the users operate the system? Does it meet their expectations, not just their requirements? Users should be involved in the development process from the beginning of the project and should have had sight of how the system is working before this point is reached so it now contains few surprises. Expectations as well as requirements have to be managed. It may or may not be helpful to involve users in earlier testing activities: it is a matter of keeping a balance between helping them to understand the way the system works, and them seeing all the faults that arise during testing and becoming disillusioned.

Acceptance testing underlines the importance of having clear requirements from the start. Users should have a well-defined acceptance test to guide them as they test whether the system meets their expressed needs. Where discrepancies are found, the fault must be recorded, the source of the problem established and the system reworked and retested.

Regression testing

Regression testing is quite different from the earlier testing processes. Regression testing needs to take place at each stage of testing. Whenever a fault is found and the offending piece of software identified, it has to be corrected. Unfortunately this may well introduce further errors or uncover ones that were masked by the first error. Regression testing involves running an agreed set of test data through the system again to confirm that the original error has been corrected and no further errors have been introduced or uncovered. The later a fault is discovered in the testing hierarchy, the more regression testing has to take place.

Regression testing is also needed whenever the system is changed, whether during development or after implementation, as changes made to one part of a system can adversely affect another. Regression testing can largely be automated by using a standard set of test data and expected results.

5.9 EVALUATING SUPPLIERS

It is usual these days for systems to be developed by third parties. It is therefore important to establish whether those third party suppliers have the necessary quality procedures in place to ensure that the software to be supplied is to the standard expected.

To do this one has to investigate the supplier, ask some searching questions, examine documentation and perhaps undertake auditing of their processes. The types of questions to be asked include.

This list is by no means exhaustive. The response to each question should be supported by evidence. For example, if it is claimed that software quality is assessed by reviews, then examples of the outputs from the reviews can be examined. Observing some of their reviews can increase confidence in their quality processes.

It must be recognised that the supplier and the customer have different business objectives. Making a project a success therefore needs both parties to see the project as a joint venture. The customer cannot simply hand it over to the supplier and wait for delivery, nor should the supplier hide the development process from the customer. Each side needs to be involved to an appropriate level for the type of project being undertaken.

5.10 ISO 9001:2008

ISO 9001:2008 is the international standard for quality management and is specifically aimed at producers and suppliers of any products and services, not necessarily software. Organisations are inspected and awarded ISO 9001 certification by accredited auditors. This means that as a potential client you can assume a particular standard of quality management without having to carry out detailed checks yourself.

However, while ISO 9001 states that a quality level should be specified, it does not say what that level should be. For example, when producing a ballpoint pen it could be stated that a quality requirement is for the ink in the pen to last three years, but equally it could state three days. If the requirement is met then the expected quality is achieved even if it is very low. Thus having a set of ISO 9001 procedures only guarantees that a level of quality has been specified, not that this level is accepted universally as being appropriate. This does allow the client of a ISO 9001:2008 supplier to negotiate the quality criteria they personally need.

ISO 9001:2008 is based on the following principles:

The TickIT Guide is designed to show the application of ISO 9001:2008 to software systems. In doing this, it also takes account of another ISO standard, ISO/IEC 90003:2004, which offers guidance on the application of ISO 9001:2008 to software engineering. The current (2012) version of the TickIT Guide, Issue 5.5, is in line with ISO 9001:2008. TickIT applies whenever software development is carried out, the software is incorporated in the delivered product or service and the organisation wishes to be ISO 9001 accredited. TickIT is managed and maintained by the British Standards Institution (BSI).

5.11 CAPABILITY MATURITY MODELS

In the discussion of ISO9001:2008, we assumed that the motivation was to allow customers to assess potential suppliers. Sometimes, however, the managers of a supplier organisation want to assess their own quality processes to find ways of improving them. They want to gauge their current level of effectiveness, but also identify what needs to be done to get it to a higher level. This brings in the concept of maturity modes, where the organisation is assessed as being at a particular level of process maturity.

On example of this is the capability maturity model (CMM) originally developed by the Software Engineering Institute at Carnegie Mellon University for the US Department of Defense. This comprises five levels.

(1) Initial. Any organisation would be at this level by default. Good quality work may be done, but customers cannot be sure that this is always the case.

(2) Managed. Some basic project management and other systems are in place.

(3) Defined. The way each task in software development is done is defined to enable consistent good practice.

(4) Quantifiably managed. Processes and their products are measured and controlled – for example, the number of errors created in each process.

(5) Optimising. The measurement data collected is analysed to find ways of improving processes.

The assessment of an organisation’s maturity level can be done internally, or external auditors could be employed so that the maturity level can be published externally, as in the case of ISO 9001.

SAMPLE QUESTIONS

1. ‘Fitness for purpose’ defines which of the following?

(a) The quality of the project deliverables

(b) The usability of the delivered IT application

(c) The capability of the staff who will implement the IT application

(d) The capability of the staff who will use the system when it is operational

2. User acceptance testing is an example of which of the following?

(a) Project control

(b) Project monitoring

(c) Quality control

(d) Quality assurance

3. Which of the following is NOT a defined role in inspections?

(a) The moderator

(b) The project manager

(c) The scribe

(d) The author

4. ISO 9001:2008 is a standard that defines which of the following?

(a) Project management standards

(b) IT project deliverables

(c) Quality management systems

(d) Software testing procedures

ANSWERS TO SAMPLE QUESTIONS

1. (a) 2. (c) 3. (b) 4. (c)

POINTERS FOR ACTIVITIES