Chapter 1

Principles of Testing

1.1 CONTEXT OF TESTING IN PRODUCING SOFTWARE

Almost everything we use today has an element of software in it. In the early days of evolution of software, the users of software formed a small number compared to the total strength of an organization. Today, in a typical workplace (and at home), just about everyone uses a computer and software. Administrative staff use office productivity software (replacing the typewriters of yesteryears). Accountants and finance people use spreadsheets and other financial packages to help them do much faster what they used to do with calculators (or even manually). Everyone in an organization and at home uses e-mail and the Internet for entertainment, education, communication, interaction, and for getting any information they want. In addition, of course, the “technical” people use programming languages, modeling tools, simulation tools, and database management systems for tasks that they were mostly executing manually a few years earlier.

 

The above examples are just some instances where the use of software is “obvious” to the users. However, software is more ubiquitous and pervasive than seen in these examples. Software today is as common as electricity was in the early part of the last century. Almost every gadget and device we have at home and at work is embedded with a significant amount of software. Mobile phones, televisions, wrist watches, and refrigerators or any kitchen equipment all have embedded software.

Another interesting dimension is that software is being used now in mission critical situations where failure is simply unacceptable. There is no way one can suggest a solution of “please shutdown and reboot the system” for a software that is in someone's pacemaker! Almost every service we have taken for granted has software. Banks, air traffic controls, cars are all powered by software that simply cannot afford to fail. These systems have to run reliably, predictably, all the time, every time.

This pervasiveness, ubiquity, and mission criticality places certain demands on the way the software is developed and deployed.

First, an organization that develops any form of software product or service must put in every effort to drastically reduce and, preferably, eliminate any defects in each delivered product or service. Users are increasingly intolerant of the hit-and-miss approach that characterized software products. From the point of view of a software development organization also, it may not be economically viable to deliver products with defects. For instance, imagine finding a defect in the software embedded in a television after it is shipped to thousands of customers. How is it possible to send “patches” to these customers and ask them to “install the patch?” Thus, the only solution is to do it right the first time, before sending a product to the customer.

Second, defects are unlikely to remain latent for long. When the number of users was limited and the way they used the product was also predictable (and highly restricted), it was quite possible that there could be defects in the software product that would never get detected or uncovered for a very long time. However, with the number of users increasing, the chances of a defect going undetected are becoming increasingly slim. If a defect is present in the product, someone will hit upon it sooner than later.

Third, the nature of usage of a product or a service is becoming increasingly unpredictable. When bespoke software is developed for a specific function for a specific organization (for example, a payroll package), the nature of usage of the product can be predictable. For example, users can only exercise the specific functionality provided in the bespoke software. In addition, the developers of the software know the users, their business functions, and the user operations. On the other hand, consider a generic application hosted on the Internet. The developers of the application have no control over how someone will use the application. They may exercise untested functionality; they may have improper hardware or software environments; or they may not be fully trained on the application and thus simply use the product in an incorrect or unintended manner. Despite all this “mishandling,” the product should work correctly.

Finally, the consequence and impact of every single defect needs analysis, especially for mission critical applications. It may be acceptable to say that 99.9% of defects are fixed in a product for a release, and only 0.1% defects are outstanding. It appears to be an excellent statistics to go ahead and release the product. However, if we map the 0.1% failure in mission critical applications, the data will look like this.

• A total of 10, 000 incorrect surgery operations per week.
• Three airplane crashes every day.
• No electricity for five hours every week.

For sure, the above data is unacceptable for any individual, organization, or government. Providing a workaround, such as “In case of fire, wear this dress,” or documenting a failure, such as “You may lose only body parts in case of a wrong airplane landing” would not be acceptable in cases of mission critical applications.

This book focuses on software testing. Traditionally, testing is defined as being narrowly confined to testing the program code. We would like to consider testing in a broader context as encompassing all activities that address the implications of producing quality products discussed above. Producing a software product entails several phases (such as requirements gathering, design, and coding) in addition to testing (in the traditional sense of the term). While testing is definitely one of the factors (and one of the phases) that contributes to a high quality product, it alone cannot add quality to a product. Proper interaction of testing with other phases is essential for a good product. These interactions and their impact are captured in the grid in Figure 1.1.

 

 

If the quality of the other phases is low and the effectiveness of testing is low (lower left-hand corner of the grid), the situation is not sustainable. The product will most likely go out of business very soon. Trying to compensate for poor quality in other phases with increased emphasis on the testing phase (upper left-hand corner of the grid) is likely to put high pressure on everyone as the defects get detected closer to the time the product is about to be released. Similarly, blindly believing other phases to be of high quality and having a poor testing phase (lower right-hand side of the grid) will lead to the risky situation of unforeseen defects being detected at the last minute. The ideal state of course is when high quality is present in all the phases including testing (upper right-hand corner of the grid). In this state, the customers feel the benefits of quality and this promotes better teamwork and success in an organization.

1.2 ABOUT THIS CHAPTER

In this chapter, we discuss some of the basic principles of testing. We believe that these principles are fundamental to the objective of testing, namely, to provide quality products to customers. These principles also form the motivation for the rest of the book. Thus this chapter acts as an anchor for the rest of the book.

The fundamental principles of testing are as follows.

1. The goal of testing is to find defects before customers find them out.
2. Exhaustive testing is not possible; program testing can only show the presence of defects, never their absence.
3. Testing applies all through the software life cycle and is not an end-of-cycle activity.
4. Understand the reason behind the test.
5. Test the tests first.
6. Tests develop immunity and have to be revised constantly.
7. Defects occur in convoys or clusters, and testing should focus on these convoys.
8. Testing encompasses defect prevention.
9. Testing is a fine balance of defect prevention and defect detection.
10. Intelligent and well-planned automation is key to realizing the benefits of testing.
11. Testing requires talented, committed people who believe in themselves and work in teams.

We will take up each of these principles in the subsequent sections. Where appropriate, we will illustrate the principle with a simple story from outside the arena of information technology to drive home the point.

1.3 THE INCOMPLETE CAR

 

Eventually, whatever a software organization develops should meet the needs of the customer. Everything else is secondary. Testing is a means of making sure that the product meets the needs of the customer.

We would like to assign a broader meaning to the term “customer.” It does not mean just external customers. There are also internal customers. For example, if a product is built using different components from different groups within an organization, the users of these different components should be considered customers, even if they are from the same organization. Having this customer perspective enhances the quality of all the activities including testing.

We can take the internal customer concept a step further where the development team considers the testing team as its internal customer. This way we can ensure that the product is built not only for usage requirements but also for testing requirements. This concept improves “testability” of the product and improves interactions between the development and testing teams.

 

We would like to urge the reader to retain these two perspectives—customer perspective and perspective of quality not being an add-on in the end, but built in every activity and component right from the beginning—throughout the book.

 

If our job is to give a complete car to the customer (and not ask the customers to paint the car) and if our intent is to make sure the car works as expected, without any (major) problems, then we should ensure that we catch and correct all the defects in the car ourselves. This is the fundamental objective of testing. Anything we do in testing, it behoves us to remember that.

1.4 DIJKSTRA's DOCTRINE

Consider a program that is supposed to accept a six-character code and ensure that the first character is numeric and rests of the characters are alphanumeric. How many combinations of input data should we test, if our goal is to test the program exhaustively?

The first character can be filled up in one of 10 ways (the digits 0-9). The second through sixth characters can each be filled up in 62 ways (digits 0-9, lower case letters a-z and capital letters A-Z). This means that we have a total of 10 x (62⁵) or 9, 161, 328, 320 valid combinations of values to test. Assuming that each combination takes 10 seconds to test, testing all these valid combinations will take approximately 2, 905 years!

Therefore, after 2, 905 years, we may conclude that all valid inputs are accepted. But that is not the end of the story—what will happen to the program when we give invalid data? Continuing the above example, if we assume there are 10 punctuation characters, then we will have to spend a total of 44, 176 years to test all the valid and invalid combinations of input data.

All this just to accept one field and test it exhaustively. Obviously, exhaustive testing of a real life program is never possible.

All the above mean that we can choose to execute only a subset of the tests. To be effective, we should choose a subset of tests that can uncover the maximum number of errors. We will discuss in Chapter 4, on Black Box Testing, and Chapter 3, on White Box Testing, some techniques such as equivalence partitioning, boundary value analysis, code path analysis, and so on which help in identifying subsets of test cases that have a higher likelihood of uncovering defects.

 

Nevertheless, regardless of which subset of test cases we choose, we can never be 100% sure that there are no defects left out. But then, to extend an old cliche, nothing can be certain other than death and taxes, yet we live and do other things by judiciously managing the uncertainties.

1.5 A TEST IN TIME!

Defects in a product can come from any phase. There could have been errors while gathering initial requirements. If a wrong or incomplete requirement forms the basis for the design and development of a product, then that functionality can never be realized correctly in the eventual product. Similarly, when a product design—which forms the basis for the product development (a la coding)—is faulty, then the code that realizes the faulty design will also not meet the requirements. Thus, an essential condition should be that every phase of software development (requirements, design, coding, and so on) should catch and correct defects at that phase, without letting the defects seep to the next stage.

Let us look at the cost implications of letting defects seep through. If, during requirements capture, some requirements are erroneously captured and the error is not detected until the product is delivered to the customer, the organization incurs extra expenses for

• performing a wrong design based on the wrong requirements;
• transforming the wrong design into wrong code during the coding phase;
• testing to make sure the product complies with the (wrong) requirement; and
• releasing the product with the wrong functionality.

In Figure 1.2 the defects in requirements are shown in gray. The coloured figure is available on Illustrations. As you can see, these gray boxes are carried forward through three of the subsequent stages—design, coding, and testing.

 

 

When this erroneous product reaches the customer after the testing phase, the customer may incur a potential downtime that can result in loss of productivity or business. This in turn would reflect as a loss of goodwill to the software product organization. On top of this loss of goodwill, the software product organization would have to redo all the steps listed above, in order to rectify the problem.

 

Similarly, when a defect is encountered during the design phase (though the requirements were captured correctly, depicted by yellow), the costs of all of the subsequent phases (coding, testing, and so on) have to be incurred multiple times. However, presumably, the costs would be lower than in the first case, where even the requirements were not captured properly. This is because the design errors (represented by yellow boxes) are carried forward only to the coding and testing phases. Similarly, a defect in the coding phase is carried forward to the testing phase (green boxes). Again, as fewer phases are affected by this defect (compared to requirements defects or design defects), we can expect that the cost of defects in coding should be less than the earlier defects. As can be inferred from the above discussion, the cost of a defect is compounded depending on the delay in detecting the defect.

Hence, smaller the lag time between defect injection (i.e., when the defect was introduced) and defect detection (i.e., when the defect was encountered and corrected), lesser are the unnecessary costs. Thus, it becomes essential to catch the defects as early as possible. Industry data has reaffirmed these findings. While there is no consensus about the costs incurred due to delay in defect detection, a defect introduced during the requirement phase that makes it to the final release may cost as much as a thousand times the cost of detecting and correcting the defect during requirements gathering itself.

 

1.6 THE CAT AND THE SAINT

 

Testing requires asking about and understanding what you are trying to test, knowing what the correct outcome is, and why you are performing any test. If we carry out tests without understanding why we are running them, we will end up in running inappropriate tests that do not address what the product should do. In fact, it may even turn out that the product is modified to make sure the tests are run successfully, even if the product does not meet the intended customer needs!

 

Understanding the rationale of why we are testing certain functionality leads to different types of tests, which we will cover in Part II of the book. We do white box testing to check the various paths in the code and make sure they are exercised correctly. Knowing which code paths should be exercised for a given test enables making necessary changes to ensure that appropriate paths are covered. Knowing the external functionality of what the product should do, we design black box tests. Integration tests are used to make sure that the different components fit together. Internationalization testing is used to ensure that the product works with multiple languages found in different parts of the world. Regression testing is done to ensure that changes work as designed and do not have any unintended side-effects.

1.7 TEST THE TESTS FIRST!

 

From the above example, it is clear that it is the audiologist who has a hearing problem, not the patient! Imagine if the doctor prescribed a treatment for the patient assuming that the latter could not hear at 20 feet and 30 feet.

Tests are also artifacts produced by human beings, much as programs and documents are. We cannot assume that the tests will be perfect either! It is important to make sure that the tests themselves are not faulty before we start using them. One way of making sure that the tests are tested is to document the inputs and expected outputs for a given test and have this description validated by an expert or get it counter-checked by some means outside the tests themselves. For example, by giving a known input value and separately tracing out the path to be followed by the program or the process, one can manually ascertain the output that should be obtained. By comparing this “known correct result” with the result produced by the product, the confidence level of the test and the product can be increased. The practices of reviews and inspection and meticulous test planning discussed in Chapter 3 and Chapter 15 provide means to test the test.

 

1.8 THE PESTICIDE PARADOX

Defects are like pests; testing is like designing the right pesticides to catch and kill the pests; and the test cases that are written are like pesticides. Just like pests, defects develop immunity against test cases! As and when we write new test cases and uncover new defects in the product, other defects that were “hiding” underneath show up.

 

There are two possible ways to explain how products develop this “immunity” against test cases. One explanation is that the initial tests go a certain distance into the code and are stopped from proceeding further because of the defects they encounter. Once these defects are fixed, the tests proceed further, encounter newer parts of the code that have not been dealt with before, and uncover new defects. This takes a “white box” or a code approach to explain why new defects get unearthed with newer tests.

 

A second explanation for immunity is that when users (testers) start using (exercising) a product, the initial defects prevent them from using the full external functionality. As tests are run, defects are uncovered, and problems are fixed, users get to explore new functionality that has not been used before and this causes newer defects to be exposed. This “black box” view takes a functionality approach to explain the cause for this “more we test more defects come up” phenomenon.

An alternative way of looking at this problem is not that the defects develop immunity but the tests go deeper to further diagnose a problem and thus eventually “kill the defects.” Unfortunately, given the complex nature of software and the interactions among multiple components, this final kill happens very rarely. Defects still survive the tests, haunt the customers, and cause untold havoc.

The need for constantly revising the tests to be run, with the intent of identifying new strains of the defects, will take us to test planning and different types of tests, especially regression tests. Regression tests acknowledge that new fixes (pesticides) can cause new “side-effects” (new strains of pests) and can also cause some older defects to appear. The challenge in designing and running regression tests centers around designing the right tests to combat new defects introduced by the immunity acquired by a program against old test cases. We will discuss regression tests in Chapter 8.

1.9 THE CONVOY AND THE RAGS

 

Defects in a program also typically display this convoy phenomenon. They occur in clusters. Glenford Myers, in his seminal work on software testing [MYER-79], proposed that the probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section.

 

This may sound counter-intuitive, but can be logically reasoned out. A fix for one defect generally introduces some instability and necessitates another fix. All these fixes produce side-effects that eventually cause the convoy of defects in certain parts of the product.

From a test planning perspective, this means that if we find defects in a particular part of product, more—not less—effort should be spent on testing that part. This will increase the return on investments in testing as the purpose of testing is find the defects. This also means that whenever a product undergoes any change, these error-prone areas need to be tested as they may get affected. We will cover these aspects in Chapter 8, Regression Testing.

 

 

A fix for a defect is made around certain lines of code. This fix can produce side-effects around the same piece of code. This sets in spiraling changes to the program, all localized to certain select portions of the code. When we look at the code that got the fixes for the convoy of defects, it is likely to look like a piece of rag! Fixing a tear in one place in a shirt would most likely cause damage in another place. The only long-term solution in such a case is to throw away the shirt and create a new one. This amounts to a re-architecting the design and rewriting the code.

1.10 THE POLICEMEN ON THE BRIDGE

 

Testers are probably best equipped to know the problems customers may encounter. Like the second police officer in the above story, they know people fall and they know why people fall. Rather than simply catch people who fall (and thereby be exposed to the risk of a missed catch), they should also look at the root cause for falling and advise preventive action. It may not be possible for testers themselves to carry out preventive action. Just as the second police officer had to enlist the help of an engineer to plug the hole, testers would have to work with development engineers to make sure the root cause of the defects are addressed. The testers should not feel that by eliminating the problems totally their jobs are at stake. Like the second policeman, their careers can be enriching and beneficial to the organization if they harness their defect detection experience and transform some of it to defect prevention initiatives.

 

Defect prevention is a part of a tester's job. A career as a tester can be enriching and rewarding, if we can balance defect prevention and defect detection activities. Some of these career path possibilities are encapsulated in a three-stage model in Chapter 13, Common People Issues. We will now visit the question of what is the right balance between defect prevention and defect detection.

1.11 THE ENDS OF THE PENDULUM

The eventual goal of any software organization is to ensure that the customers get products that are reasonably free of defects. There are two approaches to achieving this goal. One is to focus on defect detection and correction and the second is to focus on defect prevention. These are also called quality control focus and quality assurance focus.

Testing is traditionally considered as a quality control activity, with an emphasis on defect detection and correction. We would like to take a broader view of testing and believe that there are aspects of testing that are also defect prevention oriented. For example, one of the aspects of white box testing, discussed in Chapter 3, is static testing, that involves desk checking, code walkthroughs, code reviews, and inspection. Even though these are traditionally considered as “quality assurance” activities, planning for overall testing activities with an intent to deliver quality products to customers, cannot be done effectively unless we take a holistic view of what can be done using quality assurance and what can be done with quality control (or the traditional definition of testing).

Quality assurance is normally associated with process models such as CMM, CMMI, ISO 9001, and so on. Quality control, on the other hand, is associated with testing (that form the bulk of the discussions in this book). This has caused an unnatural dichotomy between these two functions. Unfortunately, organizations view these two functions as mutually exclusive, “either-or” choices. We have even heard statements such as “with good processes, testing becomes redundant” or “processes are mere overheads—we can find out everything by testing.” It is almost as if there are two schools of thought at either extremes of a pendulum—one rooting for defect prevention (quality assurance) focus and the other rooting for the defect detection (quality control) focus. It is also common to find an organization swinging from one extreme to another over time, like a pendulum (Figure 1.5).

 

 

Rather than view defect prevention and defect detection as mutually exclusive functions or ends of a pendulum, we believe it is worthwhile to view these two as supplementary activities, being done in the right mix. Figure 1.6 gives a defect prevention—defect detection grid, which views the two functions as two dimensions. The right mix of the two activities corresponds to choosing the right quadrant in this grid.

 

 

When the focus on defect prevention is low, the emphasis on the use of appropriate standards, reviews, and processes are very low. This acts as an ideal “breeding ground” for defects. Most of the effort in ensuring quality of a product is left in the hands of the testing and defect detection team. If the focus on defect detection is also low (represented by the lower left-hand quadrant), this is a bad state for an organization to be in. Lack of testing and defect detection activities does not “kill” these defects in time; hence the defects reach the customers. This is obviously not a healthy state to be in.

Even when the defect detection focus increases, with continued low defect prevention focus (upper left hand quadrant), the testing functions become a high-adrenalin rush, high-pressure job. Most defects are detected in the last minute—before the product release. Testers thus become superheroes who “save the day” by finding all the defects just in time. They may also become adversaries to developers as they always seem to find problems in what the developers do. This quadrant is better than the previous one, but ends up being difficult to sustain because the last-minute adrenalin rush burns people out faster.

 

Preventing an illness is more effective than curing it. People who prevent defects usually do not get much attention. They are usually the unsung heroes of an organization. Those who put out the fires are the ones who get visibility, not necessarily those who make sure fires do not happen in the first place. This, however, should not deter the motivation of people from defect prevention.

As we saw in the previous section, defect prevention and defect detection are not mutually exclusive. They need to be balanced properly for producing a quality product. Defect prevention improves the quality of the process producing the products while defect detection and testing is needed to catch and correct defects that escape the process. Defect prevention is thus process focused while defect detection is product focused. Defect detection acts as an extra check to augment the effectiveness of defect prevention.

An increase in defect prevention focus enables putting in place review mechanisms, upfront standards to be followed, and documented processes for performing the job. This upfront and proactive focus on doing things right to start with causes the testing (or defect detection) function to add more value, and enables catching any residual defects (that escape the defect prevention activities) before the defects reach the customers. Quality is institutionalized with this consistently high focus on both defect prevention and defect detection. An organization may have to allocate sufficient resources for sustaining a high level of both defect prevention and defect detection activities (upper right-hand quadrant in Figure 1.6).

 

However, an organization should be careful about not relying too much on defect prevention and reducing the focus on defect detection (lower right-hand quadrant in Figure 1.6). Such a high focus on defect prevention and low focus on defect detection would not create a feeling of comfort amongst the management on the quality of product released since there are likely to be minimal internal defects found. This feeling will give rise to introduction of new processes to improve the effectiveness of defect detection. Too much of processes and such defect prevention initiatives may end up being perceived as a bureaucratic exercise, not flexible or adaptable to different scenarios. While processes bring in discipline and reduce dependency on specific individuals, they—when not implemented in spirit—could also end up being double-edged swords, acting as a damper to people's drive and initiative. When an organization pays equally high emphasis to defect prevention and defect detection (upper right corner in the grid), it may appear that it is expensive but this investment is bound to have a rich payback by institutional quality internally and making the benefits visible externally to the customers.

An organization should choose the right place on each of these two—defect detection and defect prevention—dimensions and thus choose the right place in the grid. The relative emphasis to be placed on the two dimensions will vary with the type of product, closeness to the release date, and the resources available. Making a conscious choice of the balance by considering the various factors will enable an organization to produce better quality products. It is important for an organization not to over-emphasize one of these at the expense of the other, as the next section will show.

1.12 MEN IN BLACK

As we can see from all the above discussions, testing requires abundant talent in multiple dimensions. People in the testing profession should have a customer focus, understanding the implications from the customer's perspective. They should have adequate analytical skills to be able to choose the right subset of tests and be able to counter the pesticide paradox. They should think ahead in terms of defect prevention and yet be able to spot and rectify errors that crop up. Finally (as we will see in the next section), they must be able to perform automation functions.

 

Despite all these challenging technical and inter-personal skills required, testing still remains a not-much-sought-after function. There was an interesting experiment that was described by De Marco and Lister in their book, Peopleware [DEMA-1987]. The testing team was seeded with motivated people who were “free from cognitive dissonance that hampers developers when testing their own programs.” The team was given an identity (by a black dress, amidst the traditionally dressed remainder of the organization) and tremendous importance. All this increased their pride in work and made their performance grow by leaps and bounds, “almost like magic.” Long after the individual founding members left and were replaced by new people, the “Black Team” continued its existence and reputation.

The biggest bottleneck in taking up testing as a profession is the lack of self-belief. This lack of self-belief and apparent distrust of the existence of career options in testing makes people view the profession as a launching pad to do other software functions (notably, “development,” a euphemism for coding). As a result, testers do not necessarily seek a career path in testing and develop skepticism towards the profession.

We have devoted an entire chapter in Part III of the book to career aspirations and other similar issues that people face. A part of the challenge that is faced is the context of globalization—the need to harness global resources to stay competitive. We address the organizational issues arising out of this in another chapter in Part III.

1.13 AUTOMATION SYNDROME

 

If you go through the story closely there appear to be several reasons for the crop failures that are not to do with the automation intent at all. The frustration of the farmer should not be directed at automation but on the process followed for automation and the inappropriate choices made. In the second crop cycle, the reason for failure was lack of skills and in the third cycle it is due to improper tool implementation.

 

In the first crop cycle, the farmer laid off his workers immediately after the purchase of motorcycles and expected cost and time to come down. He repeated the same mistake for the third crop cycle. Automation does not yield results immediately.

The moral of the above story as it applies to testing is that automation requires careful planning, evaluation, and training. Automation may not produce immediate returns. An organization that expects immediate returns from automation may end up being disappointed and wrongly blame automation for their failures, instead of objectively looking at their level of preparedness for automation in terms of planning, evaluation, and training.

A large number of organizations fail in their automation initiatives and revert to manual testing. Unfortunately, they conclude—wrongly—that automation will never work.

Testing, by nature, involves repetitive work. Thus, it lends itself naturally to automation. However, automation is a double-edged sword. Some of the points that should be kept in mind while harping on automation are as follows.

• Know first why you want to automate and what you want to automate, before recommending automation for automation's sake.
• Evaluate multiple tools before choosing one as being most appropriate for your need.
• Try to choose tools to match your needs, rather than changing your needs to match the tool's capabilities.
• Train people first before expecting them to be productive.
• Do not expect overnight returns from automation.

1.14 PUTTING IT ALL TOGETHER

We have discussed several basic principles of testing in this chapter. These principles provide an anchor to the other chapters that we have in rest of the book. We have organized the book into five parts. The first part (which includes this chapter) is Setting the Context, which sets the context for the rest of the book. In the chapter that follows, we cover Software Development Life Cycle (SDLC) Models in the context of testing, verification and validation activities.

In Part II, Types of Testing, we cover the common types of testing. Chapters 3 through 10 cover white box testing, black box testing, integration testing, system and acceptance testing, performance testing, regression testing, internationalization testing, and ad hoc testing.

Part III, Select Topics in Specialized Testing, addresses two specific and somewhat esoteric testing topics—object oriented testing in Chapter 11 and usability and accessibility testing in Chapter 12.

Part IV, People and Organizational Issues in Testing, provides an oftignored perspective. Chapter 13 addresses the common people issues like misconceptions, career path concerns and so on. Chapter 14 address the different organizational structures in vogue to set up effective testing teams, especially in the context of globalization.

The final part, Part V, Test Management and Automation, addresses the process, management, and automation issues to ensure effective testing in an organization. Chapter 16 discusses test planning management and execution. This discusses various aspects of putting together a test plan, tracking a testing project and related issues. Chapter 17 goes into details of the benefits, challenges, and approaches in test automation—an area of emerging and increasing importance in the test community. The final chapter, Chapter 18, goes into details of what data are required to be captured and what analysis is to be performed for measuring effectiveness of testing, quality of a product and similar perspectives and how this information can be used to achieve quantifiable continuous improvement.

While we have provided the necessary theoretical foundation in different parts of the book, our emphasis throughout the book has been on the state of practice. This section should set the context for what the reader can expect in rest of the book.

One of the early seminal works on testing is [MYER-79]. In particular, the example of trying to write test cases for verifying three numbers to be the sides of a valid triangle still remains one of the best ways to bring forth the principles of testing. [DEMA-87] provides several interesting perspectives of the entire software engineering discipline. The concept of black team has been illustrated in that work. The emphasis required for process and quality assurance methodologies and the balance to be struck between quality assurance and quality control are brought out in [HUMP-86]. Some of the universally applicable quality principles are discussed in the classics [CROS-80] and [DEMI-86]. [DIJK-72], a Turing Award lecture brings out the doctrine of program testing can never prove the absence of defects. [BEIZ-90] discusses the pesticide paradox.

1. We have talked about the pervasiveness of software as a reason why defects left in a product would get detected sooner than later. Assume that televisions with embedded software were able to download, install and self-correct patches over the cable network automatically and the TV manufacturer told you that this would just take five minutes every week “at no cost to you, the consumer.” Would you agree? Give some reasons why this is not acceptable.
2. Your organization has been successful in developing a client-server application that is installed at several customer locations. You are changing the application to be a hosted, web-based application that anyone can use after a simple registration process. Outline some of the challenges that you should expect from a quality and testing perspective of the changed application.
3. The following were some of the statements made by people in a product development organization. Identify the fallacies if any in the statements and relate it to the principles discussed in this chapter.
   1. “The code for this product is generated automatically by a CASE tool — it is therefore defect - free.”
   2. “We are certified according to the latest process models — we do not need testing.”
   3. “We need to test the software with dot matrix printers because we have never released a product without testing with a dot matrix printer.”
   4. “I have run all the tests that I have been running for the last two releases and I don't need to run any more tests.”
   5. “This automation tool is being used by our competitors -hence we should also use the same tool.”
4. Assume that each defect in gathering requirements allowed to go to customers costs $10, 000, and that the corresponding costs for design defects and coding defects are $1, 000 and $100, respectively. Also, assume that current statistics indicate that on average ten new defects come from each of the phases. In addition, each phase also lets the defects from the previous phase seep through. What is the total cost of the defects under the current scenario? If you put a quality assurance process to catch 50% of the defects from each phase not to go to the next phase, what are the expected cost savings?
5. You are to write a program that adds two two-digit integers. Can you test this program exhaustively? If so, how many test cases are required? Assuming that each test case can be executed and analyzed in one second, how long would it take for you to run all the tests?
6. We argued that the number of defects left in a program is proportional to the number of defects detected. Give reasons why this argument looks counterintuitive. Also, give practical reasons why this phenomenon causes problems in testing.