Preface

This book will show you how to comprehensively test new software features, whether implemented on web pages, a desktop, or mobile applications. It covers everything from generating in-depth specifications to detailed black- and white-box testing, handling error cases and user experience considerations, to areas sometimes missed off test plans such as maintainability and operating under failure conditions.

The same bugs and defects crop up repeatedly in software development, so this book has a huge range of tests to run and scenarios to check to help you achieve great test coverage. They’re illustrated by dozens of example issues I’ve encountered during my decades in the software industry. While it’s never possible to guarantee a product is bug-free, the recommendations here give you the best chance of catching defects.

Who this book is for

Many different people can perform testing during a development cycle. In addition to dedicated testers, developers should always try their code, and product owners, support engineers, and technical writers can also provide more or less formal feedback.

This book is for anyone involved in software testing, regardless of their official job title. Everyone can use the suggestions here to increase test coverage and improve the quality of the features you release.

Testing has a lovely learning curve. You can start with manual testing on easy-to-use public interfaces that have been well polished and documented. From there, you can go deeper into the system, checking logs and metrics and using internal interfaces. Testers with the right skill set can improve code directly by performing code reviews and writing unit tests. This book describes all these approaches to improve your testing, whether you’re a novice or an expert tester.

Is this book for Agile or waterfall projects? Both!

This book does not discuss different development models such as Agile or waterfall. Whichever you are using, you will need to design a comprehensive test plan to exercise new features and discover their defects.

However, there will be significant timing differences depending on the method you’re using. Waterfall projects may extend over many months, with weeks allocated to the different areas of testing described here. In contrast, Agile projects rapidly cycle through the different types of testing, doing a little of each in cycle after cycle.

Despite their differences, both project types, and the many real-world hybrids used in practice, require a thorough test plan covering the many different aspects of system behavior. Whatever project type you work on, this book can help you write better test plans.

Is this book for manual or automated testing? Both!

This book has many proposals for what you should test. It does not cover how those tests should be run. The aim, after applying this book to your feature, is to produce a comprehensive test plan with steps to detect many defect types.

How you run that test plan is up to you. Some areas must be done manually – exploratory testing, for instance, involves a tester getting their hands on a feature to investigate how it behaves in an unstructured way. Only once you have learned how it works can you write the feature specification and do systematic testing, either manual or automated. User experience testing is also best started with a genuine user present. Conversely, load testing requires automation to reach the required load levels and cannot be done by hand.

In between much of the functional testing, security, error cases, and even some user experience testing can be automated. Where possible, that is a vital stage in the testing process to free up testers’ time for future test designs. That is especially true when following a Continuous Integration/Continuous Deployment (CI/CD) pipeline, in which changes are pushed to live systems within minutes. There, you must automate tests and carefully choose which to run on each change.

This book does not discuss how to perform manual testing, automate tests, or what tools you should use. Those are subjects for other books in the Packt catalog. Here, we will describe what tests you should run and how you should design your test plan.

Types of testing

There are several levels of testing, but this book focuses on just one – system testing. This is the hardest part of testing because it covers the whole system working together, end to end. Because it is so difficult, it requires the thought and planning described here. That does not mean it is the only kind of testing. You should supplement all the system testing here with testing at other levels since they each have different strengths.

At the lowest level is unit testing, also known as component testing, where scripts exercise individual functions or modules. This form of testing is often performed by developers, as they know what to expect from their code and are best placed to write test harnesses to drive it. Unit tests can thoroughly test individual actions, but they can’t find issues that arise when functions are used together.

To find those issues, you need integration testing. This approach involves running entire modules isolated from the rest of the system, such as user interfaces or databases. At this stage, the testing becomes much more complex. Because far more can go wrong, it allows you to find another class of issues. While both function A and function B may work well on their own, incorrect assumptions or implementations might mean they do not work together. Integration testing finds that class of bugs. It can be carried out by either developers or dedicated testers, although it requires a significant amount of automation and is unsuitable for manual testing. Using modules or clients independently of the rest of the system needs test harnesses to stand in for areas that are not being run.

Beyond integration testing is system testing, the focus of this book. Once the whole system is working together, with all modules integrated with the others, then the final class of bugs is revealed. These issues only occur when running the full system in a realistic environment, such as loading and problems between modules. On the plus side, system testing can find all the realistic bugs that will affect users. On the downside, it is difficult to translate from system behavior down to the individual functions that are failing, and systems can be complex to test fully. This book aims to give techniques and ideas to successfully navigate that challenge.

The naming convention used here follows the International Software Testing Qualification Board (ISTQB) syllabus, which refers to component, integration, and system testing. There is some overlap with the topics covered there, although this book takes a far more hands-on, practical approach rather than focusing on the theory. By selecting the most useful pieces of the ISTQB syllabus, and expanding their practical aspects, this book will show you how to thoroughly test software features.

Test process

There are many important topics that this book can’t address. This book is only about writing excellent test plans to find a wide variety of issues in your application. It does not cover test teams or anything associated with test processes.

Test prioritization is not covered in detail here. The aim is to give you a large bank of ideas and test scenarios, from which you can decide their relative priority for your application. Judging risk and the likelihood and impact of issues will depend on your circumstances. It’s up to you to apply these ideas.

This book also doesn’t cover team organization. How many testers do you need in your company? How should they be arranged to work with the product owner and developers? How much time should they devote to the different forms of testing? It also doesn’t cover managing communication, such as how bugs should be raised and their life cycle, but it will tell you how to find those bugs.

Finally, this book doesn’t describe the release process, including how test results should be evaluated, internal usage and beta programs, and how to sign off and perform releases. Again, that depends on your product and industry. Even if you don’t decide to run all the tests here, you can deliberately choose which ones you exclude so that you have the best possible idea of the risks you are taking.

The aim, by the end of this book, is that you should be able to quickly prepare a comprehensive test plan, rapidly covering many different areas of testing relevant to your product and its features. Often in software development, the factor limiting your speed is identifying what you should do, so by understanding the main types of testing and common failures, you can develop tests faster. That lets you find more issues earlier in the development cycle, save time, speed up projects, and give your customers a better user experience.

With that said, we can now see the areas this book covers in detail.

What this book covers

Part 1, Preparing to Test, covers the necessary steps before you can start writing the test plan.

Chapter 1, Making the Most of Exploratory Testing, describes how best to perform exploratory testing and when you should do it. The main goal of exploratory testing is not to find bugs but to understand a feature better in order to improve the feature specification and test plans. It also finds any blocking issues that may delay future testing.

Chapter 2, Writing Great Feature Specifications, shows how to write comprehensive and useful feature specifications. This document will be the basis for all subsequent testing, so it needs to cover all the questions that could arise. That requires input from multiple groups, so this document always needs a thorough review.

That specification review is covered in Chapter 3, How to Run Successful Specification Reviews, in which you check the specification with the developers and product owners. The developers should highlight any special cases or conditions they had to add code for, while the product owner makes decisions that could affect the user experience and the scope of the changes. Successfully running that review to get what testers need from it is your responsibility.

Armed with the specification and the experience of exploratory testing, you can start designing the test plan. Part 2, Functional Testing, of this book explains the many different types of functional testing, in which your product responds to different inputs.

Chapter 4, Test Types, Cases, and Environments, examines how best to write test cases and where you should run them. It shows how unit, integration, and system tests can work together and where they fit in the release cycle.

Chapter 5, Black-Box Functional Testing, is the most familiar area of testing, and the first one people think of. When you use the feature, does it work correctly and achieve its stated goals? Even within this type of testing, there are many ideas and suggestions for how to find common weaknesses and issues.

Chapter 6, White-Box Functional Testing, is informed by a knowledge of the underlying code and its architecture. Understanding how the code works lets you add another important set of tests.

Chapter 7, Testing of Error Cases, is devoted to invalid inputs or situations and how the application handles them. This is a large area of testing because the number of invalid inputs often massively outweighs the possible valid inputs. In addition, error cases are more likely to have defects because less time is spent thinking through their consequences than happy path scenarios.

Chapter 8, User Experience Testing, covers the unique set of considerations to ensure your application is easy to use. This is far more subjective than other areas of testing and may require judgments on, for instance, which command name is clearest, or which function will be used most and should be accessed most easily. It’s vital to get those questions right to give your users the best experience.

Chapter 9, Security Testing, lists common security vulnerabilities that should routinely be checked by your test plans, as well as different attacks you can craft to ensure the appropriate protections are in place.

Chapter 10, Maintainability, considers your application’s logging, event generation, and monitoring. If there is an issue, how easy will it be to diagnose and fix? This can be of lower importance to businesses, since it is not directly customer-facing, but is of high importance to the test team and other internal users who spend large amounts of time chasing down issues.

Those chapters conclude the functional areas of testing, where you perform a certain action and check a certain outcome. If the application under test has passed all these tests, then it will work for users under normal circumstances. Part 3, Non-Functional Testing, considers abnormal circumstances such as high load and system failures.

Chapter 11, Destructive Testsing, considers scenarios in which different parts of the system are deliberately disabled or degraded to ensure that the rest of the system behaves gracefully and can recover from issues.

Chapter 12, Load Testing, checks the behavior when your application runs at its maximum performance. While it may be able to perform individual actions correctly, is it reliable when they’re repeated many times? These tests also check that your application continues to perform well, even when some subsystems are placed under load.

Finally, Chapter 13, Stress Testing, describes tests that deliberately load the system beyond its capabilities – for example, if too many entities are configured or if the system is taken beyond its loading limits. As with destructive testing, the correct behavior is to fail gracefully and recover once the stress condition is lifted.

The Appendix contains an example test plan that puts these ideas into practice for an example feature – users signing up to a web page.

Applying this guide gives you the best chance of finding defects in your software. However, despite all the suggestions here, in my experience, you find the most interesting bugs when you go off the test plan. Trying out ideas for yourself or following up on something odd that you spot is a great way to think up test cases no one else has considered. Observation and curiosity are vital throughout testing, so always keep your eyes open.

The suggestions here aren’t a recipe to be followed but instead a guide from which you can create your own test plans. Testing is like fishing. You can’t guarantee what you’ll find, and there’ll be plenty of smaller catches among the big issues that you’re really seeking. However, by looking in the most promising places and using the best techniques, you give yourself the greatest chance of success. Happy bug hunting!

Download the color images

We also provide a PDF file that has color images of the screenshots and diagrams used in this book. You can download it here: https://packt.link/W0Uic.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “For instance, to test whether a textbox can handle inputs including spaces, we may use the string one two.”

A block of code is set as follows:

create_user(permission)
{
     user = create_user()
     if(permission == ADMIN)
     {
           make_admin(user)
     }
     return user
}

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “That limit should be apparent before users try to use it – the Add user button should be grayed out with a note saying why, for instance, instead of letting users click the button.”

Tip or Important note

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Any errata related to this book can be found at https://github.com/PacktPublishing/Software-Test-Design.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you’ve read Software Test Design, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

1. Scan the QR code or visit the link below

https://packt.link/free-ebook/9781804612569

2. Submit your proof of purchase
3. That’s it! We’ll send your free PDF and other benefits to your email directly