Chapter 20. QA and Testing in the ALM
Quality assurance (QA) and testing are essential to any software or systems lifecycle. Most technology professionals view the QA and testing process as simply executing test cases to verify and validate that requirements have been met and that the system functions as expected. But there is a lot more to QA and testing, and this chapter will help you understand how to establish effective processes to help ensure that your system functions as needed. DevOps helps us build, package, and deploy software much more quickly. Too often, the QA and testing process cannot keep up with the accelerated deployment pipeline. DevOps cannot succeed without excellent QA and testing.
20.1 Goals of QA and Testing
The goal of QA and testing is to support the iterative nature of agile ALM by ensuring that there is an automated testing process that can keep pace with the DevOps approach to application deployment. After so many process improvement successes, many organizations find that their QA and testing functions are actually left in the dust. Your goal should be to have a comprehensive approach to testing that can help your DevOps and agile ALM reach its full potential.
20.2 Why Are QA and Testing Important?
QA and testing are important because your customers want you to deliver defect-free code that accomplishes its objectives. It is absolutely vital that your testing process be both comprehensive and yet able to keep up with the rapid rate of change inherent in agile development and DevOps in particular. Without excellent testing, your developers will be mired in supporting system issues and fixing bugs. Testing must be started from the very beginning of the process, including unit testing through functional and regression testing and more. QA and testing is the lifeguard in the pool.
20.3 Where Do I Start?
Getting started with QA and testing involves understanding your existing processes along with the demands of your emerging DevOps practices. We find that teams may have adequate manual testing, but these measures won’t meet the needs of rapid application delivery when organizations adopt a DevOps approach. You must assess your ability to automate your testing throughout the software and systems lifecycle. Although unit and functional testing are a must-have, effective testing needs to include a much deeper approach, such as testing directly at the application programming interface (API) level. With this foundation, you can successfully move on to creating message proxies and implementing emerging techniques such as service virtualization testing, which we will discuss in the next section.
The nature of the agile iterative development process and the high rate of deployment enabled by DevOps are often limited by the lack of robust automated testing. Many teams are finding that even if they can deploy faster, QA and testing approvals often lag behind.
To ensure effective and timely QA and testing, you need to have a comprehensive test plan that is simultaneously pragmatic and comprehensive.
20.4 Planning the Testing Process
The people responsible for planning the testing process need to take a comprehensive view encompassing the entire agile ALM. We like the term continuous testing because it denotes the need for an approach that encompasses the entire software and systems lifecycle.
Developers benefit greatly from starting with unit testing and especially test-driven development (TDD). Functional and regression testing are also fundamental and, for many projects, another basic starting point. But there needs to be a much deeper approach to testing that provides more comprehensive test coverage. When applications have an exposed API available for use, then automated testing can utilize the published API for robust automated testing. Many testing tools make use of messaging protocols, which provide an effective and comprehensive approach to automated testing.
We have seen robust testing tools, which included proxies to capture messaging protocols, effectively creating a “record and playback” script. These tools usually allow you to make copies of the message package and modify them, effectively making test cases that follow logical paths that may not even be accessible from the user interface. A number of excellent testing frameworks allow you to effectively create virtualized test assets that function in a way similar to mocks. This approach has become known as service virtualization testing and should be on your list of must-have testing approaches. Make sure that you don’t forget to embrace performance testing and nonfunctional testing as well.
Since your testing process needs to take a comprehensive approach, we strongly recommend that you at least evaluate and consider some of the excellent testing frameworks available today
Some of these popular techniques are pretty complicated, and we have seen considerable success with taking a DevOps approach in which the developer works closely with a test engineer who is familiar with these tools, much like XP-paired programming. With this approach, you get the value of a technical resource who knows the application, combined with the expertise of an engineer who understands the test tools.
Similarly, you should take a DevOps approach to writing comprehensive test cases.
20.5 Creating the Test Cases
Test cases should explain what you are going to test and what you expect the results to be. Test cases can help supplement requirements documents or agile stories by explaining how the application is expected to work. We view the test script as providing the step-by-step instructions on how to execute the test, and this information is essential. Requirements documents are often incomplete due to real-life constraints, but your test cases and scripts must be comprehensive and up to date in order to ensure that you maintain quality.
20.6 Ensuring Quality
There are many ways to ensure quality. From defining requirements to test cases and test scripts, eliminating defects is a key goal of the agile ALM. Taking a DevOps approach to quality will help you achieve success. For example, cross-functional team code reviews are known to be extremely effective. Yet, ensuring high overall quality demands a much wider approach.
Effective version control procedures, including the best use of branching and merging, build engineering (described in Chapter 6), and continuous integration (described in Chapter 8) are a critical component. Throughout this book, we have been describing many strategies that help ensure quality. You need to ensure that you are taking a holistic and comprehensive approach. QA and testing are absolutely required, but the fact is that your entire ALM helps provide quality. Make sure that you always build in quality from the beginning of the agile ALM.
20.7 Ensuring Quality from the Beginning
The best way to approach quality is to focus on it from the very beginning. We like to invoke the term “left-shift” to denote the involvement of operations engineers early in the lifecycle. Getting the QA and testing function involved from the beginning of the software and systems delivery process is crucial. Professional test engineers know how to structure the testing process to be comprehensive and efficient. Taking a DevOps approach of pairing developers with test engineers is the best approach. Have them start with unit testing and work their way all the way through the software delivery process to production and ongoing systems support. We find that some test scripts actually form the basis of environment monitoring, discussed in Chapter 11, which is a key consideration in ensuring that systems are reliable and secure.
20.8 Conclusion
It is always best to take a comprehensive approach to QA and testing. Although certainly a necessary and good first step, unit testing is usually limited in scope. Functional and regression testing are also typically limited to the user interface and consequently also limited in scope. For optimal quality, you need to adopt automated continuous testing from the very beginning of the software and systems delivery lifecycle. We advocate embracing evolving API and service virtualization testing to create a holistic approach to automated testing that can truly enable the DevOps capabilities to deploy as often as desired.