With the wide adoption of Agile methodologies in the software development industry, delivery of software within a shorter time period has become the norm. Automation is a buzzword in today’s software world, as it is an essential part of successful Agile implementation. Every possible aspect of software development and delivery needs to be automated as much as possible to meet the demand for getting high-quality, state of the art software solutions to the market on time.
Maintaining quality standards while delivering software in shorter cadence is a challenge for any software vendor, as verifying each and every aspect of the developed software is a time-consuming effort. Validating only recent modifications and putting software components into production could result in disasters if critical functionality is broken. Automating the testing aspect of software as much as possible and handing over the testing job to computers, with a minimal need for human interaction, is the key to success in assuring high-quality software.
In this book, you will be provided with hands-on experience to get you started with automation testing using Microsoft Visual Studio in combination with Selenium and other test automation tools and frameworks. Lessons will start with simple steps, and as you read through the chapters, you will be diving into different aspects of testing, such as load and performance testing, cloud-based load testing, and test data management options. Additionally, the book will take you through simple implementation of test automation executions using containerized test execution clients.
Before moving into the lessons-based hands-on learning, this chapter will give you a broader understanding of the need for software test automation, where and when test automation is applicable, and getting the software test automation integrated with software delivery automation. Further, we will discuss the Return on Investment (ROI) aspect of test automation, which is essential to getting the buy-in from the management of any company to get the required support and sponsorship to implement software test automation.
Examining Different Software Testing Types
Software testing can be divided into two broader categories—namely, Functional Testing and Non-Functional Testing. Each of these categories has different types of testing included in them.
Functional Testing
Unit Testing: Testing of an individual module or software component that is generally performed by the developer/programmer of such module or component. This type of testing requires deeper understanding of the design and implementation of the software component or module that is being tested.
Integration Testing: Testing the combined functionality of modules integrated together in software is referred to as integration testing. The modules can be individual applications or code modules that have interdependencies to perform a function.
Smoke Testing: Smoke tests are used to validate a system after a new build to ensure there are no showstoppers for execution of the system, covering all general functional scenarios of the software system. If smoke testing fails, further testing of the system generally is not carried out until the build is fixed so that the smoke tests can be run without failures.
Sanity Testing: A new software build or version is validated for a minimal level of successful execution of its functionality to allow further testing to be carried out on the new version. If sanity tests fail, no further testing is executed until the software system version passes all sanity tests.
API Testing: As a part of integration testing, sometimes application programming interfaces (APIs) are directly tested for input and output.
Regression Testing: Testing a software system as a whole, covering all modules, is termed as Regression testing. Before delivering a version upgrade to production, software is expected to go through a regression testing cycle to ensure no functionality in any part of the system is broken.
Acceptance Testing: The software system is tested in production-equivalent environments to verify all the business requirements are satisfied with end-to-end flows of the system. This validation is performed with involvement of the client of the software system, and this testing is called User Acceptance Testing (UAT).
Exploratory Testing: Exploring the software system functionality without following a specific flow defined in a test case and identifying any issues and potential test cases/flows is referred to as exploratory testing.
Non-Functional Testing
Performance Testing: Verification of a system to ensure it meets the performance expectations. This is comprised of stress and load testing. Performance testing determines whether a system can meet performance demands while under stress or load.
Stress Testing: Stress testing is performed to check the system’s ability to handle data volume and traffic beyond its requirements or expectations so that how and when it fails can be identified.
Load Testing: Load testing verifies how much load a system can handle without a performance degrade. Unlike stress testing, the maximum load applied in load testing is generally the maximum limit of the specification or slight additional load beyond maximum load specification. The performance of system and the infrastructure of the system are monitored during the load testing to identify the bottlenecks.
Volume Testing: Verification of the system behavior with a high volume of data is performed in volume testing. The next level of volume testing is increasing the volume to stress the system in order to perform stress testing.
Reliability Testing: Reliability testing verifies whether the system can perform without any failures for a given period.
Usability Testing: Usability testing verifies the capability of a new user to easily understand the flow of the system and use it without any difficulty. The availability of proper help or documentation, such as user guides, is also validated.
Security/Vulnerability Testing: Weaknesses in software, hardware, network, etc. are verified in vulnerability testing to prevent hackers or malicious software from causing issues or controlling the systems. Systems used in military, air traffic control, and space programs, etc. are highly tested for vulnerabilities.
Recovery Testing: The capability of the system to recover from crashes or disasters is validated in recovery testing. For example, a system should be able to complete its operations without failures to its functional flow, even in a situation of sudden network failure or in a server restart, once the relevant network or server comes back online. Not even one server should be visible or impact the flow of the system.
Compliance Testing: Validation of whether a system meets the organization- or client-specified standards is known as compliance testing.
Compatibility Testing: Verification of a system’s behavior on different platforms, hardware, networking, browsers, etc. is identified as compatibility testing. This includes previous version support as well, which is known as backward compatibility testing.
Install/Uninstall Testing: Capability of a system to set up and be removed from different hardware and networks without having any issues is tested under install/uninstall testing.
Localization/Globalization Testing: A software system’s ability to work for specific culture and locale settings is verified in localization testing. Globalization testing checks whether a system is able to work in any culture or locale settings.
Of these various types of testing, this book focuses mainly on implementation of functional test automation with Visual Studio-related tools and frameworks, including Selenium. Some non-functional test areas are also explored in the book, especially focusing on Visual Studio and Azure DevOps Services to support load and performance testing.
The Importance of Testing Software Systems
Meeting the functional requirement specifications: Functional testing helps to ensure the system is developed as per the requirement specifications and it is helping to improve the client business process. Acceptance testing, especially where the client/end user is involved with the testing while using production equivalent environments, ensure that the system meets the needs of the business once it is put into production use. Additionally, testing helps to ensure all components and applications in a given system are working together to provide the required functionality.
Support platforms and other compatibilities: Testing is required to make sure the system is compatible with all platforms, components, browsers, operating systems, etc., so as to ensure all those needs are verified before reaching production environments. It is vital to find any compatibility issues well before reaching production to avoid unexpected situations once the system is put to use in production.
- Minimize critical bugs from reaching production: Extensive testing and performing regression testing on a system before sending it to production minimizes the chance of a critical bug reaching production. Bugs in different software stages have differing costs, and the highest cost occurs when a bug discovered in production (see Figure 1-1). A critical bug in production may even result in closure of project engagement with a client and could potentially lead to legal action against the software system vendors. So, performing functional and non-functional testing is required to ensure no issues are in the system when it is put into use in production.
Figure 1-1Cost of a bug
Ensuring a system is capable of handling production data volumes and traffic: A system performing non-functional testing for performance, load, stress, etc., as identified in the previous section, helps the system development teams fix the potential issues before shipping the products. This validation of system capability to cope with production data loads in a stable manner is vital for any software system to be effectively used for its intended purposes. Additionally, recoverability testing allows the system to be implemented in a way to make it work without sudden failures and to gracefully recover from unexpected situations.
Preventing disruptions for the system by hackers or malicious software: Testing is required to find any security holes or vulnerabilities in the system to prevent any hackers or malicious software from exploiting them. This is critical as, for example, a banking system hack can cause financial crisis in a country. Further, imagine if a hack to a defense system or military missile control system could cause chaos to entire world.
Ensure system is usable: Testing helps to determine if the system is user-friendly and the experience of the users of the system is pleasant and smooth. Helping the users to improve their way of work to increase productivity is the purpose of introducing software systems, and having usability is vital for any system to be effective.
Automating Testing
- Return on Investment (ROI): Even though initially the investment is higher for automation testing when compared to manual testing, in the long run automation test costs will be less than manual testing costs (see Figure 1-2). This allows more benefits against the investments made for automation as opposed to the manual testing. Manual testing will always be available, but the lesser need for manual tests will result in higher ROI with regards to testing of software.
Figure 1-2Cost of automated vs manual testing over time
Capability to run on any schedule: Automated tests can run day and night and provide results, as when they run against the systems under testing. There is no need to have human resource interaction, and you can enable them to run on their own 24/7 or on any given schedule of your preference. This capability is handy when the agility is adopted and deployment automations are in use. Once the system is deployed to a given environment, automation testing can kick in and do all the preliminary tests, such as sanity, smoke, integration, or regression tests, to ensure all previous functionalities are intact.
Need for fewer human resources and interactions: Once test automations can take over automatically testing the system for previous functionalities, the manual testing is only required for exploring and testing new functionalities. This lessens the need for human resources, which would help with cost savings.
Detecting bugs quickly: The capability to run automation tests more often and frequently allows identification of any defects quickly in the development process, which allows them to be fixed with less cost impact.
Reliability: Automations would not skip tests or test steps based on assumptions or fatigue from running the same routines over and over again, which can easily happen with human resources. This makes automated tests more reliable, as they are always performing the same steps in testing and will be using the same set of different data values.
Reusable: Automated test codes can be reusable to test similar software systems with minor changes, depending on the specifics of a particular software. This saves time and money required in the long run for any software development organization.
Continuation: Developers of test automations can utilize existing automations in development of new ones and identify what parts of functionality are already covered with automations by utilizing the clear reporting available. This, in turn, helps them to identify new or not covered areas of functionality for further automations.
Simultaneous: Test automations can run on multiple machines and distribute and run parallel tests to achieve faster execution times compared to utilizing human resources. This would enable teams discovering any issues to fix the issues quicker and provide cost savings.
High volume: Automated tests can run on many instances—for example, a mobile application being tested can run on thousands of devices parallel and faster when automation is used, which would be an impossible task to achieve with manual testing.
Hitting the limits of systems: It is impossible to stress a system or to verify the system behavior while under heavy loads and traffic with manual testing. Automations can achieve these load, performance, and volume testing requirements in an effective manner.
Considering all of these benefits, it is obvious that the implementation of software automation would be vital for any software development organization to sustain in business. The ability to meet the demand of delivering high-quality software, with state-of-the-art technological capabilities, on time to market, cannot be made a reality without using test automations.
The Impact of Future Industry Trends on Software Testing
Adoption of Agile methodology and DevOps : The recent trend of adoption of DevOps and agility in software development practices is in for a long run in the future. This causes high demand for test automation for different aspects of testing. It would be mandatory for software testers to obtain additional skills and work closely with developers to meet the demand for automation as much as possible. Software testers will have to become “Test Development Engineers” and will have to acquire skills to code and write scripts. With DevOps siloed, thinking in the software development process will disappear, and job role lines will be blurred. Testers may have to come out of testing siloes and explore other aspects of software systems, such as operational activities like deployment testing.
Cloud computing : Software systems now run everywhere, distributed in various geo regions thanks to Cloud-based computing. These systems deliver content and data depending on the end-user’s location. Testing of such distributed systems requires moving out of traditional approaches to testing and adopting methodologies to test such distributions. Adopting automation would help to produce testing solutions to cater to these distributed needs, as parallel run capability and ability to run with less human interaction would be helpful to implement testing of all geo region testing in shorter time-frames.
Rise of Artificial Intelligence (AI) and Machine Learning (ML) : The Internet of Things (IOT) is becoming part of day-to-day life in society, with wider adoption of such solutions in a smart world. Having smart phones, smart cars, and even smart houses will become a trend in the future, making testing of such scenarios critical. Automation will be vital to meeting these trends, as testing the enormous number of devices utilized in these technological areas is impossible manually.
Engineering for performance: Modern software has to be intelligent, should effectively analyze the end-user behaviors, and should be able to provide customized experience for the user. This requires high performance in application systems, which have to undergo a lot of stress and performance testing, requiring automation of such testing.
It is obvious that automation of testing would play a significant role in the successful implementation of any software system. Investment in automation of testing would be rewarding for any software company to be competitive.
Summary
In this chapter, you have explored the different types of tests, the importance of software testing, and the need for automating the testing in detail. The future industry trends and the role of software testing was also described to give more insight and help you prepare for future demands.
In the next chapter, you will learn how to set up and get ready to implement test automation with Visual Studio and related frameworks.