5.2. Functional Testing
Functional tests have many different responsibilities. Part of their role is to ensure that the system being developed meets the requirements set out by the customer. In some circumstances they ensure that goals and metrics — such as performance metrics — are met. Functional tests can also allow customers to monitor the system while it's being developed. At a low level, functional tests will be created by testers to test functionality, such as the UI and performance testing. At a higher level, acceptance tests may be created by the customer to ensure that the software meets the external requirements. Functional tests should pick up where unit tests left off and start testing the system with concrete implementations and in a more end-to-end process with the aim of catching integration problems and missing requirements. When acceptance tests have been well written, they can act as a specification because they are readable by a customer. This concept will be covered in more depth in Chapter 6.
Functional tests can be invaluable during a major refactoring of a system. When large refactorings occur, a large number of unit tests can break and it can be hard to determine which tests were expected to fail. Functional testing will help indicate to you when the application is restored to a known state and indicate if manual testing should begin.
Unlike unit tests, functional tests can take a long time to run. Often a well-defined suite of functional tests will run overnight and, in some cases, will run for days. Because functional tests take a great deal of time to run, it's imperative that some type of continuous integration system is put into place to schedule when the functional tests are run. Without such a system in place, functional tests are not run often enough and degrade over time. Although it is common practice to run unit tests upon every commit to the source control repository, functional tests are more likely to be run every night.
Many types of tests fall under the functional testing umbrella. It's understandable if you find the term functional testing confusing, because many developers/testers/customers substitute the precise name of the testing discipline with the generic term "functional test." Let's take a moment and define each type of functional test:
Accessibility testing. In this context, accessible can be thought of as the ability to access. Accessibility testing ensures that the functionality of the application is available to people with disabilities. Many organizations have published standards that applications should meet to maximize accessibility.
Ad hoc testing. Ad hoc testing is a concept that is applied to tests that are created with the intention to only be run one time. Ad hoc tests are considered exploratory tests. It is often argued that ad hoc tests should be more structured and be contained in the test harness.
Acceptance testing. Acceptance testing tests that the system does exactly what it is supposed to do. They can be used as a "sign-off" to indicate that the vendor has delivered what the customer has asked for. Acceptance tests are created from the customer's requirements and result directly in the acceptance or rejection of the final system
Capacity testing. This type of test is used to determine the maximum number of resources that can be contained inside the application under test.
Deployment testing. This type of test ensures that the package created to deploy the application works as expected. Often deployment testing occurs on virtual machines to ensure that the application will install on multiple operating systems.
Exploratory testing. Exploratory testing is a manual test that can be thought of as a style of testing that allows a developer/tester the freedom to explore a black box system by creating tests to find out how the system works.
Graphical user interface software testing. UI testing includes tests that ensure the UI meets the specifications of how the application should perform. UI tests can be automated or manual tests.
Performance testing. Performance testing measures the response of a system when it's placed under load. Common metrics for web applications is Time to First Byte (TTFB) and Requests per Second (RPS).
Recovery testing. Recovery testing simply tests how well the application under test recovers from a software crash or hardware issue.
Regression testing. Regression tests ensure that functionality that was previously working correctly has stopped working as intended.
Sanity testing. A sanity test is a quick check to ensure the validity of a function. Sanity tests help determine whether it is reasonable to continue with more thorough tests.
Scalability testing. Testing for scalability ensures that the web application under test is able to maintain the same performance characteristics (requests per second) as traffic, users, or the amount of data on the page increases.
Security testing. Security testing ensures that data and functionality is protected on the application under test.
Smoke testing. Smoke tests are the first tests run on the application under test, to provide assurance that the application will not catastrophically fail. If the application does not catastrophically fail, more in-depth tests can be run on the system. Smoke tests are often used to help flush out errors.
Stress testing. This is another name for performance testing.
Usability testing. Usability testing focuses on how users use the system. Usability testing cannot be automated, and testers have a very close communication line with the users in the test scenario.
Volume testing. Volume testing ensures that the system being tested can handle a certain amount of data. Databases or files are examples of items that would be tested under a volume test. Volume testing is also closely related to performance testing.
Some software development processes place a barrier that is stricter than we have discussed thus far between unit and functional tests. XP processes state that unit tests should be created by developers, while functional test specifications should be created by the customer with the guidance of developers and testers.