The third and final type of automated functional tests is system tests. These tests are meant to run against a fully assembled and running application. System tests come in two flavors, depending on the type of application: an API test or a UI test. System tests can take a long time to execute, and it is not uncommon for long tests with an elaborate setup of test data to take well over a minute.
System tests execute against a running application, which means that they will need configuration and setup before they can be run. The application needs to be running in a controlled environment and all of the integrations with data stores need to be fully operational. Integrations with other systems need to be either up and running or swapped out with a replacement mock to ensure that all operations that integrate with those systems will function properly.
These conditions make it less likely that developers will execute these tests on their local machines as they are making changes to the application. It is only when they are creating a new test or changing a test that they might do so. However, even then they may be executing these tests not against a locally run version of the application, but against a version that is already deployed to a test environment. This is not necessarily a good thing, but often just the reality in most teams.
When executing system tests as part of the pipeline, they are often run after the code has been deployed to at least one environment. This will often be the test environment. This implies that the system tests are on the critical path from a source code change to the deployment to production. If this path becomes too long, they can also be taken out of the pipeline. They are then run on a schedule—for example, every night. Just as with integration tests, this speeds up the pipeline, but it removes the opportunity to use system tests as a quality gate.
System tests, and UI tests in particular, are often fragile and can stop working unexpectedly after minor changes. For this reason, it is advised that you keep their number as low as possible; however, keep in mind that these are the tests that can catch particular errors, such as misconfiguration or other runtime errors, database-application mismatches, or series of operations that create error states.
Besides automated function tests, there are also manual functional tests that have value in many DevOps projects. These are discussed next.