Plugins That Change the Normal Test Run Flow
The following plugins in some way change how pytest runs your tests.
pytest-repeat: Run Tests More Than Once
To run tests more than once per session, use the pytest-repeat plugin.[35] This plugin is useful if you have an intermittent failure in a test.
Following is a normal test run of tests that start with test_list from ch7/tasks _proj_v2:
| | $ cd /path/to/code/ch7/tasks_proj_v2 |
| | $ pip install . |
| | $ pip install pytest-repeat |
| | $ pytest -v -k test_list |
| | ===================== test session starts ====================== |
| | plugins: repeat-0.4.1, mock-1.6.2 |
| | collected 62 items |
| | |
| | tests/func/test_api_exceptions.py::test_list_raises PASSED |
| | tests/unit/test_cli.py::test_list_no_args PASSED |
| | tests/unit/test_cli.py::test_list_print_empty PASSED |
| | tests/unit/test_cli.py::test_list_print_many_items PASSED |
| | tests/unit/test_cli.py::test_list_dash_o PASSED |
| | tests/unit/test_cli.py::test_list_dash_dash_owner PASSED |
| | |
| | ===================== 56 tests deselected ====================== |
| | =========== 6 passed, 56 deselected in 0.10 seconds ============ |
With the pytest-repeat plugin, you can use --count to run everything twice:
| | $ pytest --count=2 -v -k test_list |
| | ===================== test session starts ====================== |
| | plugins: repeat-0.4.1, mock-1.6.2 |
| | collected 124 items |
| | |
| | tests/func/test_api_exceptions.py::test_list_raises[1/2] PASSED |
| | tests/func/test_api_exceptions.py::test_list_raises[2/2] PASSED |
| | tests/unit/test_cli.py::test_list_no_args[1/2] PASSED |
| | tests/unit/test_cli.py::test_list_no_args[2/2] PASSED |
| | tests/unit/test_cli.py::test_list_print_empty[1/2] PASSED |
| | tests/unit/test_cli.py::test_list_print_empty[2/2] PASSED |
| | tests/unit/test_cli.py::test_list_print_many_items[1/2] PASSED |
| | tests/unit/test_cli.py::test_list_print_many_items[2/2] PASSED |
| | tests/unit/test_cli.py::test_list_dash_o[1/2] PASSED |
| | tests/unit/test_cli.py::test_list_dash_o[2/2] PASSED |
| | tests/unit/test_cli.py::test_list_dash_dash_owner[1/2] PASSED |
| | tests/unit/test_cli.py::test_list_dash_dash_owner[2/2] PASSED |
| | |
| | ===================== 112 tests deselected ===================== |
| | ========== 12 passed, 112 deselected in 0.16 seconds =========== |
You can repeat a subset of the tests or just one, and even choose to run it 1,000 times overnight if you want to see if you can catch the failure. You can also set it to stop on the first failure.
pytest-xdist: Run Tests in Parallel
Usually all tests run sequentially. And that’s just what you want if your tests hit a resource that can only be accessed by one client at a time. However, if your tests do not need access to a shared resource, you could speed up test sessions by running multiple tests in parallel. The pytest-xdist plugin allows you to do that. You can specify multiple processors and run many tests in parallel. You can even push off tests onto other machines and use more than one computer.
Here’s a test that takes at least a second to run, with parametrization such that it runs ten times:
| | import pytest |
| | import time |
| | |
| | |
| | @pytest.mark.parametrize('x', list(range(10))) |
| | def test_something(x): |
| | time.sleep(1) |
Notice that it takes over ten seconds to run normally:
| | $ pip install pytest-xdist |
| | $ cd /path/to/code/appendices/xdist |
| | $ pytest test_parallel.py |
| | ===================== test session starts ====================== |
| | plugins: xdist-1.20.0, forked-0.2 |
| | collected 10 items |
| | |
| | test_parallel.py .......... |
| | |
| | ================== 10 passed in 10.07 seconds ================= |
With the pytest-xdist plugin, you can use -n numprocesses to run each test in a subprocess, and use -n auto to automatically detect the number of CPUs on the system. Here’s the same test run on multiple processors:
| | $ pytest -n auto test_parallel.py |
| | ===================== test session starts ====================== |
| | plugins: xdist-1.20.0, forked-0.2 |
| | gw0 [10] / gw1 [10] / gw2 [10] / gw3 [10] |
| | scheduling tests via LoadScheduling |
| | .......... |
| | ================== 10 passed in 4.27 seconds =================== |
It’s not a silver bullet to speed up your test times by a factor of the number of processors you have—there is overhead time. However, many testing scenarios enable you to run tests in parallel. And when the tests are long, you may as well let them run in parallel to speed up your test time.
The pytest-xdist plugin does a lot more than we’ve covered here, including the ability to offload tests to different computers altogether, so be sure to read more about the pytest-xdist plugin[36] on PyPI.
pytest-timeout: Put Time Limits on Your Tests
There are no normal timeout periods for tests in pytest. However, if you’re working with resources that may occasionally disappear, such as web services, it’s a good idea to put some time restrictions on your tests.
The pytest-timeout plugin[37] does just that. It allows you pass a timeout period on the command line or mark individual tests with timeout periods in seconds. The mark overrides the command-line timeout so that the test can be either longer or shorter than the timeout limit.
Let’s run the tests from the previous example (with one-second sleeps) with a half-second timeout:
| | $ cd /path/to/code/appendices/xdist |
| | $ pip install pytest-timeout |
| | $ pytest --timeout=0.5 -x test_parallel.py |
| | ===================== test session starts ====================== |
| | plugins: xdist-1.20.0, timeout-1.2.0, forked-0.2 |
| | timeout: 0.5s method: signal |
| | collected 10 items |
| | |
| | test_parallel.py F |
| | |
| | =========================== FAILURES =========================== |
| | ______________________ test_something[0] _______________________ |
| | |
| | x = 0 |
| | |
| | @pytest.mark.parametrize('x', list(range(10))) |
| | def test_something(x): |
| | > time.sleep(1) |
| | E Failed: Timeout >0.5s |
| | |
| | test_parallel.py:6: Failed |
| | !!!!!!!!!!!! Interrupted: stopping after 1 failures !!!!!!!!!!!! |
| | =================== 1 failed in 0.68 seconds =================== |
The -x stops testing after the first failure.