tox: Testing Multiple Configurations
tox is a command-line tool that allows you to run your complete suite of tests in multiple environments. We’re going to use it to test the Tasks project in multiple versions of Python. However, tox is not limited to just Python versions. You can use it to test with different dependency configurations and different configurations for different operating systems.
In gross generalities, here’s a mental model for how tox works:
tox uses the setup.py file for the package under test to create an installable source distribution of your package. It looks in tox.ini for a list of environments and then for each environment…
- tox creates a virtual environment in a .tox directory.
- tox pip installs some dependencies.
- tox pip installs your package from the sdist in step 1.
- tox runs your tests.
After all of the environments are tested, tox reports a summary of how they all did.
This makes a lot more sense when you see it in action, so let’s look at how to modify the Tasks project to use tox to test Python 2.7 and 3.6. I chose versions 2.7 and 3.6 because they are both already installed on my system. If you have different versions installed, go ahead and change the envlist line to match whichever version you have or are willing to install.
The first thing we need to do to the Tasks project is add a tox.ini file at the same level as setup.py—the top project directory. I’m also going to move anything that’s in pytest.ini into tox.ini.
Here’s the abbreviated code layout:
| | tasks_proj_v2/ |
| | ├── ... |
| | ├── setup.py |
| | ├── tox.ini |
| | ├── src |
| | │ └── tasks |
| | │ ├── __init__.py |
| | │ ├── api.py |
| | │ └── ... |
| | └── tests |
| | ├── conftest.py |
| | ├── func |
| | │ ├── __init__.py |
| | │ ├── test_add.py |
| | │ └── ... |
| | └── unit |
| | ├── __init__.py |
| | ├── test_task.py |
| | └── ... |
Now, here’s what the tox.ini file looks like:
| | # tox.ini , put in same dir as setup.py |
| | |
| | [tox] |
| | envlist = py27,py36 |
| | |
| | [testenv] |
| | deps=pytest |
| | commands=pytest |
| | |
| | [pytest] |
| | addopts = -rsxX -l --tb=short --strict |
| | markers = |
| | smoke: Run the smoke test test functions |
| | get: Run the test functions that test tasks.get() |
Under [tox], we have envlist = py27,py36. This is a shorthand to tell tox to run our tests using both python2.7 and python3.6.
Under [testenv], the deps=pytest line tells tox to make sure pytest is installed. If you have multiple test dependencies, you can put them on separate lines. You can also specify which version to use.
The commands=pytest line tells tox to run pytest in each environment.
Under [pytest], we can put whatever we normally would want to put into pytest.ini to configure pytest, as discussed in Chapter 6, Configuration. In this case, addopts is used to turn on extra summary information for skips, xfails, and xpasses (-rsxX) and turn on showing local variables in stack traces (-l). It also defaults to shortened stack traces (--tb=short) and makes sure all markers used in tests are declared first (--strict). The markers section is where the markers are declared.
Before running tox, you have to make sure you install it:
| | $ pip install tox |
This can be done within a virtual environment.
Then to run tox, just run, well, tox:
| | $ cd /path/to/code/ch7/tasks_proj_v2 |
| | $ tox |
| | GLOB sdist-make: /path/to/code/ch7/tasks_proj_v2/setup.py |
| | py27 create: /path/to/code/ch7/tasks_proj_v2/.tox/py27 |
| | py27 installdeps: pytest |
| | py27 inst: /path/to/code/ch7/tasks_proj_v2/.tox/dist/tasks-0.1.1.zip |
| | py27 installed: click==6.7,funcsigs==1.0.2,mock==2.0.0, |
| | pbr==3.1.1,py==1.4.34,pytest==3.2.1, |
| | pytest-mock==1.6.2,six==1.10.0,tasks==0.1.1,tinydb==3.4.0 |
| | py27 runtests: PYTHONHASHSEED='1311894089' |
| | py27 runtests: commands[0] | pytest |
| | ================= test session starts ================== |
| | plugins: mock-1.6.2 |
| | collected 62 items |
| | |
| | tests/func/test_add.py ... |
| | tests/func/test_add_variety.py ............................ |
| | tests/func/test_add_variety2.py ............ |
| | tests/func/test_api_exceptions.py ......... |
| | tests/func/test_unique_id.py . |
| | tests/unit/test_cli.py ..... |
| | tests/unit/test_task.py .... |
| | |
| | ============== 62 passed in 0.25 seconds =============== |
| | py36 create: /path/to/code/ch7/tasks_proj_v2/.tox/py36 |
| | py36 installdeps: pytest |
| | py36 inst: /path/to/code/ch7/tasks_proj_v2/.tox/dist/tasks-0.1.1.zip |
| | py36 installed: click==6.7,py==1.4.34,pytest==3.2.1, |
| | pytest-mock==1.6.2,six==1.10.0,tasks==0.1.1,tinydb==3.4.0 |
| | py36 runtests: PYTHONHASHSEED='1311894089' |
| | py36 runtests: commands[0] | pytest |
| | ================= test session starts ================== |
| | plugins: mock-1.6.2 |
| | collected 62 items |
| | |
| | tests/func/test_add.py ... |
| | tests/func/test_add_variety.py ............................ |
| | tests/func/test_add_variety2.py ............ |
| | tests/func/test_api_exceptions.py ......... |
| | tests/func/test_unique_id.py . |
| | |
| | tests/unit/test_cli.py ..... |
| | tests/unit/test_task.py .... |
| | |
| | ============== 62 passed in 0.27 seconds =============== |
| | _______________________ summary ________________________ |
| | py27: commands succeeded |
| | py36: commands succeeded |
| | congratulations :) |
At the end, we have a nice summary of all the test environments and their outcomes:
| | _________________________ summary _________________________ |
| | py27: commands succeeded |
| | py36: commands succeeded |
| | congratulations :) |
Doesn’t that give you a nice, warm, happy feeling? We got a “congratulations” and a smiley face.
tox is much more powerful than what I’m showing here and deserves your attention if you are using pytest to test packages intended to be run in multiple environments. For more detailed information, check out the tox documentation.[27]