Parametrized Testing
Sending some values through a function and checking the output to make sure it’s correct is a common pattern in software testing. However, calling a function once with one set of values and one check for correctness isn’t enough to fully test most functions. Parametrized testing is a way to send multiple sets of data through the same test and have pytest report if any of the sets failed.
To help understand the problem parametrized testing is trying to solve, let’s take a simple test for add():
| | import pytest |
| | import tasks |
| | from tasks import Task |
| | |
| | |
| | def test_add_1(): |
| | """tasks.get() using id returned from add() works.""" |
| | task = Task('breathe', 'BRIAN', True) |
| | task_id = tasks.add(task) |
| | t_from_db = tasks.get(task_id) |
| | # everything but the id should be the same |
| | assert equivalent(t_from_db, task) |
| | |
| | |
| | def equivalent(t1, t2): |
| | """Check two tasks for equivalence.""" |
| | # Compare everything but the id field |
| | return ((t1.summary == t2.summary) and |
| | (t1.owner == t2.owner) and |
| | (t1.done == t2.done)) |
| | |
| | |
| | @pytest.fixture(autouse=True) |
| | def initialized_tasks_db(tmpdir): |
| | """Connect to db before testing, disconnect after.""" |
| | tasks.start_tasks_db(str(tmpdir), 'tiny') |
| | yield |
| | tasks.stop_tasks_db() |
When a Task object is created, its id field is set to None. After it’s added and retrieved from the database, the id field will be set. Therefore, we can’t just use == to check to see if our task was added and retrieved correctly. The equivalent() helper function checks all but the id field. The autouse fixture is included to make sure the database is accessible. Let’s make sure the test passes:
| | $ cd /path/to/code/ch2/tasks_proj/tests/func |
| | $ pytest -v test_add_variety.py::test_add_1 |
| | ===================== test session starts ====================== |
| | collected 1 item |
| | |
| | test_add_variety.py::test_add_1 PASSED |
| | |
| | =================== 1 passed in 0.03 seconds =================== |
The test seems reasonable. However, it’s just testing one example task. What if we want to test lots of variations of a task? No problem. We can use @pytest.mark.parametrize(argnames, argvalues) to pass lots of data through the same test, like this:
| | @pytest.mark.parametrize('task', |
| | [Task('sleep', done=True), |
| | Task('wake', 'brian'), |
| | Task('breathe', 'BRIAN', True), |
| | Task('exercise', 'BrIaN', False)]) |
| | def test_add_2(task): |
| | """Demonstrate parametrize with one parameter.""" |
| | task_id = tasks.add(task) |
| | t_from_db = tasks.get(task_id) |
| | assert equivalent(t_from_db, task) |
The first argument to parametrize() is a string with a comma-separated list of names—’task’, in our case. The second argument is a list of values, which in our case is a list of Task objects. pytest will run this test once for each task and report each as a separate test:
| | $ cd /path/to/code/ch2/tasks_proj/tests/func |
| | $ pytest -v test_add_variety.py::test_add_2 |
| | ===================== test session starts ====================== |
| | collected 4 items |
| | |
| | test_add_variety.py::test_add_2[task0] PASSED |
| | test_add_variety.py::test_add_2[task1] PASSED |
| | test_add_variety.py::test_add_2[task2] PASSED |
| | test_add_variety.py::test_add_2[task3] PASSED |
| | |
| | =================== 4 passed in 0.05 seconds =================== |
This use of parametrize() works for our purposes. However, let’s pass in the tasks as tuples to see how multiple test parameters would work:
| | @pytest.mark.parametrize('summary, owner, done', |
| | [('sleep', None, False), |
| | ('wake', 'brian', False), |
| | ('breathe', 'BRIAN', True), |
| | ('eat eggs', 'BrIaN', False), |
| | ]) |
| | def test_add_3(summary, owner, done): |
| | """Demonstrate parametrize with multiple parameters.""" |
| | task = Task(summary, owner, done) |
| | task_id = tasks.add(task) |
| | t_from_db = tasks.get(task_id) |
| | assert equivalent(t_from_db, task) |
When you use types that are easy for pytest to convert into strings, the test identifier uses the parameter values in the report to make it readable:
| | $ cd /path/to/code/ch2/tasks_proj/tests/func |
| | $ pytest -v test_add_variety.py::test_add_3 |
| | ===================== test session starts ====================== |
| | collected 4 items |
| | |
| | test_add_variety.py::test_add_3[sleep-None-False] PASSED |
| | test_add_variety.py::test_add_3[wake-brian-False] PASSED |
| | test_add_variety.py::test_add_3[breathe-BRIAN-True] PASSED |
| | test_add_variety.py::test_add_3[eat eggs-BrIaN-False] PASSED |
| | |
| | =================== 4 passed in 0.05 seconds =================== |
You can use that whole test identifier—called a node in pytest terminology—to re-run the test if you want:
| | $ cd /path/to/code/ch2/tasks_proj/tests/func |
| | $ pytest -v test_add_variety.py::test_add_3[sleep-None-False] |
| | ===================== test session starts ====================== |
| | collected 1 item |
| | |
| | test_add_variety.py::test_add_3[sleep-None-False] PASSED |
| | |
| | =================== 1 passed in 0.02 seconds =================== |
Be sure to use quotes if there are spaces in the identifier:
| | $ cd /path/to/code/ch2/tasks_proj/tests/func |
| | $ pytest -v "test_add_variety.py::test_add_3[eat eggs-BrIaN-False]" |
| | ===================== test session starts ====================== |
| | collected 1 item |
| | |
| | test_add_variety.py::test_add_3[eat eggs-BrIaN-False] PASSED |
| | |
| | =================== 1 passed in 0.03 seconds =================== |
Now let’s go back to the list of tasks version, but move the task list to a variable outside the function:
| | tasks_to_try = (Task('sleep', done=True), |
| | Task('wake', 'brian'), |
| | Task('wake', 'brian'), |
| | Task('breathe', 'BRIAN', True), |
| | Task('exercise', 'BrIaN', False)) |
| | |
| | |
| | @pytest.mark.parametrize('task', tasks_to_try) |
| | def test_add_4(task): |
| | """Slightly different take.""" |
| | task_id = tasks.add(task) |
| | t_from_db = tasks.get(task_id) |
| | assert equivalent(t_from_db, task) |
It’s convenient and the code looks nice. But the readability of the output is hard to interpret:
| | $ cd /path/to/code/ch2/tasks_proj/tests/func |
| | $ pytest -v test_add_variety.py::test_add_4 |
| | ===================== test session starts ====================== |
| | collected 5 items |
| | |
| | test_add_variety.py::test_add_4[task0] PASSED |
| | test_add_variety.py::test_add_4[task1] PASSED |
| | test_add_variety.py::test_add_4[task2] PASSED |
| | test_add_variety.py::test_add_4[task3] PASSED |
| | test_add_variety.py::test_add_4[task4] PASSED |
| | |
| | =================== 5 passed in 0.05 seconds =================== |
The readability of the multiple parameter version is nice, but so is the list of Task objects. To compromise, we can use the ids optional parameter to parametrize() to make our own identifiers for each task data set. The ids parameter needs to be a list of strings the same length as the number of data sets. However, because we assigned our data set to a variable name, tasks_to_try, we can use it to generate ids:
| | task_ids = ['Task({},{},{})'.format(t.summary, t.owner, t.done) |
| | for t in tasks_to_try] |
| | |
| | |
| | @pytest.mark.parametrize('task', tasks_to_try, ids=task_ids) |
| | def test_add_5(task): |
| | """Demonstrate ids.""" |
| | task_id = tasks.add(task) |
| | t_from_db = tasks.get(task_id) |
| | assert equivalent(t_from_db, task) |
Let’s run that and see how it looks:
| | $ cd /path/to/code/ch2/tasks_proj/tests/func |
| | $ pytest -v test_add_variety.py::test_add_5 |
| | ===================== test session starts ====================== |
| | collected 5 items |
| | |
| | test_add_variety.py::test_add_5[Task(sleep,None,True)] PASSED |
| | test_add_variety.py::test_add_5[Task(wake,brian,False)0] PASSED |
| | test_add_variety.py::test_add_5[Task(wake,brian,False)1] PASSED |
| | test_add_variety.py::test_add_5[Task(breathe,BRIAN,True)] PASSED |
| | test_add_variety.py::test_add_5[Task(exercise,BrIaN,False)] PASSED |
| | |
| | =================== 5 passed in 0.04 seconds =================== |
And these test identifiers can be used to run tests:
| | $ cd /path/to/code/ch2/tasks_proj/tests/func |
| | $ pytest -v "test_add_variety.py::test_add_5[Task(exercise,BrIaN,False)]" |
| | ===================== test session starts ====================== |
| | collected 1 item |
| | |
| | test_add_variety.py::test_add_5[Task(exercise,BrIaN,False)] PASSED |
| | |
| | =================== 1 passed in 0.03 seconds =================== |
We definitely need quotes for these identifiers; otherwise, the brackets and parentheses will confuse the shell.
You can apply parametrize() to classes as well. When you do that, the same data sets will be sent to all test methods in the class:
| | @pytest.mark.parametrize('task', tasks_to_try, ids=task_ids) |
| | class TestAdd(): |
| | """Demonstrate parametrize and test classes.""" |
| | |
| | def test_equivalent(self, task): |
| | """Similar test, just within a class.""" |
| | task_id = tasks.add(task) |
| | t_from_db = tasks.get(task_id) |
| | assert equivalent(t_from_db, task) |
| | def test_valid_id(self, task): |
| | """We can use the same data or multiple tests.""" |
| | task_id = tasks.add(task) |
| | t_from_db = tasks.get(task_id) |
| | assert t_from_db.id == task_id |
Here it is in action:
| | $ cd /path/to/code/ch2/tasks_proj/tests/func |
| | $ pytest -v test_add_variety.py::TestAdd |
| | ===================== test session starts ====================== |
| | collected 10 items |
| | |
| | test_add_variety.py::TestAdd::test_equivalent[Task(sleep,None,True)] PASSED |
| | test_add_variety.py::TestAdd::test_equivalent[Task(wake,brian,False)0] PASSED |
| | test_add_variety.py::TestAdd::test_equivalent[Task(wake,brian,False)1] PASSED |
| | test_add_variety.py::TestAdd::test_equivalent[Task(breathe,BRIAN,True)] PASSED |
| | test_add_variety.py::TestAdd::test_equivalent[Task(exercise,BrIaN,False)] PASSED |
| | test_add_variety.py::TestAdd::test_valid_id[Task(sleep,None,True)] PASSED |
| | test_add_variety.py::TestAdd::test_valid_id[Task(wake,brian,False)0] PASSED |
| | test_add_variety.py::TestAdd::test_valid_id[Task(wake,brian,False)1] PASSED |
| | test_add_variety.py::TestAdd::test_valid_id[Task(breathe,BRIAN,True)] PASSED |
| | test_add_variety.py::TestAdd::test_valid_id[Task(exercise,BrIaN,False)] PASSED |
| | |
| | ================== 10 passed in 0.08 seconds =================== |
You can also identify parameters by including an id right alongside the parameter value when passing in a list within the @pytest.mark.parametrize() decorator. You do this with pytest.param(<value>, id="something") syntax:
| | @pytest.mark.parametrize('task', [ |
| | pytest.param(Task('create'), id='just summary'), |
| | pytest.param(Task('inspire', 'Michelle'), id='summary/owner'), |
| | pytest.param(Task('encourage', 'Michelle', True), id='summary/owner/done')]) |
| | def test_add_6(task): |
| | """Demonstrate pytest.param and id.""" |
| | task_id = tasks.add(task) |
| | t_from_db = tasks.get(task_id) |
| | assert equivalent(t_from_db, task) |
In action:
| | $ cd /path/to/code/ch2/tasks_proj/tests/func |
| | $ pytest -v test_add_variety.py::test_add_6 |
| | =================== test session starts ==================== |
| | collected 3 items |
| | |
| | test_add_variety.py::test_add_6[just summary] PASSED |
| | test_add_variety.py::test_add_6[summary/owner] PASSED |
| | test_add_variety.py::test_add_6[summary/owner/done] PASSED |
| | |
| | ================= 3 passed in 0.05 seconds ================= |
This is useful when the id cannot be derived from the parameter value.