.. _`test generators`: .. _`parametrizing-tests`: .. _`parametrized test functions`: .. _`parametrize`: .. _`parametrize-basics`: Parametrizing fixtures and test functions ========================================================================== pytest enables test parametrization at several levels: - :py:func:`pytest.fixture` allows one to :ref:`parametrize fixture functions `. * `@pytest.mark.parametrize`_ allows one to define multiple sets of arguments and fixtures at the test function or class. * `pytest_generate_tests`_ allows one to define custom parametrization schemes or extensions. .. _parametrizemark: .. _`@pytest.mark.parametrize`: ``@pytest.mark.parametrize``: parametrizing test functions --------------------------------------------------------------------- .. regendoc: wipe .. versionadded:: 2.2 .. versionchanged:: 2.4 Several improvements. The builtin :ref:`pytest.mark.parametrize ref` decorator enables parametrization of arguments for a test function. Here is a typical example of a test function that implements checking that a certain input leads to an expected output:: # content of test_expectation.py import pytest @pytest.mark.parametrize("test_input,expected", [ ("3+5", 8), ("2+4", 6), ("6*9", 42), ]) def test_eval(test_input, expected): assert eval(test_input) == expected Here, the ``@parametrize`` decorator defines three different ``(test_input,expected)`` tuples so that the ``test_eval`` function will run three times using them in turn: .. code-block:: pytest $ pytest =========================== test session starts ============================ platform linux -- Python 3.x.y, pytest-4.x.y, py-1.x.y, pluggy-0.x.y cachedir: $PYTHON_PREFIX/.pytest_cache rootdir: $REGENDOC_TMPDIR, inifile: collected 3 items test_expectation.py ..F [100%] ================================= FAILURES ================================= ____________________________ test_eval[6*9-42] _____________________________ test_input = '6*9', expected = 42 @pytest.mark.parametrize("test_input,expected", [ ("3+5", 8), ("2+4", 6), ("6*9", 42), ]) def test_eval(test_input, expected): > assert eval(test_input) == expected E AssertionError: assert 54 == 42 E + where 54 = eval('6*9') test_expectation.py:8: AssertionError ==================== 1 failed, 2 passed in 0.12 seconds ==================== As designed in this example, only one pair of input/output values fails the simple test function. And as usual with test function arguments, you can see the ``input`` and ``output`` values in the traceback. Note that you could also use the parametrize marker on a class or a module (see :ref:`mark`) which would invoke several functions with the argument sets. It is also possible to mark individual test instances within parametrize, for example with the builtin ``mark.xfail``:: # content of test_expectation.py import pytest @pytest.mark.parametrize("test_input,expected", [ ("3+5", 8), ("2+4", 6), pytest.param("6*9", 42, marks=pytest.mark.xfail), ]) def test_eval(test_input, expected): assert eval(test_input) == expected Let's run this: .. code-block:: pytest $ pytest =========================== test session starts ============================ platform linux -- Python 3.x.y, pytest-4.x.y, py-1.x.y, pluggy-0.x.y cachedir: $PYTHON_PREFIX/.pytest_cache rootdir: $REGENDOC_TMPDIR, inifile: collected 3 items test_expectation.py ..x [100%] =================== 2 passed, 1 xfailed in 0.12 seconds ==================== The one parameter set which caused a failure previously now shows up as an "xfailed (expected to fail)" test. In case the values provided to ``parametrize`` result in an empty list - for example, if they're dynamically generated by some function - the behaviour of pytest is defined by the :confval:`empty_parameter_set_mark` option. To get all combinations of multiple parametrized arguments you can stack ``parametrize`` decorators:: import pytest @pytest.mark.parametrize("x", [0, 1]) @pytest.mark.parametrize("y", [2, 3]) def test_foo(x, y): pass This will run the test with the arguments set to ``x=0/y=2``, ``x=1/y=2``, ``x=0/y=3``, and ``x=1/y=3`` exhausting parameters in the order of the decorators. .. _`pytest_generate_tests`: Basic ``pytest_generate_tests`` example --------------------------------------------- Sometimes you may want to implement your own parametrization scheme or implement some dynamism for determining the parameters or scope of a fixture. For this, you can use the ``pytest_generate_tests`` hook which is called when collecting a test function. Through the passed in ``metafunc`` object you can inspect the requesting test context and, most importantly, you can call ``metafunc.parametrize()`` to cause parametrization. For example, let's say we want to run a test taking string inputs which we want to set via a new ``pytest`` command line option. Let's first write a simple test accepting a ``stringinput`` fixture function argument:: # content of test_strings.py def test_valid_string(stringinput): assert stringinput.isalpha() Now we add a ``conftest.py`` file containing the addition of a command line option and the parametrization of our test function:: # content of conftest.py def pytest_addoption(parser): parser.addoption("--stringinput", action="append", default=[], help="list of stringinputs to pass to test functions") def pytest_generate_tests(metafunc): if 'stringinput' in metafunc.fixturenames: metafunc.parametrize("stringinput", metafunc.config.getoption('stringinput')) If we now pass two stringinput values, our test will run twice: .. code-block:: pytest $ pytest -q --stringinput="hello" --stringinput="world" test_strings.py .. [100%] 2 passed in 0.12 seconds Let's also run with a stringinput that will lead to a failing test: .. code-block:: pytest $ pytest -q --stringinput="!" test_strings.py F [100%] ================================= FAILURES ================================= ___________________________ test_valid_string[!] ___________________________ stringinput = '!' def test_valid_string(stringinput): > assert stringinput.isalpha() E AssertionError: assert False E + where False = () E + where = '!'.isalpha test_strings.py:3: AssertionError 1 failed in 0.12 seconds As expected our test function fails. If you don't specify a stringinput it will be skipped because ``metafunc.parametrize()`` will be called with an empty parameter list: .. code-block:: pytest $ pytest -q -rs test_strings.py s [100%] ========================= short test summary info ========================== SKIPPED [1] test_strings.py: got empty parameter set ['stringinput'], function test_valid_string at $REGENDOC_TMPDIR/test_strings.py:1 1 skipped in 0.12 seconds Note that when calling ``metafunc.parametrize`` multiple times with different parameter sets, all parameter names across those sets cannot be duplicated, otherwise an error will be raised. More examples ------------- For further examples, you might want to look at :ref:`more parametrization examples `.