test_ok1/doc/en/parametrize.txt

.. _`test generators`:
.. _`parametrizing-tests`:
.. _`parametrized test functions`:
.. _`parametrize`:

.. _`parametrize-basics`:

Parametrizing fixtures and test functions
==========================================================================

pytest supports test parametrization in several well-integrated ways:

- :py:func:`pytest.fixture` allows to define :ref:`parametrization
  at the level of fixture functions <fixture-parametrize>`.

* `@pytest.mark.parametrize`_ allows to define parametrization at the
  function or class level, provides multiple argument/fixture sets 
  for a particular test function or class.

* `pytest_generate_tests`_ enables implementing your own custom 
  dynamic parametrization scheme or extensions.

.. _parametrizemark:
.. _`@pytest.mark.parametrize`:


``@pytest.mark.parametrize``: parametrizing test functions
---------------------------------------------------------------------

.. regendoc: wipe

.. versionadded:: 2.2, improved in 2.4

The builtin ``pytest.mark.parametrize`` decorator enables
parametrization of arguments for a test function.  Here is a typical example
of a test function that implements checking that a certain input leads
to an expected output::

    # content of test_expectation.py
    import pytest
    @pytest.mark.parametrize("input,expected", [
        ("3+5", 8),
        ("2+4", 6),
        ("6*9", 42),
    ])
    def test_eval(input, expected):
        assert eval(input) == expected

Here, the ``@parametrize`` decorator defines three different ``(input,output)``
tuples so that that the ``test_eval`` function will run three times using
them in turn::

    $ py.test 
    ============================= test session starts ==============================
    platform linux2 -- Python 2.7.3 -- pytest-2.4.0.dev3
    plugins: xdist, cache, cli, pep8, xprocess, cov, capturelog, bdd-splinter, rerunfailures, instafail, localserver
    collected 3 items
    
    test_expectation.py ..F
    
    =================================== FAILURES ===================================
    ______________________________ test_eval[6*9-42] _______________________________
    
    input = '6*9', expected = 42
    
        @pytest.mark.parametrize("input,expected", [
            ("3+5", 8),
            ("2+4", 6),
            ("6*9", 42),
        ])
        def test_eval(input, expected):
    >       assert eval(input) == expected
    E       assert 54 == 42
    E        +  where 54 = eval('6*9')
    
    test_expectation.py:8: AssertionError
    ====================== 1 failed, 2 passed in 0.02 seconds ======================

As designed in this example, only one pair of input/output values fails
the simple test function.  And as usual with test function arguments,
you can see the ``input`` and ``output`` values in the traceback.

Note that you could also use the parametrize marker on a class or a module
(see :ref:`mark`) which would invoke several functions with the argument sets.

It is also possible to mark individual test instances within parametrize,
for example with the builtin ``mark.xfail``::

    # content of test_expectation.py
    import pytest
    @pytest.mark.parametrize("input,expected", [
        ("3+5", 8),
        ("2+4", 6),
        pytest.mark.xfail(("6*9", 42)),
    ])
    def test_eval(input, expected):
        assert eval(input) == expected

Let's run this::

    $ py.test
    ============================= test session starts ==============================
    platform linux2 -- Python 2.7.3 -- pytest-2.4.0.dev3
    plugins: xdist, cache, cli, pep8, xprocess, cov, capturelog, bdd-splinter, rerunfailures, instafail, localserver
    collected 3 items
    
    test_expectation.py ..x
    
    ===================== 2 passed, 1 xfailed in 0.02 seconds ======================

The one parameter set which caused a failure previously now
shows up as an "xfailed (expected to fail)" test.

.. note::

    In versions prior to 2.4 one needed to specify the argument
    names as a tuple.  This remains valid but the simpler ``"name1,name2,..."``
    comma-separated-string syntax is now advertised fist because 
    it's easier to write, produces less line noise.

.. _`pytest_generate_tests`:

Basic ``pytest_generate_tests`` example
---------------------------------------------

Sometimes you may want to implement your own parametrization scheme
or implement some dynamism for determining the parameters or scope
of a fixture.   For this, you can use the ``pytest_generate_tests`` hook 
which is called when collecting a test function.  Through the passed in
`metafunc` object you can inspect the requesting test context and, most
importantly, you can call ``metafunc.parametrize()`` to cause
parametrization.  

For example, let's say we want to run a test taking string inputs which
we want to set via a new py.test command line option.  Let's first write 
a simple test accepting a ``stringinput`` fixture function argument::

    # content of test_strings.py

    def test_valid_string(stringinput):
        assert stringinput.isalpha()

Now we add a ``conftest.py`` file containing the addition of a 
command line option and the parametrization of our test function::

    # content of conftest.py

    def pytest_addoption(parser):
        parser.addoption("--stringinput", action="append", default=[],
            help="list of stringinputs to pass to test functions")

    def pytest_generate_tests(metafunc):
        if 'stringinput' in metafunc.fixturenames:
            metafunc.parametrize("stringinput", 
                                 metafunc.config.option.stringinput)

If we now pass two stringinput values, our test will run twice::

    $ py.test -q --stringinput="hello" --stringinput="world" test_strings.py
    ..

Let's also run with a stringinput that will lead to a failing test::

    $ py.test -q --stringinput="!" test_strings.py
    F
    =================================== FAILURES ===================================
    _____________________________ test_valid_string[!] _____________________________
    
    stringinput = '!'
    
        def test_valid_string(stringinput):
    >       assert stringinput.isalpha()
    E       assert <built-in method isalpha of str object at 0x7fd657390fd0>()
    E        +  where <built-in method isalpha of str object at 0x7fd657390fd0> = '!'.isalpha
    
    test_strings.py:3: AssertionError

As expected our test function fails. 

If you don't specify a stringinput it will be skipped because
``metafunc.parametrize()`` will be called with an empty parameter
listlist::

    $ py.test -q -rs test_strings.py 
    s
    =========================== short test summary info ============================
    SKIP [1] /home/hpk/p/pytest/_pytest/python.py:999: got empty parameter set, function test_valid_string at /tmp/doc-exec-2/test_strings.py:1

For further examples, you might want to look at :ref:`more
parametrization examples <paramexamples>`.

.. _`metafunc object`:

The **metafunc** object
-------------------------------------------

.. currentmodule:: _pytest.python

metafunc objects are passed to the ``pytest_generate_tests`` hook.
They help to inspect a testfunction and to generate tests
according to test configuration or values specified
in the class or module where a test function is defined:

``metafunc.fixturenames``: set of required function arguments for given function

``metafunc.function``: underlying python test function

``metafunc.cls``: class object where the test function is defined in or None.

``metafunc.module``: the module object where the test function is defined in.

``metafunc.config``: access to command line opts and general config

``metafunc.funcargnames``: alias for ``fixturenames``, for pre-2.3 compatibility

.. automethod:: Metafunc.parametrize
.. automethod:: Metafunc.addcall(funcargs=None,id=_notexists,param=_notexists)