test_ok2/doc/en/funcargs.txt

.. _resources:
.. _`funcargs`:
.. _`funcarg mechanism`:

=======================================================
funcargs: resource injection and parametrization
=======================================================

.. note::

    pytest-2.3 introduces major refinements to the test setup and funcarg
    mechanisms introduced to pytest-2.0.  All pre-2.3 usages remain
    supported and several use cases, among them scoping and parametrization
    of funcarg resources, are now easier to accomplish.  For more background, 
    see `compatibility notes`_ and the detailed :ref:`reasoning for the new 
    funcarg and setup functions <funcargcompare>`.

.. _`Dependency injection`: http://en.wikipedia.org/wiki/Dependency_injection

Introduction
====================

pytest supports the injection of test resources into test and setup functions 
and flexibly control their life cycle in relation to the overall test
execution.  Moreover, tests can get executed multiple times if you have
different variants of test resources to test with.  

The basic mechanism for injecting objects is called the *funcarg
mechanism* because objects are injected when a test or setup
**function** states it as an **argument**. The injected argument 
is created by a call to a registered **funcarg factory** for each argument
name.  This mechanism is an example of `Dependency Injection`_ 
and helps to de-couple test code from the setup of required
objects: at test writing time you do not need to care for the details of
where and how your required test resources are constructed, if they are
shared on a per-class, module or session basis, or if your test function
is invoked multiple times with differently configured resource
instances.

Funcarg dependency injection allows to organise test resources 
in a modular explicit way so that test functions state their needs
in their signature.  pytest additionally offers powerful xunit-style 
:ref:`setup functions <setup functions>` for the cases where you need 
to create implicit test state that is not passed explicitely to test functions.

When a test function is invoked multiple times with different arguments we
speak of **parametrized testing**.  You can use it e. g. to repeatedly run test
functions against different database backends or to check that certain
inputs lead to certain outputs.

Concretely, there are three main means of funcarg management:

* a `@pytest.factory`_ marker to define resource factories,
  their scoping and parametrization.   Factories can themselves 
  receive resources through their function arguments, easing
  the setup of `interdependent resources`_.  Factories can use
  the special `request`_ object to access details from where
  the factory or setup function is called and for registering finalizers.

* a `@pytest.mark.parametrize`_ marker for executing test functions
  multiple times with different argument sets,

* a `pytest_generate_tests`_ plugin hook marker for implementing
  your parametrization for a test function which may depend on 
  command line options, class/module attributes etc.

Apart from making it easy to manage your own test resources
pytest also comes with some :ref:`builtinresources` which
you can use without defining them yourself.  Third-party plugins 
offer yet more domain-specific funcarg resources (for example the
`pytest-django plugin <http://pypi.python.org/pypi/pytest-django>`_) so
that after plugin installation you can simply use them in
your test and setup functions.  This all contributes to high
re-useability of test resource management and goes far beyond what can
be done with the classical xUnit style approach which encodes resource
setup statically into the test source code, leading to duplicate and
hard-to change fixtures.

.. _`@pytest.factory`:

``@pytest.factory``: Creating parametrized, scoped resources
=====================================================================

Basic funcarg injection example
-----------------------------------------------------------

Let's look at a simple self-contained test module using a factory
and a funcarg::

    # content of ./test_simplefactory.py
    import pytest

    @pytest.factory()
    def myfuncarg():
        return 42

    def test_function(myfuncarg):
        assert myfuncarg == 17

Here, the ``test_function`` needs an object named ``myfuncarg`` and thus
py.test will discover and call the ``@pytest.factory`` marked ``myfuncarg`` 
factory function.  Running the tests looks like this::

    $ py.test test_simplefactory.py
    =========================== test session starts ============================
    platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev11
    plugins: xdist, bugzilla, cache, oejskit, cli, timeout, pep8, cov
    collecting ... collected 1 items
    
    test_simplefactory.py F
    
    ================================= FAILURES =================================
    ______________________________ test_function _______________________________
    
    myfuncarg = 42
    
        def test_function(myfuncarg):
    >       assert myfuncarg == 17
    E       assert 42 == 17
    
    test_simplefactory.py:8: AssertionError
    ========================= 1 failed in 0.01 seconds =========================

This shows that the test function was called with a ``myfuncarg``
argument value of ``42`` and the assert fails as expected.  Here is 
how py.test comes to call the test function this way:

1. py.test :ref:`finds <test discovery>` the ``test_function`` because
   of the ``test_`` prefix.  The test function needs a function argument
   named ``myfuncarg``.  A matching factory function is discovered by
   looking for a factory function named ``myfuncarg``.

2. ``myfuncarg()`` is called to create a value ``42``.

3. ``test_function(42)`` is now called and results in the above
   reported exception because of the assertion mismatch.

Note that if you misspell a function argument or want
to use one that isn't available, you'll see an error
with a list of available function arguments.

.. Note::

    You can always issue::

        py.test --funcargs test_simplefactory.py

    to see available function arguments.


Location independency of funcarg factories
----------------------------------------------------

If during implementing your tests you realize that you
want to use a factory from multiple test files you can move it
to a :ref:`conftest.py <conftest.py>` file or even separately installable
:ref:`plugins <plugins>` without changing test code.  The discovery of
funcarg factories starts at test classes, then test modules, then
``conftest.py`` files and finally builtin and 3-rd party plugins.

Creating and using a session-shared funcarg
-----------------------------------------------------------------

.. regendoc:wipe

.. versionadded:: 2.3

The `@pytest.factory`_ marker allows to

* mark a function as a factory for resources, useable by test and setup
  functions 

* define parameters in order to run tests multiple times with
  different resource instances

* declare a scope which determines the level of caching, i.e. how often
  the factory will be called. Valid scopes are ``session``, ``module``, 
  ``class`` and ``function``.

Here is a simple example of a factory creating a shared ``smtplib.SMTP``
connection resource which test functions then may use across the whole
test session::

    # content of conftest.py
    import pytest
    import smtplib

    @pytest.factory(scope="session")
    def smtp(request):
        return smtplib.SMTP("merlinux.eu")

The name of the factory is ``smtp`` (the factory function name) 
and you can access its result by listing the name ``smtp`` as
an input parameter in any test or setup function::

    # content of test_module.py
    def test_ehlo(smtp):
        response = smtp.ehlo()
        assert response[0] == 250 
        assert "merlinux" in response[1]
        assert 0  # for demo purposes

    def test_noop(smtp):
        response = smtp.noop()
        assert response[0] == 250
        assert 0  # for demo purposes

We deliberately insert failing ``assert 0`` statements in order to
inspect what is going on and can now run the tests::

    $ py.test -q test_module.py
    collecting ... collected 2 items
    FF
    ================================= FAILURES =================================
    ________________________________ test_ehlo _________________________________
    
    smtp = <smtplib.SMTP instance at 0x31bce18>
    
        def test_ehlo(smtp):
            response = smtp.ehlo()
            assert response[0] == 250
            assert "merlinux" in response[1]
    >       assert 0  # for demo purposes
    E       assert 0
    
    test_module.py:5: AssertionError
    ________________________________ test_noop _________________________________
    
    smtp = <smtplib.SMTP instance at 0x31bce18>
    
        def test_noop(smtp):
            response = smtp.noop()
            assert response[0] == 250
    >       assert 0  # for demo purposes
    E       assert 0
    
    test_module.py:10: AssertionError
    2 failed in 0.26 seconds

you see the two ``assert 0`` failing and can also see that
the same (session-scoped) object was passed into the two test functions.


Parametrizing a session-shared funcarg resource
-----------------------------------------------------------------

Extending the previous example, we can flag the factory to create
two ``smtp`` values which will cause all tests using it to
run twice with two different values.  The factory function gets
access to each parameter through the special `request`_ object::

    # content of conftest.py
    import pytest
    import smtplib

    @pytest.factory(scope="session", 
                    params=["merlinux.eu", "mail.python.org"])
    def smtp(request):
        return smtplib.SMTP(request.param)

The main change is the definition of a ``params`` list in the
``factory``-marker and the ``request.param`` access within the
factory function.  No test function code needs to change.  
So let's just do another run::

    $ py.test -q
    collecting ... collected 4 items
    FFFF
    ================================= FAILURES =================================
    __________________________ test_ehlo[merlinux.eu] __________________________
    
    smtp = <smtplib.SMTP instance at 0x28dc5a8>
    
        def test_ehlo(smtp):
            response = smtp.ehlo()
            assert response[0] == 250
            assert "merlinux" in response[1]
    >       assert 0  # for demo purposes
    E       assert 0
    
    test_module.py:5: AssertionError
    __________________________ test_noop[merlinux.eu] __________________________
    
    smtp = <smtplib.SMTP instance at 0x28dc5a8>
    
        def test_noop(smtp):
            response = smtp.noop()
            assert response[0] == 250
    >       assert 0  # for demo purposes
    E       assert 0
    
    test_module.py:10: AssertionError
    ________________________ test_ehlo[mail.python.org] ________________________
    
    smtp = <smtplib.SMTP instance at 0x28e3e18>
    
        def test_ehlo(smtp):
            response = smtp.ehlo()
            assert response[0] == 250
    >       assert "merlinux" in response[1]
    E       assert 'merlinux' in 'mail.python.org\nSIZE 10240000\nETRN\nSTARTTLS\nENHANCEDSTATUSCODES\n8BITMIME\nDSN'
    
    test_module.py:4: AssertionError
    ________________________ test_noop[mail.python.org] ________________________
    
    smtp = <smtplib.SMTP instance at 0x28e3e18>
    
        def test_noop(smtp):
            response = smtp.noop()
            assert response[0] == 250
    >       assert 0  # for demo purposes
    E       assert 0
    
    test_module.py:10: AssertionError
    4 failed in 6.17 seconds

We get four failures because we are running the two tests twice with
different ``smtp`` instantiations as defined on the factory.
Note that with the ``mail.python.org`` connection the second test
fails in ``test_ehlo`` because it expects a specific server string.
    
Adding a finalizer to the parametrized resource
--------------------------------------------------------

Further extending the ``smtp`` example, we now want to properly
close a smtp server connection after the last test using it
has been run.  We can do this by calling the ``request.addfinalizer()``
helper::

    # content of conftest.py
    import pytest
    import smtplib

    @pytest.factory(scope="session", 
                    params=["merlinux.eu", "mail.python.org"])
    def smtp(request):
        smtp = smtplib.SMTP(request.param)
        def fin():
            print ("finalizing %s" % smtp)
            smtp.close()
        request.addfinalizer(fin)
        return smtp

We also add a print call and then run py.test without the default
output capturing and disabled traceback reporting::

    $ py.test -s -q --tb=no
    collecting ... collected 4 items
    FFFF
    4 failed in 6.40 seconds
    finalizing <smtplib.SMTP instance at 0x125d3b0>
    finalizing <smtplib.SMTP instance at 0x1265b90>

We see that the two ``smtp`` instances are finalized appropriately.

Looking at test collection without running tests
------------------------------------------------------

You can also look at what tests pytest collects without running them::

    $ py.test --collectonly
    =========================== test session starts ============================
    platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev11
    plugins: xdist, bugzilla, cache, oejskit, cli, timeout, pep8, cov
    collecting ... collected 4 items
    <Module 'test_module.py'>
      <Function 'test_ehlo[merlinux.eu]'>
      <Function 'test_noop[merlinux.eu]'>
      <Function 'test_ehlo[mail.python.org]'>
      <Function 'test_noop[mail.python.org]'>
    
    =============================  in 0.01 seconds =============================

Note that pytest orders your test run by resource usage, minimizing
the number of active resources at any given time.


.. _`interdependent resources`:

Interdepdendent resources
----------------------------------------------------------

You can not only use funcargs in test functions but also in their factories
themselves.  Extending the previous example we can instantiate another
object ``app`` and stick the ``smtp`` resource into it like this::

    # content of test_appsetup.py
   
    import pytest

    @pytest.factory(scope="module")
    class app:
        def __init__(self, smtp):
            self.smtp = smtp

    def test_exists(app):
        assert app.smtp

Here we declare an ``app`` class to be a factory and have its init-method
use the previously defined ``smtp`` resource.  Let's run it::

    $ py.test -v test_appsetup.py
    =========================== test session starts ============================
    platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev11 -- /home/hpk/venv/1/bin/python
    cachedir: /home/hpk/tmp/doc-exec-423/.cache
    plugins: xdist, bugzilla, cache, oejskit, cli, timeout, pep8, cov
    collecting ... collected 2 items
    
    test_appsetup.py:12: test_exists[merlinux.eu] PASSED
    test_appsetup.py:12: test_exists[mail.python.org] PASSED
    
    ========================= 2 passed in 6.82 seconds =========================

Due to the parametrization of ``smtp`` the test will run twice with two
different ``App`` instances and respective smtp servers.  There is no
need for the ``app`` factory to be aware of the parametrization.  Note
that the ``app`` factory has a scope of ``module`` but uses the
session-scoped ``smtp`` object: it is fine for factories to use
"broader" scoped resources but not the other way round.  A
session-scoped resource could not use a module-scoped resource in a
meaningful way.

.. _`automatic per-resource grouping`:

Grouping tests by resource parameters
----------------------------------------------------------

.. regendoc: wipe

pytest minimizes the number of active resources during test runs.
If you have a parametrized resource, then all the tests using one
resource instance will execute one after another.  Then any finalizers
are called for that resource instance and then the next parametrized
resource instance is created and its tests are run.  Among other things,
this eases testing of applications which create and use global state.

The following example uses two parametrized funcargs, one of which is 
scoped on a per-module basis, and all the functions perform ``print`` call s
to show the flow of calls::

    # content of test_module.py
    import pytest

    @pytest.factory(scope="module", params=["mod1", "mod2"])
    def modarg(request):
        param = request.param
        print "create", param
        def fin():
            print "fin", param
        request.addfinalizer(fin)
        return param

    @pytest.factory(scope="function", params=[1,2])
    def otherarg(request):
        return request.param

    def test_0(otherarg):
        print "  test0", otherarg
    def test_1(modarg):
        print "  test1", modarg
    def test_2(otherarg, modarg):
        print "  test2", otherarg, modarg

Let's run the tests in verbose mode and with looking at the print-output::

    $ py.test -v -s test_module.py
    =========================== test session starts ============================
    platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev11 -- /home/hpk/venv/1/bin/python
    cachedir: /home/hpk/tmp/doc-exec-423/.cache
    plugins: xdist, bugzilla, cache, oejskit, cli, timeout, pep8, cov
    collecting ... collected 8 items
    
    test_module.py:16: test_0[1] PASSED
    test_module.py:16: test_0[2] PASSED
    test_module.py:18: test_1[mod1] PASSED
    test_module.py:20: test_2[1-mod1] PASSED
    test_module.py:20: test_2[2-mod1] PASSED
    test_module.py:18: test_1[mod2] PASSED
    test_module.py:20: test_2[1-mod2] PASSED
    test_module.py:20: test_2[2-mod2] PASSED
    
    ========================= 8 passed in 0.02 seconds =========================
      test0 1
      test0 2
    create mod1
      test1 mod1
      test2 1 mod1
      test2 2 mod1
    fin mod1
    create mod2
      test1 mod2
      test2 1 mod2
      test2 2 mod2
    fin mod2

You can see that the parametrized module-scoped ``modarg`` resource caused
an ordering of test execution that lead to the fewest possible "active" resources. The finalizer for the ``mod1`` parametrized resource was executed 
before the ``mod2`` resource was setup.

.. currentmodule:: _pytest.python
.. _`request`:

``request``: interacting with test invocation context
-------------------------------------------------------

The ``request`` object may be received by `@pytest.factory`_ or
:ref:`@pytest.setup <setup>` marked functions and provides methods

* to inspect attributes of the requesting test context, such as 
  ``function``, ``cls``, ``module``, ``session`` and the pytest 
  ``config`` object.  A request object passed to a parametrized factory
  will also carry a ``request.param`` object (A parametrized factory and
  all of its dependent tests will be called with each of the factory-specified
  ``params``).

* to add finalizers/teardowns to be invoked when the last
  test of the requesting test context executes

.. autoclass:: _pytest.python.FuncargRequest()
    :members:


.. _`test generators`:
.. _`parametrizing-tests`:
.. _`parametrized test functions`:

Parametrizing test functions
==========================================================================

While the `@pytest.factory`_ decorator allows to define parametrization
of funcarg resources at the factory-level, there are also means to
define parametrization at test functions directly:

* `@pytest.mark.parametrize`_ to provide multiple argument sets 
  for a particular test function or class.

* `pytest_generate_tests`_ to implement your own custom parametrization 
  scheme or extensions.

.. _`@pytest.mark.parametrize`:

``@pytest.mark.parametrize``: parametrizing test functions
---------------------------------------------------------------------

.. regendoc: wipe

.. versionadded:: 2.2

The builtin ``pytest.mark.parametrize`` decorator enables
parametrization of arguments for a test function.  Here is a typical example
of a test function that wants check for expected output given a certain input::

    # content of test_expectation.py
    import pytest
    @pytest.mark.parametrize(("input", "expected"), [
        ("3+5", 8),
        ("2+4", 6),
        ("6*9", 42),
    ])
    def test_eval(input, expected):
        assert eval(input) == expected

The ``@parametrize`` decorator defines three different argument sets for the 
two ``(input, output)`` arguments of ``test_eval`` function so the latter
will be run three times::

    $ py.test -q
    collecting ... collected 13 items
    ....F........
    ================================= FAILURES =================================
    ____________________________ test_eval[6*9-42] _____________________________
    
    input = '6*9', expected = 42
    
        @pytest.mark.parametrize(("input", "expected"), [
            ("3+5", 8),
            ("2+4", 6),
            ("6*9", 42),
        ])
        def test_eval(input, expected):
    >       assert eval(input) == expected
    E       assert 54 == 42
    E        +  where 54 = eval('6*9')
    
    test_expectation.py:8: AssertionError
    1 failed, 12 passed in 6.41 seconds

As expected only one pair of input/output values fails the simple test function.
As usual you can see the ``input`` and ``output`` values in the traceback.

Note that there are various ways how you can mark groups of functions,
see :ref:`mark`.


.. _`pytest_generate_tests`:

Basic ``pytest_generate_tests`` example
---------------------------------------------

..  XXX

    > line 598 "Basic ``pytest_generate_tests`` example" - I think this is
    > not a very basic example! I think it is copied from parametrize.txt
    > page, where it might make more sense. Here is what I would consider a
    > basic example.
    > 
    > # code
    > def isSquare(n):
    >     n = n ** 0.5
    >     return int(n) == n
    > 
    > # test file
    > def pytest_generate_tests(metafunc):
    >     squares = [1, 4, 9, 16, 25, 36, 49]
    >     for n in range(1, 50):
    >         expected = n in squares
    >         if metafunc.function.__name__ == 'test_isSquare':
    >             metafunc.addcall(id=n, funcargs=dict(n=n,
    >             expected=expected))
    > 
    > 
    > def test_isSquare(n, expected):
    >     assert isSquare(n) == expected


.. XXX
    consider adding more examples, also mixed (factory-parametrized/test-function-parametrized, see mail from Brianna)

The ``pytest_generate_tests`` hook is typically used if you want
to go beyond what ``@pytest.mark.parametrize`` offers.  For example,
let's say we want to execute a test with different computation
parameters and the parameter range shall be determined by a command
line argument.  Let's first write a simple (do-nothing) computation test::

    # content of test_compute.py

    def test_compute(param1):
        assert param1 < 4

Now we add a ``conftest.py`` file containing the addition of a 
command line option and the generation of tests depending on
that option::

    # content of conftest.py

    def pytest_addoption(parser):
        parser.addoption("--all", action="store_true",
            help="run all combinations")

    def pytest_generate_tests(metafunc):
        if 'param1' in metafunc.funcargnames:
            if metafunc.config.option.all:
                end = 5
            else:
                end = 2
            metafunc.parametrize("param1", range(end))

This means that we only run two tests if no option is passed::

    $ py.test -q test_compute.py
    collecting ... collected 2 items
    ..
    2 passed in 0.01 seconds

And we run five tests if we add the ``--all`` option::

    $ py.test -q --all test_compute.py
    collecting ... collected 5 items
    ....F
    ================================= FAILURES =================================
    _____________________________ test_compute[4] ______________________________
    
    param1 = 4
    
        def test_compute(param1):
    >       assert param1 < 4
    E       assert 4 < 4
    
    test_compute.py:3: AssertionError
    1 failed, 4 passed in 0.02 seconds

As expected when running the full range of ``param1`` values
we'll get an error on the last one.

You might want to look at :ref:`more parametrization examples <paramexamples>`.


.. _`metafunc object`:

The **metafunc** object
-------------------------------------------

metafunc objects are passed to the ``pytest_generate_tests`` hook.
They help to inspect a testfunction and to generate tests
according to test configuration or values specified
in the class or module where a test function is defined:

``metafunc.funcargnames``: set of required function arguments for given function

``metafunc.function``: underlying python test function

``metafunc.cls``: class object where the test function is defined in or None.

``metafunc.module``: the module object where the test function is defined in.

``metafunc.config``: access to command line opts and general config

.. automethod:: Metafunc.parametrize
.. automethod:: Metafunc.addcall(funcargs=None,id=_notexists,param=_notexists)


.. regendoc:wipe


.. _`compatibility notes`:

.. _`funcargscompat`:

Compatibility notes
============================================================

**Funcargs** were originally introduced to pytest-2.0.  In pytest-2.3 
the mechanism was extended and refined:

* previously funcarg factories were specified with a special 
  ``pytest_funcarg__NAME`` prefix instead of using the 
  ``@pytest.factory`` decorator.

* Factories received a `request`_ object which managed caching through
  ``request.cached_setup()`` calls and allowed using other funcargs via 
  ``request.getfuncargvalue()`` calls.  These intricate APIs made it hard 
  to do proper parametrization and implement resource caching. The
  new ``@pytest.factory`` decorator allows to simply declare the scope
  and let pytest figure things out for you.

* if you used parametrization and funcarg factories which made use of
  ``request.cached_setup()`` it is recommeneded to invest a few minutes
  and simplify your funcarg factory code to use the `@pytest.factory`_ 
  decorator instead.  This will also allow to take advantage of 
  the `automatic per-resource grouping`_ of tests.

.. note::

    Throughout the pytest documents the ``pytest_funcarg__NAME`` way of
    defining a funcarg factory is often termed "old-style".  Their
    use remains fully supported and existing code using it should run
    unmodified.