.. _resources: .. _`funcargs`: .. _`funcarg mechanism`: ======================================================= funcargs: resource injection and parametrization ======================================================= .. note:: pytest-2.3 introduces major refinements to the original funcarg mechanism introduced to pytest-2.0. While the old way remains fully supported, it is recommended to use the refined mechanisms. See also the `compatibility notes`_ and the detailed :ref:`reasoning for the new funcarg and setup functions `. .. _`Dependency injection`: http://en.wikipedia.org/wiki/Dependency_injection Introduction ==================== py.test supports the injection of objects into test and setup functions and flexibly control their life cycle in relation to the overall test execution. Moreover, you can run a test function multiple times injecting different objects. The basic mechanism for injecting objects is called the *funcarg mechanism* because objects are **injected** when a test or setup function states it as an **argument**. The injected argument is created by a call to a registered **funcarg factory**. This approach is an example of `Dependency Injection`_ and helps to de-couple test code from the setup of required objects: at test writing time you do not need to care for the details of where and how your required resources are constructed, if they are shared on a per-class, module or session basis, or if your test function is invoked multiple times with differently configured resource instances. When a test function is invoked multiple times with different arguments we speak of **parametrized testing**. This is useful if you want to test e.g. against different database backends or want to write a parametrized test function, checking that certain inputs lead to certain outputs. You can parametrize funcarg factories, parametrize test function arguments or even implement your own parametrization scheme through a plugin hook. pytest additionally offers powerful xunit-style :ref:`setup functions ` for the cases where you need to create implicit test state that is not passed explicitely to test functions. Concretely, there are three main means of funcarg management: * a `@pytest.factory`_ marker to define resource factories, their scoping and parametrization. Factories can themselves receive resources through their function arguments, easing the setup of `interdependent resources`_. Factories can use the special `testcontext`_ object to access details from where the factory or setup function is called and for registering finalizers. * a `@pytest.mark.parametrize`_ marker for executing test functions multiple times with different argument sets, * a `pytest_generate_tests`_ plugin hook marker for implementing your parametrization for a test function which may depend on command line options, class/module attributes etc. Apart from making it easy to manage your own test resources pytest also comes with some :ref:`builtinresources` which you can use without defining them yourself. Third-party plugins offer yet more domain-specific funcarg resources (for example the `pytest-django plugin `_) so that after plugin installation you can simply use them in your test and setup functions. This all contributes to high re-useability of test resource management and goes far beyond what can be done with the classical xUnit style approach which encodes resource setup statically into the test source code, leading to duplicate and hard-to change fixtures. .. _`@pytest.factory`: ``@pytest.factory``: Creating parametrized, scoped resources ===================================================================== Basic funcarg injection example ----------------------------------------------------------- Let's look at a simple self-contained test module using a factory and a funcarg:: # content of ./test_simplefactory.py import pytest @pytest.factory() def myfuncarg(): return 42 def test_function(myfuncarg): assert myfuncarg == 17 Here, the ``test_function`` needs an object named ``myfuncarg`` and thus py.test will discover and call the ``@pytest.factory`` marked ``myfuncarg`` factory function. Running the tests looks like this:: $ py.test test_simplefactory.py =========================== test session starts ============================ platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev8 plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov collecting ... collected 1 items test_simplefactory.py F ================================= FAILURES ================================= ______________________________ test_function _______________________________ myfuncarg = 42 def test_function(myfuncarg): > assert myfuncarg == 17 E assert 42 == 17 test_simplefactory.py:8: AssertionError ========================= 1 failed in 0.02 seconds ========================= This shows that the test function was called with a ``myfuncarg`` argument value of ``42`` and the assert fails as expected. Here is how py.test comes to call the test function this way: 1. py.test :ref:`finds ` the ``test_function`` because of the ``test_`` prefix. The test function needs a function argument named ``myfuncarg``. A matching factory function is discovered by looking for a factory function named ``myfuncarg``. 2. ``myfuncarg()`` is called to create a value ``42``. 3. ``test_function(42)`` is now called and results in the above reported exception because of the assertion mismatch. Note that if you misspell a function argument or want to use one that isn't available, you'll see an error with a list of available function arguments. .. Note:: You can always issue:: py.test --funcargs test_simplefactory.py to see available function arguments. Location independency of funcarg factories ---------------------------------------------------- If during implementing your tests you realize that you want to use a factory from multiple test files you can move it to a :ref:`conftest.py ` file or even separately installable :ref:`plugins ` without changing test code. The discovery of funcarg factories starts at test classes, then test modules, then ``conftest.py`` files and finally builtin and 3-rd party plugins. Creating and using a session-shared funcarg ----------------------------------------------------------------- .. regendoc:wipe .. versionadded:: 2.3 The `@pytest.factory`_ marker allows to * mark a function as a factory for resources, useable by test and setup functions * define parameters in order to run tests multiple times with different resource instances * declare a scope which determines the level of caching, i.e. how often the factory will be called. Valid scopes are ``session``, ``module``, ``class`` and ``function``. Here is a simple example of a factory creating a shared ``smtplib.SMTP`` connection resource which test functions then may use across the whole test session:: # content of conftest.py import pytest import smtplib @pytest.factory(scope="session") def smtp(testcontext): return smtplib.SMTP("merlinux.eu") The name of the factory is ``smtp`` (the factory function name) and you can access its result by listing the name ``smtp`` as an input parameter in any test or setup function:: # content of test_module.py def test_ehlo(smtp): response = smtp.ehlo() assert response[0] == 250 assert "merlinux" in response[1] assert 0 # for demo purposes def test_noop(smtp): response = smtp.noop() assert response[0] == 250 assert 0 # for demo purposes We deliberately insert failing ``assert 0`` statements in order to inspect what is going on and can now run the tests:: $ py.test -q test_module.py collecting ... collected 2 items FF ================================= FAILURES ================================= ________________________________ test_ehlo _________________________________ smtp = def test_ehlo(smtp): response = smtp.ehlo() assert response[0] == 250 assert "merlinux" in response[1] > assert 0 # for demo purposes E assert 0 test_module.py:5: AssertionError ________________________________ test_noop _________________________________ smtp = def test_noop(smtp): response = smtp.noop() assert response[0] == 250 > assert 0 # for demo purposes E assert 0 test_module.py:10: AssertionError 2 failed in 0.20 seconds you see the two ``assert 0`` failing and can also see that the same (session-scoped) object was passed into the two test functions. Parametrizing a session-shared funcarg resource ----------------------------------------------------------------- Extending the previous example, we can flag the factory to create two ``smtp`` values which will cause all tests using it to run twice with two different values. The factory function gets access to each parameter through the special `testcontext`_ object:: # content of conftest.py import pytest import smtplib @pytest.factory(scope="session", params=["merlinux.eu", "mail.python.org"]) def smtp(testcontext): return smtplib.SMTP(testcontext.param) The main change is the definition of a ``params`` list in the ``factory``-marker and the ``testcontext.param`` access within the factory function. No test function code needs to change. So let's just do another run:: $ py.test -q collecting ... collected 4 items FFFF ================================= FAILURES ================================= __________________________ test_ehlo[merlinux.eu] __________________________ smtp = def test_ehlo(smtp): response = smtp.ehlo() assert response[0] == 250 assert "merlinux" in response[1] > assert 0 # for demo purposes E assert 0 test_module.py:5: AssertionError __________________________ test_noop[merlinux.eu] __________________________ smtp = def test_noop(smtp): response = smtp.noop() assert response[0] == 250 > assert 0 # for demo purposes E assert 0 test_module.py:10: AssertionError ________________________ test_ehlo[mail.python.org] ________________________ smtp = def test_ehlo(smtp): response = smtp.ehlo() assert response[0] == 250 > assert "merlinux" in response[1] E assert 'merlinux' in 'mail.python.org\nSIZE 10240000\nETRN\nSTARTTLS\nENHANCEDSTATUSCODES\n8BITMIME\nDSN' test_module.py:4: AssertionError ________________________ test_noop[mail.python.org] ________________________ smtp = def test_noop(smtp): response = smtp.noop() assert response[0] == 250 > assert 0 # for demo purposes E assert 0 test_module.py:10: AssertionError 4 failed in 6.00 seconds We get four failures because we are running the two tests twice with different ``smtp`` instantiations as defined on the factory. Note that with the ``mail.python.org`` connection the second test fails in ``test_ehlo`` because it expects a specific server string. Adding a finalizer to the parametrized resource -------------------------------------------------------- Further extending the ``smtp`` example, we now want to properly close a smtp server connection after the last test using it has been run. We can do this by calling the ``testcontext.addfinalizer()`` helper:: # content of conftest.py import pytest import smtplib @pytest.factory(scope="session", params=["merlinux.eu", "mail.python.org"]) def smtp(testcontext): smtp = smtplib.SMTP(testcontext.param) def fin(): print ("finalizing %s" % smtp) smtp.close() testcontext.addfinalizer(fin) return smtp We also add a print call and then run py.test without the default output capturing and disabled traceback reporting:: $ py.test -s -q --tb=no collecting ... collected 4 items FFFF 4 failed in 5.62 seconds finalizing finalizing We see that the two ``smtp`` instances are finalized appropriately. Looking at test collection without running tests ------------------------------------------------------ You can also look at what tests pytest collects without running them:: $ py.test --collectonly =========================== test session starts ============================ platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev8 plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov collecting ... collected 4 items ============================= in 0.02 seconds ============================= Note that pytest orders your test run by resource usage, minimizing the number of active resources at any given time. .. _`interdependent resources`: Interdepdendent resources ---------------------------------------------------------- You can not only use funcargs in test functions but also in their factories themselves. Extending the previous example we can instantiate another object ``app`` and stick the ``smtp`` resource into it like this:: # content of test_appsetup.py import pytest class App: def __init__(self, smtp): self.smtp = smtp @pytest.factory(scope="module") def app(smtp): return App(smtp) def test_exists(app): assert app.smtp Here we define the factory local to the test module and make it access the ``smtp`` resource by listing it as an input parameter. Let's run this:: $ py.test -v test_appsetup.py =========================== test session starts ============================ platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev8 -- /home/hpk/venv/1/bin/python cachedir: /home/hpk/tmp/doc-exec-414/.cache plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov collecting ... collected 2 items test_appsetup.py:12: test_exists[merlinux.eu] PASSED test_appsetup.py:12: test_exists[mail.python.org] PASSED ========================= 2 passed in 5.37 seconds ========================= Due to the parametrization of ``smtp`` the test will run twice with two different ``App`` instances and respective smtp servers. There is no need for the ``app`` factory to be aware of the parametrization. Note that the ``app`` factory has a scope of ``module`` whereas it uses the session-scoped ``smtp`` object: it is fine for factories to use "broader" scoped resources but not the other way round. A session-scoped resource could not use a module-scoped resource in a meaningful way. .. _`automatic per-resource grouping`: Grouping tests by resource parameters ---------------------------------------------------------- .. regendoc: wipe pytest minimizes the number of active resources during test runs. If you have a parametrized resource, then all the tests using one resource instance will execute one after another. Then any finalizers are called for that resource instance and then the next parametrized resource instance is created and its tests are run. Among other things, this eases testing of applications which create and use global state. The following example uses two parametrized funcargs, one of which is scoped on a per-module basis, and all the functions perform ``print`` call s to show the flow of calls:: # content of test_module.py import pytest @pytest.factory(scope="module", params=["mod1", "mod2"]) def modarg(testcontext): param = testcontext.param print "create", param def fin(): print "fin", param testcontext.addfinalizer(fin) return param @pytest.factory(scope="function", params=[1,2]) def otherarg(testcontext): return testcontext.param def test_0(otherarg): print " test0", otherarg def test_1(modarg): print " test1", modarg def test_2(otherarg, modarg): print " test2", otherarg, modarg Let's run the tests in verbose mode and with looking at the print-output:: $ py.test -v -s test_module.py =========================== test session starts ============================ platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev8 -- /home/hpk/venv/1/bin/python cachedir: /home/hpk/tmp/doc-exec-414/.cache plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov collecting ... collected 8 items test_module.py:16: test_0[1] PASSED test_module.py:16: test_0[2] PASSED test_module.py:18: test_1[mod1] PASSED test_module.py:20: test_2[1-mod1] PASSED test_module.py:20: test_2[2-mod1] PASSED test_module.py:18: test_1[mod2] PASSED test_module.py:20: test_2[1-mod2] PASSED test_module.py:20: test_2[2-mod2] PASSED ========================= 8 passed in 0.02 seconds ========================= test0 1 test0 2 create mod1 test1 mod1 test2 1 mod1 test2 2 mod1 fin mod1 create mod2 test1 mod2 test2 1 mod2 test2 2 mod2 fin mod2 You can see that the parametrized module-scoped ``modarg`` resource caused a re-ordering of test execution. The finalizer for the ``mod1`` parametrized resource was executed before the ``mod2`` resource was setup. .. currentmodule:: _pytest.python .. _`testcontext`: ``testcontext``: interacting with test context --------------------------------------------------- The ``testcontext`` object may be used by `@pytest.factory`_ or :ref:`@pytest.setup ` marked functions. It contains information relating to the test context within which the marked function executes. Moreover, you can call ``testcontext.addfinalizer(myfinalizer)`` in order to trigger a call to ``myfinalizer`` after the last test in the test context has executed. If passed to a parametrized factory ``testcontext.param`` will contain a parameter (one value out of the ``params`` list specified with the `@pytest.factory`_ marker). .. autoclass:: _pytest.python.TestContext() :members: .. _`test generators`: .. _`parametrizing-tests`: .. _`parametrized test functions`: Parametrizing test functions ========================================================================== While the `@pytest.factory`_ decorator allows to define parametrization of funcarg resources at the factory-level, there are also means to define parametrization at test functions directly: * `@pytest.mark.parametrize`_ to provide multiple argument sets for a particular test function or class. * `pytest_generate_tests`_ to implement your own custom parametrization scheme or extensions. .. _`@pytest.mark.parametrize`: ``@pytest.mark.parametrize``: parametrizing test functions --------------------------------------------------------------------- .. regendoc: wipe .. versionadded:: 2.2 The builtin ``pytest.mark.parametrize`` decorator enables parametrization of arguments for a test function. Here is a typical example of a test function that wants check for expected output given a certain input:: # content of test_expectation.py import pytest @pytest.mark.parametrize(("input", "expected"), [ ("3+5", 8), ("2+4", 6), ("6*9", 42), ]) def test_eval(input, expected): assert eval(input) == expected The ``@parametrize`` decorator defines three different argument sets for the two ``(input, output)`` arguments of ``test_eval`` function so the latter will be run three times:: $ py.test -q collecting ... collected 13 items ....F........ ================================= FAILURES ================================= ____________________________ test_eval[6*9-42] _____________________________ input = '6*9', expected = 42 @pytest.mark.parametrize(("input", "expected"), [ ("3+5", 8), ("2+4", 6), ("6*9", 42), ]) def test_eval(input, expected): > assert eval(input) == expected E assert 54 == 42 E + where 54 = eval('6*9') test_expectation.py:8: AssertionError 1 failed, 12 passed in 5.76 seconds As expected only one pair of input/output values fails the simple test function. As usual you can see the ``input`` and ``output`` values in the traceback. Note that there are various ways how you can mark groups of functions, see :ref:`mark`. .. _`pytest_generate_tests`: Basic ``pytest_generate_tests`` example --------------------------------------------- The ``pytest_generate_tests`` hook is typically used if you want to go beyond what ``@pytest.mark.parametrize`` offers. For example, let's say we want to execute a test with different computation parameters and the parameter range shall be determined by a command line argument. Let's first write a simple (do-nothing) computation test:: # content of test_compute.py def test_compute(param1): assert param1 < 4 Now we add a ``conftest.py`` file containing the addition of a command line option and the generation of tests depending on that option:: # content of conftest.py def pytest_addoption(parser): parser.addoption("--all", action="store_true", help="run all combinations") def pytest_generate_tests(metafunc): if 'param1' in metafunc.funcargnames: if metafunc.config.option.all: end = 5 else: end = 2 metafunc.parametrize("param1", range(end)) This means that we only run two tests if no option is passed:: $ py.test -q test_compute.py collecting ... collected 2 items .. 2 passed in 0.02 seconds And we run five tests if we add the ``--all`` option:: $ py.test -q --all collecting ... collected 5 items ....F ================================= FAILURES ================================= _____________________________ test_compute[4] ______________________________ param1 = 4 def test_compute(param1): > assert param1 < 4 E assert 4 < 4 test_compute.py:3: AssertionError 1 failed, 4 passed in 0.03 seconds As expected when running the full range of ``param1`` values we'll get an error on the last one. You might want to look at :ref:`more parametrization examples `. .. _`metafunc object`: The **metafunc** object ------------------------------------------- metafunc objects are passed to the ``pytest_generate_tests`` hook. They help to inspect a testfunction and to generate tests according to test configuration or values specified in the class or module where a test function is defined: ``metafunc.funcargnames``: set of required function arguments for given function ``metafunc.function``: underlying python test function ``metafunc.cls``: class object where the test function is defined in or None. ``metafunc.module``: the module object where the test function is defined in. ``metafunc.config``: access to command line opts and general config .. automethod:: Metafunc.parametrize .. automethod:: Metafunc.addcall(funcargs=None,id=_notexists,param=_notexists) .. regendoc:wipe .. _`compatibility notes`: .. _`funcargscompat`: Compatibility notes ============================================================ **Funcargs** were originally introduced to pytest-2.0. In pytest-2.3 the mechanism was extended and refined: * previously funcarg factories were specified with a special ``pytest_funcarg__NAME`` prefix instead of using the ``@pytest.factory`` decorator. * Factories received a `request`_ object which managed caching through ``request.cached_setup()`` calls and allowed using other funcargs via ``request.getfuncargvalue()`` calls. These intricate APIs made it hard to do proper parametrization and implement resource caching. The new ``@pytest.factory`` decorator allows to simply declare the scope and let pytest figure things out for you. * if you used parametrization and funcarg factories which made use of ``request.cached_setup()`` it is recommeneded to invest a few minutes and simplify your funcarg factory code to use the `@pytest.factory`_ decorator instead. This will also allow to take advantage of the `automatic per-resource grouping`_ of tests. .. note:: Throughout the pytest documents the ``pytest_funcarg__NAME`` way of defining a funcarg factory is often termed "old-style". Their use remains fully supported and existing code using it should run unmodified. .. _request: The request object passed to old-style factories ----------------------------------------------------------------- Old-style funcarg factory definitions can receive a :py:class:`~_pytest.python.FuncargRequest` object which provides methods to manage caching and finalization in the context of the test invocation as well as several attributes of the the underlying test item.