488 lines
16 KiB
Plaintext
488 lines
16 KiB
Plaintext
|
|
.. _resources:
|
|
|
|
=======================================================
|
|
test resource injection and parametrization
|
|
=======================================================
|
|
|
|
.. _`Dependency injection`: http://en.wikipedia.org/wiki/Dependency_injection
|
|
|
|
.. versionadded: 2.3
|
|
|
|
pytest offers very flexible means for managing test resources and
|
|
test parametrization.
|
|
|
|
The pytest resource management mechanism is an example of `Dependency
|
|
Injection`_ because test and :ref:`setup functions <xunitsetup>` receive
|
|
resources simply by stating them as an input argument. Therefore and
|
|
also for historic reasons, they are often called **funcargs**. At test
|
|
writing time you typically do not need to care for the details of how
|
|
your required resources are constructed, if they live through a
|
|
function, class, module or session scope or if the test will be called
|
|
multiple times with different resource instances.
|
|
|
|
To create a value with which to call a test function a resource factory
|
|
function is called which gets full access to the test context and can
|
|
register finalizers which are to be run after the last test in that context
|
|
finished. Resource factories can be implemented in same test class or
|
|
test module, in a per-directory ``conftest.py`` file or in an external
|
|
plugin. This allows **total de-coupling of test and setup code**,
|
|
lowering the cost of refactoring.
|
|
|
|
A test function may be invoked multiple times in which case we
|
|
speak of parametrization. You can parametrize resources or parametrize
|
|
test function arguments directly or even implement your own parametrization
|
|
scheme through a plugin hook.
|
|
|
|
A resource has a **name** under which test and setup functions
|
|
can access it by listing it as an input argument. Due to this and
|
|
also for historic reasons, resources are often called **funcargs**.
|
|
A resource is created by a factory which can be flagged with a **scope**
|
|
to only create resources on a per-class/per-module/per-session basis
|
|
instead of the default per-function scope.
|
|
|
|
Concretely, there are three means of resource and parametrization management:
|
|
|
|
* a `@pytest.factory`_ marker to define resource factories,
|
|
their scoping and parametrization. Factories can themselves
|
|
receive resources through their function arguments, easing
|
|
the setup of interdependent resources. They can also use
|
|
the special `testcontext`_ object to access details n which
|
|
the factory/setup is called and for registering finalizers.
|
|
|
|
* a `@pytest.mark.parametrize`_ marker for executing test functions
|
|
multiple times with different parameter sets
|
|
|
|
* a `pytest_generate_tests`_ plugin hook marker for implementing
|
|
your parametrization for a test function which may depend on
|
|
command line options, class/module attributes etc.
|
|
|
|
Finally, pytest comes with some :ref:`builtinresources` which
|
|
you can use without defining them yourself. Moreover, third-party
|
|
plugins offer their own resources so that after installation
|
|
you can simply use them in your test and setup functions.
|
|
|
|
.. _`@pytest.factory`:
|
|
|
|
``@pytest.factory``: Creating parametrized, scoped resources
|
|
-----------------------------------------------------------------
|
|
|
|
.. regendoc:wipe
|
|
|
|
.. versionadded:: 2.3
|
|
|
|
The `@pytest.factory`_ marker allows to
|
|
|
|
* mark a function as a factory for resources, useable by test and setup functions
|
|
* define parameters in order to run tests multiple times with different
|
|
resource instances
|
|
* set a scope which determines the level of caching, i.e. how often
|
|
the factory will be called. Valid scopes are ``session``, ``module``,
|
|
``class`` and ``function``.
|
|
|
|
Here is a simple example of a factory creating a shared ``smtplib.SMTP``
|
|
connection resource which test functions then may use across the whole
|
|
test session::
|
|
|
|
# content of conftest.py
|
|
import pytest
|
|
import smtplib
|
|
|
|
@pytest.factory(scope="session")
|
|
def smtp(testcontext):
|
|
smtp = smtplib.SMTP("merlinux.eu")
|
|
testcontext.addfinalizer(smtp.close)
|
|
return smtp
|
|
|
|
The name of the resource is ``smtp`` (the factory function name)
|
|
and you can now access the ``smtp`` resource by listing it as
|
|
an input parameter in any test function below the directory where
|
|
``conftest.py`` is located::
|
|
|
|
# content of test_module.py
|
|
def test_ehlo(smtp):
|
|
response = smtp.ehlo()
|
|
assert response[0] == 250
|
|
assert "merlinux" in response[1]
|
|
assert 0 # for demo purposes
|
|
|
|
def test_noop(smtp):
|
|
response = smtp.noop()
|
|
assert response[0] == 250
|
|
assert 0 # for demo purposes
|
|
|
|
If you run the tests::
|
|
|
|
$ py.test -q
|
|
collecting ... collected 2 items
|
|
FF
|
|
================================= FAILURES =================================
|
|
________________________________ test_ehlo _________________________________
|
|
|
|
smtp = <smtplib.SMTP instance at 0x1c7c638>
|
|
|
|
def test_ehlo(smtp):
|
|
response = smtp.ehlo()
|
|
assert response[0] == 250
|
|
assert "merlinux" in response[1]
|
|
> assert 0 # for demo purposes
|
|
E assert 0
|
|
|
|
test_module.py:5: AssertionError
|
|
________________________________ test_noop _________________________________
|
|
|
|
smtp = <smtplib.SMTP instance at 0x1c7c638>
|
|
|
|
def test_noop(smtp):
|
|
response = smtp.noop()
|
|
assert response[0] == 250
|
|
> assert 0 # for demo purposes
|
|
E assert 0
|
|
|
|
test_module.py:10: AssertionError
|
|
2 failed in 0.18 seconds
|
|
|
|
you will see the two ``assert 0`` failing and can see that
|
|
the same (session-scoped) object was passed into the two test functions.
|
|
|
|
If you now want to test multiple servers you can simply parametrize
|
|
the ``smtp`` factory::
|
|
|
|
# content of conftest.py
|
|
import pytest
|
|
import smtplib
|
|
|
|
@pytest.factory(scope="session",
|
|
params=["merlinux.eu", "mail.python.org"])
|
|
def smtp(testcontext):
|
|
smtp = smtplib.SMTP(testcontext.param)
|
|
def fin():
|
|
smtp.close()
|
|
testcontext.addfinalizer(fin)
|
|
return smtp
|
|
|
|
The main change is the definition of a ``params`` list in the
|
|
``factory``-marker and the ``testcontext.param`` access within the
|
|
factory function. No test code needs to change. So let's just do another
|
|
run::
|
|
|
|
$ py.test -q
|
|
collecting ... collected 4 items
|
|
FFFF
|
|
================================= FAILURES =================================
|
|
__________________________ test_ehlo[merlinux.eu] __________________________
|
|
|
|
smtp = <smtplib.SMTP instance at 0x1d162d8>
|
|
|
|
def test_ehlo(smtp):
|
|
response = smtp.ehlo()
|
|
assert response[0] == 250
|
|
assert "merlinux" in response[1]
|
|
> assert 0 # for demo purposes
|
|
E assert 0
|
|
|
|
test_module.py:5: AssertionError
|
|
__________________________ test_noop[merlinux.eu] __________________________
|
|
|
|
smtp = <smtplib.SMTP instance at 0x1d162d8>
|
|
|
|
def test_noop(smtp):
|
|
response = smtp.noop()
|
|
assert response[0] == 250
|
|
> assert 0 # for demo purposes
|
|
E assert 0
|
|
|
|
test_module.py:10: AssertionError
|
|
________________________ test_ehlo[mail.python.org] ________________________
|
|
|
|
smtp = <smtplib.SMTP instance at 0x1d1f098>
|
|
|
|
def test_ehlo(smtp):
|
|
response = smtp.ehlo()
|
|
assert response[0] == 250
|
|
> assert "merlinux" in response[1]
|
|
E assert 'merlinux' in 'mail.python.org\nSIZE 10240000\nETRN\nSTARTTLS\nENHANCEDSTATUSCODES\n8BITMIME\nDSN'
|
|
|
|
test_module.py:4: AssertionError
|
|
________________________ test_noop[mail.python.org] ________________________
|
|
|
|
smtp = <smtplib.SMTP instance at 0x1d1f098>
|
|
|
|
def test_noop(smtp):
|
|
response = smtp.noop()
|
|
assert response[0] == 250
|
|
> assert 0 # for demo purposes
|
|
E assert 0
|
|
|
|
test_module.py:10: AssertionError
|
|
4 failed in 6.42 seconds
|
|
|
|
We get four failures because we are running the two tests twice with
|
|
different ``smtp`` instantiations as defined on the factory.
|
|
Note that with the ``mail.python.org`` connection the second test
|
|
fails in ``test_ehlo`` because it expects a specific server string.
|
|
|
|
You can also look at what tests pytest collects without running them::
|
|
|
|
$ py.test --collectonly
|
|
=========================== test session starts ============================
|
|
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev7
|
|
plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov
|
|
collecting ... collected 4 items
|
|
<Module 'test_module.py'>
|
|
<Function 'test_ehlo[merlinux.eu]'>
|
|
<Function 'test_noop[merlinux.eu]'>
|
|
<Function 'test_ehlo[mail.python.org]'>
|
|
<Function 'test_noop[mail.python.org]'>
|
|
|
|
============================= in 0.02 seconds =============================
|
|
|
|
Note that pytest orders your test run by resource usage, minimizing
|
|
the number of active resources at any given time.
|
|
|
|
|
|
Interdepdendent resources
|
|
----------------------------------------------------------
|
|
|
|
You can not only use resources in test functions but also in resource factories
|
|
themselves. Extending the previous example we can instantiate an application
|
|
object by sticking the ``smtp`` resource into it::
|
|
|
|
# content of test_appsetup.py
|
|
|
|
import pytest
|
|
|
|
class App:
|
|
def __init__(self, smtp):
|
|
self.smtp = smtp
|
|
|
|
@pytest.factory(scope="module")
|
|
def app(smtp):
|
|
return App(smtp)
|
|
|
|
def test_exists(app):
|
|
assert app.smtp
|
|
|
|
Let's run this::
|
|
|
|
$ py.test -v test_appsetup.py
|
|
=========================== test session starts ============================
|
|
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev7 -- /home/hpk/venv/1/bin/python
|
|
cachedir: /home/hpk/tmp/doc-exec-398/.cache
|
|
plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov
|
|
collecting ... collected 2 items
|
|
|
|
test_appsetup.py:12: test_exists[merlinux.eu] PASSED
|
|
test_appsetup.py:12: test_exists[mail.python.org] PASSED
|
|
|
|
========================= 2 passed in 5.96 seconds =========================
|
|
|
|
Due to the parametrization of ``smtp`` the test will
|
|
run twice with two different ``App`` instances and respective smtp servers.
|
|
There is no need for the ``app`` factory to be aware of the parametrization.
|
|
|
|
|
|
|
|
Grouping tests by resource parameters
|
|
----------------------------------------------------------
|
|
|
|
.. regendoc: wipe
|
|
|
|
pytest minimizes the number of active resources during test runs.
|
|
If you have a parametrized resource, then all the tests using one
|
|
resource instance will execute one after another. Then any finalizers
|
|
are called for that resource instance and then the next parametrized
|
|
resource instance is created and its tests are run. Among other things,
|
|
this eases testing of applications which create and use global state.
|
|
|
|
The following example uses two parametrized funcargs, one of which is
|
|
scoped on a per-module basis::
|
|
|
|
# content of test_module.py
|
|
import pytest
|
|
|
|
@pytest.factory(scope="module", params=["mod1", "mod2"])
|
|
def modarg(testcontext):
|
|
param = testcontext.param
|
|
print "create", param
|
|
def fin():
|
|
print "fin", param
|
|
testcontext.addfinalizer(fin)
|
|
return param
|
|
|
|
@pytest.factory(scope="function", params=[1,2])
|
|
def otherarg(testcontext):
|
|
return testcontext.param
|
|
|
|
def test_0(otherarg):
|
|
print " test0", otherarg
|
|
def test_1(modarg):
|
|
print " test1", modarg
|
|
def test_2(otherarg, modarg):
|
|
print " test2", otherarg, modarg
|
|
|
|
Let's run the tests in verbose mode and with looking at the print-output::
|
|
|
|
$ py.test -v -s
|
|
=========================== test session starts ============================
|
|
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev7 -- /home/hpk/venv/1/bin/python
|
|
cachedir: /home/hpk/tmp/doc-exec-398/.cache
|
|
plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov
|
|
collecting ... collected 8 items
|
|
|
|
test_module.py:16: test_0[1] PASSED
|
|
test_module.py:16: test_0[2] PASSED
|
|
test_module.py:18: test_1[mod1] PASSED
|
|
test_module.py:20: test_2[1-mod1] PASSED
|
|
test_module.py:20: test_2[2-mod1] PASSED
|
|
test_module.py:18: test_1[mod2] PASSED
|
|
test_module.py:20: test_2[1-mod2] PASSED
|
|
test_module.py:20: test_2[2-mod2] PASSED
|
|
|
|
========================= 8 passed in 0.03 seconds =========================
|
|
test0 1
|
|
test0 2
|
|
create mod1
|
|
test1 mod1
|
|
test2 1 mod1
|
|
test2 2 mod1
|
|
fin mod1
|
|
create mod2
|
|
test1 mod2
|
|
test2 1 mod2
|
|
test2 2 mod2
|
|
fin mod2
|
|
|
|
You can see that the parametrized module-scoped ``modarg`` resource caused
|
|
a re-ordering of test execution. The finalizer for the ``mod1`` parametrized
|
|
resource was executed before the ``mod2`` resource was setup.
|
|
|
|
.. currentmodule:: _pytest.python
|
|
.. _`testcontext`:
|
|
|
|
``testcontext``: interacting with test context
|
|
---------------------------------------------------
|
|
|
|
The ``testcontext`` object may be received by `@pytest.factory`_ or
|
|
`@pytest.setup`_ marked functions. It contains information relating
|
|
to the test context within which the function executes. Moreover, you
|
|
can call ``testcontext.addfinalizer(myfinalizer)`` in order to trigger
|
|
a call to ``myfinalizer`` after the last test in the test context has executed.
|
|
If passed to a parametrized factory ``testcontext.param`` will contain
|
|
a parameter (one value out of the ``params`` list specified with the
|
|
`@pytest.factory`_ marker).
|
|
|
|
.. autoclass:: _pytest.python.TestContext()
|
|
:members:
|
|
|
|
.. _`@pytest.mark.parametrize`:
|
|
|
|
``@pytest.mark.parametrize``: directly parametrizing test functions
|
|
----------------------------------------------------------------------------
|
|
|
|
.. versionadded:: 2.2
|
|
|
|
The builtin ``pytest.mark.parametrize`` decorator enables
|
|
parametrization of arguments for a test function. Here is an example
|
|
of a test function that wants check for expected output given a certain input::
|
|
|
|
# content of test_expectation.py
|
|
import pytest
|
|
@pytest.mark.parametrize(("input", "expected"), [
|
|
("3+5", 8),
|
|
("2+4", 6),
|
|
("6*9", 42),
|
|
])
|
|
def test_eval(input, expected):
|
|
assert eval(input) == expected
|
|
|
|
we parametrize two arguments of the test function so that the test
|
|
function is called three times. Let's run it::
|
|
|
|
$ py.test -q
|
|
collecting ... collected 11 items
|
|
..F........
|
|
================================= FAILURES =================================
|
|
____________________________ test_eval[6*9-42] _____________________________
|
|
|
|
input = '6*9', expected = 42
|
|
|
|
@pytest.mark.parametrize(("input", "expected"), [
|
|
("3+5", 8),
|
|
("2+4", 6),
|
|
("6*9", 42),
|
|
])
|
|
def test_eval(input, expected):
|
|
> assert eval(input) == expected
|
|
E assert 54 == 42
|
|
E + where 54 = eval('6*9')
|
|
|
|
test_expectation.py:8: AssertionError
|
|
1 failed, 10 passed in 0.04 seconds
|
|
|
|
As expected only one pair of input/output values fails the simple test function.
|
|
|
|
Note that there are various ways how you can mark groups of functions,
|
|
see :ref:`mark`.
|
|
|
|
|
|
.. _`pytest_generate_tests`:
|
|
|
|
``pytest_generate_test``: implementing your own parametrization scheme
|
|
----------------------------------------------------------------------------
|
|
|
|
.. regendoc:wipe
|
|
|
|
Let's say we want to execute a test with different computation
|
|
parameters and the parameter range shall be determined by a command
|
|
line argument. Let's first write a simple (do-nothing) computation test::
|
|
|
|
# content of test_compute.py
|
|
|
|
def test_compute(param1):
|
|
assert param1 < 4
|
|
|
|
Now we add a test configuration like this::
|
|
|
|
# content of conftest.py
|
|
|
|
def pytest_addoption(parser):
|
|
parser.addoption("--all", action="store_true",
|
|
help="run all combinations")
|
|
|
|
def pytest_generate_tests(metafunc):
|
|
if 'param1' in metafunc.funcargnames:
|
|
if metafunc.config.option.all:
|
|
end = 5
|
|
else:
|
|
end = 2
|
|
metafunc.parametrize("param1", range(end))
|
|
|
|
This means that we only run 2 tests if we do not pass ``--all``::
|
|
|
|
$ py.test -q test_compute.py
|
|
collecting ... collected 2 items
|
|
..
|
|
2 passed in 0.03 seconds
|
|
|
|
We run only two computations, so we see two dots.
|
|
let's run the full monty::
|
|
|
|
$ py.test -q --all
|
|
collecting ... collected 5 items
|
|
....F
|
|
================================= FAILURES =================================
|
|
_____________________________ test_compute[4] ______________________________
|
|
|
|
param1 = 4
|
|
|
|
def test_compute(param1):
|
|
> assert param1 < 4
|
|
E assert 4 < 4
|
|
|
|
test_compute.py:3: AssertionError
|
|
1 failed, 4 passed in 0.03 seconds
|
|
|
|
As expected when running the full range of ``param1`` values
|
|
we'll get an error on the last one.
|