majorly refine funcargs docs and rename "resources.txt" to "funcargs.txt" so that existing web links will eventually land at this new page when pytest is released. Also integrated the detailed reasoning and update setup function docs
to reflect latest discussions and feedback gathered on py-dev mailing list.
This commit is contained in:
holger krekel
parent
46dc7eeacb
commit
6746a00cb8
|
|
@ -907,12 +907,15 @@ class Function(FunctionMixin, pytest.Item):
|
||||||
|
|
||||||
|
|
||||||
class FuncargRequest:
|
class FuncargRequest:
|
||||||
""" A request for function arguments from a test function.
|
""" (old-style) A request for function arguments from a test function.
|
||||||
|
|
||||||
Note that there is an optional ``param`` attribute in case
|
Note that there is an optional ``param`` attribute in case
|
||||||
there was an invocation to metafunc.addcall(param=...).
|
there was an invocation to metafunc.addcall(param=...).
|
||||||
If no such call was done in a ``pytest_generate_tests``
|
If no such call was done in a ``pytest_generate_tests``
|
||||||
hook, the attribute will not be present.
|
hook, the attribute will not be present. Note that
|
||||||
|
as of pytest-2.3 you probably rather want to use the
|
||||||
|
testcontext object and mark your factory with a ``@pytest.factory``
|
||||||
|
marker.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, pyfuncitem):
|
def __init__(self, pyfuncitem):
|
||||||
|
|
|
||||||
|
|
@ -11,6 +11,8 @@ py.test reference documentation
|
||||||
customize.txt
|
customize.txt
|
||||||
assert.txt
|
assert.txt
|
||||||
funcargs.txt
|
funcargs.txt
|
||||||
|
funcarg_compare.txt
|
||||||
|
setup.txt
|
||||||
xunit_setup.txt
|
xunit_setup.txt
|
||||||
capture.txt
|
capture.txt
|
||||||
monkeypatch.txt
|
monkeypatch.txt
|
||||||
|
|
|
||||||
|
|
@ -17,7 +17,7 @@
|
||||||
#
|
#
|
||||||
# The full version, including alpha/beta/rc tags.
|
# The full version, including alpha/beta/rc tags.
|
||||||
# The short X.Y version.
|
# The short X.Y version.
|
||||||
version = release = "2.3.0.dev8"
|
version = release = "2.3.0.dev9"
|
||||||
|
|
||||||
import sys, os
|
import sys, os
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -21,4 +21,3 @@ need more examples or have questions. Also take a look at the :ref:`comprehensiv
|
||||||
markers.txt
|
markers.txt
|
||||||
pythoncollection.txt
|
pythoncollection.txt
|
||||||
nonpython.txt
|
nonpython.txt
|
||||||
newexamples.txt
|
|
||||||
|
|
|
||||||
|
|
@ -1,55 +1,23 @@
|
||||||
|
|
||||||
V5: changes to new resource/setup facilities
|
.. _`funcargcompare`:
|
||||||
|
|
||||||
|
=============================================================
|
||||||
|
pytest-2.3: reasoning for the new funcarg and setup functions
|
||||||
=============================================================
|
=============================================================
|
||||||
|
|
||||||
**Target audience**: Reading this document requires basic knowledge of
|
**Target audience**: Reading this document requires basic knowledge of
|
||||||
python testing, xUnit setup methods and the basic pytest funcarg mechanism,
|
python testing, xUnit setup methods and the (previous) basic pytest
|
||||||
see http://pytest.org/latest/funcargs.html
|
funcarg mechanism, see http://pytest.org/2.2.4/funcargs.html
|
||||||
|
|
||||||
|
|
||||||
**Changes**: This V5 draft is based on incorporating and thinking about
|
|
||||||
feedback on previous versions provided by Floris Bruynooghe, Carl Meyer,
|
|
||||||
Ronny Pfannschmidt and Samuele Pedroni. I have also now implemented it
|
|
||||||
which triggered a number of refinements as well. The main changes are:
|
|
||||||
|
|
||||||
* Collapse funcarg factory decorators into a single "@resource" one.
|
|
||||||
You can specify scopes and params with it. When using the decorator
|
|
||||||
the "pytest_funcarg__" prefix is not allowed and the old-style
|
|
||||||
``request`` object cannot be received.
|
|
||||||
|
|
||||||
* funcarg resource factories can now use funcargs themselves
|
|
||||||
|
|
||||||
* Drop setup/directory scope from this draft
|
|
||||||
|
|
||||||
* introduce a new @setup decorator similar to the @funcarg one
|
|
||||||
except that setup-markers cannot define parametriation themselves.
|
|
||||||
Instead they can easily depend on a parametrized funcarg (which
|
|
||||||
must not be visible at test function signatures).
|
|
||||||
|
|
||||||
* drop consideration of setup_X support for funcargs because
|
|
||||||
it is less flexible and probably causes more implementation
|
|
||||||
troubles than the current @setup approach which can share
|
|
||||||
a lot of logic with the @funcarg one.
|
|
||||||
|
|
||||||
* tests are grouped by parametrized funcargs and according to scope
|
|
||||||
(sounds like a small thing but is a big deal)
|
|
||||||
|
|
||||||
* make the new-style funcargs/setup use a "testcontext" object
|
|
||||||
which offers test context info and addfinalizer() methods but no
|
|
||||||
getfuncargvalue()/cached_setup()/applymarker anymore. Reason
|
|
||||||
being that getfuncargvalue()/cached_setup breaks other features
|
|
||||||
such as sorting by resource-scope and parametrization
|
|
||||||
|
|
||||||
|
|
||||||
.. currentmodule:: _pytest
|
.. currentmodule:: _pytest
|
||||||
|
|
||||||
Shortcomings of the previous pytest_funcarg__ mechanism
|
Shortcomings of the previous pytest_funcarg__ mechanism
|
||||||
---------------------------------------------------------
|
===========================================================
|
||||||
|
|
||||||
The previous funcarg mechanism calls a factory each time a
|
The pre pytest-2.3 funcarg mechanism calls a factory each time a
|
||||||
funcarg for a test function is testcontexted. If a factory wants
|
funcarg for a test function is required. If a factory wants to
|
||||||
t re-use a resource across different scopes, it often used
|
re-use a resource across different scopes, it often used
|
||||||
the ``testcontext.cached_setup()`` helper to manage caching of
|
the ``request.cached_setup()`` helper to manage caching of
|
||||||
resources. Here is a basic example how we could implement
|
resources. Here is a basic example how we could implement
|
||||||
a per-session Database object::
|
a per-session Database object::
|
||||||
|
|
||||||
|
|
@ -95,10 +63,9 @@ new facilities.
|
||||||
Direct scoping of funcarg factories
|
Direct scoping of funcarg factories
|
||||||
--------------------------------------------------------
|
--------------------------------------------------------
|
||||||
|
|
||||||
Instead of calling cached_setup(), you can decorate your factory
|
Instead of calling cached_setup(), you can use the :ref:`@pytest.factory <@pytest.factory>` decorator and directly state the scope::
|
||||||
to state its scope::
|
|
||||||
|
|
||||||
@pytest.mark.resource(scope="session")
|
@pytest.factory(scope="session")
|
||||||
def db(testcontext):
|
def db(testcontext):
|
||||||
# factory will only be invoked once per session -
|
# factory will only be invoked once per session -
|
||||||
db = DataBase()
|
db = DataBase()
|
||||||
|
|
@ -108,25 +75,21 @@ to state its scope::
|
||||||
This factory implementation does not need to call ``cached_setup()`` anymore
|
This factory implementation does not need to call ``cached_setup()`` anymore
|
||||||
because it will only be invoked once per session. Moreover, the
|
because it will only be invoked once per session. Moreover, the
|
||||||
``testcontext.addfinalizer()`` registers a finalizer according to the specified
|
``testcontext.addfinalizer()`` registers a finalizer according to the specified
|
||||||
resource scope on which the factory function is operating. With this new
|
resource scope on which the factory function is operating.
|
||||||
scoping, the still existing ``cached_setup()`` should be much less used
|
|
||||||
but will remain for compatibility reasons and for the case where you
|
|
||||||
still want to have your factory get called on a per-item basis.
|
|
||||||
|
|
||||||
|
|
||||||
Direct parametrization of funcarg resource factories
|
Direct parametrization of funcarg resource factories
|
||||||
----------------------------------------------------------
|
----------------------------------------------------------
|
||||||
|
|
||||||
.. note:: Implemented
|
|
||||||
|
|
||||||
Previously, funcarg factories could not directly cause parametrization.
|
Previously, funcarg factories could not directly cause parametrization.
|
||||||
You needed to specify a ``@parametrize`` or implement a ``pytest_generate_tests`` hook to perform parametrization, i.e. calling a test multiple times
|
You needed to specify a ``@parametrize`` decorator on your test function
|
||||||
with different value sets. pytest-2.X introduces a decorator for use
|
or implement a ``pytest_generate_tests`` hook to perform
|
||||||
on the factory itself::
|
parametrization, i.e. calling a test multiple times with different value
|
||||||
|
sets. pytest-2.3 introduces a decorator for use on the factory itself::
|
||||||
|
|
||||||
@pytest.mark.resource(params=["mysql", "pg"])
|
@pytest.factory(params=["mysql", "pg"])
|
||||||
def pytest_funcarg__db(testcontext):
|
def pytest_funcarg__db(testcontext):
|
||||||
...
|
... # use testcontext.param
|
||||||
|
|
||||||
Here the factory will be invoked twice (with the respective "mysql"
|
Here the factory will be invoked twice (with the respective "mysql"
|
||||||
and "pg" values set as ``testcontext.param`` attributes) and and all of
|
and "pg" values set as ``testcontext.param`` attributes) and and all of
|
||||||
|
|
@ -141,7 +104,7 @@ functions/classes were parametrized via
|
||||||
|
|
||||||
Of course it's perfectly fine to combine parametrization and scoping::
|
Of course it's perfectly fine to combine parametrization and scoping::
|
||||||
|
|
||||||
@pytest.mark.resource(scope="session", params=["mysql", "pg"])
|
@pytest.factory(scope="session", params=["mysql", "pg"])
|
||||||
def pytest_funcarg__db(testcontext):
|
def pytest_funcarg__db(testcontext):
|
||||||
if testcontext.param == "mysql":
|
if testcontext.param == "mysql":
|
||||||
db = MySQL()
|
db = MySQL()
|
||||||
|
|
@ -155,26 +118,23 @@ receiving the values created by the two respective invocations to the
|
||||||
factory function.
|
factory function.
|
||||||
|
|
||||||
|
|
||||||
No ``pytest_funcarg__`` prefix when using @resource decorator
|
No ``pytest_funcarg__`` prefix when using @factory decorator
|
||||||
-------------------------------------------------------------------
|
-------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
.. note:: Implemented
|
|
||||||
|
|
||||||
When using the ``@funcarg`` decorator the name of the function
|
When using the ``@funcarg`` decorator the name of the function
|
||||||
does not need to (and in fact cannot) use the ``pytest_funcarg__``
|
denotes the name under which the resource can be accessed as a function
|
||||||
naming::
|
argument::
|
||||||
|
|
||||||
@pytest.mark.resource
|
@pytest.factory()
|
||||||
def db(testcontext):
|
def db(testcontext):
|
||||||
...
|
...
|
||||||
|
|
||||||
The name under which the funcarg resource can be requested is ``db``.
|
The name under which the funcarg resource can be requested is ``db``.
|
||||||
|
|
||||||
You can also use the "old" non-decorator way of specifying funcarg factories
|
You can still use the "old" non-decorator way of specifying funcarg factories
|
||||||
aka::
|
aka::
|
||||||
|
|
||||||
def pytest_funcarg__db(testcontext):
|
def pytest_funcarg__db(request):
|
||||||
...
|
...
|
||||||
|
|
||||||
It is recommended to use the resource decorator, however.
|
It is recommended to use the resource decorator, however.
|
||||||
|
|
@ -183,8 +143,6 @@ It is recommended to use the resource decorator, however.
|
||||||
solving per-session setup / the new @setup marker
|
solving per-session setup / the new @setup marker
|
||||||
--------------------------------------------------------------
|
--------------------------------------------------------------
|
||||||
|
|
||||||
.. note:: Implemented, at least working for basic situations.
|
|
||||||
|
|
||||||
pytest for a long time offered a pytest_configure and a pytest_sessionstart
|
pytest for a long time offered a pytest_configure and a pytest_sessionstart
|
||||||
hook which are often used to setup global resources. This suffers from
|
hook which are often used to setup global resources. This suffers from
|
||||||
several problems:
|
several problems:
|
||||||
|
|
@ -201,7 +159,7 @@ several problems:
|
||||||
fact that this hook is actually used for reporting, in particular
|
fact that this hook is actually used for reporting, in particular
|
||||||
the test-header with platform/custom information.
|
the test-header with platform/custom information.
|
||||||
|
|
||||||
Moreover, it is today not easy to define a scoped setup from plugins or
|
Moreover, it was not easy to define a scoped setup from plugins or
|
||||||
conftest files other than to implement a ``pytest_runtest_setup()`` hook
|
conftest files other than to implement a ``pytest_runtest_setup()`` hook
|
||||||
and caring for scoping/caching yourself. And it's virtually impossible
|
and caring for scoping/caching yourself. And it's virtually impossible
|
||||||
to do this with parametrization as ``pytest_runtest_setup()`` is called
|
to do this with parametrization as ``pytest_runtest_setup()`` is called
|
||||||
|
|
@ -209,20 +167,17 @@ during test execution and parametrization happens at collection time.
|
||||||
|
|
||||||
It follows that pytest_configure/session/runtest_setup are often not
|
It follows that pytest_configure/session/runtest_setup are often not
|
||||||
appropriate for implementing common fixture needs. Therefore,
|
appropriate for implementing common fixture needs. Therefore,
|
||||||
pytest-2.X introduces a new :ref:`@pytest.setup` marker which takes
|
pytest-2.3 introduces a new :ref:`@pytest.setup <setup>` marker
|
||||||
an optional "scope" parameter.
|
for setup functions and it accepts an optional "scope" parameter.
|
||||||
|
|
||||||
See :ref:`new_setup` for examples.
|
See :ref:`setup` for more explanation and examples.
|
||||||
|
|
||||||
funcarg and setup discovery now happens at collection time
|
funcarg and setup discovery now happens at collection time
|
||||||
---------------------------------------------------------------------
|
---------------------------------------------------------------------
|
||||||
|
|
||||||
.. note::
|
pytest-2.3 takes care to discover funcarg factories and @setup methods
|
||||||
Partially implemented - collectonly shows no extra information however.
|
|
||||||
|
|
||||||
pytest-2.X takes care to discover funcarg factories and @setup methods
|
|
||||||
at collection time. This is more efficient especially for large test suites.
|
at collection time. This is more efficient especially for large test suites.
|
||||||
Moreover, a call to "py.test --collectonly" should be able to show
|
Moreover, a call to "py.test --collectonly" should be able to in the future
|
||||||
a lot of setup-information and thus presents a nice method to get an
|
show a lot of setup-information and thus presents a nice method to get an
|
||||||
overview of resource management in your project.
|
overview of resource management in your project.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,75 +1,110 @@
|
||||||
==============================================================
|
|
||||||
Injecting objects into test functions (funcargs)
|
|
||||||
==============================================================
|
|
||||||
|
|
||||||
.. currentmodule:: _pytest.python
|
|
||||||
|
|
||||||
|
|
||||||
|
.. _resources:
|
||||||
.. _`funcargs`:
|
.. _`funcargs`:
|
||||||
.. _`funcarg mechanism`:
|
.. _`funcarg mechanism`:
|
||||||
|
|
||||||
Dependency injection through function arguments
|
=======================================================
|
||||||
=================================================
|
funcargs: resource injection and parametrization
|
||||||
|
=======================================================
|
||||||
|
|
||||||
.. note::
|
.. note::
|
||||||
|
|
||||||
This section describes the pytest mechanisms prior
|
pytest-2.3 introduces major refinements to the original funcarg
|
||||||
to the pytest-2.3 release. If you haven't used these
|
mechanism introduced to pytest-2.0. While the old way
|
||||||
features yet, it makes more sense to stop here and read
|
remains fully supported, it is recommended to use the refined
|
||||||
:ref:`resources` instead.
|
mechanisms. See also the `compatibility notes`_ and the detailed
|
||||||
|
:ref:`reasoning for the new funcarg and setup functions <funcargcompare>`.
|
||||||
py.test lets you inject objects into test invocations and precisely
|
|
||||||
control their life cycle in relation to the overall test execution. Moreover,
|
|
||||||
you can run a test function multiple times injecting different objects.
|
|
||||||
|
|
||||||
The basic mechanism for injecting objects is also called the
|
|
||||||
*funcarg mechanism* because objects are ultimately injected
|
|
||||||
by calling a test function with it as an argument. Unlike the
|
|
||||||
classical xUnit approach *funcargs* relate more to `Dependency Injection`_
|
|
||||||
because they help to de-couple test code from objects required for
|
|
||||||
them to execute. At test writing time you do not need to care for the
|
|
||||||
details of how your required resources are constructed or if they
|
|
||||||
live through a function, class, module or session scope.
|
|
||||||
|
|
||||||
.. _`Dependency injection`: http://en.wikipedia.org/wiki/Dependency_injection
|
.. _`Dependency injection`: http://en.wikipedia.org/wiki/Dependency_injection
|
||||||
|
|
||||||
To create a value with which to call a test function a factory function
|
Introduction
|
||||||
is called which gets full access to the test function context and can
|
====================
|
||||||
register finalizers or invoke lifecycle-caching helpers. The factory
|
|
||||||
can be implemented in same test class or test module, in a
|
|
||||||
per-directory ``conftest.py`` file or in an external plugin. This
|
|
||||||
allows total de-coupling of test and setup code.
|
|
||||||
|
|
||||||
A test function may be invoked multiple times in which case we
|
py.test supports the injection of objects into test and setup functions
|
||||||
speak of :ref:`parametrized testing <parametrizing-tests>`. This can be
|
and flexibly control their life cycle in relation to the overall test
|
||||||
very useful if you want to test e.g. against different database backends
|
execution. Moreover, you can run a test function multiple times
|
||||||
or with multiple numerical arguments sets and want to reuse the same set
|
injecting different objects.
|
||||||
of test functions.
|
|
||||||
|
|
||||||
py.test comes with some :ref:`builtinresources` and there are some refined usages in the examples section.
|
The basic mechanism for injecting objects is called the *funcarg
|
||||||
|
mechanism* because objects are **injected** when a test or setup
|
||||||
|
function states it as an **argument**. The injected argument is
|
||||||
|
created by a call to a registered **funcarg factory**. This approach is
|
||||||
|
an example of `Dependency Injection`_ and helps to de-couple test code
|
||||||
|
from the setup of required objects: at test writing time you do not need
|
||||||
|
to care for the details of where and how your required resources are
|
||||||
|
constructed, if they are shared on a per-class, module or session basis,
|
||||||
|
or if your test function is invoked multiple times with differently
|
||||||
|
configured resource instances.
|
||||||
|
|
||||||
.. _funcarg:
|
When a test function is invoked multiple times with different arguments we
|
||||||
|
speak of **parametrized testing**. This is useful if you want to test
|
||||||
|
e.g. against different database backends or want to write a parametrized
|
||||||
|
test function, checking that certain inputs lead to certain outputs.
|
||||||
|
You can parametrize funcarg factories, parametrize test function
|
||||||
|
arguments or even implement your own parametrization scheme through a
|
||||||
|
plugin hook.
|
||||||
|
|
||||||
Basic injection example
|
pytest additionally offers powerful xunit-style :ref:`setup functions <setup
|
||||||
--------------------------------
|
functions>` for the cases where you need to create implicit test state
|
||||||
|
that is not passed explicitely to test functions.
|
||||||
|
|
||||||
Let's look at a simple self-contained test module::
|
Concretely, there are three main means of funcarg management:
|
||||||
|
|
||||||
|
* a `@pytest.factory`_ marker to define resource factories,
|
||||||
|
their scoping and parametrization. Factories can themselves
|
||||||
|
receive resources through their function arguments, easing
|
||||||
|
the setup of `interdependent resources`_. Factories can use
|
||||||
|
the special `testcontext`_ object to access details from where
|
||||||
|
the factory or setup function is called and for registering finalizers.
|
||||||
|
|
||||||
|
* a `@pytest.mark.parametrize`_ marker for executing test functions
|
||||||
|
multiple times with different argument sets,
|
||||||
|
|
||||||
|
* a `pytest_generate_tests`_ plugin hook marker for implementing
|
||||||
|
your parametrization for a test function which may depend on
|
||||||
|
command line options, class/module attributes etc.
|
||||||
|
|
||||||
|
Apart from making it easy to manage your own test resources
|
||||||
|
pytest also comes with some :ref:`builtinresources` which
|
||||||
|
you can use without defining them yourself. Third-party plugins
|
||||||
|
offer yet more domain-specific funcarg resources (for example the
|
||||||
|
`pytest-django plugin <http://pypi.python.org/pypi/pytest-django>`_) so
|
||||||
|
that after plugin installation you can simply use them in
|
||||||
|
your test and setup functions. This all contributes to high
|
||||||
|
re-useability of test resource management and goes far beyond what can
|
||||||
|
be done with the classical xUnit style approach which encodes resource
|
||||||
|
setup statically into the test source code, leading to duplicate and
|
||||||
|
hard-to change fixtures.
|
||||||
|
|
||||||
|
.. _`@pytest.factory`:
|
||||||
|
|
||||||
|
``@pytest.factory``: Creating parametrized, scoped resources
|
||||||
|
=====================================================================
|
||||||
|
|
||||||
|
Basic funcarg injection example
|
||||||
|
-----------------------------------------------------------
|
||||||
|
|
||||||
|
Let's look at a simple self-contained test module using a factory
|
||||||
|
and a funcarg::
|
||||||
|
|
||||||
# content of ./test_simplefactory.py
|
# content of ./test_simplefactory.py
|
||||||
def pytest_funcarg__myfuncarg(request):
|
import pytest
|
||||||
|
|
||||||
|
@pytest.factory()
|
||||||
|
def myfuncarg():
|
||||||
return 42
|
return 42
|
||||||
|
|
||||||
def test_function(myfuncarg):
|
def test_function(myfuncarg):
|
||||||
assert myfuncarg == 17
|
assert myfuncarg == 17
|
||||||
|
|
||||||
This test function needs an injected object named ``myfuncarg``.
|
Here, the ``test_function`` needs an object named ``myfuncarg`` and thus
|
||||||
py.test will automatically discover and call the ``pytest_funcarg__myfuncarg``
|
py.test will discover and call the ``@pytest.factory`` marked ``myfuncarg``
|
||||||
factory. Running the test looks like this::
|
factory function. Running the tests looks like this::
|
||||||
|
|
||||||
$ py.test test_simplefactory.py
|
$ py.test test_simplefactory.py
|
||||||
=========================== test session starts ============================
|
=========================== test session starts ============================
|
||||||
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev2
|
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev8
|
||||||
plugins: xdist, bugzilla, cache, oejskit, pep8, cov
|
plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov
|
||||||
collecting ... collected 1 items
|
collecting ... collected 1 items
|
||||||
|
|
||||||
test_simplefactory.py F
|
test_simplefactory.py F
|
||||||
|
|
@ -83,7 +118,7 @@ factory. Running the test looks like this::
|
||||||
> assert myfuncarg == 17
|
> assert myfuncarg == 17
|
||||||
E assert 42 == 17
|
E assert 42 == 17
|
||||||
|
|
||||||
test_simplefactory.py:5: AssertionError
|
test_simplefactory.py:8: AssertionError
|
||||||
========================= 1 failed in 0.02 seconds =========================
|
========================= 1 failed in 0.02 seconds =========================
|
||||||
|
|
||||||
This shows that the test function was called with a ``myfuncarg``
|
This shows that the test function was called with a ``myfuncarg``
|
||||||
|
|
@ -93,14 +128,12 @@ how py.test comes to call the test function this way:
|
||||||
1. py.test :ref:`finds <test discovery>` the ``test_function`` because
|
1. py.test :ref:`finds <test discovery>` the ``test_function`` because
|
||||||
of the ``test_`` prefix. The test function needs a function argument
|
of the ``test_`` prefix. The test function needs a function argument
|
||||||
named ``myfuncarg``. A matching factory function is discovered by
|
named ``myfuncarg``. A matching factory function is discovered by
|
||||||
looking for the name ``pytest_funcarg__myfuncarg``.
|
looking for a factory function named ``myfuncarg``.
|
||||||
|
|
||||||
2. ``pytest_funcarg__myfuncarg(request)`` is called and
|
2. ``myfuncarg()`` is called to create a value ``42``.
|
||||||
returns the value for ``myfuncarg``.
|
|
||||||
|
|
||||||
3. the test function can now be called: ``test_function(42)``.
|
3. ``test_function(42)`` is now called and results in the above
|
||||||
This results in the above exception because of the assertion
|
reported exception because of the assertion mismatch.
|
||||||
mismatch.
|
|
||||||
|
|
||||||
Note that if you misspell a function argument or want
|
Note that if you misspell a function argument or want
|
||||||
to use one that isn't available, you'll see an error
|
to use one that isn't available, you'll see an error
|
||||||
|
|
@ -114,130 +147,517 @@ with a list of available function arguments.
|
||||||
|
|
||||||
to see available function arguments.
|
to see available function arguments.
|
||||||
|
|
||||||
The request object passed to factories
|
|
||||||
-----------------------------------------
|
|
||||||
|
|
||||||
Each funcarg factory receives a :py:class:`~_pytest.python.FuncargRequest` object which
|
Location independency of funcarg factories
|
||||||
provides methods to manage caching and finalization in the context of the
|
----------------------------------------------------
|
||||||
test invocation as well as several attributes of the the underlying test item. In fact, as of version pytest-2.3, the request API is implemented on all Item
|
|
||||||
objects and therefore the request object has general :py:class:`Node attributes and methods <_pytest.main.Node>` attributes. This is a backward compatible
|
If during implementing your tests you realize that you
|
||||||
change so no changes are neccessary for pre-2.3 funcarg factories.
|
want to use a factory from multiple test files you can move it
|
||||||
|
to a :ref:`conftest.py <conftest.py>` file or even separately installable
|
||||||
|
:ref:`plugins <plugins>` without changing test code. The discovery of
|
||||||
|
funcarg factories starts at test classes, then test modules, then
|
||||||
|
``conftest.py`` files and finally builtin and 3-rd party plugins.
|
||||||
|
|
||||||
|
Creating and using a session-shared funcarg
|
||||||
|
-----------------------------------------------------------------
|
||||||
|
|
||||||
|
.. regendoc:wipe
|
||||||
|
|
||||||
|
.. versionadded:: 2.3
|
||||||
|
|
||||||
|
The `@pytest.factory`_ marker allows to
|
||||||
|
|
||||||
|
* mark a function as a factory for resources, useable by test and setup
|
||||||
|
functions
|
||||||
|
|
||||||
|
* define parameters in order to run tests multiple times with
|
||||||
|
different resource instances
|
||||||
|
|
||||||
|
* declare a scope which determines the level of caching, i.e. how often
|
||||||
|
the factory will be called. Valid scopes are ``session``, ``module``,
|
||||||
|
``class`` and ``function``.
|
||||||
|
|
||||||
|
Here is a simple example of a factory creating a shared ``smtplib.SMTP``
|
||||||
|
connection resource which test functions then may use across the whole
|
||||||
|
test session::
|
||||||
|
|
||||||
|
# content of conftest.py
|
||||||
|
import pytest
|
||||||
|
import smtplib
|
||||||
|
|
||||||
|
@pytest.factory(scope="session")
|
||||||
|
def smtp(testcontext):
|
||||||
|
return smtplib.SMTP("merlinux.eu")
|
||||||
|
|
||||||
|
The name of the factory is ``smtp`` (the factory function name)
|
||||||
|
and you can access its result by listing the name ``smtp`` as
|
||||||
|
an input parameter in any test or setup function::
|
||||||
|
|
||||||
|
# content of test_module.py
|
||||||
|
def test_ehlo(smtp):
|
||||||
|
response = smtp.ehlo()
|
||||||
|
assert response[0] == 250
|
||||||
|
assert "merlinux" in response[1]
|
||||||
|
assert 0 # for demo purposes
|
||||||
|
|
||||||
|
def test_noop(smtp):
|
||||||
|
response = smtp.noop()
|
||||||
|
assert response[0] == 250
|
||||||
|
assert 0 # for demo purposes
|
||||||
|
|
||||||
|
We deliberately insert failing ``assert 0`` statements in order to
|
||||||
|
inspect what is going on and can now run the tests::
|
||||||
|
|
||||||
|
$ py.test -q test_module.py
|
||||||
|
collecting ... collected 2 items
|
||||||
|
FF
|
||||||
|
================================= FAILURES =================================
|
||||||
|
________________________________ test_ehlo _________________________________
|
||||||
|
|
||||||
|
smtp = <smtplib.SMTP instance at 0x25e7b48>
|
||||||
|
|
||||||
|
def test_ehlo(smtp):
|
||||||
|
response = smtp.ehlo()
|
||||||
|
assert response[0] == 250
|
||||||
|
assert "merlinux" in response[1]
|
||||||
|
> assert 0 # for demo purposes
|
||||||
|
E assert 0
|
||||||
|
|
||||||
|
test_module.py:5: AssertionError
|
||||||
|
________________________________ test_noop _________________________________
|
||||||
|
|
||||||
|
smtp = <smtplib.SMTP instance at 0x25e7b48>
|
||||||
|
|
||||||
|
def test_noop(smtp):
|
||||||
|
response = smtp.noop()
|
||||||
|
assert response[0] == 250
|
||||||
|
> assert 0 # for demo purposes
|
||||||
|
E assert 0
|
||||||
|
|
||||||
|
test_module.py:10: AssertionError
|
||||||
|
2 failed in 0.20 seconds
|
||||||
|
|
||||||
|
you see the two ``assert 0`` failing and can also see that
|
||||||
|
the same (session-scoped) object was passed into the two test functions.
|
||||||
|
|
||||||
|
|
||||||
.. _`parametrizing tests, generalized`: http://tetamap.wordpress.com/2009/05/13/parametrizing-python-tests-generalized/
|
Parametrizing a session-shared funcarg resource
|
||||||
|
-----------------------------------------------------------------
|
||||||
|
|
||||||
.. _`blog post about the monkeypatch funcarg`: http://tetamap.wordpress.com/2009/03/03/monkeypatching-in-unit-tests-done-right/
|
Extending the previous example, we can flag the factory to create
|
||||||
.. _`xUnit style`: xunit_setup.html
|
two ``smtp`` values which will cause all tests using it to
|
||||||
|
run twice with two different values. The factory function gets
|
||||||
|
access to each parameter through the special `testcontext`_ object::
|
||||||
|
|
||||||
|
# content of conftest.py
|
||||||
|
import pytest
|
||||||
|
import smtplib
|
||||||
|
|
||||||
|
@pytest.factory(scope="session",
|
||||||
|
params=["merlinux.eu", "mail.python.org"])
|
||||||
|
def smtp(testcontext):
|
||||||
|
return smtplib.SMTP(testcontext.param)
|
||||||
|
|
||||||
|
The main change is the definition of a ``params`` list in the
|
||||||
|
``factory``-marker and the ``testcontext.param`` access within the
|
||||||
|
factory function. No test function code needs to change.
|
||||||
|
So let's just do another run::
|
||||||
|
|
||||||
|
$ py.test -q
|
||||||
|
collecting ... collected 4 items
|
||||||
|
FFFF
|
||||||
|
================================= FAILURES =================================
|
||||||
|
__________________________ test_ehlo[merlinux.eu] __________________________
|
||||||
|
|
||||||
|
smtp = <smtplib.SMTP instance at 0x2960050>
|
||||||
|
|
||||||
|
def test_ehlo(smtp):
|
||||||
|
response = smtp.ehlo()
|
||||||
|
assert response[0] == 250
|
||||||
|
assert "merlinux" in response[1]
|
||||||
|
> assert 0 # for demo purposes
|
||||||
|
E assert 0
|
||||||
|
|
||||||
|
test_module.py:5: AssertionError
|
||||||
|
__________________________ test_noop[merlinux.eu] __________________________
|
||||||
|
|
||||||
|
smtp = <smtplib.SMTP instance at 0x2960050>
|
||||||
|
|
||||||
|
def test_noop(smtp):
|
||||||
|
response = smtp.noop()
|
||||||
|
assert response[0] == 250
|
||||||
|
> assert 0 # for demo purposes
|
||||||
|
E assert 0
|
||||||
|
|
||||||
|
test_module.py:10: AssertionError
|
||||||
|
________________________ test_ehlo[mail.python.org] ________________________
|
||||||
|
|
||||||
|
smtp = <smtplib.SMTP instance at 0x296a128>
|
||||||
|
|
||||||
|
def test_ehlo(smtp):
|
||||||
|
response = smtp.ehlo()
|
||||||
|
assert response[0] == 250
|
||||||
|
> assert "merlinux" in response[1]
|
||||||
|
E assert 'merlinux' in 'mail.python.org\nSIZE 10240000\nETRN\nSTARTTLS\nENHANCEDSTATUSCODES\n8BITMIME\nDSN'
|
||||||
|
|
||||||
|
test_module.py:4: AssertionError
|
||||||
|
________________________ test_noop[mail.python.org] ________________________
|
||||||
|
|
||||||
|
smtp = <smtplib.SMTP instance at 0x296a128>
|
||||||
|
|
||||||
|
def test_noop(smtp):
|
||||||
|
response = smtp.noop()
|
||||||
|
assert response[0] == 250
|
||||||
|
> assert 0 # for demo purposes
|
||||||
|
E assert 0
|
||||||
|
|
||||||
|
test_module.py:10: AssertionError
|
||||||
|
4 failed in 6.00 seconds
|
||||||
|
|
||||||
|
We get four failures because we are running the two tests twice with
|
||||||
|
different ``smtp`` instantiations as defined on the factory.
|
||||||
|
Note that with the ``mail.python.org`` connection the second test
|
||||||
|
fails in ``test_ehlo`` because it expects a specific server string.
|
||||||
|
|
||||||
|
Adding a finalizer to the parametrized resource
|
||||||
|
--------------------------------------------------------
|
||||||
|
|
||||||
|
Further extending the ``smtp`` example, we now want to properly
|
||||||
|
close a smtp server connection after the last test using it
|
||||||
|
has been run. We can do this by calling the ``testcontext.addfinalizer()``
|
||||||
|
helper::
|
||||||
|
|
||||||
|
# content of conftest.py
|
||||||
|
import pytest
|
||||||
|
import smtplib
|
||||||
|
|
||||||
|
@pytest.factory(scope="session",
|
||||||
|
params=["merlinux.eu", "mail.python.org"])
|
||||||
|
def smtp(testcontext):
|
||||||
|
smtp = smtplib.SMTP(testcontext.param)
|
||||||
|
def fin():
|
||||||
|
print ("finalizing %s" % smtp)
|
||||||
|
smtp.close()
|
||||||
|
testcontext.addfinalizer(fin)
|
||||||
|
return smtp
|
||||||
|
|
||||||
|
We also add a print call and then run py.test without the default
|
||||||
|
output capturing and disabled traceback reporting::
|
||||||
|
|
||||||
|
$ py.test -s -q --tb=no
|
||||||
|
collecting ... collected 4 items
|
||||||
|
FFFF
|
||||||
|
4 failed in 5.62 seconds
|
||||||
|
finalizing <smtplib.SMTP instance at 0x2d60368>
|
||||||
|
finalizing <smtplib.SMTP instance at 0x2d68e60>
|
||||||
|
|
||||||
|
We see that the two ``smtp`` instances are finalized appropriately.
|
||||||
|
|
||||||
|
Looking at test collection without running tests
|
||||||
|
------------------------------------------------------
|
||||||
|
|
||||||
|
You can also look at what tests pytest collects without running them::
|
||||||
|
|
||||||
|
$ py.test --collectonly
|
||||||
|
=========================== test session starts ============================
|
||||||
|
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev8
|
||||||
|
plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov
|
||||||
|
collecting ... collected 4 items
|
||||||
|
<Module 'test_module.py'>
|
||||||
|
<Function 'test_ehlo[merlinux.eu]'>
|
||||||
|
<Function 'test_noop[merlinux.eu]'>
|
||||||
|
<Function 'test_ehlo[mail.python.org]'>
|
||||||
|
<Function 'test_noop[mail.python.org]'>
|
||||||
|
|
||||||
|
============================= in 0.02 seconds =============================
|
||||||
|
|
||||||
|
Note that pytest orders your test run by resource usage, minimizing
|
||||||
|
the number of active resources at any given time.
|
||||||
|
|
||||||
|
|
||||||
|
.. _`interdependent resources`:
|
||||||
|
|
||||||
|
Interdepdendent resources
|
||||||
|
----------------------------------------------------------
|
||||||
|
|
||||||
|
You can not only use funcargs in test functions but also in their factories
|
||||||
|
themselves. Extending the previous example we can instantiate another
|
||||||
|
object ``app`` and stick the ``smtp`` resource into it like this::
|
||||||
|
|
||||||
|
# content of test_appsetup.py
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
class App:
|
||||||
|
def __init__(self, smtp):
|
||||||
|
self.smtp = smtp
|
||||||
|
|
||||||
|
@pytest.factory(scope="module")
|
||||||
|
def app(smtp):
|
||||||
|
return App(smtp)
|
||||||
|
|
||||||
|
def test_exists(app):
|
||||||
|
assert app.smtp
|
||||||
|
|
||||||
|
Here we define the factory local to the test module and make it access
|
||||||
|
the ``smtp`` resource by listing it as an input parameter. Let's run this::
|
||||||
|
|
||||||
|
$ py.test -v test_appsetup.py
|
||||||
|
=========================== test session starts ============================
|
||||||
|
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev8 -- /home/hpk/venv/1/bin/python
|
||||||
|
cachedir: /home/hpk/tmp/doc-exec-414/.cache
|
||||||
|
plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov
|
||||||
|
collecting ... collected 2 items
|
||||||
|
|
||||||
|
test_appsetup.py:12: test_exists[merlinux.eu] PASSED
|
||||||
|
test_appsetup.py:12: test_exists[mail.python.org] PASSED
|
||||||
|
|
||||||
|
========================= 2 passed in 5.37 seconds =========================
|
||||||
|
|
||||||
|
Due to the parametrization of ``smtp`` the test will run twice with two
|
||||||
|
different ``App`` instances and respective smtp servers. There is no
|
||||||
|
need for the ``app`` factory to be aware of the parametrization. Note
|
||||||
|
that the ``app`` factory has a scope of ``module`` whereas it uses the
|
||||||
|
session-scoped ``smtp`` object: it is fine for factories to use
|
||||||
|
"broader" scoped resources but not the other way round. A
|
||||||
|
session-scoped resource could not use a module-scoped resource in a
|
||||||
|
meaningful way.
|
||||||
|
|
||||||
|
.. _`automatic per-resource grouping`:
|
||||||
|
|
||||||
|
Grouping tests by resource parameters
|
||||||
|
----------------------------------------------------------
|
||||||
|
|
||||||
|
.. regendoc: wipe
|
||||||
|
|
||||||
|
pytest minimizes the number of active resources during test runs.
|
||||||
|
If you have a parametrized resource, then all the tests using one
|
||||||
|
resource instance will execute one after another. Then any finalizers
|
||||||
|
are called for that resource instance and then the next parametrized
|
||||||
|
resource instance is created and its tests are run. Among other things,
|
||||||
|
this eases testing of applications which create and use global state.
|
||||||
|
|
||||||
|
The following example uses two parametrized funcargs, one of which is
|
||||||
|
scoped on a per-module basis, and all the functions perform ``print`` call
|
||||||
|
it to document the flow of calls::
|
||||||
|
|
||||||
|
# content of test_module.py
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
@pytest.factory(scope="module", params=["mod1", "mod2"])
|
||||||
|
def modarg(testcontext):
|
||||||
|
param = testcontext.param
|
||||||
|
print "create", param
|
||||||
|
def fin():
|
||||||
|
print "fin", param
|
||||||
|
testcontext.addfinalizer(fin)
|
||||||
|
return param
|
||||||
|
|
||||||
|
@pytest.factory(scope="function", params=[1,2])
|
||||||
|
def otherarg(testcontext):
|
||||||
|
return testcontext.param
|
||||||
|
|
||||||
|
def test_0(otherarg):
|
||||||
|
print " test0", otherarg
|
||||||
|
def test_1(modarg):
|
||||||
|
print " test1", modarg
|
||||||
|
def test_2(otherarg, modarg):
|
||||||
|
print " test2", otherarg, modarg
|
||||||
|
|
||||||
|
Let's run the tests in verbose mode and with looking at the print-output::
|
||||||
|
|
||||||
|
$ py.test -v -s test_module.py
|
||||||
|
=========================== test session starts ============================
|
||||||
|
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev8 -- /home/hpk/venv/1/bin/python
|
||||||
|
cachedir: /home/hpk/tmp/doc-exec-414/.cache
|
||||||
|
plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov
|
||||||
|
collecting ... collected 8 items
|
||||||
|
|
||||||
|
test_module.py:16: test_0[1] PASSED
|
||||||
|
test_module.py:16: test_0[2] PASSED
|
||||||
|
test_module.py:18: test_1[mod1] PASSED
|
||||||
|
test_module.py:20: test_2[1-mod1] PASSED
|
||||||
|
test_module.py:20: test_2[2-mod1] PASSED
|
||||||
|
test_module.py:18: test_1[mod2] PASSED
|
||||||
|
test_module.py:20: test_2[1-mod2] PASSED
|
||||||
|
test_module.py:20: test_2[2-mod2] PASSED
|
||||||
|
|
||||||
|
========================= 8 passed in 0.02 seconds =========================
|
||||||
|
test0 1
|
||||||
|
test0 2
|
||||||
|
create mod1
|
||||||
|
test1 mod1
|
||||||
|
test2 1 mod1
|
||||||
|
test2 2 mod1
|
||||||
|
fin mod1
|
||||||
|
create mod2
|
||||||
|
test1 mod2
|
||||||
|
test2 1 mod2
|
||||||
|
test2 2 mod2
|
||||||
|
fin mod2
|
||||||
|
|
||||||
|
You can see that the parametrized module-scoped ``modarg`` resource caused
|
||||||
|
a re-ordering of test execution. The finalizer for the ``mod1`` parametrized
|
||||||
|
resource was executed before the ``mod2`` resource was setup.
|
||||||
|
|
||||||
|
.. currentmodule:: _pytest.python
|
||||||
|
.. _`testcontext`:
|
||||||
|
|
||||||
|
``testcontext``: interacting with test context
|
||||||
|
---------------------------------------------------
|
||||||
|
|
||||||
|
The ``testcontext`` object may be used by `@pytest.factory`_ or
|
||||||
|
:ref:`@pytest.setup <setup>` marked functions. It contains
|
||||||
|
information relating to the test context within which the function
|
||||||
|
executes. Moreover, you can call
|
||||||
|
``testcontext.addfinalizer(myfinalizer)`` in order to trigger a call to
|
||||||
|
``myfinalizer`` after the last test in the test context has executed.
|
||||||
|
If passed to a parametrized factory ``testcontext.param`` will contain a
|
||||||
|
parameter (one value out of the ``params`` list specified with the
|
||||||
|
`@pytest.factory`_ marker).
|
||||||
|
|
||||||
|
.. autoclass:: _pytest.python.TestContext()
|
||||||
|
:members:
|
||||||
|
|
||||||
|
|
||||||
.. _`funcarg factory`:
|
|
||||||
.. _factory:
|
|
||||||
|
|
||||||
.. _`test generators`:
|
.. _`test generators`:
|
||||||
.. _`parametrizing-tests`:
|
.. _`parametrizing-tests`:
|
||||||
.. _`parametrized test functions`:
|
.. _`parametrized test functions`:
|
||||||
|
|
||||||
Parametrizing multiple calls to a test function
|
Parametrizing test functions
|
||||||
===========================================================
|
==========================================================================
|
||||||
|
|
||||||
You can parametrize multiple runs of the same test
|
While the `@pytest.factory`_ decorator allows to define parametrization
|
||||||
function by adding new test function calls with different
|
of funcarg resources at the factory-level, there are also means to
|
||||||
function argument values. Let's look at a simple self-contained
|
define parametrization at test functions directly:
|
||||||
example:
|
|
||||||
|
|
||||||
Basic generated test example
|
* `@pytest.mark.parametrize`_ to provide multiple argument sets
|
||||||
----------------------------
|
for a particular test function or class.
|
||||||
|
|
||||||
Let's consider a test module which uses the ``pytest_generate_tests``
|
* `pytest_generate_tests`_ to implement your own custom parametrization
|
||||||
hook to generate several calls to the same test function::
|
scheme or extensions.
|
||||||
|
|
||||||
|
.. _`@pytest.mark.parametrize`:
|
||||||
|
|
||||||
|
``@pytest.mark.parametrize``: parametrizing test functions
|
||||||
|
---------------------------------------------------------------------
|
||||||
|
|
||||||
|
.. regendoc: wipe
|
||||||
|
|
||||||
|
.. versionadded:: 2.2
|
||||||
|
|
||||||
|
The builtin ``pytest.mark.parametrize`` decorator enables
|
||||||
|
parametrization of arguments for a test function. Here is a typical example
|
||||||
|
of a test function that wants check for expected output given a certain input::
|
||||||
|
|
||||||
|
# content of test_expectation.py
|
||||||
|
import pytest
|
||||||
|
@pytest.mark.parametrize(("input", "expected"), [
|
||||||
|
("3+5", 8),
|
||||||
|
("2+4", 6),
|
||||||
|
("6*9", 42),
|
||||||
|
])
|
||||||
|
def test_eval(input, expected):
|
||||||
|
assert eval(input) == expected
|
||||||
|
|
||||||
|
The ``@parametrize`` decorator defines three different argument sets for the
|
||||||
|
two ``(input, output)`` arguments of ``test_eval`` function so the latter
|
||||||
|
will be run three times::
|
||||||
|
|
||||||
|
$ py.test -q
|
||||||
|
|
||||||
|
collecting ... collected 13 items
|
||||||
|
....F........
|
||||||
|
================================= FAILURES =================================
|
||||||
|
____________________________ test_eval[6*9-42] _____________________________
|
||||||
|
|
||||||
|
input = '6*9', expected = 42
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(("input", "expected"), [
|
||||||
|
("3+5", 8),
|
||||||
|
("2+4", 6),
|
||||||
|
("6*9", 42),
|
||||||
|
])
|
||||||
|
def test_eval(input, expected):
|
||||||
|
> assert eval(input) == expected
|
||||||
|
E assert 54 == 42
|
||||||
|
E + where 54 = eval('6*9')
|
||||||
|
|
||||||
|
test_expectation.py:8: AssertionError
|
||||||
|
1 failed, 12 passed in 5.76 seconds
|
||||||
|
|
||||||
|
As expected only one pair of input/output values fails the simple test function.
|
||||||
|
As usual you can see the ``input`` and ``output`` values in the traceback.
|
||||||
|
|
||||||
|
Note that there are various ways how you can mark groups of functions,
|
||||||
|
see :ref:`mark`.
|
||||||
|
|
||||||
|
|
||||||
|
.. _`pytest_generate_tests`:
|
||||||
|
|
||||||
|
Basic ``pytest_generate_tests`` example
|
||||||
|
---------------------------------------------
|
||||||
|
|
||||||
|
The ``pytest_generate_tests`` hook is typically used if you want
|
||||||
|
to go beyond what ``@pytest.mark.parametrize`` offers. For example,
|
||||||
|
let's say we want to execute a test with different computation
|
||||||
|
parameters and the parameter range shall be determined by a command
|
||||||
|
line argument. Let's first write a simple (do-nothing) computation test::
|
||||||
|
|
||||||
|
# content of test_compute.py
|
||||||
|
|
||||||
|
def test_compute(param1):
|
||||||
|
assert param1 < 4
|
||||||
|
|
||||||
|
Now we add a ``conftest.py`` file containing the addition of a
|
||||||
|
command line option and the generation of tests depending on
|
||||||
|
that option::
|
||||||
|
|
||||||
|
# content of conftest.py
|
||||||
|
|
||||||
|
def pytest_addoption(parser):
|
||||||
|
parser.addoption("--all", action="store_true",
|
||||||
|
help="run all combinations")
|
||||||
|
|
||||||
# content of test_example.py
|
|
||||||
def pytest_generate_tests(metafunc):
|
def pytest_generate_tests(metafunc):
|
||||||
if "numiter" in metafunc.funcargnames:
|
if 'param1' in metafunc.funcargnames:
|
||||||
metafunc.parametrize("numiter", range(10))
|
if metafunc.config.option.all:
|
||||||
|
end = 5
|
||||||
|
else:
|
||||||
|
end = 2
|
||||||
|
metafunc.parametrize("param1", range(end))
|
||||||
|
|
||||||
def test_func(numiter):
|
This means that we only run two tests if no option is passed::
|
||||||
assert numiter < 9
|
|
||||||
|
|
||||||
Running this will generate ten invocations of ``test_func`` passing in each of the items in the list of ``range(10)``::
|
$ py.test -q test_compute.py
|
||||||
|
collecting ... collected 2 items
|
||||||
|
..
|
||||||
|
2 passed in 0.02 seconds
|
||||||
|
|
||||||
$ py.test test_example.py
|
And we run five tests if we add the ``--all`` option::
|
||||||
=========================== test session starts ============================
|
|
||||||
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev2
|
|
||||||
plugins: xdist, bugzilla, cache, oejskit, pep8, cov
|
|
||||||
collecting ... collected 10 items
|
|
||||||
|
|
||||||
test_example.py .........F
|
|
||||||
|
|
||||||
|
$ py.test -q --all
|
||||||
|
collecting ... collected 5 items
|
||||||
|
....F
|
||||||
================================= FAILURES =================================
|
================================= FAILURES =================================
|
||||||
_______________________________ test_func[9] _______________________________
|
_____________________________ test_compute[4] ______________________________
|
||||||
|
|
||||||
numiter = 9
|
param1 = 4
|
||||||
|
|
||||||
def test_func(numiter):
|
def test_compute(param1):
|
||||||
> assert numiter < 9
|
> assert param1 < 4
|
||||||
E assert 9 < 9
|
E assert 4 < 4
|
||||||
|
|
||||||
test_example.py:6: AssertionError
|
test_compute.py:3: AssertionError
|
||||||
==================== 1 failed, 9 passed in 0.03 seconds ====================
|
1 failed, 4 passed in 0.03 seconds
|
||||||
|
|
||||||
Obviously, only when ``numiter`` has the value of ``9`` does the test fail. Note that the ``pytest_generate_tests(metafunc)`` hook is called during
|
As expected when running the full range of ``param1`` values
|
||||||
the test collection phase which is separate from the actual test running.
|
we'll get an error on the last one.
|
||||||
Let's just look at what is collected::
|
|
||||||
|
|
||||||
$ py.test --collectonly test_example.py
|
|
||||||
=========================== test session starts ============================
|
|
||||||
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev2
|
|
||||||
plugins: xdist, bugzilla, cache, oejskit, pep8, cov
|
|
||||||
collecting ... collected 10 items
|
|
||||||
<Module 'test_example.py'>
|
|
||||||
<Function 'test_func[0]'>
|
|
||||||
<Function 'test_func[1]'>
|
|
||||||
<Function 'test_func[2]'>
|
|
||||||
<Function 'test_func[3]'>
|
|
||||||
<Function 'test_func[4]'>
|
|
||||||
<Function 'test_func[5]'>
|
|
||||||
<Function 'test_func[6]'>
|
|
||||||
<Function 'test_func[7]'>
|
|
||||||
<Function 'test_func[8]'>
|
|
||||||
<Function 'test_func[9]'>
|
|
||||||
|
|
||||||
============================= in 0.02 seconds =============================
|
|
||||||
|
|
||||||
If you want to select only the run with the value ``7`` you could do::
|
|
||||||
|
|
||||||
$ py.test -v -k 7 test_example.py # or -k test_func[7]
|
|
||||||
=========================== test session starts ============================
|
|
||||||
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev2 -- /home/hpk/venv/1/bin/python
|
|
||||||
cachedir: /home/hpk/tmp/doc-exec-271/.cache
|
|
||||||
plugins: xdist, bugzilla, cache, oejskit, pep8, cov
|
|
||||||
collecting ... collected 10 items
|
|
||||||
|
|
||||||
test_example.py:5: test_func[0] PASSED
|
|
||||||
test_example.py:5: test_func[1] PASSED
|
|
||||||
test_example.py:5: test_func[2] PASSED
|
|
||||||
test_example.py:5: test_func[3] PASSED
|
|
||||||
test_example.py:5: test_func[4] PASSED
|
|
||||||
test_example.py:5: test_func[5] PASSED
|
|
||||||
test_example.py:5: test_func[6] PASSED
|
|
||||||
test_example.py:5: test_func[7] PASSED
|
|
||||||
test_example.py:5: test_func[8] PASSED
|
|
||||||
test_example.py:5: test_func[9] FAILED
|
|
||||||
|
|
||||||
================================= FAILURES =================================
|
|
||||||
_______________________________ test_func[9] _______________________________
|
|
||||||
|
|
||||||
numiter = 9
|
|
||||||
|
|
||||||
def test_func(numiter):
|
|
||||||
> assert numiter < 9
|
|
||||||
E assert 9 < 9
|
|
||||||
|
|
||||||
test_example.py:6: AssertionError
|
|
||||||
==================== 1 failed, 9 passed in 0.03 seconds ====================
|
|
||||||
|
|
||||||
You might want to look at :ref:`more parametrization examples <paramexamples>`.
|
You might want to look at :ref:`more parametrization examples <paramexamples>`.
|
||||||
|
|
||||||
|
|
||||||
.. _`metafunc object`:
|
.. _`metafunc object`:
|
||||||
|
|
||||||
The **metafunc** object
|
The **metafunc** object
|
||||||
|
|
@ -260,3 +680,52 @@ in the class or module where a test function is defined:
|
||||||
|
|
||||||
.. automethod:: Metafunc.parametrize
|
.. automethod:: Metafunc.parametrize
|
||||||
.. automethod:: Metafunc.addcall(funcargs=None,id=_notexists,param=_notexists)
|
.. automethod:: Metafunc.addcall(funcargs=None,id=_notexists,param=_notexists)
|
||||||
|
|
||||||
|
|
||||||
|
.. regendoc:wipe
|
||||||
|
|
||||||
|
|
||||||
|
.. _`compatibility notes`:
|
||||||
|
|
||||||
|
.. _`funcargscompat`:
|
||||||
|
|
||||||
|
Compatibility notes
|
||||||
|
============================================================
|
||||||
|
|
||||||
|
**Funcargs** were originally introduced to pytest-2.0. In pytest-2.3
|
||||||
|
the mechanism were extended and refined. Here are some related notes:
|
||||||
|
|
||||||
|
* previously funcarg factories were specified with a special
|
||||||
|
``pytest_funcarg__NAME`` prefix instead of using the
|
||||||
|
``@pytest.factory`` decorator.
|
||||||
|
|
||||||
|
* Factories received a `request`_ object which managed caching through
|
||||||
|
``request.cached_setup()`` calls and allowed using other funcargs via
|
||||||
|
``request.getfuncargvalue()`` calls. These intricate APIs made it hard
|
||||||
|
to do proper parametrization and implement resource caching. The
|
||||||
|
new ``@pytest.factory`` decorator allows to simply declare the scope
|
||||||
|
and let pytest figure things out for you.
|
||||||
|
|
||||||
|
* if you used parametrization and funcarg factories which made use of
|
||||||
|
``request.cached_setup()`` it is recommeneded to invest a few minutes
|
||||||
|
and simplify your funcarg factory code to use the `@pytest.factory`_
|
||||||
|
decorator instead. This will also allow to take advantage of
|
||||||
|
the `automatic per-resource grouping`_ of tests.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
Throughout the pytest documents the ``pytest_funcarg__NAME`` way of
|
||||||
|
defining a funcarg factory is often termed "old-style". Their
|
||||||
|
use remains fully supported and existing code using it should run
|
||||||
|
unmodified.
|
||||||
|
|
||||||
|
.. _request:
|
||||||
|
|
||||||
|
The request object passed to old-style factories
|
||||||
|
-----------------------------------------------------------------
|
||||||
|
|
||||||
|
Old-style funcarg factory definitions can receive a :py:class:`~_pytest.python.FuncargRequest` object which
|
||||||
|
provides methods to manage caching and finalization in the context of the
|
||||||
|
test invocation as well as several attributes of the the underlying test item.
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -27,7 +27,7 @@ Welcome to pytest!
|
||||||
|
|
||||||
- (new in 2.3) :ref:`easy test resource management, scoping and
|
- (new in 2.3) :ref:`easy test resource management, scoping and
|
||||||
parametrization <resources>`
|
parametrization <resources>`
|
||||||
- (new in 2.3) :ref:`xunitsetup`.
|
- (new in 2.3) :ref:`setup functions`.
|
||||||
- (new in 2.2) :ref:`durations`
|
- (new in 2.2) :ref:`durations`
|
||||||
- (much improved in 2.2) :ref:`marking and test selection <mark>`
|
- (much improved in 2.2) :ref:`marking and test selection <mark>`
|
||||||
- (improved in 2.2) :ref:`parametrized test functions <parametrized test functions>`
|
- (improved in 2.2) :ref:`parametrized test functions <parametrized test functions>`
|
||||||
|
|
|
||||||
|
|
@ -1,487 +0,0 @@
|
||||||
|
|
||||||
.. _resources:
|
|
||||||
|
|
||||||
=======================================================
|
|
||||||
test resource injection and parametrization
|
|
||||||
=======================================================
|
|
||||||
|
|
||||||
.. _`Dependency injection`: http://en.wikipedia.org/wiki/Dependency_injection
|
|
||||||
|
|
||||||
.. versionadded: 2.3
|
|
||||||
|
|
||||||
pytest offers very flexible means for managing test resources and
|
|
||||||
test parametrization.
|
|
||||||
|
|
||||||
The pytest resource management mechanism is an example of `Dependency
|
|
||||||
Injection`_ because test and :ref:`setup functions <xunitsetup>` receive
|
|
||||||
resources simply by stating them as an input argument. Therefore and
|
|
||||||
also for historic reasons, they are often called **funcargs**. At test
|
|
||||||
writing time you typically do not need to care for the details of how
|
|
||||||
your required resources are constructed, if they live through a
|
|
||||||
function, class, module or session scope or if the test will be called
|
|
||||||
multiple times with different resource instances.
|
|
||||||
|
|
||||||
To create a value with which to call a test function a resource factory
|
|
||||||
function is called which gets full access to the test context and can
|
|
||||||
register finalizers which are to be run after the last test in that context
|
|
||||||
finished. Resource factories can be implemented in same test class or
|
|
||||||
test module, in a per-directory ``conftest.py`` file or in an external
|
|
||||||
plugin. This allows **total de-coupling of test and setup code**,
|
|
||||||
lowering the cost of refactoring.
|
|
||||||
|
|
||||||
A test function may be invoked multiple times in which case we
|
|
||||||
speak of parametrization. You can parametrize resources or parametrize
|
|
||||||
test function arguments directly or even implement your own parametrization
|
|
||||||
scheme through a plugin hook.
|
|
||||||
|
|
||||||
A resource has a **name** under which test and setup functions
|
|
||||||
can access it by listing it as an input argument. Due to this and
|
|
||||||
also for historic reasons, resources are often called **funcargs**.
|
|
||||||
A resource is created by a factory which can be flagged with a **scope**
|
|
||||||
to only create resources on a per-class/per-module/per-session basis
|
|
||||||
instead of the default per-function scope.
|
|
||||||
|
|
||||||
Concretely, there are three means of resource and parametrization management:
|
|
||||||
|
|
||||||
* a `@pytest.factory`_ marker to define resource factories,
|
|
||||||
their scoping and parametrization. Factories can themselves
|
|
||||||
receive resources through their function arguments, easing
|
|
||||||
the setup of interdependent resources. They can also use
|
|
||||||
the special `testcontext`_ object to access details n which
|
|
||||||
the factory/setup is called and for registering finalizers.
|
|
||||||
|
|
||||||
* a `@pytest.mark.parametrize`_ marker for executing test functions
|
|
||||||
multiple times with different parameter sets
|
|
||||||
|
|
||||||
* a `pytest_generate_tests`_ plugin hook marker for implementing
|
|
||||||
your parametrization for a test function which may depend on
|
|
||||||
command line options, class/module attributes etc.
|
|
||||||
|
|
||||||
Finally, pytest comes with some :ref:`builtinresources` which
|
|
||||||
you can use without defining them yourself. Moreover, third-party
|
|
||||||
plugins offer their own resources so that after installation
|
|
||||||
you can simply use them in your test and setup functions.
|
|
||||||
|
|
||||||
.. _`@pytest.factory`:
|
|
||||||
|
|
||||||
``@pytest.factory``: Creating parametrized, scoped resources
|
|
||||||
-----------------------------------------------------------------
|
|
||||||
|
|
||||||
.. regendoc:wipe
|
|
||||||
|
|
||||||
.. versionadded:: 2.3
|
|
||||||
|
|
||||||
The `@pytest.factory`_ marker allows to
|
|
||||||
|
|
||||||
* mark a function as a factory for resources, useable by test and setup functions
|
|
||||||
* define parameters in order to run tests multiple times with different
|
|
||||||
resource instances
|
|
||||||
* set a scope which determines the level of caching, i.e. how often
|
|
||||||
the factory will be called. Valid scopes are ``session``, ``module``,
|
|
||||||
``class`` and ``function``.
|
|
||||||
|
|
||||||
Here is a simple example of a factory creating a shared ``smtplib.SMTP``
|
|
||||||
connection resource which test functions then may use across the whole
|
|
||||||
test session::
|
|
||||||
|
|
||||||
# content of conftest.py
|
|
||||||
import pytest
|
|
||||||
import smtplib
|
|
||||||
|
|
||||||
@pytest.factory(scope="session")
|
|
||||||
def smtp(testcontext):
|
|
||||||
smtp = smtplib.SMTP("merlinux.eu")
|
|
||||||
testcontext.addfinalizer(smtp.close)
|
|
||||||
return smtp
|
|
||||||
|
|
||||||
The name of the resource is ``smtp`` (the factory function name)
|
|
||||||
and you can now access the ``smtp`` resource by listing it as
|
|
||||||
an input parameter in any test function below the directory where
|
|
||||||
``conftest.py`` is located::
|
|
||||||
|
|
||||||
# content of test_module.py
|
|
||||||
def test_ehlo(smtp):
|
|
||||||
response = smtp.ehlo()
|
|
||||||
assert response[0] == 250
|
|
||||||
assert "merlinux" in response[1]
|
|
||||||
assert 0 # for demo purposes
|
|
||||||
|
|
||||||
def test_noop(smtp):
|
|
||||||
response = smtp.noop()
|
|
||||||
assert response[0] == 250
|
|
||||||
assert 0 # for demo purposes
|
|
||||||
|
|
||||||
If you run the tests::
|
|
||||||
|
|
||||||
$ py.test -q
|
|
||||||
collecting ... collected 2 items
|
|
||||||
FF
|
|
||||||
================================= FAILURES =================================
|
|
||||||
________________________________ test_ehlo _________________________________
|
|
||||||
|
|
||||||
smtp = <smtplib.SMTP instance at 0x1c7c638>
|
|
||||||
|
|
||||||
def test_ehlo(smtp):
|
|
||||||
response = smtp.ehlo()
|
|
||||||
assert response[0] == 250
|
|
||||||
assert "merlinux" in response[1]
|
|
||||||
> assert 0 # for demo purposes
|
|
||||||
E assert 0
|
|
||||||
|
|
||||||
test_module.py:5: AssertionError
|
|
||||||
________________________________ test_noop _________________________________
|
|
||||||
|
|
||||||
smtp = <smtplib.SMTP instance at 0x1c7c638>
|
|
||||||
|
|
||||||
def test_noop(smtp):
|
|
||||||
response = smtp.noop()
|
|
||||||
assert response[0] == 250
|
|
||||||
> assert 0 # for demo purposes
|
|
||||||
E assert 0
|
|
||||||
|
|
||||||
test_module.py:10: AssertionError
|
|
||||||
2 failed in 0.18 seconds
|
|
||||||
|
|
||||||
you will see the two ``assert 0`` failing and can see that
|
|
||||||
the same (session-scoped) object was passed into the two test functions.
|
|
||||||
|
|
||||||
If you now want to test multiple servers you can simply parametrize
|
|
||||||
the ``smtp`` factory::
|
|
||||||
|
|
||||||
# content of conftest.py
|
|
||||||
import pytest
|
|
||||||
import smtplib
|
|
||||||
|
|
||||||
@pytest.factory(scope="session",
|
|
||||||
params=["merlinux.eu", "mail.python.org"])
|
|
||||||
def smtp(testcontext):
|
|
||||||
smtp = smtplib.SMTP(testcontext.param)
|
|
||||||
def fin():
|
|
||||||
smtp.close()
|
|
||||||
testcontext.addfinalizer(fin)
|
|
||||||
return smtp
|
|
||||||
|
|
||||||
The main change is the definition of a ``params`` list in the
|
|
||||||
``factory``-marker and the ``testcontext.param`` access within the
|
|
||||||
factory function. No test code needs to change. So let's just do another
|
|
||||||
run::
|
|
||||||
|
|
||||||
$ py.test -q
|
|
||||||
collecting ... collected 4 items
|
|
||||||
FFFF
|
|
||||||
================================= FAILURES =================================
|
|
||||||
__________________________ test_ehlo[merlinux.eu] __________________________
|
|
||||||
|
|
||||||
smtp = <smtplib.SMTP instance at 0x1d162d8>
|
|
||||||
|
|
||||||
def test_ehlo(smtp):
|
|
||||||
response = smtp.ehlo()
|
|
||||||
assert response[0] == 250
|
|
||||||
assert "merlinux" in response[1]
|
|
||||||
> assert 0 # for demo purposes
|
|
||||||
E assert 0
|
|
||||||
|
|
||||||
test_module.py:5: AssertionError
|
|
||||||
__________________________ test_noop[merlinux.eu] __________________________
|
|
||||||
|
|
||||||
smtp = <smtplib.SMTP instance at 0x1d162d8>
|
|
||||||
|
|
||||||
def test_noop(smtp):
|
|
||||||
response = smtp.noop()
|
|
||||||
assert response[0] == 250
|
|
||||||
> assert 0 # for demo purposes
|
|
||||||
E assert 0
|
|
||||||
|
|
||||||
test_module.py:10: AssertionError
|
|
||||||
________________________ test_ehlo[mail.python.org] ________________________
|
|
||||||
|
|
||||||
smtp = <smtplib.SMTP instance at 0x1d1f098>
|
|
||||||
|
|
||||||
def test_ehlo(smtp):
|
|
||||||
response = smtp.ehlo()
|
|
||||||
assert response[0] == 250
|
|
||||||
> assert "merlinux" in response[1]
|
|
||||||
E assert 'merlinux' in 'mail.python.org\nSIZE 10240000\nETRN\nSTARTTLS\nENHANCEDSTATUSCODES\n8BITMIME\nDSN'
|
|
||||||
|
|
||||||
test_module.py:4: AssertionError
|
|
||||||
________________________ test_noop[mail.python.org] ________________________
|
|
||||||
|
|
||||||
smtp = <smtplib.SMTP instance at 0x1d1f098>
|
|
||||||
|
|
||||||
def test_noop(smtp):
|
|
||||||
response = smtp.noop()
|
|
||||||
assert response[0] == 250
|
|
||||||
> assert 0 # for demo purposes
|
|
||||||
E assert 0
|
|
||||||
|
|
||||||
test_module.py:10: AssertionError
|
|
||||||
4 failed in 6.42 seconds
|
|
||||||
|
|
||||||
We get four failures because we are running the two tests twice with
|
|
||||||
different ``smtp`` instantiations as defined on the factory.
|
|
||||||
Note that with the ``mail.python.org`` connection the second test
|
|
||||||
fails in ``test_ehlo`` because it expects a specific server string.
|
|
||||||
|
|
||||||
You can also look at what tests pytest collects without running them::
|
|
||||||
|
|
||||||
$ py.test --collectonly
|
|
||||||
=========================== test session starts ============================
|
|
||||||
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev7
|
|
||||||
plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov
|
|
||||||
collecting ... collected 4 items
|
|
||||||
<Module 'test_module.py'>
|
|
||||||
<Function 'test_ehlo[merlinux.eu]'>
|
|
||||||
<Function 'test_noop[merlinux.eu]'>
|
|
||||||
<Function 'test_ehlo[mail.python.org]'>
|
|
||||||
<Function 'test_noop[mail.python.org]'>
|
|
||||||
|
|
||||||
============================= in 0.02 seconds =============================
|
|
||||||
|
|
||||||
Note that pytest orders your test run by resource usage, minimizing
|
|
||||||
the number of active resources at any given time.
|
|
||||||
|
|
||||||
|
|
||||||
Interdepdendent resources
|
|
||||||
----------------------------------------------------------
|
|
||||||
|
|
||||||
You can not only use resources in test functions but also in resource factories
|
|
||||||
themselves. Extending the previous example we can instantiate an application
|
|
||||||
object by sticking the ``smtp`` resource into it::
|
|
||||||
|
|
||||||
# content of test_appsetup.py
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
class App:
|
|
||||||
def __init__(self, smtp):
|
|
||||||
self.smtp = smtp
|
|
||||||
|
|
||||||
@pytest.factory(scope="module")
|
|
||||||
def app(smtp):
|
|
||||||
return App(smtp)
|
|
||||||
|
|
||||||
def test_exists(app):
|
|
||||||
assert app.smtp
|
|
||||||
|
|
||||||
Let's run this::
|
|
||||||
|
|
||||||
$ py.test -v test_appsetup.py
|
|
||||||
=========================== test session starts ============================
|
|
||||||
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev7 -- /home/hpk/venv/1/bin/python
|
|
||||||
cachedir: /home/hpk/tmp/doc-exec-398/.cache
|
|
||||||
plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov
|
|
||||||
collecting ... collected 2 items
|
|
||||||
|
|
||||||
test_appsetup.py:12: test_exists[merlinux.eu] PASSED
|
|
||||||
test_appsetup.py:12: test_exists[mail.python.org] PASSED
|
|
||||||
|
|
||||||
========================= 2 passed in 5.96 seconds =========================
|
|
||||||
|
|
||||||
Due to the parametrization of ``smtp`` the test will
|
|
||||||
run twice with two different ``App`` instances and respective smtp servers.
|
|
||||||
There is no need for the ``app`` factory to be aware of the parametrization.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Grouping tests by resource parameters
|
|
||||||
----------------------------------------------------------
|
|
||||||
|
|
||||||
.. regendoc: wipe
|
|
||||||
|
|
||||||
pytest minimizes the number of active resources during test runs.
|
|
||||||
If you have a parametrized resource, then all the tests using one
|
|
||||||
resource instance will execute one after another. Then any finalizers
|
|
||||||
are called for that resource instance and then the next parametrized
|
|
||||||
resource instance is created and its tests are run. Among other things,
|
|
||||||
this eases testing of applications which create and use global state.
|
|
||||||
|
|
||||||
The following example uses two parametrized funcargs, one of which is
|
|
||||||
scoped on a per-module basis::
|
|
||||||
|
|
||||||
# content of test_module.py
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
@pytest.factory(scope="module", params=["mod1", "mod2"])
|
|
||||||
def modarg(testcontext):
|
|
||||||
param = testcontext.param
|
|
||||||
print "create", param
|
|
||||||
def fin():
|
|
||||||
print "fin", param
|
|
||||||
testcontext.addfinalizer(fin)
|
|
||||||
return param
|
|
||||||
|
|
||||||
@pytest.factory(scope="function", params=[1,2])
|
|
||||||
def otherarg(testcontext):
|
|
||||||
return testcontext.param
|
|
||||||
|
|
||||||
def test_0(otherarg):
|
|
||||||
print " test0", otherarg
|
|
||||||
def test_1(modarg):
|
|
||||||
print " test1", modarg
|
|
||||||
def test_2(otherarg, modarg):
|
|
||||||
print " test2", otherarg, modarg
|
|
||||||
|
|
||||||
Let's run the tests in verbose mode and with looking at the print-output::
|
|
||||||
|
|
||||||
$ py.test -v -s
|
|
||||||
=========================== test session starts ============================
|
|
||||||
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev7 -- /home/hpk/venv/1/bin/python
|
|
||||||
cachedir: /home/hpk/tmp/doc-exec-398/.cache
|
|
||||||
plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov
|
|
||||||
collecting ... collected 8 items
|
|
||||||
|
|
||||||
test_module.py:16: test_0[1] PASSED
|
|
||||||
test_module.py:16: test_0[2] PASSED
|
|
||||||
test_module.py:18: test_1[mod1] PASSED
|
|
||||||
test_module.py:20: test_2[1-mod1] PASSED
|
|
||||||
test_module.py:20: test_2[2-mod1] PASSED
|
|
||||||
test_module.py:18: test_1[mod2] PASSED
|
|
||||||
test_module.py:20: test_2[1-mod2] PASSED
|
|
||||||
test_module.py:20: test_2[2-mod2] PASSED
|
|
||||||
|
|
||||||
========================= 8 passed in 0.03 seconds =========================
|
|
||||||
test0 1
|
|
||||||
test0 2
|
|
||||||
create mod1
|
|
||||||
test1 mod1
|
|
||||||
test2 1 mod1
|
|
||||||
test2 2 mod1
|
|
||||||
fin mod1
|
|
||||||
create mod2
|
|
||||||
test1 mod2
|
|
||||||
test2 1 mod2
|
|
||||||
test2 2 mod2
|
|
||||||
fin mod2
|
|
||||||
|
|
||||||
You can see that the parametrized module-scoped ``modarg`` resource caused
|
|
||||||
a re-ordering of test execution. The finalizer for the ``mod1`` parametrized
|
|
||||||
resource was executed before the ``mod2`` resource was setup.
|
|
||||||
|
|
||||||
.. currentmodule:: _pytest.python
|
|
||||||
.. _`testcontext`:
|
|
||||||
|
|
||||||
``testcontext``: interacting with test context
|
|
||||||
---------------------------------------------------
|
|
||||||
|
|
||||||
The ``testcontext`` object may be received by `@pytest.factory`_ or
|
|
||||||
`@pytest.setup`_ marked functions. It contains information relating
|
|
||||||
to the test context within which the function executes. Moreover, you
|
|
||||||
can call ``testcontext.addfinalizer(myfinalizer)`` in order to trigger
|
|
||||||
a call to ``myfinalizer`` after the last test in the test context has executed.
|
|
||||||
If passed to a parametrized factory ``testcontext.param`` will contain
|
|
||||||
a parameter (one value out of the ``params`` list specified with the
|
|
||||||
`@pytest.factory`_ marker).
|
|
||||||
|
|
||||||
.. autoclass:: _pytest.python.TestContext()
|
|
||||||
:members:
|
|
||||||
|
|
||||||
.. _`@pytest.mark.parametrize`:
|
|
||||||
|
|
||||||
``@pytest.mark.parametrize``: directly parametrizing test functions
|
|
||||||
----------------------------------------------------------------------------
|
|
||||||
|
|
||||||
.. versionadded:: 2.2
|
|
||||||
|
|
||||||
The builtin ``pytest.mark.parametrize`` decorator enables
|
|
||||||
parametrization of arguments for a test function. Here is an example
|
|
||||||
of a test function that wants check for expected output given a certain input::
|
|
||||||
|
|
||||||
# content of test_expectation.py
|
|
||||||
import pytest
|
|
||||||
@pytest.mark.parametrize(("input", "expected"), [
|
|
||||||
("3+5", 8),
|
|
||||||
("2+4", 6),
|
|
||||||
("6*9", 42),
|
|
||||||
])
|
|
||||||
def test_eval(input, expected):
|
|
||||||
assert eval(input) == expected
|
|
||||||
|
|
||||||
we parametrize two arguments of the test function so that the test
|
|
||||||
function is called three times. Let's run it::
|
|
||||||
|
|
||||||
$ py.test -q
|
|
||||||
collecting ... collected 11 items
|
|
||||||
..F........
|
|
||||||
================================= FAILURES =================================
|
|
||||||
____________________________ test_eval[6*9-42] _____________________________
|
|
||||||
|
|
||||||
input = '6*9', expected = 42
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(("input", "expected"), [
|
|
||||||
("3+5", 8),
|
|
||||||
("2+4", 6),
|
|
||||||
("6*9", 42),
|
|
||||||
])
|
|
||||||
def test_eval(input, expected):
|
|
||||||
> assert eval(input) == expected
|
|
||||||
E assert 54 == 42
|
|
||||||
E + where 54 = eval('6*9')
|
|
||||||
|
|
||||||
test_expectation.py:8: AssertionError
|
|
||||||
1 failed, 10 passed in 0.04 seconds
|
|
||||||
|
|
||||||
As expected only one pair of input/output values fails the simple test function.
|
|
||||||
|
|
||||||
Note that there are various ways how you can mark groups of functions,
|
|
||||||
see :ref:`mark`.
|
|
||||||
|
|
||||||
|
|
||||||
.. _`pytest_generate_tests`:
|
|
||||||
|
|
||||||
``pytest_generate_test``: implementing your own parametrization scheme
|
|
||||||
----------------------------------------------------------------------------
|
|
||||||
|
|
||||||
.. regendoc:wipe
|
|
||||||
|
|
||||||
Let's say we want to execute a test with different computation
|
|
||||||
parameters and the parameter range shall be determined by a command
|
|
||||||
line argument. Let's first write a simple (do-nothing) computation test::
|
|
||||||
|
|
||||||
# content of test_compute.py
|
|
||||||
|
|
||||||
def test_compute(param1):
|
|
||||||
assert param1 < 4
|
|
||||||
|
|
||||||
Now we add a test configuration like this::
|
|
||||||
|
|
||||||
# content of conftest.py
|
|
||||||
|
|
||||||
def pytest_addoption(parser):
|
|
||||||
parser.addoption("--all", action="store_true",
|
|
||||||
help="run all combinations")
|
|
||||||
|
|
||||||
def pytest_generate_tests(metafunc):
|
|
||||||
if 'param1' in metafunc.funcargnames:
|
|
||||||
if metafunc.config.option.all:
|
|
||||||
end = 5
|
|
||||||
else:
|
|
||||||
end = 2
|
|
||||||
metafunc.parametrize("param1", range(end))
|
|
||||||
|
|
||||||
This means that we only run 2 tests if we do not pass ``--all``::
|
|
||||||
|
|
||||||
$ py.test -q test_compute.py
|
|
||||||
collecting ... collected 2 items
|
|
||||||
..
|
|
||||||
2 passed in 0.03 seconds
|
|
||||||
|
|
||||||
We run only two computations, so we see two dots.
|
|
||||||
let's run the full monty::
|
|
||||||
|
|
||||||
$ py.test -q --all
|
|
||||||
collecting ... collected 5 items
|
|
||||||
....F
|
|
||||||
================================= FAILURES =================================
|
|
||||||
_____________________________ test_compute[4] ______________________________
|
|
||||||
|
|
||||||
param1 = 4
|
|
||||||
|
|
||||||
def test_compute(param1):
|
|
||||||
> assert param1 < 4
|
|
||||||
E assert 4 < 4
|
|
||||||
|
|
||||||
test_compute.py:3: AssertionError
|
|
||||||
1 failed, 4 passed in 0.03 seconds
|
|
||||||
|
|
||||||
As expected when running the full range of ``param1`` values
|
|
||||||
we'll get an error on the last one.
|
|
||||||
|
|
@ -1,5 +1,7 @@
|
||||||
.. _xunitsetup:
|
.. _xunitsetup:
|
||||||
.. _setup:
|
.. _setup:
|
||||||
|
.. _`setup functions`:
|
||||||
|
.. _`@pytest.setup`:
|
||||||
|
|
||||||
``@setup`` functions or: xunit on steroids
|
``@setup`` functions or: xunit on steroids
|
||||||
========================================================
|
========================================================
|
||||||
|
|
@ -18,19 +20,20 @@ it has finished. Unlike :ref:`injected resources <resources>` setup
|
||||||
functions work indirectly by causing global side effects or
|
functions work indirectly by causing global side effects or
|
||||||
setting test case attributes which test methods can then access.
|
setting test case attributes which test methods can then access.
|
||||||
|
|
||||||
pytest originally introduced in 2005 a fine-grained model of detecting
|
pytest originally introduced in 2005 a scope-specific model of detecting
|
||||||
setup and teardown functions on a per-module, class or function basis.
|
setup and teardown functions on a per-module, class or function basis.
|
||||||
The Python unittest module and nose have subsequently incorporated them.
|
The Python unittest package and nose have subsequently incorporated them.
|
||||||
This model remains supported as :ref:`old-style xunit`.
|
This model remains supported by pytest as :ref:`old-style xunit`.
|
||||||
|
|
||||||
With pytest-2.3 a new ``pytest.setup()`` decorator is introduced
|
Moreover, pytest-2.3 introduces a new ``pytest.setup()`` decorator
|
||||||
to mark functions as setup functions which:
|
to mark functions as setup functions which allow to implement everything
|
||||||
|
you can do with the old-style and much more. Specifically setup functions:
|
||||||
|
|
||||||
- can receive resources through funcargs,
|
- can receive :ref:`resources through funcargs <resources>`,
|
||||||
- fully interoperate with parametrized resources,
|
- fully interoperate with parametrized resources,
|
||||||
- can be defined in a plugin or conftest.py file and get called
|
- can be defined in a plugin or :ref:`conftest.py` file and get called
|
||||||
on a per-session, per-module, per-class or per-function basis,
|
on a per-session, per-module, per-class or per-function basis,
|
||||||
- can access the full :ref:`testcontext` for which the setup is called,
|
- can access the :ref:`testcontext <testcontext>` for which the setup is called,
|
||||||
- can precisely control teardown by registering one or multiple
|
- can precisely control teardown by registering one or multiple
|
||||||
teardown functions as soon as they have performed some actions
|
teardown functions as soon as they have performed some actions
|
||||||
which need undoing, eliminating the no need for a separate
|
which need undoing, eliminating the no need for a separate
|
||||||
|
|
@ -75,7 +78,7 @@ Our ``mydir`` setup function is executed on a per-function basis,
|
||||||
the default scope used by the ``pytest.setup`` decorator.
|
the default scope used by the ``pytest.setup`` decorator.
|
||||||
It accesses the ``tmpdir`` resource which provides a new empty
|
It accesses the ``tmpdir`` resource which provides a new empty
|
||||||
directory path object. The ``test_function2`` here checks that
|
directory path object. The ``test_function2`` here checks that
|
||||||
it executes with a fresh directory and specifically
|
it executes with a fresh directory and that it
|
||||||
does not see the previously created ``anotherfile``. We can
|
does not see the previously created ``anotherfile``. We can
|
||||||
thus expect two passing tests::
|
thus expect two passing tests::
|
||||||
|
|
||||||
|
|
@ -115,6 +118,11 @@ empty directory ahead of running a test.
|
||||||
test modules accessing a global resource
|
test modules accessing a global resource
|
||||||
-------------------------------------------------------
|
-------------------------------------------------------
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
Relying on `global state is considered bad programming practise <http://en.wikipedia.org/wiki/Global_variable>`_ but when you work with an application
|
||||||
|
that relies on it you often have no choice.
|
||||||
|
|
||||||
If you want test modules to access a global resource,
|
If you want test modules to access a global resource,
|
||||||
you can stick the resource to the module globals in
|
you can stick the resource to the module globals in
|
||||||
a per-module setup function. We use a :ref:`resource factory
|
a per-module setup function. We use a :ref:`resource factory
|
||||||
|
|
|
||||||
|
|
@ -4,10 +4,13 @@
|
||||||
Old-style xunit-style setup
|
Old-style xunit-style setup
|
||||||
========================================
|
========================================
|
||||||
|
|
||||||
This section describes the old way how you can implement setup and
|
.. note::
|
||||||
teardown on a per-module/class/function basis. It remains fully
|
|
||||||
supported but it is recommended to rather use :ref:`@setup functions <setup>`
|
This section describes the old way how you can implement setup and
|
||||||
or :ref:`injected resources <resources>` for implementing your setup needs.
|
teardown on a per-module/class/function basis. It remains fully
|
||||||
|
supported but it is recommended to rather use :ref:`@setup functions
|
||||||
|
<setup>` or :ref:`injected resources <resources>` for implementing your
|
||||||
|
setup needs.
|
||||||
|
|
||||||
Module level setup/teardown
|
Module level setup/teardown
|
||||||
--------------------------------------
|
--------------------------------------
|
||||||
Loading…
Reference in New Issue