refine documentation, move setup to own "setup" page and provide
some more examples. move old setup_module/... to xunit_old page.
This commit is contained in:
holger krekel
parent
5fd84c35dd
commit
ae241a5071
|
|
@ -25,7 +25,9 @@ Welcome to pytest!
|
|||
|
||||
- **supports functional testing and complex test setups**
|
||||
|
||||
- (new in 2.3) :ref:`easy test resource management and generalized xUnit setup <resources>`
|
||||
- (new in 2.3) :ref:`easy test resource management, scoping and
|
||||
parametrization <resources>`
|
||||
- (new in 2.3) :ref:`xunitsetup`.
|
||||
- (new in 2.2) :ref:`durations`
|
||||
- (much improved in 2.2) :ref:`marking and test selection <mark>`
|
||||
- (improved in 2.2) :ref:`parametrized test functions <parametrized test functions>`
|
||||
|
|
|
|||
|
|
@ -1,74 +1,66 @@
|
|||
|
||||
.. _resources:
|
||||
|
||||
test resource management and xUnit setup (on steroids)
|
||||
=======================================================
|
||||
test resource injection and parametrization
|
||||
=======================================================
|
||||
|
||||
.. _`Dependency injection`: http://en.wikipedia.org/wiki/Dependency_injection
|
||||
|
||||
.. versionadded: 2.3
|
||||
|
||||
pytest offers advanced resource parametrization and injection mechanisms
|
||||
including a fully integrated generalization of the popular xUnit
|
||||
setup-style methods. A resource is created by a ``@pytest.mark.factory``
|
||||
marked function and its name is the name of the function. A resource
|
||||
is injected into test or setup functions if they use the name
|
||||
in their signature. Therefore and also for historic reasons, resources are
|
||||
sometimes called "funcargs" because they ultimately appear as
|
||||
function arguments.
|
||||
|
||||
The pytest resource management and setup features are exposed through
|
||||
three decorators:
|
||||
|
||||
* a `@pytest.mark.factory`_ marker to define resource factories,
|
||||
their scoping and parametrization.
|
||||
|
||||
* a `@pytest.mark.setup`_ marker to define setup functions and their
|
||||
scoping.
|
||||
|
||||
* a `@pytest.mark.parametrize`_ marker for executing test functions
|
||||
multiple times with different parameter sets
|
||||
|
||||
Generally, resource factories and setup functions:
|
||||
|
||||
- can be defined in test modules, test classes, conftest.py files or
|
||||
in plugins.
|
||||
|
||||
- can themselves receive resources through their function arguments,
|
||||
simplifying the setup and use of interdependent resources.
|
||||
|
||||
- can use the special `testcontext`_ object for access to the
|
||||
context in which the factory/setup is called and for registering
|
||||
finalizers.
|
||||
|
||||
This document showcases these features through some basic examples.
|
||||
|
||||
Note that pytest also comes with some :ref:`builtinresources` which
|
||||
you can use without defining them yourself.
|
||||
|
||||
Background and terms
|
||||
---------------------------
|
||||
pytest offers very flexible means for managing test resources and
|
||||
test parametrization.
|
||||
|
||||
The pytest resource management mechanism is an example of `Dependency
|
||||
Injection`_ which helps to de-couple test code from resource
|
||||
instantiation code required for them to execute. At test writing time
|
||||
you typically do not need to care for the details of how your required
|
||||
resources are constructed, if they live through a function, class,
|
||||
module or session scope or if the test will be called multiple times
|
||||
with different resource instances.
|
||||
Injection`_ because test and :ref:`setup functions <xunitsetup>` receive
|
||||
resources simply by stating them as an input argument. Therefore and
|
||||
also for historic reasons, they are often called **funcargs**. At test
|
||||
writing time you typically do not need to care for the details of how
|
||||
your required resources are constructed, if they live through a
|
||||
function, class, module or session scope or if the test will be called
|
||||
multiple times with different resource instances.
|
||||
|
||||
To create a value with which to call a test function a resource factory
|
||||
function is called which gets full access to the test context and can
|
||||
register finalizers which are to be run after the last test in that context
|
||||
finished. Resource factories can be implemented in same test class or
|
||||
test module, in a per-directory ``conftest.py`` file or in an external plugin. This allows total de-coupling of test and setup code.
|
||||
test module, in a per-directory ``conftest.py`` file or in an external
|
||||
plugin. This allows **total de-coupling of test and setup code**,
|
||||
lowering the cost of refactoring.
|
||||
|
||||
A test function may be invoked multiple times in which case we
|
||||
speak of :ref:`parametrized testing <parametrizing-tests>`. This can be
|
||||
very useful if you want to test e.g. against different database backends
|
||||
or with multiple numerical arguments sets and want to reuse the same set
|
||||
of test functions.
|
||||
speak of parametrization. You can parametrize resources or parametrize
|
||||
test function arguments directly or even implement your own parametrization
|
||||
scheme through a plugin hook.
|
||||
|
||||
A resource has a **name** under which test and setup functions
|
||||
can access it by listing it as an input argument. Due to this and
|
||||
also for historic reasons, resources are often called **funcargs**.
|
||||
A resource is created by a factory which can be flagged with a **scope**
|
||||
to only create resources on a per-class/per-module/per-session basis
|
||||
instead of the default per-function scope.
|
||||
|
||||
Concretely, there are three means of resource and parametrization management:
|
||||
|
||||
* a `@pytest.mark.factory`_ marker to define resource factories,
|
||||
their scoping and parametrization. Factories can themselves
|
||||
receive resources through their function arguments, easing
|
||||
the setup of interdependent resources. They can also use
|
||||
the special `testcontext`_ object to access details n which
|
||||
the factory/setup is called and for registering finalizers.
|
||||
|
||||
* a `@pytest.mark.parametrize`_ marker for executing test functions
|
||||
multiple times with different parameter sets
|
||||
|
||||
* a `pytest_generate_tests`_ plugin hook marker for implementing
|
||||
your parametrization for a test function which may depend on
|
||||
command line options, class/module attributes etc.
|
||||
|
||||
Finally, pytest comes with some :ref:`builtinresources` which
|
||||
you can use without defining them yourself. Moreover, third-party
|
||||
plugins offer their own resources so that after installation
|
||||
you can simply use them in your test and setup functions.
|
||||
|
||||
.. _`@pytest.mark.factory`:
|
||||
|
||||
|
|
@ -81,11 +73,12 @@ of test functions.
|
|||
|
||||
The `@pytest.mark.factory`_ marker allows to
|
||||
|
||||
* mark a function as a factory for resources used by test and setup functions
|
||||
* define parametrization to run tests multiple times with different
|
||||
* mark a function as a factory for resources, useable by test and setup functions
|
||||
* define parameters in order to run tests multiple times with different
|
||||
resource instances
|
||||
* set a scope which determines the level of caching. valid scopes
|
||||
are ``session``, ``module``, ``class`` and ``function``.
|
||||
* set a scope which determines the level of caching, i.e. how often
|
||||
the factory will be called. Valid scopes are ``session``, ``module``,
|
||||
``class`` and ``function``.
|
||||
|
||||
Here is a simple example of a factory creating a shared ``smtplib.SMTP``
|
||||
connection resource which test functions then may use across the whole
|
||||
|
|
@ -248,12 +241,12 @@ Note that pytest orders your test run by resource usage, minimizing
|
|||
the number of active resources at any given time.
|
||||
|
||||
|
||||
Accessing resources from a factory function
|
||||
Interdepdendent resources
|
||||
----------------------------------------------------------
|
||||
|
||||
You can directly use resources as funcargs in resource factories.
|
||||
Extending the previous example we can instantiate an application
|
||||
object and stick the live ``smtp`` resource into it::
|
||||
You can not only use resources in test functions but also in resource factories
|
||||
themselves. Extending the previous example we can instantiate an application
|
||||
object by sticking the ``smtp`` resource into it::
|
||||
|
||||
# content of test_appsetup.py
|
||||
|
||||
|
|
@ -289,114 +282,6 @@ run twice with two different ``App`` instances and respective smtp servers.
|
|||
There is no need for the ``app`` factory to be aware of the parametrization.
|
||||
|
||||
|
||||
.. _`new_setup`:
|
||||
.. _`@pytest.mark.setup`:
|
||||
|
||||
``@pytest.mark.setup``: xUnit setup methods on steroids
|
||||
-----------------------------------------------------------------
|
||||
|
||||
.. regendoc:wipe
|
||||
|
||||
.. versionadded:: 2.3
|
||||
|
||||
The ``@pytest.mark.setup`` marker allows
|
||||
|
||||
* to define setup-functions close to test code or in conftest.py files
|
||||
or plugins.
|
||||
* to mark a function as a setup method; the function can itself
|
||||
receive funcargs and will execute multiple times if the funcargs
|
||||
are parametrized
|
||||
* to set a scope which influences when the setup function going to be
|
||||
called. valid scopes are ``session``, ``module``, ``class`` and ``function``.
|
||||
|
||||
Here is a simple example. First we define a global ``globdir`` resource::
|
||||
|
||||
# content of conftest.py
|
||||
import pytest
|
||||
|
||||
@pytest.mark.factory(scope="module")
|
||||
def globdir(testcontext, tmpdir):
|
||||
def fin():
|
||||
print "finalize", tmpdir
|
||||
testcontext.addfinalizer(fin)
|
||||
print "created resource", tmpdir
|
||||
return tmpdir
|
||||
|
||||
And then we write a test file containing a setup-marked function
|
||||
taking this resource and setting it as a module global::
|
||||
|
||||
# content of test_module.py
|
||||
import pytest
|
||||
|
||||
@pytest.mark.setup(scope="module")
|
||||
def setresource(testcontext, globdir):
|
||||
print "setupresource", globdir
|
||||
testcontext.module.myresource = globdir
|
||||
|
||||
def test_1():
|
||||
assert myresource
|
||||
print "using myresource", myresource
|
||||
|
||||
def test_2():
|
||||
assert myresource
|
||||
print "using myresource", myresource
|
||||
|
||||
Let's run this module::
|
||||
|
||||
$ py.test -qs
|
||||
collecting ... collected 2 items
|
||||
..
|
||||
2 passed in 0.26 seconds
|
||||
created resource /home/hpk/tmp/pytest-4427/test_10
|
||||
setupresource /home/hpk/tmp/pytest-4427/test_10
|
||||
using myresource /home/hpk/tmp/pytest-4427/test_10
|
||||
using myresource /home/hpk/tmp/pytest-4427/test_10
|
||||
finalize /home/hpk/tmp/pytest-4427/test_10
|
||||
|
||||
The two test functions in the module use the same global ``myresource``
|
||||
object because the ``setresource`` set it as a module attribute.
|
||||
|
||||
The ``globdir`` factory can now become parametrized without any test
|
||||
or setup code needing to change::
|
||||
|
||||
# content of conftest.py
|
||||
import pytest
|
||||
|
||||
@pytest.mark.factory(scope="module", params=["aaa", "bbb"])
|
||||
def globdir(testcontext, tmpdir):
|
||||
newtmp = tmpdir.join(testcontext.param)
|
||||
def fin():
|
||||
print "finalize", newtmp
|
||||
testcontext.addfinalizer(fin)
|
||||
print "created resource", newtmp
|
||||
return newtmp
|
||||
|
||||
Running the unchanged previous test files now runs four tests::
|
||||
|
||||
$ py.test -qs
|
||||
collecting ... collected 4 items
|
||||
....
|
||||
4 passed in 0.26 seconds
|
||||
created resource /home/hpk/tmp/pytest-4428/test_1_aaa_0/aaa
|
||||
setupresource /home/hpk/tmp/pytest-4428/test_1_aaa_0/aaa
|
||||
using myresource /home/hpk/tmp/pytest-4428/test_1_aaa_0/aaa
|
||||
using myresource /home/hpk/tmp/pytest-4428/test_1_aaa_0/aaa
|
||||
finalize /home/hpk/tmp/pytest-4428/test_1_aaa_0/aaa
|
||||
created resource /home/hpk/tmp/pytest-4428/test_1_bbb_0/bbb
|
||||
setupresource /home/hpk/tmp/pytest-4428/test_1_bbb_0/bbb
|
||||
using myresource /home/hpk/tmp/pytest-4428/test_1_bbb_0/bbb
|
||||
using myresource /home/hpk/tmp/pytest-4428/test_1_bbb_0/bbb
|
||||
finalize /home/hpk/tmp/pytest-4428/test_1_bbb_0/bbb
|
||||
|
||||
Each parameter causes the creation of a respective resource and the
|
||||
unchanged test module uses it in its ``@setup`` decorated method.
|
||||
|
||||
.. note::
|
||||
|
||||
Tests using a particular parametrized resource instance will
|
||||
executed next to each other. Any finalizers will be run before the
|
||||
next parametrized resource instance is being setup and tests
|
||||
are rerun.
|
||||
|
||||
Grouping tests by resource parameters
|
||||
----------------------------------------------------------
|
||||
|
|
@ -540,7 +425,10 @@ As expected only one pair of input/output values fails the simple test function.
|
|||
Note that there are various ways how you can mark groups of functions,
|
||||
see :ref:`mark`.
|
||||
|
||||
Generating parameters combinations, depending on command line
|
||||
|
||||
.. _`pytest_generate_tests`:
|
||||
|
||||
``pytest_generate_test``: implementing your own parametrization scheme
|
||||
----------------------------------------------------------------------------
|
||||
|
||||
.. regendoc:wipe
|
||||
|
|
|
|||
|
|
@ -0,0 +1,204 @@
|
|||
.. _xunitsetup:
|
||||
.. _setup:
|
||||
|
||||
``@setup`` functions or: xunit on steroids
|
||||
========================================================
|
||||
|
||||
.. versionadded:: 2.3
|
||||
|
||||
.. _`funcargs`: funcargs.html
|
||||
.. _`test parametrization`: funcargs.html#parametrizing-tests
|
||||
.. _`unittest plugin`: plugin/unittest.html
|
||||
.. _`xUnit`: http://en.wikipedia.org/wiki/XUnit
|
||||
|
||||
Python, Java and many other languages support a so called xUnit_ style
|
||||
of resource setup. This typically involves the call of a ``setup``
|
||||
("fixture") method before running a test function and ``teardown`` after
|
||||
it has finished. Unlike :ref:`injected resources <resources>` setup
|
||||
functions work indirectly by causing global side effects or
|
||||
setting test case attributes which test methods can then access.
|
||||
|
||||
pytest originally introduced in 2005 a fine-grained model of detecting
|
||||
setup and teardown functions on a per-module, class or function basis.
|
||||
The Python unittest module and nose have subsequently incorporated them.
|
||||
This model remains supported as :ref:`old-style xunit`.
|
||||
|
||||
With pytest-2.3 a new ``pytest.setup()`` decorator is introduced
|
||||
to mark functions as setup functions which:
|
||||
|
||||
- can receive resources through funcargs,
|
||||
- fully interoperate with parametrized resources,
|
||||
- can be defined in a plugin or conftest.py file and get called
|
||||
on a per-session, per-module, per-class or per-function basis,
|
||||
- can access the full :ref:`testcontext` for which the setup is called,
|
||||
- can precisely control teardown by registering one or multiple
|
||||
teardown functions as soon as they have performed some actions
|
||||
which need undoing, eliminating the no need for a separate
|
||||
"teardown" decorator.
|
||||
- allow to separate different setup concerns even if they
|
||||
happen to work in the same scope
|
||||
|
||||
All of these features are now demonstrated by little examples.
|
||||
|
||||
.. _`new_setup`:
|
||||
.. _`@pytest.mark.setup`:
|
||||
|
||||
basic per-function setup
|
||||
-------------------------------
|
||||
|
||||
.. regendoc:wipe
|
||||
|
||||
Suppose you want to have a clean directory with a single
|
||||
file entry for each test function in a module and have
|
||||
the test execute with this directory as current working dir::
|
||||
|
||||
# content of test_funcdir.py
|
||||
import pytest
|
||||
import os
|
||||
|
||||
@pytest.mark.setup()
|
||||
def mydir(tmpdir):
|
||||
tmpdir.join("myfile").write("example content")
|
||||
old = tmpdir.chdir()
|
||||
|
||||
def test_function1():
|
||||
assert os.path.exists("myfile")
|
||||
f = open("anotherfile", "w")
|
||||
f.write("")
|
||||
f.close()
|
||||
|
||||
def test_function2():
|
||||
assert os.path.exists("myfile")
|
||||
assert not os.path.exists("anotherfile")
|
||||
|
||||
Our ``mydir`` setup function is executed on a per-function basis,
|
||||
the default scope used by the ``pytest.mark.setup`` decorator.
|
||||
It accesses the ``tmpdir`` resource which provides a new empty
|
||||
directory path object. The ``test_function2`` here checks that
|
||||
it executes with a fresh directory and specifically
|
||||
does not see the previously created ``anotherfile``. We can
|
||||
thus expect two passing tests::
|
||||
|
||||
$ py.test -v
|
||||
=========================== test session starts ============================
|
||||
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev7 -- /home/hpk/venv/1/bin/python
|
||||
cachedir: /home/hpk/tmp/doc-exec-410/.cache
|
||||
plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov
|
||||
collecting ... collected 2 items
|
||||
|
||||
test_funcdir.py:9: test_function1 PASSED
|
||||
test_funcdir.py:15: test_function2 PASSED
|
||||
|
||||
========================= 2 passed in 0.26 seconds =========================
|
||||
|
||||
per-function setup, for every function of a project
|
||||
------------------------------------------------------------
|
||||
|
||||
If you want to define a setup per-function but want to apply
|
||||
it to every function in your project you don't need to duplicate
|
||||
the setup-definition into each test module. Instead you can put
|
||||
it into a ``conftest.py`` file into the root of your project::
|
||||
|
||||
# content of conftest.py
|
||||
import pytest
|
||||
import os
|
||||
|
||||
@pytest.mark.setup()
|
||||
def cleandir(tmpdir):
|
||||
old = tmpdir.chdir()
|
||||
|
||||
The ``cleandir`` setup function will be called for every test function
|
||||
below the directory tree where ``conftest.py`` resides. In this
|
||||
case it just uses the builtin ``tmpdir`` resource to change to the
|
||||
empty directory ahead of running a test.
|
||||
|
||||
test modules accessing a global resource
|
||||
-------------------------------------------------------
|
||||
|
||||
If you want test modules to access a global resource,
|
||||
you can stick the resource to the module globals in
|
||||
a per-module setup function. We use a :ref:`resource factory
|
||||
<@pytest.mark.factory>` to create our global resource::
|
||||
|
||||
# content of conftest.py
|
||||
import pytest
|
||||
|
||||
class GlobalResource:
|
||||
def __init__(self):
|
||||
pass
|
||||
|
||||
@pytest.mark.factory(scope="session")
|
||||
def globresource():
|
||||
return GlobalResource()
|
||||
|
||||
@pytest.mark.setup(scope="module")
|
||||
def setresource(testcontext, globresource):
|
||||
testcontext.module.globresource = globresource
|
||||
|
||||
Now any test module can access ``globresource`` as a module global::
|
||||
|
||||
# content of test_glob.py
|
||||
|
||||
def test_1():
|
||||
print ("test_1 %s" % globresource)
|
||||
def test_2():
|
||||
print ("test_2 %s" % globresource)
|
||||
|
||||
Let's run this module without output-capturing::
|
||||
|
||||
$ py.test -qs test_glob.py
|
||||
collecting ... collected 2 items
|
||||
..
|
||||
2 passed in 0.02 seconds
|
||||
test_1 <conftest.GlobalResource instance at 0x13197e8>
|
||||
test_2 <conftest.GlobalResource instance at 0x13197e8>
|
||||
|
||||
The two tests see the same global ``globresource`` object.
|
||||
|
||||
Parametrizing the global resource
|
||||
+++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
We extend the previous example and add parametrization to the globresource
|
||||
factory and also add a finalizer::
|
||||
|
||||
# content of conftest.py
|
||||
|
||||
import pytest
|
||||
|
||||
class GlobalResource:
|
||||
def __init__(self, param):
|
||||
self.param = param
|
||||
|
||||
@pytest.mark.factory(scope="session", params=[1,2])
|
||||
def globresource(testcontext):
|
||||
g = GlobalResource(testcontext.param)
|
||||
def fin():
|
||||
print "finalizing", g
|
||||
testcontext.addfinalizer(fin)
|
||||
return g
|
||||
|
||||
@pytest.mark.setup(scope="module")
|
||||
def setresource(testcontext, globresource):
|
||||
testcontext.module.globresource = globresource
|
||||
|
||||
And then re-run our test module::
|
||||
|
||||
$ py.test -qs test_glob.py
|
||||
collecting ... collected 4 items
|
||||
....
|
||||
4 passed in 0.02 seconds
|
||||
test_1 <conftest.GlobalResource instance at 0x1922e18>
|
||||
test_2 <conftest.GlobalResource instance at 0x1922e18>
|
||||
finalizing <conftest.GlobalResource instance at 0x1922e18>
|
||||
test_1 <conftest.GlobalResource instance at 0x1925518>
|
||||
test_2 <conftest.GlobalResource instance at 0x1925518>
|
||||
finalizing <conftest.GlobalResource instance at 0x1925518>
|
||||
|
||||
We are now running the two tests twice with two different global resource
|
||||
instances. Note that the tests are ordered such that only
|
||||
one instance is active at any given time: the finalizer of
|
||||
the first globresource instance is called before the second
|
||||
instance is created and sent to the setup functions.
|
||||
|
||||
|
||||
|
||||
|
|
@ -1,23 +1,16 @@
|
|||
.. _xunitsetup:
|
||||
|
||||
====================================
|
||||
Extended xUnit style setup fixtures
|
||||
====================================
|
||||
.. _`old-style xunit`:
|
||||
|
||||
.. _`funcargs`: funcargs.html
|
||||
.. _`test parametrization`: funcargs.html#parametrizing-tests
|
||||
.. _`unittest plugin`: plugin/unittest.html
|
||||
.. _`xUnit`: http://en.wikipedia.org/wiki/XUnit
|
||||
|
||||
Python, Java and many other languages support xUnit_ style testing.
|
||||
This typically involves the call of a ``setup`` ("fixture") method
|
||||
before running a test function and ``teardown`` after it has finished.
|
||||
``py.test`` supports a more fine-grained model of setup/teardown
|
||||
handling by optionally calling per-module and per-class hooks.
|
||||
Old-style xunit-style setup
|
||||
========================================
|
||||
|
||||
This section describes the old way how you can implement setup and
|
||||
teardown on a per-module/class/function basis. It remains fully
|
||||
supported but it is recommended to rather use :ref:`@setup functions <setup>`
|
||||
or :ref:`injected resources <resources>` for implementing your setup needs.
|
||||
|
||||
Module level setup/teardown
|
||||
=============================================
|
||||
--------------------------------------
|
||||
|
||||
If you have multiple test functions and test classes in a single
|
||||
module you can optionally implement the following fixture methods
|
||||
|
|
@ -32,7 +25,7 @@ which will usually be called once for all the functions::
|
|||
"""
|
||||
|
||||
Class level setup/teardown
|
||||
=============================================
|
||||
----------------------------------
|
||||
|
||||
Similarly, the following methods are called at class level before
|
||||
and after all test methods of the class are called::
|
||||
|
|
@ -50,7 +43,7 @@ and after all test methods of the class are called::
|
|||
"""
|
||||
|
||||
Method and function level setup/teardown
|
||||
=============================================
|
||||
-----------------------------------------------
|
||||
|
||||
Similarly, the following methods are called around each method invocation::
|
||||
|
||||
Loading…
Reference in New Issue