updating docs wrt funcargs and deprecations - still need updated implementation
--HG-- branch : trunk
This commit is contained in:
parent
6843c64a95
commit
6c861c273e
|
@ -120,6 +120,7 @@ name. Given a filesystem ``fspath`` it is constructed as follows:
|
||||||
the trailing ``.py``.
|
the trailing ``.py``.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Plugin hooks and events
|
Plugin hooks and events
|
||||||
=======================================
|
=======================================
|
||||||
|
|
||||||
|
|
|
@ -30,7 +30,6 @@ modules a leading ``test_`` or trailing ``_test`` filename.
|
||||||
From each test module every function with a leading ``test_``
|
From each test module every function with a leading ``test_``
|
||||||
or class with a leading ``Test`` name is collected.
|
or class with a leading ``Test`` name is collected.
|
||||||
|
|
||||||
.. _`generative tests`:
|
|
||||||
.. _`collection process`: ext.html#collection-process
|
.. _`collection process`: ext.html#collection-process
|
||||||
|
|
||||||
Rapidly write integration, functional, unit tests
|
Rapidly write integration, functional, unit tests
|
||||||
|
@ -259,11 +258,17 @@ For example, in order to avoid running some tests on Win32::
|
||||||
def test_xxx(self):
|
def test_xxx(self):
|
||||||
...
|
...
|
||||||
|
|
||||||
|
.. _`test generators`: funcargs.html#test-generators
|
||||||
|
|
||||||
|
.. _`generative tests`:
|
||||||
|
|
||||||
generative tests: yielding parametrized tests
|
generative tests: yielding parametrized tests
|
||||||
====================================================
|
====================================================
|
||||||
|
|
||||||
|
**deprecated since 1.0 in favour of `test generators`_**
|
||||||
|
|
||||||
*Generative tests* are test methods that are *generator functions* which
|
*Generative tests* are test methods that are *generator functions* which
|
||||||
``yield`` callables and their arguments. This is most useful for running a
|
``yield`` callables and their arguments. This is useful for running a
|
||||||
test function multiple times against different parameters. Example::
|
test function multiple times against different parameters. Example::
|
||||||
|
|
||||||
def test_generative():
|
def test_generative():
|
||||||
|
|
|
@ -1,27 +1,42 @@
|
||||||
======================================================
|
======================================================
|
||||||
**funcargs**: powerful test setup and parametrization
|
**funcargs**: test setup and parametrization
|
||||||
======================================================
|
======================================================
|
||||||
|
|
||||||
Since version 1.0 it is possible to provide arguments to test functions,
|
Since version 1.0 test functions can make great use of
|
||||||
often called "funcargs". The funcarg mechanisms were developed with
|
their arguments or "funcargs" for short. py.test helps
|
||||||
these goals in mind:
|
to setup or generate argument values with the goal
|
||||||
|
of making it easy to:
|
||||||
|
|
||||||
* **no boilerplate**: cleanly encapsulate test setup and fixtures
|
* separate test function code from test setup/fixtures
|
||||||
* **flexibility**: easily setup test state depending on command line options or environment
|
* manage test value setup and teardown depending on
|
||||||
* **readability**: write simple to read and debug test functions
|
command line options or configuration
|
||||||
* **parametrizing tests**: run a test function multiple times with different parameters
|
* parametrize multiple runs of the same test functions
|
||||||
|
* present useful debug info if something goes wrong
|
||||||
|
|
||||||
|
Using funcargs, test functions become more expressive,
|
||||||
|
more "templaty" and more test-aspect oriented. In fact,
|
||||||
|
funcarg mechanisms are meant to be complete and
|
||||||
|
convenient enough to
|
||||||
|
|
||||||
|
* substitute most usages of `xUnit style`_ setup
|
||||||
|
|
||||||
|
* substitute all usages of `old-style generative tests`_,
|
||||||
|
i.e. test functions that use the "yield" statement.
|
||||||
|
Using yield in test functions is deprecated since 1.0.
|
||||||
|
|
||||||
|
|
||||||
.. contents:: Contents:
|
.. _`xUnit style`: xunit_setup.html
|
||||||
:depth: 2
|
.. _`old-style generative tests`:
|
||||||
|
|
||||||
Basic mechanisms by example
|
.. _`funcarg provider`:
|
||||||
=============================================
|
|
||||||
|
|
||||||
providing single function arguments as needed
|
funcarg providers: setting up test function arguments
|
||||||
---------------------------------------------------------
|
==============================================================
|
||||||
|
|
||||||
Let's look at a simple example of using funcargs within a test module:
|
Test functions can specify one ore more arguments ("funcargs")
|
||||||
|
and a test module or plugin can define functions that provide
|
||||||
|
the function argument. Let's look at a self-contained example
|
||||||
|
that you can put into a test module:
|
||||||
|
|
||||||
.. sourcecode:: python
|
.. sourcecode:: python
|
||||||
|
|
||||||
|
@ -31,66 +46,30 @@ Let's look at a simple example of using funcargs within a test module:
|
||||||
def test_function(myfuncarg):
|
def test_function(myfuncarg):
|
||||||
assert myfuncarg == 42
|
assert myfuncarg == 42
|
||||||
|
|
||||||
1. To setup the running of the ``test_function()`` call, py.test
|
|
||||||
looks up a provider for the ``myfuncarg`` argument.
|
|
||||||
The provider method is recognized by its ``pytest_funcarg__`` prefix
|
|
||||||
followed by the requested function argument name.
|
|
||||||
The `request object`_ gives access to test context.
|
|
||||||
|
|
||||||
2. The ``test_function(42)`` call is executed. If the test fails
|
|
||||||
one can see the original provided value.
|
|
||||||
|
|
||||||
|
|
||||||
generating test functions call
|
|
||||||
----------------------------------------------------------------------
|
|
||||||
|
|
||||||
You can parametrize multiple runs of a test function by
|
|
||||||
providing multiple values for function arguments. Here
|
|
||||||
is an example for running the same test function three times.
|
|
||||||
|
|
||||||
.. sourcecode:: python
|
|
||||||
|
|
||||||
def pytest_genfunc(funcspec):
|
|
||||||
if "arg1" in funcspec.funcargnames:
|
|
||||||
for value in range(3):
|
|
||||||
funcspec.addcall(arg1=value, arg2=range(10))
|
|
||||||
|
|
||||||
def test_function(arg1, arg2):
|
|
||||||
assert arg1 in arg2
|
|
||||||
|
|
||||||
Here is what happens:
|
Here is what happens:
|
||||||
|
|
||||||
1. The ``pytest_genfunc()`` hook will be called once for each test
|
1. **lookup funcarg provider**: The ``test_function`` needs an value for
|
||||||
function during the collection process. The if-statement makes
|
``myfuncarg`` to run. The provider is found by its special
|
||||||
sure that we only add calls for functions that actually need the
|
name, ``pytest_funcarg__`` followed by the function
|
||||||
provided value. The `funcspec object`_ provides access to context
|
argument argument name. If a provider cannot be found,
|
||||||
information.
|
a list of all available function arguments is presented.
|
||||||
|
|
||||||
2. Subsequently the ``test_function()`` will be called three times
|
2. **setup funcarg value**: ``pytest_funcarg__myfuncarg(request)`` is
|
||||||
with three different values for ``arg1``.
|
called to setup the value for ``myfuncarg``.
|
||||||
|
|
||||||
Funcarg rules and support objects
|
3. **execute test** ``test_function(42)`` call is executed.
|
||||||
====================================
|
If the test fails one can see the original provided
|
||||||
|
value in the traceback at the top.
|
||||||
It is helpful to understand when function arguments are setup and when
|
|
||||||
function calls are added. Both providers deal with function arguments
|
|
||||||
but at different phases of the testing process. New Function calls
|
|
||||||
can only be added during the test collection phase whereas function
|
|
||||||
arguments are provided during the actual setup of (already collected)
|
|
||||||
test functions.
|
|
||||||
|
|
||||||
.. _`request object`:
|
.. _`request object`:
|
||||||
|
|
||||||
funcarg request objects
|
funcarg request objects
|
||||||
------------------------
|
------------------------------------------
|
||||||
|
|
||||||
Request objects encapsulate a request for a function argument from a
|
Request objects are passed to funcarg providers. Request objects
|
||||||
specific test function. Request objects provide access to command line
|
encapsulate a request for a function argument for a
|
||||||
options, the underlying python function and allow interaction
|
specific test function. Request objects allow providers to access
|
||||||
with other providers and the test running process.
|
test configuration and test context:
|
||||||
|
|
||||||
Attributes of request objects
|
|
||||||
++++++++++++++++++++++++++++++++++++++++
|
|
||||||
|
|
||||||
``request.argname``: name of the requested function argument
|
``request.argname``: name of the requested function argument
|
||||||
|
|
||||||
|
@ -98,12 +77,15 @@ Attributes of request objects
|
||||||
|
|
||||||
``request.cls``: class object where the test function is defined in or None.
|
``request.cls``: class object where the test function is defined in or None.
|
||||||
|
|
||||||
``funcspec.module``: module object where the test function is defined in.
|
``request.module``: module object where the test function is defined in.
|
||||||
|
|
||||||
``request.config``: access to command line opts and general config
|
``request.config``: access to command line opts and general config
|
||||||
|
|
||||||
|
``request.param``: if exists is the argument passed by a `parametrizing test generator`_
|
||||||
|
|
||||||
|
|
||||||
cleanup after test function execution
|
cleanup after test function execution
|
||||||
++++++++++++++++++++++++++++++++++++++++
|
---------------------------------------------
|
||||||
|
|
||||||
Request objects allow to **register a finalizer method** which is
|
Request objects allow to **register a finalizer method** which is
|
||||||
called after a test function has finished running.
|
called after a test function has finished running.
|
||||||
|
@ -139,58 +121,94 @@ is no next provider left. See the `decorator example`_
|
||||||
for a use of this method.
|
for a use of this method.
|
||||||
|
|
||||||
|
|
||||||
.. _`lookup order`:
|
.. _`test generators`:
|
||||||
|
.. _`parametrizing test generator`:
|
||||||
|
|
||||||
Order of provider and test generator lookup
|
generating parametrized tests with funcargs
|
||||||
----------------------------------------------
|
===========================================================
|
||||||
|
|
||||||
Both test generators as well as funcarg providers
|
You can parametrize multiple runs of the same test function
|
||||||
are looked up in the following three scopes:
|
by schedulings new test function calls which get different
|
||||||
|
funcarg values. Let's look at a simple self-contained
|
||||||
|
example:
|
||||||
|
|
||||||
1. test module
|
.. sourcecode:: python
|
||||||
2. local plugins
|
|
||||||
3. global plugins
|
|
||||||
|
|
||||||
Using multiple funcargs
|
def pytest_generate_tests(metafunc):
|
||||||
----------------------------------------
|
if "numiter" in metafunc.funcargs:
|
||||||
|
for i in range(10):
|
||||||
|
metafunc.addcall(param=i)
|
||||||
|
|
||||||
Test functions can have multiple arguments
|
def pytest_funcarg__numiter(request):
|
||||||
which can either come from a test generator
|
return request.param
|
||||||
or from a provider.
|
|
||||||
|
|
||||||
.. _`funcspec object`:
|
def test_func(numiter):
|
||||||
|
assert numiter < 10
|
||||||
|
|
||||||
funcspec objects
|
Here is what happens in detail:
|
||||||
------------------------
|
|
||||||
|
|
||||||
Funcspecs help to inspect a testfunction and
|
1. **add test function calls**:
|
||||||
to generate tests with combinations of function argument values.
|
``pytest_generate_tests(metafunc)`` hook is called once for each test
|
||||||
|
function. The `metafunc object`_ has context information.
|
||||||
|
``metafunc.addcall(param=i)`` schedules a new test call
|
||||||
|
such that function argument providers will see an additional
|
||||||
|
``arg`` attribute on their request object.
|
||||||
|
|
||||||
Calling ``funcspec.addcall(**funcargs)`` will add
|
2. **setup funcarg values**: the ``pytest_funcarg__arg1(request)`` provider is called
|
||||||
a tests function call using the given dictionary
|
10 times with ten different request objects all pointing to
|
||||||
of function arguments. This addition of a call
|
the same test function. Our provider here simply returns
|
||||||
happens during test collection.
|
the ``arg`` value but we could of course also setup more
|
||||||
|
heavyweight resources here.
|
||||||
|
|
||||||
Note that you can provide a name for your test run
|
3. **execute tests**: ``test_func(numiter)`` is called ten times with
|
||||||
by passing a ``_id=...`` parameter. This name
|
ten different arguments.
|
||||||
will be shown during verbose and traceback
|
|
||||||
reporting.
|
|
||||||
|
|
||||||
Attributes of funcspec objects
|
.. _`metafunc object`:
|
||||||
++++++++++++++++++++++++++++++++++++++++
|
|
||||||
|
|
||||||
``funcspec.funcargnames``: set of required function arguments for given function
|
test generators and metafunc objects
|
||||||
|
-------------------------------------------
|
||||||
|
|
||||||
``funcspec.function``: underlying python test function
|
metafunc objects are passed to the ``pytest_generate_tests`` hook.
|
||||||
|
They help to inspect a testfunction and to generate tests
|
||||||
|
according to test configuration or values specified
|
||||||
|
in the class or module where a test function is defined:
|
||||||
|
|
||||||
``funcspec.cls``: class object where the test function is defined in or None.
|
``metafunc.funcargnames``: set of required function arguments for given function
|
||||||
|
|
||||||
``funcspec.module``: the module object where the test function is defined in.
|
``metafunc.function``: underlying python test function
|
||||||
|
|
||||||
``funcspec.config``: access to command line opts and general config
|
``metafunc.cls``: class object where the test function is defined in or None.
|
||||||
|
|
||||||
|
``metafunc.module``: the module object where the test function is defined in.
|
||||||
|
|
||||||
|
``metafunc.config``: access to command line opts and general config
|
||||||
|
|
||||||
|
|
||||||
Useful Funcarg Tutorial Examples
|
the ``metafunc.addcall()`` method
|
||||||
|
-----------------------------------------------
|
||||||
|
|
||||||
|
.. sourcecode:: python
|
||||||
|
|
||||||
|
def addcall(id=None, param=None):
|
||||||
|
""" trigger a later test function call. """
|
||||||
|
|
||||||
|
The specified ``param`` will be seen by the
|
||||||
|
`funcarg provider`_ as a ``request.param`` attribute.
|
||||||
|
|
||||||
|
If you provide an `id`` it will be used for reporting
|
||||||
|
and identification purposes. If you don't supply an `id`
|
||||||
|
the stringified counter of the list of added calls will be used.
|
||||||
|
``id`` values needs to be unique between all
|
||||||
|
invocations for a given test function.
|
||||||
|
|
||||||
|
*Test generators are called during test collection which
|
||||||
|
is separate from the actual test setup and test run.
|
||||||
|
With distributed testing setting up funcargs will
|
||||||
|
even happen in a different process. Therefore one should
|
||||||
|
defer setup of heavyweight objects to funcarg providers.*
|
||||||
|
|
||||||
|
|
||||||
|
Funcarg Tutorial Examples
|
||||||
=======================================
|
=======================================
|
||||||
|
|
||||||
application specific test setup
|
application specific test setup
|
||||||
|
@ -237,9 +255,6 @@ following code into a local ``conftest.py``:
|
||||||
def myapp(self):
|
def myapp(self):
|
||||||
return MyApp()
|
return MyApp()
|
||||||
|
|
||||||
py.test finds the ``pytest_funcarg__mysetup`` method by
|
|
||||||
name, see also `lookup order`_.
|
|
||||||
|
|
||||||
To run the example we put a pseudo MyApp object into ``myapp.py``:
|
To run the example we put a pseudo MyApp object into ``myapp.py``:
|
||||||
|
|
||||||
.. sourcecode:: python
|
.. sourcecode:: python
|
||||||
|
@ -369,11 +384,11 @@ extend the `accept example`_ by putting this in our test class:
|
||||||
def test_sometest(self, accept):
|
def test_sometest(self, accept):
|
||||||
assert accept.tmpdir.join("special").check()
|
assert accept.tmpdir.join("special").check()
|
||||||
|
|
||||||
According to the the `lookup order`_ our module level provider
|
Our module level provider will be invoked first and it can
|
||||||
will be invoked first and it can ask ask its request object to
|
ask its request object to call the next provider and then
|
||||||
call the next provider and then decorate its result. This
|
decorate its result. This mechanism allows us to stay
|
||||||
mechanism allows us to stay ignorant of how/where the
|
ignorant of how/where the function argument is provided -
|
||||||
function argument is provided.
|
in our example from a ConftestPlugin but could be any plugin.
|
||||||
|
|
||||||
sidenote: the temporary directory used here are instances of
|
sidenote: the temporary directory used here are instances of
|
||||||
the `py.path.local`_ class which provides many of the os.path
|
the `py.path.local`_ class which provides many of the os.path
|
||||||
|
@ -382,62 +397,6 @@ methods in a convenient way.
|
||||||
.. _`py.path.local`: ../path.html#local
|
.. _`py.path.local`: ../path.html#local
|
||||||
|
|
||||||
|
|
||||||
parametrize test functions with multiple func args
|
|
||||||
--------------------------------------------------------------------------
|
|
||||||
|
|
||||||
You can trigger calling test functions which take more
|
|
||||||
than one function argument. Consider this example:
|
|
||||||
|
|
||||||
.. sourcecode:: python
|
|
||||||
|
|
||||||
def pytest_genfunc(funcspec):
|
|
||||||
for arg1 in range(2):
|
|
||||||
for arg2 in (10, 20):
|
|
||||||
funcspec.addcall(arg1=arg1, arg2=arg2)
|
|
||||||
|
|
||||||
# the actual test function
|
|
||||||
|
|
||||||
def test_function(arg1, arg2):
|
|
||||||
pass
|
|
||||||
|
|
||||||
Running this test module will result in ``test_function``
|
|
||||||
being called four times, with the following arguments::
|
|
||||||
|
|
||||||
test_function(0, 10)
|
|
||||||
test_function(0, 20)
|
|
||||||
test_function(1, 10)
|
|
||||||
test_function(2, 20)
|
|
||||||
|
|
||||||
|
|
||||||
example: test functions with pre-generated and provided funcargs
|
|
||||||
-------------------------------------------------------------------
|
|
||||||
|
|
||||||
You can mix generated function arguments and normally
|
|
||||||
provided ones. Consider this module:
|
|
||||||
|
|
||||||
.. sourcecode:: python
|
|
||||||
|
|
||||||
def pytest_genfunc(funcspec):
|
|
||||||
if "arg1" in funcspec.funcargnames: # test_function2 does not have it
|
|
||||||
funcspec.addcall(arg1=10)
|
|
||||||
funcspec.addcall(arg1=20)
|
|
||||||
|
|
||||||
def pytest_funcarg__arg2(request):
|
|
||||||
return [10, 20]
|
|
||||||
|
|
||||||
def test_function(arg1, arg2):
|
|
||||||
assert arg1 in arg2
|
|
||||||
|
|
||||||
def test_function2(arg2):
|
|
||||||
assert args2 == [10, 20]
|
|
||||||
|
|
||||||
Running this test module will result in ``test_function``
|
|
||||||
being called twice, with these arguments::
|
|
||||||
|
|
||||||
test_function(10, [10, 20])
|
|
||||||
test_function(20, [10, 20])
|
|
||||||
|
|
||||||
|
|
||||||
Questions and Answers
|
Questions and Answers
|
||||||
==================================
|
==================================
|
||||||
|
|
||||||
|
|
Loading…
Reference in New Issue