V3 draft of resource api
This commit is contained in:
holger krekel
parent
38b18c44e9
commit
4766497515
|
|
@ -610,6 +610,7 @@ class Session(FSCollector):
|
||||||
yield x
|
yield x
|
||||||
node.ihook.pytest_collectreport(report=rep)
|
node.ihook.pytest_collectreport(report=rep)
|
||||||
|
|
||||||
|
# XXX not used yet
|
||||||
def register_resource_factory(self, name, factoryfunc,
|
def register_resource_factory(self, name, factoryfunc,
|
||||||
matchscope=None,
|
matchscope=None,
|
||||||
cachescope=None):
|
cachescope=None):
|
||||||
|
|
@ -634,4 +635,3 @@ class Session(FSCollector):
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -27,6 +27,7 @@ import sys, os
|
||||||
#sys.path.insert(0, os.path.abspath('.'))
|
#sys.path.insert(0, os.path.abspath('.'))
|
||||||
|
|
||||||
autodoc_member_order = "bysource"
|
autodoc_member_order = "bysource"
|
||||||
|
todo_include_todos = 1
|
||||||
|
|
||||||
# -- General configuration -----------------------------------------------------
|
# -- General configuration -----------------------------------------------------
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -23,5 +23,6 @@ Full pytest documentation
|
||||||
:hidden:
|
:hidden:
|
||||||
|
|
||||||
changelog.txt
|
changelog.txt
|
||||||
examples/resources.txt
|
resources
|
||||||
|
example/resources_attic
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,369 +0,0 @@
|
||||||
|
|
||||||
V2: Creating and working with parametrized test resources
|
|
||||||
===============================================================
|
|
||||||
|
|
||||||
pytest-2.X provides generalized resource parametrization, unifying
|
|
||||||
and extending all existing funcarg and parametrization features of
|
|
||||||
previous pytest versions. Existing test suites and plugins written
|
|
||||||
for previous pytest versions shall run unmodified.
|
|
||||||
|
|
||||||
This V2 draft focuses on incorporating feedback provided by Floris Bruynooghe,
|
|
||||||
Carl Meyer and Ronny Pfannschmidt. It remains as draft documentation, pending
|
|
||||||
further refinements and changes according to implementation or backward
|
|
||||||
compatibility issues. The main changes to V1 are:
|
|
||||||
|
|
||||||
* changed API names (atnode -> scopenode)
|
|
||||||
* register_factory now happens at Node.collect_init() or pytest_collection_init
|
|
||||||
time. It will raise an Error if called during the runtestloop
|
|
||||||
(which performs setup/call/teardown for each collected test).
|
|
||||||
* new examples and notes related to @parametrize and metafunc.parametrize()
|
|
||||||
* use 2.X as the version for introduction - not sure if 2.3 or 2.4 will
|
|
||||||
actually bring it.
|
|
||||||
* examples/uses which were previously not possible to implement easily
|
|
||||||
are marked with "NEW" in the title.
|
|
||||||
|
|
||||||
(NEW) the init_collection and init_runtestloop hooks
|
|
||||||
------------------------------------------------------
|
|
||||||
|
|
||||||
pytest for a long time offers a pytest_configure and a pytest_sessionstart
|
|
||||||
hook which are often used to setup global resources. This suffers from
|
|
||||||
several problems:
|
|
||||||
|
|
||||||
1. in distributed testing the master process would setup test resources
|
|
||||||
that are never needed because it only co-ordinates the test run
|
|
||||||
activities of the slave processes.
|
|
||||||
|
|
||||||
2. In large test suites resources are created which might not be needed
|
|
||||||
for the concrete test run.
|
|
||||||
|
|
||||||
3. Thirdly, even if you only perform a collection (with "--collectonly")
|
|
||||||
resource-setup will be executed.
|
|
||||||
|
|
||||||
4. there is no place way to allow global parametrized collection and setup
|
|
||||||
|
|
||||||
The existing hooks are not a good place regarding these issues. pytest-2.X
|
|
||||||
solves all these issues through the introduction of two specific hooks
|
|
||||||
(and the new register_factory/getresource API)::
|
|
||||||
|
|
||||||
def pytest_init_collection(session):
|
|
||||||
# called ahead of pytest_collection, which implements the
|
|
||||||
# collection process
|
|
||||||
|
|
||||||
def pytest_init_runtestloop(session):
|
|
||||||
# called ahead of pytest_runtestloop() which executes the
|
|
||||||
# setup and calling of tests
|
|
||||||
|
|
||||||
The pytest_init_collection hook can be used for registering resources,
|
|
||||||
see `global resource management`_ and `parametrizing global resources`_.
|
|
||||||
|
|
||||||
The init_runtests can be used to setup and/or interact with global
|
|
||||||
resources. If you just use a global resource, you may explicitely
|
|
||||||
use it in a function argument or through a `class resource attribute`_.
|
|
||||||
|
|
||||||
.. _`global resource management`:
|
|
||||||
|
|
||||||
managing a global database resource
|
|
||||||
---------------------------------------------------------------
|
|
||||||
|
|
||||||
If you have one database object which you want to use in tests
|
|
||||||
you can write the following into a conftest.py file::
|
|
||||||
|
|
||||||
# contest of conftest.py
|
|
||||||
|
|
||||||
class Database:
|
|
||||||
def __init__(self):
|
|
||||||
print ("database instance created")
|
|
||||||
def destroy(self):
|
|
||||||
print ("database instance destroyed")
|
|
||||||
|
|
||||||
def factory_db(name, node):
|
|
||||||
db = Database()
|
|
||||||
node.addfinalizer(db.destroy)
|
|
||||||
return db
|
|
||||||
|
|
||||||
def pytest_init_collection(session):
|
|
||||||
session.register_factory("db", factory_db)
|
|
||||||
|
|
||||||
You can then access the constructed resource in a test by specifying
|
|
||||||
the pre-registered name in your function definition::
|
|
||||||
|
|
||||||
def test_something(db):
|
|
||||||
...
|
|
||||||
|
|
||||||
The "db" function argument will lead to a lookup and call of the respective
|
|
||||||
factory function and its result will be passed to the function body.
|
|
||||||
As the factory is registered on the session, it will by default only
|
|
||||||
get called once per session and its value will thus be re-used across
|
|
||||||
the whole test session.
|
|
||||||
|
|
||||||
Previously, factories would need to call the ``request.cached_setup()``
|
|
||||||
method to manage caching. Here is how we could implement the above
|
|
||||||
with traditional funcargs::
|
|
||||||
|
|
||||||
# content of conftest.py
|
|
||||||
class DataBase:
|
|
||||||
... as above
|
|
||||||
|
|
||||||
def pytest_funcarg__db(request):
|
|
||||||
return request.cached_setup(setup=DataBase,
|
|
||||||
teardown=lambda db: db.destroy,
|
|
||||||
scope="session")
|
|
||||||
|
|
||||||
As the funcarg factory is automatically registered by detecting its
|
|
||||||
name and because it is called each time "db" is requested, it needs
|
|
||||||
to care for caching itself, here by calling the cached_setup() method
|
|
||||||
to manage it. As it encodes the caching scope in the factory code body,
|
|
||||||
py.test has no way to report this via e. g. "py.test --funcargs".
|
|
||||||
More seriously, it's not exactly trivial to provide parametrization:
|
|
||||||
we would need to add a "parametrize" decorator where the resource is
|
|
||||||
used or implement a pytest_generate_tests(metafunc) hook to
|
|
||||||
call metafunc.parametrize() with the "db" argument, and then the
|
|
||||||
factory would need to care to pass the appropriate "extrakey" into
|
|
||||||
cached_setup(). By contrast, the new way just requires a modified
|
|
||||||
call to register factories::
|
|
||||||
|
|
||||||
def pytest_init_collection(session):
|
|
||||||
session.register_factory("db", [factory_mysql, factory_pg])
|
|
||||||
|
|
||||||
and no other code needs to change or get decorated.
|
|
||||||
|
|
||||||
(NEW) instantiating one database for each test module
|
|
||||||
---------------------------------------------------------------
|
|
||||||
|
|
||||||
If you want one database instance per test module you can restrict
|
|
||||||
caching by modifying the "scopenode" parameter of the registration
|
|
||||||
call above:
|
|
||||||
|
|
||||||
def pytest_init_collection(session):
|
|
||||||
session.register_factory("db", factory_db, scopenode=pytest.Module)
|
|
||||||
|
|
||||||
Neither the tests nor the factory function will need to change.
|
|
||||||
This means that you can decide the scoping of resources at runtime -
|
|
||||||
e.g. based on a command line option: for developer settings you might
|
|
||||||
want per-session and for Continous Integration runs you might prefer
|
|
||||||
per-module or even per-function scope like this::
|
|
||||||
|
|
||||||
def pytest_init_collection(session):
|
|
||||||
session.register_factory("db", factory_db,
|
|
||||||
scopenode=pytest.Function)
|
|
||||||
|
|
||||||
Using a resource from another resource factory
|
|
||||||
----------------------------------------------
|
|
||||||
|
|
||||||
You can use the database resource from a another resource factory through
|
|
||||||
the ``node.getresource()`` method. Let's add a resource factory for
|
|
||||||
a "db_users" table at module-level, extending the previous db-example::
|
|
||||||
|
|
||||||
def pytest_init_collection(session):
|
|
||||||
...
|
|
||||||
# this factory will be using a scopenode=pytest.Module because
|
|
||||||
# it is defined in a test module.
|
|
||||||
session.register_factory("db_users", createusers)
|
|
||||||
|
|
||||||
def createusers(name, node):
|
|
||||||
db = node.getresource("db")
|
|
||||||
table = db.create_table("users", ...)
|
|
||||||
node.addfinalizer(lambda: db.destroy_table("users")
|
|
||||||
|
|
||||||
def test_user_creation(db_users):
|
|
||||||
...
|
|
||||||
|
|
||||||
The create-users will be called for each module. After the tests in
|
|
||||||
that module finish execution, the table will be destroyed according
|
|
||||||
to registered finalizer. Note that calling getresource() for a resource
|
|
||||||
which has a tighter scope will raise a LookupError because the
|
|
||||||
is not available at a more general scope. Concretely, if you
|
|
||||||
table is defined as a per-session resource and the database object as a
|
|
||||||
per-module one, the table creation cannot work on a per-session basis.
|
|
||||||
|
|
||||||
amending/decorating a resource / funcarg__ compatibility
|
|
||||||
----------------------------------------------------------------------
|
|
||||||
|
|
||||||
If you want to decorate a session-registered resource with
|
|
||||||
a test-module one, you can do the following::
|
|
||||||
|
|
||||||
# content of conftest.py
|
|
||||||
def pytest_init_collection(session):
|
|
||||||
session.register_factory("db_users", createusers)
|
|
||||||
|
|
||||||
This will register a db_users method on a per-session basis.
|
|
||||||
If you want to create a dummy user such that all test
|
|
||||||
methods in a test module can work with it::
|
|
||||||
|
|
||||||
# content of test_user_admin.py
|
|
||||||
def setup_class(cls, db_users):
|
|
||||||
|
|
||||||
def pytest_init_collection(session):
|
|
||||||
session.register_factory("db_users", createcreate_users,
|
|
||||||
scopenode=pytest.Module)
|
|
||||||
|
|
||||||
def create_users(name, node):
|
|
||||||
# get the session-managed resource
|
|
||||||
db_users = node.getresource(name)
|
|
||||||
# add a user and define a remove_user undo function
|
|
||||||
...
|
|
||||||
node.addfinalizer(remove_user)
|
|
||||||
return db_users
|
|
||||||
|
|
||||||
def test_user_fields(db_users):
|
|
||||||
# work with db_users with a pre-created entry
|
|
||||||
...
|
|
||||||
|
|
||||||
Using the pytest_funcarg__ mechanism, you can do the equivalent::
|
|
||||||
|
|
||||||
# content of test_user_admin.py
|
|
||||||
|
|
||||||
def pytest_funcarg__db_users(request):
|
|
||||||
def create_user():
|
|
||||||
db_users = request.getfuncargvalue("db_users")
|
|
||||||
# add a user
|
|
||||||
return db_users
|
|
||||||
def remove_user(db_users):
|
|
||||||
...
|
|
||||||
return request.cached_setup(create_user, remove_user, scope="module")
|
|
||||||
|
|
||||||
As the funcarg mechanism is implemented in terms of the new API
|
|
||||||
it's also possible to mix - use register_factory/getresource at plugin-level
|
|
||||||
and pytest_funcarg__ factories at test module level.
|
|
||||||
|
|
||||||
As discussed previously with `global resource management`_, the funcarg-factory
|
|
||||||
does not easily extend to provide parametrization.
|
|
||||||
|
|
||||||
|
|
||||||
.. _`class resource attributes`:
|
|
||||||
|
|
||||||
(NEW) Setting resources as class attributes
|
|
||||||
-------------------------------------------
|
|
||||||
|
|
||||||
If you want to make an attribute available on a test class, you can
|
|
||||||
use a new mark::
|
|
||||||
|
|
||||||
@pytest.mark.class_resource("db")
|
|
||||||
class TestClass:
|
|
||||||
def test_something(self):
|
|
||||||
#use self.db
|
|
||||||
|
|
||||||
Note that this way of using resources work with unittest.TestCase-style
|
|
||||||
tests as well. If you have defined "db" as a parametrized resource,
|
|
||||||
the functions of the Test class will be run multiple times with different
|
|
||||||
values found in "self.db".
|
|
||||||
|
|
||||||
Previously, pytest could not offer its resource management features
|
|
||||||
since those were tied to passing function arguments ("funcargs") and
|
|
||||||
this cannot be easily integrated with the unittest framework and its
|
|
||||||
common per-project customizations.
|
|
||||||
|
|
||||||
|
|
||||||
.. _`parametrizing global resources`:
|
|
||||||
|
|
||||||
(NEW) parametrizing global resources
|
|
||||||
----------------------------------------------------
|
|
||||||
|
|
||||||
If you want to rerun tests with different resource values you can specify
|
|
||||||
a list of factories instead of just one::
|
|
||||||
|
|
||||||
def pytest_init_collection(session):
|
|
||||||
session.register_factory("db", [factory1, factory2])
|
|
||||||
|
|
||||||
In this case all tests that require the "db" resource will be run twice
|
|
||||||
using the respective values obtained from the two factory functions.
|
|
||||||
|
|
||||||
For reporting purposes you might want to also define identifiers
|
|
||||||
for the db values::
|
|
||||||
|
|
||||||
def pytest_init_collection(session):
|
|
||||||
session.register_factory("db", [factory1, factory2],
|
|
||||||
ids=["mysql", "pg"])
|
|
||||||
|
|
||||||
This will make pytest use the respective id values when reporting
|
|
||||||
nodeids.
|
|
||||||
|
|
||||||
|
|
||||||
(New) Declaring resource usage / implicit parametrization
|
|
||||||
----------------------------------------------------------
|
|
||||||
|
|
||||||
Sometimes you may have a resource that can work in multiple variants,
|
|
||||||
like using different database backends. As another use-case,
|
|
||||||
pytest's own test suite uses a "testdir" funcarg which helps to setup
|
|
||||||
example scenarios, perform a subprocess-pytest run and check the output.
|
|
||||||
However, there are many features that should also work with the pytest-xdist
|
|
||||||
mode, distributing tests to multiple CPUs or hosts. The invocation
|
|
||||||
variants are not visible in the function signature and cannot be easily
|
|
||||||
addressed through a "parametrize" decorator or call. Nevertheless we want
|
|
||||||
to have both invocation variants to be collected and executed.
|
|
||||||
|
|
||||||
The solution is to tell pytest that you are using a resource implicitely::
|
|
||||||
|
|
||||||
@pytest.mark.uses_resource("invocation-option")
|
|
||||||
class TestClass:
|
|
||||||
def test_method(self, testdir):
|
|
||||||
...
|
|
||||||
|
|
||||||
When the testdir factory gets the parametrized "invocation-option"
|
|
||||||
resource, it will see different values, depending on what the respective
|
|
||||||
factories provide. To register the invocation-mode factory you would write::
|
|
||||||
|
|
||||||
# content of conftest.py
|
|
||||||
def pytest_init_collection(session):
|
|
||||||
session.register_factory("invocation-option",
|
|
||||||
[lambda **kw: "", lambda **kw: "-n1"])
|
|
||||||
|
|
||||||
The testdir factory can then access it easily::
|
|
||||||
|
|
||||||
option = node.getresource("invocation-option", "")
|
|
||||||
...
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
apart from the "uses_resource" decoration none of the already
|
|
||||||
written test functions needs to be modified for the new API.
|
|
||||||
|
|
||||||
The implicit "testdir" parametrization only happens for the tests
|
|
||||||
which declare use of the invocation-option resource. All other
|
|
||||||
tests will get the default value passed as the second parameter
|
|
||||||
to node.getresource() above. You can thus restrict
|
|
||||||
running the variants to particular tests or test sets.
|
|
||||||
|
|
||||||
To conclude, these three code fragments work together to allow efficient
|
|
||||||
cross-session resource parametrization.
|
|
||||||
|
|
||||||
|
|
||||||
Implementation and compatibility notes
|
|
||||||
============================================================
|
|
||||||
|
|
||||||
The new API is designed to support all existing resource parametrization
|
|
||||||
and funcarg usages. This chapter discusses implementation aspects.
|
|
||||||
Feel free to choose ignorance and only consider the above usage-level.
|
|
||||||
|
|
||||||
Implementing the funcarg mechanism in terms of the new API
|
|
||||||
-------------------------------------------------------------
|
|
||||||
|
|
||||||
Prior to pytest-2.X, pytest mainly advertised the "funcarg" mechanism
|
|
||||||
for resource management. It provides automatic registration of
|
|
||||||
factories through discovery of ``pytest_funcarg__NAME`` factory methods
|
|
||||||
on plugins, test modules, classes and functions. Those factories are be
|
|
||||||
called *each time* a resource (funcarg) is required, hence the support
|
|
||||||
for a ``request.cached_setup" method which helps to cache resources
|
|
||||||
across calls. Request objects internally keep a (item, requested_name,
|
|
||||||
remaining-factories) state. The "reamaining-factories" state is
|
|
||||||
used for implementing decorating factories; a factory for a given
|
|
||||||
name can call ``getfuncargvalue(name)`` to invoke the next-matching
|
|
||||||
factory factories and then amend the return value.
|
|
||||||
|
|
||||||
In order to implement the existing funcarg mechanism through
|
|
||||||
the new API, the new API needs to internally keep around similar
|
|
||||||
state. XXX
|
|
||||||
|
|
||||||
As an example let's consider the Module.setup_collect() method::
|
|
||||||
|
|
||||||
class Module(PyCollector):
|
|
||||||
def setup_collect(self):
|
|
||||||
for name, func in self.obj.__dict__.items():
|
|
||||||
if name.startswith("pytest_funcarg__"):
|
|
||||||
resourcename = name[len("pytest_funcarg__"):]
|
|
||||||
self.register_factory(resourcename,
|
|
||||||
RequestAdapter(self, name, func))
|
|
||||||
|
|
||||||
The request adapater takes care to provide the pre-2.X API for funcarg
|
|
||||||
factories, i.e. request.cached_setup/addfinalizer/getfuncargvalue
|
|
||||||
methods and some attributes.
|
|
||||||
|
|
@ -110,12 +110,13 @@ with a list of available function arguments.
|
||||||
The request object passed to factories
|
The request object passed to factories
|
||||||
-----------------------------------------
|
-----------------------------------------
|
||||||
|
|
||||||
Each funcarg factory receives a :py:class:`~_pytest.main.Request` object which
|
Each funcarg factory receives a :py:class:`~_pytest.python.Request` object which
|
||||||
provides methods to manage caching and finalization in the context of the
|
provides methods to manage caching and finalization in the context of the
|
||||||
test invocation as well as several attributes of the the underlying test item. In fact, as of version pytest-2.3, the request API is implemented on all Item
|
test invocation as well as several attributes of the the underlying test item. In fact, as of version pytest-2.3, the request API is implemented on all Item
|
||||||
objects and therefore the request object has general :py:class:`Node attributes and methods <_pytest.main.Node>` attributes. This is a backward compatible
|
objects and therefore the request object has general :py:class:`Node attributes and methods <_pytest.main.Node>` attributes. This is a backward compatible
|
||||||
change so no changes are neccessary for pre-2.3 funcarg factories.
|
change so no changes are neccessary for pre-2.3 funcarg factories.
|
||||||
|
|
||||||
|
|
||||||
.. _`parametrizing tests, generalized`: http://tetamap.wordpress.com/2009/05/13/parametrizing-python-tests-generalized/
|
.. _`parametrizing tests, generalized`: http://tetamap.wordpress.com/2009/05/13/parametrizing-python-tests-generalized/
|
||||||
|
|
||||||
.. _`blog post about the monkeypatch funcarg`: http://tetamap.wordpress.com/2009/03/03/monkeypatching-in-unit-tests-done-right/
|
.. _`blog post about the monkeypatch funcarg`: http://tetamap.wordpress.com/2009/03/03/monkeypatching-in-unit-tests-done-right/
|
||||||
|
|
|
||||||
|
|
@ -296,6 +296,7 @@ into interactive debugging when a test failure occurs.
|
||||||
|
|
||||||
The :py:mod:`_pytest.terminal` reported specifically uses
|
The :py:mod:`_pytest.terminal` reported specifically uses
|
||||||
the reporting hook to print information about a test run.
|
the reporting hook to print information about a test run.
|
||||||
|
|
||||||
Collection hooks
|
Collection hooks
|
||||||
------------------------------
|
------------------------------
|
||||||
|
|
||||||
|
|
@ -309,6 +310,7 @@ For influencing the collection of objects in Python modules
|
||||||
you can use the following hook:
|
you can use the following hook:
|
||||||
|
|
||||||
.. autofunction:: pytest_pycollect_makeitem
|
.. autofunction:: pytest_pycollect_makeitem
|
||||||
|
.. autofunction:: pytest_generate_tests
|
||||||
|
|
||||||
|
|
||||||
Reporting hooks
|
Reporting hooks
|
||||||
|
|
@ -329,7 +331,7 @@ test execution:
|
||||||
Reference of objects involved in hooks
|
Reference of objects involved in hooks
|
||||||
===========================================================
|
===========================================================
|
||||||
|
|
||||||
.. autoclass:: _pytest.main.Request()
|
.. autoclass:: _pytest.python.Request()
|
||||||
:members:
|
:members:
|
||||||
|
|
||||||
.. autoclass:: _pytest.config.Config()
|
.. autoclass:: _pytest.config.Config()
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,426 @@
|
||||||
|
|
||||||
|
V3: Creating and working with parametrized test resources
|
||||||
|
===============================================================
|
||||||
|
|
||||||
|
**Target audience**: Reading this document requires basic knowledge of
|
||||||
|
python testing, xUnit setup methods and the basic pytest funcarg mechanism,
|
||||||
|
see http://pytest.org/latest/funcargs.html
|
||||||
|
|
||||||
|
**Abstract**: pytest-2.X provides more powerful and more flexible funcarg
|
||||||
|
and setup machinery. It does so by introducing a new @funcarg and a
|
||||||
|
new @setup marker which allows to define scoping and parametrization
|
||||||
|
parameters. If using ``@funcarg``, following the ``pytest_funcarg__``
|
||||||
|
naming pattern becomes optional. Functions decorated with ``@setup``
|
||||||
|
are called independenlty from the definition of funcargs but can
|
||||||
|
access funcarg values if needed. This allows for ultimate flexibility
|
||||||
|
in designing your test fixtures and their parametrization. Also,
|
||||||
|
you can now use ``py.test --collectonly`` to inspect your fixture
|
||||||
|
setup. Nonwithstanding these extensions, pre-existing test suites
|
||||||
|
and plugins written to work for previous pytest versions shall run unmodified.
|
||||||
|
|
||||||
|
|
||||||
|
**Changes**: This V3 draft is based on incorporating and thinking about
|
||||||
|
feedback provided by Floris Bruynooghe, Carl Meyer and Samuele Pedroni.
|
||||||
|
It remains as draft documentation, pending further refinements and
|
||||||
|
changes according to implementation or backward compatibility issues.
|
||||||
|
The main changes to V2 are:
|
||||||
|
|
||||||
|
* Collapse funcarg factory decorator into a single "@funcarg" one.
|
||||||
|
You can specify scopes and params with it. Moreover, if you supply
|
||||||
|
a "name" you do not need to follow the "pytest_funcarg__NAME" naming
|
||||||
|
pattern. Keeping with "funcarg" naming arguable now makes more
|
||||||
|
sense since the main interface using these resources are test and
|
||||||
|
setup functions. Keeping it probably causes the least semantic friction.
|
||||||
|
|
||||||
|
* Drop setup_directory/setup_session and introduce a new @setup
|
||||||
|
decorator similar to the @funcarg one but accepting funcargs.
|
||||||
|
|
||||||
|
* cosnider the extended setup_X funcargs for dropping because
|
||||||
|
the new @setup decorator probably is more flexible and introduces
|
||||||
|
less implementation complexity.
|
||||||
|
|
||||||
|
.. currentmodule:: _pytest
|
||||||
|
|
||||||
|
|
||||||
|
Shortcomings of the previous pytest_funcarg__ mechanism
|
||||||
|
---------------------------------------------------------
|
||||||
|
|
||||||
|
The previous funcarg mechanism calls a factory each time a
|
||||||
|
funcarg for a test function is requested. If a factory wants
|
||||||
|
t re-use a resource across different scopes, it often used
|
||||||
|
the ``request.cached_setup()`` helper to manage caching of
|
||||||
|
resources. Here is a basic example how we could implement
|
||||||
|
a per-session Database object::
|
||||||
|
|
||||||
|
# content of conftest.py
|
||||||
|
class Database:
|
||||||
|
def __init__(self):
|
||||||
|
print ("database instance created")
|
||||||
|
def destroy(self):
|
||||||
|
print ("database instance destroyed")
|
||||||
|
|
||||||
|
def pytest_funcarg__db(request):
|
||||||
|
return request.cached_setup(setup=DataBase,
|
||||||
|
teardown=lambda db: db.destroy,
|
||||||
|
scope="session")
|
||||||
|
|
||||||
|
There are some problems with this approach:
|
||||||
|
|
||||||
|
1. Scoping resource creation is not straight forward, instead one must
|
||||||
|
understand the intricate cached_setup() method mechanics.
|
||||||
|
|
||||||
|
2. parametrizing the "db" resource is not straight forward:
|
||||||
|
you need to apply a "parametrize" decorator or implement a
|
||||||
|
:py:func:`~hookspec.pytest_generate_tests` hook
|
||||||
|
calling :py:func:`~python.Metafunc.parametrize` which
|
||||||
|
performs parametrization at the places where the resource
|
||||||
|
is used. Moreover, you need to modify the factory to use an
|
||||||
|
``extrakey`` parameter containing ``request.param`` to the
|
||||||
|
:py:func:`~python.Request.cached_setup` call.
|
||||||
|
|
||||||
|
3. the current implementation is inefficient: it performs factory discovery
|
||||||
|
each time a "db" argument is required. This discovery wrongly happens at
|
||||||
|
setup-time.
|
||||||
|
|
||||||
|
4. there is no way how you can use funcarg factories, let alone
|
||||||
|
parametrization, when your tests use the xUnit setup_X approach.
|
||||||
|
|
||||||
|
5. there is no way to specify a per-directory scope for caching.
|
||||||
|
|
||||||
|
In the following sections, API extensions are presented to solve
|
||||||
|
each of these problems.
|
||||||
|
|
||||||
|
|
||||||
|
Direct scoping of funcarg factories
|
||||||
|
--------------------------------------------------------
|
||||||
|
|
||||||
|
Instead of calling cached_setup(), you can decorate your factory
|
||||||
|
to state its scope::
|
||||||
|
|
||||||
|
@pytest.mark.funcarg(scope="session")
|
||||||
|
def pytest_funcarg__db(request):
|
||||||
|
# factory will only be invoked once per session -
|
||||||
|
db = DataBase()
|
||||||
|
request.addfinalizer(db.destroy) # destroy when session is finished
|
||||||
|
return db
|
||||||
|
|
||||||
|
This factory implementation does not need to call ``cached_setup()`` anymore
|
||||||
|
because it will only be invoked once per session. Moreover, the
|
||||||
|
``request.addfinalizer()`` registers a finalizer according to the specified
|
||||||
|
resource scope on which the factory function is operating. With this new
|
||||||
|
scoping, the still existing ``cached_setup()`` should be much less used
|
||||||
|
but will remain for compatibility reasons and for the case where you
|
||||||
|
still want to have your factory get called on a per-item basis.
|
||||||
|
|
||||||
|
|
||||||
|
Direct parametrization of funcarg resource factories
|
||||||
|
----------------------------------------------------------
|
||||||
|
|
||||||
|
Previously, funcarg factories could not directly cause parametrization.
|
||||||
|
You needed to specify a ``@parametrize`` or implement a ``pytest_generate_tests`` hook to perform parametrization, i.e. calling a test multiple times
|
||||||
|
with different value sets. pytest-2.X introduces a decorator for use
|
||||||
|
on the factory itself::
|
||||||
|
|
||||||
|
@pytest.mark.funcarg(params=["mysql", "pg"])
|
||||||
|
def pytest_funcarg__db(request):
|
||||||
|
...
|
||||||
|
|
||||||
|
Here the factory will be invoked twice (with the respective "mysql"
|
||||||
|
and "pg" values set as ``request.param`` attributes) and and all of
|
||||||
|
the tests requiring "db" will run twice as well. The "mysql" and
|
||||||
|
"pg" values will also be used for reporting the test-invocation variants.
|
||||||
|
|
||||||
|
This new way of parametrizing funcarg factories should in many cases
|
||||||
|
allow to re-use already written factories because effectively
|
||||||
|
``request.param`` are already the parametrization attribute for test
|
||||||
|
functions/classes were parametrized via
|
||||||
|
:py:func:`~_pytest.python.Metafunc.parametrize(indirect=True)` calls.
|
||||||
|
|
||||||
|
Of course it's perfectly fine to combine parametrization and scoping::
|
||||||
|
|
||||||
|
@pytest.mark.funcarg(scope="session", params=["mysql", "pg"])
|
||||||
|
def pytest_funcarg__db(request):
|
||||||
|
if request.param == "mysql":
|
||||||
|
db = MySQL()
|
||||||
|
elif request.param == "pg":
|
||||||
|
db = PG()
|
||||||
|
request.addfinalizer(db.destroy) # destroy when session is finished
|
||||||
|
return db
|
||||||
|
|
||||||
|
This would execute all tests requiring the per-session "db" resource twice,
|
||||||
|
receiving the values created by the two respective invocations to the
|
||||||
|
factory function.
|
||||||
|
|
||||||
|
Direct usage of funcargs with funcargs factories
|
||||||
|
----------------------------------------------------------
|
||||||
|
|
||||||
|
You can now directly use funcargs in funcarg factories. Example::
|
||||||
|
|
||||||
|
@pytest.mark.funcarg(scope="session")
|
||||||
|
def db(request, tmpdir):
|
||||||
|
# tmpdir is a session-specific tempdir
|
||||||
|
|
||||||
|
Apart from convenience it also solves an issue when your factory
|
||||||
|
depends on a parametrized funcarg. Previously, a call to
|
||||||
|
``request.getfuncargvalue()`` would not allow pytest to know
|
||||||
|
at collection time about the fact that a required resource is
|
||||||
|
actually parametrized.
|
||||||
|
|
||||||
|
The "pytest_funcarg__" prefix becomes optional
|
||||||
|
-----------------------------------------------------
|
||||||
|
|
||||||
|
When using the ``@funcarg`` decorator you do not need to use
|
||||||
|
the ``pytest_funcarg__`` prefix any more::
|
||||||
|
|
||||||
|
@pytest.mark.funcarg
|
||||||
|
def db(request):
|
||||||
|
...
|
||||||
|
|
||||||
|
The name under which the funcarg resource can be requested is ``db``.
|
||||||
|
Any ``pytest_funcarg__`` prefix will be stripped. Note that a an
|
||||||
|
unqualified funcarg-marker implies a scope of "function" meaning
|
||||||
|
that the funcarg factory will be called for each test function invocation.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
support for a new @setup marker
|
||||||
|
------------------------------------------------------
|
||||||
|
|
||||||
|
pytest for a long time offered a pytest_configure and a pytest_sessionstart
|
||||||
|
hook which are often used to setup global resources. This suffers from
|
||||||
|
several problems:
|
||||||
|
|
||||||
|
1. in distributed testing the master process would setup test resources
|
||||||
|
that are never needed because it only co-ordinates the test run
|
||||||
|
activities of the slave processes.
|
||||||
|
|
||||||
|
2. if you only perform a collection (with "--collectonly")
|
||||||
|
resource-setup will still be executed.
|
||||||
|
|
||||||
|
3. If a pytest_sessionstart is contained in some subdirectories
|
||||||
|
conftest.py file, it will not be called. This stems from the
|
||||||
|
fact that this hook is actually used for reporting, in particular
|
||||||
|
the test-header with platform/custom information.
|
||||||
|
|
||||||
|
4. there is no direct way how you can restrict setup to a directory scope.
|
||||||
|
|
||||||
|
Moreover, it is today not easy to define scoped setup from plugins or
|
||||||
|
conftest files other than to implement a ``pytest_runtest_setup()`` hook
|
||||||
|
and caring for scoping/caching yourself. And it's virtually impossible
|
||||||
|
to do this with parametrization as ``pytest_runtest_setup()`` is called
|
||||||
|
during test execution and parametrization happens at collection time.
|
||||||
|
|
||||||
|
It follows that pytest_configure/session/runtest_setup are often not
|
||||||
|
appropriate for implementing common fixture needs. Therefore,
|
||||||
|
pytest-2.X introduces a new "@pytest.mark.setup" marker, accepting
|
||||||
|
the same parameters as the @funcargs decorator. The difference is
|
||||||
|
that the decorated function can accept function arguments itself
|
||||||
|
Example::
|
||||||
|
|
||||||
|
# content of conftest.py
|
||||||
|
import pytest
|
||||||
|
@pytest.mark.setup(scope="session")
|
||||||
|
def mysetup(db):
|
||||||
|
...
|
||||||
|
|
||||||
|
This ``mysetup`` function is going to be executed when the first
|
||||||
|
test in the directory tree executes. It is going to be executed once
|
||||||
|
per-session and it receives the ``db`` funcarg which must be of same
|
||||||
|
of higher scope; you e. g. generally cannot use a per-module or per-function
|
||||||
|
scoped resource in a session-scoped setup function.
|
||||||
|
|
||||||
|
You can also use ``@setup`` inside a test module or class::
|
||||||
|
|
||||||
|
# content of test_module.py
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
@pytest.mark.setup(scope="module", params=[1,2,3])
|
||||||
|
def modes(tmpdir, request):
|
||||||
|
# ...
|
||||||
|
|
||||||
|
This would execute the ``modes`` function once for each parameter.
|
||||||
|
In addition to normal funcargs you can also receive the "request"
|
||||||
|
funcarg which represents a takes on each of the values in the
|
||||||
|
``params=[1,2,3]`` decorator argument.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
For each scope, the funcargs will be setup and then the setup functions
|
||||||
|
will be called. This allows @setup-decorated functions to depend
|
||||||
|
on already setup funcarg values by accessing ``request.funcargs``.
|
||||||
|
|
||||||
|
Using funcarg resources in xUnit setup methods
|
||||||
|
------------------------------------------------------------
|
||||||
|
|
||||||
|
XXX Consider this feature in contrast to the @setup feature - probably
|
||||||
|
introducing one of them is better and the @setup decorator is more flexible.
|
||||||
|
|
||||||
|
For a long time, pytest has recommended the usage of funcarg
|
||||||
|
factories as a primary means for managing resources in your test run.
|
||||||
|
It is a better approach than the jUnit-based approach in many cases, even
|
||||||
|
more with the new pytest-2.X features, because the funcarg resource factory
|
||||||
|
provides a single place to determine scoping and parametrization. Your tests
|
||||||
|
do not need to encode setup/teardown details in every test file's
|
||||||
|
setup_module/class/method.
|
||||||
|
|
||||||
|
However, the jUnit methods originally introduced by pytest to Python,
|
||||||
|
remain popoular with nose and unittest-based test suites. Without question,
|
||||||
|
there are large existing test suites using this paradigm. pytest-2.X
|
||||||
|
recognizes this fact and now offers direct integration with funcarg resources. Here is a basic example for getting a per-module tmpdir::
|
||||||
|
|
||||||
|
def setup_module(mod, tmpdir):
|
||||||
|
mod.tmpdir = tmpdir
|
||||||
|
|
||||||
|
This will trigger pytest's funcarg mechanism to create a value of
|
||||||
|
"tmpdir" which can then be used throughout the module as a global.
|
||||||
|
|
||||||
|
The new extension to setup_X methods also works in case a resource is
|
||||||
|
parametrized. For example, let's consider an setup_class example using
|
||||||
|
our "db" resource::
|
||||||
|
|
||||||
|
class TestClass:
|
||||||
|
def setup_class(cls, db):
|
||||||
|
cls.db = db
|
||||||
|
# perform some extra things on db
|
||||||
|
# so that test methods can work with it
|
||||||
|
|
||||||
|
With pytest-2.X the setup* methods will be discovered at collection-time,
|
||||||
|
allowing to seemlessly integrate this approach with parametrization,
|
||||||
|
allowing the factory specification to determine all details. The
|
||||||
|
setup_class itself does not itself need to be aware of the fact that
|
||||||
|
"db" might be a mysql/PG database.
|
||||||
|
Note that if the specified resource is provided only as a per-testfunction
|
||||||
|
resource, collection would early on report a ScopingMismatch error.
|
||||||
|
|
||||||
|
|
||||||
|
the "directory" caching scope
|
||||||
|
--------------------------------------------
|
||||||
|
|
||||||
|
All API accepting a scope (:py:func:`cached_setup()` and
|
||||||
|
the new funcarg/setup decorators) now also accept a "directory"
|
||||||
|
specification. This allows to restrict/cache resource values on a
|
||||||
|
per-directory level.
|
||||||
|
|
||||||
|
funcarg and setup discovery now happens at collection time
|
||||||
|
---------------------------------------------------------------------
|
||||||
|
|
||||||
|
pytest-2.X takes care to discover funcarg factories and setup_X methods
|
||||||
|
at collection time. This is more efficient especially for large test suites.
|
||||||
|
Moreover, a call to "py.test --collectonly" should be able to show
|
||||||
|
a lot of setup-information and thus presents a nice method to get an
|
||||||
|
overview of resource management in your project.
|
||||||
|
|
||||||
|
Implementation level
|
||||||
|
===================================================================
|
||||||
|
|
||||||
|
To implement the above new features, pytest-2.X grows some new hooks and
|
||||||
|
methods. At the time of writing V2 and without actually implementing
|
||||||
|
it, it is not clear how much of this new internal API will also be
|
||||||
|
exposed and advertised e. g. for plugin writers.
|
||||||
|
|
||||||
|
The main effort, however, will lie in revising what is done at
|
||||||
|
collection and what at test setup time. All funcarg factories and
|
||||||
|
xUnit setup methods need to be discovered at collection time
|
||||||
|
for the above mechanism to work. Additionally all test function
|
||||||
|
signatures need to be parsed in order to know which resources are
|
||||||
|
used. On the plus side, all previously collected fixtures and
|
||||||
|
test functions only need to be called, no discovery is neccessary
|
||||||
|
is required anymore.
|
||||||
|
|
||||||
|
the "request" object incorporates scope-specific behaviour
|
||||||
|
------------------------------------------------------------------
|
||||||
|
|
||||||
|
funcarg factories receive a request object to help with implementing
|
||||||
|
finalization and inspection of the requesting-context. If there is
|
||||||
|
no scoping is in effect, nothing much will change of the API behaviour.
|
||||||
|
However, with scoping the request object represents the according context.
|
||||||
|
Let's consider this example::
|
||||||
|
|
||||||
|
@pytest.mark.factory_scope("class")
|
||||||
|
def pytest_funcarg__db(request):
|
||||||
|
# ...
|
||||||
|
request.getfuncargvalue(...)
|
||||||
|
#
|
||||||
|
request.addfinalizer(db)
|
||||||
|
|
||||||
|
Due to the class-scope, the request object will:
|
||||||
|
|
||||||
|
- provide a ``None`` value for the ``request.function`` attribute.
|
||||||
|
- default to per-class finalization with the addfinalizer() call.
|
||||||
|
- raise a ScopeMismatchError if a more broadly scoped factory
|
||||||
|
wants to use a more tighly scoped factory (e.g. per-function)
|
||||||
|
|
||||||
|
In fact, the request object is likely going to provide a "node"
|
||||||
|
attribute, denoting the current collection node on which it internally
|
||||||
|
operates. (Prior to pytest-2.3 there already was an internal
|
||||||
|
_pyfuncitem).
|
||||||
|
|
||||||
|
As these are rather intuitive extensions, not much friction is expected
|
||||||
|
for test/plugin writers using the new scoping and parametrization mechanism.
|
||||||
|
It's, however, a serious internal effort to reorganize the pytest
|
||||||
|
implementation.
|
||||||
|
|
||||||
|
|
||||||
|
node.register_factory/getresource() methods
|
||||||
|
--------------------------------------------------------
|
||||||
|
|
||||||
|
In order to implement factory- and setup-method discovery at
|
||||||
|
collection time, a new node API will be introduced to allow
|
||||||
|
for factory registration and a getresource() call to obtain
|
||||||
|
created values. The exact details of this API remain subject
|
||||||
|
to experimentation. The basic idea is to introduce two new
|
||||||
|
methods to the Session class which is already available on all nodes
|
||||||
|
through the ``node.session`` attribute::
|
||||||
|
|
||||||
|
class Session:
|
||||||
|
def register_resource_factory(self, name, factory_or_list, scope):
|
||||||
|
""" register a resource factory for the given name.
|
||||||
|
|
||||||
|
:param name: Name of the resource.
|
||||||
|
:factory_or_list: a function or a list of functions creating
|
||||||
|
one or multiple resource values.
|
||||||
|
:param scope: a node instance. The factory will be only visisble
|
||||||
|
available for all descendant nodes.
|
||||||
|
specify the "session" instance for global availability
|
||||||
|
"""
|
||||||
|
|
||||||
|
def getresource(self, name, node):
|
||||||
|
""" get a named resource for the give node.
|
||||||
|
|
||||||
|
This method looks up a matching funcarg resource factory
|
||||||
|
and calls it.
|
||||||
|
"""
|
||||||
|
|
||||||
|
.. todo::
|
||||||
|
|
||||||
|
XXX While this new API (or some variant of it) may suffices to implement
|
||||||
|
all of the described new usage-level features, it remains unclear how the
|
||||||
|
existing "@parametrize" or "metafunc.parametrize()" calls will map to it.
|
||||||
|
These parametrize-approaches tie resource parametrization to the
|
||||||
|
function/funcargs-usage rather than to the factories.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
ISSUES
|
||||||
|
--------------------------
|
||||||
|
|
||||||
|
decorating a parametrized funcarg factory:
|
||||||
|
|
||||||
|
@pytest.mark.funcarg(scope="session", params=["mysql", "pg"])
|
||||||
|
def db(request):
|
||||||
|
...
|
||||||
|
class TestClass:
|
||||||
|
@pytest.mark.funcarg(scope="function")
|
||||||
|
def something(self, request):
|
||||||
|
session_db = request.getfuncargvalue("db")
|
||||||
|
...
|
||||||
|
|
||||||
|
Here the function-scoped "something" factory uses the session-scoped
|
||||||
|
"db" factory to perform some additional steps. The dependency, however,
|
||||||
|
is only visible at setup-time, when the factory actually gets called.
|
||||||
|
|
||||||
|
In order to allow parametrization at collection-time I see two ways:
|
||||||
|
|
||||||
|
- allow specifying dependencies in the funcarg-marker
|
||||||
|
- allow funcargs for factories as well
|
||||||
|
|
||||||
Loading…
Reference in New Issue