492 lines
16 KiB
Plaintext
492 lines
16 KiB
Plaintext
================================================
|
|
Customizing and Extending py.test
|
|
================================================
|
|
|
|
.. contents::
|
|
:local:
|
|
:depth: 2
|
|
|
|
basic test configuration
|
|
===================================
|
|
|
|
available command line options
|
|
---------------------------------
|
|
|
|
You can see command line options by running::
|
|
|
|
py.test -h
|
|
|
|
This will display all available command line options
|
|
in your specific environment.
|
|
|
|
.. _`project-specific test configuration`:
|
|
.. _`collect_ignore`:
|
|
|
|
conftest.py: project specific hooks and configuration
|
|
--------------------------------------------------------
|
|
|
|
A unique feature of py.test are its ``conftest.py`` files which
|
|
allow to:
|
|
|
|
* `set option defaults`_
|
|
|
|
* `implement hooks`_
|
|
|
|
* `specify funcargs`_
|
|
|
|
or set particular variables to influence the testing process:
|
|
|
|
* ``pytest_plugins``: list of named plugins to load
|
|
|
|
* ``collect_ignore``: list of paths to ignore during test collection, relative to the containing ``conftest.py`` file
|
|
|
|
* ``rsyncdirs``: list of to-be-rsynced directories for distributed
|
|
testing, relative to the containing ``conftest.py`` file.
|
|
|
|
You may put a conftest.py files in your project root directory or into
|
|
your package directory if you want to add project-specific test options.
|
|
|
|
``py.test`` loads all ``conftest.py`` files upwards from the command
|
|
line file arguments. It usually looks up configuration values
|
|
right-to-left, i.e. the closer conftest files will be checked first.
|
|
This means you can have a ``conftest.py`` in your very home directory to
|
|
have some global configuration values.
|
|
|
|
.. _`specify funcargs`: funcargs.html#application-setup-tutorial-example
|
|
|
|
.. _`set option defaults`:
|
|
|
|
setting persistent option defaults
|
|
------------------------------------
|
|
|
|
py.test will lookup option values in this order:
|
|
|
|
* command line
|
|
* conftest.py files
|
|
* environment variables
|
|
|
|
To find out about the particular switches and type::
|
|
|
|
py.test --help-config
|
|
|
|
This will print information about all options in your
|
|
environment, including your local plugins.
|
|
|
|
.. _`basetemp`:
|
|
|
|
Temporary directories
|
|
-------------------------------------------
|
|
|
|
You can create directories by calling one of two methods
|
|
on the config object:
|
|
|
|
- ``config.mktemp(basename)``: create and return a new tempdir
|
|
|
|
- ``config.ensuretemp(basename)``: create or return a new tempdir
|
|
|
|
temporary directories are created as sub directories of a per-session
|
|
testdir and will keep around the directories of the last three test
|
|
runs. You can set the base temporary directory through the command line
|
|
`--basetemp`` option. When distributing tests on the same machine,
|
|
``py.test`` takes care to configure a basetemp directory for the sub
|
|
processes such that all temporary data lands below below a single
|
|
per-test run basetemp directory.
|
|
|
|
.. _`function arguments`: funcargs.html
|
|
.. _`extensions`:
|
|
|
|
Plugin basics and project configuration
|
|
=============================================
|
|
|
|
.. _`local plugin`:
|
|
|
|
project specific "local" or named "global" plugins
|
|
--------------------------------------------------------------
|
|
|
|
py.test implements much of its functionality by calling `well specified
|
|
hooks`_. Python modules which contain such hook functions are called
|
|
plugins. Hook functions are discovered in ``conftest.py`` files or in
|
|
`named plugins`_. ``conftest.py`` files are sometimes called
|
|
"anonymous" or conftest plugins. They are useful for keeping test
|
|
extensions close to your application. Named plugins are normal python
|
|
modules or packages that can be distributed separately. Named plugins
|
|
need to follow a naming pattern; they have an all lowercase ``pytest_``
|
|
prefixed name. While conftest plugins are discovered automatically,
|
|
named plugins must be explicitely specified.
|
|
|
|
.. _`named plugins`: plugin/index.html
|
|
|
|
.. _`tool startup`:
|
|
.. _`loaded at tool startup`:
|
|
.. _`test tool starts up`:
|
|
|
|
Plugin discovery at tool startup
|
|
--------------------------------------------
|
|
|
|
py.test loads plugin modules at tool startup in the following way:
|
|
|
|
* by loading all plugins registered through `setuptools entry points`_.
|
|
|
|
* by reading the ``PYTEST_PLUGINS`` environment variable
|
|
and importing the comma-separated list of named plugins.
|
|
|
|
* by pre-scanning the command line for the ``-p name`` option
|
|
and loading the specified plugin before actual command line parsing.
|
|
|
|
* by loading all `conftest.py plugin`_ files as inferred by the command line
|
|
invocation (test files and all of its parent directories).
|
|
Note that ``conftest.py`` files from sub directories are loaded
|
|
during test collection and not at tool startup.
|
|
|
|
* by recursively loading all plugins specified by the
|
|
``pytest_plugins`` variable in a ``conftest.py`` file
|
|
|
|
Specifying plugins in a test module or plugin
|
|
-----------------------------------------------
|
|
|
|
You can specify plugins in a test module or a plugin like this:
|
|
|
|
.. sourcecode:: python
|
|
|
|
pytest_plugins = "name1", "name2",
|
|
|
|
When the test module or plugin is loaded the specified plugins
|
|
will be loaded. If you specify plugins without the ``pytest_``
|
|
prefix it will be automatically added. All plugin names
|
|
must be lowercase.
|
|
|
|
.. _`conftest.py plugin`:
|
|
.. _`conftestplugin`:
|
|
|
|
Writing per-project plugins (conftest.py)
|
|
------------------------------------------------------
|
|
|
|
The purpose of ``conftest.py`` files is to allow `project-specific
|
|
test configuration`_. They thus make for a good place to implement
|
|
project-specific test related features through hooks. For example you may
|
|
set the `collect_ignore`_ variable depending on a command line option
|
|
by defining the following hook in a ``conftest.py`` file:
|
|
|
|
.. _`exclude-file-example`:
|
|
|
|
.. sourcecode:: python
|
|
|
|
# ./conftest.py in your root or package dir
|
|
collect_ignore = ['hello', 'test_world.py']
|
|
def pytest_addoption(parser):
|
|
parser.addoption("--runall", action="store_true", default=False)
|
|
def pytest_configure(config):
|
|
if config.getvalue("runall"):
|
|
collect_ignore[:] = []
|
|
|
|
.. _`setuptools entry points`:
|
|
|
|
Writing setuptools-registered plugins
|
|
------------------------------------------------------
|
|
|
|
.. _`Distribute`: http://pypi.python.org/pypi/distribute
|
|
.. _`setuptools`: http://pypi.python.org/pypi/setuptools
|
|
|
|
If you want to make your plugin publically available, you
|
|
can use `setuptools`_ or `Distribute`_ which both allow
|
|
to register an entry point. ``py.test`` will register
|
|
all objects with the ``pytest11`` entry point.
|
|
To make your plugin available you may insert the following
|
|
lines in your setuptools/distribute-based setup-invocation:
|
|
|
|
.. sourcecode:: python
|
|
|
|
# sample ./setup.py file
|
|
from setuptools import setup
|
|
|
|
setup(
|
|
name="myproject",
|
|
packages = ['myproject']
|
|
|
|
# the following makes a plugin available to py.test
|
|
entry_points = {
|
|
'pytest11': [
|
|
'name_of_plugin = myproject.pluginmodule',
|
|
]
|
|
},
|
|
)
|
|
|
|
If a package is installed with this setup, py.test will load
|
|
``myproject.pluginmodule`` under the ``name_of_plugin`` name
|
|
and use it as a plugin.
|
|
|
|
Accessing another plugin by name
|
|
--------------------------------------------
|
|
|
|
If a plugin wants to collaborate with code from
|
|
another plugin it can obtain a reference through
|
|
the plugin manager like this:
|
|
|
|
.. sourcecode:: python
|
|
|
|
plugin = config.pluginmanager.getplugin("name_of_plugin")
|
|
|
|
If you want to look at the names of existing plugins, use
|
|
the ``--traceconfig`` option.
|
|
|
|
.. _`well specified hooks`:
|
|
.. _`implement hooks`:
|
|
|
|
Important py.test hooks
|
|
====================================
|
|
|
|
py.test calls hooks functions to implement its `test collection`_,
|
|
running and reporting process. When py.test loads a plugin it validates
|
|
that all hook functions conform to the `hook definition specification`_.
|
|
|
|
The hook function name and its
|
|
argument names need to match exactly but it is allowed for an implementation
|
|
to accept *less* parameters. You'll get useful errors on mistyped hook or
|
|
argument names. Read on for some introductory information on particular
|
|
hooks. It's sensible to look at existing plugins so see example usages
|
|
and start off with your own plugin.
|
|
|
|
.. _`hook definition specification`: plugin/hookspec.html
|
|
|
|
.. _`configuration hooks`:
|
|
|
|
command line parsing and configuration hooks
|
|
--------------------------------------------------------------------
|
|
|
|
When the `test tool starts up`_ it will invoke all hooks that add
|
|
command line options in the python standard optparse style.
|
|
|
|
.. sourcecode:: python
|
|
|
|
def pytest_addoption(parser):
|
|
""" add command line options. """"
|
|
parser.addoption("--myopt", dest="myopt", action="store_true")
|
|
|
|
After all these hooks have been called, the command line is parser
|
|
and a ``config`` object is created and another hook is invoked,
|
|
for example:
|
|
|
|
.. sourcecode:: python
|
|
|
|
def pytest_configure(config):
|
|
config.getvalue("myopt")
|
|
|
|
When the test run finishes this corresponding finalizer hook is called:
|
|
|
|
.. sourcecode:: python
|
|
|
|
def pytest_unconfigure(config):
|
|
...
|
|
|
|
|
|
adding global py.test helpers and functionality
|
|
--------------------------------------------------------------------
|
|
|
|
If you want to make global helper functions or objects available
|
|
to your test code you can implement:
|
|
|
|
.. sourcecode:: python
|
|
|
|
def pytest_namespace():
|
|
""" return dictionary with items to be made available on py.test. namespace """
|
|
|
|
All such returned items will be made available directly on
|
|
the ``py.test`` namespace.
|
|
|
|
If you want to provide helpers that are specific to a test function run or need
|
|
to be setup per test function run, please refer to the `funcargs mechanism`_.
|
|
|
|
.. _`funcargs mechanism`: funcargs.html
|
|
|
|
|
|
generic "runtest" hooks
|
|
------------------------------
|
|
|
|
Each test item is usually executed by calling the following three hooks:
|
|
|
|
.. sourcecode:: python
|
|
|
|
pytest_runtest_setup(item)
|
|
pytest_runtest_call(item)
|
|
pytest_runtest_teardown(item)
|
|
|
|
For each of the three invocations a `call object`_ encapsulates
|
|
information about the outcome of the call and is subsequently used
|
|
to make a report object:
|
|
|
|
.. sourcecode:: python
|
|
|
|
report = hook.pytest_runtest_makereport(item, call)
|
|
|
|
For example, the `pytest_pdb plugin`_ uses this hook to activate
|
|
interactive debugging on failures when ``--pdb`` is specified on the
|
|
command line.
|
|
|
|
Usually three reports will be generated for a single test item for each
|
|
of the three runtest hooks respectively. If ``pytest_runtest_setup``
|
|
fails then ``pytest_runtest_teardown`` will be called but not
|
|
``pytest_runtest_call``.
|
|
|
|
Each of the up to three reports is eventually fed to the logreport hook:
|
|
|
|
.. sourcecode:: python
|
|
|
|
pytest_runtest_logreport(report)
|
|
|
|
A ``report`` object contains status and reporting information:
|
|
|
|
.. sourcecode:: python
|
|
|
|
report.longrepr = string/lines/object to print
|
|
report.when = "setup", "call" or "teardown"
|
|
report.shortrepr = letter for progress-report
|
|
report.passed = True or False
|
|
report.failed = True or False
|
|
report.skipped = True or False
|
|
|
|
The `terminal plugin`_ uses this hook to print information
|
|
about a test run.
|
|
|
|
The whole protocol described here is implemented via this hook:
|
|
|
|
.. sourcecode:: python
|
|
|
|
pytest_runtest_protocol(item) -> True
|
|
|
|
.. _`call object`:
|
|
|
|
The call object contains information about a performed call:
|
|
|
|
.. sourcecode:: python
|
|
|
|
call.excinfo = ExceptionInfo object or None
|
|
call.when = "setup", "call" or "teardown"
|
|
call.outerr = None or tuple of strings representing captured stdout/stderr
|
|
|
|
.. _`pytest_pdb plugin`: plugin/pdb.html
|
|
.. _`terminal plugin`: plugin/terminal.html
|
|
|
|
|
|
generic collection hooks
|
|
------------------------------
|
|
|
|
py.test calls the following two fundamental hooks for collecting files and directories:
|
|
|
|
.. sourcecode:: python
|
|
|
|
def pytest_collect_directory(path, parent):
|
|
""" return Collection node or None for the given path. """
|
|
|
|
def pytest_collect_file(path, parent):
|
|
""" return Collection node or None for the given path. """
|
|
|
|
Both return a `collection node`_ for a given path. All returned
|
|
nodes from all hook implementations will participate in the
|
|
collection and running protocol. The ``parent`` object is
|
|
the parent node and may be used to access command line
|
|
options via the ``parent.config`` object.
|
|
|
|
|
|
Python test function and module hooks
|
|
----------------------------------------------------
|
|
|
|
For influencing the collection of objects in Python modules
|
|
you can use the following hook:
|
|
|
|
.. sourcecode:: python
|
|
|
|
def pytest_pycollect_makeitem(collector, name, obj):
|
|
""" return custom item/collector for a python object in a module, or None. """
|
|
|
|
This hook will be called for each Python object in a collected
|
|
Python module. The return value is a custom `collection node`_ or None.
|
|
|
|
.. XXX or ``False`` if you want to indicate that the given item should not be collected.
|
|
|
|
|
|
Gateway initialization (distributed testing)
|
|
----------------------------------------------------
|
|
|
|
(alpha) For distributed testing it can be useful to prepare the
|
|
remote environment. For this you can implement the newgateway hook:
|
|
|
|
.. sourcecode:: python
|
|
|
|
def pytest_gwmanage_newgateway(gateway, platinfo):
|
|
""" called after a gateway is instantiated. """
|
|
|
|
The ``gateway`` object here has a ``spec`` attribute which is an ``execnet.XSpec``
|
|
object, which has attributes that map key/values as specified from a ``--txspec``
|
|
option. The platinfo object is a dictionary with information about the remote process:
|
|
|
|
* ``version``: remote python's ``sys.version_info``
|
|
* ``platform``: remote ``sys.platform``
|
|
* ``cwd``: remote ``os.getcwd``
|
|
|
|
|
|
.. _`collection process`:
|
|
.. _`collection node`:
|
|
.. _`test collection`:
|
|
|
|
|
|
Test Collection process
|
|
======================================================
|
|
|
|
the collection tree
|
|
---------------------------------
|
|
|
|
The collecting process is iterative so that distribution
|
|
and execution of tests can start as soon as the first test
|
|
item is collected. Collection nodes with children are
|
|
called "Collectors" and terminal nodes are called "Items".
|
|
Here is an example of such a tree, generated with the
|
|
command ``py.test --collectonly py/xmlobj``::
|
|
|
|
<Directory 'xmlobj'>
|
|
<Directory 'testing'>
|
|
<Module 'test_html.py' (py.__.xmlobj.testing.test_html)>
|
|
<Function 'test_html_name_stickyness'>
|
|
<Function 'test_stylenames'>
|
|
<Function 'test_class_None'>
|
|
<Function 'test_alternating_style'>
|
|
<Module 'test_xml.py' (py.__.xmlobj.testing.test_xml)>
|
|
<Function 'test_tag_with_text'>
|
|
<Function 'test_class_identity'>
|
|
<Function 'test_tag_with_text_and_attributes'>
|
|
<Function 'test_tag_with_subclassed_attr_simple'>
|
|
<Function 'test_tag_nested'>
|
|
<Function 'test_tag_xmlname'>
|
|
|
|
By default all directories not starting with a dot are traversed,
|
|
looking for ``test_*.py`` and ``*_test.py`` files. Those Python
|
|
files are imported under their `package name`_.
|
|
|
|
The Module collector looks for test functions
|
|
and test classes and methods. Test functions and methods
|
|
are prefixed ``test`` by default. Test classes must
|
|
start with a capitalized ``Test`` prefix.
|
|
|
|
Customizing error messages
|
|
-------------------------------------------------
|
|
|
|
On test and collection nodes ``py.test`` will invoke
|
|
the ``node.repr_failure(excinfo)`` function which
|
|
you may override and make it return an error
|
|
representation string of your choice. It
|
|
will be reported as a (red) string.
|
|
|
|
.. _`package name`:
|
|
|
|
constructing the package name for test modules
|
|
-------------------------------------------------
|
|
|
|
Test modules are imported under their fully qualified
|
|
name. Given a filesystem ``fspath`` it is constructed as follows:
|
|
|
|
* walk the directories up to the last one that contains
|
|
an ``__init__.py`` file.
|
|
|
|
* perform ``sys.path.insert(0, basedir)``.
|
|
|
|
* import the root package as ``root``
|