374 lines
12 KiB
Plaintext
374 lines
12 KiB
Plaintext
================================================
|
|
Customizing and Extending py.test
|
|
================================================
|
|
|
|
basic test configuration
|
|
===================================
|
|
|
|
Command line options
|
|
---------------------------------
|
|
|
|
You can see command line options by running::
|
|
|
|
py.test -h
|
|
|
|
This will display all available command line options
|
|
in your specific environment.
|
|
|
|
|
|
setting persistent option defaults
|
|
------------------------------------
|
|
|
|
py.test will lookup option values in this order:
|
|
|
|
* command line
|
|
* conftest.py files
|
|
* environment variables
|
|
|
|
To get an overview on existing names and settings type::
|
|
|
|
py.test --help-config
|
|
|
|
This will print information about all available options
|
|
in your environment, including your local plugins and
|
|
command line options.
|
|
|
|
.. _`function arguments`: funcargs.html
|
|
.. _`extensions`:
|
|
|
|
Plugin basics and project configuration
|
|
=============================================
|
|
|
|
.. _`local plugin`:
|
|
|
|
py.test implements all aspects of its functionality by calling `well specified
|
|
hooks`_. Hook functions are discovered in :file:`conftest.py` files or in
|
|
`named plugins`_. :file:`conftest.py` files are useful for keeping test
|
|
extensions and customizations close to test code.
|
|
|
|
local conftest.py plugins
|
|
--------------------------------------------------------------
|
|
|
|
local `conftest.py` plugins are usually automatically loaded and
|
|
registered but its contained hooks are only called when collecting or
|
|
running tests in files or directories next to or below the ``conftest.py``
|
|
file. Assume the following layout and content of files::
|
|
|
|
a/conftest.py:
|
|
def pytest_runtest_setup(item):
|
|
print ("setting up", item)
|
|
|
|
a/test_in_subdir.py:
|
|
def test_sub():
|
|
pass
|
|
|
|
test_flat.py:
|
|
def test_flat():
|
|
pass
|
|
|
|
Here is how you might run it::
|
|
|
|
py.test test_flat.py # will not show "setting up"
|
|
py.test a/test_sub.py # will show "setting up"
|
|
|
|
``py.test`` loads all ``conftest.py`` files upwards from the command
|
|
line file arguments. It usually looks up configuration values or hooks
|
|
right-to-left, i.e. the closer conftest files are checked before
|
|
the further away ones. This means you can have a ``conftest.py``
|
|
in your home directory to provide global configuration values.
|
|
|
|
.. Note::
|
|
if you have ``conftest.py`` files which do not reside in a
|
|
python package directory (one containing an ``__init__.py``) then
|
|
"import conftest" will be ambigous and should be avoided. **If you
|
|
want to import anything from a ``conftest.py`` file it is better
|
|
to put it inside a package to disambiguate.
|
|
|
|
.. _`named plugins`: plugin/index.html
|
|
|
|
Plugin discovery at tool startup
|
|
--------------------------------------------
|
|
|
|
py.test loads plugin modules at tool startup in the following way:
|
|
|
|
* by loading all plugins registered through `setuptools entry points`_.
|
|
|
|
* by reading the ``PYTEST_PLUGINS`` environment variable
|
|
and importing the comma-separated list of named plugins.
|
|
|
|
* by pre-scanning the command line for the ``-p name`` option
|
|
and loading the specified plugin before actual command line parsing.
|
|
|
|
* by loading all :file:`conftest.py` files as inferred by the command line
|
|
invocation (test files and all of its *parent* directories).
|
|
Note that ``conftest.py`` files from *sub* directories are by default
|
|
not loaded at tool startup.
|
|
|
|
* by recursively loading all plugins specified by the
|
|
``pytest_plugins`` variable in a ``conftest.py`` file
|
|
|
|
Requiring/Loading plugins in a test module or plugin
|
|
-------------------------------------------------------------
|
|
|
|
You can specify plugins in a test module or a plugin like this::
|
|
|
|
pytest_plugins = "name1", "name2",
|
|
|
|
When the test module or plugin is loaded the specified plugins
|
|
will be loaded. If you specify plugins without the ``pytest_``
|
|
prefix it will be automatically added. All plugin names
|
|
must be lowercase.
|
|
|
|
.. _`conftest.py plugin`:
|
|
.. _`conftestplugin`:
|
|
|
|
Writing per-project plugins (conftest.py)
|
|
------------------------------------------------------
|
|
|
|
The purpose of :file:`conftest.py` files is to allow project-specific
|
|
test customization. They thus make for a good place to implement
|
|
project-specific test related features through hooks. For example you may
|
|
set the `collect_ignore`_ variable depending on a command line option
|
|
by defining the following hook in a ``conftest.py`` file::
|
|
|
|
# ./conftest.py in your root or package dir
|
|
collect_ignore = ['hello', 'test_world.py']
|
|
def pytest_addoption(parser):
|
|
parser.addoption("--runall", action="store_true", default=False)
|
|
def pytest_configure(config):
|
|
if config.getvalue("runall"):
|
|
collect_ignore[:] = []
|
|
|
|
.. _`setuptools entry points`:
|
|
.. _registered:
|
|
|
|
Writing setuptools-registered plugins
|
|
------------------------------------------------------
|
|
|
|
.. _`Distribute`: http://pypi.python.org/pypi/distribute
|
|
.. _`setuptools`: http://pypi.python.org/pypi/setuptools
|
|
|
|
If you want to make your plugin publically available, you
|
|
can use `setuptools`_ or `Distribute`_ which both allow
|
|
to register an entry point. ``py.test`` will register
|
|
all objects with the ``pytest11`` entry point.
|
|
To make your plugin available you may insert the following
|
|
lines in your setuptools/distribute-based setup-invocation:
|
|
|
|
.. sourcecode:: python
|
|
|
|
# sample ./setup.py file
|
|
from setuptools import setup
|
|
|
|
setup(
|
|
name="myproject",
|
|
packages = ['myproject']
|
|
|
|
# the following makes a plugin available to py.test
|
|
entry_points = {
|
|
'pytest11': [
|
|
'name_of_plugin = myproject.pluginmodule',
|
|
]
|
|
},
|
|
)
|
|
|
|
If a package is installed with this setup, py.test will load
|
|
``myproject.pluginmodule`` under the ``name_of_plugin`` name
|
|
and use it as a plugin.
|
|
|
|
Accessing another plugin by name
|
|
--------------------------------------------
|
|
|
|
If a plugin wants to collaborate with code from
|
|
another plugin it can obtain a reference through
|
|
the plugin manager like this:
|
|
|
|
.. sourcecode:: python
|
|
|
|
plugin = config.pluginmanager.getplugin("name_of_plugin")
|
|
|
|
If you want to look at the names of existing plugins, use
|
|
the ``--traceconfig`` option.
|
|
|
|
.. _`well specified hooks`:
|
|
|
|
py.test hook reference
|
|
====================================
|
|
|
|
hook specification and validation
|
|
-----------------------------------------
|
|
|
|
py.test calls hook functions to implement initialization, running,
|
|
test execution and reporting. When py.test loads a plugin it validates
|
|
that all hook functions conform to their respective hook specification.
|
|
Each hook function name and its argument names need to match a hook
|
|
specification exactly but it is allowed for a hook function to accept
|
|
*less* parameters than specified. If you mistype argument names or the
|
|
hook name itself you get useful errors.
|
|
|
|
initialisation, command line and configuration hooks
|
|
--------------------------------------------------------------------
|
|
|
|
.. currentmodule:: pytest.hookspec
|
|
|
|
.. autofunction:: pytest_cmdline_parse
|
|
.. autofunction:: pytest_namespace
|
|
.. autofunction:: pytest_addoption
|
|
.. autofunction:: pytest_cmdline_main
|
|
.. autofunction:: pytest_configure
|
|
.. autofunction:: pytest_unconfigure
|
|
|
|
generic "runtest" hooks
|
|
------------------------------
|
|
|
|
All all runtest related hooks receive a :py:class:`pytest.collect.Item` object.
|
|
|
|
.. autofunction:: pytest_runtest_protocol
|
|
.. autofunction:: pytest_runtest_setup
|
|
.. autofunction:: pytest_runtest_call
|
|
.. autofunction:: pytest_runtest_teardown
|
|
.. autofunction:: pytest_runtest_makereport
|
|
|
|
For deeper understanding you may look at the default implementation of
|
|
these hooks in :py:mod:`pytest.plugin.runner` and maybe also
|
|
in :py:mod:`pytest.plugin.pdb` which intercepts creation
|
|
of reports in order to drop to interactive debugging.
|
|
|
|
The :py:mod:`pytest.plugin.terminal` reported specifically uses
|
|
the reporting hook to print information about a test run.
|
|
|
|
collection hooks
|
|
------------------------------
|
|
|
|
py.test calls the following hooks for collecting files and directories:
|
|
|
|
.. autofunction:: pytest_ignore_collect
|
|
.. autofunction:: pytest_collect_directory
|
|
.. autofunction:: pytest_collect_file
|
|
|
|
For influencing the collection of objects in Python modules
|
|
you can use the following hook:
|
|
|
|
.. autofunction:: pytest_pycollect_makeitem
|
|
|
|
|
|
reporting hooks
|
|
------------------------------
|
|
|
|
Collection related reporting hooks:
|
|
|
|
.. autofunction: pytest_collectstart
|
|
.. autofunction: pytest_log_itemcollect
|
|
.. autofunction: pytest_collectreport
|
|
.. autofunction: pytest_deselected
|
|
|
|
And here is the central hook for reporting about
|
|
test execution:
|
|
|
|
.. autofunction: pytest_runtest_logreport
|
|
|
|
The test collection tree
|
|
======================================================
|
|
|
|
|
|
Default filesystem test discovery
|
|
-----------------------------------------------
|
|
|
|
Test collection starts from specified paths or from the current
|
|
directory. All tests are collected ahead of running the first test.
|
|
(This used to be different in earlier versions of ``py.test`` where
|
|
collection and running was interweaved which made test randomization
|
|
and distributed testing harder).
|
|
|
|
Collection nodes which have children are called "Collectors" and otherwise
|
|
they are called "Items" or "test items". Here is an example of such a
|
|
tree::
|
|
|
|
example $ py.test --collectonly test_collectonly.py
|
|
<Directory 'example'>
|
|
<Module 'test_collectonly.py'>
|
|
<Function 'test_function'>
|
|
<Class 'TestClass'>
|
|
<Instance '()'>
|
|
<Function 'test_method'>
|
|
<Function 'test_anothermethod'>
|
|
|
|
By default all directories not starting with a dot are traversed,
|
|
looking for ``test_*.py`` and ``*_test.py`` files. Those Python
|
|
files are imported under their `package name`_.
|
|
|
|
The Module collector looks for test functions
|
|
and test classes and methods. Test functions and methods
|
|
are prefixed ``test`` by default. Test classes must
|
|
start with a capitalized ``Test`` prefix.
|
|
|
|
Customizing error messages
|
|
-------------------------------------------------
|
|
|
|
On test and collection nodes ``py.test`` will invoke
|
|
the ``node.repr_failure(excinfo)`` function which
|
|
you may override and make it return an error
|
|
representation string of your choice. It
|
|
will be reported as a (red) string.
|
|
|
|
.. _`package name`:
|
|
|
|
constructing the package name for test modules
|
|
-------------------------------------------------
|
|
|
|
Test modules are imported under their fully qualified
|
|
name. Given a filesystem ``fspath`` it is constructed as follows:
|
|
|
|
* walk the directories up to the last one that contains
|
|
an ``__init__.py`` file.
|
|
|
|
* perform ``sys.path.insert(0, basedir)``.
|
|
|
|
* import the root package as ``root``
|
|
|
|
Reference of important objects involved in hooks
|
|
===========================================================
|
|
|
|
.. autoclass:: pytest.plugin.config.Config
|
|
:members:
|
|
|
|
.. autoclass:: pytest.plugin.session.Item
|
|
:inherited-members:
|
|
|
|
.. autoclass:: pytest.plugin.session.Node
|
|
:members:
|
|
|
|
.. autoclass:: pytest.plugin.runner.CallInfo
|
|
:members:
|
|
|
|
.. autoclass:: pytest.plugin.runner.TestReport
|
|
:members:
|
|
|
|
|
|
conftest.py configuration files
|
|
=================================================
|
|
|
|
conftest.py reference docs
|
|
|
|
A unique feature of py.test are its ``conftest.py`` files which allow
|
|
project and directory specific customizations to testing.
|
|
|
|
* `set option defaults`_
|
|
|
|
or set particular variables to influence the testing process:
|
|
|
|
* ``pytest_plugins``: list of named plugins to load
|
|
|
|
* ``collect_ignore``: list of paths to ignore during test collection, relative to the containing ``conftest.py`` file
|
|
|
|
* ``rsyncdirs``: list of to-be-rsynced directories for distributed
|
|
testing, relative to the containing ``conftest.py`` file.
|
|
|
|
You may put a conftest.py files in your project root directory or into
|
|
your package directory if you want to add project-specific test options.
|
|
|
|
|
|
.. _`specify funcargs`: funcargs.html#application-setup-tutorial-example
|
|
|
|
.. _`set option defaults`:
|