p53209761
/
test_ok2
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Moved more sections from reference to how-to. 

* re-ordered and grouped how-to guides in how-to landing page
* retitled moved documents appropriately
* separated writing plugins/writing hook functions into two documents
This commit is contained in:
Daniele Procida 2021-03-13 23:11:48 +00:00
parent 63bc49de70
commit d171a3f52d
15 changed files with 400 additions and 369 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

39
doc/en/contents.rst
View File

@ -24,18 +24,30 @@ How-to guides
:maxdepth: 2
how-to/usage
how-to/existingtestsuite
how-to/assert
how-to/mark
how-to/monkeypatch
how-to/tmpdir
how-to/capture
how-to/skipping
how-to/parametrize
how-to/plugins
how-to/nose
how-to/bash-completion
how-to/fixtures
how-to/mark
how-to/parametrize
how-to/tmpdir
how-to/monkeypatch
how-to/doctest
how-to/cache
how-to/logging
how-to/capture-stdout-stderr
how-to/capture-warnings
how-to/skipping
how-to/plugins
how-to/writing_plugins
how-to/writing_hook_functions
how-to/existingtestsuite
how-to/unittest
how-to/nose
how-to/xunit_setup
how-to/bash-completion
Reference guides
@ -45,14 +57,7 @@ Reference guides
:maxdepth: 2
reference/fixtures
reference/warnings
reference/doctest
reference/cache
reference/unittest
reference/xunit_setup
reference/plugin_list
reference/writing_plugins
reference/logging
reference/customize
reference/reference

1
doc/en/explanation/fixtures.rst
View File

@ -6,6 +6,7 @@ About fixtures
.. seealso:: :ref:`how-to-fixtures`
.. seealso:: :ref:`Fixtures reference <reference-fixtures>`
pytest fixtures are designed to be explicit, modular and scalable.
What fixtures are
-----------------

6
doc/en/how-to/assert.rst
View File

@ -1,16 +1,14 @@
.. _`assert`:
How to write and report assertions in tests
==================================================
.. _`assertfeedback`:
.. _`assert with the assert statement`:
.. _`assert`:
Asserting with the ``assert`` statement
---------------------------------------------------------
``pytest`` allows you to use the standard python ``assert`` for verifying
``pytest`` allows you to use the standard Python ``assert`` for verifying
expectations and values in Python tests. For example, you can write the
following:

4
doc/en/reference/cache.rst → doc/en/how-to/cache.rst
View File

@ -2,8 +2,8 @@
.. _cache:
Cache: working with cross-testrun state
=======================================
How to re-run failed tests and maintain state between test runs
===============================================================

0
doc/en/how-to/capture.rst → doc/en/how-to/capture-stdout-stderr.rst
View File

4
doc/en/reference/warnings.rst → doc/en/how-to/capture-warnings.rst
View File

@ -1,7 +1,7 @@
.. _`warnings`:
Warnings Capture
================
How to capture warnings
=======================

2
doc/en/reference/doctest.rst → doc/en/how-to/doctest.rst
View File

@ -1,6 +1,6 @@
.. _doctest:
Doctest integration for modules and test files
How to run doctests
=========================================================
By default, all files matching the ``test*.txt`` pattern will

61
doc/en/how-to/index.rst
View File

@ -3,19 +3,60 @@
How-to guides
================
Core pytest functionality
-------------------------
.. toctree::
:maxdepth: 1
usage
existingtestsuite
assert
mark
monkeypatch
tmpdir
capture
skipping
parametrize
plugins
nose
bash-completion
fixtures
mark
parametrize
tmpdir
monkeypatch
doctest
cache
test output and outcomes
----------------------------
.. toctree::
:maxdepth: 1
logging
capture-stdout-stderr
capture-warnings
skipping
Plugins
----------------------------
.. toctree::
:maxdepth: 1
plugins
writing_plugins
writing_hook_functions
pytest and other test systems
-----------------------------
.. toctree::
:maxdepth: 1
existingtestsuite
unittest
nose
xunit_setup
pytest development environment
------------------------------
.. toctree::
:maxdepth: 1
bash-completion

7
doc/en/reference/logging.rst → doc/en/how-to/logging.rst
View File

@ -1,10 +1,7 @@
.. _logging:
Logging
-------
How to manage logging
---------------------
pytest captures log messages of level ``WARNING`` or above automatically and displays them in their own section
for each failed test in the same manner as captured stdout and stderr.

4
doc/en/reference/unittest.rst → doc/en/how-to/unittest.rst
View File

@ -2,8 +2,8 @@
.. _`unittest.TestCase`:
.. _`unittest`:
unittest.TestCase Support
=========================
How to use ``unittest``-based tests with pytest
===============================================
``pytest`` supports running Python ``unittest``-based tests out of the box.
It's meant for leveraging existing ``unittest``-based test suites

313
doc/en/how-to/writing_hook_functions.rst Normal file
View File

@ -0,0 +1,313 @@
.. _`writinghooks`:
Writing hook functions
======================
.. _validation:
hook function validation and execution
--------------------------------------
pytest calls hook functions from registered plugins for any
given hook specification. Let's look at a typical hook function
for the ``pytest_collection_modifyitems(session, config,
items)`` hook which pytest calls after collection of all test items is
completed.
When we implement a ``pytest_collection_modifyitems`` function in our plugin
pytest will during registration verify that you use argument
names which match the specification and bail out if not.
Let's look at a possible implementation:
.. code-block:: python
def pytest_collection_modifyitems(config, items):
# called after collection is completed
# you can modify the ``items`` list
...
Here, ``pytest`` will pass in ``config`` (the pytest config object)
and ``items`` (the list of collected test items) but will not pass
in the ``session`` argument because we didn't list it in the function
signature. This dynamic "pruning" of arguments allows ``pytest`` to
be "future-compatible": we can introduce new hook named parameters without
breaking the signatures of existing hook implementations. It is one of
the reasons for the general long-lived compatibility of pytest plugins.
Note that hook functions other than ``pytest_runtest_*`` are not
allowed to raise exceptions. Doing so will break the pytest run.
.. _firstresult:
firstresult: stop at first non-None result
-------------------------------------------
Most calls to ``pytest`` hooks result in a **list of results** which contains
all non-None results of the called hook functions.
Some hook specifications use the ``firstresult=True`` option so that the hook
call only executes until the first of N registered functions returns a
non-None result which is then taken as result of the overall hook call.
The remaining hook functions will not be called in this case.
.. _`hookwrapper`:
hookwrapper: executing around other hooks
-------------------------------------------------
.. currentmodule:: _pytest.core
pytest plugins can implement hook wrappers which wrap the execution
of other hook implementations. A hook wrapper is a generator function
which yields exactly once. When pytest invokes hooks it first executes
hook wrappers and passes the same arguments as to the regular hooks.
At the yield point of the hook wrapper pytest will execute the next hook
implementations and return their result to the yield point in the form of
a :py:class:`Result <pluggy._Result>` instance which encapsulates a result or
exception info. The yield point itself will thus typically not raise
exceptions (unless there are bugs).
Here is an example definition of a hook wrapper:
.. code-block:: python
import pytest
@pytest.hookimpl(hookwrapper=True)
def pytest_pyfunc_call(pyfuncitem):
do_something_before_next_hook_executes()
outcome = yield
# outcome.excinfo may be None or a (cls, val, tb) tuple
res = outcome.get_result() # will raise if outcome was exception
post_process_result(res)
outcome.force_result(new_res) # to override the return value to the plugin system
Note that hook wrappers don't return results themselves, they merely
perform tracing or other side effects around the actual hook implementations.
If the result of the underlying hook is a mutable object, they may modify
that result but it's probably better to avoid it.
For more information, consult the
:ref:`pluggy documentation about hookwrappers <pluggy:hookwrappers>`.
.. _plugin-hookorder:
Hook function ordering / call example
-------------------------------------
For any given hook specification there may be more than one
implementation and we thus generally view ``hook`` execution as a
``1:N`` function call where ``N`` is the number of registered functions.
There are ways to influence if a hook implementation comes before or
after others, i.e. the position in the ``N``-sized list of functions:
.. code-block:: python
# Plugin 1
@pytest.hookimpl(tryfirst=True)
def pytest_collection_modifyitems(items):
# will execute as early as possible
...
# Plugin 2
@pytest.hookimpl(trylast=True)
def pytest_collection_modifyitems(items):
# will execute as late as possible
...
# Plugin 3
@pytest.hookimpl(hookwrapper=True)
def pytest_collection_modifyitems(items):
# will execute even before the tryfirst one above!
outcome = yield
# will execute after all non-hookwrappers executed
Here is the order of execution:
1. Plugin3's pytest_collection_modifyitems called until the yield point
because it is a hook wrapper.
2. Plugin1's pytest_collection_modifyitems is called because it is marked
with ``tryfirst=True``.
3. Plugin2's pytest_collection_modifyitems is called because it is marked
with ``trylast=True`` (but even without this mark it would come after
Plugin1).
4. Plugin3's pytest_collection_modifyitems then executing the code after the yield
point. The yield receives a :py:class:`Result <pluggy._Result>` instance which encapsulates
the result from calling the non-wrappers. Wrappers shall not modify the result.
It's possible to use ``tryfirst`` and ``trylast`` also in conjunction with
``hookwrapper=True`` in which case it will influence the ordering of hookwrappers
among each other.
Declaring new hooks
------------------------
.. note::
This is a quick overview on how to add new hooks and how they work in general, but a more complete
overview can be found in `the pluggy documentation <https://pluggy.readthedocs.io/en/latest/>`__.
.. currentmodule:: _pytest.hookspec
Plugins and ``conftest.py`` files may declare new hooks that can then be
implemented by other plugins in order to alter behaviour or interact with
the new plugin:
.. autofunction:: pytest_addhooks
:noindex:
Hooks are usually declared as do-nothing functions that contain only
documentation describing when the hook will be called and what return values
are expected. The names of the functions must start with `pytest_` otherwise pytest won't recognize them.
Here's an example. Let's assume this code is in the ``sample_hook.py`` module.
.. code-block:: python
def pytest_my_hook(config):
"""
Receives the pytest config and does things with it
"""
To register the hooks with pytest they need to be structured in their own module or class. This
class or module can then be passed to the ``pluginmanager`` using the ``pytest_addhooks`` function
(which itself is a hook exposed by pytest).
.. code-block:: python
def pytest_addhooks(pluginmanager):
""" This example assumes the hooks are grouped in the 'sample_hook' module. """
from my_app.tests import sample_hook
pluginmanager.add_hookspecs(sample_hook)
For a real world example, see `newhooks.py`_ from `xdist <https://github.com/pytest-dev/pytest-xdist>`_.
.. _`newhooks.py`: https://github.com/pytest-dev/pytest-xdist/blob/974bd566c599dc6a9ea291838c6f226197208b46/xdist/newhooks.py
Hooks may be called both from fixtures or from other hooks. In both cases, hooks are called
through the ``hook`` object, available in the ``config`` object. Most hooks receive a
``config`` object directly, while fixtures may use the ``pytestconfig`` fixture which provides the same object.
.. code-block:: python
@pytest.fixture()
def my_fixture(pytestconfig):
# call the hook called "pytest_my_hook"
# 'result' will be a list of return values from all registered functions.
result = pytestconfig.hook.pytest_my_hook(config=pytestconfig)
.. note::
Hooks receive parameters using only keyword arguments.
Now your hook is ready to be used. To register a function at the hook, other plugins or users must
now simply define the function ``pytest_my_hook`` with the correct signature in their ``conftest.py``.
Example:
.. code-block:: python
def pytest_my_hook(config):
"""
Print all active hooks to the screen.
"""
print(config.hook)
.. _`addoptionhooks`:
Using hooks in pytest_addoption
-------------------------------
Occasionally, it is necessary to change the way in which command line options
are defined by one plugin based on hooks in another plugin. For example,
a plugin may expose a command line option for which another plugin needs
to define the default value. The pluginmanager can be used to install and
use hooks to accomplish this. The plugin would define and add the hooks
and use pytest_addoption as follows:
.. code-block:: python
# contents of hooks.py
# Use firstresult=True because we only want one plugin to define this
# default value
@hookspec(firstresult=True)
def pytest_config_file_default_value():
""" Return the default value for the config file command line option. """
# contents of myplugin.py
def pytest_addhooks(pluginmanager):
""" This example assumes the hooks are grouped in the 'hooks' module. """
from . import hook
pluginmanager.add_hookspecs(hook)
def pytest_addoption(parser, pluginmanager):
default_value = pluginmanager.hook.pytest_config_file_default_value()
parser.addoption(
"--config-file",
help="Config file to use, defaults to %(default)s",
default=default_value,
)
The conftest.py that is using myplugin would simply define the hook as follows:
.. code-block:: python
def pytest_config_file_default_value():
return "config.yaml"
Optionally using hooks from 3rd party plugins
---------------------------------------------
Using new hooks from plugins as explained above might be a little tricky
because of the standard :ref:`validation mechanism <validation>`:
if you depend on a plugin that is not installed, validation will fail and
the error message will not make much sense to your users.
One approach is to defer the hook implementation to a new plugin instead of
declaring the hook functions directly in your plugin module, for example:
.. code-block:: python
# contents of myplugin.py
class DeferPlugin:
"""Simple plugin to defer pytest-xdist hook functions."""
def pytest_testnodedown(self, node, error):
"""standard xdist hook function."""
def pytest_configure(config):
if config.pluginmanager.hasplugin("xdist"):
config.pluginmanager.register(DeferPlugin())
This has the added benefit of allowing you to conditionally install hooks
depending on which plugins are installed.

317
doc/en/reference/writing_plugins.rst → doc/en/how-to/writing_plugins.rst
View File

@ -454,320 +454,3 @@ Additionally it is possible to copy examples for an example folder before runnin
For more information about the result object that ``runpytest()`` returns, and
the methods that it provides please check out the :py:class:`RunResult
<_pytest.pytester.RunResult>` documentation.
.. _`writinghooks`:
Writing hook functions
======================
.. _validation:
hook function validation and execution
--------------------------------------
pytest calls hook functions from registered plugins for any
given hook specification. Let's look at a typical hook function
for the ``pytest_collection_modifyitems(session, config,
items)`` hook which pytest calls after collection of all test items is
completed.
When we implement a ``pytest_collection_modifyitems`` function in our plugin
pytest will during registration verify that you use argument
names which match the specification and bail out if not.
Let's look at a possible implementation:
.. code-block:: python
def pytest_collection_modifyitems(config, items):
# called after collection is completed
# you can modify the ``items`` list
...
Here, ``pytest`` will pass in ``config`` (the pytest config object)
and ``items`` (the list of collected test items) but will not pass
in the ``session`` argument because we didn't list it in the function
signature. This dynamic "pruning" of arguments allows ``pytest`` to
be "future-compatible": we can introduce new hook named parameters without
breaking the signatures of existing hook implementations. It is one of
the reasons for the general long-lived compatibility of pytest plugins.
Note that hook functions other than ``pytest_runtest_*`` are not
allowed to raise exceptions. Doing so will break the pytest run.
.. _firstresult:
firstresult: stop at first non-None result
-------------------------------------------
Most calls to ``pytest`` hooks result in a **list of results** which contains
all non-None results of the called hook functions.
Some hook specifications use the ``firstresult=True`` option so that the hook
call only executes until the first of N registered functions returns a
non-None result which is then taken as result of the overall hook call.
The remaining hook functions will not be called in this case.
.. _`hookwrapper`:
hookwrapper: executing around other hooks
-------------------------------------------------
.. currentmodule:: _pytest.core
pytest plugins can implement hook wrappers which wrap the execution
of other hook implementations. A hook wrapper is a generator function
which yields exactly once. When pytest invokes hooks it first executes
hook wrappers and passes the same arguments as to the regular hooks.
At the yield point of the hook wrapper pytest will execute the next hook
implementations and return their result to the yield point in the form of
a :py:class:`Result <pluggy._Result>` instance which encapsulates a result or
exception info. The yield point itself will thus typically not raise
exceptions (unless there are bugs).
Here is an example definition of a hook wrapper:
.. code-block:: python
import pytest
@pytest.hookimpl(hookwrapper=True)
def pytest_pyfunc_call(pyfuncitem):
do_something_before_next_hook_executes()
outcome = yield
# outcome.excinfo may be None or a (cls, val, tb) tuple
res = outcome.get_result() # will raise if outcome was exception
post_process_result(res)
outcome.force_result(new_res) # to override the return value to the plugin system
Note that hook wrappers don't return results themselves, they merely
perform tracing or other side effects around the actual hook implementations.
If the result of the underlying hook is a mutable object, they may modify
that result but it's probably better to avoid it.
For more information, consult the
:ref:`pluggy documentation about hookwrappers <pluggy:hookwrappers>`.
.. _plugin-hookorder:
Hook function ordering / call example
-------------------------------------
For any given hook specification there may be more than one
implementation and we thus generally view ``hook`` execution as a
``1:N`` function call where ``N`` is the number of registered functions.
There are ways to influence if a hook implementation comes before or
after others, i.e. the position in the ``N``-sized list of functions:
.. code-block:: python
# Plugin 1
@pytest.hookimpl(tryfirst=True)
def pytest_collection_modifyitems(items):
# will execute as early as possible
...
# Plugin 2
@pytest.hookimpl(trylast=True)
def pytest_collection_modifyitems(items):
# will execute as late as possible
...
# Plugin 3
@pytest.hookimpl(hookwrapper=True)
def pytest_collection_modifyitems(items):
# will execute even before the tryfirst one above!
outcome = yield
# will execute after all non-hookwrappers executed
Here is the order of execution:
1. Plugin3's pytest_collection_modifyitems called until the yield point
because it is a hook wrapper.
2. Plugin1's pytest_collection_modifyitems is called because it is marked
with ``tryfirst=True``.
3. Plugin2's pytest_collection_modifyitems is called because it is marked
with ``trylast=True`` (but even without this mark it would come after
Plugin1).
4. Plugin3's pytest_collection_modifyitems then executing the code after the yield
point. The yield receives a :py:class:`Result <pluggy._Result>` instance which encapsulates
the result from calling the non-wrappers. Wrappers shall not modify the result.
It's possible to use ``tryfirst`` and ``trylast`` also in conjunction with
``hookwrapper=True`` in which case it will influence the ordering of hookwrappers
among each other.
Declaring new hooks
------------------------
.. note::
This is a quick overview on how to add new hooks and how they work in general, but a more complete
overview can be found in `the pluggy documentation <https://pluggy.readthedocs.io/en/latest/>`__.
.. currentmodule:: _pytest.hookspec
Plugins and ``conftest.py`` files may declare new hooks that can then be
implemented by other plugins in order to alter behaviour or interact with
the new plugin:
.. autofunction:: pytest_addhooks
:noindex:
Hooks are usually declared as do-nothing functions that contain only
documentation describing when the hook will be called and what return values
are expected. The names of the functions must start with `pytest_` otherwise pytest won't recognize them.
Here's an example. Let's assume this code is in the ``sample_hook.py`` module.
.. code-block:: python
def pytest_my_hook(config):
"""
Receives the pytest config and does things with it
"""
To register the hooks with pytest they need to be structured in their own module or class. This
class or module can then be passed to the ``pluginmanager`` using the ``pytest_addhooks`` function
(which itself is a hook exposed by pytest).
.. code-block:: python
def pytest_addhooks(pluginmanager):
""" This example assumes the hooks are grouped in the 'sample_hook' module. """
from my_app.tests import sample_hook
pluginmanager.add_hookspecs(sample_hook)
For a real world example, see `newhooks.py`_ from `xdist <https://github.com/pytest-dev/pytest-xdist>`_.
.. _`newhooks.py`: https://github.com/pytest-dev/pytest-xdist/blob/974bd566c599dc6a9ea291838c6f226197208b46/xdist/newhooks.py
Hooks may be called both from fixtures or from other hooks. In both cases, hooks are called
through the ``hook`` object, available in the ``config`` object. Most hooks receive a
``config`` object directly, while fixtures may use the ``pytestconfig`` fixture which provides the same object.
.. code-block:: python
@pytest.fixture()
def my_fixture(pytestconfig):
# call the hook called "pytest_my_hook"
# 'result' will be a list of return values from all registered functions.
result = pytestconfig.hook.pytest_my_hook(config=pytestconfig)
.. note::
Hooks receive parameters using only keyword arguments.
Now your hook is ready to be used. To register a function at the hook, other plugins or users must
now simply define the function ``pytest_my_hook`` with the correct signature in their ``conftest.py``.
Example:
.. code-block:: python
def pytest_my_hook(config):
"""
Print all active hooks to the screen.
"""
print(config.hook)
.. _`addoptionhooks`:
Using hooks in pytest_addoption
-------------------------------
Occasionally, it is necessary to change the way in which command line options
are defined by one plugin based on hooks in another plugin. For example,
a plugin may expose a command line option for which another plugin needs
to define the default value. The pluginmanager can be used to install and
use hooks to accomplish this. The plugin would define and add the hooks
and use pytest_addoption as follows:
.. code-block:: python
# contents of hooks.py
# Use firstresult=True because we only want one plugin to define this
# default value
@hookspec(firstresult=True)
def pytest_config_file_default_value():
""" Return the default value for the config file command line option. """
# contents of myplugin.py
def pytest_addhooks(pluginmanager):
""" This example assumes the hooks are grouped in the 'hooks' module. """
from . import hook
pluginmanager.add_hookspecs(hook)
def pytest_addoption(parser, pluginmanager):
default_value = pluginmanager.hook.pytest_config_file_default_value()
parser.addoption(
"--config-file",
help="Config file to use, defaults to %(default)s",
default=default_value,
)
The conftest.py that is using myplugin would simply define the hook as follows:
.. code-block:: python
def pytest_config_file_default_value():
return "config.yaml"
Optionally using hooks from 3rd party plugins
---------------------------------------------
Using new hooks from plugins as explained above might be a little tricky
because of the standard :ref:`validation mechanism <validation>`:
if you depend on a plugin that is not installed, validation will fail and
the error message will not make much sense to your users.
One approach is to defer the hook implementation to a new plugin instead of
declaring the hook functions directly in your plugin module, for example:
.. code-block:: python
# contents of myplugin.py
class DeferPlugin:
"""Simple plugin to defer pytest-xdist hook functions."""
def pytest_testnodedown(self, node, error):
"""standard xdist hook function."""
def pytest_configure(config):
if config.pluginmanager.hasplugin("xdist"):
config.pluginmanager.register(DeferPlugin())
This has the added benefit of allowing you to conditionally install hooks
depending on which plugins are installed.

2
doc/en/reference/xunit_setup.rst → doc/en/how-to/xunit_setup.rst
View File

@ -2,7 +2,7 @@
.. _`classic xunit`:
.. _xunitsetup:
classic xunit-style setup
How to implement xunit-style set-up
========================================
This section describes a classic and popular way how you can implement

2
doc/en/reference/fixtures.rst
View File

@ -5,7 +5,7 @@
.. _`pytest.fixture`:
pytest fixtures: explicit, modular, scalable
Fixtures reference
========================================================
.. seealso:: :ref:`about-fixtures`

7
doc/en/reference/index.rst
View File

@ -7,13 +7,6 @@ Reference guides
:maxdepth: 1
fixtures
warnings
doctest
cache
unittest
xunit_setup
plugin_list
writing_plugins
logging
customize
reference