test_ok1/doc/customize.txt

================================================
Customizing and Extending py.test
================================================

basic test configuration
===================================

available command line options
---------------------------------

You can see command line options by running::

    py.test -h

This will display all available command line options
in your specific environment.

.. _`project-specific test configuration`:
.. _`collect_ignore`:

conftest.py: project specific hooks and configuration
--------------------------------------------------------

A unique feature of py.test are its ``conftest.py`` files which
allow to:

* `set option defaults`_

* `implement hooks`_

* `specify funcargs`_

or set particular variables to influence the testing process:

* ``pytest_plugins``: list of named plugins to load

* ``collect_ignore``: list of paths to ignore during test collection, relative to the containing ``conftest.py`` file

* ``rsyncdirs``: list of to-be-rsynced directories for distributed
  testing, relative to the containing ``conftest.py`` file.

You may put a conftest.py files in your project root directory or into
your package directory if you want to add project-specific test options.

``py.test`` loads all ``conftest.py`` files upwards from the command
line file arguments.  It usually looks up configuration values
right-to-left, i.e. the closer conftest files will be checked first.
This means you can have a ``conftest.py`` in your very home directory to
have some global configuration values.

.. _`specify funcargs`: funcargs.html#application-setup-tutorial-example

.. _`set option defaults`:

setting persistent option defaults
------------------------------------

py.test will lookup option values in this order:

* command line
* conftest.py files
* environment variables

To find out about the particular switches and type::

    py.test --help-config

This will print information about all options in your
environment, including your local plugins.

.. _`basetemp`:

Temporary directories
-------------------------------------------

You can create directories by calling one of two methods
on the config object:

- ``config.mktemp(basename)``: create and return a new tempdir

- ``config.ensuretemp(basename)``: create or return a new tempdir

temporary directories are created as sub directories of a per-session
testdir and will keep around the directories of the last three test
runs.  You can set the base temporary directory through the command line
`--basetemp`` option.  When distributing tests on the same machine,
``py.test`` takes care to configure a basetemp directory for the sub
processes such that all temporary data lands below below a single
per-test run basetemp directory.

.. _`function arguments`: funcargs.html
.. _`extensions`:

Plugin basics and project configuration
=============================================

.. _`local plugin`:

project specific "local" or named "global" plugins
--------------------------------------------------------------

py.test implements much of its functionality by calling `well specified
hooks`_.  Python modules which contain such hook functions are called
plugins.  Hook functions are discovered in ``conftest.py`` files or in
`named plugins`_.  ``conftest.py`` files are sometimes called
"anonymous" or conftest plugins. They are useful for keeping test
extensions close to your application.  Named plugins are normal python
modules or packages that can be distributed separately.   Named plugins
need to follow a naming pattern; they have an all lowercase ``pytest_``
prefixed name.  While conftest plugins are discovered automatically,
named plugins must be explicitely specified.

.. _`named plugins`: plugin/index.html

.. _`tool startup`:
.. _`loaded at tool startup`:
.. _`test tool starts up`:

Plugin discovery at tool startup
--------------------------------------------

py.test loads plugin modules at tool startup in the following way:

* by loading all plugins registered through `setuptools entry points`_.

* by reading the ``PYTEST_PLUGINS`` environment variable
  and importing the comma-separated list of named plugins.

* by pre-scanning the command line for the ``-p name`` option
  and loading the specified plugin before actual command line parsing.

* by loading all `conftest.py plugin`_ files as inferred by the command line
  invocation (test files and all of its parent directories).
  Note that ``conftest.py`` files from sub directories are loaded
  during test collection and not at tool startup.

* by recursively loading all plugins specified by the
  ``pytest_plugins`` variable in a ``conftest.py`` file

Specifying plugins in a test module or plugin
-----------------------------------------------

You can specify plugins in a test module or a plugin like this:

.. sourcecode:: python

    pytest_plugins = "name1", "name2",

When the test module or plugin is loaded the specified plugins
will be loaded.  If you specify plugins without the ``pytest_``
prefix it will be automatically added.  All plugin names
must be lowercase.

.. _`conftest.py plugin`:
.. _`conftestplugin`:

Writing per-project plugins (conftest.py)
------------------------------------------------------

The purpose of ``conftest.py`` files is to allow `project-specific
test configuration`_.  They thus make for a good place to implement
project-specific test related features through hooks.  For example you may
set the `collect_ignore`_ variable depending on a command line option
by defining the following hook in a ``conftest.py`` file:

.. _`exclude-file-example`:

.. sourcecode:: python

    # ./conftest.py in your root or package dir
    collect_ignore = ['hello', 'test_world.py']
    def pytest_addoption(parser):
        parser.addoption("--runall", action="store_true", default=False)
    def pytest_configure(config):
        if config.getvalue("runall"):
            collect_ignore[:] = []

.. _`setuptools entry points`:

Writing setuptools-registered plugins
------------------------------------------------------

.. _`Distribute`: http://pypi.python.org/pypi/distribute
.. _`setuptools`: http://pypi.python.org/pypi/setuptools

If you want to make your plugin publically available, you
can use `setuptools`_ or `Distribute`_ which both allow
to register an entry point.  ``py.test`` will register
all objects with the ``pytest11`` entry point.
To make your plugin available you may insert the following
lines in your setuptools/distribute-based setup-invocation:

.. sourcecode:: python

    # sample ./setup.py file
    from setuptools import setup

    setup(
        name="myproject",
        packages = ['myproject']

        # the following makes a plugin available to py.test
        entry_points = {
            'pytest11': [
                'name_of_plugin = myproject.pluginmodule',
            ]
        },
    )

If a package is installed with this setup, py.test will load
``myproject.pluginmodule`` under the ``name_of_plugin`` name
and use it as a plugin.

Accessing another plugin by name
--------------------------------------------

If a plugin wants to collaborate with code from
another plugin it can obtain a reference through
the plugin manager like this:

.. sourcecode:: python

    plugin = config.pluginmanager.getplugin("name_of_plugin")

If you want to look at the names of existing plugins, use
the ``--traceconfig`` option.

.. _`well specified hooks`:
.. _`implement hooks`:

py.test hook reference
====================================

hook specification and validation
-----------------------------------------

py.test calls hook functions to implement initialization, running,
test execution and reporting.  When py.test loads a plugin it validates
that all hook functions conform to their respective hook specification.
Each hook function name and its argument names need to match a hook
specification exactly but it is allowed for a hook function to accept 
*less* parameters than specified.  If you mistype argument names or the
hook name itself you get useful errors.

initialisation, command line and configuration hooks
--------------------------------------------------------------------

.. currentmodule:: pytest.hookspec

.. autofunction:: pytest_namespace
.. autofunction:: pytest_addoption
.. autofunction:: pytest_cmdline_main
.. autofunction:: pytest_configure
.. autofunction:: pytest_unconfigure

generic "runtest" hooks
------------------------------

Each test item is usually executed by calling the following three hooks:

.. autofunction:: pytest_runtest_setup
.. autofunction:: pytest_runtest_call
.. autofunction:: pytest_runtest_teardown

For each of the three invocations a `call object`_ encapsulates
information about the outcome of the call and is subsequently used
to make a report object:

.. sourcecode:: python

    report = hook.pytest_runtest_makereport(item, call)

For example, the `pytest_pdb plugin`_ uses this hook to activate
interactive debugging on failures when ``--pdb`` is specified on the
command line.

Usually three reports will be generated for a single test item for each
of the three runtest hooks respectively.  If ``pytest_runtest_setup``
fails then ``pytest_runtest_teardown`` will be called but not
``pytest_runtest_call``.

Each of the up to three reports is eventually fed to the logreport hook:

.. sourcecode:: python

    pytest_runtest_logreport(report)

A ``report`` object contains status and reporting information:

.. sourcecode:: python

    report.longrepr = string/lines/object to print
    report.when = "setup", "call" or "teardown"
    report.shortrepr = letter for progress-report
    report.passed = True or False
    report.failed = True or False
    report.skipped = True or False

The `terminal plugin`_ uses this hook to print information
about a test run.

The whole protocol described here is implemented via this hook:

.. sourcecode:: python

    pytest_runtest_protocol(item) -> True

.. _`call object`:

The call object contains information about a performed call:

.. sourcecode:: python

    call.excinfo = ExceptionInfo object or None
    call.when = "setup", "call" or "teardown"
    call.outerr = None or tuple of strings representing captured stdout/stderr

.. _`pytest_pdb plugin`: plugin/pdb.html
.. _`terminal plugin`: plugin/terminal.html


generic collection hooks
------------------------------

py.test calls the following two fundamental hooks for collecting files and directories:

.. sourcecode:: python

    def pytest_collect_directory(path, parent):
        """ return Collection node or None for the given path. """

    def pytest_collect_file(path, parent):
        """ return Collection node or None for the given path. """

Both return a `collection node`_ for a given path.  All returned
nodes from all hook implementations will participate in the
collection and running protocol.  The ``parent`` object is
the parent node and may be used to access command line
options via the ``parent.config`` object.


Python test function and module hooks
----------------------------------------------------

For influencing the collection of objects in Python modules
you can use the following hook:

.. sourcecode:: python

    def pytest_pycollect_makeitem(collector, name, obj):
        """ return custom item/collector for a python object in a module, or None.  """

This hook will be called for each Python object in a collected
Python module.  The return value is a custom `collection node`_ or None.

.. XXX or ``False`` if you want to indicate that the given item should not be collected.


Gateway initialization (distributed testing)
----------------------------------------------------

(alpha) For distributed testing it can be useful to prepare the
remote environment.  For this you can implement the newgateway hook:

.. sourcecode:: python

    def pytest_gwmanage_newgateway(gateway, platinfo):
        """ called after a gateway is instantiated. """

The ``gateway`` object here has a ``spec`` attribute which is an ``execnet.XSpec``
object, which has attributes that map key/values as specified from a ``--txspec``
option. The platinfo object is a dictionary with information about the remote process:

* ``version``: remote python's ``sys.version_info``
* ``platform``: remote ``sys.platform``
* ``cwd``: remote ``os.getcwd``


.. _`collection process`:
.. _`collection node`:
.. _`test collection`:


Test Collection process
======================================================

the collection tree
---------------------------------

The collecting process is iterative so that distribution
and execution of tests can start as soon as the first test
item is collected.  Collection nodes with children are
called "Collectors" and terminal nodes are called "Items".
Here is an example of such a tree, generated with the
command ``py.test --collectonly py/xmlobj``::

    <Directory 'xmlobj'>
        <Directory 'testing'>
            <Module 'test_html.py' (py.__.xmlobj.testing.test_html)>
                <Function 'test_html_name_stickyness'>
                <Function 'test_stylenames'>
                <Function 'test_class_None'>
                <Function 'test_alternating_style'>
            <Module 'test_xml.py' (py.__.xmlobj.testing.test_xml)>
                <Function 'test_tag_with_text'>
                <Function 'test_class_identity'>
                <Function 'test_tag_with_text_and_attributes'>
                <Function 'test_tag_with_subclassed_attr_simple'>
                <Function 'test_tag_nested'>
                <Function 'test_tag_xmlname'>

By default all directories not starting with a dot are traversed,
looking for ``test_*.py`` and ``*_test.py`` files.  Those Python
files are imported under their `package name`_.

The Module collector looks for test functions
and test classes and methods. Test functions and methods
are prefixed ``test`` by default.  Test classes must
start with a capitalized ``Test`` prefix.

Customizing error messages
-------------------------------------------------

On test and collection nodes ``py.test`` will invoke
the ``node.repr_failure(excinfo)`` function which
you may override and make it return an error
representation string of your choice.  It
will be reported as a (red) string.

.. _`package name`:

constructing the package name for test modules
-------------------------------------------------

Test modules are imported under their fully qualified
name.  Given a filesystem ``fspath`` it is constructed as follows:

* walk the directories up to the last one that contains
  an ``__init__.py`` file.

* perform ``sys.path.insert(0, basedir)``.

* import the root package as ``root``