================================================ Customizing and Extending py.test ================================================ basic test configuration =================================== Command line options and configuration file settings ----------------------------------------------------------------- You can get help on options and configuration options by running:: py.test -h # prints options _and_ config file settings This will display command line and configuration file settings which were registered by installed plugins. how test configuration is read from setup/tox ini-files -------------------------------------------------------- py.test looks for the first ``[pytest]`` section in either the first ``setup.cfg`` or the first ``tox.ini`` file found upwards from the arguments. Example:: py.test path/to/testdir will look in the following dirs for a config file:: path/to/testdir/setup.cfg path/to/setup.cfg path/setup.cfg setup.cfg ... # up until root of filesystem path/to/testdir/tox.ini path/to/tox.ini path/tox.ini ... # up until root of filesystem If no path was provided at all the current working directory is used for the lookup. builtin configuration file options ---------------------------------------------- .. confval:: minversion = VERSTRING specifies the minimal pytest version that is needed for this test suite. minversion = 2.1 # will fail if we run with pytest-2.0 .. confval:: addargs = OPTS add the specified ``OPTS`` to the set of command line arguments as if they had been specified by the user. Example: if you have this ini file content:: [pytest] addargs = --maxfail=2 -rf # exit after 2 failures, report fail info issuing ``py.test test_hello.py`` actually means:: py.test --maxfail=2 -rf test_hello.py .. _`function arguments`: funcargs.html .. _`extensions`: Plugin basics and project configuration ============================================= .. _`local plugin`: py.test implements all aspects of its functionality by calling `well specified hooks`_. Hook functions are discovered in :file:`conftest.py` files or in `named plugins`_. :file:`conftest.py` files are useful for keeping test extensions and customizations close to test code. local conftest.py plugins -------------------------------------------------------------- local ``conftest.py`` plugins contain directory-specific hook implemenations. Its contained runtest- and collection- related hooks are called when collecting or running tests in files or directories next to or below the ``conftest.py`` file. Example: Assume the following layout and content of files:: a/conftest.py: def pytest_runtest_setup(item): print ("setting up", item) a/test_in_subdir.py: def test_sub(): pass test_flat.py: def test_flat(): pass Here is how you might run it:: py.test test_flat.py # will not show "setting up" py.test a/test_sub.py # will show "setting up" ``py.test`` loads all ``conftest.py`` files upwards from the command line file arguments. It usually performs look up right-to-left, i.e. the hooks in "closer" conftest files will be called earlier than further away ones. This means you can even have a ``conftest.py`` file in your home directory to customize test functionality globally for all of your projects. .. Note:: If you have ``conftest.py`` files which do not reside in a python package directory (i.e. one containing an ``__init__.py``) then "import conftest" can be ambigous because there might be other ``conftest.py`` files as well on your PYTHONPATH or ``sys.path``. It is good practise for projects to put ``conftest.py`` within a package scope or to never import anything from the conftest.py file. .. _`named plugins`: plugin/index.html Plugin discovery at tool startup -------------------------------------------- py.test loads plugin modules at tool startup in the following way: * by loading all plugins registered through `setuptools entry points`_. * by pre-scanning the command line for the ``-p name`` option and loading the specified plugin before actual command line parsing. * by loading all :file:`conftest.py` files as inferred by the command line invocation (test files and all of its *parent* directories). Note that ``conftest.py`` files from *sub* directories are by default not loaded at tool startup. * by recursively loading all plugins specified by the ``pytest_plugins`` variable in ``conftest.py`` files Requiring/Loading plugins in a test module or conftest file ------------------------------------------------------------- You can require plugins in a test module or a conftest file like this:: pytest_plugins = "name1", "name2", When the test module or conftest plugin is loaded the specified plugins will be loaded as well. You can also use dotted path like this:: pytest_plugins = "myapp.testsupport.myplugin" which will import the specified module as a py.test plugin. .. _`setuptools entry points`: .. _registered: Writing setuptools-registered plugins ------------------------------------------------------ .. _`Distribute`: http://pypi.python.org/pypi/distribute .. _`setuptools`: http://pypi.python.org/pypi/setuptools If you want to make your plugin publically available, you can use `setuptools`_ or `Distribute`_ which both allow to register an entry point. ``py.test`` will register all objects with the ``pytest11`` entry point. To make your plugin available you may insert the following lines in your setuptools/distribute-based setup-invocation: .. sourcecode:: python # sample ./setup.py file from setuptools import setup setup( name="myproject", packages = ['myproject'] # the following makes a plugin available to py.test entry_points = { 'pytest11': [ 'name_of_plugin = myproject.pluginmodule', ] }, ) If a package is installed with this setup, py.test will load ``myproject.pluginmodule`` under the ``name_of_plugin`` name and use it as a plugin. Accessing another plugin by name -------------------------------------------- If a plugin wants to collaborate with code from another plugin it can obtain a reference through the plugin manager like this: .. sourcecode:: python plugin = config.pluginmanager.getplugin("name_of_plugin") If you want to look at the names of existing plugins, use the ``--traceconfig`` option. .. _`well specified hooks`: py.test hook reference ==================================== hook specification and validation ----------------------------------------- py.test calls hook functions to implement initialization, running, test execution and reporting. When py.test loads a plugin it validates that all hook functions conform to their respective hook specification. Each hook function name and its argument names need to match a hook specification exactly but it is allowed for a hook function to accept *less* parameters than specified. If you mistype argument names or the hook name itself you get useful errors. initialisation, command line and configuration hooks -------------------------------------------------------------------- .. currentmodule:: pytest.hookspec .. autofunction:: pytest_cmdline_parse .. autofunction:: pytest_namespace .. autofunction:: pytest_addoption .. autofunction:: pytest_cmdline_main .. autofunction:: pytest_configure .. autofunction:: pytest_unconfigure generic "runtest" hooks ------------------------------ All all runtest related hooks receive a :py:class:`pytest.collect.Item` object. .. autofunction:: pytest_runtest_protocol .. autofunction:: pytest_runtest_setup .. autofunction:: pytest_runtest_call .. autofunction:: pytest_runtest_teardown .. autofunction:: pytest_runtest_makereport For deeper understanding you may look at the default implementation of these hooks in :py:mod:`pytest.plugin.runner` and maybe also in :py:mod:`pytest.plugin.pdb` which intercepts creation of reports in order to drop to interactive debugging. The :py:mod:`pytest.plugin.terminal` reported specifically uses the reporting hook to print information about a test run. collection hooks ------------------------------ py.test calls the following hooks for collecting files and directories: .. autofunction:: pytest_ignore_collect .. autofunction:: pytest_collect_directory .. autofunction:: pytest_collect_file For influencing the collection of objects in Python modules you can use the following hook: .. autofunction:: pytest_pycollect_makeitem reporting hooks ------------------------------ Collection related reporting hooks: .. autofunction: pytest_collectstart .. autofunction: pytest_log_itemcollect .. autofunction: pytest_collectreport .. autofunction: pytest_deselected And here is the central hook for reporting about test execution: .. autofunction: pytest_runtest_logreport The test collection tree ====================================================== Default filesystem test discovery ----------------------------------------------- Test collection starts from specified paths or from the current directory. All tests are collected ahead of running the first test. (This used to be different in earlier versions of ``py.test`` where collection and running was interweaved which made test randomization and distributed testing harder). Collection nodes which have children are called "Collectors" and otherwise they are called "Items" or "test items". Here is an example of such a tree:: example $ py.test --collectonly test_collectonly.py By default all directories not starting with a dot are traversed, looking for ``test_*.py`` and ``*_test.py`` files. Those Python files are imported under their `package name`_. The Module collector looks for test functions and test classes and methods. Test functions and methods are prefixed ``test`` by default. Test classes must start with a capitalized ``Test`` prefix. Customizing error messages ------------------------------------------------- On test and collection nodes ``py.test`` will invoke the ``node.repr_failure(excinfo)`` function which you may override and make it return an error representation string of your choice. It will be reported as a (red) string. .. _`package name`: constructing the package name for test modules ------------------------------------------------- Test modules are imported under their fully qualified name. Given a filesystem ``fspath`` it is constructed as follows: * walk the directories up to the last one that contains an ``__init__.py`` file. * perform ``sys.path.insert(0, basedir)``. * import the root package as ``root`` Reference of important objects involved in hooks =========================================================== .. autoclass:: pytest.plugin.config.Config :members: .. autoclass:: pytest.plugin.config.Parser :members: .. autoclass:: pytest.plugin.session.Node(name, parent) :members: .. .. autoclass:: pytest.plugin.session.File(fspath, parent) :members: .. autoclass:: pytest.plugin.session.Item(name, parent) :members: .. autoclass:: pytest.plugin.python.Module(name, parent) :members: .. autoclass:: pytest.plugin.python.Class(name, parent) :members: .. autoclass:: pytest.plugin.python.Function(name, parent) :members: .. autoclass:: pytest.plugin.runner.CallInfo :members: .. autoclass:: pytest.plugin.runner.TestReport :members: