test_ok2/doc/en/goodpractises.rst

.. highlightlang:: python
.. _`goodpractises`:

Good Integration Practices
=================================================

Work with virtual environments
-----------------------------------------------------------

We recommend to use virtualenv_ environments and use pip_
(or easy_install_) for installing your application and any dependencies
as well as the ``pytest`` package itself.  This way you will get an isolated
and reproducible environment.    Given you have installed virtualenv_
and execute it from the command line, here is an example session for unix
or windows::

    virtualenv .   # create a virtualenv directory in the current directory

    source bin/activate  # on unix

    scripts/activate     # on Windows

We can now install pytest::

    pip install pytest

Due to the ``activate`` step above the ``pip`` will come from
the virtualenv directory and install any package into the isolated
virtual environment.

Choosing a test layout / import rules
------------------------------------------

``pytest`` supports two common test layouts:

* putting tests into an extra directory outside your actual application
  code, useful if you have many functional tests or for other reasons
  want to keep tests separate from actual application code (often a good
  idea)::

    setup.py   # your distutils/setuptools Python package metadata
    mypkg/
        __init__.py
        appmodule.py
    tests/
        test_app.py
        ...


* inlining test directories into your application package, useful if you
  have direct relation between (unit-)test and application modules and
  want to distribute your tests along with your application::

    setup.py   # your distutils/setuptools Python package metadata
    mypkg/
        __init__.py
        appmodule.py
        ...
        test/
            test_app.py
            ...

Important notes relating to both schemes:

- **make sure that "mypkg" is importable**, for example by typing once::

     pip install -e .   # install package using setup.py in editable mode

- **avoid "__init__.py" files in your test directories**.
  This way your tests can run easily against an installed version
  of ``mypkg``, independently from the installed package if it contains
  the tests or not.

- With inlined tests you might put ``__init__.py`` into test
  directories and make them installable as part of your application.
  Using the ``py.test --pyargs mypkg`` invocation pytest will
  discover where mypkg is installed and collect tests from there.
  With the "external" test you can still distribute tests but they
  will not be installed or become importable.

Typically you can run tests by pointing to test directories or modules::

    py.test tests/test_app.py       # for external test dirs
    py.test mypkg/test/test_app.py  # for inlined test dirs
    py.test mypkg                   # run tests in all below test directories
    py.test                         # run all tests below current dir
    ...

Because of the above ``editable install`` mode you can change your
source code (both tests and the app) and rerun tests at will.
Once you are done with your work, you can `use tox`_ to make sure
that the package is really correct and tests pass in all
required configurations.

.. note::

    You can use Python3 namespace packages (PEP420) for your application
    but pytest will still perform `test package name`_ discovery based on the
    presence of ``__init__.py`` files.  If you use one of the
    two recommended file system layouts above but leave away the ``__init__.py``
    files from your directories it should just work on Python3.3 and above.  From
    "inlined tests", however, you will need to use absolute imports for
    getting at your application code.

.. _`test package name`:

.. note::

    If ``pytest`` finds a "a/b/test_module.py" test file while
    recursing into the filesystem it determines the import name
    as follows:

    * determine ``basedir``: this is the first "upward" (towards the root)
      directory not containing an ``__init__.py``.  If e.g. both ``a``
      and ``b`` contain an ``__init__.py`` file then the parent directory
      of ``a`` will become the ``basedir``.

    * perform ``sys.path.insert(0, basedir)`` to make the test module
      importable under the fully qualified import name.

    * ``import a.b.test_module`` where the path is determined
      by converting path separators ``/`` into "." characters.  This means
      you must follow the convention of having directory and file
      names map directly to the import names.

    The reason for this somewhat evolved importing technique is
    that in larger projects multiple test modules might import
    from each other and thus deriving a canonical import name helps
    to avoid surprises such as a test modules getting imported twice.


.. _`virtualenv`: http://pypi.python.org/pypi/virtualenv
.. _`buildout`: http://www.buildout.org/
.. _pip: http://pypi.python.org/pypi/pip

.. _`use tox`:

Use tox and Continuous Integration servers
-------------------------------------------------

If you frequently release code and want to make sure that your actual
package passes all tests you may want to look into `tox`_, the
virtualenv test automation tool and its `pytest support
<http://testrun.org/tox/latest/example/pytest.html>`_.
Tox helps you to setup virtualenv environments with pre-defined
dependencies and then executing a pre-configured test command with
options.  It will run tests against the installed package and not
against your source code checkout, helping to detect packaging
glitches.

If you want to use Jenkins_ you can use the ``--junitxml=PATH`` option
to create a JUnitXML file that Jenkins_ can pick up and generate reports.

.. _standalone:
.. _`genscript method`:

(deprecated) Create a pytest standalone script
-----------------------------------------------

If you are a maintainer or application developer and want people
who don't deal with python much to easily run tests you may generate
a standalone ``pytest`` script::

    py.test --genscript=runtests.py

This generates a ``runtests.py`` script which is a fully functional basic
``pytest`` script, running unchanged under Python2 and Python3.
You can tell people to download the script and then e.g.  run it like this::

    python runtests.py

.. note::

   You must have pytest and its dependencies installed as an sdist, not
   as wheels because genscript need the source code for generating a
   standalone script.




Integrating with distutils / ``python setup.py test``
--------------------------------------------------------

You can integrate test runs into your distutils or
setuptools based project.  Use the `genscript method`_
to generate a standalone ``pytest`` script::

    py.test --genscript=runtests.py

and make this script part of your distribution and then add
this to your ``setup.py`` file::

    from distutils.core import setup, Command
    # you can also import from setuptools

    class PyTest(Command):
        user_options = []
        def initialize_options(self):
            pass

        def finalize_options(self):
            pass

        def run(self):
            import subprocess
            import sys
            errno = subprocess.call([sys.executable, 'runtests.py'])
            raise SystemExit(errno)


    setup(
        #...,
        cmdclass = {'test': PyTest},
        #...,
    )

If you now type::

    python setup.py test

this will execute your tests using ``runtests.py``. As this is a
standalone version of ``pytest`` no prior installation whatsoever is
required for calling the test command. You can also pass additional
arguments to the subprocess-calls such as your test directory or other
options.


Integration with setuptools test commands
----------------------------------------------------

Setuptools supports writing our own Test command for invoking pytest.
Most often it is better to use tox_ instead, but here is how you can
get started with setuptools integration::

    import sys

    from setuptools.command.test import test as TestCommand


    class PyTest(TestCommand):
        user_options = [('pytest-args=', 'a', "Arguments to pass to py.test")]

        def initialize_options(self):
            TestCommand.initialize_options(self)
            self.pytest_args = []

        def finalize_options(self):
            TestCommand.finalize_options(self)
            self.test_args = []
            self.test_suite = True

        def run_tests(self):
            #import here, cause outside the eggs aren't loaded
            import pytest
            errno = pytest.main(self.pytest_args)
            sys.exit(errno)


    setup(
        #...,
        tests_require=['pytest'],
        cmdclass = {'test': PyTest},
        )

Now if you run::

    python setup.py test

this will download ``pytest`` if needed and then run your tests
as you would expect it to. You can pass a single string of arguments
using the ``--pytest-args`` or ``-a`` command-line option. For example::

    python setup.py test -a "--durations=5"

is equivalent to running ``py.test --durations=5``.

.. seealso::

    For a more powerful solution, take a look at the
    `pytest-runner <https://pypi.python.org/pypi/pytest-runner>`_ plugin.

.. _`test discovery`:
.. _`Python test discovery`:

Conventions for Python test discovery
-------------------------------------------------

``pytest`` implements the following standard test discovery:

* collection starts from paths specified in :confval:`testpaths` if configured,
  otherwise from initial command line arguments which may be directories,
  filenames or test ids. If :confval:`testpaths` is not configured and no
  directories or files were given in the command line, start collection from
  the current directory.
* recurse into directories, unless they match :confval:`norecursedirs`
* ``test_*.py`` or ``*_test.py`` files, imported by their `test package name`_.
* ``Test`` prefixed test classes (without an ``__init__`` method)
* ``test_`` prefixed test functions or methods are test items

For examples of how to customize your test discovery :doc:`example/pythoncollection`.

Within Python modules, ``pytest`` also discovers tests using the standard
:ref:`unittest.TestCase <unittest.TestCase>` subclassing technique.

.. include:: links.inc