From cd8e69e33c5a4cf3bf866491484b58a0ad0a0a7b Mon Sep 17 00:00:00 2001 From: holger krekel Date: Tue, 10 Dec 2013 14:54:13 +0100 Subject: [PATCH] close issue240 - rework "good practises" document and discuss discuss the two common test directory layouts in more detail, including an explicit note on how it interacts with PEP420-namespace packages. --- CHANGELOG | 4 + doc/en/goodpractises.txt | 227 +++++++++++++++++++++++++-------------- 2 files changed, 148 insertions(+), 83 deletions(-) diff --git a/CHANGELOG b/CHANGELOG index 3ba08a485..3d625b665 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -26,6 +26,10 @@ Unreleased with repeated same values (sometimes useful to to test if calling a second time works as with the first time). +- close issue240 - document precisely how pytest module importing + works, discuss the two common test directory layouts, and how it + interacts with PEP420-namespace packages. + - fix issue246 fix finalizer order to be LIFO on independent fixtures depending on a parametrized higher-than-function scoped fixture. (was quite some effort so please bear with the complexity of this sentence :) diff --git a/doc/en/goodpractises.txt b/doc/en/goodpractises.txt index 8b2ccec56..edb271be1 100644 --- a/doc/en/goodpractises.txt +++ b/doc/en/goodpractises.txt @@ -8,24 +8,151 @@ Good Integration Practises Work with virtual environments ----------------------------------------------------------- -We recommend to use virtualenv_ environments and use easy_install_ -(or pip_) for installing your application dependencies as well as -the ``pytest`` package itself. This way you will get a much more reproducible -environment. A good tool to help you automate test runs against multiple -dependency configurations or Python interpreters is `tox`_. +We recommend to use virtualenv_ environments and use pip_ +(or easy_install_) for installing your application and any dependencies +as well as the ``pytest`` package itself. This way you will get an isolated +and reproducible environment. Given you have installed virtualenv_ +and execute it from the command line, here is an example session for unix +or windows:: + + virtualenv . # create a virtualenv directory in the current directory + + source bin/activate # on unix + + scripts/activate # on Windows + +We can now install pytest:: + + pip install pytest + +Due to the ``activate`` step above the ``pip`` will come from +the virtualenv directory and install any package into the isolated +virtual environment. + +Choosing a test layout / import rules +------------------------------------------ + +py.test supports two common test layouts: + +* putting tests into an extra directory outside your actual application + code, useful if you have many functional tests or for other reasons + want to keep tests separate from actual application code (often a good + idea):: + + setup.py # your distutils/setuptools Python package metadata + mypkg/ + __init__.py + appmodule.py + tests/ + test_app.py + ... + + +* inlining test directories into your application package, useful if you + have direct relation between (unit-)test and application modules and + want to distribute your tests along with your application:: + + setup.py # your distutils/setuptools Python package metadata + mypkg/ + __init__.py + appmodule.py + ... + test/ + test_app.py + ... + +Important notes relating to both schemes: + +- **make sure that "mypkg" is importable**, for example by typing once:: + + pip install -e . # install package using setup.py in editable mode + +- **avoid "__init__.py" files in your test directories**. + This way your tests can run easily against an installed version + of ``mypkg``, independently from if the installed version contains + the tests or not. + +- With inlined tests you might put ``__init__.py`` into test + directories and make them installable as part of your application. + Using the ``py.test --pyargs mypkg`` invocation pytest will + discover where mypkg is installed and collect tests from there. + With the "external" test you can still distribute tests but they + will not be installed or become importable. + +Typically you can run tests by pointing to test directories or modules:: + + py.test tests/test_app.py # for external test dirs + py.test mypkg/test/test_app.py # for inlined test dirs + py.test mypkg # run tests in all below test directories + py.test # run all tests below current dir + ... + +Because of the above ``editable install`` mode you can change your +source code (both tests and the app) and rerun tests at will. +Once you are done with your work, you can `use tox`_ to make sure +that the package is really correct and tests pass in all +required configurations. + +.. note:: + + You can use Python3 namespace packages (PEP420) for your application + but pytest will still perform `package name`_ discovery based on the + presence of ``__init__.py`` files. If you use one of the above + two recommended file system layouts but leave away the ``__init__.py`` + files it should just work on Python3.3 and above. When using + "inlined tests", however, you will need to use absolute imports for + getting at your application code because the test modules will be + imported directly, without any application context. The latter allows + your tests to run against an installed version of your package. + +.. _`package name`: + +.. note:: + + If py.test finds a "a/b/test_module.py" test file while + recursing into the filesystem it determines the import name + as follows: + + * determine ``basedir``: this is the first "upward" (towards the root) + directory not containing an ``__init__.py``. If e.g. both ``a`` + and ``b`` contain an ``__init__.py`` file then the parent directory + of ``a`` will become the ``basedir``. + + * perform ``sys.path.insert(0, basedir)`` to make the test module + importable under the fully qualified import name. + + * ``import a.b.test_module`` where the path is determined + by converting path separators ``/`` into "." characters. This means + you must follow the convention of having directory and file + names map directly to the import names. + + The reason for this somewhat evolved importing technique is + that in larger projects multiple test modules might import + from each other and thus deriving a canonical import name helps + to avoid surprises such as a test modules getting imported twice. + .. _`virtualenv`: http://pypi.python.org/pypi/virtualenv .. _`buildout`: http://www.buildout.org/ .. _pip: http://pypi.python.org/pypi/pip +.. _`use tox`: + Use tox and Continuous Integration servers ------------------------------------------------- -If you frequently release code to the public you -may want to look into `tox`_, the virtualenv test automation -tool and its `pytest support `_. -The basic idea is to generate a JUnitXML file through the ``--junitxml=PATH`` option and have a continuous integration server like Jenkins_ pick it up -and generate reports. +If you frequently release code and want to make sure that your actual +package passes all tests you may want to look into `tox`_, the +virtualenv test automation tool and its `pytest support +`_. +Tox helps you to setup virtualenv environments with pre-defined +dependencies and then executing a pre-configured test command with +options. It will run tests against the installed package and not +against your source code checkout, helping to detect packaging +glitches. + +If you want to use Jenkins_ you can use the ``--junitxml=PATH`` option +to create a JUnitXML file that Jenkins_ can pick up and generate reports. .. _standalone: .. _`genscript method`: @@ -33,21 +160,19 @@ and generate reports. Create a py.test standalone script ------------------------------------------- -If you are a maintainer or application developer and want others -to easily run tests you can generate a completely standalone "py.test" -script:: +If you are a maintainer or application developer and want people +who don't deal with python much to easily run tests you may generate +a standalone "py.test" script:: py.test --genscript=runtests.py -generates a ``runtests.py`` script which is a fully functional basic +This generates a ``runtests.py`` script which is a fully functional basic ``py.test`` script, running unchanged under Python2 and Python3. You can tell people to download the script and then e.g. run it like this:: python runtests.py - - Integrating with distutils / ``python setup.py test`` -------------------------------------------------------- @@ -93,8 +218,9 @@ options. Integration with setuptools test commands ---------------------------------------------------- -Setuptools supports writing our own Test command for invoking -pytest:: +Setuptools supports writing our own Test command for invoking pytest. +Most often it is better to use tox_ instead, but here is how you can +get started with setuptools integration:: from setuptools.command.test import test as TestCommand import sys @@ -143,69 +269,4 @@ For examples of how to customize your test discovery :doc:`example/pythoncollect Within Python modules, py.test also discovers tests using the standard :ref:`unittest.TestCase ` subclassing technique. -Choosing a test layout / import rules ------------------------------------------- - -py.test supports common test layouts: - -* inlining test directories into your application package, useful if you want to - keep (unit) tests and actually tested code close together:: - - mypkg/ - __init__.py - appmodule.py - ... - test/ - test_app.py - ... - -* putting tests into an extra directory outside your actual application - code, useful if you have many functional tests or want to keep - tests separate from actual application code:: - - mypkg/ - __init__.py - appmodule.py - tests/ - test_app.py - ... - -In both cases you usually need to make sure that ``mypkg`` is importable, -for example by using the setuptools ``python setup.py develop`` method. - -You can run your tests by pointing to it:: - - py.test tests/test_app.py # for external test dirs - py.test mypkg/test/test_app.py # for inlined test dirs - py.test mypkg # run tests in all below test directories - py.test # run all tests below current dir - ... - -.. _`package name`: - -.. note:: - - If py.test finds a "a/b/test_module.py" test file while - recursing into the filesystem it determines the import name - as follows: - - * find ``basedir`` -- this is the first "upward" (towards the root) - directory not containing an ``__init__.py``. If both the ``a`` - and ``b`` directories contain an ``__init__.py`` the basedir will - be the parent dir of ``a``. - - * perform ``sys.path.insert(0, basedir)`` to make the test module - importable under the fully qualified import name. - - * ``import a.b.test_module`` where the path is determined - by converting path separators ``/`` into "." characters. This means - you must follow the convention of having directory and file - names map directly to the import names. - - The reason for this somewhat evolved importing technique is - that in larger projects multiple test modules might import - from each other and thus deriving a canonical import name helps - to avoid surprises such as a test modules getting imported twice. - - .. include:: links.inc