From 841f73170751b237a15f64a489988f783da7982d Mon Sep 17 00:00:00 2001 From: Bruno Oliveira Date: Thu, 9 Mar 2017 20:41:33 -0300 Subject: [PATCH 1/4] Attempt to clarify the confusion regarding __init__ files and unique test names Fix #529 --- doc/en/goodpractices.rst | 85 ++++++++++++++++++---------------------- tox.ini | 2 + 2 files changed, 40 insertions(+), 47 deletions(-) diff --git a/doc/en/goodpractices.rst b/doc/en/goodpractices.rst index 48f80a9e2..65c5624fa 100644 --- a/doc/en/goodpractices.rst +++ b/doc/en/goodpractices.rst @@ -30,68 +30,58 @@ Within Python modules, ``pytest`` also discovers tests using the standard Choosing a test layout / import rules ------------------------------------------- +------------------------------------- ``pytest`` supports two common test layouts: -* putting tests into an extra directory outside your actual application - code, useful if you have many functional tests or for other reasons - want to keep tests separate from actual application code (often a good - idea):: +Tests outside application code +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - setup.py # your setuptools Python package metadata +Putting tests into an extra directory outside your actual application code +might be useful if you have many functional tests or for other reasons want +to keep tests separate from actual application code (often a good idea):: + + setup.py mypkg/ __init__.py - appmodule.py + app.py + view.py tests/ test_app.py + test_view.py ... +This way your tests can run easily against an installed version +of ``mypkg``. -* inlining test directories into your application package, useful if you - have direct relation between (unit-)test and application modules and - want to distribute your tests along with your application:: +Note that using this scheme your test files must have **unique names**, because +``pytest`` will import them as *top-level* modules since there are no packages +to derive a full package name from. - setup.py # your setuptools Python package metadata +Tests as part of application code +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Inlining test directories into your application package +is useful if you have direct relation between tests and application modules and +want to distribute them along with your application:: + + setup.py mypkg/ __init__.py - appmodule.py - ... + app.py + view.py test/ + __init__.py test_app.py + test_view.py ... -Important notes relating to both schemes: +In this scheme, it is easy to your tests using the ``--pyargs`` option:: -- **make sure that "mypkg" is importable**, for example by typing once:: + pytest --pyargs mypkg - pip install -e . # install package using setup.py in editable mode +``pytest`` will discover where ``mypkg`` is installed and collect tests from there. -- **avoid "__init__.py" files in your test directories**. - This way your tests can run easily against an installed version - of ``mypkg``, independently from the installed package if it contains - the tests or not. - -- With inlined tests you might put ``__init__.py`` into test - directories and make them installable as part of your application. - Using the ``pytest --pyargs mypkg`` invocation pytest will - discover where mypkg is installed and collect tests from there. - With the "external" test you can still distribute tests but they - will not be installed or become importable. - -Typically you can run tests by pointing to test directories or modules:: - - pytest tests/test_app.py # for external test dirs - pytest mypkg/test/test_app.py # for inlined test dirs - pytest mypkg # run tests in all below test directories - pytest # run all tests below current dir - ... - -Because of the above ``editable install`` mode you can change your -source code (both tests and the app) and rerun tests at will. -Once you are done with your work, you can `use tox`_ to make sure -that the package is really correct and tests pass in all -required configurations. .. note:: @@ -144,7 +134,13 @@ for installing your application and any dependencies as well as the ``pytest`` package itself. This ensures your code and dependencies are isolated from the system Python installation. -If you frequently release code and want to make sure that your actual +You can then install your package in "editable" mode:: + + pip install -e . + +which lets you change your source code (both tests and application) and rerun tests at will. + +Once you are done with your work and want to make sure that your actual package passes all tests you may want to look into `tox`_, the virtualenv test automation tool and its `pytest support `_. @@ -154,11 +150,6 @@ options. It will run tests against the installed package and not against your source code checkout, helping to detect packaging glitches. -Continuous integration services such as Jenkins_ can make use of the -``--junitxml=PATH`` option to create a JUnitXML file and generate reports (e.g. -by publishing the results in a nice format with the `Jenkins xUnit Plugin -`_). - Integrating with setuptools / ``python setup.py test`` / ``pytest-runner`` -------------------------------------------------------------------------- diff --git a/tox.ini b/tox.ini index 1b9fb9f5a..a47fc5132 100644 --- a/tox.ini +++ b/tox.ini @@ -106,6 +106,8 @@ commands= pytest -ra {posargs:testing/test_unittest.py} [testenv:docs] +skipsdist=True +usedevelop=True basepython=python changedir=doc/en deps= From 581857aab631b2a5a17232b3a332dd44b832e689 Mon Sep 17 00:00:00 2001 From: Bruno Oliveira Date: Thu, 9 Mar 2017 20:46:22 -0300 Subject: [PATCH 2/4] Fix typo --- doc/en/goodpractices.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/en/goodpractices.rst b/doc/en/goodpractices.rst index 65c5624fa..5f6ea710b 100644 --- a/doc/en/goodpractices.rst +++ b/doc/en/goodpractices.rst @@ -76,7 +76,7 @@ want to distribute them along with your application:: test_view.py ... -In this scheme, it is easy to your tests using the ``--pyargs`` option:: +In this scheme, it is easy to your run tests using the ``--pyargs`` option:: pytest --pyargs mypkg From 4a9348324d414f4f7a68b8a096a9ff960101aa1f Mon Sep 17 00:00:00 2001 From: Bruno Oliveira Date: Tue, 14 Mar 2017 07:59:01 -0300 Subject: [PATCH 3/4] Add more information to test-layout docs as discussed during PR --- doc/en/goodpractices.rst | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/doc/en/goodpractices.rst b/doc/en/goodpractices.rst index 5f6ea710b..ff71f1c9c 100644 --- a/doc/en/goodpractices.rst +++ b/doc/en/goodpractices.rst @@ -56,7 +56,44 @@ of ``mypkg``. Note that using this scheme your test files must have **unique names**, because ``pytest`` will import them as *top-level* modules since there are no packages -to derive a full package name from. +to derive a full package name from. In other words, the test files in the example above will +be imported as ``test_app`` and ``test_view`` top-level modules by adding ``tests/`` to +``sys.path``. + +If you need to have test modules with the same name, you might add ``__init__.py`` files to your +``tests`` folder and subfolders, changing them to packages:: + + mypkg/ + ... + tests/ + __init__.py + foo/ + __init__.py + test_view.py + bar/ + __init__.py + test_view.py + +Now pytest will load the modules as ``tests.foo.test_view`` and ``tests.bar.test_view``, allowing +you to have modules with the same name. But now this introduces a subtle problem: in order to load +the test modules from the ``tests`` directory, pytest prepends the root of the repository to +``sys.path``, which adds the side-effect that now ``mypkg`` is also importable. +This is problematic if you are using a tool like `tox`_ to test your package in a virtual environment, +because you want to test the *installed* version of your package, not the local code from the repository. + +There are a couple of solutions to this: + +* Do not add the ``tests/__init__.py`` file. This will cause your test modules to be loaded as + ``foo.test_view`` and ``bar.test_view``, which is usually fine and avoids having the root of + the repository being added to the ``PYTHONPATH``. + +* Add ``addopts=--import-mode=append`` to your ``pytest.ini`` file. This will cause ``pytest`` to + load your modules by *appending* the root directory instead of prepending. + +* Alternatively, consider using the ``src`` layout explained in detail in this excellent + `blog post by Ionel Cristian Mărieș `_ + which prevents a lot of the pitfalls described here. + Tests as part of application code ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ From 272aba98e2009aaa37841594691ff88cb866b7d3 Mon Sep 17 00:00:00 2001 From: Bruno Oliveira Date: Tue, 14 Mar 2017 18:39:02 -0300 Subject: [PATCH 4/4] Mention the src layout as recommended practice --- doc/en/goodpractices.rst | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/doc/en/goodpractices.rst b/doc/en/goodpractices.rst index ff71f1c9c..c17fabbc9 100644 --- a/doc/en/goodpractices.rst +++ b/doc/en/goodpractices.rst @@ -63,6 +63,7 @@ be imported as ``test_app`` and ``test_view`` top-level modules by adding ``test If you need to have test modules with the same name, you might add ``__init__.py`` files to your ``tests`` folder and subfolders, changing them to packages:: + setup.py mypkg/ ... tests/ @@ -81,19 +82,27 @@ the test modules from the ``tests`` directory, pytest prepends the root of the r This is problematic if you are using a tool like `tox`_ to test your package in a virtual environment, because you want to test the *installed* version of your package, not the local code from the repository. -There are a couple of solutions to this: +In this situation, it is **strongly** suggested to use a ``src`` layout where application root package resides in a +sub-directory of your root:: -* Do not add the ``tests/__init__.py`` file. This will cause your test modules to be loaded as - ``foo.test_view`` and ``bar.test_view``, which is usually fine and avoids having the root of - the repository being added to the ``PYTHONPATH``. + setup.py + src/ + mypkg/ + __init__.py + app.py + view.py + tests/ + __init__.py + foo/ + __init__.py + test_view.py + bar/ + __init__.py + test_view.py -* Add ``addopts=--import-mode=append`` to your ``pytest.ini`` file. This will cause ``pytest`` to - load your modules by *appending* the root directory instead of prepending. - -* Alternatively, consider using the ``src`` layout explained in detail in this excellent - `blog post by Ionel Cristian Mărieș `_ - which prevents a lot of the pitfalls described here. +This layout prevents a lot of common pitfalls and has many benefits, which are better explained in this excellent +`blog post by Ionel Cristian Mărieș `_. Tests as part of application code ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -119,6 +128,8 @@ In this scheme, it is easy to your run tests using the ``--pyargs`` option:: ``pytest`` will discover where ``mypkg`` is installed and collect tests from there. +Note that this layout also works in conjunction with the ``src`` layout mentioned in the previous section. + .. note::