From 15cdf137d57e9b83176cad20f9b5daf2dd9facd0 Mon Sep 17 00:00:00 2001 From: Bruno Oliveira Date: Thu, 3 Aug 2017 22:00:11 -0300 Subject: [PATCH 1/2] Document which pytest features work with `unittest` Fix #2626 --- changelog/2626.doc | 1 + doc/en/mark.rst | 1 + doc/en/skipping.rst | 1 + doc/en/unittest.rst | 124 ++++++++++++++++++++++++++++---------------- doc/en/usage.rst | 7 +++ 5 files changed, 90 insertions(+), 44 deletions(-) create mode 100644 changelog/2626.doc diff --git a/changelog/2626.doc b/changelog/2626.doc new file mode 100644 index 000000000..d244ca12c --- /dev/null +++ b/changelog/2626.doc @@ -0,0 +1 @@ +Explicitly document which pytest features work with ``unittest``. diff --git a/doc/en/mark.rst b/doc/en/mark.rst index ab9546d31..0b0e072a0 100644 --- a/doc/en/mark.rst +++ b/doc/en/mark.rst @@ -10,6 +10,7 @@ By using the ``pytest.mark`` helper you can easily set metadata on your test functions. There are some builtin markers, for example: +* :ref:`skip ` - always skip a test function * :ref:`skipif ` - skip a test function if a certain condition is met * :ref:`xfail ` - produce an "expected failure" outcome if a certain condition is met diff --git a/doc/en/skipping.rst b/doc/en/skipping.rst index f3c292a18..8690035a3 100644 --- a/doc/en/skipping.rst +++ b/doc/en/skipping.rst @@ -27,6 +27,7 @@ corresponding to the "short" letters shown in the test progress:: (See :ref:`how to change command line options defaults`) .. _skipif: +.. _skip: .. _`condition booleans`: Skipping test functions diff --git a/doc/en/unittest.rst b/doc/en/unittest.rst index 56721ff9b..a2813333f 100644 --- a/doc/en/unittest.rst +++ b/doc/en/unittest.rst @@ -2,58 +2,80 @@ .. _`unittest.TestCase`: .. _`unittest`: -Support for unittest.TestCase / Integration of fixtures -===================================================================== +unittest.TestCase Support +========================= -.. _`unittest.py style`: http://docs.python.org/library/unittest.html +.. _`unittest.py style`: https://docs.python.org/3/library/unittest.html -``pytest`` has support for running Python `unittest.py style`_ tests. -It's meant for leveraging existing unittest-style projects -to use pytest features. Concretely, pytest will automatically -collect ``unittest.TestCase`` subclasses and their ``test`` methods in -test files. It will invoke typical setup/teardown methods and -generally try to make test suites written to run on unittest, to also -run using ``pytest``. We assume here that you are familiar with writing -``unittest.TestCase`` style tests and rather focus on -integration aspects. +``pytest`` supports running Python `unittest.py style`_ tests out of the box. +It's meant for leveraging existing ``unittest``-style test suites +to use pytest as a test runner and also allow to incrementally adapt +the test suite to take full advantage of pytest features. -Note that this is meant as a provisional way of running your test code -until you fully convert to pytest-style tests. To fully take advantage of -:ref:`fixtures `, :ref:`parametrization ` and -:ref:`hooks ` you should convert (tools like `unittest2pytest -`__ are helpful). -Also, not all 3rd party pluging are expected to work best with -``unittest.TestCase`` style tests. +To run an existing ``unittest``-style test suite using ``pytest``, simply type:: -Usage -------------------------------------------------------------------- + pytest tests -After :ref:`installation` type:: - pytest +pytest will automatically collect ``unittest.TestCase`` subclasses and +their ``test`` methods in ``test_*.py`` or ``*_test.py`` files. -and you should be able to run your unittest-style tests if they -are contained in ``test_*`` modules. If that works for you then -you can make use of most :ref:`pytest features `, for example -``--pdb`` debugging in failures, using :ref:`plain assert-statements `, -:ref:`more informative tracebacks `, stdout-capturing or -distributing tests to multiple CPUs via the ``-nNUM`` option if you -installed the ``pytest-xdist`` plugin. Please refer to -the general ``pytest`` documentation for many more examples. +Almost all ``unittest`` features are supported, like ``@unittest.skip`` style decorators, ``setUp/tearDown``, +``setUpClass/tearDownClass()``, etc (see :ref:`unittest-limitations`). -.. note:: +Benefits +-------- - Running tests from ``unittest.TestCase`` subclasses with ``--pdb`` will - disable tearDown and cleanup methods for the case that an Exception - occurs. This allows proper post mortem debugging for all applications - which have significant logic in their tearDown machinery. However, - supporting this feature has the following side effect: If people - overwrite ``unittest.TestCase`` ``__call__`` or ``run``, they need to - to overwrite ``debug`` in the same way (this is also true for standard - unittest). +You can make use of several pytest features, most without changing any existing code: -Mixing pytest fixtures into unittest.TestCase style tests ------------------------------------------------------------ +* Use :ref:`plain assert-statements ` instead of ``self.assert*`` functions (`unittest2pytest + `__ is immensely helpful in this); +* Obtain :ref:`more informative tracebacks `; +* :ref:`stdout and stderr ` capturing; +* :ref:`Test selection options ` using ``-k`` and ``-m`` flags; +* :ref:`maxfail`; +* :ref:`--pdb ` command-line option for debugging on test failures + (see :ref:`note ` below); +* Distribute tests to multiple CPUs using the `pytest-xdist `_ plugin; + +.. _unittest-limitations: + +Limitations +----------- + +.. _`load_tests protocol`: https://docs.python.org/3/library/unittest.html#load-tests-protocol +.. _`setUpModule/tearDownModule`: https://docs.python.org/3/library/unittest.html#setupmodule-and-teardownmodule +.. _`subtests`: https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests + +Pytest currently does not support the following ``unittest`` features: + +* `load_tests protocol`_; +* `setUpModule/tearDownModule`_; +* `subtests`_; + + +pytest features in ``unittest.TestCase`` subclasses +--------------------------------------------------- + +The following pytest features work in ``unittest.TestCase`` subclasses: + +* :ref:`Marks `: :ref:`skip `, :ref:`skipif `, :ref:`xfail `; +* :ref:`Auto-use fixtures `; + +The following pytest features **do not** work, and probably +never will due to different design philosophies: + +* :ref:`Fixtures ` (except for ``autouse`` fixtures, see :ref:`below `); +* :ref:`Parametrization `; +* :ref:`Custom hooks `; + + +Third party plugins may or may not work well, depending on the plugin and the test suite. + +.. _mixing-fixtures: + +Mixing pytest fixtures into ``unittest.TestCase`` subclasses using marks +------------------------------------------------------------------------ Running your unittest with ``pytest`` allows you to use its :ref:`fixture mechanism ` with ``unittest.TestCase`` style @@ -143,8 +165,8 @@ share the same ``self.db`` instance which was our intention when writing the class-scoped fixture function above. -autouse fixtures and accessing other fixtures -------------------------------------------------------------------- +Using autouse fixtures and accessing other fixtures +--------------------------------------------------- Although it's usually better to explicitly declare use of fixtures you need for a given test, you may sometimes want to have fixtures that are @@ -165,6 +187,7 @@ creation of a per-test temporary directory:: import unittest class MyTest(unittest.TestCase): + @pytest.fixture(autouse=True) def initdir(self, tmpdir): tmpdir.chdir() # change to pytest-provided temporary directory @@ -200,3 +223,16 @@ was executed ahead of the ``test_method``. You can also gradually move away from subclassing from ``unittest.TestCase`` to *plain asserts* and then start to benefit from the full pytest feature set step by step. + +.. _pdb-unittest-note: + +.. note:: + + Running tests from ``unittest.TestCase`` subclasses with ``--pdb`` will + disable tearDown and cleanup methods for the case that an Exception + occurs. This allows proper post mortem debugging for all applications + which have significant logic in their tearDown machinery. However, + supporting this feature has the following side effect: If people + overwrite ``unittest.TestCase`` ``__call__`` or ``run``, they need to + to overwrite ``debug`` in the same way (this is also true for standard + unittest). diff --git a/doc/en/usage.rst b/doc/en/usage.rst index e19415892..9360a8ed2 100644 --- a/doc/en/usage.rst +++ b/doc/en/usage.rst @@ -41,6 +41,8 @@ Getting help on version, option names, environment variables pytest -h | --help # show help on command line and config file options +.. _maxfail: + Stopping after the first (or N) failures --------------------------------------------------- @@ -49,6 +51,8 @@ To stop the testing process after the first (N) failures:: pytest -x # stop after first failure pytest --maxfail=2 # stop after two failures +.. _select-tests: + Specifying tests / selecting tests --------------------------------------------------- @@ -135,6 +139,9 @@ with Ctrl+C to find out where the tests are *hanging*. By default no output will be shown (because KeyboardInterrupt is caught by pytest). By using this option you make sure a trace is shown. + +.. _pdb-option: + Dropping to PDB_ (Python Debugger) on failures ----------------------------------------------- From 37353a854e2545a9b1d8894660a6cf3a12294eba Mon Sep 17 00:00:00 2001 From: Bruno Oliveira Date: Fri, 4 Aug 2017 17:56:13 -0300 Subject: [PATCH 2/2] Implement suggestions by Raphael --- doc/en/unittest.rst | 53 +++++++++++++++++++++------------------------ 1 file changed, 25 insertions(+), 28 deletions(-) diff --git a/doc/en/unittest.rst b/doc/en/unittest.rst index a2813333f..92b9e653e 100644 --- a/doc/en/unittest.rst +++ b/doc/en/unittest.rst @@ -5,14 +5,12 @@ unittest.TestCase Support ========================= -.. _`unittest.py style`: https://docs.python.org/3/library/unittest.html - -``pytest`` supports running Python `unittest.py style`_ tests out of the box. -It's meant for leveraging existing ``unittest``-style test suites +``pytest`` supports running Python ``unittest``-based tests out of the box. +It's meant for leveraging existing ``unittest``-based test suites to use pytest as a test runner and also allow to incrementally adapt -the test suite to take full advantage of pytest features. +the test suite to take full advantage of pytest's features. -To run an existing ``unittest``-style test suite using ``pytest``, simply type:: +To run an existing ``unittest``-style test suite using ``pytest``, type:: pytest tests @@ -20,16 +18,28 @@ To run an existing ``unittest``-style test suite using ``pytest``, simply type:: pytest will automatically collect ``unittest.TestCase`` subclasses and their ``test`` methods in ``test_*.py`` or ``*_test.py`` files. -Almost all ``unittest`` features are supported, like ``@unittest.skip`` style decorators, ``setUp/tearDown``, -``setUpClass/tearDownClass()``, etc (see :ref:`unittest-limitations`). +Almost all ``unittest`` features are supported: -Benefits --------- +* ``@unittest.skip`` style decorators; +* ``setUp/tearDown``; +* ``setUpClass/tearDownClass()``; -You can make use of several pytest features, most without changing any existing code: +.. _`load_tests protocol`: https://docs.python.org/3/library/unittest.html#load-tests-protocol +.. _`setUpModule/tearDownModule`: https://docs.python.org/3/library/unittest.html#setupmodule-and-teardownmodule +.. _`subtests`: https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests + +Up to this point pytest does not have support for the following features: + +* `load_tests protocol`_; +* `setUpModule/tearDownModule`_; +* `subtests`_; + +Benefits out of the box +----------------------- + +By running your test suite with pytest you can make use of several features, +in most cases without having to modify existing code: -* Use :ref:`plain assert-statements ` instead of ``self.assert*`` functions (`unittest2pytest - `__ is immensely helpful in this); * Obtain :ref:`more informative tracebacks `; * :ref:`stdout and stderr ` capturing; * :ref:`Test selection options ` using ``-k`` and ``-m`` flags; @@ -37,21 +47,8 @@ You can make use of several pytest features, most without changing any existing * :ref:`--pdb ` command-line option for debugging on test failures (see :ref:`note ` below); * Distribute tests to multiple CPUs using the `pytest-xdist `_ plugin; - -.. _unittest-limitations: - -Limitations ------------ - -.. _`load_tests protocol`: https://docs.python.org/3/library/unittest.html#load-tests-protocol -.. _`setUpModule/tearDownModule`: https://docs.python.org/3/library/unittest.html#setupmodule-and-teardownmodule -.. _`subtests`: https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests - -Pytest currently does not support the following ``unittest`` features: - -* `load_tests protocol`_; -* `setUpModule/tearDownModule`_; -* `subtests`_; +* Use :ref:`plain assert-statements ` instead of ``self.assert*`` functions (`unittest2pytest + `__ is immensely helpful in this); pytest features in ``unittest.TestCase`` subclasses