244 lines
8.2 KiB
Plaintext
244 lines
8.2 KiB
Plaintext
|
|
automatic collection of tests on all levels
|
|
-------------------------------------------
|
|
|
|
The automated test collection process walks the current
|
|
directory (or the directory given as a command line argument)
|
|
and all its subdirectories and collects python modules with a
|
|
leading ``test_`` or trailing ``_test`` filename. From each
|
|
test module every function with a leading ``test_`` or class with
|
|
a leading ``Test`` name is collected. The collecting process can
|
|
be customized at directory, module or class level. (see
|
|
`collection process`_ for some implementation details).
|
|
|
|
.. _`generative tests`:
|
|
.. _`collection process`: impl-test.html#collection-process
|
|
|
|
assert with the ``assert`` statement
|
|
------------------------------------
|
|
|
|
``py.test`` allows to use the standard python
|
|
``assert statement`` for verifying expectations
|
|
and values in Python tests. For example, you can
|
|
write the following in your tests::
|
|
|
|
assert hasattr(x, 'attribute')
|
|
|
|
to state that your object has a certain ``attribute``. In case this
|
|
assertion fails you will see the value of ``x``. Intermediate
|
|
values are computed by executing the assert expression a second time.
|
|
If you execute code with side effects, e.g. read from a file like this::
|
|
|
|
assert f.read() != '...'
|
|
|
|
then you may get a warning from pytest if that assertions
|
|
first failed and then succeeded.
|
|
|
|
asserting expected exceptions
|
|
----------------------------------------------
|
|
|
|
In order to write assertions about exceptions, you use
|
|
one of two forms::
|
|
|
|
py.test.raises(Exception, func, *args, **kwargs)
|
|
py.test.raises(Exception, "func(*args, **kwargs)")
|
|
|
|
both of which execute the given function with args and kwargs and
|
|
asserts that the given ``Exception`` is raised. The reporter will
|
|
provide you with helpful output in case of failures such as *no
|
|
exception* or *wrong exception*.
|
|
|
|
dynamically skipping tests
|
|
----------------------------------------
|
|
|
|
If you want to skip tests you can use ``py.test.skip`` within
|
|
test or setup functions. Example::
|
|
|
|
py.test.skip("message")
|
|
|
|
You can also use a helper to skip on a failing import::
|
|
|
|
docutils = py.test.importorskip("docutils")
|
|
|
|
or to skip if a library does not have the right version::
|
|
|
|
docutils = py.test.importorskip("docutils", minversion="0.3")
|
|
|
|
The version will be read from the module's ``__version__`` attribute.
|
|
|
|
|
|
generative tests: yielding more tests
|
|
-------------------------------------
|
|
|
|
*Generative tests* are test methods that are *generator functions* which
|
|
``yield`` callables and their arguments. This is most useful for running a
|
|
test function multiple times against different parameters. Example::
|
|
|
|
def test_generative():
|
|
for x in (42,17,49):
|
|
yield check, x
|
|
|
|
def check(arg):
|
|
assert arg % 7 == 0 # second generated tests fails!
|
|
|
|
Note that ``test_generative()`` will cause three tests
|
|
to get run, notably ``check(42)``, ``check(17)`` and ``check(49)``
|
|
of which the middle one will obviously fail.
|
|
|
|
To make it easier to distinguish the generated tests it is possible to specify an explicit name for them, like for example::
|
|
|
|
def test_generative():
|
|
for x in (42,17,49):
|
|
yield "case %d" % x, check, x
|
|
|
|
|
|
.. _`selection by keyword`:
|
|
|
|
selecting/unselecting tests by keyword
|
|
---------------------------------------------
|
|
|
|
Pytest's keyword mechanism provides a powerful way to
|
|
group and selectively run tests in your test code base.
|
|
You can selectively run tests by specifiying a keyword
|
|
on the command line. Examples:
|
|
|
|
py.test -k test_simple
|
|
py.test -k "-test_simple"
|
|
|
|
will run all tests matching (or not matching) the
|
|
"test_simple" keyword. Note that you need to quote
|
|
the keyword if "-" is recognized as an indicator
|
|
for a commandline option. Lastly, you may use
|
|
|
|
py.test. -k "test_simple:"
|
|
|
|
which will run all tests after the expression has *matched once*, i.e.
|
|
all tests that are seen after a test that matches the "test_simple"
|
|
keyword.
|
|
|
|
By default, all filename parts and
|
|
class/function names of a test function are put into the set
|
|
of keywords for a given test. You may specify additional
|
|
kewords like this::
|
|
|
|
@py.test.mark(webtest=True)
|
|
def test_send_http():
|
|
...
|
|
|
|
testing with multiple python versions / executables
|
|
---------------------------------------------------
|
|
|
|
With ``--tx EXECUTABLE`` you can specify a python
|
|
executable (e.g. ``python2.2``) with which the tests
|
|
will be executed.
|
|
|
|
|
|
testing starts immediately
|
|
--------------------------
|
|
|
|
Testing starts as soon as the first ``test item``
|
|
is collected. The collection process is iterative
|
|
and does not need to complete before your first
|
|
test items are executed.
|
|
|
|
support for modules containing tests
|
|
--------------------------------------
|
|
|
|
As ``py.test`` operates as a separate cmdline
|
|
tool you can easily have a command line utility and
|
|
some tests in the same file.
|
|
|
|
debug with the ``print`` statement
|
|
----------------------------------
|
|
|
|
By default, ``py.test`` catches text written to stdout/stderr during
|
|
the execution of each individual test. This output will only be
|
|
displayed however if the test fails; you will not see it
|
|
otherwise. This allows you to put debugging print statements in your
|
|
code without being overwhelmed by all the output that might be
|
|
generated by tests that do not fail.
|
|
|
|
Each failing test that produced output during the running of the test
|
|
will have its output displayed in the ``recorded stdout`` section.
|
|
|
|
The catching of stdout/stderr output can be disabled using the
|
|
``--nocapture`` option to the ``py.test`` tool. Any output will
|
|
in this case be displayed as soon as it is generated.
|
|
|
|
test execution order
|
|
--------------------------------
|
|
|
|
Tests usually run in the order in which they appear in the files.
|
|
However, tests should not rely on running one after another, as
|
|
this prevents more advanced usages: running tests
|
|
distributedly or selectively, or in "looponfailing" mode,
|
|
will cause them to run in random order.
|
|
|
|
useful tracebacks, recursion detection
|
|
--------------------------------------
|
|
|
|
A lot of care is taken to present nice tracebacks in case of test
|
|
failure. Try::
|
|
|
|
py.test py/doc/example/pytest/failure_demo.py
|
|
|
|
to see a variety of 17 tracebacks, each tailored to a different
|
|
failure situation.
|
|
|
|
``py.test`` uses the same order for presenting tracebacks as Python
|
|
itself: the oldest function call is shown first, and the most recent call is
|
|
shown last. A ``py.test`` reported traceback starts with your
|
|
failing test function. If the maximum recursion depth has been
|
|
exceeded during the running of a test, for instance because of
|
|
infinite recursion, ``py.test`` will indicate where in the
|
|
code the recursion was taking place. You can inhibit
|
|
traceback "cutting" magic by supplying ``--fulltrace``.
|
|
|
|
There is also the possibility of using ``--tb=short`` to get regular CPython
|
|
tracebacks. Or you can use ``--tb=no`` to not show any tracebacks at all.
|
|
|
|
no inheritance requirement
|
|
--------------------------
|
|
|
|
Test classes are recognized by their leading ``Test`` name. Unlike
|
|
``unitest.py``, you don't need to inherit from some base class to make
|
|
them be found by the test runner. Besides being easier, it also allows
|
|
you to write test classes that subclass from application level
|
|
classes.
|
|
|
|
disabling a test class
|
|
----------------------
|
|
|
|
If you want to disable a complete test class you
|
|
can set the class-level attribute ``disabled``.
|
|
For example, in order to avoid running some tests on Win32::
|
|
|
|
class TestPosixOnly:
|
|
disabled = sys.platform == 'win32'
|
|
|
|
def test_xxx(self):
|
|
...
|
|
|
|
testing for deprecated APIs
|
|
------------------------------
|
|
|
|
In your tests you can use ``py.test.deprecated_call(func, *args, **kwargs)``
|
|
to test that a particular function call triggers a DeprecationWarning.
|
|
This is useful for testing phasing out of old APIs in your projects.
|
|
|
|
doctest support
|
|
-------------------
|
|
|
|
If you want to integrate doctests, ``py.test`` now by default
|
|
picks up files matching the ``test_*.txt`` or ``*_test.txt``
|
|
patterns and processes them as text files containing doctests.
|
|
This is an experimental feature and likely to change
|
|
its implementation.
|
|
|
|
|
|
.. _`reStructured Text`: http://docutils.sourceforge.net
|
|
.. _`Python debugger`: http://docs.python.org/lib/module-pdb.html
|
|
|
|
|
|
.. _nose: http://somethingaboutorange.com/mrl/projects/nose/
|