237 lines
8.1 KiB
Plaintext
237 lines
8.1 KiB
Plaintext
==================================================
|
|
py.test feature overview
|
|
==================================================
|
|
|
|
.. contents::
|
|
:local:
|
|
:depth: 1
|
|
|
|
mature command line testing tool
|
|
====================================================
|
|
|
|
py.test is a command line tool to collect and run automated tests. It
|
|
runs well on Linux, Windows and OSX Python 2.4 through to 2.6 versions.
|
|
It can distribute a single test run to multiple machines. It is used in
|
|
many projects, ranging from running 10 thousands of tests integrated
|
|
with buildbot to a few inlined tests on a command line script.
|
|
|
|
.. _`autocollect`:
|
|
|
|
automatically collects and executes tests
|
|
===============================================
|
|
|
|
py.test discovers tests automatically by looking at
|
|
specified directories and its files for common
|
|
naming patterns. As ``py.test`` operates as a separate
|
|
cmdline tool you can easily have a command line utility and
|
|
some tests in the same file.
|
|
|
|
supports many testing practises and methods
|
|
==================================================================
|
|
|
|
py.test supports many testing methods conventionally used in
|
|
the Python community. It runs traditional `unittest.py`_,
|
|
`doctest.py`_, supports `xUnit style setup`_ and nose_ specific
|
|
setups and test suites. It offers minimal no-boilerplate model
|
|
for configuring and deploying tests written as simple Python
|
|
functions or methods. It also integrates `coverage testing
|
|
with figleaf`_ or `Javasript unit- and functional testing`_.
|
|
|
|
.. _`Javasript unit- and functional testing`: plugin/oejskit.html
|
|
.. _`coverage testing with figleaf`: plugin/figleaf.html
|
|
|
|
no-boilerplate test functions with Python
|
|
===================================================
|
|
|
|
automatic Python test discovery
|
|
------------------------------------
|
|
|
|
By default, all python modules with a ``test_*.py``
|
|
filename are inspected for finding tests:
|
|
|
|
* functions with a name beginning with ``test_``
|
|
* classes with a leading ``Test`` name and ``test`` prefixed methods.
|
|
* ``unittest.TestCase`` subclasses
|
|
|
|
test functions can run with different argument sets
|
|
-----------------------------------------------------------
|
|
|
|
py.test offers the unique `funcargs mechanism`_ for setting up
|
|
and passing project-specific objects to Python test functions.
|
|
Test Parametrization happens by triggering a call to the same test
|
|
functions with different argument values.
|
|
|
|
per-test capturing of output, including subprocesses
|
|
----------------------------------------------------
|
|
|
|
By default, ``py.test`` captures all writes to stdout/stderr.
|
|
Output from ``print`` statements as well as from subprocesses
|
|
is captured_. When a test fails, the associated captured outputs are shown.
|
|
This allows you to put debugging print statements in your code without
|
|
being overwhelmed by all the output that might be generated by tests
|
|
that do not fail.
|
|
|
|
.. _captured: plugin/capture.html
|
|
|
|
assert with the ``assert`` statement
|
|
----------------------------------------
|
|
|
|
``py.test`` allows to use the standard python
|
|
``assert statement`` for verifying expectations
|
|
and values in Python tests. For example, you can
|
|
write the following in your tests::
|
|
|
|
assert hasattr(x, 'attribute')
|
|
|
|
to state that your object has a certain ``attribute``. In case this
|
|
assertion fails you will see the value of ``x``. Intermediate
|
|
values are computed by executing the assert expression a second time.
|
|
If you execute code with side effects, e.g. read from a file like this::
|
|
|
|
assert f.read() != '...'
|
|
|
|
then you may get a warning from pytest if that assertions
|
|
first failed and then succeeded.
|
|
|
|
asserting expected exceptions
|
|
----------------------------------------
|
|
|
|
In order to write assertions about exceptions, you use
|
|
one of two forms::
|
|
|
|
py.test.raises(Exception, func, *args, **kwargs)
|
|
py.test.raises(Exception, "func(*args, **kwargs)")
|
|
|
|
both of which execute the specified function with args and kwargs and
|
|
asserts that the given ``Exception`` is raised. The reporter will
|
|
provide you with helpful output in case of failures such as *no
|
|
exception* or *wrong exception*.
|
|
|
|
|
|
information-rich tracebacks, PDB introspection
|
|
-------------------------------------------------------
|
|
|
|
.. _`example tracebacks`: http://paste.pocoo.org/show/134814/
|
|
|
|
A lot of care is taken to present useful failure information
|
|
and in particular nice and concise Python tracebacks. This
|
|
is especially useful if you need to regularly look at failures
|
|
from nightly runs, i.e. are detached from the actual test
|
|
running session. Here are `example tracebacks`_ for a number of failing
|
|
test functions. You can modify traceback printing styles through the
|
|
command line. Using the `--pdb`` option you can automatically activate
|
|
a PDB `Python debugger`_ when a test fails.
|
|
|
|
advanced skipping of tests
|
|
-------------------------------
|
|
|
|
If you want to skip tests you can use ``py.test.skip`` within
|
|
test or setup functions. Example::
|
|
|
|
def test_hello():
|
|
if sys.platform != "win32":
|
|
py.test.skip("only win32 supported")
|
|
|
|
You can also use a helper to skip on a failing import::
|
|
|
|
docutils = py.test.importorskip("docutils")
|
|
|
|
or to skip if a library does not have the right version::
|
|
|
|
docutils = py.test.importorskip("docutils", minversion="0.3")
|
|
|
|
The version will be read from the specified module's ``__version__`` attribute.
|
|
|
|
.. _`funcargs mechanism`: funcargs.html
|
|
.. _`unittest.py`: http://docs.python.org/library/unittest.html
|
|
.. _`doctest.py`: http://docs.python.org/library/doctest.html
|
|
.. _`xUnit style setup`: xunit_setup.html
|
|
.. _`pytest_nose`: plugin/nose.html
|
|
|
|
load-balance test runs to multiple CPUs
|
|
========================================
|
|
|
|
For large test suites you can distribute your
|
|
tests to multiple CPUs by issuing for example::
|
|
|
|
py.test -n 3
|
|
|
|
Read more on `distributed testing`_.
|
|
|
|
.. _`distributed testing`: dist.html
|
|
|
|
ad-hoc run tests cross-platform
|
|
==================================================
|
|
|
|
py.test supports the sending of tests to
|
|
remote ssh-accounts, socket servers.
|
|
It can `ad-hoc run your test on multiple
|
|
platforms one a single test run`. Ad-hoc
|
|
means that there are **no installation
|
|
requirements whatsoever** on the remote side.
|
|
|
|
.. _`ad-hoc run your test on multiple platforms one a single test run`: dist.html#atonce
|
|
|
|
advanced test selection and running modes
|
|
=========================================================
|
|
|
|
.. _`selection by keyword`:
|
|
|
|
``py.test --looponfailing`` allows to run a test suite,
|
|
memorize all failures and then loop over the failing set
|
|
of tests until they all pass. It will re-start running
|
|
the tests when it detects file changes in your project.
|
|
|
|
You can selectively run tests by specifiying a keyword
|
|
on the command line. Examples::
|
|
|
|
py.test -k test_simple
|
|
py.test -k "-test_simple"
|
|
|
|
will run all tests matching (or not matching) the
|
|
"test_simple" keyword. Note that you need to quote
|
|
the keyword if "-" is recognized as an indicator
|
|
for a commandline option. Lastly, you may use::
|
|
|
|
py.test. -k "test_simple:"
|
|
|
|
which will run all tests after the expression has *matched once*, i.e.
|
|
all tests that are seen after a test that matches the "test_simple"
|
|
keyword.
|
|
|
|
By default, all filename parts and
|
|
class/function names of a test function are put into the set
|
|
of keywords for a given test. You can specify additional
|
|
kewords like this::
|
|
|
|
@py.test.mark.webtest
|
|
def test_send_http():
|
|
...
|
|
|
|
and then use those keywords to select tests. See the `pytest_keyword`_
|
|
plugin for more information.
|
|
|
|
.. _`pytest_keyword`: plugin/keyword.html
|
|
|
|
easy to extend
|
|
=========================================
|
|
|
|
py.test has advanced `extension mechanisms`_
|
|
with a growing `list of default plugins`_.
|
|
One can can easily modify or add aspects for for
|
|
purposes such as:
|
|
|
|
* reporting extensions
|
|
* customizing collection and execution of tests
|
|
* running and managing non-python tests
|
|
* managing domain-specific test state setup
|
|
|
|
.. _`list of default plugins`: plugin/index.html
|
|
.. _`extension mechanisms`: customize.html#extensions
|
|
|
|
.. _`reStructured Text`: http://docutils.sourceforge.net
|
|
.. _`Python debugger`: http://docs.python.org/lib/module-pdb.html
|
|
|
|
|
|
.. _nose: http://somethingaboutorange.com/mrl/projects/nose/
|