test_ok1/doc/test/features.txt

==================================================
py.test features
==================================================

py.test is an extensible tool for running all kinds
of tests on one or more machines.  It supports a variety
of testing methods including unit, functional, integration 
and doc-testing.  It is used in projects that run more 
than 10 thousand tests regularly as well as in single-file projects. 

py.test presents a clean and powerful command line interface
and strives to generally make testing a fun no-boilerplate effort. 
It works and is tested against linux, windows and osx 
on CPython 2.3 - CPython 2.6. 

.. contents:: List of Contents
    :depth: 1

.. _`autocollect`:

automatically collects and executes tests 
===============================================

py.test discovers tests automatically by inspect specified
directories or files.  By default, it collects all python
modules a leading ``test_`` or trailing ``_test`` filename.
From each test module every function with a leading ``test_``
or class with a leading ``Test`` name is collected.  

.. _`collection process`: extend.html#collection-process


funcargs and xUnit style setups 
===================================================

py.test provides powerful means for managing test
state and fixtures.  Apart from the `traditional 
xUnit style setup`_ for unittests it features the 
simple and powerful `funcargs mechanism`_ for handling
both complex and simple test scenarious. 

.. _`funcargs mechanism`: funcargs.html
.. _`traditional xUnit style setup`: xunit_setup.html

load-balance tests to multiple CPUs
===================================

For large test suites you can distribute your 
tests to multiple CPUs by issuing for example::

    py.test -n 3 

Read more on `distributed testing`_. 

.. _`distributed testing`: dist.html

Distribute tests across machines
===================================

py.test supports the sending of tests to 
remote ssh-accounts or socket servers. 
It can `ad-hoc run your test on multiple
platforms one a single test run`. Ad-hoc
means that there are **no installation 
requirements whatsoever** on the remote side. 

.. _`ad-hoc run your test on multiple platforms one a single test run`: dist.html#atonce

extensive debugging support 
===================================

testing starts immediately 
--------------------------

Testing starts as soon as the first ``test item`` 
is collected.  The collection process is iterative 
and does not need to complete before your first 
test items are executed. 

support for modules containing tests
--------------------------------------

As ``py.test`` operates as a separate cmdline 
tool you can easily have a command line utility and
some tests in the same file.  

debug with the ``print`` statement
----------------------------------

By default, ``py.test`` catches text written to stdout/stderr during
the execution of each individual test.  This output will only be
displayed however if the test fails; you will not see it
otherwise. This allows you to put debugging print statements in your
code without being overwhelmed by all the output that might be
generated by tests that do not fail.

Each failing test that produced output during the running of the test
function will have its output displayed in the ``recorded stdout`` section.

During Setup and Teardown ("Fixture") capturing is performed separately so 
that you will only see this output if the actual fixture functions fail. 

The catching of stdout/stderr output can be disabled using the 
``--nocapture`` or ``-s`` option to the ``py.test`` tool.  Any output will 
in this case be displayed as soon as it is generated.

test execution order
--------------------------------

Tests usually run in the order in which they appear in the files. 
However, tests should not rely on running one after another, as
this prevents more advanced usages: running tests
distributedly or selectively, or in "looponfailing" mode,
will cause them to run in random order. 

assert with the ``assert`` statement
----------------------------------------

``py.test`` allows to use the standard python
``assert statement`` for verifying expectations 
and values in Python tests.  For example, you can 
write the following in your tests:: 

     assert hasattr(x, 'attribute') 

to state that your object has a certain ``attribute``. In case this
assertion fails you will see the value of ``x``.  Intermediate
values are computed by executing the assert expression a second time. 
If you execute code with side effects, e.g. read from a file like this::

        assert f.read() != '...'

then you may get a warning from pytest if that assertions
first failed and then succeeded. 

asserting expected exceptions 
----------------------------------------

In order to write assertions about exceptions, you use
one of two forms::

    py.test.raises(Exception, func, *args, **kwargs) 
    py.test.raises(Exception, "func(*args, **kwargs)")

both of which execute the given function with args and kwargs and
asserts that the given ``Exception`` is raised.  The reporter will
provide you with helpful output in case of failures such as *no
exception* or *wrong exception*.

useful tracebacks, recursion detection 
--------------------------------------

A lot of care is taken to present nice tracebacks in case of test
failure. Try::

    py.test py/doc/example/pytest/failure_demo.py

to see a variety of tracebacks, each representing a different
failure situation.

``py.test`` uses the same order for presenting tracebacks as Python
itself: the oldest function call is shown first, and the most recent call is
shown last. A ``py.test`` reported traceback starts with your
failing test function.  If the maximum recursion depth has been
exceeded during the running of a test, for instance because of
infinite recursion, ``py.test`` will indicate where in the
code the recursion was taking place.  You can  inhibit
traceback "cutting" magic by supplying ``--fulltrace``. 

There is also the possibility of using ``--tb=short`` to get regular CPython
tracebacks. Or you can use ``--tb=no`` to not show any tracebacks at all.

no inheritance requirement 
--------------------------

Test classes are recognized by their leading ``Test`` name.  Unlike
``unitest.py``, you don't need to inherit from some base class to make
them be found by the test runner. Besides being easier, it also allows
you to write test classes that subclass from application level
classes.

testing for deprecated APIs
------------------------------

In your tests you can use ``py.test.deprecated_call(func, *args, **kwargs)``
to test that a particular function call triggers a DeprecationWarning. 
This is useful for testing phasing out of old APIs in your projects. 


advanced test selection / skipping 
=========================================================

dynamically skipping tests 
-------------------------------

If you want to skip tests you can use ``py.test.skip`` within
test or setup functions.  Example::

    py.test.skip("message")

You can also use a helper to skip on a failing import::

    docutils = py.test.importorskip("docutils")

or to skip if a library does not have the right version::

    docutils = py.test.importorskip("docutils", minversion="0.3")

The version will be read from the module's ``__version__`` attribute. 

.. _`selection by keyword`: 

selecting/unselecting tests by keyword 
---------------------------------------------

Pytest's keyword mechanism provides a powerful way to 
group and selectively run tests in your test code base. 
You can selectively run tests by specifiying a keyword 
on the command line.  Examples::

    py.test -k test_simple   
    py.test -k "-test_simple"

will run all tests matching (or not matching) the
"test_simple" keyword.  Note that you need to quote 
the keyword if "-" is recognized as an indicator 
for a commandline option.  Lastly, you may use::

    py.test. -k "test_simple:"

which will run all tests after the expression has *matched once*, i.e. 
all tests that are seen after a test that matches the "test_simple" 
keyword. 

By default, all filename parts and
class/function names of a test function are put into the set
of keywords for a given test.  You can specify additional 
kewords like this::

    @py.test.mark(webtest=True)
    def test_send_http():
        ... 

and then use those keywords to select tests. 

disabling a test class
---------------------- 

If you want to disable a complete test class you
can set the class-level attribute ``disabled``. 
For example, in order to avoid running some tests on Win32:: 

    class TestPosixOnly: 
        disabled = sys.platform == 'win32'
    
        def test_xxx(self):
            ... 

.. _`test generators`: funcargs.html#test-generators

.. _`generative tests`: 

generative tests: yielding parametrized tests
====================================================

Deprecated since 1.0 in favour of `test generators`_.

*Generative tests* are test methods that are *generator functions* which
``yield`` callables and their arguments.  This is useful for running a
test function multiple times against different parameters.  Example::

    def test_generative(): 
        for x in (42,17,49): 
            yield check, x 
    
    def check(arg): 
        assert arg % 7 == 0   # second generated tests fails!

Note that ``test_generative()`` will cause three tests 
to get run, notably ``check(42)``, ``check(17)`` and ``check(49)``
of which the middle one will obviously fail. 

To make it easier to distinguish the generated tests it is possible to specify an explicit name for them, like for example::

    def test_generative(): 
        for x in (42,17,49): 
            yield "case %d" % x, check, x 

easy to extend 
=========================================

Since 1.0 py.test has advanced `extension mechanisms`_
and a growing `list of plugins`_. 
One can can easily modify or add aspects for for 
purposes such as: 

* reporting extensions
* customizing collection and execution of tests 
* running non-python tests
* managing custom test state setup 

.. _`list of plugins`: plugin/index.html
.. _`extension mechanisms`: extend.html

.. _`reStructured Text`: http://docutils.sourceforge.net
.. _`Python debugger`: http://docs.python.org/lib/module-pdb.html


.. _nose: http://somethingaboutorange.com/mrl/projects/nose/