improve and clarify skipping docs
This commit is contained in:
parent
318e8a404b
commit
22fac92ca0
10
ISSUES.txt
10
ISSUES.txt
|
@ -88,6 +88,16 @@ etc. Idea is to allow Python expressions which can operate
|
||||||
on common spellings for operating systems and python
|
on common spellings for operating systems and python
|
||||||
interpreter versions.
|
interpreter versions.
|
||||||
|
|
||||||
|
pytest.mark.xfail signature change
|
||||||
|
-------------------------------------------------------
|
||||||
|
tags: feature 2.1
|
||||||
|
|
||||||
|
change to pytest.mark.xfail(reason, (optional)condition)
|
||||||
|
to better implement the word meaning. It also signals
|
||||||
|
better that we always have some kind of an implementation
|
||||||
|
reason that can be formualated.
|
||||||
|
Compatibility? Maybe rename to "pytest.mark.xfail"?
|
||||||
|
|
||||||
introduce py.test.mark registration
|
introduce py.test.mark registration
|
||||||
-----------------------------------------
|
-----------------------------------------
|
||||||
tags: feature 2.1
|
tags: feature 2.1
|
||||||
|
|
|
@ -1,20 +1,22 @@
|
||||||
|
|
||||||
.. _`skip and xfail`:
|
.. _`skip and xfail`:
|
||||||
|
|
||||||
skip and xfail mechanisms
|
skip and xfail: dealing with tests that can not succeed
|
||||||
=====================================================================
|
=====================================================================
|
||||||
|
|
||||||
You can skip or "xfail" test functions, either by marking functions
|
If you have test functions that cannot be run on certain platforms
|
||||||
with a decorator or by calling the ``pytest.skip|xfail`` functions.
|
or that you expect to fail you can mark them accordingly or you
|
||||||
|
may call helper functions during execution of setup or test functions.
|
||||||
|
|
||||||
A *skip* means that you expect your test to pass unless a certain configuration or condition (e.g. wrong Python interpreter, missing dependency) prevents it to run. And *xfail* means that you expect your test to fail because there is an
|
A *skip* means that you expect your test to pass unless a certain
|
||||||
implementation problem. py.test counts and lists *xfailing* tests separately
|
configuration or condition (e.g. wrong Python interpreter, missing
|
||||||
and it is possible to give additional information, such as bug number or a URL.
|
dependency) prevents it to run. And *xfail* means that your test
|
||||||
|
can run but you expect it to fail because there is an implementation problem.
|
||||||
|
|
||||||
Detailed information about skipped/xfailed tests is by default not shown
|
py.test counts and lists *skip* and *xfail* tests separately. However,
|
||||||
at the end of a test run to avoid cluttering the output. You can use
|
detailed information about skipped/xfailed tests is not shown by default
|
||||||
the ``-r`` option to see details corresponding to the "short" letters
|
to avoid cluttering the output. You can use the ``-r`` option to see
|
||||||
shown in the test progress::
|
details corresponding to the "short" letters shown in the test
|
||||||
|
progress::
|
||||||
|
|
||||||
py.test -rxs # show extra info on skips and xfails
|
py.test -rxs # show extra info on skips and xfails
|
||||||
|
|
||||||
|
@ -22,7 +24,7 @@ shown in the test progress::
|
||||||
|
|
||||||
.. _skipif:
|
.. _skipif:
|
||||||
|
|
||||||
Skipping a single function
|
Marking a test function to be skipped
|
||||||
-------------------------------------------
|
-------------------------------------------
|
||||||
|
|
||||||
Here is an example of marking a test function to be skipped
|
Here is an example of marking a test function to be skipped
|
||||||
|
@ -34,9 +36,9 @@ when run on a Python3 interpreter::
|
||||||
...
|
...
|
||||||
|
|
||||||
During test function setup the skipif condition is
|
During test function setup the skipif condition is
|
||||||
evaluated by calling ``eval(expr, namespace)``. The namespace
|
evaluated by calling ``eval('sys.version_info >= (3,0)', namespace)``.
|
||||||
contains all the module globals of the test function so that
|
(*New in version 2.0.2*) The namespace contains all the module globals of the test function so that
|
||||||
you can for example check for versions::
|
you can for example check for versions of a module you are using::
|
||||||
|
|
||||||
import mymodule
|
import mymodule
|
||||||
|
|
||||||
|
@ -44,23 +46,15 @@ you can for example check for versions::
|
||||||
def test_function():
|
def test_function():
|
||||||
...
|
...
|
||||||
|
|
||||||
The test function will be skipped and not run if
|
The test function will not be run ("skipped") if
|
||||||
mymodule is below the specified version. The reason
|
``mymodule`` is below the specified version. The reason
|
||||||
for specifying the condition as a string is mainly that
|
for specifying the condition as a string is mainly that
|
||||||
you can see more detailed reporting of xfail/skip reasons.
|
py.test can report a summary of skip conditions.
|
||||||
|
For information on the construction of the ``namespace``
|
||||||
|
see `evaluation of skipif/xfail conditions`_.
|
||||||
|
|
||||||
Actually, the namespace is first initialized by
|
You can of course create a shortcut for your conditional skip
|
||||||
putting the ``sys`` and ``os`` modules and the test
|
decorator at module level like this::
|
||||||
``config`` object into it. And is then updated with
|
|
||||||
the module globals. The latter allows you to skip based
|
|
||||||
on a test configuration value::
|
|
||||||
|
|
||||||
@pytest.mark.skipif("not config.getvalue('db')")
|
|
||||||
def test_function(...):
|
|
||||||
...
|
|
||||||
|
|
||||||
You can create a shortcut for your conditional skip decorator
|
|
||||||
at module level like this::
|
|
||||||
|
|
||||||
win32only = pytest.mark.skipif("sys.platform != 'win32'")
|
win32only = pytest.mark.skipif("sys.platform != 'win32'")
|
||||||
|
|
||||||
|
@ -68,11 +62,10 @@ at module level like this::
|
||||||
def test_function():
|
def test_function():
|
||||||
...
|
...
|
||||||
|
|
||||||
|
|
||||||
skip all test functions of a class
|
skip all test functions of a class
|
||||||
--------------------------------------
|
--------------------------------------
|
||||||
|
|
||||||
As with all function :ref:`mark` you can skip test functions at the
|
As with all function :ref:`marking <mark>` you can skip test functions at the
|
||||||
`whole class- or module level`_. Here is an example
|
`whole class- or module level`_. Here is an example
|
||||||
for skipping all methods of a test class based on the platform::
|
for skipping all methods of a test class based on the platform::
|
||||||
|
|
||||||
|
@ -82,9 +75,10 @@ for skipping all methods of a test class based on the platform::
|
||||||
def test_function(self):
|
def test_function(self):
|
||||||
"will not be setup or run under 'win32' platform"
|
"will not be setup or run under 'win32' platform"
|
||||||
|
|
||||||
The ``pytestmark`` decorator will be applied to each test function.
|
The ``pytestmark`` special name tells py.test to apply it to each test
|
||||||
If your code targets python2.6 or above you can equivalently use
|
function in the class. If your code targets python2.6 or above you can
|
||||||
the skipif decorator on classes::
|
more naturally use the skipif decorator (and any other marker) on
|
||||||
|
classes::
|
||||||
|
|
||||||
@pytest.mark.skipif("sys.platform == 'win32'")
|
@pytest.mark.skipif("sys.platform == 'win32'")
|
||||||
class TestPosixCalls:
|
class TestPosixCalls:
|
||||||
|
@ -155,6 +149,31 @@ Running it with the report-on-xfail option gives this output::
|
||||||
|
|
||||||
======================== 6 xfailed in 0.06 seconds =========================
|
======================== 6 xfailed in 0.06 seconds =========================
|
||||||
|
|
||||||
|
.. _`evaluation of skipif/xfail conditions`:
|
||||||
|
|
||||||
|
evaluation of skipif/xfail expressions
|
||||||
|
----------------------------------------------------
|
||||||
|
|
||||||
|
.. versionadded:: 2.0.2
|
||||||
|
|
||||||
|
The evaluation of a condition string in ``pytest.mark.skipif(conditionstring)``
|
||||||
|
or ``pytest.mark.xfail(conditionstring)`` takes place in a namespace
|
||||||
|
dictionary which is constructed as follows:
|
||||||
|
|
||||||
|
* the namespace is initialized by putting the ``sys`` and ``os`` modules
|
||||||
|
and the pytest ``config`` object into it.
|
||||||
|
|
||||||
|
* updated with the module globals of the test function for which the
|
||||||
|
expression is applied.
|
||||||
|
|
||||||
|
The pytest ``config`` object allows you to skip based on a test configuration value
|
||||||
|
which you might have added::
|
||||||
|
|
||||||
|
@pytest.mark.skipif("not config.getvalue('db')")
|
||||||
|
def test_function(...):
|
||||||
|
...
|
||||||
|
|
||||||
|
|
||||||
imperative xfail from within a test or setup function
|
imperative xfail from within a test or setup function
|
||||||
------------------------------------------------------
|
------------------------------------------------------
|
||||||
|
|
||||||
|
|
Loading…
Reference in New Issue