diff --git a/ISSUES.txt b/ISSUES.txt index e92dd38e5..e39946dcc 100644 --- a/ISSUES.txt +++ b/ISSUES.txt @@ -88,6 +88,16 @@ etc. Idea is to allow Python expressions which can operate on common spellings for operating systems and python interpreter versions. +pytest.mark.xfail signature change +------------------------------------------------------- +tags: feature 2.1 + +change to pytest.mark.xfail(reason, (optional)condition) +to better implement the word meaning. It also signals +better that we always have some kind of an implementation +reason that can be formualated. +Compatibility? Maybe rename to "pytest.mark.xfail"? + introduce py.test.mark registration ----------------------------------------- tags: feature 2.1 diff --git a/doc/skipping.txt b/doc/skipping.txt index 94164ebd2..ee68f3027 100644 --- a/doc/skipping.txt +++ b/doc/skipping.txt @@ -1,20 +1,22 @@ - .. _`skip and xfail`: -skip and xfail mechanisms +skip and xfail: dealing with tests that can not succeed ===================================================================== -You can skip or "xfail" test functions, either by marking functions -with a decorator or by calling the ``pytest.skip|xfail`` functions. +If you have test functions that cannot be run on certain platforms +or that you expect to fail you can mark them accordingly or you +may call helper functions during execution of setup or test functions. -A *skip* means that you expect your test to pass unless a certain configuration or condition (e.g. wrong Python interpreter, missing dependency) prevents it to run. And *xfail* means that you expect your test to fail because there is an -implementation problem. py.test counts and lists *xfailing* tests separately -and it is possible to give additional information, such as bug number or a URL. +A *skip* means that you expect your test to pass unless a certain +configuration or condition (e.g. wrong Python interpreter, missing +dependency) prevents it to run. And *xfail* means that your test +can run but you expect it to fail because there is an implementation problem. -Detailed information about skipped/xfailed tests is by default not shown -at the end of a test run to avoid cluttering the output. You can use -the ``-r`` option to see details corresponding to the "short" letters -shown in the test progress:: +py.test counts and lists *skip* and *xfail* tests separately. However, +detailed information about skipped/xfailed tests is not shown by default +to avoid cluttering the output. You can use the ``-r`` option to see +details corresponding to the "short" letters shown in the test +progress:: py.test -rxs # show extra info on skips and xfails @@ -22,7 +24,7 @@ shown in the test progress:: .. _skipif: -Skipping a single function +Marking a test function to be skipped ------------------------------------------- Here is an example of marking a test function to be skipped @@ -34,9 +36,9 @@ when run on a Python3 interpreter:: ... During test function setup the skipif condition is -evaluated by calling ``eval(expr, namespace)``. The namespace -contains all the module globals of the test function so that -you can for example check for versions:: +evaluated by calling ``eval('sys.version_info >= (3,0)', namespace)``. +(*New in version 2.0.2*) The namespace contains all the module globals of the test function so that +you can for example check for versions of a module you are using:: import mymodule @@ -44,23 +46,15 @@ you can for example check for versions:: def test_function(): ... -The test function will be skipped and not run if -mymodule is below the specified version. The reason +The test function will not be run ("skipped") if +``mymodule`` is below the specified version. The reason for specifying the condition as a string is mainly that -you can see more detailed reporting of xfail/skip reasons. +py.test can report a summary of skip conditions. +For information on the construction of the ``namespace`` +see `evaluation of skipif/xfail conditions`_. -Actually, the namespace is first initialized by -putting the ``sys`` and ``os`` modules and the test -``config`` object into it. And is then updated with -the module globals. The latter allows you to skip based -on a test configuration value:: - - @pytest.mark.skipif("not config.getvalue('db')") - def test_function(...): - ... - -You can create a shortcut for your conditional skip decorator -at module level like this:: +You can of course create a shortcut for your conditional skip +decorator at module level like this:: win32only = pytest.mark.skipif("sys.platform != 'win32'") @@ -68,11 +62,10 @@ at module level like this:: def test_function(): ... - skip all test functions of a class -------------------------------------- -As with all function :ref:`mark` you can skip test functions at the +As with all function :ref:`marking ` you can skip test functions at the `whole class- or module level`_. Here is an example for skipping all methods of a test class based on the platform:: @@ -82,9 +75,10 @@ for skipping all methods of a test class based on the platform:: def test_function(self): "will not be setup or run under 'win32' platform" -The ``pytestmark`` decorator will be applied to each test function. -If your code targets python2.6 or above you can equivalently use -the skipif decorator on classes:: +The ``pytestmark`` special name tells py.test to apply it to each test +function in the class. If your code targets python2.6 or above you can +more naturally use the skipif decorator (and any other marker) on +classes:: @pytest.mark.skipif("sys.platform == 'win32'") class TestPosixCalls: @@ -155,6 +149,31 @@ Running it with the report-on-xfail option gives this output:: ======================== 6 xfailed in 0.06 seconds ========================= +.. _`evaluation of skipif/xfail conditions`: + +evaluation of skipif/xfail expressions +---------------------------------------------------- + +.. versionadded:: 2.0.2 + +The evaluation of a condition string in ``pytest.mark.skipif(conditionstring)`` +or ``pytest.mark.xfail(conditionstring)`` takes place in a namespace +dictionary which is constructed as follows: + +* the namespace is initialized by putting the ``sys`` and ``os`` modules + and the pytest ``config`` object into it. + +* updated with the module globals of the test function for which the + expression is applied. + +The pytest ``config`` object allows you to skip based on a test configuration value +which you might have added:: + + @pytest.mark.skipif("not config.getvalue('db')") + def test_function(...): + ... + + imperative xfail from within a test or setup function ------------------------------------------------------