improve and clarify skipping docs

ISSUES.txt

@@ -88,6 +88,16 @@ etc. Idea is to allow Python expressions which can operate
 on common spellings for operating systems and python
 interpreter versions.
 
+pytest.mark.xfail signature change
+-------------------------------------------------------
+tags: feature 2.1
+
+change to pytest.mark.xfail(reason, (optional)condition)
+to better implement the word meaning.  It also signals
+better that we always have some kind of an implementation
+reason that can be formualated.
+Compatibility? Maybe rename to "pytest.mark.xfail"?
+
 introduce py.test.mark registration
 -----------------------------------------
 tags: feature 2.1

doc/skipping.txt

@@ -1,20 +1,22 @@
-
 .. _`skip and xfail`:
 
-skip and xfail mechanisms
+skip and xfail: dealing with tests that can not succeed
 =====================================================================
 
-You can skip or "xfail" test functions, either by marking functions
-with a decorator or by calling the ``pytest.skip|xfail`` functions.
+If you have test functions that cannot be run on certain platforms
+or that you expect to fail you can mark them accordingly or you
+may call helper functions during execution of setup or test functions.
 
-A *skip* means that you expect your test to pass unless a certain configuration or condition (e.g. wrong Python interpreter, missing dependency) prevents it to run.  And *xfail* means that you expect your test to fail because there is an
-implementation problem.  py.test counts and lists *xfailing* tests separately
-and it is possible to give additional information, such as bug number or a URL.
+A *skip* means that you expect your test to pass unless a certain
+configuration or condition (e.g. wrong Python interpreter, missing
+dependency) prevents it to run.  And *xfail* means that your test
+can run but you expect it to fail because there is an implementation problem.
 
-Detailed information about skipped/xfailed tests is by default not shown
-at the end of a test run to avoid cluttering the output.  You can use
-the ``-r`` option to see details corresponding to the "short" letters
-shown in the test progress::
+py.test counts and lists *skip* and *xfail* tests separately. However,
+detailed information about skipped/xfailed tests is not shown by default
+to avoid cluttering the output.  You can use the ``-r`` option to see
+details corresponding to the "short" letters shown in the test
+progress::
 
     py.test -rxs  # show extra info on skips and xfails
 
@@ -22,7 +24,7 @@ shown in the test progress::
 
 .. _skipif:
 
-Skipping a single function
+Marking a test function to be skipped
 -------------------------------------------
 
 Here is an example of marking a test function to be skipped
@@ -34,9 +36,9 @@ when run on a Python3 interpreter::
         ...
 
 During test function setup the skipif condition is
-evaluated by calling ``eval(expr, namespace)``.  The namespace
-contains all the module globals of the test function so that
-you can for example check for versions::
+evaluated by calling ``eval('sys.version_info >= (3,0)', namespace)``.
+(*New in version 2.0.2*) The namespace contains all the module globals of the test function so that
+you can for example check for versions of a module you are using::
 
     import mymodule
 
@@ -44,23 +46,15 @@ you can for example check for versions::
     def test_function():
         ...
   
-The test function will be skipped and not run if
-mymodule is below the specified version.  The reason
+The test function will not be run ("skipped") if
+``mymodule`` is below the specified version.  The reason
 for specifying the condition as a string is mainly that
-you can see more detailed reporting of xfail/skip reasons.
+py.test can report a summary of skip conditions.
+For information on the construction of the ``namespace``
+see `evaluation of skipif/xfail conditions`_.
 
-Actually, the namespace is first initialized by
-putting the ``sys`` and ``os`` modules and the test
-``config`` object into it.  And is then updated with
-the module globals.  The latter allows you to skip based
-on a test configuration value::
-
-    @pytest.mark.skipif("not config.getvalue('db')")
-    def test_function(...):
-        ...
-
-You can create a shortcut for your conditional skip decorator
-at module level like this::
+You can of course create a shortcut for your conditional skip
+decorator at module level like this::
 
     win32only = pytest.mark.skipif("sys.platform != 'win32'")
 
@@ -68,11 +62,10 @@ at module level like this::
     def test_function():
         ...
 
-
 skip all test functions of a class
 --------------------------------------
 
-As with all function :ref:`mark` you can skip test functions at the
+As with all function :ref:`marking <mark>` you can skip test functions at the
 `whole class- or module level`_.  Here is an example
 for skipping all methods of a test class based on the platform::
 
@@ -82,9 +75,10 @@ for skipping all methods of a test class based on the platform::
         def test_function(self):
             "will not be setup or run under 'win32' platform"
 
-The ``pytestmark`` decorator will be applied to each test function.
-If your code targets python2.6 or above you can equivalently use
-the skipif decorator on classes::
+The ``pytestmark`` special name tells py.test to apply it to each test
+function in the class.  If your code targets python2.6 or above you can
+more naturally use the skipif decorator (and any other marker) on
+classes::
 
     @pytest.mark.skipif("sys.platform == 'win32'")
     class TestPosixCalls:
@@ -155,6 +149,31 @@ Running it with the report-on-xfail option gives this output::
     
     ======================== 6 xfailed in 0.06 seconds =========================
 
+.. _`evaluation of skipif/xfail conditions`:
+
+evaluation of skipif/xfail expressions
+----------------------------------------------------
+
+.. versionadded:: 2.0.2
+
+The evaluation of a condition string in ``pytest.mark.skipif(conditionstring)``
+or ``pytest.mark.xfail(conditionstring)`` takes place in a namespace
+dictionary which is constructed as follows:
+
+* the namespace is initialized by putting the ``sys`` and ``os`` modules
+  and the pytest ``config`` object into it.
+  
+* updated with the module globals of the test function for which the
+  expression is applied.
+
+The pytest ``config`` object allows you to skip based on a test configuration value
+which you might have added::
+
+    @pytest.mark.skipif("not config.getvalue('db')")
+    def test_function(...):
+        ...
+
+
 imperative xfail from within a test or setup function
 ------------------------------------------------------