test_ok1/doc/en/example/parametrize.txt

.. _paramexamples:

Parametrizing tests
=================================================

.. currentmodule:: _pytest.python

``pytest`` allows to easily parametrize test functions.
For basic docs, see :ref:`parametrize-basics`.

In the following we provide some examples using
the builtin mechanisms.

Generating parameters combinations, depending on command line
----------------------------------------------------------------------------

.. regendoc:wipe

Let's say we want to execute a test with different computation
parameters and the parameter range shall be determined by a command
line argument.  Let's first write a simple (do-nothing) computation test::

    # content of test_compute.py

    def test_compute(param1):
        assert param1 < 4

Now we add a test configuration like this::

    # content of conftest.py

    def pytest_addoption(parser):
        parser.addoption("--all", action="store_true",
            help="run all combinations")

    def pytest_generate_tests(metafunc):
        if 'param1' in metafunc.fixturenames:
            if metafunc.config.option.all:
                end = 5
            else:
                end = 2
            metafunc.parametrize("param1", range(end))

This means that we only run 2 tests if we do not pass ``--all``::

    $ py.test -q test_compute.py
    ..
    2 passed in 0.01 seconds

We run only two computations, so we see two dots.
let's run the full monty::

    $ py.test -q --all
    ....F
    ================================= FAILURES =================================
    _____________________________ test_compute[4] ______________________________

    param1 = 4

        def test_compute(param1):
    >       assert param1 < 4
    E       assert 4 < 4

    test_compute.py:3: AssertionError
    1 failed, 4 passed in 0.01 seconds

As expected when running the full range of ``param1`` values
we'll get an error on the last one.


Different options for test IDs
------------------------------------

pytest will build a string that is the test ID for each set of values in a
parametrized test. These IDs can be used with ``-k`` to select specific cases
to run, and they will also identify the specific case when one is failing.
Running pytest with ``--collect-only`` will show the generated IDs.

Numbers, strings, booleans and None will have their usual string representation
used in the test ID. For other objects, pytest will make a string based on
the argument name::

    # contents of test_time.py

    from datetime import datetime, timedelta

    testdata = [(datetime(2001, 12, 12), datetime(2001, 12, 11), timedelta(1)),
	       	(datetime(2001, 12, 11), datetime(2001, 12, 12), timedelta(-1)),
		]


    @pytest.mark.parametrize("a,b,expected", testdata)
    def test_timedistance_v0(a, b, expected):
	diff = a - b
	assert diff == expected


    @pytest.mark.parametrize("a,b,expected", testdata, ids=["forward", "backward"])
    def test_timedistance_v1(a, b, expected):
	diff = a - b
	assert diff == expected


    def idfn(val):
	if isinstance(val, (datetime,)):
	    # note this wouldn't show any hours/minutes/seconds
	    return val.strftime('%Y%m%d')


    @pytest.mark.parametrize("a,b,expected", testdata, ids=idfn)
    def test_timedistance_v2(a, b, expected):
	diff = a - b
	assert diff == expected


In ``test_timedistance_v0``, we let pytest generate the test IDs.

In ``test_timedistance_v1``, we specified ``ids`` as a list of strings which were
used as the test IDs. These are succinct, but can be a pain to maintain.

In ``test_timedistance_v2``, we specified ``ids`` as a function that can generate a
string representation to make part of the test ID. So our ``datetime`` values use the
label generated by ``idfn``, but because we didn't generate a label for ``timedelta``
objects, they are still using the default pytest representation::


    $ py.test test_time.py --collect-only
    ============================ test session starts =============================
    platform linux2 -- Python 2.7.3 -- py-1.4.20 -- pytest-2.6.0.dev1
    plugins: cache
    collected 6 items
    <Module 'test_time.py'>
      <Function 'test_timedistance_v0[a0-b0-expected0]'>
      <Function 'test_timedistance_v0[a1-b1-expected1]'>
      <Function 'test_timedistance_v1[forward]'>
      <Function 'test_timedistance_v1[backward]'>
      <Function 'test_timedistance_v2[20011212-20011211-expected0]'>
      <Function 'test_timedistance_v2[20011211-20011212-expected1]'>

    ============================== in 0.04 seconds ===============================




A quick port of "testscenarios"
------------------------------------

.. _`test scenarios`: http://pypi.python.org/pypi/testscenarios/

Here is a quick port to run tests configured with `test scenarios`_,
an add-on from Robert Collins for the standard unittest framework. We
only have to work a bit to construct the correct arguments for pytest's
:py:func:`Metafunc.parametrize`::

    # content of test_scenarios.py

    def pytest_generate_tests(metafunc):
        idlist = []
        argvalues = []
        for scenario in metafunc.cls.scenarios:
            idlist.append(scenario[0])
            items = scenario[1].items()
            argnames = [x[0] for x in items]
            argvalues.append(([x[1] for x in items]))
        metafunc.parametrize(argnames, argvalues, ids=idlist, scope="class")

    scenario1 = ('basic', {'attribute': 'value'})
    scenario2 = ('advanced', {'attribute': 'value2'})

    class TestSampleWithScenarios:
        scenarios = [scenario1, scenario2]

        def test_demo1(self, attribute):
            assert isinstance(attribute, str)

        def test_demo2(self, attribute):
            assert isinstance(attribute, str)

this is a fully self-contained example which you can run with::

    $ py.test test_scenarios.py
    =========================== test session starts ============================
    platform linux -- Python 3.4.0 -- py-1.4.26 -- pytest-2.6.4
    collected 4 items
    
    test_scenarios.py ....
    
    ========================= 4 passed in 0.01 seconds =========================

If you just collect tests you'll also nicely see 'advanced' and 'basic' as variants for the test function::


    $ py.test --collect-only test_scenarios.py
    =========================== test session starts ============================
    platform linux -- Python 3.4.0 -- py-1.4.26 -- pytest-2.6.4
    collected 4 items
    <Module 'test_scenarios.py'>
      <Class 'TestSampleWithScenarios'>
        <Instance '()'>
          <Function 'test_demo1[basic]'>
          <Function 'test_demo2[basic]'>
          <Function 'test_demo1[advanced]'>
          <Function 'test_demo2[advanced]'>
    
    =============================  in 0.01 seconds =============================

Note that we told ``metafunc.parametrize()`` that your scenario values
should be considered class-scoped.  With pytest-2.3 this leads to a
resource-based ordering.

Deferring the setup of parametrized resources
---------------------------------------------------

.. regendoc:wipe

The parametrization of test functions happens at collection
time.  It is a good idea to setup expensive resources like DB
connections or subprocess only when the actual test is run.
Here is a simple example how you can achieve that, first
the actual test requiring a ``db`` object::

    # content of test_backends.py

    import pytest
    def test_db_initialized(db):
        # a dummy test
        if db.__class__.__name__ == "DB2":
            pytest.fail("deliberately failing for demo purposes")

We can now add a test configuration that generates two invocations of
the ``test_db_initialized`` function and also implements a factory that
creates a database object for the actual test invocations::

    # content of conftest.py
    import pytest

    def pytest_generate_tests(metafunc):
        if 'db' in metafunc.fixturenames:
            metafunc.parametrize("db", ['d1', 'd2'], indirect=True)

    class DB1:
        "one database object"
    class DB2:
        "alternative database object"

    @pytest.fixture
    def db(request):
        if request.param == "d1":
            return DB1()
        elif request.param == "d2":
            return DB2()
        else:
            raise ValueError("invalid internal test config")

Let's first see how it looks like at collection time::

    $ py.test test_backends.py --collect-only
    =========================== test session starts ============================
    platform linux -- Python 3.4.0 -- py-1.4.26 -- pytest-2.6.4
    collected 2 items
    <Module 'test_backends.py'>
      <Function 'test_db_initialized[d1]'>
      <Function 'test_db_initialized[d2]'>
    
    =============================  in 0.00 seconds =============================

And then when we run the test::

    $ py.test -q test_backends.py
    .F
    ================================= FAILURES =================================
    _________________________ test_db_initialized[d2] __________________________
    
    db = <conftest.DB2 object at 0x2b04d7936be0>
    
        def test_db_initialized(db):
            # a dummy test
            if db.__class__.__name__ == "DB2":
    >           pytest.fail("deliberately failing for demo purposes")
    E           Failed: deliberately failing for demo purposes
    
    test_backends.py:6: Failed
    1 failed, 1 passed in 0.01 seconds

The first invocation with ``db == "DB1"`` passed while the second with ``db == "DB2"`` failed.  Our ``db`` fixture function has instantiated each of the DB values during the setup phase while the ``pytest_generate_tests`` generated two according calls to the ``test_db_initialized`` during the collection phase.

.. regendoc:wipe

Parametrizing test methods through per-class configuration
--------------------------------------------------------------

.. _`unittest parameterizer`: http://code.google.com/p/unittest-ext/source/browse/trunk/params.py


Here is an example ``pytest_generate_function`` function implementing a
parametrization scheme similar to Michael Foord's `unittest
parameterizer`_ but in a lot less code::

    # content of ./test_parametrize.py
    import pytest

    def pytest_generate_tests(metafunc):
        # called once per each test function
        funcarglist = metafunc.cls.params[metafunc.function.__name__]
        argnames = list(funcarglist[0])
        metafunc.parametrize(argnames, [[funcargs[name] for name in argnames]
                for funcargs in funcarglist])

    class TestClass:
        # a map specifying multiple argument sets for a test method
        params = {
            'test_equals': [dict(a=1, b=2), dict(a=3, b=3), ],
            'test_zerodivision': [dict(a=1, b=0), ],
        }

        def test_equals(self, a, b):
            assert a == b

        def test_zerodivision(self, a, b):
            pytest.raises(ZeroDivisionError, "a/b")

Our test generator looks up a class-level definition which specifies which
argument sets to use for each test function.  Let's run it::

    $ py.test -q
    F..
    ================================= FAILURES =================================
    ________________________ TestClass.test_equals[2-1] ________________________
    
    self = <test_parametrize.TestClass object at 0x2af4cdee0da0>, a = 1, b = 2
    
        def test_equals(self, a, b):
    >       assert a == b
    E       assert 1 == 2
    
    test_parametrize.py:18: AssertionError
    1 failed, 2 passed in 0.01 seconds

Indirect parametrization with multiple fixtures
--------------------------------------------------------------

Here is a stripped down real-life example of using parametrized
testing for testing serialization of objects between different python
interpreters.  We define a ``test_basic_objects`` function which
is to be run with different sets of arguments for its three arguments:

* ``python1``: first python interpreter, run to pickle-dump an object to a file
* ``python2``: second interpreter, run to pickle-load an object from a file
* ``obj``: object to be dumped/loaded

.. literalinclude:: multipython.py

Running it results in some skips if we don't have all the python interpreters installed and otherwise runs all combinations (5 interpreters times 5 interpreters times 3 objects to serialize/deserialize)::

   . $ py.test -rs -q multipython.py
   ..................FFFFFF...
   ================================= FAILURES =================================
   ________________ test_basic_objects[python3.4-python2.6-42] ________________
   
   python1 = <multipython.Python object at 0x2afc7d8c2828>
   python2 = <multipython.Python object at 0x2afc7d8c2588>, obj = 42
   
       @pytest.mark.parametrize("obj", [42, {}, {1:3},])
       def test_basic_objects(python1, python2, obj):
           python1.dumps(obj)
   >       python2.load_and_is_true("obj == %s" % obj)
   
   multipython.py:51: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   multipython.py:46: in load_and_is_true
       py.process.cmdexec("%s %s" %(self.pythonpath, loadfile))
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   
   cmd = '/home/hpk/bin/python2.6 /tmp/pytest-111/test_basic_objects_python3_4_p0/load.py'
   
       def cmdexec(cmd):
           """ return unicode output of executing 'cmd' in a separate process.
       
           raise cmdexec.Error exeception if the command failed.
           the exception will provide an 'err' attribute containing
           the error-output from the command.
           if the subprocess module does not provide a proper encoding/unicode strings
           sys.getdefaultencoding() will be used, if that does not exist, 'UTF-8'.
           """
           process = subprocess.Popen(cmd, shell=True,
                   universal_newlines=True,
                   stdout=subprocess.PIPE, stderr=subprocess.PIPE)
           out, err = process.communicate()
           if sys.version_info[0] < 3: # on py3 we get unicode strings, on py2 not
               try:
                   default_encoding = sys.getdefaultencoding() # jython may not have it
               except AttributeError:
                   default_encoding = sys.stdout.encoding or 'UTF-8'
               out = unicode(out, process.stdout.encoding or default_encoding)
               err = unicode(err, process.stderr.encoding or default_encoding)
           status = process.poll()
           if status:
   >           raise ExecutionFailed(status, status, cmd, out, err)
   E           py.process.cmdexec.Error: ExecutionFailed: 1  /home/hpk/bin/python2.6 /tmp/pytest-111/test_basic_objects_python3_4_p0/load.py
   E           Traceback (most recent call last):
   E             File "/tmp/pytest-111/test_basic_objects_python3_4_p0/load.py", line 4, in <module>
   E               obj = pickle.load(f)
   E             File "/usr/lib/python2.6/pickle.py", line 1370, in load
   E               return Unpickler(file).load()
   E             File "/usr/lib/python2.6/pickle.py", line 858, in load
   E               dispatch[key](self)
   E             File "/usr/lib/python2.6/pickle.py", line 886, in load_proto
   E               raise ValueError, "unsupported pickle protocol: %d" % proto
   E           ValueError: unsupported pickle protocol: 3
   
   ../../../.tox/regen/lib/python3.4/site-packages/py/_process/cmdexec.py:28: Error
   --------------------------- Captured stdout call ---------------------------
   /tmp/pytest-111/test_basic_objects_python3_4_p0/load.py
   _______________ test_basic_objects[python3.4-python2.6-obj1] _______________
   
   python1 = <multipython.Python object at 0x2afc7d8b8c88>
   python2 = <multipython.Python object at 0x2afc7d8b8e48>, obj = {}
   
       @pytest.mark.parametrize("obj", [42, {}, {1:3},])
       def test_basic_objects(python1, python2, obj):
           python1.dumps(obj)
   >       python2.load_and_is_true("obj == %s" % obj)
   
   multipython.py:51: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   multipython.py:46: in load_and_is_true
       py.process.cmdexec("%s %s" %(self.pythonpath, loadfile))
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   
   cmd = '/home/hpk/bin/python2.6 /tmp/pytest-111/test_basic_objects_python3_4_p1/load.py'
   
       def cmdexec(cmd):
           """ return unicode output of executing 'cmd' in a separate process.
       
           raise cmdexec.Error exeception if the command failed.
           the exception will provide an 'err' attribute containing
           the error-output from the command.
           if the subprocess module does not provide a proper encoding/unicode strings
           sys.getdefaultencoding() will be used, if that does not exist, 'UTF-8'.
           """
           process = subprocess.Popen(cmd, shell=True,
                   universal_newlines=True,
                   stdout=subprocess.PIPE, stderr=subprocess.PIPE)
           out, err = process.communicate()
           if sys.version_info[0] < 3: # on py3 we get unicode strings, on py2 not
               try:
                   default_encoding = sys.getdefaultencoding() # jython may not have it
               except AttributeError:
                   default_encoding = sys.stdout.encoding or 'UTF-8'
               out = unicode(out, process.stdout.encoding or default_encoding)
               err = unicode(err, process.stderr.encoding or default_encoding)
           status = process.poll()
           if status:
   >           raise ExecutionFailed(status, status, cmd, out, err)
   E           py.process.cmdexec.Error: ExecutionFailed: 1  /home/hpk/bin/python2.6 /tmp/pytest-111/test_basic_objects_python3_4_p1/load.py
   E           Traceback (most recent call last):
   E             File "/tmp/pytest-111/test_basic_objects_python3_4_p1/load.py", line 4, in <module>
   E               obj = pickle.load(f)
   E             File "/usr/lib/python2.6/pickle.py", line 1370, in load
   E               return Unpickler(file).load()
   E             File "/usr/lib/python2.6/pickle.py", line 858, in load
   E               dispatch[key](self)
   E             File "/usr/lib/python2.6/pickle.py", line 886, in load_proto
   E               raise ValueError, "unsupported pickle protocol: %d" % proto
   E           ValueError: unsupported pickle protocol: 3
   
   ../../../.tox/regen/lib/python3.4/site-packages/py/_process/cmdexec.py:28: Error
   --------------------------- Captured stdout call ---------------------------
   /tmp/pytest-111/test_basic_objects_python3_4_p1/load.py
   _______________ test_basic_objects[python3.4-python2.6-obj2] _______________
   
   python1 = <multipython.Python object at 0x2afc7d8bf6d8>
   python2 = <multipython.Python object at 0x2afc7d8bf860>, obj = {1: 3}
   
       @pytest.mark.parametrize("obj", [42, {}, {1:3},])
       def test_basic_objects(python1, python2, obj):
           python1.dumps(obj)
   >       python2.load_and_is_true("obj == %s" % obj)
   
   multipython.py:51: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   multipython.py:46: in load_and_is_true
       py.process.cmdexec("%s %s" %(self.pythonpath, loadfile))
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   
   cmd = '/home/hpk/bin/python2.6 /tmp/pytest-111/test_basic_objects_python3_4_p2/load.py'
   
       def cmdexec(cmd):
           """ return unicode output of executing 'cmd' in a separate process.
       
           raise cmdexec.Error exeception if the command failed.
           the exception will provide an 'err' attribute containing
           the error-output from the command.
           if the subprocess module does not provide a proper encoding/unicode strings
           sys.getdefaultencoding() will be used, if that does not exist, 'UTF-8'.
           """
           process = subprocess.Popen(cmd, shell=True,
                   universal_newlines=True,
                   stdout=subprocess.PIPE, stderr=subprocess.PIPE)
           out, err = process.communicate()
           if sys.version_info[0] < 3: # on py3 we get unicode strings, on py2 not
               try:
                   default_encoding = sys.getdefaultencoding() # jython may not have it
               except AttributeError:
                   default_encoding = sys.stdout.encoding or 'UTF-8'
               out = unicode(out, process.stdout.encoding or default_encoding)
               err = unicode(err, process.stderr.encoding or default_encoding)
           status = process.poll()
           if status:
   >           raise ExecutionFailed(status, status, cmd, out, err)
   E           py.process.cmdexec.Error: ExecutionFailed: 1  /home/hpk/bin/python2.6 /tmp/pytest-111/test_basic_objects_python3_4_p2/load.py
   E           Traceback (most recent call last):
   E             File "/tmp/pytest-111/test_basic_objects_python3_4_p2/load.py", line 4, in <module>
   E               obj = pickle.load(f)
   E             File "/usr/lib/python2.6/pickle.py", line 1370, in load
   E               return Unpickler(file).load()
   E             File "/usr/lib/python2.6/pickle.py", line 858, in load
   E               dispatch[key](self)
   E             File "/usr/lib/python2.6/pickle.py", line 886, in load_proto
   E               raise ValueError, "unsupported pickle protocol: %d" % proto
   E           ValueError: unsupported pickle protocol: 3
   
   ../../../.tox/regen/lib/python3.4/site-packages/py/_process/cmdexec.py:28: Error
   --------------------------- Captured stdout call ---------------------------
   /tmp/pytest-111/test_basic_objects_python3_4_p2/load.py
   ________________ test_basic_objects[python3.4-python2.7-42] ________________
   
   python1 = <multipython.Python object at 0x2afc7d8b8710>
   python2 = <multipython.Python object at 0x2afc7d8b8748>, obj = 42
   
       @pytest.mark.parametrize("obj", [42, {}, {1:3},])
       def test_basic_objects(python1, python2, obj):
           python1.dumps(obj)
   >       python2.load_and_is_true("obj == %s" % obj)
   
   multipython.py:51: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   multipython.py:46: in load_and_is_true
       py.process.cmdexec("%s %s" %(self.pythonpath, loadfile))
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   
   cmd = '/home/hpk/venv/0/bin/python2.7 /tmp/pytest-111/test_basic_objects_python3_4_p3/load.py'
   
       def cmdexec(cmd):
           """ return unicode output of executing 'cmd' in a separate process.
       
           raise cmdexec.Error exeception if the command failed.
           the exception will provide an 'err' attribute containing
           the error-output from the command.
           if the subprocess module does not provide a proper encoding/unicode strings
           sys.getdefaultencoding() will be used, if that does not exist, 'UTF-8'.
           """
           process = subprocess.Popen(cmd, shell=True,
                   universal_newlines=True,
                   stdout=subprocess.PIPE, stderr=subprocess.PIPE)
           out, err = process.communicate()
           if sys.version_info[0] < 3: # on py3 we get unicode strings, on py2 not
               try:
                   default_encoding = sys.getdefaultencoding() # jython may not have it
               except AttributeError:
                   default_encoding = sys.stdout.encoding or 'UTF-8'
               out = unicode(out, process.stdout.encoding or default_encoding)
               err = unicode(err, process.stderr.encoding or default_encoding)
           status = process.poll()
           if status:
   >           raise ExecutionFailed(status, status, cmd, out, err)
   E           py.process.cmdexec.Error: ExecutionFailed: 1  /home/hpk/venv/0/bin/python2.7 /tmp/pytest-111/test_basic_objects_python3_4_p3/load.py
   E           Traceback (most recent call last):
   E             File "/tmp/pytest-111/test_basic_objects_python3_4_p3/load.py", line 4, in <module>
   E               obj = pickle.load(f)
   E             File "/usr/lib/python2.7/pickle.py", line 1378, in load
   E               return Unpickler(file).load()
   E             File "/usr/lib/python2.7/pickle.py", line 858, in load
   E               dispatch[key](self)
   E             File "/usr/lib/python2.7/pickle.py", line 886, in load_proto
   E               raise ValueError, "unsupported pickle protocol: %d" % proto
   E           ValueError: unsupported pickle protocol: 3
   
   ../../../.tox/regen/lib/python3.4/site-packages/py/_process/cmdexec.py:28: Error
   --------------------------- Captured stdout call ---------------------------
   /tmp/pytest-111/test_basic_objects_python3_4_p3/load.py
   _______________ test_basic_objects[python3.4-python2.7-obj1] _______________
   
   python1 = <multipython.Python object at 0x2afc7d8bfb38>
   python2 = <multipython.Python object at 0x2afc7d8bf3c8>, obj = {}
   
       @pytest.mark.parametrize("obj", [42, {}, {1:3},])
       def test_basic_objects(python1, python2, obj):
           python1.dumps(obj)
   >       python2.load_and_is_true("obj == %s" % obj)
   
   multipython.py:51: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   multipython.py:46: in load_and_is_true
       py.process.cmdexec("%s %s" %(self.pythonpath, loadfile))
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   
   cmd = '/home/hpk/venv/0/bin/python2.7 /tmp/pytest-111/test_basic_objects_python3_4_p4/load.py'
   
       def cmdexec(cmd):
           """ return unicode output of executing 'cmd' in a separate process.
       
           raise cmdexec.Error exeception if the command failed.
           the exception will provide an 'err' attribute containing
           the error-output from the command.
           if the subprocess module does not provide a proper encoding/unicode strings
           sys.getdefaultencoding() will be used, if that does not exist, 'UTF-8'.
           """
           process = subprocess.Popen(cmd, shell=True,
                   universal_newlines=True,
                   stdout=subprocess.PIPE, stderr=subprocess.PIPE)
           out, err = process.communicate()
           if sys.version_info[0] < 3: # on py3 we get unicode strings, on py2 not
               try:
                   default_encoding = sys.getdefaultencoding() # jython may not have it
               except AttributeError:
                   default_encoding = sys.stdout.encoding or 'UTF-8'
               out = unicode(out, process.stdout.encoding or default_encoding)
               err = unicode(err, process.stderr.encoding or default_encoding)
           status = process.poll()
           if status:
   >           raise ExecutionFailed(status, status, cmd, out, err)
   E           py.process.cmdexec.Error: ExecutionFailed: 1  /home/hpk/venv/0/bin/python2.7 /tmp/pytest-111/test_basic_objects_python3_4_p4/load.py
   E           Traceback (most recent call last):
   E             File "/tmp/pytest-111/test_basic_objects_python3_4_p4/load.py", line 4, in <module>
   E               obj = pickle.load(f)
   E             File "/usr/lib/python2.7/pickle.py", line 1378, in load
   E               return Unpickler(file).load()
   E             File "/usr/lib/python2.7/pickle.py", line 858, in load
   E               dispatch[key](self)
   E             File "/usr/lib/python2.7/pickle.py", line 886, in load_proto
   E               raise ValueError, "unsupported pickle protocol: %d" % proto
   E           ValueError: unsupported pickle protocol: 3
   
   ../../../.tox/regen/lib/python3.4/site-packages/py/_process/cmdexec.py:28: Error
   --------------------------- Captured stdout call ---------------------------
   /tmp/pytest-111/test_basic_objects_python3_4_p4/load.py
   _______________ test_basic_objects[python3.4-python2.7-obj2] _______________
   
   python1 = <multipython.Python object at 0x2afc7d8b86a0>
   python2 = <multipython.Python object at 0x2afc7d8c2a90>, obj = {1: 3}
   
       @pytest.mark.parametrize("obj", [42, {}, {1:3},])
       def test_basic_objects(python1, python2, obj):
           python1.dumps(obj)
   >       python2.load_and_is_true("obj == %s" % obj)
   
   multipython.py:51: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   multipython.py:46: in load_and_is_true
       py.process.cmdexec("%s %s" %(self.pythonpath, loadfile))
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   
   cmd = '/home/hpk/venv/0/bin/python2.7 /tmp/pytest-111/test_basic_objects_python3_4_p5/load.py'
   
       def cmdexec(cmd):
           """ return unicode output of executing 'cmd' in a separate process.
       
           raise cmdexec.Error exeception if the command failed.
           the exception will provide an 'err' attribute containing
           the error-output from the command.
           if the subprocess module does not provide a proper encoding/unicode strings
           sys.getdefaultencoding() will be used, if that does not exist, 'UTF-8'.
           """
           process = subprocess.Popen(cmd, shell=True,
                   universal_newlines=True,
                   stdout=subprocess.PIPE, stderr=subprocess.PIPE)
           out, err = process.communicate()
           if sys.version_info[0] < 3: # on py3 we get unicode strings, on py2 not
               try:
                   default_encoding = sys.getdefaultencoding() # jython may not have it
               except AttributeError:
                   default_encoding = sys.stdout.encoding or 'UTF-8'
               out = unicode(out, process.stdout.encoding or default_encoding)
               err = unicode(err, process.stderr.encoding or default_encoding)
           status = process.poll()
           if status:
   >           raise ExecutionFailed(status, status, cmd, out, err)
   E           py.process.cmdexec.Error: ExecutionFailed: 1  /home/hpk/venv/0/bin/python2.7 /tmp/pytest-111/test_basic_objects_python3_4_p5/load.py
   E           Traceback (most recent call last):
   E             File "/tmp/pytest-111/test_basic_objects_python3_4_p5/load.py", line 4, in <module>
   E               obj = pickle.load(f)
   E             File "/usr/lib/python2.7/pickle.py", line 1378, in load
   E               return Unpickler(file).load()
   E             File "/usr/lib/python2.7/pickle.py", line 858, in load
   E               dispatch[key](self)
   E             File "/usr/lib/python2.7/pickle.py", line 886, in load_proto
   E               raise ValueError, "unsupported pickle protocol: %d" % proto
   E           ValueError: unsupported pickle protocol: 3
   
   ../../../.tox/regen/lib/python3.4/site-packages/py/_process/cmdexec.py:28: Error
   --------------------------- Captured stdout call ---------------------------
   /tmp/pytest-111/test_basic_objects_python3_4_p5/load.py
   6 failed, 21 passed in 1.66 seconds

Indirect parametrization of optional implementations/imports
--------------------------------------------------------------------

If you want to compare the outcomes of several implementations of a given
API, you can write test functions that receive the already imported implementations
and get skipped in case the implementation is not importable/available.  Let's
say we have a "base" implementation and the other (possibly optimized ones)
need to provide similar results::

    # content of conftest.py

    import pytest

    @pytest.fixture(scope="session")
    def basemod(request):
        return pytest.importorskip("base")

    @pytest.fixture(scope="session", params=["opt1", "opt2"])
    def optmod(request):
        return pytest.importorskip(request.param)

And then a base implementation of a simple function::

    # content of base.py
    def func1():
        return 1

And an optimized version::

    # content of opt1.py
    def func1():
        return 1.0001

And finally a little test module::

    # content of test_module.py

    def test_func1(basemod, optmod):
        assert round(basemod.func1(), 3) == round(optmod.func1(), 3)


If you run this with reporting for skips enabled::

    $ py.test -rs test_module.py
    =========================== test session starts ============================
    platform linux -- Python 3.4.0 -- py-1.4.26 -- pytest-2.6.4
    collected 2 items
    
    test_module.py .s
    ========================= short test summary info ==========================
    SKIP [1] /tmp/doc-exec-70/conftest.py:10: could not import 'opt2'
    
    =================== 1 passed, 1 skipped in 0.01 seconds ====================

You'll see that we don't have a ``opt2`` module and thus the second test run
of our ``test_func1`` was skipped.  A few notes:

- the fixture functions in the ``conftest.py`` file are "session-scoped" because we
  don't need to import more than once

- if you have multiple test functions and a skipped import, you will see
  the ``[1]`` count increasing in the report

- you can put :ref:`@pytest.mark.parametrize <@pytest.mark.parametrize>` style
  parametrization on the test functions to parametrize input/output
  values as well.