2010-11-21 04:35:55 +08:00
|
|
|
|
2011-02-09 21:55:21 +08:00
|
|
|
.. _paramexamples:
|
|
|
|
|
2010-11-21 04:35:55 +08:00
|
|
|
parametrizing tests
|
|
|
|
=================================================
|
|
|
|
|
|
|
|
py.test allows to easily implement your own custom
|
|
|
|
parametrization scheme for tests. Here we provide
|
|
|
|
some examples for inspiration and re-use.
|
|
|
|
|
2011-02-09 21:55:21 +08:00
|
|
|
generating parameters combinations, depending on command line
|
|
|
|
----------------------------------------------------------------------------
|
|
|
|
|
|
|
|
.. regendoc:wipe
|
|
|
|
|
|
|
|
Let's say we want to execute a test with different parameters
|
|
|
|
and the parameter range shall be determined by a command
|
|
|
|
line argument. Let's first write a simple computation test::
|
|
|
|
|
|
|
|
# content of test_compute.py
|
|
|
|
|
|
|
|
def test_compute(param1):
|
|
|
|
assert param1 < 4
|
|
|
|
|
|
|
|
Now we add a test configuration like this::
|
|
|
|
|
|
|
|
# content of conftest.py
|
|
|
|
|
|
|
|
def pytest_addoption(parser):
|
|
|
|
parser.addoption("--all", action="store_true",
|
|
|
|
help="run all combinations")
|
|
|
|
|
|
|
|
def pytest_generate_tests(metafunc):
|
|
|
|
if 'param1' in metafunc.funcargnames:
|
|
|
|
if metafunc.config.option.all:
|
|
|
|
end = 5
|
|
|
|
else:
|
|
|
|
end = 2
|
|
|
|
for i in range(end):
|
|
|
|
metafunc.addcall(funcargs={'param1': i})
|
|
|
|
|
|
|
|
This means that we only run 2 tests if we do not pass ``--all``::
|
|
|
|
|
|
|
|
$ py.test -q test_compute.py
|
|
|
|
collecting ... collected 2 items
|
|
|
|
..
|
|
|
|
2 passed in 0.01 seconds
|
|
|
|
|
|
|
|
We run only two computations, so we see two dots.
|
|
|
|
let's run the full monty::
|
|
|
|
|
|
|
|
$ py.test -q --all
|
|
|
|
collecting ... collected 5 items
|
|
|
|
....F
|
|
|
|
================================= FAILURES =================================
|
|
|
|
_____________________________ test_compute[4] ______________________________
|
|
|
|
|
|
|
|
param1 = 4
|
|
|
|
|
|
|
|
def test_compute(param1):
|
|
|
|
> assert param1 < 4
|
|
|
|
E assert 4 < 4
|
|
|
|
|
|
|
|
test_compute.py:3: AssertionError
|
2011-05-01 18:38:56 +08:00
|
|
|
1 failed, 4 passed in 0.03 seconds
|
2011-02-09 21:55:21 +08:00
|
|
|
|
|
|
|
As expected when running the full range of ``param1`` values
|
|
|
|
we'll get an error on the last one.
|
|
|
|
|
2011-02-17 21:46:40 +08:00
|
|
|
Deferring the setup of parametrizing resources
|
2011-02-09 21:55:21 +08:00
|
|
|
---------------------------------------------------
|
|
|
|
|
|
|
|
.. regendoc:wipe
|
|
|
|
|
|
|
|
The parametrization of test functions happens at collection
|
|
|
|
time. It is often a good idea to setup possibly expensive
|
|
|
|
resources only when the actual test is run. Here is a simple
|
|
|
|
example how you can achieve that::
|
|
|
|
|
|
|
|
# content of test_backends.py
|
|
|
|
|
|
|
|
import pytest
|
|
|
|
def test_db_initialized(db):
|
|
|
|
# a dummy test
|
|
|
|
if db.__class__.__name__ == "DB2":
|
|
|
|
pytest.fail("deliberately failing for demo purposes")
|
|
|
|
|
|
|
|
Now we add a test configuration that takes care to generate
|
|
|
|
two invocations of the ``test_db_initialized`` function and
|
|
|
|
furthermore a factory that creates a database object when
|
|
|
|
each test is actually run::
|
|
|
|
|
|
|
|
# content of conftest.py
|
|
|
|
|
|
|
|
def pytest_generate_tests(metafunc):
|
|
|
|
if 'db' in metafunc.funcargnames:
|
|
|
|
metafunc.addcall(param="d1")
|
|
|
|
metafunc.addcall(param="d2")
|
|
|
|
|
|
|
|
class DB1:
|
|
|
|
"one database object"
|
|
|
|
class DB2:
|
|
|
|
"alternative database object"
|
|
|
|
|
|
|
|
def pytest_funcarg__db(request):
|
|
|
|
if request.param == "d1":
|
|
|
|
return DB1()
|
|
|
|
elif request.param == "d2":
|
|
|
|
return DB2()
|
|
|
|
else:
|
|
|
|
raise ValueError("invalid internal test config")
|
|
|
|
|
|
|
|
Let's first see how it looks like at collection time::
|
|
|
|
|
|
|
|
$ py.test test_backends.py --collectonly
|
2011-03-09 17:58:36 +08:00
|
|
|
=========================== test session starts ============================
|
2011-05-01 18:38:56 +08:00
|
|
|
platform linux2 -- Python 2.6.6 -- pytest-2.0.3
|
2011-03-09 17:58:36 +08:00
|
|
|
collecting ... collected 2 items
|
2011-02-09 21:55:21 +08:00
|
|
|
<Module 'test_backends.py'>
|
|
|
|
<Function 'test_db_initialized[0]'>
|
|
|
|
<Function 'test_db_initialized[1]'>
|
2011-03-09 17:58:36 +08:00
|
|
|
|
2011-05-01 18:38:56 +08:00
|
|
|
============================= in 0.01 seconds =============================
|
2011-02-09 21:55:21 +08:00
|
|
|
|
|
|
|
And then when we run the test::
|
|
|
|
|
|
|
|
$ py.test -q test_backends.py
|
|
|
|
collecting ... collected 2 items
|
|
|
|
.F
|
|
|
|
================================= FAILURES =================================
|
|
|
|
__________________________ test_db_initialized[1] __________________________
|
|
|
|
|
2011-05-01 18:38:56 +08:00
|
|
|
db = <conftest.DB2 instance at 0x2bf7bd8>
|
2011-02-09 21:55:21 +08:00
|
|
|
|
|
|
|
def test_db_initialized(db):
|
|
|
|
# a dummy test
|
|
|
|
if db.__class__.__name__ == "DB2":
|
|
|
|
> pytest.fail("deliberately failing for demo purposes")
|
|
|
|
E Failed: deliberately failing for demo purposes
|
|
|
|
|
|
|
|
test_backends.py:6: Failed
|
2011-05-01 18:38:56 +08:00
|
|
|
1 failed, 1 passed in 0.03 seconds
|
2011-02-09 21:55:21 +08:00
|
|
|
|
|
|
|
Now you see that one invocation of the test passes and another fails,
|
|
|
|
as it to be expected.
|
|
|
|
|
2010-11-21 04:35:55 +08:00
|
|
|
Parametrizing test methods through per-class configuration
|
|
|
|
--------------------------------------------------------------
|
|
|
|
|
|
|
|
.. _`unittest parameterizer`: http://code.google.com/p/unittest-ext/source/browse/trunk/params.py
|
|
|
|
|
|
|
|
Here is an example ``pytest_generate_function`` function implementing a
|
|
|
|
parametrization scheme similar to Michael Foords `unittest
|
|
|
|
parameterizer`_ in a lot less code::
|
|
|
|
|
|
|
|
# content of ./test_parametrize.py
|
|
|
|
import pytest
|
|
|
|
|
|
|
|
def pytest_generate_tests(metafunc):
|
|
|
|
# called once per each test function
|
|
|
|
for funcargs in metafunc.cls.params[metafunc.function.__name__]:
|
|
|
|
# schedule a new test function run with applied **funcargs
|
|
|
|
metafunc.addcall(funcargs=funcargs)
|
|
|
|
|
|
|
|
class TestClass:
|
|
|
|
# a map specifying multiple argument sets for a test method
|
|
|
|
params = {
|
|
|
|
'test_equals': [dict(a=1, b=2), dict(a=3, b=3), ],
|
|
|
|
'test_zerodivision': [dict(a=1, b=0), dict(a=3, b=2)],
|
|
|
|
}
|
|
|
|
|
|
|
|
def test_equals(self, a, b):
|
|
|
|
assert a == b
|
|
|
|
|
|
|
|
def test_zerodivision(self, a, b):
|
|
|
|
pytest.raises(ZeroDivisionError, "a/b")
|
|
|
|
|
|
|
|
Running it means we are two tests for each test functions, using
|
|
|
|
the respective settings::
|
|
|
|
|
|
|
|
$ py.test -q
|
2011-02-09 21:55:21 +08:00
|
|
|
collecting ... collected 6 items
|
|
|
|
.FF..F
|
2010-11-21 04:35:55 +08:00
|
|
|
================================= FAILURES =================================
|
2011-02-09 21:55:21 +08:00
|
|
|
__________________________ test_db_initialized[1] __________________________
|
|
|
|
|
2011-05-01 18:38:56 +08:00
|
|
|
db = <conftest.DB2 instance at 0x19bcb90>
|
2011-02-09 21:55:21 +08:00
|
|
|
|
|
|
|
def test_db_initialized(db):
|
|
|
|
# a dummy test
|
|
|
|
if db.__class__.__name__ == "DB2":
|
|
|
|
> pytest.fail("deliberately failing for demo purposes")
|
|
|
|
E Failed: deliberately failing for demo purposes
|
|
|
|
|
|
|
|
test_backends.py:6: Failed
|
2010-11-21 04:35:55 +08:00
|
|
|
_________________________ TestClass.test_equals[0] _________________________
|
|
|
|
|
2011-05-01 18:38:56 +08:00
|
|
|
self = <test_parametrize.TestClass instance at 0x19ca8c0>, a = 1, b = 2
|
2010-11-21 04:35:55 +08:00
|
|
|
|
|
|
|
def test_equals(self, a, b):
|
|
|
|
> assert a == b
|
|
|
|
E assert 1 == 2
|
|
|
|
|
|
|
|
test_parametrize.py:17: AssertionError
|
|
|
|
______________________ TestClass.test_zerodivision[1] ______________________
|
|
|
|
|
2011-05-01 18:38:56 +08:00
|
|
|
self = <test_parametrize.TestClass instance at 0x19cd4d0>, a = 3, b = 2
|
2010-11-21 04:35:55 +08:00
|
|
|
|
|
|
|
def test_zerodivision(self, a, b):
|
|
|
|
> pytest.raises(ZeroDivisionError, "a/b")
|
|
|
|
E Failed: DID NOT RAISE
|
|
|
|
|
|
|
|
test_parametrize.py:20: Failed
|
2011-05-01 18:38:56 +08:00
|
|
|
3 failed, 3 passed in 0.05 seconds
|
2010-11-21 04:35:55 +08:00
|
|
|
|
|
|
|
Parametrizing test methods through a decorator
|
|
|
|
--------------------------------------------------------------
|
|
|
|
|
|
|
|
Modifying the previous example we can also allow decorators
|
|
|
|
for parametrizing test methods::
|
|
|
|
|
|
|
|
# content of test_parametrize2.py
|
|
|
|
|
|
|
|
import pytest
|
|
|
|
|
|
|
|
# test support code
|
|
|
|
def params(funcarglist):
|
|
|
|
def wrapper(function):
|
|
|
|
function.funcarglist = funcarglist
|
|
|
|
return function
|
|
|
|
return wrapper
|
|
|
|
|
|
|
|
def pytest_generate_tests(metafunc):
|
|
|
|
for funcargs in getattr(metafunc.function, 'funcarglist', ()):
|
|
|
|
metafunc.addcall(funcargs=funcargs)
|
|
|
|
|
|
|
|
# actual test code
|
|
|
|
class TestClass:
|
|
|
|
@params([dict(a=1, b=2), dict(a=3, b=3), ])
|
|
|
|
def test_equals(self, a, b):
|
|
|
|
assert a == b
|
|
|
|
|
|
|
|
@params([dict(a=1, b=0), dict(a=3, b=2)])
|
|
|
|
def test_zerodivision(self, a, b):
|
|
|
|
pytest.raises(ZeroDivisionError, "a/b")
|
|
|
|
|
|
|
|
Running it gives similar results as before::
|
|
|
|
|
|
|
|
$ py.test -q test_parametrize2.py
|
2010-11-26 20:26:56 +08:00
|
|
|
collecting ... collected 4 items
|
2010-11-21 04:35:55 +08:00
|
|
|
F..F
|
|
|
|
================================= FAILURES =================================
|
|
|
|
_________________________ TestClass.test_equals[0] _________________________
|
|
|
|
|
2011-05-01 18:38:56 +08:00
|
|
|
self = <test_parametrize2.TestClass instance at 0x1cf1170>, a = 1, b = 2
|
2010-11-21 04:35:55 +08:00
|
|
|
|
|
|
|
@params([dict(a=1, b=2), dict(a=3, b=3), ])
|
|
|
|
def test_equals(self, a, b):
|
|
|
|
> assert a == b
|
|
|
|
E assert 1 == 2
|
|
|
|
|
|
|
|
test_parametrize2.py:19: AssertionError
|
|
|
|
______________________ TestClass.test_zerodivision[1] ______________________
|
|
|
|
|
2011-05-01 18:38:56 +08:00
|
|
|
self = <test_parametrize2.TestClass instance at 0x1d02170>, a = 3, b = 2
|
2010-11-21 04:35:55 +08:00
|
|
|
|
|
|
|
@params([dict(a=1, b=0), dict(a=3, b=2)])
|
|
|
|
def test_zerodivision(self, a, b):
|
|
|
|
> pytest.raises(ZeroDivisionError, "a/b")
|
|
|
|
E Failed: DID NOT RAISE
|
|
|
|
|
|
|
|
test_parametrize2.py:23: Failed
|
2011-05-01 18:38:56 +08:00
|
|
|
2 failed, 2 passed in 0.03 seconds
|
2010-11-21 04:35:55 +08:00
|
|
|
|
|
|
|
checking serialization between Python interpreters
|
|
|
|
--------------------------------------------------------------
|
|
|
|
|
|
|
|
Here is a stripped down real-life example of using parametrized
|
2011-02-17 21:46:40 +08:00
|
|
|
testing for testing serialization between different interpreters.
|
2010-11-21 04:35:55 +08:00
|
|
|
We define a ``test_basic_objects`` function which is to be run
|
|
|
|
with different sets of arguments for its three arguments::
|
|
|
|
|
|
|
|
* ``python1``: first python interpreter
|
|
|
|
* ``python2``: second python interpreter
|
|
|
|
* ``obj``: object to be dumped from first interpreter and loaded into second interpreter
|
|
|
|
|
|
|
|
.. literalinclude:: multipython.py
|
|
|
|
|
|
|
|
Running it (with Python-2.4 through to Python2.7 installed)::
|
|
|
|
|
|
|
|
. $ py.test -q multipython.py
|
2010-11-26 20:26:56 +08:00
|
|
|
collecting ... collected 75 items
|
2010-11-21 04:35:55 +08:00
|
|
|
....s....s....s....ssssss....s....s....s....ssssss....s....s....s....ssssss
|
2011-05-01 18:38:56 +08:00
|
|
|
48 passed, 27 skipped in 2.04 seconds
|