write down some thoughts on parametrization - still not completely clear what the next stage should look like ...
This commit is contained in:
holger krekel
parent
1c746e0819
commit
b92176024c
108
ISSUES.txt
108
ISSUES.txt
|
|
@ -10,28 +10,27 @@ refine parametrize API
|
|||
-------------------------------------------------------------
|
||||
tags: critical feature
|
||||
|
||||
extend metafunc.parametrize to better support indirection
|
||||
by specifying a setupfunc(request, val) which will _substitute_
|
||||
the funcarg factory. Here is an example:
|
||||
extend metafunc.parametrize to directly support indirection, example:
|
||||
|
||||
def setupdb(request, val):
|
||||
def setupdb(request, config):
|
||||
# setup "resource" based on test request and the values passed
|
||||
# in to parametrize. setupfunc is called for each such value.
|
||||
# you may use request.addfinalizer() or request.cached_setup ...
|
||||
return db
|
||||
return dynamic_setup_database(val)
|
||||
|
||||
@pytest.mark.parametrize("db", ["pg", "mysql"], setupfunc=setupdb)
|
||||
def test_heavy_functional_test(db):
|
||||
...
|
||||
|
||||
There would be no need to write or explain funcarg factories here.
|
||||
There would be no need to write or explain funcarg factories and
|
||||
their special __ syntax.
|
||||
|
||||
The examples and improvements should also show how to put the parametrize
|
||||
decorator to a class, to a module or even to a directory. For the directory
|
||||
part a conftest.py content like this::
|
||||
|
||||
pytestmark = [
|
||||
@pytest.mark.parametrize("db", ...),
|
||||
@pytest.mark.parametrize_setup("db", ...),
|
||||
]
|
||||
|
||||
probably makes sense in order to keep the declarative nature. This mirrors
|
||||
|
|
@ -39,13 +38,87 @@ the marker-mechanism with respect to a test module but puts it to a directory
|
|||
scale.
|
||||
|
||||
When doing larger scoped parametrization it probably becomes neccessary
|
||||
to allow parametrization to not match - currently any parametrized argument
|
||||
that is not present in a function will cause a ValueError. With
|
||||
to allow parametrization to be ignored if the according parameter is not
|
||||
used (currently any parametrized argument that is not present in a function will cause a ValueError). Example:
|
||||
|
||||
@pytest.mark.parametrize("db", ..., mustmatch=False)
|
||||
|
||||
we can tell pytest to not raise an error but simply ignore the parametrization
|
||||
if the signature of a decorated function does not match.
|
||||
means to not raise an error but simply ignore the parametrization
|
||||
if the signature of a decorated function does not match. XXX is it
|
||||
not sufficient to always allow non-matches?
|
||||
|
||||
|
||||
unify item/request classes, generalize items
|
||||
---------------------------------------------------------------
|
||||
tags: 2.4 wish
|
||||
|
||||
in lieu of extended parametrization and the new way to specify resource
|
||||
factories in terms of the parametrize decorator, consider unification
|
||||
of the item and request class. This also is connected with allowing
|
||||
funcargs in setup functions. Example of new item API:
|
||||
|
||||
item.getresource("db") # alias for request.getfuncargvalue
|
||||
item.addfinalizer(...)
|
||||
item.cached_setup(...)
|
||||
item.applymarker(...)
|
||||
|
||||
test classes/modules could then use this api via::
|
||||
|
||||
def pytest_runtest_setup(item):
|
||||
use item API ...
|
||||
|
||||
introduction of this new method needs to be _fully_ backward compatible -
|
||||
and the documentation needs to change along to mention this new way of
|
||||
doing things.
|
||||
|
||||
impl note: probably Request._fillfuncargs would be called from the
|
||||
python plugins own pytest_runtest_setup(item) and would call
|
||||
item.getresource(X) for all X in the funcargs of a function.
|
||||
|
||||
XXX is it possible to even put the above item API to Nodes, i.e. also
|
||||
to Directorty/module/file/class collectors? Problem is that current
|
||||
funcarg factories presume they are called with a per-function (even
|
||||
per-funcarg-per-function) scope. Could there be small tweaks to the new
|
||||
API that lift this restriction?
|
||||
|
||||
consider::
|
||||
|
||||
def setup_class(cls, tmpdir):
|
||||
# would get a per-class tmpdir because tmpdir parametrization
|
||||
# would know that it is called with a class scope
|
||||
#
|
||||
#
|
||||
#
|
||||
this looks very difficult because those setup functions are also used
|
||||
by nose etc. Rather consider introduction of a new setup hook:
|
||||
|
||||
def setup_test(self, item):
|
||||
self.db = item.cached_setup(..., scope='class')
|
||||
self.tmpdir = item.getresource("tmpdir")
|
||||
|
||||
this should be compatible to unittest/nose and provide much of what
|
||||
"testresources" provide. XXX This would not allow full parametrization
|
||||
such that test function could be run multiple times with different
|
||||
values. See "parametrized attributes" issue.
|
||||
|
||||
allow parametrized attributes on classes
|
||||
--------------------------------------------------
|
||||
|
||||
tags: wish 2.4
|
||||
|
||||
example:
|
||||
|
||||
@pytest.mark.parametrize_attr("db", setupfunc, [1,2,3], scope="class")
|
||||
@pytest.mark.parametrize_attr("tmp", setupfunc, scope="...")
|
||||
class TestMe:
|
||||
def test_hello(self):
|
||||
access self.db ...
|
||||
|
||||
this would run the test_hello() function three times with three
|
||||
different values for self.db. This could also work with unittest/nose
|
||||
style tests, i.e. it leverages existing test suites without needing
|
||||
to rewrite them. Together with the previously mentioned setup_test()
|
||||
maybe the setupfunc could be ommitted?
|
||||
|
||||
checks / deprecations for next release
|
||||
---------------------------------------------------------------
|
||||
|
|
@ -74,18 +147,6 @@ appropriately to avoid this issue. Moreover/Alternatively, we could
|
|||
record which implementations of a hook succeeded and only call their
|
||||
teardown.
|
||||
|
||||
do early-teardown of test modules
|
||||
-----------------------------------------
|
||||
tags: feature
|
||||
|
||||
currently teardowns are called when the next tests is setup
|
||||
except for the function/method level where interally
|
||||
"teardown_exact" tears down immediately. Generalize
|
||||
this to perform the "neccessary" teardown compared to
|
||||
the "next" test item during teardown - this should
|
||||
get rid of some irritations because otherwise e.g.
|
||||
prints of teardown-code appear in the setup of the next test.
|
||||
|
||||
consider and document __init__ file usage in test directories
|
||||
---------------------------------------------------------------
|
||||
tags: bug core
|
||||
|
|
@ -167,6 +228,7 @@ attempt will be made).
|
|||
consider introducing a hook/mechanism that allows to apply marks
|
||||
from conftests or plugins. (See extended parametrization)
|
||||
|
||||
|
||||
explicit referencing of conftest.py files
|
||||
-----------------------------------------
|
||||
tags: feature
|
||||
|
|
|
|||
Loading…
Reference in New Issue