improve / add to dependency/test resource injection ------------------------------------------------------------- tags: wish feature docs write up better examples showing the connection between the two. refine parametrize API ------------------------------------------------------------- tags: critical feature extend metafunc.parametrize to directly support indirection, example: def setupdb(request, config): # setup "resource" based on test request and the values passed # in to parametrize. setupfunc is called for each such value. # you may use request.addfinalizer() or request.cached_setup ... return dynamic_setup_database(val) @pytest.mark.parametrize("db", ["pg", "mysql"], setupfunc=setupdb) def test_heavy_functional_test(db): ... There would be no need to write or explain funcarg factories and their special __ syntax. The examples and improvements should also show how to put the parametrize decorator to a class, to a module or even to a directory. For the directory part a conftest.py content like this:: pytestmark = [ @pytest.mark.parametrize_setup("db", ...), ] probably makes sense in order to keep the declarative nature. This mirrors the marker-mechanism with respect to a test module but puts it to a directory scale. When doing larger scoped parametrization it probably becomes neccessary to allow parametrization to be ignored if the according parameter is not used (currently any parametrized argument that is not present in a function will cause a ValueError). Example: @pytest.mark.parametrize("db", ..., mustmatch=False) means to not raise an error but simply ignore the parametrization if the signature of a decorated function does not match. XXX is it not sufficient to always allow non-matches? unify item/request classes, generalize items --------------------------------------------------------------- tags: 2.4 wish in lieu of extended parametrization and the new way to specify resource factories in terms of the parametrize decorator, consider unification of the item and request class. This also is connected with allowing funcargs in setup functions. Example of new item API: item.getresource("db") # alias for request.getfuncargvalue item.addfinalizer(...) item.cached_setup(...) item.applymarker(...) test classes/modules could then use this api via:: def pytest_runtest_setup(item): use item API ... introduction of this new method needs to be _fully_ backward compatible - and the documentation needs to change along to mention this new way of doing things. impl note: probably Request._fillfuncargs would be called from the python plugins own pytest_runtest_setup(item) and would call item.getresource(X) for all X in the funcargs of a function. XXX is it possible to even put the above item API to Nodes, i.e. also to Directorty/module/file/class collectors? Problem is that current funcarg factories presume they are called with a per-function (even per-funcarg-per-function) scope. Could there be small tweaks to the new API that lift this restriction? consider:: def setup_class(cls, tmpdir): # would get a per-class tmpdir because tmpdir parametrization # would know that it is called with a class scope # # # this looks very difficult because those setup functions are also used by nose etc. Rather consider introduction of a new setup hook: def setup_test(self, item): self.db = item.cached_setup(..., scope='class') self.tmpdir = item.getresource("tmpdir") this should be compatible to unittest/nose and provide much of what "testresources" provide. XXX This would not allow full parametrization such that test function could be run multiple times with different values. See "parametrized attributes" issue. allow parametrized attributes on classes -------------------------------------------------- tags: wish 2.4 example: @pytest.mark.parametrize_attr("db", setupfunc, [1,2,3], scope="class") @pytest.mark.parametrize_attr("tmp", setupfunc, scope="...") class TestMe: def test_hello(self): access self.db ... this would run the test_hello() function three times with three different values for self.db. This could also work with unittest/nose style tests, i.e. it leverages existing test suites without needing to rewrite them. Together with the previously mentioned setup_test() maybe the setupfunc could be ommitted? checks / deprecations for next release --------------------------------------------------------------- tags: bug 2.4 core xdist * check oejskit plugin compatibility * move pytest_nose out of pylib because it implicitely extends the protocol now - setup/teardown is called at module level. consider making calling of setup/teardown configurable optimizations --------------------------------------------------------------- tags: 2.4 core - look at ihook optimization such that all lookups for hooks relating to the same fspath are cached. fix start/finish partial finailization problem --------------------------------------------------------------- tags: bug core if a configure/runtest_setup/sessionstart/... hook invocation partially fails the sessionfinishes is not called. Each hook implementation should better be repsonsible for registering a cleanup/finalizer appropriately to avoid this issue. Moreover/Alternatively, we could record which implementations of a hook succeeded and only call their teardown. consider and document __init__ file usage in test directories --------------------------------------------------------------- tags: bug core Currently, a test module is imported with its fully qualified package path, determined by checking __init__ files upwards. This has the side effect that a source package at the root of the test dir could be imported as well. This is somewhat convenient but complicates the picture for running tests against different versions of a package. Also, implicit sys.path manipulations are problematic per-se. Maybe factorting out a pytest_addsyspath hook which can be disabled from the command line makes sense. In any case documentation/recommendations for certain scenarios makes sense. relax requirement to have tests/testing contain an __init__ ---------------------------------------------------------------- tags: feature bb: http://bitbucket.org/hpk42/py-trunk/issue/64 A local test run of a "tests" directory may work but a remote one fail because the tests directory does not contain an "__init__.py". Either give an error or make it work without the __init__.py i.e. port the nose-logic of unloading a test module. customize test function collection ------------------------------------------------------- tags: feature - introduce py.test.mark.nocollect for not considering a function for test collection at all. maybe also introduce a py.test.mark.test to explicitely mark a function to become a tested one. Lookup JUnit ways of tagging tests. introduce pytest.mark.importorskip ------------------------------------------------------- tags: feature in addition to the imperative pytest.importorskip also introduce a pytest.mark.importorskip so that the test count is more correct. introduce py.test.mark.platform ------------------------------------------------------- tags: feature Introduce nice-to-spell platform-skipping, examples: @py.test.mark.platform("python3") @py.test.mark.platform("not python3") @py.test.mark.platform("win32 and not python3") @py.test.mark.platform("darwin") @py.test.mark.platform("not (jython and win32)") @py.test.mark.platform("not (jython and win32)", xfail=True) etc. Idea is to allow Python expressions which can operate on common spellings for operating systems and python interpreter versions. pytest.mark.xfail signature change ------------------------------------------------------- tags: feature change to pytest.mark.xfail(reason, (optional)condition) to better implement the word meaning. It also signals better that we always have some kind of an implementation reason that can be formualated. Compatibility? how to introduce a new name/keep compat? allow to non-intrusively apply skipfs/xfail/marks --------------------------------------------------- tags: feature use case: mark a module or directory structures to be skipped on certain platforms (i.e. no import attempt will be made). consider introducing a hook/mechanism that allows to apply marks from conftests or plugins. (See extended parametrization) explicit referencing of conftest.py files ----------------------------------------- tags: feature allow to name conftest.py files (in sub directories) that should be imported early, as to include command line options. improve central py.test ini file ---------------------------------- tags: feature introduce more declarative configuration options: - (to-be-collected test directories) - required plugins - test func/class/file matching patterns - skip/xfail (non-intrusive) - pytest.ini and tox.ini and setup.cfg configuration in the same file new documentation ---------------------------------- tags: feature - logo py.test - examples for unittest or functional testing - resource management for functional testing - patterns: page object have imported module mismatch honour relative paths -------------------------------------------------------- tags: bug With 1.1.1 py.test fails at least on windows if an import is relative and compared against an absolute conftest.py path. Normalize. consider globals: py.test.ensuretemp and config -------------------------------------------------------------- tags: experimental-wish consider deprecating py.test.ensuretemp and py.test.config to further reduce py.test globality. Also consider having py.test.config and ensuretemp coming from a plugin rather than being there from the start. consider allowing funcargs for setup methods -------------------------------------------------------------- tags: experimental-wish Users have expressed the wish to have funcargs available to setup functions. Experiment with allowing funcargs there - it might also help to make the py.test.ensuretemp and config deprecation. For filling funcargs for setup methods, we could call funcarg factories with a request object that not have a cls/function attributes. However, how to handle parametrized test functions and funcargs? maybe introduce a setup method like: setup_invocation(self, request) which has full access to the test invocation through "request" through which you can get funcargvalues, use cached_setup etc. Therefore, the access to funcargs would be indirect but it could be consistently implemented. setup_invocation() would be a "glue" function for bringing together the xUnit and funcargs world. consider pytest_addsyspath hook ----------------------------------------- tags: py.test could call a new pytest_addsyspath() in order to systematically allow manipulation of sys.path and to inhibit it via --no-addsyspath in order to more easily run against installed packages. Alternatively it could also be done via the config object and pytest_configure. show plugin information in test header ---------------------------------------------------------------- tags: feature Now that external plugins are becoming more numerous it would be useful to have external plugins along with their versions displayed as a header line. deprecate global py.test.config usage ---------------------------------------------------------------- tags: feature py.test.ensuretemp and py.test.config are probably the last objects containing global state. Often using them is not neccessary. This is about trying to get rid of them, i.e. deprecating them and checking with PyPy's usages as well as others. remove deprecated bits in collect.py ------------------------------------------------------------------- tags: feature In an effort to further simplify code, review and remove deprecated bits in collect.py. Probably good: - inline consider_file/dir methods, no need to have them subclass-overridable because of hooks implement fslayout decorator --------------------------------- tags: feature Improve the way how tests can work with pre-made examples, keeping the layout close to the test function: @pytest.mark.fslayout(""" conftest.py: # empty tests/ test_%(NAME)s: # becomes test_run1.py def test_function(self): pass """) def test_run(pytester, fslayout): p = fslayout.findone("test_*.py") result = pytester.runpytest(p) assert result.ret == 0 assert result.passed == 1 Another idea is to allow to define a full scenario including the run in one content string:: runscenario(""" test_{TESTNAME}.py: import pytest @pytest.mark.xfail def test_that_fails(): assert 0 @pytest.mark.skipif("True") def test_hello(): pass conftest.py: import pytest def pytest_runsetup_setup(item): pytest.skip("abc") runpytest -rsxX *SKIP*{TESTNAME}* *1 skipped* """) This could be run with at least three different ways to invoke pytest: through the shell, through "python -m pytest" and inlined. As inlined would be the fastest it could be run first (or "--fast" mode).