checks / deprecations for next release --------------------------------------------------------------- tags: bug 2.4 core xdist * check oejskit plugin compatibility * move pytest_nose out of pylib because it implicitely extends the protocol now - setup/teardown is called at module level. consider making calling of setup/teardown configurable optimizations --------------------------------------------------------------- tags: 2.4 core - look at ihook optimization such that all lookups for hooks relating to the same fspath are cached. fix start/finish partial finailization problem --------------------------------------------------------------- tags: bug core if a configure/runtest_setup/sessionstart/... hook invocation partially fails the sessionfinishes is not called. Each hook implementation should better be repsonsible for registering a cleanup/finalizer appropriately to avoid this issue. Moreover/Alternatively, we could record which implementations of a hook succeeded and only call their teardown. do early-teardown of test modules ----------------------------------------- tags: feature 2.2 currently teardowns are called when the next tests is setup except for the function/method level where interally "teardown_exact" tears down immediately. Generalize this to perform the "neccessary" teardown compared to the "next" test item during teardown - this should get rid of some irritations because otherwise e.g. prints of teardown-code appear in the setup of the next test. consider and document __init__ file usage in test directories --------------------------------------------------------------- tags: bug 2.2 core Currently, a test module is imported with its fully qualified package path, determined by checking __init__ files upwards. This has the side effect that a source package at the root of the test dir could be imported as well. This is somewhat convenient but complicates the picture for running tests against different versions of a package. Also, implicit sys.path manipulations are problematic per-se. Maybe factorting out a pytest_addsyspath hook which can be disabled from the command line makes sense. In any case documentation/recommendations for certain scenarios makes sense. relax requirement to have tests/testing contain an __init__ ---------------------------------------------------------------- tags: feature 2.2 bb: http://bitbucket.org/hpk42/py-trunk/issue/64 A local test run of a "tests" directory may work but a remote one fail because the tests directory does not contain an "__init__.py". Either give an error or make it work without the __init__.py i.e. port the nose-logic of unloading a test module. customize test function collection ------------------------------------------------------- tags: feature 2.2 - introduce py.test.mark.nocollect for not considering a function for test collection at all. maybe also introduce a py.test.mark.test to explicitely mark a function to become a tested one. Lookup JUnit ways of tagging tests. introduce pytest.mark.importorskip ------------------------------------------------------- tags: feature 2.2 in addition to the imperative pytest.importorskip also introduce a pytest.mark.importorskip so that the test count is more correct. introduce py.test.mark.platform ------------------------------------------------------- tags: feature 2.2 Introduce nice-to-spell platform-skipping, examples: @py.test.mark.platform("python3") @py.test.mark.platform("not python3") @py.test.mark.platform("win32 and not python3") @py.test.mark.platform("darwin") @py.test.mark.platform("not (jython and win32)") @py.test.mark.platform("not (jython and win32)", xfail=True) etc. Idea is to allow Python expressions which can operate on common spellings for operating systems and python interpreter versions. pytest.mark.xfail signature change ------------------------------------------------------- tags: feature 2.2 change to pytest.mark.xfail(reason, (optional)condition) to better implement the word meaning. It also signals better that we always have some kind of an implementation reason that can be formualated. Compatibility? Maybe rename to "pytest.mark.xfail"? introduce py.test.mark registration ----------------------------------------- tags: feature 2.2 introduce a hook that allows to register a named mark decorator with documentation and add "py.test --marks" to get a list of available marks. Deprecate "dynamic" mark definitions. allow to non-intrusively apply skipfs/xfail/marks --------------------------------------------------- tags: feature 2.2 use case: mark a module or directory structures to be skipped on certain platforms (i.e. no import attempt will be made). consider introducing a hook/mechanism that allows to apply marks from conftests or plugins. explicit referencing of conftest.py files ----------------------------------------- tags: feature 2.2 allow to name conftest.py files (in sub directories) that should be imported early, as to include command line options. improve central py.test ini file ---------------------------------- tags: feature 2.2 introduce more declarative configuration options: - (to-be-collected test directories) - required plugins - test func/class/file matching patterns - skip/xfail (non-intrusive) - pytest.ini and tox.ini and setup.cfg configuration in the same file new documentation ---------------------------------- tags: feature 2.2 - logo py.test - examples for unittest or functional testing - resource management for functional testing - patterns: page object - parametrized testing - better / more integrated plugin docs generalize parametrized testing to generate combinations ------------------------------------------------------------- tags: feature 2.2 think about extending metafunc.addcall or add a new method to allow to generate tests with combinations of all generated versions - what to do about "id" and "param" in such combinations though? introduce py.test.mark.multi ----------------------------------------- tags: feature 1.3 introduce py.test.mark.multi to specify a number of values for a given function argument. have imported module mismatch honour relative paths -------------------------------------------------------- tags: bug 2.2 With 1.1.1 py.test fails at least on windows if an import is relative and compared against an absolute conftest.py path. Normalize. call termination with small timeout ------------------------------------------------- tags: feature 2.2 test: testing/pytest/dist/test_dsession.py - test_terminate_on_hanging_node Call gateway group termination with a small timeout if available. Should make dist-testing less likely to leave lost processes. consider globals: py.test.ensuretemp and config -------------------------------------------------------------- tags: experimental-wish 2.2 consider deprecating py.test.ensuretemp and py.test.config to further reduce py.test globality. Also consider having py.test.config and ensuretemp coming from a plugin rather than being there from the start. consider allowing funcargs for setup methods -------------------------------------------------------------- tags: experimental-wish 2.2 Users have expressed the wish to have funcargs available to setup functions. Experiment with allowing funcargs there - it might also help to make the py.test.ensuretemp and config deprecation. For filling funcargs for setup methods, we could call funcarg factories with a request object that not have a cls/function attributes. However, how to handle parametrized test functions and funcargs? setup_function -> request can be like it is now setup_class -> request has no request.function setup_module -> request has no request.cls consider pytest_addsyspath hook ----------------------------------------- tags: 2.2 py.test could call a new pytest_addsyspath() in order to systematically allow manipulation of sys.path and to inhibit it via --no-addsyspath in order to more easily run against installed packages. Alternatively it could also be done via the config object and pytest_configure. show plugin information in test header ---------------------------------------------------------------- tags: feature 2.2 Now that external plugins are becoming more numerous it would be useful to have external plugins along with their versions displayed as a header line. deprecate global py.test.config usage ---------------------------------------------------------------- tags: feature 2.2 py.test.ensuretemp and py.test.config are probably the last objects containing global state. Often using them is not neccessary. This is about trying to get rid of them, i.e. deprecating them and checking with PyPy's usages as well as others. remove deprecated bits in collect.py ------------------------------------------------------------------- tags: feature 2.2 In an effort to further simplify code, review and remove deprecated bits in collect.py. Probably good: - inline consider_file/dir methods, no need to have them subclass-overridable because of hooks