diff --git a/py/doc/impl-test.txt b/py/doc/impl-test.txt new file mode 100644 index 000000000..6636107df --- /dev/null +++ b/py/doc/impl-test.txt @@ -0,0 +1,273 @@ +=============================================== +Implementation and Customization of ``py.test`` +=============================================== + +.. contents:: +.. sectnum:: + + + +.. _`basicpicture`: + + +Collecting and running tests / implementation remarks +====================================================== + +In order to customize ``py.test`` it's good to understand +its basic architure (WARNING: these are not guaranteed +yet to stay the way they are now!):: + + ___________________ + | | + | Collector | + |___________________| + / \ + | Item.run() + | ^ + receive test Items / + | /execute test Item + | / + ___________________/ + | | + | Session | + |___________________| + + ............................. + . conftest.py configuration . + . cmdline options . + ............................. + + +The *Session* basically receives test *Items* from a *Collector*, +and executes them via the ``Item.run()`` method. It monitors +the outcome of the test and reports about failures and successes. + +.. _`collection process`: + +Collectors and the test collection process +------------------------------------------ + +The collecting process is iterative, i.e. the session +traverses and generates a *collector tree*. Here is an example of such +a tree, generated with the command ``py.test --collectonly py/xmlobj``:: + + + + + + + + + + + + + + + + + +By default all directories not starting with a dot are traversed, +looking for ``test_*.py`` and ``*_test.py`` files. Those files +are imported under their `package name`_. + +.. _`collector API`: + +test items are collectors as well +--------------------------------- + +To make the reporting life simple for the session object +items offer a ``run()`` method as well. In fact the session +distinguishes "collectors" from "items" solely by interpreting +their return value. If it is a list, then we recurse into +it, otherwise we consider the "test" as passed. + +.. _`package name`: + +constructing the package name for modules +----------------------------------------- + +Test modules are imported under their fully qualified +name. Given a module ``path`` the fully qualified package +name is constructed as follows: + +* determine the last "upward" directory from ``path`` that + contains an ``__init__.py`` file. Going upwards + means repeatedly calling the ``dirpath()`` method + on a path object (which returns the parent directory + as a path object). + +* insert this base directory into the sys.path list + as its first element + +* import the root package + +* determine the fully qualified name for the module located + at ``path`` ... + + * if the imported root package has a __package__ object + then call ``__package__.getimportname(path)`` + + * otherwise use the relative path of the module path to + the base dir and turn slashes into dots and strike + the trailing ``.py``. + +The Module collector will eventually trigger +``__import__(mod_fqdnname, ...)`` to finally get to +the live module object. + +Side note: this whole logic is performed by local path +object's ``pyimport()`` method. + +Module Collector +----------------- + +The default Module collector looks for test functions +and test classes and methods. Test functions and methods +are prefixed ``test`` by default. Test classes must +start with a capitalized ``Test`` prefix. + + +Customizing the testing process +=============================== + +writing conftest.py files +----------------------------------- + +XXX + +adding custom options ++++++++++++++++++++++++ + +To register a project-specific command line option +you may have the following code within a ``conftest.py`` file:: + + import py + Option = py.test.config.Option + option = py.test.config.addoptions("pypy options", + Option('-V', '--view', action="store_true", dest="view", default=False, + help="view translation tests' flow graphs with Pygame"), + ) + +and you can then access ``option.view`` like this:: + + if option.view: + print "view this!" + +The option will be available if you type ``py.test -h`` +Note that you may only register upper case short +options. ``py.test`` reserves all lower +case short options for its own cross-project usage. + +customizing the collecting and running process +----------------------------------------------- + +To introduce different test items you can create +one or more ``conftest.py`` files in your project. +When the collection process traverses directories +and modules the default collectors will produce +custom Collectors and Items if they are found +in a local ``conftest.py`` file. + +example: perform additional ReST checs +++++++++++++++++++++++++++++++++++++++ + +With your custom collectors or items you can completely +derive from the standard way of collecting and running +tests in a localized manner. Let's look at an example. +If you invoke ``py.test --collectonly py/documentation`` +then you get:: + + + + + + + + + + + + + + + + + ... + +In ``py/documentation/conftest.py`` you find the following +customization:: + + class DocDirectory(py.test.collect.Directory): + + def run(self): + results = super(DocDirectory, self).run() + for x in self.fspath.listdir('*.txt', sort=True): + results.append(x.basename) + return results + + def join(self, name): + if not name.endswith('.txt'): + return super(DocDirectory, self).join(name) + p = self.fspath.join(name) + if p.check(file=1): + return ReSTChecker(p, parent=self) + + Directory = DocDirectory + +The existence of the 'Directory' name in the +``pypy/documentation/conftest.py`` module makes the collection +process defer to our custom "DocDirectory" collector. We extend +the set of collected test items by ``ReSTChecker`` instances +which themselves create ``ReSTSyntaxTest`` and ``LinkCheckerMaker`` +items. All of this instances (need to) follow the `collector API`_. + + +Customizing the collection process in a module +---------------------------------------------- + + REPEATED WARNING: details of the collection and running process are + still subject to refactorings and thus details will change. + If you are customizing py.test at "Item" level then you + definitely want to be subscribed to the `py-dev mailing list`_ + to follow ongoing development. + +If you have a module where you want to take responsibility for +collecting your own test Items and possibly even for executing +a test then you can provide `generative tests`_ that yield +callables and possibly arguments as a tuple. This should +serve some immediate purposes like paramtrized tests. + +.. _`generative tests`: test.html#generative-tests + +The other extension possibility goes deeper into the machinery +and allows you to specify a custom test ``Item`` class which +is responsible for setting up and executing an underlying +test. [XXX not working: You can integrate your custom ``py.test.Item`` subclass +by binding an ``Item`` name to a test class.] Or you can +extend the collection process for a whole directory tree +by putting Items in a ``conftest.py`` configuration file. +The collection process constantly looks at according names +in the *chain of conftest.py* modules to determine collectors +and items at ``Directory``, ``Module``, ``Class``, ``Function`` +or ``Generator`` level. Note that, right now, except for ``Function`` +items all classes are pure collectors, i.e. will return a list +of names (possibly empty). + +XXX implement doctests as alternatives to ``Function`` items. + +Customizing execution of Functions +---------------------------------- + +- Function test items allow total control of executing their + contained test method. ``function.run()`` will get called by the + session in order to actually run a test. The method is responsible + for performing proper setup/teardown ("Test Fixtures") for a + Function test. + +- ``Function.execute(target, *args)`` methods are invoked by + the default ``Function.run()`` to actually execute a python + function with the given (usually empty set of) arguments. + + +.. _`py-dev mailing list`: http://codespeak.net/mailman/listinfo/py-dev diff --git a/py/doc/test.txt b/py/doc/test.txt index c7e079e47..7510993bf 100644 --- a/py/doc/test.txt +++ b/py/doc/test.txt @@ -5,6 +5,12 @@ The ``py.test`` tool and library .. contents:: .. sectnum:: + +This document is about the *usage* of the ``py.test`` testing tool. There is +also document describing the `implementation and the extending of py.test`_. + +.. _`implementation and the extending of py.test`: impl-test.html + starting point: ``py.test`` command line tool ============================================= @@ -40,6 +46,9 @@ start with ``test_`` or ends with ``_test`` from the directory and any subdirectories, starting with the current directory, and run them. Each Python test module is inspected for test methods starting with ``test_``. +.. _`getting started`: getting-started.html + + .. _features: Basic Features of ``py.test`` @@ -94,6 +103,7 @@ be customized at directory, module or class level. (see `collection process`_ for some implementation details). .. _`generative tests`: +.. _`collection process`: impl-test.html#collection-process generative tests: yielding more tests ------------------------------------- @@ -206,6 +216,10 @@ instance because of infinite recursion, ``py.test`` will indicate where in the code the recursion was taking place. You can inhibit traceback "cutting" magic by supplying ``--fulltrace``. +There is also the possibility of usind ``--tb=short`` to get the regular Python +tracebacks (which can sometimes be useful when they are extremely long). Or you +can use ``--tb=no`` to not show any tracebacks at all. + no inheritance requirement -------------------------- @@ -356,270 +370,6 @@ to insert ``setup_class = classmethod(setup_class)`` to make your setup function callable. Did we mention that lazyness is a virtue? -.. _`basicpicture`: - - -Collecting and running tests / implementation remarks -====================================================== - -In order to customize ``py.test`` it's good to understand -its basic architure (WARNING: these are not guaranteed -yet to stay the way they are now!):: - - ___________________ - | | - | Collector | - |___________________| - / \ - | Item.run() - | ^ - receive test Items / - | /execute test Item - | / - ___________________/ - | | - | Session | - |___________________| - - ............................. - . conftest.py configuration . - . cmdline options . - ............................. - - -The *Session* basically receives test *Items* from a *Collector*, -and executes them via the ``Item.run()`` method. It monitors -the outcome of the test and reports about failures and successes. - -.. _`collection process`: - -Collectors and the test collection process ------------------------------------------- - -The collecting process is iterative, i.e. the session -traverses and generates a *collector tree*. Here is an example of such -a tree, generated with the command ``py.test --collectonly py/xmlobj``:: - - - - - - - - - - - - - - - - - -By default all directories not starting with a dot are traversed, -looking for ``test_*.py`` and ``*_test.py`` files. Those files -are imported under their `package name`_. - -.. _`collector API`: - -test items are collectors as well ---------------------------------- - -To make the reporting life simple for the session object -items offer a ``run()`` method as well. In fact the session -distinguishes "collectors" from "items" solely by interpreting -their return value. If it is a list, then we recurse into -it, otherwise we consider the "test" as passed. - -.. _`package name`: - -constructing the package name for modules ------------------------------------------ - -Test modules are imported under their fully qualified -name. Given a module ``path`` the fully qualified package -name is constructed as follows: - -* determine the last "upward" directory from ``path`` that - contains an ``__init__.py`` file. Going upwards - means repeatedly calling the ``dirpath()`` method - on a path object (which returns the parent directory - as a path object). - -* insert this base directory into the sys.path list - as its first element - -* import the root package - -* determine the fully qualified name for the module located - at ``path`` ... - - * if the imported root package has a __package__ object - then call ``__package__.getimportname(path)`` - - * otherwise use the relative path of the module path to - the base dir and turn slashes into dots and strike - the trailing ``.py``. - -The Module collector will eventually trigger -``__import__(mod_fqdnname, ...)`` to finally get to -the live module object. - -Side note: this whole logic is performed by local path -object's ``pyimport()`` method. - -Module Collector ------------------ - -The default Module collector looks for test functions -and test classes and methods. Test functions and methods -are prefixed ``test`` by default. Test classes must -start with a capitalized ``Test`` prefix. - - -Customizing the testing process -=============================== - -writing conftest.py files ------------------------------------ - -XXX - -adding custom options -+++++++++++++++++++++++ - -To register a project-specific command line option -you may have the following code within a ``conftest.py`` file:: - - import py - Option = py.test.config.Option - option = py.test.config.addoptions("pypy options", - Option('-V', '--view', action="store_true", dest="view", default=False, - help="view translation tests' flow graphs with Pygame"), - ) - -and you can then access ``option.view`` like this:: - - if option.view: - print "view this!" - -The option will be available if you type ``py.test -h`` -Note that you may only register upper case short -options. ``py.test`` reserves all lower -case short options for its own cross-project usage. - -customizing the collecting and running process ------------------------------------------------ - -To introduce different test items you can create -one or more ``conftest.py`` files in your project. -When the collection process traverses directories -and modules the default collectors will produce -custom Collectors and Items if they are found -in a local ``conftest.py`` file. - -example: perform additional ReST checs -++++++++++++++++++++++++++++++++++++++ - -With your custom collectors or items you can completely -derive from the standard way of collecting and running -tests in a localized manner. Let's look at an example. -If you invoke ``py.test --collectonly py/documentation`` -then you get:: - - - - - - - - - - - - - - - - - ... - -In ``py/documentation/conftest.py`` you find the following -customization:: - - class DocDirectory(py.test.collect.Directory): - - def run(self): - results = super(DocDirectory, self).run() - for x in self.fspath.listdir('*.txt', sort=True): - results.append(x.basename) - return results - - def join(self, name): - if not name.endswith('.txt'): - return super(DocDirectory, self).join(name) - p = self.fspath.join(name) - if p.check(file=1): - return ReSTChecker(p, parent=self) - - Directory = DocDirectory - -The existence of the 'Directory' name in the -``pypy/documentation/conftest.py`` module makes the collection -process defer to our custom "DocDirectory" collector. We extend -the set of collected test items by ``ReSTChecker`` instances -which themselves create ``ReSTSyntaxTest`` and ``LinkCheckerMaker`` -items. All of this instances (need to) follow the `collector API`_. - - -Customizing the collection process in a module ----------------------------------------------- - - REPEATED WARNING: details of the collection and running process are - still subject to refactorings and thus details will change. - If you are customizing py.test at "Item" level then you - definitely want to be subscribed to the `py-dev mailing list`_ - to follow ongoing development. - -If you have a module where you want to take responsibility for -collecting your own test Items and possibly even for executing -a test then you can provide `generative tests`_ that yield -callables and possibly arguments as a tuple. This should -serve some immediate purposes like paramtrized tests. - -The other extension possibility goes deeper into the machinery -and allows you to specify a custom test ``Item`` class which -is responsible for setting up and executing an underlying -test. [XXX not working: You can integrate your custom ``py.test.Item`` subclass -by binding an ``Item`` name to a test class.] Or you can -extend the collection process for a whole directory tree -by putting Items in a ``conftest.py`` configuration file. -The collection process constantly looks at according names -in the *chain of conftest.py* modules to determine collectors -and items at ``Directory``, ``Module``, ``Class``, ``Function`` -or ``Generator`` level. Note that, right now, except for ``Function`` -items all classes are pure collectors, i.e. will return a list -of names (possibly empty). - -XXX implement doctests as alternatives to ``Function`` items. - -Customizing execution of Functions ----------------------------------- - -- Function test items allow total control of executing their - contained test method. ``function.run()`` will get called by the - session in order to actually run a test. The method is responsible - for performing proper setup/teardown ("Test Fixtures") for a - Function test. - -- ``Function.execute(target, *args)`` methods are invoked by - the default ``Function.run()`` to actually execute a python - function with the given (usually empty set of) arguments. - -.. _`getting started`: getting-started.html -.. _`py-dev mailing list`: http://codespeak.net/mailman/listinfo/py-dev - - Automated Distributed Testing ================================== @@ -710,8 +460,10 @@ Sample configuration:: dist_maxwait = 100 dist_taskspernode = 10 -Running server is done by ``-w`` command line option or ``--startserver`` -(the former might change at some point due to conflicts). +To use the browser-based reporter (with a nice AJAX interface) you have to tell +``py.test`` to run a small server locally using the ``-w`` or ``--startserver`` +command line options. Afterwards you can point your browser to localhost:8000 +to see the progress of the testing. Development Notes -----------------