p53209761
/
test_ok2
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[svn r37987] split the py.test docs into implementation and customization docs and normal 

user docs.

--HG--
branch : trunk
This commit is contained in:
cfbolz 2007-02-06 00:18:22 +01:00
parent cc0dfc1c3c
commit a430e2b8f5
2 changed files with 291 additions and 266 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

273
py/doc/impl-test.txt Normal file
View File

@ -0,0 +1,273 @@
===============================================
Implementation and Customization of ``py.test``
===============================================
.. contents::
.. sectnum::
.. _`basicpicture`:
Collecting and running tests / implementation remarks
======================================================
In order to customize ``py.test`` it's good to understand
its basic architure (WARNING: these are not guaranteed
yet to stay the way they are now!)::
___________________
| |
| Collector |
|___________________|
/ \
| Item.run()
| ^
receive test Items /
| /execute test Item
| /
___________________/
| |
| Session |
|___________________|
.............................
. conftest.py configuration .
. cmdline options .
.............................
The *Session* basically receives test *Items* from a *Collector*,
and executes them via the ``Item.run()`` method. It monitors
the outcome of the test and reports about failures and successes.
.. _`collection process`:
Collectors and the test collection process
------------------------------------------
The collecting process is iterative, i.e. the session
traverses and generates a *collector tree*. Here is an example of such
a tree, generated with the command ``py.test --collectonly py/xmlobj``::
<Directory 'xmlobj'>
<Directory 'testing'>
<Module 'test_html.py' (py.__.xmlobj.testing.test_html)>
<Function 'test_html_name_stickyness'>
<Function 'test_stylenames'>
<Function 'test_class_None'>
<Function 'test_alternating_style'>
<Module 'test_xml.py' (py.__.xmlobj.testing.test_xml)>
<Function 'test_tag_with_text'>
<Function 'test_class_identity'>
<Function 'test_tag_with_text_and_attributes'>
<Function 'test_tag_with_subclassed_attr_simple'>
<Function 'test_tag_nested'>
<Function 'test_tag_xmlname'>
By default all directories not starting with a dot are traversed,
looking for ``test_*.py`` and ``*_test.py`` files. Those files
are imported under their `package name`_.
.. _`collector API`:
test items are collectors as well
---------------------------------
To make the reporting life simple for the session object
items offer a ``run()`` method as well. In fact the session
distinguishes "collectors" from "items" solely by interpreting
their return value. If it is a list, then we recurse into
it, otherwise we consider the "test" as passed.
.. _`package name`:
constructing the package name for modules
-----------------------------------------
Test modules are imported under their fully qualified
name. Given a module ``path`` the fully qualified package
name is constructed as follows:
* determine the last "upward" directory from ``path`` that
contains an ``__init__.py`` file. Going upwards
means repeatedly calling the ``dirpath()`` method
on a path object (which returns the parent directory
as a path object).
* insert this base directory into the sys.path list
as its first element
* import the root package
* determine the fully qualified name for the module located
at ``path`` ...
* if the imported root package has a __package__ object
then call ``__package__.getimportname(path)``
* otherwise use the relative path of the module path to
the base dir and turn slashes into dots and strike
the trailing ``.py``.
The Module collector will eventually trigger
``__import__(mod_fqdnname, ...)`` to finally get to
the live module object.
Side note: this whole logic is performed by local path
object's ``pyimport()`` method.
Module Collector
-----------------
The default Module collector looks for test functions
and test classes and methods. Test functions and methods
are prefixed ``test`` by default. Test classes must
start with a capitalized ``Test`` prefix.
Customizing the testing process
===============================
writing conftest.py files
-----------------------------------
XXX
adding custom options
+++++++++++++++++++++++
To register a project-specific command line option
you may have the following code within a ``conftest.py`` file::
import py
Option = py.test.config.Option
option = py.test.config.addoptions("pypy options",
Option('-V', '--view', action="store_true", dest="view", default=False,
help="view translation tests' flow graphs with Pygame"),
)
and you can then access ``option.view`` like this::
if option.view:
print "view this!"
The option will be available if you type ``py.test -h``
Note that you may only register upper case short
options. ``py.test`` reserves all lower
case short options for its own cross-project usage.
customizing the collecting and running process
-----------------------------------------------
To introduce different test items you can create
one or more ``conftest.py`` files in your project.
When the collection process traverses directories
and modules the default collectors will produce
custom Collectors and Items if they are found
in a local ``conftest.py`` file.
example: perform additional ReST checs
++++++++++++++++++++++++++++++++++++++
With your custom collectors or items you can completely
derive from the standard way of collecting and running
tests in a localized manner. Let's look at an example.
If you invoke ``py.test --collectonly py/documentation``
then you get::
<DocDirectory 'documentation'>
<DocDirectory 'example'>
<DocDirectory 'pytest'>
<Module 'test_setup_flow_example.py' (test_setup_flow_example)>
<Class 'TestStateFullThing'>
<Instance '()'>
<Function 'test_42'>
<Function 'test_23'>
<ReSTChecker 'TODO.txt'>
<ReSTSyntaxTest 'TODO.txt'>
<LinkCheckerMaker 'checklinks'>
<ReSTChecker 'api.txt'>
<ReSTSyntaxTest 'api.txt'>
<LinkCheckerMaker 'checklinks'>
<CheckLink 'getting-started.html'>
...
In ``py/documentation/conftest.py`` you find the following
customization::
class DocDirectory(py.test.collect.Directory):
def run(self):
results = super(DocDirectory, self).run()
for x in self.fspath.listdir('*.txt', sort=True):
results.append(x.basename)
return results
def join(self, name):
if not name.endswith('.txt'):
return super(DocDirectory, self).join(name)
p = self.fspath.join(name)
if p.check(file=1):
return ReSTChecker(p, parent=self)
Directory = DocDirectory
The existence of the 'Directory' name in the
``pypy/documentation/conftest.py`` module makes the collection
process defer to our custom "DocDirectory" collector. We extend
the set of collected test items by ``ReSTChecker`` instances
which themselves create ``ReSTSyntaxTest`` and ``LinkCheckerMaker``
items. All of this instances (need to) follow the `collector API`_.
Customizing the collection process in a module
----------------------------------------------
REPEATED WARNING: details of the collection and running process are
still subject to refactorings and thus details will change.
If you are customizing py.test at "Item" level then you
definitely want to be subscribed to the `py-dev mailing list`_
to follow ongoing development.
If you have a module where you want to take responsibility for
collecting your own test Items and possibly even for executing
a test then you can provide `generative tests`_ that yield
callables and possibly arguments as a tuple. This should
serve some immediate purposes like paramtrized tests.
.. _`generative tests`: test.html#generative-tests
The other extension possibility goes deeper into the machinery
and allows you to specify a custom test ``Item`` class which
is responsible for setting up and executing an underlying
test. [XXX not working: You can integrate your custom ``py.test.Item`` subclass
by binding an ``Item`` name to a test class.] Or you can
extend the collection process for a whole directory tree
by putting Items in a ``conftest.py`` configuration file.
The collection process constantly looks at according names
in the *chain of conftest.py* modules to determine collectors
and items at ``Directory``, ``Module``, ``Class``, ``Function``
or ``Generator`` level. Note that, right now, except for ``Function``
items all classes are pure collectors, i.e. will return a list
of names (possibly empty).
XXX implement doctests as alternatives to ``Function`` items.
Customizing execution of Functions
----------------------------------
- Function test items allow total control of executing their
contained test method. ``function.run()`` will get called by the
session in order to actually run a test. The method is responsible
for performing proper setup/teardown ("Test Fixtures") for a
Function test.
- ``Function.execute(target, *args)`` methods are invoked by
the default ``Function.run()`` to actually execute a python
function with the given (usually empty set of) arguments.
.. _`py-dev mailing list`: http://codespeak.net/mailman/listinfo/py-dev

284
py/doc/test.txt
View File

@ -5,6 +5,12 @@ The ``py.test`` tool and library
.. contents:: .. contents::
.. sectnum:: .. sectnum::
This document is about the *usage* of the ``py.test`` testing tool. There is
also document describing the `implementation and the extending of py.test`_.
.. _`implementation and the extending of py.test`: impl-test.html
starting point: ``py.test`` command line tool starting point: ``py.test`` command line tool
============================================= =============================================
@ -40,6 +46,9 @@ start with ``test_`` or ends with ``_test`` from the directory and any
subdirectories, starting with the current directory, and run them. Each subdirectories, starting with the current directory, and run them. Each
Python test module is inspected for test methods starting with ``test_``. Python test module is inspected for test methods starting with ``test_``.
.. _`getting started`: getting-started.html
.. _features: .. _features:
Basic Features of ``py.test`` Basic Features of ``py.test``
@ -94,6 +103,7 @@ be customized at directory, module or class level. (see
`collection process`_ for some implementation details). `collection process`_ for some implementation details).
.. _`generative tests`: .. _`generative tests`:
.. _`collection process`: impl-test.html#collection-process
generative tests: yielding more tests generative tests: yielding more tests
------------------------------------- -------------------------------------
@ -206,6 +216,10 @@ instance because of infinite recursion, ``py.test`` will indicate
where in the code the recursion was taking place. You can where in the code the recursion was taking place. You can
inhibit traceback "cutting" magic by supplying ``--fulltrace``. inhibit traceback "cutting" magic by supplying ``--fulltrace``.
There is also the possibility of usind ``--tb=short`` to get the regular Python
tracebacks (which can sometimes be useful when they are extremely long). Or you
can use ``--tb=no`` to not show any tracebacks at all.
no inheritance requirement no inheritance requirement
-------------------------- --------------------------
@ -356,270 +370,6 @@ to insert ``setup_class = classmethod(setup_class)`` to make
your setup function callable. Did we mention that lazyness your setup function callable. Did we mention that lazyness
is a virtue? is a virtue?
.. _`basicpicture`:
Collecting and running tests / implementation remarks
======================================================
In order to customize ``py.test`` it's good to understand
its basic architure (WARNING: these are not guaranteed
yet to stay the way they are now!)::
___________________
| |
| Collector |
|___________________|
/ \
| Item.run()
| ^
receive test Items /
| /execute test Item
| /
___________________/
| |
| Session |
|___________________|
.............................
. conftest.py configuration .
. cmdline options .
.............................
The *Session* basically receives test *Items* from a *Collector*,
and executes them via the ``Item.run()`` method. It monitors
the outcome of the test and reports about failures and successes.
.. _`collection process`:
Collectors and the test collection process
------------------------------------------
The collecting process is iterative, i.e. the session
traverses and generates a *collector tree*. Here is an example of such
a tree, generated with the command ``py.test --collectonly py/xmlobj``::
<Directory 'xmlobj'>
<Directory 'testing'>
<Module 'test_html.py' (py.__.xmlobj.testing.test_html)>
<Function 'test_html_name_stickyness'>
<Function 'test_stylenames'>
<Function 'test_class_None'>
<Function 'test_alternating_style'>
<Module 'test_xml.py' (py.__.xmlobj.testing.test_xml)>
<Function 'test_tag_with_text'>
<Function 'test_class_identity'>
<Function 'test_tag_with_text_and_attributes'>
<Function 'test_tag_with_subclassed_attr_simple'>
<Function 'test_tag_nested'>
<Function 'test_tag_xmlname'>
By default all directories not starting with a dot are traversed,
looking for ``test_*.py`` and ``*_test.py`` files. Those files
are imported under their `package name`_.
.. _`collector API`:
test items are collectors as well
---------------------------------
To make the reporting life simple for the session object
items offer a ``run()`` method as well. In fact the session
distinguishes "collectors" from "items" solely by interpreting
their return value. If it is a list, then we recurse into
it, otherwise we consider the "test" as passed.
.. _`package name`:
constructing the package name for modules
-----------------------------------------
Test modules are imported under their fully qualified
name. Given a module ``path`` the fully qualified package
name is constructed as follows:
* determine the last "upward" directory from ``path`` that
contains an ``__init__.py`` file. Going upwards
means repeatedly calling the ``dirpath()`` method
on a path object (which returns the parent directory
as a path object).
* insert this base directory into the sys.path list
as its first element
* import the root package
* determine the fully qualified name for the module located
at ``path`` ...
* if the imported root package has a __package__ object
then call ``__package__.getimportname(path)``
* otherwise use the relative path of the module path to
the base dir and turn slashes into dots and strike
the trailing ``.py``.
The Module collector will eventually trigger
``__import__(mod_fqdnname, ...)`` to finally get to
the live module object.
Side note: this whole logic is performed by local path
object's ``pyimport()`` method.
Module Collector
-----------------
The default Module collector looks for test functions
and test classes and methods. Test functions and methods
are prefixed ``test`` by default. Test classes must
start with a capitalized ``Test`` prefix.
Customizing the testing process
===============================
writing conftest.py files
-----------------------------------
XXX
adding custom options
+++++++++++++++++++++++
To register a project-specific command line option
you may have the following code within a ``conftest.py`` file::
import py
Option = py.test.config.Option
option = py.test.config.addoptions("pypy options",
Option('-V', '--view', action="store_true", dest="view", default=False,
help="view translation tests' flow graphs with Pygame"),
)
and you can then access ``option.view`` like this::
if option.view:
print "view this!"
The option will be available if you type ``py.test -h``
Note that you may only register upper case short
options. ``py.test`` reserves all lower
case short options for its own cross-project usage.
customizing the collecting and running process
-----------------------------------------------
To introduce different test items you can create
one or more ``conftest.py`` files in your project.
When the collection process traverses directories
and modules the default collectors will produce
custom Collectors and Items if they are found
in a local ``conftest.py`` file.
example: perform additional ReST checs
++++++++++++++++++++++++++++++++++++++
With your custom collectors or items you can completely
derive from the standard way of collecting and running
tests in a localized manner. Let's look at an example.
If you invoke ``py.test --collectonly py/documentation``
then you get::
<DocDirectory 'documentation'>
<DocDirectory 'example'>
<DocDirectory 'pytest'>
<Module 'test_setup_flow_example.py' (test_setup_flow_example)>
<Class 'TestStateFullThing'>
<Instance '()'>
<Function 'test_42'>
<Function 'test_23'>
<ReSTChecker 'TODO.txt'>
<ReSTSyntaxTest 'TODO.txt'>
<LinkCheckerMaker 'checklinks'>
<ReSTChecker 'api.txt'>
<ReSTSyntaxTest 'api.txt'>
<LinkCheckerMaker 'checklinks'>
<CheckLink 'getting-started.html'>
...
In ``py/documentation/conftest.py`` you find the following
customization::
class DocDirectory(py.test.collect.Directory):
def run(self):
results = super(DocDirectory, self).run()
for x in self.fspath.listdir('*.txt', sort=True):
results.append(x.basename)
return results
def join(self, name):
if not name.endswith('.txt'):
return super(DocDirectory, self).join(name)
p = self.fspath.join(name)
if p.check(file=1):
return ReSTChecker(p, parent=self)
Directory = DocDirectory
The existence of the 'Directory' name in the
``pypy/documentation/conftest.py`` module makes the collection
process defer to our custom "DocDirectory" collector. We extend
the set of collected test items by ``ReSTChecker`` instances
which themselves create ``ReSTSyntaxTest`` and ``LinkCheckerMaker``
items. All of this instances (need to) follow the `collector API`_.
Customizing the collection process in a module
----------------------------------------------
REPEATED WARNING: details of the collection and running process are
still subject to refactorings and thus details will change.
If you are customizing py.test at "Item" level then you
definitely want to be subscribed to the `py-dev mailing list`_
to follow ongoing development.
If you have a module where you want to take responsibility for
collecting your own test Items and possibly even for executing
a test then you can provide `generative tests`_ that yield
callables and possibly arguments as a tuple. This should
serve some immediate purposes like paramtrized tests.
The other extension possibility goes deeper into the machinery
and allows you to specify a custom test ``Item`` class which
is responsible for setting up and executing an underlying
test. [XXX not working: You can integrate your custom ``py.test.Item`` subclass
by binding an ``Item`` name to a test class.] Or you can
extend the collection process for a whole directory tree
by putting Items in a ``conftest.py`` configuration file.
The collection process constantly looks at according names
in the *chain of conftest.py* modules to determine collectors
and items at ``Directory``, ``Module``, ``Class``, ``Function``
or ``Generator`` level. Note that, right now, except for ``Function``
items all classes are pure collectors, i.e. will return a list
of names (possibly empty).
XXX implement doctests as alternatives to ``Function`` items.
Customizing execution of Functions
----------------------------------
- Function test items allow total control of executing their
contained test method. ``function.run()`` will get called by the
session in order to actually run a test. The method is responsible
for performing proper setup/teardown ("Test Fixtures") for a
Function test.
- ``Function.execute(target, *args)`` methods are invoked by
the default ``Function.run()`` to actually execute a python
function with the given (usually empty set of) arguments.
.. _`getting started`: getting-started.html
.. _`py-dev mailing list`: http://codespeak.net/mailman/listinfo/py-dev
Automated Distributed Testing Automated Distributed Testing
================================== ==================================
@ -710,8 +460,10 @@ Sample configuration::
dist_maxwait = 100 dist_maxwait = 100
dist_taskspernode = 10 dist_taskspernode = 10
Running server is done by ``-w`` command line option or ``--startserver`` To use the browser-based reporter (with a nice AJAX interface) you have to tell
(the former might change at some point due to conflicts). ``py.test`` to run a small server locally using the ``-w`` or ``--startserver``
command line options. Afterwards you can point your browser to localhost:8000
to see the progress of the testing.
Development Notes Development Notes
----------------- -----------------