test_ok1/py/doc/apigen.txt

apigen - API documentation generation tool
===========================================

What is it?
------------

Apigen is a tool for automatically generating API reference documentation for
Python projects.  It works by examining code at runtime rather than at 
compile time.  This way it is capable of displaying information 
about the code base after initialization.  A drawback is that
you cannot easily document source code that automatically
starts server processes or has some other irreversible effects upon getting imported. 

The apigen functionality can either be used from code, or from
py.test, in the latter case it will gather information about
modules, classes to export explicitely by using provided script. 

Please note that apigen is currently geared towards documenting the
py library itself, making it nicely work for other projects
may still require a bit of adaption and refinement work. 

Using from code
----------------

The library provides a simple API to generate a py.rest.rst tree (which
represents a ReStructuredText document), along with some helper classes to 
control the output. The most important objects are the Tracer, which traces
code execution by registering itself with sys.settrace, the DocStorage class,
that stores Tracer information, and the RestGen class which creates a ReST
tree (see py.rest.rst).

Gathering information
++++++++++++++++++++++

To gather information about documentation, you will first need to tell the tool
what objects it should investigate. Only information for registered objects
will be stored. An example::

  >>> import py
  >>> from py.__.apigen.tracer.docstorage import DocStorage, DocStorageAccessor
  >>> from py.__.apigen.tracer.tracer import Tracer
  >>> toregister = {'py.path.local': py.path.local,
  ...               'py.path.svnwc': py.path.svnwc}
  >>> ds = DocStorage().from_dict(toregister)
  >>> t = Tracer(ds)
  >>> t.start_tracing()
  >>> p = py.path.local('.')
  >>> p.check(dir=True)
  True
  >>> t.end_tracing()

Now the 'ds' variable should contain all kinds of information about both the
py.path.local and the py.path.svnwc class (it will walk through 'toregister' to
find information about all it contains), and things like call stacks, and
possible argument types, etc. as additional information about
py.path.local.check() (since it was called from the traced code).

Viewing information
++++++++++++++++++++

Viewing the information stored in the DocStorage instance isn't very hard
either. As explained there is a RestGen class that creates a py.rest.rst tree,
which can directly be serialized to ReStructuredText, which can in turn be
converted to other formats. Also the py.rest.rst tree can be manipulated
directly, or converted to formats other than ReST (currently only HTML) using
special transformers.

There are several helper classes available that wrap the output format 
generation. There are two types of helper classes, 'LinkWriters' and 'Writers'.
The first are responsible for generating external links (for viewing source),
the second for generating the actual output from the py.rest.rst tree, and
for generating internal links (which is directly related to generating output).
Instances of these classes are passed to the RestGen class as arguments on
instantiation.

An example of creating a directory with seperate ReST files (using DirWriter)
from the 'ds' DocumentStorage instance we created below, without any external
links (using DirectPaste).
::

  >>> from py.__.apigen.rest.genrest import RestGen, DirectPaste, DirWriter
  >>> # create a temp dir in /tmp/pytest-<userid>
  >>> tempdir = py.test.ensuretemp('apigen_example')
  >>> rg = RestGen(DocStorageAccessor(ds), DirectPaste(), DirWriter(tempdir))
  >>> rg.write()

An example of a somewhat more 'real-life' use case, writing to a directory of
HTML files (this uses py.rest.transform), generating links to ViewVC source
views::

  >>> from py.__.apigen.rest.genrest import ViewVC, HTMLDirWriter
  >>> from py.__.apigen.rest.htmlhandlers import HTMLHandler
  >>> tempdir = py.test.ensuretemp('apigen_example_2')
  >>> rg = RestGen(DocStorageAccessor(ds), ViewVC('http://some.host.com/viewvc/myproj/trunk/'),
  ...              HTMLDirWriter(HTMLHandler, HTMLHandler, tempdir))
  >>> rg.write()

Using from py.test
-------------------

Running unit tests forms an ideal opportunity for apigen to find out about what
happens when code is executed (assuming you have proper test coverage ;). There
are hooks built into py.test that allow you to do that:

* Write down a python script which contains at least two functions

  - `get_documentable_items() -> {}` - function which will return dictionary
     of name to object of exported items

  - `build(pkgpath, docstorageaccessor)` - function which will be invoked afterwards
    with DocStorageAccessor instance as an argument (you should read DocStorageAccessor
    interface to know how you can access it)

XXX: Write down some example usage after guido implement the script

Comparison with other documentation generation tools
----------------------------------------------------

Apigen is of course not the only documentation generation tool available for
Python. Although we knew in advance that our tool had certain features the
others do not offer, we decided to investigate a bit so that we could do a
proper comparison.

Tools examined
++++++++++++++

After some 'googling around', it turned out that the amount of documentation
generation tools available was surprisingly low. There were only 5 packages
I could find, of which 1 (called 'HappyDoc') seems dead (last release 2001),
one (called 'Pudge') not yet born (perhaps DOA even? most of the links on the
website are dead), and one (called 'Endo') specific to the Enthought suite.
The remaining two were Epydoc, which is widely used [1]_, and PyDoctor, which is
used only by (and written for) the Twisted project, but can be used seperately.

Epydoc
~~~~~~

http://epydoc.sourceforge.net/

Epydoc is the best known, and most widely used, documentation generation tool
for Python. It builds a documentation tree by inspecting imported modules and
using Python's introspection features. This way it can display information like
containment, inheritance, and docstrings.

The tool is relatively sophisticated, with support for generating HTML and PDF,
choosing different styles (CSS), generating graphs using Graphviz, etc. Also
it allows using markup (which can be ReST, JavaDoc, or their own 'epytext'
format) inside docstrings for displaying rich text in the result.

Quick overview:

  * builds docs from object tree
  * displays relatively little information, just inheritance trees, API and
    docstrings
  * supports some markup (ReST, 'epytext', JavaDoc) in docstrings

PyDoctor
~~~~~~~~

http://codespeak.net/~mwh/pydoctor/

This tool is written by Michael Hudson for the Twisted project. The major
difference between this and Epydoc is that it browses the AST (Abstract Syntax
Tree) instead of using 'live' objects, which means that code that uses special
import mechanisms, or depends on other code that is not available can still be
inspected. On the other hand, code that, for example, puts bound methods into a
module namespace is not documented.

The tool is relatively simple and doesn't support the more advanced features
that Epydoc offers. It was written for Twisted and there are no current plans to
promote its use for unrelated projects.

Quick overview:

  * inspects AST rather than object tree
  * again not a lot of information, the usual API docstrings, class inheritance
    and module structure, but that's it
  * rather heavy dependencies (depends on Twisted/Nevow (trunk version))
  * written for Twisted, but quite nice output with other applications

Quick overview lists of the other tools
+++++++++++++++++++++++++++++++++++++++

HappyDoc
~~~~~~~~

http://happydoc.sourceforge.net/

  * dead
  * inspects AST
  * quite flexible, different output formats (HTML, XML, SGML, PDF)
  * pluggable docstring parsers

Pudge
~~~~~

http://pudge.lesscode.org/

  * immature, dead?
  * builds docs from live object tree (I think?)
  * supports ReST
  * uses Kid templates

Endo
~~~~

https://svn.enthought.com/enthought/wiki/EndoHowTo

  * inspects object tree (I think?)
  * 'traits' aware (see https://svn.enthought.com/enthought/wiki/Traits)
  * customizable HTML output with custom templating engine
  * little documentation, seems like it's written for Enthought's own use
    mostly
  * heavy dependencies

.. [1] Epydoc doesn't seem to be developed anymore, either, but it's so
       widely used it can not be ignored...

Questions, remarks, etc.
-------------------------

For more information, questions, remarks, etc. see http://codespeak.net/py.
This website also contains links to mailing list and IRC channel.

.. _`initpkg`: http://codespeak.net/svn/py/dist/py/initpkg.py
.. _`py.test documentation`: http://codespeak.net/svn/py/dist/py/documentation/test.txt