apigen - API documentation generation tool =========================================== What is it? ------------ Apigen is a tool for automatically generating API reference documentation for Python projects. It works by examining code at runtime rather than at compile time. This way it is capable of displaying information about the code base after initialization. A drawback is that you cannot easily document source code that automatically starts server processes or has some other irreversible effects upon getting imported. The apigen functionality can either be used from code, or from py.test, in the latter case it will gather information about modules, classes to export explicitely by using provided script. Please note that apigen is currently geared towards documenting the py library itself, making it nicely work for other projects may still require a bit of adaption and refinement work. Using from code ---------------- The library provides a simple API to generate a py.rest.rst tree (which represents a ReStructuredText document), along with some helper classes to control the output. The most important objects are the Tracer, which traces code execution by registering itself with sys.settrace, the DocStorage class, that stores Tracer information, and the RestGen class which creates a ReST tree (see py.rest.rst). Gathering information ++++++++++++++++++++++ To gather information about documentation, you will first need to tell the tool what objects it should investigate. Only information for registered objects will be stored. An example:: >>> import py >>> from py.__.apigen.tracer.docstorage import DocStorage, DocStorageAccessor >>> from py.__.apigen.tracer.tracer import Tracer >>> toregister = {'py.path.local': py.path.local, ... 'py.path.svnwc': py.path.svnwc} >>> ds = DocStorage().from_dict(toregister) >>> t = Tracer(ds) >>> t.start_tracing() >>> p = py.path.local('.') >>> p.check(dir=True) True >>> t.end_tracing() Now the 'ds' variable should contain all kinds of information about both the py.path.local and the py.path.svnwc class (it will walk through 'toregister' to find information about all it contains), and things like call stacks, and possible argument types, etc. as additional information about py.path.local.check() (since it was called from the traced code). Viewing information ++++++++++++++++++++ Viewing the information stored in the DocStorage instance isn't very hard either. As explained there is a RestGen class that creates a py.rest.rst tree, which can directly be serialized to ReStructuredText, which can in turn be converted to other formats. Also the py.rest.rst tree can be manipulated directly, or converted to formats other than ReST (currently only HTML) using special transformers. There are several helper classes available that wrap the output format generation. There are two types of helper classes, 'LinkWriters' and 'Writers'. The first are responsible for generating external links (for viewing source), the second for generating the actual output from the py.rest.rst tree, and for generating internal links (which is directly related to generating output). Instances of these classes are passed to the RestGen class as arguments on instantiation. An example of creating a directory with seperate ReST files (using DirWriter) from the 'ds' DocumentStorage instance we created below, without any external links (using DirectPaste). :: >>> from py.__.apigen.rest.genrest import RestGen, DirectPaste, DirWriter >>> # create a temp dir in /tmp/pytest- >>> tempdir = py.test.ensuretemp('apigen_example') >>> rg = RestGen(DocStorageAccessor(ds), DirectPaste(), DirWriter(tempdir)) >>> rg.write() An example of a somewhat more 'real-life' use case, writing to a directory of HTML files (this uses py.rest.transform), generating links to ViewVC source views:: >>> from py.__.apigen.rest.genrest import ViewVC, HTMLDirWriter >>> from py.__.apigen.rest.htmlhandlers import HTMLHandler >>> tempdir = py.test.ensuretemp('apigen_example_2') >>> rg = RestGen(DocStorageAccessor(ds), ViewVC('http://some.host.com/viewvc/myproj/trunk/'), ... HTMLDirWriter(HTMLHandler, HTMLHandler, tempdir)) >>> rg.write() Using from py.test ------------------- Running unit tests forms an ideal opportunity for apigen to find out about what happens when code is executed (assuming you have proper test coverage ;). There are hooks built into py.test that allow you to do that: * Write down a python script which contains at least two functions - `get_documentable_items() -> {}` - function which will return dictionary of name to object of exported items - `build(pkgpath, docstorageaccessor)` - function which will be invoked afterwards with DocStorageAccessor instance as an argument (you should read DocStorageAccessor interface to know how you can access it) XXX: Write down some example usage after guido implement the script Comparison with other documentation generation tools ---------------------------------------------------- Apigen is of course not the only documentation generation tool available for Python. Although we knew in advance that our tool had certain features the others do not offer, we decided to investigate a bit so that we could do a proper comparison. Tools examined ++++++++++++++ After some 'googling around', it turned out that the amount of documentation generation tools available was surprisingly low. There were only 5 packages I could find, of which 1 (called 'HappyDoc') seems dead (last release 2001), one (called 'Pudge') not yet born (perhaps DOA even? most of the links on the website are dead), and one (called 'Endo') specific to the Enthought suite. The remaining two were Epydoc, which is widely used [1]_, and PyDoctor, which is used only by (and written for) the Twisted project, but can be used seperately. Epydoc ~~~~~~ http://epydoc.sourceforge.net/ Epydoc is the best known, and most widely used, documentation generation tool for Python. It builds a documentation tree by inspecting imported modules and using Python's introspection features. This way it can display information like containment, inheritance, and docstrings. The tool is relatively sophisticated, with support for generating HTML and PDF, choosing different styles (CSS), generating graphs using Graphviz, etc. Also it allows using markup (which can be ReST, JavaDoc, or their own 'epytext' format) inside docstrings for displaying rich text in the result. Quick overview: * builds docs from object tree * displays relatively little information, just inheritance trees, API and docstrings * supports some markup (ReST, 'epytext', JavaDoc) in docstrings PyDoctor ~~~~~~~~ http://codespeak.net/~mwh/pydoctor/ This tool is written by Michael Hudson for the Twisted project. The major difference between this and Epydoc is that it browses the AST (Abstract Syntax Tree) instead of using 'live' objects, which means that code that uses special import mechanisms, or depends on other code that is not available can still be inspected. On the other hand, code that, for example, puts bound methods into a module namespace is not documented. The tool is relatively simple and doesn't support the more advanced features that Epydoc offers. It was written for Twisted and there are no current plans to promote its use for unrelated projects. Quick overview: * inspects AST rather than object tree * again not a lot of information, the usual API docstrings, class inheritance and module structure, but that's it * rather heavy dependencies (depends on Twisted/Nevow (trunk version)) * written for Twisted, but quite nice output with other applications Quick overview lists of the other tools +++++++++++++++++++++++++++++++++++++++ HappyDoc ~~~~~~~~ http://happydoc.sourceforge.net/ * dead * inspects AST * quite flexible, different output formats (HTML, XML, SGML, PDF) * pluggable docstring parsers Pudge ~~~~~ http://pudge.lesscode.org/ * immature, dead? * builds docs from live object tree (I think?) * supports ReST * uses Kid templates Endo ~~~~ https://svn.enthought.com/enthought/wiki/EndoHowTo * inspects object tree (I think?) * 'traits' aware (see https://svn.enthought.com/enthought/wiki/Traits) * customizable HTML output with custom templating engine * little documentation, seems like it's written for Enthought's own use mostly * heavy dependencies .. [1] Epydoc doesn't seem to be developed anymore, either, but it's so widely used it can not be ignored... Questions, remarks, etc. ------------------------- For more information, questions, remarks, etc. see http://codespeak.net/py. This website also contains links to mailing list and IRC channel. .. _`initpkg`: http://codespeak.net/svn/py/dist/py/initpkg.py .. _`py.test documentation`: http://codespeak.net/svn/py/dist/py/documentation/test.txt