201 lines
6.2 KiB
Plaintext
201 lines
6.2 KiB
Plaintext
|
.. include:: <s5defs.txt>
|
||
|
|
|
||
|
|
=================================================
|
||
|
|
py.test - flexible and powerful automated testing
|
||
|
|
=================================================
|
||
|
|
|
||
|
|
:Authors: Holger Krekel, merlinux GmbH
|
||
|
|
:Date: 13th March 2006
|
||
|
|
|
||
|
|
Intro: Benefits of Automated Testing
|
||
|
|
======================================
|
||
|
|
|
||
|
|
- prove that code changes actually fix a certain issue
|
||
|
|
- minimizing Time to Feedback for developers
|
||
|
|
- reducing overall Time to Market
|
||
|
|
- document usage of plugins
|
||
|
|
- tests as a means of communication
|
||
|
|
- easing entry for newcomers
|
||
|
|
|
||
|
|
py.test Purposes & Goals
|
||
|
|
===============================
|
||
|
|
|
||
|
|
- automated cross-project open source testing tool
|
||
|
|
- flexible per-project customization
|
||
|
|
- reusing test methods/reporting across projects
|
||
|
|
- various iterative test collection methods
|
||
|
|
- support for distributed testing
|
||
|
|
- py lib is a development support library
|
||
|
|
|
||
|
|
What is Python?
|
||
|
|
===============================
|
||
|
|
|
||
|
|
- easy-to-learn flexible OO high level language
|
||
|
|
- glue-language for connecting C++, Java and scripting
|
||
|
|
- used e.g. by Google for deployment/testing/implementation
|
||
|
|
- used by BIND (mainstream DNS internet server) for testing
|
||
|
|
- Jython provides Python for JVM
|
||
|
|
- IronPython provides Python for .NET
|
||
|
|
- CPython is mainstream C-based platform
|
||
|
|
- PyPy - Python in Python implementation
|
||
|
|
|
||
|
|
Main drivers of py.test development
|
||
|
|
======================================
|
||
|
|
|
||
|
|
- PyPy project testing needs (part of EU project)
|
||
|
|
- needs by individual (freely contributing) projects
|
||
|
|
- at least 20 projects using py.test and py lib
|
||
|
|
- higher level innovation plans by merlinux & experts
|
||
|
|
- commercial needs
|
||
|
|
- almost three years of (non-fulltime) development
|
||
|
|
|
||
|
|
Authors & copyrights
|
||
|
|
==============================================
|
||
|
|
|
||
|
|
- initial: Holger Krekel, Armin Rigo
|
||
|
|
- major contributors: Jan Balster, Brian Dorsey, Grig
|
||
|
|
Gheorghiu
|
||
|
|
- many others with small patches
|
||
|
|
- MIT license
|
||
|
|
|
||
|
|
who is merlinux?
|
||
|
|
===============================
|
||
|
|
|
||
|
|
- small company founded in 2004 by Holger Krekel and Laura
|
||
|
|
Creighton
|
||
|
|
|
||
|
|
- purpose: research and development / open source technologies
|
||
|
|
|
||
|
|
- 7 employees (no win32 experts!), 6 freelancers
|
||
|
|
|
||
|
|
- three larger projects:
|
||
|
|
|
||
|
|
- PyPy - next generation Python implementation
|
||
|
|
- mailwitness - digital invoicing/signatures
|
||
|
|
- provider of development servers
|
||
|
|
|
||
|
|
- technologies: virtualization, deployment and testing
|
||
|
|
|
||
|
|
Main Features of py.test
|
||
|
|
===============================
|
||
|
|
|
||
|
|
- simplest possible ``assert`` approach
|
||
|
|
- clean setup/teardown semantics
|
||
|
|
- stdout/stderr capturing per test
|
||
|
|
- per-project/directory cmdline options (many predefined)
|
||
|
|
- test selection support
|
||
|
|
- customizable auto-collection of tests
|
||
|
|
- `more features`_ ...
|
||
|
|
|
||
|
|
.. _`more features`: ../test.html#features
|
||
|
|
|
||
|
|
Main User-Level entry points
|
||
|
|
===============================
|
||
|
|
|
||
|
|
- ``py.test.raises(Exc, func, *args, **kwargs)``
|
||
|
|
- ``py.test.fail(msg)`` -> fail a test
|
||
|
|
- ``py.test.skip(msg)`` -> skip a test
|
||
|
|
- ``py.test.ensuretemp(prefix)`` -> per-test session temporary directory
|
||
|
|
- ``conftest.py`` can modify almost arbitrary testing aspects
|
||
|
|
(but it's a bit involved)
|
||
|
|
|
||
|
|
some py lib components
|
||
|
|
===============================
|
||
|
|
|
||
|
|
- ``py.execnet`` provides ad-hoc means to distribute programs
|
||
|
|
- ``py.path`` objects abstract local and svn files
|
||
|
|
- ``py.log`` offers (preliminary) logging support
|
||
|
|
- ``py.xml.html`` for programmatic html generation
|
||
|
|
- lazy import ...``import py`` is enough
|
||
|
|
|
||
|
|
py.test Implementation
|
||
|
|
===============================
|
||
|
|
|
||
|
|
- `basic picture`_
|
||
|
|
- Session objects (Terminal and Tcl-GUI)
|
||
|
|
- reporting hooks are on session objects
|
||
|
|
- Collector hierarchy yield iteratively tests
|
||
|
|
- uses py lib extensively (py.path/py.execnet)
|
||
|
|
- "conftest.py" per-directory configuration mechanism
|
||
|
|
|
||
|
|
.. _`basic picture`: ../test.html#basicpicture
|
||
|
|
|
||
|
|
Session objects
|
||
|
|
===============================
|
||
|
|
|
||
|
|
- responsible for driving the testing process
|
||
|
|
- make use of iterative Collector hierarchies
|
||
|
|
- responsible for reporting (XXX)
|
||
|
|
- can be split to a Frontend and BackendSession
|
||
|
|
for distributed testing (GUI frontend uses it)
|
||
|
|
|
||
|
|
Collector objects
|
||
|
|
===============================
|
||
|
|
|
||
|
|
- Collectors / Test Items form a tree
|
||
|
|
- the tree is build iteratively (driven from Sessions)
|
||
|
|
- collector tree can be viewed with ``--collectonly``
|
||
|
|
- ``run()`` returns list of (test) names or runs the test
|
||
|
|
- ``join(name)`` returns a sub collector/item
|
||
|
|
- various helper methods to e.g. determine file/location
|
||
|
|
|
||
|
|
Extensions: ReST documentation checking
|
||
|
|
=========================================
|
||
|
|
|
||
|
|
- `py/documentation/conftest.py`_ provides test
|
||
|
|
items for checking documentation and link integrity
|
||
|
|
|
||
|
|
- uses its own collector/testitem hierarchy
|
||
|
|
|
||
|
|
- invokes ``docutils`` processing, reports errors
|
||
|
|
|
||
|
|
.. _`py/documentation/conftest.py`: ../conftest.py
|
||
|
|
|
||
|
|
Extensions: Distributed Testing
|
||
|
|
==============================================
|
||
|
|
|
||
|
|
- using py.execnet to dispatch on different python versions
|
||
|
|
- using py.execnet to dispatch tests on other hosts/platforms
|
||
|
|
- currently: Popen, SSH and Socket gateways
|
||
|
|
- missing support pushing tests to "the other side"
|
||
|
|
- missing for deployment on multiple machines
|
||
|
|
- but it's already possible ...
|
||
|
|
|
||
|
|
Example using pywinauto from linux
|
||
|
|
==============================================
|
||
|
|
|
||
|
|
- start socketserver.py on windows
|
||
|
|
- connect a SocketGateway e.g. from linux
|
||
|
|
- send tests, execute and report tracebacks through the
|
||
|
|
gateway
|
||
|
|
- remotely use pywinauto to automate testing of GUI work flow
|
||
|
|
- interactive example ...
|
||
|
|
|
||
|
|
Status of py lib
|
||
|
|
===============================
|
||
|
|
|
||
|
|
- mostly developed on linux/OSX
|
||
|
|
- basically all tests pass on win32 as well
|
||
|
|
- but missing some win32 convenience
|
||
|
|
- some support for generation of html/ReST/PDFs reports
|
||
|
|
- py.execnet works rather reliably (pending deeper win32 testing)
|
||
|
|
- flexible configuration but sometimes non-obvious/documented
|
||
|
|
(requires understanding of internals)
|
||
|
|
|
||
|
|
Next Steps py lib / py.test
|
||
|
|
===============================
|
||
|
|
|
||
|
|
- refined py.execnet distribution of programs
|
||
|
|
- more configurable and customizable reporting
|
||
|
|
- implement support for testing distribution
|
||
|
|
- explore refined win32 support
|
||
|
|
- automated collection of unittest.py based tests
|
||
|
|
- make spawning processes/gateways more robust
|
||
|
|
- doctest support
|
||
|
|
- unify logging approaches (py.log.*)
|
||
|
|
- ...
|
||
|
|
|
||
|
|
|
||
|
|
.. |bullet| unicode:: U+02022
|
||
|
|
.. footer:: Holger Krekel (merlinux) |bullet| |bullet| 13th March 2006
|