test_ok2/doc/test/funcargs.txt

==============================================================
**funcargs**: advanced test fixtures and parametrization
==============================================================

.. contents:: 
    :local:
    :depth: 2

what is a "funcarg"? 
=================================================

A *funcarg* is the short name for "test function argument".  Each python test function invocation may receive one or multiple function arguments.  Function argument values can be created next to the test code or in separate test configuration files which allows test functions to remain ignorant of how its base test values are created.  A test function can also be called multiple times with different sets of function arguments, allowing for arbitrary parametrization.  A Funcarg parameter can be any value, a simple number or an application object.  

.. _`contact possibilities`: ../contact.html
.. _`parametrizing tests, generalized`: http://tetamap.wordpress.com/2009/05/13/parametrizing-python-tests-generalized/ 

.. _`blog post about the monkeypatch funcarg`: http://tetamap.wordpress.com/2009/03/03/monkeypatching-in-unit-tests-done-right/
.. _`xUnit style`: xunit_setup.html 


.. _`funcarg factory`:
.. _factory:
 
funcarg factories: creating test function arguments 
==============================================================

Test functions can specify one ore more arguments ("funcargs") 
and a test module or plugin can define factory functions that provide
the function argument.  Let's look at a simple self-contained 
example that you can put into a test module: 

.. sourcecode:: python

    # ./test_simplefactory.py 
    def pytest_funcarg__myfuncarg(request):
        return 42

    def test_function(myfuncarg):
        assert myfuncarg == 17

If you run this with ``py.test test_simplefactory.py`` you see something like this:

.. sourcecode:: python

    =========================== test session starts ============================
    python: platform linux2 -- Python 2.6.2
    test object 1: /home/hpk/hg/py/trunk/example/funcarg/test_simplefactory.py

    test_simplefactory.py F

    ================================ FAILURES ==================================
    ______________________________ test_function _______________________________

    myfuncarg = 42

        def test_function(myfuncarg):
    >       assert myfuncarg == 17
    E       assert 42 == 17

    test_simplefactory.py:6: AssertionError
    ======================== 1 failed in 0.11 seconds ==========================


This means that the test function was called with a ``myfuncarg`` value
of ``42`` and the assert fails accordingly.  Here is how py.test 
calls the test function: 

1. py.test discovers the ``test_function`` because of the ``test_`` prefix. 
   The test function needs a function argument named ``myfuncarg``.  
   A matching factory function is discovered by looking for the 
   name ``pytest_funcarg__myfuncarg``. 

2. ``pytest_funcarg__myfuncarg(request)`` is called and 
   returns the value for ``myfuncarg``.   

3. ``test_function(42)`` call is executed. 

Note that if you misspell a function argument or want
to use one that isn't available, you'll see an error 
with a list of available function arguments. You can 
also issue::

    py.test --funcargs test_simplefactory.py

to see available function arguments (which you can also
think of as "resources"). 

Factory functions receive a `request object`_ 
which they can use to register setup/teardown 
functions or access meta data about a test. 

.. _`request object`: 

funcarg factory request objects
------------------------------------------

Request objects represents a handle on a specific python test function call.  A request object is passed to a funcarg factory and provides access to test configuration and context as well as some `useful caching and finalization helpers`_. Here is a list of attributes: 

``request.function``: python function object requesting the argument

``request.cls``: class object where the test function is defined in or None. 

``request.module``: module object where the test function is defined in. 

``request.config``: access to command line opts and general config

``request.param``: if exists was passed by a previous `metafunc.addcall`_

.. _`useful caching and finalization helpers`: 


registering funcarg related finalizers/cleanup 
----------------------------------------------------

.. sourcecode:: python 

    def addfinalizer(func):
        """ call a finalizer function when test function finishes. """ 
     
Calling ``request.addfinalizer()`` is useful for scheduling teardown 
functions.  Here is an example for providing a ``myfile`` 
object that is to be closed when the execution of a 
test function finishes. 

.. sourcecode:: python

    def pytest_funcarg__myfile(self, request):
        # ... create and open a unique per-function "myfile" object ...
        request.addfinalizer(lambda: myfile.close())
        return myfile


managing fixtures across test modules and test runs
----------------------------------------------------------

.. sourcecode:: python 

    def cached_setup(setup, teardown=None, scope="module", extrakey=None):
        """ cache and return result of calling setup().  

        The requested argument name, the scope and the ``extrakey`` 
        determine the cache key.  The scope also determines when 
        teardown(result) will be called.  valid scopes are: 
        scope == 'function': when the single test function run finishes. 
        scope == 'module': when tests in a different module are run
        scope == 'session': when tests of the session have run. 
        """

Calling ``request.cached_setup()`` helps you to manage fixture 
objects across several scopes.  For example, for creating a Database object 
that is to be setup only once during a test session you can use the helper
like this:

.. sourcecode:: python 
  
    def pytest_funcarg__database(request):
        return request.cached_setup(
                 setup=lambda: Database("..."), 
                 teardown=lambda val: val.close(), 
                 scope="session"
        )


requesting values of other funcargs 
---------------------------------------------

.. sourcecode:: python 

    def getfuncargvalue(name): 
        """ Lookup and call function argument factory for the given name. 
        Each function argument is only created once per function setup. 
        """ 

``request.getfuncargvalue(name)`` calls another funcarg factory function. 
You can use this function if you want to `decorate a funcarg`_, i.e. 
you want to provide the "normal" value but add something
extra.  If a factory cannot be found a ``request.Error`` 
exception will be raised. 

.. _`test generators`: 
.. _`parametrizing-tests`:

generating parametrized tests 
===========================================================

You can parametrize multiple runs of the same test 
function by adding new test function calls with different
function argument values. Let's look at a simple self-contained 
example: 

.. sourcecode:: python

    # ./test_example.py 
    def pytest_generate_tests(metafunc):
        if "numiter" in metafunc.funcargnames: 
            for i in range(10):
                metafunc.addcall(funcargs=dict(numiter=i))

    def test_func(numiter):
        assert numiter < 9

If you run this with ``py.test test_example.py`` you'll get:

.. sourcecode:: python

    ============================= test session starts ==========================
    python: platform linux2 -- Python 2.6.2
    test object 1: /home/hpk/hg/py/trunk/test_example.py

    test_example.py .........F

    ================================ FAILURES ==================================
    __________________________ test_func.test_func[9] __________________________

    numiter = 9

        def test_func(numiter):
    >       assert numiter < 9
    E       assert 9 < 9

    /home/hpk/hg/py/trunk/test_example.py:10: AssertionError


Here is what happens in detail:

1. ``pytest_generate_tests(metafunc)`` hook is called once for each test
   function.  It adds ten new function calls with explicit function arguments. 

2. **execute tests**: ``test_func(numiter)`` is called ten times with 
   ten different arguments. 

.. _`metafunc object`:

test generators and metafunc objects
-------------------------------------------

metafunc objects are passed to the ``pytest_generate_tests`` hook. 
They help to inspect a testfunction and to generate tests 
according to test configuration or values specified 
in the class or module where a test function is defined: 

``metafunc.funcargnames``: set of required function arguments for given function

``metafunc.function``: underlying python test function 

``metafunc.cls``: class object where the test function is defined in or None. 

``metafunc.module``: the module object where the test function is defined in. 

``metafunc.config``: access to command line opts and general config


.. _`metafunc.addcall`: 

the ``metafunc.addcall()`` method 
-----------------------------------------------

.. sourcecode:: python

    def addcall(funcargs={}, id=None, param=None):
        """ trigger a new test function call. """ 

``funcargs`` can be a dictionary of argument names
mapped to values - providing it is called *direct parametrization*. 

If you provide an `id`` it will be used for reporting
and identification purposes.  If you don't supply an `id` 
the stringified counter of the list of added calls will be used. 
``id`` values needs to be unique between all 
invocations for a given test function. 

``param`` if specified will be seen by any 
`funcarg factory`_ as a ``request.param`` attribute. 
Setting it is called *indirect parametrization*.  

Indirect parametrization is preferable if test values are 
expensive to setup or can only be created in certain environments.  
Test generators and thus ``addcall()`` invocations are performed 
during test collection which is separate from the actual test 
setup and test run phase.  With distributed testing collection 
and test setup/run happens in different process.  



.. _`tutorial examples`: 

Tutorial Examples
=======================================

To see how you can implement custom paramtrization schemes, 
see  e.g.  `parametrizing tests, generalized`_ (blog post).

To enable creation of test support code that can flexibly
register setup/teardown functions see the `blog post about 
the monkeypatch funcarg`_. 

If you find issues or have further suggestions for improving
the mechanism you are welcome to checkout `contact possibilities`_ page. 

.. _`application setup tutorial example`: 
.. _appsetup:

application specific test setup and fixtures
---------------------------------------------------------

Here is a basic useful step-wise example for handling application
specific test setup.  The goal is to have one place where we have the 
glue and test support code for bootstrapping and configuring application objects and allow
test modules and test functions to stay ignorant of involved details. 

step 1: use and implement a test/app-specific "mysetup"
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Let's write a simple test function living in a test file
``test_sample.py`` that uses a ``mysetup`` funcarg for accessing test
specific setup. 

.. sourcecode:: python

    # ./test_sample.py
    def test_answer(mysetup): 
        app = mysetup.myapp()
        answer = app.question()
        assert answer == 42

To run this test py.test needs to find and call a factory to 
obtain the required ``mysetup`` function argument.  The test 
function interacts with the provided application specific setup. 

To provide the ``mysetup`` function argument we write down
a factory method in a `local plugin`_ by putting the
following code into a local ``conftest.py``:

.. sourcecode:: python

    # ./conftest.py 
    
    from myapp import MyApp

    def pytest_funcarg__mysetup(request):
        return MySetup()

    class MySetup:
        def myapp(self):
            return MyApp()
    
To run the example we represent our application by putting a pseudo MyApp object into ``myapp.py``:

.. sourcecode:: python

    # ./myapp.py
    class MyApp:
        def question(self):
            return 6 * 9

You can now run the test with ``py.test test_sample.py`` which will
show this failure:

.. sourcecode:: python

    ========================= test session starts =========================
    python: platform linux2 -- Python 2.6.2
    test object 1: /home/hpk/hg/py/trunk/example/funcarg/mysetup

    test_sample.py F

    ============================== FAILURES ===============================
    _____________________________ test_answer _____________________________

    mysetup = <mysetup.conftest.MySetup instance at 0xa020eac>

        def test_answer(mysetup):
            app = mysetup.myapp()
            answer = app.question()
    >       assert answer == 42
    E       assert 54 == 42

    test_sample.py:5: AssertionError
    ====================== 1 failed in 0.11 seconds =======================

This means that our ``mysetup`` object was successfully instantiated, 
we asked it to provide an application instance and checking
its ``question`` method resulted in the wrong answer.  If you are 
confused as to what the concrete question or answers actually mean, 
please see here_ :)  Otherwise proceed to step 2. 

.. _here: http://uncyclopedia.wikia.com/wiki/The_Hitchhiker's_Guide_to_the_Galaxy
.. _`local plugin`: customize.html#local-plugin

.. _`tut-cmdlineoption`:

step 2: adding a command line option 
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

If you provide a "funcarg" from a plugin you can easily make methods
depend on command line options or environment settings.  
To add a command line option we update the conftest.py of
the previous example to add a command line option 
and to offer a new mysetup method:

.. sourcecode:: python

    # ./conftest.py 
    import py
    from myapp import MyApp

    def pytest_funcarg__mysetup(request):
        return MySetup(request)

    def pytest_addoption(parser):
        parser.addoption("--ssh", action="store", default=None,
            help="specify ssh host to run tests with")
        

    class MySetup:
        def __init__(self, request):
            self.config = request.config 

        def myapp(self):
            return MyApp()

        def getsshconnection(self):
            host = self.config.option.ssh
            if host is None:
                py.test.skip("specify ssh host with --ssh")
            return execnet.SshGateway(host)


Now any test function can use the ``mysetup.getsshconnection()`` method like this:

.. sourcecode:: python

    # ./test_ssh.py 
    class TestClass:
        def test_function(self, mysetup):
            conn = mysetup.getsshconnection()
            # work with conn 
      
Running ``py.test test_ssh.py`` without specifying a command line option will result in a skipped test_function: 

.. sourcecode:: python

    ========================= test session starts =========================
    python: platform linux2 -- Python 2.6.2
    test object 1: test_ssh.py

    test_ssh.py s

    ________________________ skipped test summary _________________________
    conftest.py:23: [1] Skipped: 'specify ssh host with --ssh'
    ====================== 1 skipped in 0.11 seconds ======================

Note especially how the test function could stay clear knowing about how to construct test state values or when to skip and with what message. The test function can concentrate on actual test code and test state factories can interact with execution of tests. 

If you specify a command line option like ``py.test --ssh=python.org`` the test will get un-skipped and actually execute. 

.. _`accept example`: 

example: specifying and selecting acceptance tests 
--------------------------------------------------------------

.. sourcecode:: python

    # ./conftest.py 
    def pytest_option(parser):
        group = parser.getgroup("myproject")
        group.addoption("-A", dest="acceptance", action="store_true", 
            help="run (slow) acceptance tests")

    def pytest_funcarg__accept(request):
        return AcceptFuncarg(request)

    class AcceptFuncarg:
        def __init__(self, request):
            if not request.config.option.acceptance:
                py.test.skip("specify -A to run acceptance tests")
            self.tmpdir = request.config.mktemp(request.function.__name__, numbered=True)
            
        def run(self, cmd):
            """ called by test code to execute an acceptance test. """ 
            self.tmpdir.chdir() 
            return py.process.cmdexec(cmd)

   
and the actual test function example:

.. sourcecode:: python

    def test_some_acceptance_aspect(accept):
        accept.tmpdir.mkdir("somesub")
        result = accept.run("ls -la")
        assert "somesub" in result
   
If you run this test without specifying a command line option
the test will get skipped with an appropriate message. Otherwise
you can start to add convenience and test support methods 
to your AcceptFuncarg and drive running of tools or
applications and provide ways to do assertions about
the output. 

.. _`decorate a funcarg`: 

example: decorating a funcarg in a test module
--------------------------------------------------------------

For larger scale setups it's sometimes useful to decorare 
a funcarg just for a particular test module.  We can 
extend the `accept example`_ by putting this in our test module:

.. sourcecode:: python

    def pytest_funcarg__accept(request):
        # call the next factory (living in our conftest.py) 
        arg = request.getfuncargvalue("accept") 
        # create a special layout in our tempdir
        arg.tmpdir.mkdir("special")
        return arg 

    class TestSpecialAcceptance:
        def test_sometest(self, accept):
            assert accept.tmpdir.join("special").check()

Our module level factory will be invoked first and it can 
ask its request object to call the next factory and then 
decorate its result.  This mechanism allows us to stay 
ignorant of how/where the function argument is provided - 
in our example from a `conftest plugin`_. 

sidenote: the temporary directory used here are instances of 
the `py.path.local`_ class which provides many of the os.path 
methods in a convenient way. 

.. _`py.path.local`: ../path.html#local
.. _`conftest plugin`: customize.html#conftestplugin