test_ok1/py/doc/execnet.txt

The py.execnet library 
======================

.. contents::
.. sectnum::

A new view on distributed execution
----------------------------------- 

``py.execnet`` supports ad-hoc distribution of parts of
a program across process and network barriers.  *Ad-hoc*
means that the client side may completely control 

* which parts of a program execute remotely and 

* which data protocols are used between them 

without requiring any prior manual installation 
of user program code on the remote side.  In fact,
not even a prior installation of any server code
is required, provided there is a way to get 
an input/output connection to a python interpreter
(for example via "ssh" and a "python" executable). 

By comparison, traditional Remote Method Based (RMI) 
require prior installation and manual rather
heavy processes of setup, distribution and 
communication between program parts.  


What about Security? Are you completely nuts? 
---------------------------------------------

We'll talk about that later :-) 

Basic Features
==============

With ''py.execnet'' you get the means 

- to execute python code fragements in remote processes and 
- to interchange data between asynchronously executing code fragments 


Available Gateways
-----------------------------------------

You may use one of the following connection methods:

* :api:`py.execnet.PopenGateway` a subprocess on the local 
  machine.  Useful for jailing certain parts of a program
  or for making use of multiple processors. 

* :api:`py.execnet.SshGateway` a way to connect to 
  a remote ssh server and distribute execution to it. 

* :api:`py.execnet.SocketGateway` a way to connect to 
  a remote Socket based server. *Note* that this method
  requires a manually started 
  :source:py/execnet/script/socketserver.py
  script.  You can run this "server script" without 
  having the py lib installed on that remote system. 


executing code remotely 
-------------------------------------

All gateways offer remote code execution via this high level function:

    def remote_exec(source): 
        """return channel object for communicating with the asynchronously 
        executing 'source' code which will have a corresponding 'channel' 
        object in its executing namespace."""

With `remote_exec` you send source code to the other
side and get both a local and a remote Channel_ object,
which you can use to have the local and remote site
communicate data in a structured way.   Here is 
an example: 

  >>> import py 
  >>> gw = py.execnet.PopenGateway()
  >>> channel = gw.remote_exec("""
  ...     import os
  ...     channel.send(os.getpid())
  ... """)
  >>> remote_pid = channel.receive()
  >>> remote_pid != py.std.os.getpid()
  True

.. _`Channel`: 
.. _`channel-api`: 
.. _`exchange data`: 

Bidirectionally exchange data between hosts 
-------------------------------------------------------------

A channel object allows to send and receive data between 
two asynchronously running programs.  When calling
`remote_exec` you will get a channel object back and 
the code fragement running on the other side will 
see a channel object in its global namespace. 

Here is the interface of channel objects::

    #
    # API for sending and receiving anonymous values
    #
    channel.send(item): 
        sends the given item to the other side of the channel, 
        possibly blocking if the sender queue is full. 
        Note that items need to be marshallable (all basic 
        python types are).

    channel.receive():
        receives an item that was sent from the other side, 
        possibly blocking if there is none. 
        Note that exceptions from the other side will be 
        reraised as gateway.RemoteError exceptions containing 
        a textual representation of the remote traceback. 

    channel.waitclose(timeout=None): 
        wait until this channel is closed.  Note that a closed
        channel may still hold items that will be received or 
        send. Note that exceptions from the other side will be 
        reraised as gateway.RemoteError exceptions containing 
        a textual representation of the remote traceback. 

    channel.close(): 
        close this channel on both the local and the remote side. 
        A remote side blocking on receive() on this channel 
        will get woken up and see an EOFError exception. 

Instantiating a gateway from a string-specification
---------------------------------------------------------

To specify Gateways with a String::

    >>> import py
    >>> gwspec = py.execnet.GatewaySpec("popen")
    >>> gw = gwspec.makegateway()
    >>> ex = gw.remote_exec("import sys ; channel.send(sys.executable)").receive()
    >>> assert ex == py.std.sys.executable, (ex, py.std.sys.executable)
    >>>

current gateway types and specifications
+++++++++++++++++++++++++++++++++++++++++++++++

+------------------------+-------------------------------------------+
| ssh host               | ssh:host:pythonexecutable:path            |
+------------------------+-------------------------------------------+
| local subprocess       | popen:python_executable:path              |
+------------------------+-------------------------------------------+
| remote socket process  | socket:host:port:python_executable:path   |
+------------------------+-------------------------------------------+

examples of valid specifications
++++++++++++++++++++++++++++++++++++++

``ssh:wyvern:python2.4`` signifies a connection to a Python process on the machine reached via "ssh wyvern", current dir will be the 'pyexecnet-cache' subdirectory. 

``socket:192.168.1.4`` signifies a connection to a Python Socket server process to the given IP on the default port 8888; current dir will be the 'pyexecnet-cache' directory. 

``popen:python2.5`` signifies a connection to a python2.5 subprocess; current dir will be the current dir of the instantiator. 

``popen::pytest-cache1`` signifies a connection to a subprocess using ``sys.executable``; current dir will be the `pytest-cache1`. 


Examples for execnet usage
-------------------------------------------

Example: compare cwd() of Popen Gateways
++++++++++++++++++++++++++++++++++++++++

A PopenGateway has the same working directory as the instantiatior::

    >>> import py, os
    >>> gw = py.execnet.PopenGateway()
    >>> ch = gw.remote_exec("import os; channel.send(os.getcwd())")
    >>> res = ch.receive()
    >>> assert res == os.getcwd()
    >>> gw.exit()

Example: multichannels 
++++++++++++++++++++++++++++++++++++++++

MultiChannels manage 1-n execution and communication:

    >>> import py
    >>> ch1 = py.execnet.PopenGateway().remote_exec("channel.send(1)")
    >>> ch2 = py.execnet.PopenGateway().remote_exec("channel.send(2)")
    >>> mch = py.execnet.MultiChannel([ch1, ch2])
    >>> l = mch.receive_each()
    >>> assert len(l) == 2
    >>> assert 1 in l 
    >>> assert 2 in l 

MultiGateways help with sending code to multiple remote places: 

    >>> import py
    >>> mgw = py.execnet.MultiGateway([py.execnet.PopenGateway() for x in range(2)])
    >>> mch = mgw.remote_exec("import os; channel.send(os.getcwd())")
    >>> res = mch.receive_each()
    >>> assert res == [os.getcwd()] * 2, res
    >>> mgw.exit()


Example: receiving file contents from remote places
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
problem: retrieving contents of remote files::

    import py
    # open a gateway to a fresh child process 
    gw = py.execnet.SshGateway('codespeak.net') 
    channel = gw.remote_exec("""
            for fn in channel:
                f = open(fn, 'rb')
                channel.send(f.read())
                f.close()
    """) 

    for fn in somefilelist: 
        channel.send(fn) 
        content = channel.receive()
        # process content 
     
    # later you can exit / close down the gateway
    contentgateway.exit()


A more complicated "nested" Gateway Example 
........................................... 

The following example opens a PopenGateway, i.e. a python
child process, starts a socket server within that process and
then opens a SocketGateway to the freshly started
socketserver.  Thus it forms a "triangle":: 


    CLIENT < ... >  PopenGateway() 
        <             .
         .            .
          .          .
           .        .
            > SocketGateway() 
            
The below "socketserver" mentioned script is a small script that 
basically listens and accepts socket connections, receives one 
liners and executes them. 

Here are 20 lines of code making the above triangle happen::

    import py 
    port = 7770
    socketserverbootstrap = py.code.Source(
        mypath.dirpath().dirpath('bin', 'socketserver.py').read(),
        """
        import socket
        sock = bind_and_listen(("localhost", %r)) 
        channel.send("ok") 
        startserver(sock)
    """ % port) 
    # open a gateway to a fresh child process
    proxygw = py.execnet.PopenGateway()

    # execute asynchronously the above socketserverbootstrap on the other
    channel = proxygw.remote_exec(socketserverbootstrap)

    # the other side should start the socket server now 
    assert channel.receive() == "ok" 
    gw = py.execnet.SocketGateway('localhost', cls.port)
    print "initialized socket gateway to port", cls.port