265 lines
9.2 KiB
Plaintext
265 lines
9.2 KiB
Plaintext
==============================================================================
|
|
py.execnet: *elastic* distributed programming
|
|
==============================================================================
|
|
|
|
``execnet`` helps you to:
|
|
|
|
* ad-hoc instantiate local or remote Python Processes
|
|
* send code for execution in one or many processes
|
|
* send and receive data between processes through channels
|
|
|
|
One of it's unique features is that it uses a **zero-install**
|
|
technique: no manual installation steps are required on
|
|
remote places, only a basic working Python interpreter
|
|
and some input/output connection to it.
|
|
|
|
There is a `EuroPython2009 talk`_ from July 2009 with
|
|
examples and some pictures.
|
|
|
|
.. contents::
|
|
:local:
|
|
:depth: 2
|
|
|
|
.. _`EuroPython2009 talk`: http://codespeak.net/download/py/ep2009-execnet.pdf
|
|
|
|
Gateways: immediately spawn local or remote process
|
|
===================================================
|
|
|
|
In order to send code to a remote place or a subprocess
|
|
you need to instantiate a so-called Gateway object.
|
|
There are currently three Gateway classes:
|
|
|
|
* :api:`py.execnet.PopenGateway` to open a subprocess
|
|
on the local machine. Useful for making use
|
|
of multiple processors to to contain code execution
|
|
in a separated environment.
|
|
|
|
* :api:`py.execnet.SshGateway` to connect to
|
|
a remote ssh server and distribute execution to it.
|
|
|
|
* :api:`py.execnet.SocketGateway` a way to connect to
|
|
a remote Socket based server. *Note* that this method
|
|
requires a manually started
|
|
:source:py/execnet/script/socketserver.py
|
|
script. You can run this "server script" without
|
|
having the py lib installed on the remote system
|
|
and you can setup it up as permanent service.
|
|
|
|
|
|
remote_exec: execute source code remotely
|
|
===================================================
|
|
|
|
All gateways offer remote code execution via this high level function::
|
|
|
|
def remote_exec(source):
|
|
"""return channel object for communicating with the asynchronously
|
|
executing 'source' code which will have a corresponding 'channel'
|
|
object in its executing namespace."""
|
|
|
|
With `remote_exec` you send source code to the other
|
|
side and get both a local and a remote Channel_ object,
|
|
which you can use to have the local and remote site
|
|
communicate data in a structured way. Here is
|
|
an example for reading the PID::
|
|
|
|
>>> import py
|
|
>>> gw = py.execnet.PopenGateway()
|
|
>>> channel = gw.remote_exec("""
|
|
... import os
|
|
... channel.send(os.getpid())
|
|
... """)
|
|
>>> remote_pid = channel.receive()
|
|
>>> remote_pid != py.std.os.getpid()
|
|
True
|
|
|
|
.. _`Channel`:
|
|
.. _`channel-api`:
|
|
.. _`exchange data`:
|
|
|
|
Channels: bidirectionally exchange data between hosts
|
|
=======================================================
|
|
|
|
A channel object allows to send and receive data between
|
|
two asynchronously running programs. When calling
|
|
`remote_exec` you will get a channel object back and
|
|
the code fragment running on the other side will
|
|
see a channel object in its global namespace.
|
|
|
|
Here is the interface of channel objects::
|
|
|
|
#
|
|
# API for sending and receiving anonymous values
|
|
#
|
|
channel.send(item):
|
|
sends the given item to the other side of the channel,
|
|
possibly blocking if the sender queue is full.
|
|
Note that items need to be marshallable (all basic
|
|
python types are).
|
|
|
|
channel.receive():
|
|
receives an item that was sent from the other side,
|
|
possibly blocking if there is none.
|
|
Note that exceptions from the other side will be
|
|
reraised as gateway.RemoteError exceptions containing
|
|
a textual representation of the remote traceback.
|
|
|
|
channel.waitclose(timeout=None):
|
|
wait until this channel is closed. Note that a closed
|
|
channel may still hold items that will be received or
|
|
send. Note that exceptions from the other side will be
|
|
reraised as gateway.RemoteError exceptions containing
|
|
a textual representation of the remote traceback.
|
|
|
|
channel.close():
|
|
close this channel on both the local and the remote side.
|
|
A remote side blocking on receive() on this channel
|
|
will get woken up and see an EOFError exception.
|
|
|
|
|
|
.. _xspec:
|
|
|
|
|
|
XSpec: string specification for gateway type and configuration
|
|
===============================================================
|
|
|
|
``py.execnet`` supports a simple extensible format for
|
|
specifying and configuring Gateways for remote execution.
|
|
You can use a string specification to instantiate a new gateway,
|
|
for example a new SshGateway::
|
|
|
|
gateway = py.execnet.makegateway("ssh=myhost")
|
|
|
|
Let's look at some examples for valid specifications.
|
|
Specification for an ssh connection to `wyvern`, running on python2.4 in the (newly created) 'mycache' subdirectory::
|
|
|
|
ssh=wyvern//python=python2.4//chdir=mycache
|
|
|
|
Specification of a python2.5 subprocess; with a low CPU priority ("nice" level). Current dir will be the current dir of the instantiator (that's true for all 'popen' specifications unless they specify 'chdir')::
|
|
|
|
popen//python=2.5//nice=20
|
|
|
|
Specification of a Python Socket server process that listens on 192.168.1.4:8888; current dir will be the 'pyexecnet-cache' sub directory which is used a default for all remote processes::
|
|
|
|
socket=192.168.1.4:8888
|
|
|
|
More generally, a specification string has this general format::
|
|
|
|
key1=value1//key2=value2//key3=value3
|
|
|
|
If you omit a value, a boolean true value is assumed. Currently
|
|
the following key/values are supported:
|
|
|
|
* ``popen`` for a PopenGateway
|
|
* ``ssh=host`` for a SshGateway
|
|
* ``socket=address:port`` for a SocketGateway
|
|
* ``python=executable`` for specifying Python executables
|
|
* ``chdir=path`` change remote working dir to given relative or absolute path
|
|
* ``nice=value`` decrease remote nice level if platforms supports it
|
|
|
|
|
|
Examples of py.execnet usage
|
|
===============================================================
|
|
|
|
Compare cwd() of Popen Gateways
|
|
----------------------------------------
|
|
|
|
A PopenGateway has the same working directory as the instantiatior::
|
|
|
|
>>> import py, os
|
|
>>> gw = py.execnet.PopenGateway()
|
|
>>> ch = gw.remote_exec("import os; channel.send(os.getcwd())")
|
|
>>> res = ch.receive()
|
|
>>> assert res == os.getcwd()
|
|
>>> gw.exit()
|
|
|
|
Synchronously receive results from two sub processes
|
|
-----------------------------------------------------
|
|
|
|
Use MultiChannels for receiving multiple results from remote code::
|
|
|
|
>>> import py
|
|
>>> ch1 = py.execnet.PopenGateway().remote_exec("channel.send(1)")
|
|
>>> ch2 = py.execnet.PopenGateway().remote_exec("channel.send(2)")
|
|
>>> mch = py.execnet.MultiChannel([ch1, ch2])
|
|
>>> l = mch.receive_each()
|
|
>>> assert len(l) == 2
|
|
>>> assert 1 in l
|
|
>>> assert 2 in l
|
|
|
|
Asynchronously receive results from two sub processes
|
|
-----------------------------------------------------
|
|
|
|
Use ``MultiChannel.make_receive_queue()`` for asynchronously receiving
|
|
multiple results from remote code. This standard Queue provides
|
|
``(channel, result)`` tuples which allows to determine where
|
|
a result comes from::
|
|
|
|
>>> import py
|
|
>>> ch1 = py.execnet.PopenGateway().remote_exec("channel.send(1)")
|
|
>>> ch2 = py.execnet.PopenGateway().remote_exec("channel.send(2)")
|
|
>>> mch = py.execnet.MultiChannel([ch1, ch2])
|
|
>>> queue = mch.make_receive_queue()
|
|
>>> chan1, res1 = queue.get() # you may also specify a timeout
|
|
>>> chan2, res2 = queue.get()
|
|
>>> res1 + res2
|
|
3
|
|
>>> assert chan1 in (ch1, ch2)
|
|
>>> assert chan2 in (ch1, ch2)
|
|
>>> assert chan1 != chan2
|
|
|
|
Receive file contents from remote SSH account
|
|
-----------------------------------------------------
|
|
|
|
Here is a small program that you can use to retrieve
|
|
contents of remote files::
|
|
|
|
import py
|
|
# open a gateway to a fresh child process
|
|
gw = py.execnet.SshGateway('codespeak.net')
|
|
channel = gw.remote_exec("""
|
|
for fn in channel:
|
|
f = open(fn, 'rb')
|
|
channel.send(f.read())
|
|
f.close()
|
|
""")
|
|
|
|
for fn in somefilelist:
|
|
channel.send(fn)
|
|
content = channel.receive()
|
|
# process content
|
|
|
|
# later you can exit / close down the gateway
|
|
gw.exit()
|
|
|
|
|
|
Instantiate a socket server in a new subprocess
|
|
-----------------------------------------------------
|
|
|
|
The following example opens a PopenGateway, i.e. a python
|
|
child process, and starts a socket server within that process
|
|
and then opens a second gateway to the freshly started
|
|
socketserver::
|
|
|
|
import py
|
|
|
|
popengw = py.execnet.PopenGateway()
|
|
socketgw = py.execnet.SocketGateway.new_remote(popengw, ("127.0.0.1", 0))
|
|
|
|
print socketgw._rinfo() # print some info about the remote environment
|
|
|
|
|
|
Sending a module / checking if run through remote_exec
|
|
--------------------------------------------------------------
|
|
|
|
You can pass a module object to ``remote_exec`` in which case
|
|
its source code will be sent. No dependencies will be transferred
|
|
so the module must be self-contained or only use modules that are
|
|
installed on the "other" side. Module code can detect if it is
|
|
running in a remote_exec situation by checking for the special
|
|
``__name__`` attribute like this::
|
|
|
|
if __name__ == '__channelexec__':
|
|
# ... call module functions ...
|
|
|
|
|