275 lines
9.1 KiB
Plaintext
275 lines
9.1 KiB
Plaintext
The py.execnet library
|
|
======================
|
|
|
|
.. contents::
|
|
.. sectnum::
|
|
|
|
A new view on distributed execution
|
|
-----------------------------------
|
|
|
|
``py.execnet`` supports ad-hoc distribution of parts of
|
|
a program across process and network barriers. *Ad-hoc*
|
|
means that the client side may completely control
|
|
|
|
* which parts of a program execute remotely and
|
|
|
|
* which data protocols are used between them
|
|
|
|
without requiring any prior manual installation
|
|
of user program code on the remote side. In fact,
|
|
not even a prior installation of any server code
|
|
is required, provided there is a way to get
|
|
an input/output connection to a python interpreter
|
|
(for example via "ssh" and a "python" executable).
|
|
|
|
By comparison, traditional Remote Method Based (RMI)
|
|
require prior installation and manual rather
|
|
heavy processes of setup, distribution and
|
|
communication between program parts.
|
|
|
|
|
|
What about Security? Are you completely nuts?
|
|
---------------------------------------------
|
|
|
|
We'll talk about that later :-)
|
|
|
|
Basic Features
|
|
==============
|
|
|
|
With ''py.execnet'' you get the means
|
|
|
|
- to execute python code fragements in remote processes and
|
|
- to interchange data between asynchronously executing code fragments
|
|
|
|
|
|
Available Gateways
|
|
-----------------------------------------
|
|
|
|
You may use one of the following connection methods:
|
|
|
|
* :api:`py.execnet.PopenGateway` a subprocess on the local
|
|
machine. Useful for jailing certain parts of a program
|
|
or for making use of multiple processors.
|
|
|
|
* :api:`py.execnet.SshGateway` a way to connect to
|
|
a remote ssh server and distribute execution to it.
|
|
|
|
* :api:`py.execnet.SocketGateway` a way to connect to
|
|
a remote Socket based server. *Note* that this method
|
|
requires a manually started
|
|
:source:py/execnet/script/socketserver.py
|
|
script. You can run this "server script" without
|
|
having the py lib installed on that remote system.
|
|
|
|
|
|
executing code remotely
|
|
-------------------------------------
|
|
|
|
All gateways offer remote code execution via this high level function:
|
|
|
|
def remote_exec(source):
|
|
"""return channel object for communicating with the asynchronously
|
|
executing 'source' code which will have a corresponding 'channel'
|
|
object in its executing namespace."""
|
|
|
|
With `remote_exec` you send source code to the other
|
|
side and get both a local and a remote Channel_ object,
|
|
which you can use to have the local and remote site
|
|
communicate data in a structured way. Here is
|
|
an example:
|
|
|
|
>>> import py
|
|
>>> gw = py.execnet.PopenGateway()
|
|
>>> channel = gw.remote_exec("""
|
|
... import os
|
|
... channel.send(os.getpid())
|
|
... """)
|
|
>>> remote_pid = channel.receive()
|
|
>>> remote_pid != py.std.os.getpid()
|
|
True
|
|
|
|
.. _`Channel`:
|
|
.. _`channel-api`:
|
|
.. _`exchange data`:
|
|
|
|
Bidirectionally exchange data between hosts
|
|
-------------------------------------------------------------
|
|
|
|
A channel object allows to send and receive data between
|
|
two asynchronously running programs. When calling
|
|
`remote_exec` you will get a channel object back and
|
|
the code fragement running on the other side will
|
|
see a channel object in its global namespace.
|
|
|
|
Here is the interface of channel objects::
|
|
|
|
#
|
|
# API for sending and receiving anonymous values
|
|
#
|
|
channel.send(item):
|
|
sends the given item to the other side of the channel,
|
|
possibly blocking if the sender queue is full.
|
|
Note that items need to be marshallable (all basic
|
|
python types are).
|
|
|
|
channel.receive():
|
|
receives an item that was sent from the other side,
|
|
possibly blocking if there is none.
|
|
Note that exceptions from the other side will be
|
|
reraised as gateway.RemoteError exceptions containing
|
|
a textual representation of the remote traceback.
|
|
|
|
channel.waitclose(timeout=None):
|
|
wait until this channel is closed. Note that a closed
|
|
channel may still hold items that will be received or
|
|
send. Note that exceptions from the other side will be
|
|
reraised as gateway.RemoteError exceptions containing
|
|
a textual representation of the remote traceback.
|
|
|
|
channel.close():
|
|
close this channel on both the local and the remote side.
|
|
A remote side blocking on receive() on this channel
|
|
will get woken up and see an EOFError exception.
|
|
|
|
Instantiating a gateway from a string-specification
|
|
---------------------------------------------------------
|
|
|
|
To specify Gateways with a String::
|
|
|
|
>>> import py
|
|
>>> gwspec = py.execnet.GatewaySpec("popen")
|
|
>>> gw = gwspec.makegateway()
|
|
>>> ex = gw.remote_exec("import sys ; channel.send(sys.executable)").receive()
|
|
>>> assert ex == py.std.sys.executable, (ex, py.std.sys.executable)
|
|
>>>
|
|
|
|
current gateway types and specifications
|
|
+++++++++++++++++++++++++++++++++++++++++++++++
|
|
|
|
+------------------------+-------------------------------------------+
|
|
| ssh host | ssh:host:pythonexecutable:path |
|
|
+------------------------+-------------------------------------------+
|
|
| local subprocess | popen:python_executable:path |
|
|
+------------------------+-------------------------------------------+
|
|
| remote socket process | socket:host:port:python_executable:path |
|
|
+------------------------+-------------------------------------------+
|
|
|
|
examples of valid specifications
|
|
++++++++++++++++++++++++++++++++++++++
|
|
|
|
``ssh:wyvern:python2.4`` signifies a connection to a Python process on the machine reached via "ssh wyvern", current dir will be the 'pyexecnet-cache' subdirectory.
|
|
|
|
``socket:192.168.1.4`` signifies a connection to a Python Socket server process to the given IP on the default port 8888; current dir will be the 'pyexecnet-cache' directory.
|
|
|
|
``popen:python2.5`` signifies a connection to a python2.5 subprocess; current dir will be the current dir of the instantiator.
|
|
|
|
``popen::pytest-cache1`` signifies a connection to a subprocess using ``sys.executable``; current dir will be the `pytest-cache1`.
|
|
|
|
|
|
Examples for execnet usage
|
|
-------------------------------------------
|
|
|
|
Example: compare cwd() of Popen Gateways
|
|
++++++++++++++++++++++++++++++++++++++++
|
|
|
|
A PopenGateway has the same working directory as the instantiatior::
|
|
|
|
>>> import py, os
|
|
>>> gw = py.execnet.PopenGateway()
|
|
>>> ch = gw.remote_exec("import os; channel.send(os.getcwd())")
|
|
>>> res = ch.receive()
|
|
>>> assert res == os.getcwd()
|
|
>>> gw.exit()
|
|
|
|
Example: multichannels
|
|
++++++++++++++++++++++++++++++++++++++++
|
|
|
|
MultiChannels manage 1-n execution and communication:
|
|
|
|
>>> import py
|
|
>>> ch1 = py.execnet.PopenGateway().remote_exec("channel.send(1)")
|
|
>>> ch2 = py.execnet.PopenGateway().remote_exec("channel.send(2)")
|
|
>>> mch = py.execnet.MultiChannel([ch1, ch2])
|
|
>>> l = mch.receive_each()
|
|
>>> assert len(l) == 2
|
|
>>> assert 1 in l
|
|
>>> assert 2 in l
|
|
|
|
MultiGateways help with sending code to multiple remote places:
|
|
|
|
>>> import py
|
|
>>> mgw = py.execnet.MultiGateway([py.execnet.PopenGateway() for x in range(2)])
|
|
>>> mch = mgw.remote_exec("import os; channel.send(os.getcwd())")
|
|
>>> res = mch.receive_each()
|
|
>>> assert res == [os.getcwd()] * 2, res
|
|
>>> mgw.exit()
|
|
|
|
|
|
Example: receiving file contents from remote places
|
|
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
|
problem: retrieving contents of remote files::
|
|
|
|
import py
|
|
# open a gateway to a fresh child process
|
|
gw = py.execnet.SshGateway('codespeak.net')
|
|
channel = gw.remote_exec("""
|
|
for fn in channel:
|
|
f = open(fn, 'rb')
|
|
channel.send(f.read())
|
|
f.close()
|
|
""")
|
|
|
|
for fn in somefilelist:
|
|
channel.send(fn)
|
|
content = channel.receive()
|
|
# process content
|
|
|
|
# later you can exit / close down the gateway
|
|
contentgateway.exit()
|
|
|
|
|
|
A more complicated "nested" Gateway Example
|
|
...........................................
|
|
|
|
The following example opens a PopenGateway, i.e. a python
|
|
child process, starts a socket server within that process and
|
|
then opens a SocketGateway to the freshly started
|
|
socketserver. Thus it forms a "triangle"::
|
|
|
|
|
|
CLIENT < ... > PopenGateway()
|
|
< .
|
|
. .
|
|
. .
|
|
. .
|
|
> SocketGateway()
|
|
|
|
The below "socketserver" mentioned script is a small script that
|
|
basically listens and accepts socket connections, receives one
|
|
liners and executes them.
|
|
|
|
Here are 20 lines of code making the above triangle happen::
|
|
|
|
import py
|
|
port = 7770
|
|
socketserverbootstrap = py.code.Source(
|
|
mypath.dirpath().dirpath('bin', 'socketserver.py').read(),
|
|
"""
|
|
import socket
|
|
sock = bind_and_listen(("localhost", %r))
|
|
channel.send("ok")
|
|
startserver(sock)
|
|
""" % port)
|
|
# open a gateway to a fresh child process
|
|
proxygw = py.execnet.PopenGateway()
|
|
|
|
# execute asynchronously the above socketserverbootstrap on the other
|
|
channel = proxygw.remote_exec(socketserverbootstrap)
|
|
|
|
# the other side should start the socket server now
|
|
assert channel.receive() == "ok"
|
|
gw = py.execnet.SocketGateway('localhost', cls.port)
|
|
print "initialized socket gateway to port", cls.port
|
|
|
|
|