207 lines
7.3 KiB
Plaintext
207 lines
7.3 KiB
Plaintext
|
The py.execnet library
|
||
|
|
======================
|
||
|
|
|
||
|
|
.. contents::
|
||
|
|
.. sectnum::
|
||
|
|
|
||
|
|
Execnet deals with letting your python programs execute and
|
||
|
|
communicate across process and computer barriers. At the
|
||
|
|
core it is a very simple and powerful mechanism: executing
|
||
|
|
source code at "the other side" and communicating with
|
||
|
|
remote parts of your program.
|
||
|
|
|
||
|
|
A warning note: We are doing documentation-driven development
|
||
|
|
in some ways. So some of the described features are not
|
||
|
|
there yet. You may refer to the `py API`_ reference for
|
||
|
|
further information.
|
||
|
|
|
||
|
|
The "shpy" way, historical remarks
|
||
|
|
----------------------------------
|
||
|
|
|
||
|
|
Some of you may have seen the pygame-based editor **shpy**
|
||
|
|
that Armin Rigo, Holger Krekel and Michael Hudson demonstrated
|
||
|
|
at EuroPython 2004 during the lightning talks. It is a
|
||
|
|
multiline interactive python environment which allows multiple
|
||
|
|
remote users to have their own cursors and shared interaction
|
||
|
|
with the graphical shell which can execute python code, of
|
||
|
|
course.
|
||
|
|
|
||
|
|
**py.execnet** extracts and refines the basic mechanism that
|
||
|
|
was originally developed from the one-week hack that is the
|
||
|
|
current **shpy**. No, the latter is not released but hopefully
|
||
|
|
Armin, Holger and others get to do a second one-week sprint at
|
||
|
|
some point to be able to release it. If you are interested
|
||
|
|
drop them a note. This is real fun if you are willing to
|
||
|
|
bend your mind a bit.
|
||
|
|
|
||
|
|
A new view on distributed execution
|
||
|
|
-----------------------------------
|
||
|
|
|
||
|
|
**py.execnet** takes the view of **asynchronously executing
|
||
|
|
client-provided code fragments** to help address a core
|
||
|
|
problem of distributed programs. A core feature of
|
||
|
|
**py.execnet** is that **the communication protocols can be
|
||
|
|
defined by the client side**. Usually, with server/client
|
||
|
|
apps and especially RMI systems you often have to upgrade your
|
||
|
|
server if you upgrade your client.
|
||
|
|
|
||
|
|
What about Security? Are you completely nuts?
|
||
|
|
---------------------------------------------
|
||
|
|
|
||
|
|
We'll talk about that later :-)
|
||
|
|
|
||
|
|
Basic Features
|
||
|
|
==============
|
||
|
|
|
||
|
|
With ''py.execnet'' you get the means
|
||
|
|
|
||
|
|
- to navigate through the network with Process, Thread, SSH
|
||
|
|
and Socket- gateways that allow you ...
|
||
|
|
|
||
|
|
- to distribute your program across a network and define
|
||
|
|
communication protocols from the client side, making
|
||
|
|
server maintenance superflous. In fact, there is no such
|
||
|
|
thing as a server. It's just another computer ... if it
|
||
|
|
doesn't run in a kernel-level jail [#]_ in which case
|
||
|
|
even that is virtualized.
|
||
|
|
|
||
|
|
High Level Interface: **remote_exec**
|
||
|
|
-------------------------------------
|
||
|
|
|
||
|
|
These gateways offer one main high level interface::
|
||
|
|
|
||
|
|
def remote_exec(source):
|
||
|
|
"""return channel object for communicating with the asynchronously
|
||
|
|
executing 'source' code which will have a corresponding 'channel'
|
||
|
|
object in its executing namespace."""
|
||
|
|
|
||
|
|
The most important property of execnet gateways is that the
|
||
|
|
protocol they speak with each other ``is determined by the
|
||
|
|
client side``. If you open a gateway on some server in order
|
||
|
|
to coordinate with some other programs you determine your
|
||
|
|
communication protocol. Multiple clients can run their own
|
||
|
|
gateway versions in the same remote process (e.g. connecting
|
||
|
|
through their version of a SocketGateway).
|
||
|
|
|
||
|
|
You should not need to maintain software on the other sides
|
||
|
|
you are running your code at.
|
||
|
|
|
||
|
|
.. _`channel-api`:
|
||
|
|
.. _`exchange data`:
|
||
|
|
|
||
|
|
The **Channel** interface for exchanging data across gateways
|
||
|
|
-------------------------------------------------------------
|
||
|
|
|
||
|
|
While executing custom strings on "the other side" is simple enough
|
||
|
|
it is often tricky to deal with. Therefore we want a way
|
||
|
|
to send data items to and fro between the distributedly running
|
||
|
|
program. The idea is to inject a Channel object for each
|
||
|
|
execution of source code. This Channel object allows two
|
||
|
|
program parts to send data to each other.
|
||
|
|
Here is the current interface::
|
||
|
|
|
||
|
|
#
|
||
|
|
# API for sending and receiving anonymous values
|
||
|
|
#
|
||
|
|
channel.send(item):
|
||
|
|
sends the given item to the other side of the channel,
|
||
|
|
possibly blocking if the sender queue is full.
|
||
|
|
Note that items need to be marshallable (all basic
|
||
|
|
python types are):
|
||
|
|
|
||
|
|
channel.receive():
|
||
|
|
receives an item that was sent from the other side,
|
||
|
|
possibly blocking if there is none.
|
||
|
|
Note that exceptions from the other side will be
|
||
|
|
reraised as gateway.RemoteError exceptions containing
|
||
|
|
a textual representation of the remote traceback.
|
||
|
|
|
||
|
|
channel.waitclose(timeout=None):
|
||
|
|
wait until this channel is closed. Note that a closed
|
||
|
|
channel may still hold items that will be received or
|
||
|
|
send. Note that exceptions from the other side will be
|
||
|
|
reraised as gateway.RemoteError exceptions containing
|
||
|
|
a textual representation of the remote traceback.
|
||
|
|
|
||
|
|
channel.close():
|
||
|
|
close this channel on both the local and the remote side.
|
||
|
|
A remote side blocking on receive() on this channel
|
||
|
|
will get woken up and see an EOFError exception.
|
||
|
|
|
||
|
|
A simple and useful Example for Channels
|
||
|
|
........................................
|
||
|
|
|
||
|
|
problem: retrieving contents of remote files::
|
||
|
|
|
||
|
|
import py
|
||
|
|
contentserverbootstrap = py.code.Source(
|
||
|
|
"""
|
||
|
|
for fn in channel:
|
||
|
|
f = open(fn, 'rb')
|
||
|
|
try:
|
||
|
|
channel.send(f.read())
|
||
|
|
finally:
|
||
|
|
f.close()
|
||
|
|
""")
|
||
|
|
# open a gateway to a fresh child process
|
||
|
|
contentgateway = py.execnet.SshGateway('codespeak.net')
|
||
|
|
channel = contentgateway.remote_exec(contentserverbootstrap)
|
||
|
|
|
||
|
|
for fn in somefilelist:
|
||
|
|
channel.send(fn)
|
||
|
|
content = channel.receive()
|
||
|
|
# process content
|
||
|
|
|
||
|
|
# later you can exit / close down the gateway
|
||
|
|
contentgateway.exit()
|
||
|
|
|
||
|
|
A more complicated "nested" Gateway Example
|
||
|
|
...........................................
|
||
|
|
|
||
|
|
The following example opens a PopenGateway, i.e. a python
|
||
|
|
child process, starts a socket server within that process and
|
||
|
|
then opens a SocketGateway to the freshly started
|
||
|
|
socketserver. Thus it forms a "triangle"::
|
||
|
|
|
||
|
|
|
||
|
|
CLIENT < ... > PopenGateway()
|
||
|
|
< .
|
||
|
|
. .
|
||
|
|
. .
|
||
|
|
. .
|
||
|
|
> SocketGateway()
|
||
|
|
|
||
|
|
The below "socketserver" mentioned script is a small script that
|
||
|
|
basically listens and accepts socket connections, receives one
|
||
|
|
liners and executes them.
|
||
|
|
|
||
|
|
Here are 20 lines of code making the above triangle happen::
|
||
|
|
|
||
|
|
import py
|
||
|
|
port = 7770
|
||
|
|
socketserverbootstrap = py.code.Source(
|
||
|
|
mypath.dirpath().dirpath('bin', 'socketserver.py').read(),
|
||
|
|
"""
|
||
|
|
import socket
|
||
|
|
sock = bind_and_listen(("localhost", %r))
|
||
|
|
channel.send("ok")
|
||
|
|
startserver(sock)
|
||
|
|
""" % port)
|
||
|
|
# open a gateway to a fresh child process
|
||
|
|
proxygw = py.execnet.PopenGateway()
|
||
|
|
|
||
|
|
# execute asynchronously the above socketserverbootstrap on the other
|
||
|
|
channel = proxygw.remote_exec(socketserverbootstrap)
|
||
|
|
|
||
|
|
# the other side should start the socket server now
|
||
|
|
assert channel.receive() == "ok"
|
||
|
|
gw = py.execnet.SocketGateway('localhost', cls.port)
|
||
|
|
print "initialized socket gateway to port", cls.port
|
||
|
|
|
||
|
|
.. _`py API`: api.html
|
||
|
|
|
||
|
|
.. [#] There is an interesting emerging `Jail`_ linux technology
|
||
|
|
as well as a host of others, of course.
|
||
|
|
|
||
|
|
.. _`Jail`: http://books.rsbac.org/unstable/x2223.html
|