I'm using Twisted's Perspective Broker for RMI between a process and subprocess.
Rather than listen on a TCP socket (such as by passing reactor.listenTCP() an instance of PBServerFactory) and have the subprocess connect to it, I'd prefer to use the subprocess's stdin and stdout.
I've found twisted.internet.stdio.StandardIO, but if that's the way to go, I'm not sure how to set everything up.
Is it feasible to use PB over stdio instead of TCP? How?
Wait, why?
The subprocess is for running untrusted code. It's sandboxed, but needs to be able to communicate back with the parent process in limited ways. Some form of RMI is by far the cleanest option for the specific use case, and PB has an access model that looks right. But the sandboxed process doesn't have -- and shouldn't need -- network access. That RMI is its only communication with the outside world, and piping it through stdin/stdout seems like a clean way of doing business.
But if I'm not going about this the right way, that's a perfectly valid answer too.
Using a protocol like PB between a parent and child process over a stdio-like connection has two pieces. One piece is in the child process, using file descriptors 0 and 1 to communicate with the parent. The other piece is the parent process, using whatever file descriptors correspond to the child's 0 and 1.
StandardIO is the first piece. You still need the second piece - that's IReactorProcess.spawnProcess.
However, the newer endpoints APIs are a better way to access this functionality.
The basics of endpoints are that a client endpoint lets you connect to a server without caring exactly how that connection is established and a server endpoint lets you accept connections from clients without caring exactly how those clients are connecting.
There is a child process client endpoint and a stdio server endpoint. This means you can write your client something like:
factory = PBClientFactory(...)
d = factory.getRootObject()
...
clientEndpoint.connect(factory)
And your server something like:
factory = PBServerFactory(...)
...
serverEndpoint.listen(factory)
And you now have a client and server that will talk to each other, but you haven't actually specified how they talk to each other yet. Perhaps it's TCP or perhaps it's stdio.
Then all you need is to pick the right endpoints to use. To stick with your idea of communicating over stdio:
clientEndpoint = ProcessEndpoint(reactor, "/path/to/child", ("argv",), ...)
serverEndpoint = StandardIOEndpoint(reactor)
If you change your mind later, then switching to - say - TCP is as easy as:
clientEndpoint = TCP4ClientEndpoint(reactor, "1.2.3.4", 12345)
serverEndpoint = TCP4ServerEndpoint(reactor, 12345)
Or you can use the plugin mechanism for string descriptions of endpoints to turn this into configuration instead:
clientEndpoint = clientFromString(reactor, options["client-endpoint"])
serverEndpoint = serverFromString(reactor, options["server-endpoint"])
Where options["client-endpoint"] and options["server-endpoint"] are strings like "tcp:host=1.2.3.4:port=12345" and "tcp:port=12345".
For more, see the complete endpoints howto.
Related
I would like to have a server process (preferably Python) that accepts simple messages and multiple clients (again, preferably Python) that connect to the server and send messages to it. The server and clients will only ever be running on the same local machine and the OS is Linux based. The server will be automatically started by the OS and the clients started later independent of the server. I strongly want to avoid installing a whole separate messaging framework/server to do this. The messages will be simple strings such as "kick" or even just a single byte representing the message type. It also needs to know when a connection is made and lost.
From these requirements, I think named pipes would be a feasible solution, with a new instance of that pipe created for each client connection. However, when I search for examples, all of the ones I have come across deal with processes that are spawned from the same parent process and not independently started which means they can pass a parent reference to the child.
Windows seems to allow multiple instances of a named pipe (one for each client connection), but I'm unsure if this is possible on a Linux based OS?
Please could someone point me in the right direction, preferably with a basic example, even if it's just pseudo-code.
I've looked at the multiprocessing module in Python, but this seems to be oriented around the server and client sharing the same process or having one spawn the other.
Edit
May be important, the host device is not guaranteed to have networking capabilities (embedded device).
I've used zeromq for this sort of thing before. it's a relatively lightweight library that exposes this sort of functionality
otherwise, you could implement it yourself by binding a socket in the server process and having clients connect to it. this works fine for unix domain sockets, just pass AF_UNIX when creating the socket, e.g:
import socket
with socket.socket(socket.AF_UNIX) as s:
s.bind('/tmp/srv')
s.listen(1)
(c, addr) = s.accept()
with c:
c.send(b"hello world")
for the server, and:
with socket.socket(socket.AF_UNIX) as c:
c.connect('/tmp/srv')
print(c.recv(8192))
for the client.
writing a protocol around this is more involved, which is where things like zmq really help where you can easily push JSON messages around
I am using ZMQ to facilitate communications between one server and multiple clients. Is there a method to have the clients automatically find the ZMQ server if they are on the same internal network? My goal would be to have the client be able to automatically detect the IP and Port it should connect to.
It's not possible to do this in any sort of scalable way without some sort of broker or manager that will manage your communications system.
The way that would work is that you have your broker on a known IP:port, and as your server and clients spin up, they connect to the broker, and the broker then tells your endpoints how to communicate to each other.
There are some circumstances where such a communication pattern could make sense, generally when the server and clients are controlled by different entities, and maybe even different from the entity controlling the broker. In your case, it sounds like a dramatic amount of over-engineering. The only other way to do what you're looking for that I'm aware of is to just start brute forcing the network to find open IP:port combinations that respond the way you are looking for. Ick.
I suggest you just define the IP:port you want to use, probably through some method of static configuration you can change manually as necessary, or that can act as sort of a flat-file broker that both ends of the communication can access.
I have a Python program which spawns several other Python programs as subprocesses. One of these subprocesses is supposed to open and bind a ZMQ publisher socket, such that other subprocesses can subscribe to it.
I cannot give guarantees about which tcp ports will be available, so when I bind to a random port in the subprocess, my main program will not know what to tell the other subprocesses.
Is there a way to bind the socket in the main process and then somehow pass the socket to my subprocess? Or either some other way to preregister the socket or a standard way to pass the port information from the subprocess back to my main process (stdout and stderr are already used by other data)?
Just checking for a free port in the main process and passing that to the subprocess is not really optimal, because this could still fail if the socket is being assigned in the meantime. Also, since my program should work on Unix and Windows, I cannot really use ipc sockets, which would otherwise solve my problem.
The simplest is to create a logic for a pool-of-ports manager ( rather avoid attempts to share / pass ZeroMQ sockets to / among other processes )
One may create a persistent, a-priori known, tcp://A.B.C.D:8765-transport-class based .bind() access-point, exposed to all client processes as a port-assignment service, to which client processes .connect(), handshake in whatever manner is needed to proof an identity/credentials/purpose/etc and .recv() in a coordinated manner one actually free messaging/signalling-service port number, that is system-wide guaranteed to not be used at the very moment / until returned to the port-manager ( a rotating pool of ports is centrally managed, under your code-control, whereas all the sockets, created locally in the distributed process(es)/thread(s) .connect() / .bind()-ing to the pool-manager announced port#, and thus will still remain, and ought remain, consistently within ZeroMQ advice, not to be shared per-se ).
I have a python application , to be more precise a Network Application that can't go down this means i can't kill the PID since it actually talks with other servers and clients and so on ... many € per minute of downtime , you know the usual 24/7 system.
Anyway in my hobby projects i also work a lot with WSGI frameworks and i noticed that i have the same problem even during off-peak hours.
Anyway imagine a normal server using TCP/UDP ( put here your favourite WSGI/SIP/Classified Information Server/etc).
Now you perform a git pull in the remote server and there goes the new python files into the server (these files will of course ONLY affect the data processing and not the actual sockets so there is no need to re-raise the sockets or touch in any way the network part).
I don't usually use File monitors since i prefer to use SIGNAL to wakeup the internal app updater.
Now imagine the following code
from mysuper.app import handler
while True:
data = socket.recv()
if data:
socket.send(handler(data))
Lets imagine that handler is a APP with DB connections, cache connections , etc.
What is the best way to update the handler.
Is it safe to call reload(handler) ?
Will this break DB connections ?
Will DB Connections survive to this restart ?
Will current transactions be lost ?
Will this create anti-matter ?
What is the best-pratice patterns that you guys usually use if there are any ?
It's safe to call reload(handler).
Depends where you initialize your connections. If you make the connections inside handler(), then yes, they'll be garbage collected when the handler() object falls out of scope. But you wouldn't be connecting inside your main loop, would you? I'd highly recommend something like:
dbconnection = connect(...)
while True:
...
socket.send(handler(data, dbconnection))
if for no other reason than that you won't be making an expensive connection inside a tight loop.
That said, I'd recommend going with an entirely different architecture. Make a listener process that does basically nothing more than listen for UDP datagrams, sends them to a messaging queue like RabbitMQ, then waits for the reply message to send the results back to the client. Then write your actual servers that get their requests from the messaging queue, process them, and send a reply message back.
If you want to upgrade the UDP server, launch the new instance listening on another port. Update your firewall rules to redirect incoming traffic to the new port. Reload the rules. Kill the old process. Voila: seamless cutover.
The real win is from uncoupling your backend. Since multiple processes can listen for the same messages from your frontend "proxy" service, you can run several in parallel - on different machines, if you want to. To upgrade the backend, start a new instance then kill the old one so that there's no time when at least one instance isn't running.
To scale your proxy, have multiple instances running on different ports or different hosts, and configure your firewall to randomly redirect incoming datagrams to one of the proxies.
To scale your backend, run more instances.
I have a Python server which is not running as root, which fronts an application I am developing. However there are some application features which require access to RAW sockets which means root privileges.
Obviously I do not want to run the main server as root, and so my solution is to create a daemon process or command line script which runs as root providing guarded access to said features.
However I want put aside stdin/stdout communication and use an RPC style of interaction such as Pyro. But this exposes the RPC interface to anyone with network access to the machine, whereas I know that the process calling the RPC methods will be another process on the same machine.
Is there not a sort of inter-process procedure call standard which could be used in a similar (local machine only) fashion? I imagine the server doing something like this:
# Server not running as root
pythonically, returned, values = other_process_running_as_root.some_method()
And the process running as root exposing a method:
# Daemon running as root
#expose_this_method
def some_method():
# Play with RAW sockets
return pythonically, returned, values
Is anything like this possible?
Following my comment, I was interested to see if it was possible, so I had a go at putting this together: https://github.com/takowl/ZeroRPC
Bear in mind that this is thrown together in an hour or so, so it's almost certainly inferior to any serious solution (e.g. any errors on the server side will crash it...). But it works as you suggested:
Server:
rpcserver = zerorpc.Server("ipc://myrpc.ipc")
#rpcserver.expose
def product(a, b):
return a * b
rpcserver.run()
Client:
rpcclient = zerorpc.Client("ipc://myrpc.ipc")
print(rpcclient.product(5, 7))
rpcclient._stopserver()
This is an easy problem. You should be able to get what you want from any RPC mechanism that can use Unix sockets, or use regular TCP sockets but only accept connections from the loopback interface (listen on 127.0.0.1).
The multiprocessing library in the Python standard library supports local IPC, too. http://docs.python.org/library/multiprocessing.html#module-multiprocessing.connection
Pyro has a number of security features specifically to limit the access to the RPC interface. Are these too much of a performance burden to use?