I would like to have a server process (preferably Python) that accepts simple messages and multiple clients (again, preferably Python) that connect to the server and send messages to it. The server and clients will only ever be running on the same local machine and the OS is Linux based. The server will be automatically started by the OS and the clients started later independent of the server. I strongly want to avoid installing a whole separate messaging framework/server to do this. The messages will be simple strings such as "kick" or even just a single byte representing the message type. It also needs to know when a connection is made and lost.
From these requirements, I think named pipes would be a feasible solution, with a new instance of that pipe created for each client connection. However, when I search for examples, all of the ones I have come across deal with processes that are spawned from the same parent process and not independently started which means they can pass a parent reference to the child.
Windows seems to allow multiple instances of a named pipe (one for each client connection), but I'm unsure if this is possible on a Linux based OS?
Please could someone point me in the right direction, preferably with a basic example, even if it's just pseudo-code.
I've looked at the multiprocessing module in Python, but this seems to be oriented around the server and client sharing the same process or having one spawn the other.
Edit
May be important, the host device is not guaranteed to have networking capabilities (embedded device).
I've used zeromq for this sort of thing before. it's a relatively lightweight library that exposes this sort of functionality
otherwise, you could implement it yourself by binding a socket in the server process and having clients connect to it. this works fine for unix domain sockets, just pass AF_UNIX when creating the socket, e.g:
import socket
with socket.socket(socket.AF_UNIX) as s:
s.bind('/tmp/srv')
s.listen(1)
(c, addr) = s.accept()
with c:
c.send(b"hello world")
for the server, and:
with socket.socket(socket.AF_UNIX) as c:
c.connect('/tmp/srv')
print(c.recv(8192))
for the client.
writing a protocol around this is more involved, which is where things like zmq really help where you can easily push JSON messages around
Related
I have a Python program which spawns several other Python programs as subprocesses. One of these subprocesses is supposed to open and bind a ZMQ publisher socket, such that other subprocesses can subscribe to it.
I cannot give guarantees about which tcp ports will be available, so when I bind to a random port in the subprocess, my main program will not know what to tell the other subprocesses.
Is there a way to bind the socket in the main process and then somehow pass the socket to my subprocess? Or either some other way to preregister the socket or a standard way to pass the port information from the subprocess back to my main process (stdout and stderr are already used by other data)?
Just checking for a free port in the main process and passing that to the subprocess is not really optimal, because this could still fail if the socket is being assigned in the meantime. Also, since my program should work on Unix and Windows, I cannot really use ipc sockets, which would otherwise solve my problem.
The simplest is to create a logic for a pool-of-ports manager ( rather avoid attempts to share / pass ZeroMQ sockets to / among other processes )
One may create a persistent, a-priori known, tcp://A.B.C.D:8765-transport-class based .bind() access-point, exposed to all client processes as a port-assignment service, to which client processes .connect(), handshake in whatever manner is needed to proof an identity/credentials/purpose/etc and .recv() in a coordinated manner one actually free messaging/signalling-service port number, that is system-wide guaranteed to not be used at the very moment / until returned to the port-manager ( a rotating pool of ports is centrally managed, under your code-control, whereas all the sockets, created locally in the distributed process(es)/thread(s) .connect() / .bind()-ing to the pool-manager announced port#, and thus will still remain, and ought remain, consistently within ZeroMQ advice, not to be shared per-se ).
I'm running a client which makes remote calls to my server, both written with twisted. The methods running on the server side can be quite long to return, and they're eating mostly CPU in Python code so threading won't be of any help here.
I've tried a lot of stuff but eventually I think I'm going to run several instances of twisted servers to distribute the tasks.
So I'm telling my servers to listen on several sockets (let's say I create them using serverFromString on socket_1 for server 1 and on socket_2 for server 2), and I'm connecting my client on these sockets with 2 calls to connectUNIX with socket_1 and socket_2 as arguments.
So far I managed to create the servers listening on the ports I want them to, but I'm not sure how to tell my client to distribute callRemote across the sockets. When I compute several callRemote it seems that only one server is actually being used. How do I do that ?
P.S. : I tried using multiprocessing but my methods on the server side are full of unpicklable objects so no chance ; also the API of spawnProcess is utterly non compliant with the code I'm calling. Also I'm not willing to use an undocumented unmaintained project so Ampoule is not an option here.
Edit : No answer so I guess the question wasn't clear enough. Basically it all boils down to : can I pass a 'port' or 'socket' argument to callRemote so I can manage on which server I'm running the remote calls ?
Basically I have a django/uwsgi client in which I call reactor.connectUNIX(my_port). I redirect all the calls made in my python code to a threads.blockingCallFromThread(reactor, callRemote, args). The remote calls go to my_port, though I don't really know where/how the argument is passed. On the server side the application is launched with twisted.scripts._twistd_unix.UnixApplicationRunner, and the server listens on my_port
I'd like to start several servers on different addresses and have my client dispatch the remote calls among the servers. I don't know if I'm clear yet, I would gladly add more precisions.
I have hardware that connects to raw TCP socket on any given IP and port combination. It then continually sends characters. The following piece of Python code may give you an idea of what the hardware does.
import socket
serverIP = '*server IP or domain*'
serverPort = 60000
Sock = socket(AF_INET, SOCK_STREAM)
Sock.connect((serverIP, serverPort))
while (1):
f = open ("send-data.txt","r")
while 1:
c = f.readline()
if not c:
break
Sock.send(c + '\n')
Sock.shutdown(0)
Sock.close()
When this code is run it exactly behaves like my hardware system. The send-data.txt file contains characters similar to what hardware sends.
I have written a socket server in Python using SocketServer library. It allows connections, receives character stream, and stores it into a local (newly created) file. Currently, I am running this code on my system, as localhost and it works. I would like to serve these files through a webpage.
I want to be able to do the same on remote server. As you can see, my hardware limits me to use only raw TCP sockets. From what I understand, I'll need low-level access to the server machine like IaaS. I tried pythonanywhere, but I guess they don't allow simple python sockets. Heroku also requires you to write a web app, and I don't know how to go about that or whether it'll work with my hardware.
What hosting/Cloud solution out there could act as above-mentioned socket server and also as HTTP server which would later serve these files and webpages.
If I understand your question correctly, you'd like to know which affordable hosting solution would allow you to communicate via arbitrary TCP sockets.
The answer is simple: Pretty much any VPS (Virtual Private Server) company or IaaS provider. Since you tagged your question with Amazon-EC2, yes they do too, but the learning curve to get your first instance running and the security groups (read: firewall rules, which live outside your VM) configured, is rather steep. That said, you do have a so-called "Free Tier" there for one year, which enables you to try out most of their services free-of-charge.
Other providers might be more suitable. (I'm not sure if it's allowed to suggest providers here, but you could for example look at Linode or Rackspace Cloud; they offer much less flexibility than EC2, but it's a whole lot easier to get started.)
As with any IaaS option, it would be beneficial to know Linux, networking and some security basics (at the very least) as you will be solely responsible for the things you create.
Talking about security...
If that piece of code you posted has a similarly rudimentary receiving end, you're setting yourself up for trouble as soon as it's out there in public, as the communication is done in plain text [*] and doesn't seem to require any kind of authentication. Anybody could probably telnet to the receiving end and just inject some lines of text?
(That's exactly why considerably sane PaaS providers often don't let you communicate over arbitrary ports and sockets :-) )
[*] I am guessing that, because you use readline. If any encryption was involved, you'd likely write/read in chunks of bytes.
I have a python application , to be more precise a Network Application that can't go down this means i can't kill the PID since it actually talks with other servers and clients and so on ... many € per minute of downtime , you know the usual 24/7 system.
Anyway in my hobby projects i also work a lot with WSGI frameworks and i noticed that i have the same problem even during off-peak hours.
Anyway imagine a normal server using TCP/UDP ( put here your favourite WSGI/SIP/Classified Information Server/etc).
Now you perform a git pull in the remote server and there goes the new python files into the server (these files will of course ONLY affect the data processing and not the actual sockets so there is no need to re-raise the sockets or touch in any way the network part).
I don't usually use File monitors since i prefer to use SIGNAL to wakeup the internal app updater.
Now imagine the following code
from mysuper.app import handler
while True:
data = socket.recv()
if data:
socket.send(handler(data))
Lets imagine that handler is a APP with DB connections, cache connections , etc.
What is the best way to update the handler.
Is it safe to call reload(handler) ?
Will this break DB connections ?
Will DB Connections survive to this restart ?
Will current transactions be lost ?
Will this create anti-matter ?
What is the best-pratice patterns that you guys usually use if there are any ?
It's safe to call reload(handler).
Depends where you initialize your connections. If you make the connections inside handler(), then yes, they'll be garbage collected when the handler() object falls out of scope. But you wouldn't be connecting inside your main loop, would you? I'd highly recommend something like:
dbconnection = connect(...)
while True:
...
socket.send(handler(data, dbconnection))
if for no other reason than that you won't be making an expensive connection inside a tight loop.
That said, I'd recommend going with an entirely different architecture. Make a listener process that does basically nothing more than listen for UDP datagrams, sends them to a messaging queue like RabbitMQ, then waits for the reply message to send the results back to the client. Then write your actual servers that get their requests from the messaging queue, process them, and send a reply message back.
If you want to upgrade the UDP server, launch the new instance listening on another port. Update your firewall rules to redirect incoming traffic to the new port. Reload the rules. Kill the old process. Voila: seamless cutover.
The real win is from uncoupling your backend. Since multiple processes can listen for the same messages from your frontend "proxy" service, you can run several in parallel - on different machines, if you want to. To upgrade the backend, start a new instance then kill the old one so that there's no time when at least one instance isn't running.
To scale your proxy, have multiple instances running on different ports or different hosts, and configure your firewall to randomly redirect incoming datagrams to one of the proxies.
To scale your backend, run more instances.
I have a Python server which is not running as root, which fronts an application I am developing. However there are some application features which require access to RAW sockets which means root privileges.
Obviously I do not want to run the main server as root, and so my solution is to create a daemon process or command line script which runs as root providing guarded access to said features.
However I want put aside stdin/stdout communication and use an RPC style of interaction such as Pyro. But this exposes the RPC interface to anyone with network access to the machine, whereas I know that the process calling the RPC methods will be another process on the same machine.
Is there not a sort of inter-process procedure call standard which could be used in a similar (local machine only) fashion? I imagine the server doing something like this:
# Server not running as root
pythonically, returned, values = other_process_running_as_root.some_method()
And the process running as root exposing a method:
# Daemon running as root
#expose_this_method
def some_method():
# Play with RAW sockets
return pythonically, returned, values
Is anything like this possible?
Following my comment, I was interested to see if it was possible, so I had a go at putting this together: https://github.com/takowl/ZeroRPC
Bear in mind that this is thrown together in an hour or so, so it's almost certainly inferior to any serious solution (e.g. any errors on the server side will crash it...). But it works as you suggested:
Server:
rpcserver = zerorpc.Server("ipc://myrpc.ipc")
#rpcserver.expose
def product(a, b):
return a * b
rpcserver.run()
Client:
rpcclient = zerorpc.Client("ipc://myrpc.ipc")
print(rpcclient.product(5, 7))
rpcclient._stopserver()
This is an easy problem. You should be able to get what you want from any RPC mechanism that can use Unix sockets, or use regular TCP sockets but only accept connections from the loopback interface (listen on 127.0.0.1).
The multiprocessing library in the Python standard library supports local IPC, too. http://docs.python.org/library/multiprocessing.html#module-multiprocessing.connection
Pyro has a number of security features specifically to limit the access to the RPC interface. Are these too much of a performance burden to use?