I have a flask-socketio server running on multiple pods, using redis as a message queue. I want to ensure that emits from external processes reach their destination 100% of the time, or to know when they have failed.
When process A emits an event to a socket that's connected to process B, the event goes through the message queue to process B, to the client. Is there any way I can intercept the outgoing emit on process B? Ideally i'd then use a worker to check after a few seconds if the message reached the client (via a confirm event emitted from the client) or it will be emitted again.
This code runs on process A:
#app.route('/ex')
def ex_route():
socketio.emit('external', {'text': f'sender: {socket.gethostname()}, welcome!'}, room='some_room')
return jsonify(f'sending message to room "some_room" from {socket.gethostname()}')
This is the output from process A
INFO:socketio.server:emitting event "external" to some_room [/]
INFO:geventwebsocket.handler:127.0.0.1 - - [2019-01-11 13:33:44] "GET /ex HTTP/1.1" 200 177 0.003196
This is the output from process B
INFO:engineio.server:9aab2215a0da4816a45e3fdc1e449fce: Sending packet MESSAGE data 2["external",{"text":"sender: *******, welcome!"}]
There is currently no mechanism to do what you ask, unfortunately.
I think you basically have two approaches to go about this:
Always run your emits from the main server(s). If you need to emit from an auxiliary process, use an IPC mechanism to notify the server so that it can run the emit on its behalf. And now you can use callbacks.
Ignore the callbacks, and instead have the client acknowledge receipt of the event by emitting back to the server.
Adding callback support for auxiliary processes should not be terribly difficult, by the way. I never needed that functionality myself and you are the first to ask about it. Maybe I should look into that at some point.
Edit: after some thought, I came up with a 3rd option:
You can connect your external process to the server as a client, instead of using the "emit-only" option. If this process is a client, it can emit an event to the server, which in turn the server can relay to the external client. When the client replies to the server, the server can once again relay the response to the external process, which is not another client and has full send and receive capabilities.
Using IPC is not very robust, especially in case of server receiving a lot of requests there might be an issue where you receive a message and don't retranslate it and it's vital.
Use either celery or zmq or redis itself for interconnect. The most natural is using socketio itself like mentioned by Miguel as it's already waiting for the requests has the environment and can emit anytime.
I've used a greenlet hack over threads - where greenlet is lighter than threads and runs in the same environment allowing it to send the message while your main thread awaits socket in non-blocking mode. Basically you write a thread, then apply eventlet or gevent to the whole code via monkeypatching and the thread becomes a greenlet - an inbetween function call. You put a sleep on it so it doesn't hog all resources and you have your sender because greenlets share environment easily, they are not bound by io, just cpu (which is the same for threads in Python but greenlets are even more lightweight due to no OS-level context change at all).
But as soon as CPU load increased I switched over to client/server. Imbuing IPC would require massive rewrites from the ground up.
Related
Short brief about my situation:
I'm writing a server (twisted powered) which handles WebSocket connections with multiple clients (over 1000). Messages send from server to client are handled via Redis pub/subinterface (because messages can be applied via REST) in that flow:
REST appends command to the client and publishes,
twisted is getting poked because it subscribes that redis channel,
the message is added to the client queue and waits for further processing
Now, as client connects, gets registered I'm launching inlineCallback for each client to sweep throught the queue, like this:
#inlineCallbacks
def client_queue_handler(self, uuid):
queue = self.send_queue[uuid]
client = self.get_client(uuid)
while True:
uniqueID = yield queue.get()
client_context = self.redis_handler.get_single(uuid)
msg_context = next(iter([msg
for msg in client_context
if msg['unique'] == unique]),
None)
client.sendMessage(msg_context)
As I said previously, many clients may connect. Is this perfectly fine, that each client has it's own inlineCallback which performs an infinite loop? As far as I know, twisted has customizable thread pool limit. What will happen if there will be more clients (inlineCallbacks) than threads in threadpool? Will queue.get() block/sleep that "virtual thread" and pass control to the other one? Maybe one "global" thread which sweeps all clients is a better option?
inlineCallbacks doesn't start any OS threads. It's just a different interface to using Deferred. A Deferred is just an API for dealing with callbacks.
queue.get() returns a Deferred. When you yield it, inlineCallbacks internally adds a callback to it and your function remains suspended. When the callback fires, inlineCallbacks resumes your function with the value passed to the callback - which is the "result" of the Deferred you yielded.
All that's happening is some Deferred objects are being created and some callbacks are being added to them. Somewhere inside the implementation of your redis client, some event sources are "firing" the Deferred with a result which starts the process of calling its callbacks.
You can have as many of these:
* as you have system memory to hold
* as the redis client can keep track of at a time
I don't know the details of how your redis client is implemented. If it has to open a socket per queue then you'll probably be limited to the number of file descriptors you can open or the number of sockets your system can support. These numbers will be somewhere in the tens of thousands and when you bump into them, there are tricks you can deploy to raise the limit further.
If it doesn't have to open a socket per queue (for example, if it can multiplex notification for all the queues over a single socket) then it probably has a limit that's much, much higher (maybe imposed by algorithmic complexity of the slowest part of its implementation).
I want to write an Java Server may be using Netty or anything else suggested.
The whole purpose is that I want to queue incoming HTTP Request for a while because the systems I'm targeting are doing Super Memory and Compute intensive tasks so if they are burdened with heavy load they eventually tend to get crashed.
I want to have a queue in place that will actually allow only max upto 5 requests passed to destination at any given time and hold the rest of the requests in queue.
Can this be achieved using Netty in Java, I'm equally open for an implementation in Scala, Python or clojure.
I did something similar with Scala Akka actors. Instead of HTTP Request I had unlimited number of job requests come in and get added to a queue (regular Queue). Worker Manager would manage that queue and dispatch work to worker actors whenever they are done processing previous tasks. Workers would notify Worker Manager that task is complete and it would send them a new one from the queue. So in this case there is no busy waiting or looping, everything happens on message reception. You can do the same with your HTTP Requests. Akka can be used from Scala or Java and a process I described is easier to implement than it sounds.
As a web server you could use anything really. It can be Jetty, or some Servlet Container like Tomcat, or even Spray-can. All it needs to do is to receive a request and send a message to Worker Manager. The whole system would be asynchronous and non-blocking.
I am randomly getting error 1006 ( (I failed the WebSocket connection by dropping the TCP connection) when trying to write messages from threads using Tornado's websocket server handler.
I created N threads and passed my ws_handler to them.
But when I start using
self.ws_handler.write_message(jsondata)
for a large number of threads, I keep getting the same error.
From what I understand, 1006 is TCP connection dropped when a 'heartbeat' communication is skipped between websockets. I am guessing this is because of threads running in parallel and trying to send messages. I tested it using 2-3 threads and it works fine but for large number it doesn't.
I wonder if there's any method to achieve message sending within threads.( meaning lock being handled internally by ws_handler and sending accordingly).
One solution I am thinking of is to push jsondata into a queue and have another single thread push the messages, but I fear that would create a bottleneck.
My client is AutobahnPython.
Tornado is based on a single-threaded event loop; all interactions with Tornado objects must be on the event loop's thread. Use IOLoop.current().add_callback() from another thread when you need to transfer control back to the event loop.
See also http://www.tornadoweb.org/en/stable/web.html#thread-safety-notes
I have an idea. Write a WebSocket based RPC that would process messages according to the scenario below.
Client connects to a WS (web socket) server
Client sends a message to the WS server
WS server puts the message into the incoming queue (can be a multiprocessing.Queue or RabbitMQ queue)
One of the workers in the process pool picks up the message for processing
Message is being processed (can be blazingly fast or extremely slow - it is irrelevant for the WS server)
After the message is processed, results of the processing are pushed to the outcoming queue
WS server pops the result from the queue and sends it to the client
NOTE: the key point is that the WS server should be non-blocking and responsible only for:
connection acceptance
getting messages from the client and puting them into the incoming queue
popping messages from the outcoming queue and sending them back to the client
NOTE2: it might be a good idea to store client identifier somehow and pass it around with the message from the client
NOTE3: it is completely fine that because of queueing the messages back and forth the speed of simple message processing (e.g. get message as input and push it back as a result) shall become lower. Target goal is to be able to run processor expensive operations (rough non-practical example: several nested “for” loops) in the pool with the same code style as handling fast messages. I.e. pop message from the input queue together with some sort of client identifier, process it (might take a while) and push the processing results together with client ID to the output queue.
Questions:
In TornadoWeb, if I have a queue (multiprocessing or Rabit), how can
I make Tornado’s IOLoop trigger some callback whenever there is a new
item in that queue? Can you navigate me to some existing
implementation if there is any?
Is there any ready implementation of such a design? (Not necessarily with Tornado)
Maybe I should use another language (not python) to implement such a design?
Acknowledgments:
Recommendations to use REST and WSGI for whatever goal I aim to achieve are not welcome
Comments like “Here is a link to the code that I found by googling for 2 seconds. It has some imports from tornado and multiprocessing.I am not sure what it does, however I am for 99% certain that it isexactly what you need” are not welcome neither
Recommendations to use asynchronous libraries instead of normal blocking ones are ... :)
Tornado's IOLoop allows you handling events from any file object by its file descriptor, so you could try this:
connect with each of your workers processes through multiprocessing.Pipe
call add_handler for each pipe's parent end (using the connection's fileno())
make the workers write some random garbage each time they put something into the output queue, no matter if that's multiprocessing.Queue of any MQ.
handle the answers form the workers in the event handlers
I'm working on TCP client-server application using the IntNReceiver protocol. Server is accepting multiple TCP connections from client. I would like to let other threads use the protocol's sendString method, on both client and the server. I tried to use synchronized queue, monitored in separate thread and reactor.callFromThread() to call the sendString from there. This seems to work but there is a weird delay of about 20 seconds before the actual sendString actually sends the string. It does not block, returns immediately. I ran strace and the send() system call is definitely delayed. What is the proper way to do this kind of thing with twisted?
Just use callFromThread directly as your queue. The reactor is already synchronizing on and monitoring it. Anywhere you want to call foo.sendString() from a non-reactor thread, just do reactor.callFromThread(foo.sendString). Building additional infrastructure to do this (your own custom synchronized queues, for example) is just additional code that might break – as you have already discovered.