I am new to the multithreading web server programming
Now I am writing a server program that:
Receive messages (in self-defined data format) from tcp socket
Process these messages (which takes time)
Send corresponding responses to the socket
Provide ACK mechanism for receiving messages and sending responses, that is every message contains a unique seq number and I should include the ack (same as seq) in the corresponding response. The other side also implements this mechanism. If I did not receive ACK from the other side for 5 min, I should re-send the message that I expected to receive corresponding ACK from.
My thought was to use a while loop to receive messages from the socket, then process the messages and send responses.
The problem is, processing messages takes time and I may receive multiple messages in a short period. So if I call the process_message() function in this while loop and wait for its finish, it will be blocking and I will definitely waste time. So I need non-blocking way.
I have done some research. I supposed I may use two common techs: thread pool and message queue.
For thread pool, my idea goes like the following pseudo code:
def process_message():
process_message // takes time
send_response(socket)
while True:
message = recv(socket)
thread = thread_pool.get_one()
thread.start(target=process_message)
For message queue, I am not sure, but my idea would be having producer thread and consumer thread:
def consumer:
// only one consumer thread?
message = queue.poll()
consumer_thread.process_message(message)
send_response(socket)
while True:
// only one producer thread?
message = recv(socket)
producer_thread.put_message_to_queue()
Hope my idea is clear. Can anyone provide some typical solution?
Then, the tricker part, any thoughts on how to implement the ACK mechanism?
Thank you!
This is rather broad because there is still too much to implement.
The general idea is indeed to implement:
a TCP server, that will receive incoming messages and write them (including the socket from which they were received) in a queue
a pool of worker threads that will get a message from the queue, process the message, and pass the response to an object in charge of sending the message and wait for the acknowledgement
an object that will send the responses, store the sequence number, the socket and the message until the response has been acknowledged. A thread would be handy to process the list of message waiting for acknowledgement and sent them again when the timeout is exhausted.
But each part requires a consequent amount of work, and can be implemented in different ways (select, TCPServer or threads processing accepted sockets for the first, which data structure to store the messages waiting for acknowledgement for the third, and which pool implementation for the second). I have done some tests and realized that a complete answer would be far beyond what is expected on this site. IMHO, you'd better break the question in smaller answerable pieces, keeping this one as the general context.
You should also say whether the incoming messages should be immediately acknowledged when received or will be implicitely acknowledged by the response.
Related
I have a requirement where I am going to receive asynchronous data (JSON) from client over http, which I've to process whenever it is received and then send it to a device over TCP connection, and get the responses back (1 or 2 responses based on request). This response I have to send it to client back.
My question is how can I do that? i.e. should I run to while True: loop and wait for data is put on the Queue, by keep checking it if it is not empty, once received , I will collect data from Queue and send it over TCP, But how second loop (recv data from TCP connection) should be run? Same while True: loop for waiting for TCP response? and once response is received how do I send it to HTTP client?
If yes, then how will it work? Can someone provide an example? I thought running two processes, one for write_to_queue and read_from_queue, but still can't comprehend how to implement and how will it work.
I open a single TCP connection to a gateway PC. My application will send messages to the gateway which will process the payload and pass on the message to another computer (based on payload) for processing.
For example, I send message A over the TCP connection which will be routed to computer A for response. But I may also need to send message B which goes to computer B.
Currently I simply use send(messageA) and then use recv() to wait for the response. The downside is that the recv() will block which means I can't send message B until something is received (and I can't do any other tasks).
I have read about the following options but am confused to the best for my use case.
Make the socket non-blocking. I send message A, call recv() (let's assume there is some delay in processing such that nothing is received immediately; so code moves on), move on to send message B and again call recv(). Now, it could be A or B that responds, which I can handle. But I need to call recv() again since only one response received so far; but what if computer A is down and never responds -- at some point I need to decide to stop calling recv(), right? On what basis would I do this?
Set a timeout on the socket. Again, send message A but assume computer A is down, so the code will wait for timeout before moving on which is wasted time.
Use select. Since I have only one socket and I don't think that helps here; plus, I understand select will block unless a timeout is set so no different, in this case, to the option above?
Use multithreading. Have one thread to process the main application and do the sending. And another thread that just calls recv() in an infinite loop (or a long timeout) that calls a callback whenever data is available. But then if the connection is closed from the main thread, will the recv thread cause an exception or hang?
I am really not sure what best practices are or the pitfalls of the options above. Which would be best option or is there another option?
(I'm using Python, in case it makes a difference).
Im trying to make a tcp communication, where the server sends a message every x seconds through a socket, and should stop sending those messages on a certain condition where the client isnt sending any message for 5 seconds.
To be more detailed, the client also sends constant messages which are all ignored by the server on the same socket as above, and can stop sending them at any unknown time. The messages are, for simplicity, used as alive messages to inform the server that the communication is still relevant.
The problem is that if i want to send repeated messages from the server, i cannot allow it to "get busy" and start receiving messages instead, thus i cannot detect when a new messages arrives from the other side and act accordingly.
The problem is independent of the programming language, but to be more specific im using python, and cannot access the code of the client.
Is there any option of receiving and sending messages on a single socket simultaneously?
Thanks!
Option 1
Use two threads, one will write to the socket and the second will read from it.
This works since sockets are full-duplex (allow bi-directional simultaneous access).
Option 2
Use a single thread that manages all keep alives using select.epoll. This way one thread can handle multiple clients. Remember though, that if this isn't the only thread that uses the sockets, you might need to handle thread safety on your own
As discussed in another answer, threads are one common approach. The other approach is to use an event loop and nonblocking I/O. Recent versions of Python (I think starting at 3.4) include a package called asyncio that supports this.
You can call the create_connection method on an event_loop to create an asyncio connection. See this example for a simple server that reads and writes over TCP.
In many cases an event loop can permit higher performance than threads, but it has the disadvantage of requiring most or all of your code to be aware of the event model.
I have a server process which receives requests from a web clients.
The server has to call an external worker process ( another .py ) which streams data to the server and the server streams back to the client.
The server has to monitor these worker processes and send messages to them ( basically kill them or send messages to control which kind of data gets streamed ). These messages are asynchronous ( e.g. depend on the web client )
I thought in using ZeroMQ sockets over an ipc://-transport-class , but the call for socket.recv() method is blocking.
Should I use two sockets ( one for streaming data to the server and another to receive control messages from server )?
Using a separate socket for signalling and messaging is always better
While a Poller-instance will help a bit, the cardinal step is to use separate socket for signalling and another one for data-streaming. Always. The point is, that in such setup, both the Poller.poll() and the event-loop can remain socket-specific and spent not more than a predefined amount of time, during a real-time controlled code-execution.
So, do not hesitate to setup a bit richer signalling/messaging infrastructure as an environment where you will only enjoy the increased simplicity of control, separation of concerns and clarity of intents.
ZeroMQ is an excellent tool for doing this - including per-socket IO-thread affinity, so indeed a fine-grain performance tuning is available at your fingertips.
I think if figured out a solution, but I don't know if there is a better (more efficient, safer, ...) way of doing this.
The client makes a request to the server, which spawns N processes worker to attend the request.
This is the relevant excerpt from worker.py:
for i in range(start_counter,10):
# Check if there is any message from server
while True:
try:
msg = worker.recv(zmq.DONTWAIT)
print("Received {} from server".format(msg))
except zmq.Again:
break
# Send data to server
worker.send(b"Message {} from {}".format(i, worker_id))
# Take some sleep
time.sleep(random.uniform(0.3, 1.1))
In this way, the worker a) does not need a separate socket and b) does not need a separate thread to process messages from server.
In the real implementation, worker must stream 128 byte messages at 100Hz to the server, and the server must receive lots of this messages (many clients asking requests that need 3-10 worker each).
Will this approach suffer a performance hit if implemented this way?
I just found out Twitter streaming endpoints support detection of slow connections somehow.
Reference: https://dev.twitter.com/docs/streaming-apis/parameters#stall_warnings (and bottom of page)
Idea is that socket send will probably process data one by one. And it knows when one packet is received by client so it can maintain queue and always know of it's size.
It's easy when client sends some confirmation packets for each of them. But that is not the case with Twitter Streaming API - it's a one-way transfer.
My question is: how did they achieve that? I can't see a way to do it without some very low level raw socket support - but I may be forgetting something here. With some low level support we could probably get ACKs for each packets. Is that even possible? Can ACKs be somehow traced?
Any other ideas how this was done?
Any way to do this e.g. in Python? Or any other language example would be appreciated.
Or maybe I am over my head here and it simply uses to track how many bytes are not yet processed through socket.send? But isn't it a poor indication of client's connection?
I started off thinking along the same lines as you but I think the implementation is actually much easier than we both expect.
Twitter's API docs state:-
"A client reads data too slowly. Every streaming connection is backed by a queue of messages to be sent to the client. If this queue grows too large over time, the connection will be closed." - https://dev.twitter.com/docs/streaming-apis/connecting#Disconnections
Based on the above I imagine Twitter will have a thread that is pushing tweets onto a queue and a long lived http connection to a client (kept open with a while loop) that pops a message off the queue and writes the data to the http response during each loop iteration.
Now if you imagine what happens inside the while loop and you think in terms of buffers, Twitter will pop an item off the queue then write the tweet data to some kind of output buffer, that buffer will get flushed and then fill up a TCP buffer for transport to the client.
If a client is reading data slowly from its TCP buffer then the server's TCP send buffer will fill up meaning that when the server's output buffer is flushed it will block because the data cannot be written to the TCP buffer which consequently means that the while loop is not popping tweets off the queue as often (because it is being blocked when data is being flushed) causing the tweet queue to fill up.
Now you would just need a check at the beginning of each loop iteration to check whether the Tweet queue has reached some predefined threshold.