I just found out Twitter streaming endpoints support detection of slow connections somehow.
Reference: https://dev.twitter.com/docs/streaming-apis/parameters#stall_warnings (and bottom of page)
Idea is that socket send will probably process data one by one. And it knows when one packet is received by client so it can maintain queue and always know of it's size.
It's easy when client sends some confirmation packets for each of them. But that is not the case with Twitter Streaming API - it's a one-way transfer.
My question is: how did they achieve that? I can't see a way to do it without some very low level raw socket support - but I may be forgetting something here. With some low level support we could probably get ACKs for each packets. Is that even possible? Can ACKs be somehow traced?
Any other ideas how this was done?
Any way to do this e.g. in Python? Or any other language example would be appreciated.
Or maybe I am over my head here and it simply uses to track how many bytes are not yet processed through socket.send? But isn't it a poor indication of client's connection?
I started off thinking along the same lines as you but I think the implementation is actually much easier than we both expect.
Twitter's API docs state:-
"A client reads data too slowly. Every streaming connection is backed by a queue of messages to be sent to the client. If this queue grows too large over time, the connection will be closed." - https://dev.twitter.com/docs/streaming-apis/connecting#Disconnections
Based on the above I imagine Twitter will have a thread that is pushing tweets onto a queue and a long lived http connection to a client (kept open with a while loop) that pops a message off the queue and writes the data to the http response during each loop iteration.
Now if you imagine what happens inside the while loop and you think in terms of buffers, Twitter will pop an item off the queue then write the tweet data to some kind of output buffer, that buffer will get flushed and then fill up a TCP buffer for transport to the client.
If a client is reading data slowly from its TCP buffer then the server's TCP send buffer will fill up meaning that when the server's output buffer is flushed it will block because the data cannot be written to the TCP buffer which consequently means that the while loop is not popping tweets off the queue as often (because it is being blocked when data is being flushed) causing the tweet queue to fill up.
Now you would just need a check at the beginning of each loop iteration to check whether the Tweet queue has reached some predefined threshold.
Related
Is there a simple method or library to allow a websocket to drop certain messages if bandwidth doesn't allow? Or any one of the following?
to measure the queue size of outgoing messages that haven't yet reached a particular client
to measure the approximate bitrate that a client has been receiving recent messages at
to measure the time that a particular write_message finished being transmitted to the client
I'm using Tornado on the server side (tornado.websocket.WebSocketHandler) and vanilla JS on the client side. In my use case it's really only important that the server realize that a client is slow and throttle its messages (or use lossier compression) when it realizes that condition.
You can implement this on top of what you have by having the client confirm every message it gets and then use that information on the server to adapt the sending of messages to each client.
This is the only way you will know which outgoing messages haven't yet reached the client, be able to approximate bitrate or figure out the time it took for the message to reach the client. You must consider that the message back to the server will also take time and that if you use timestamps on the client, they will likely not match your servers as clients have their time set incorrectly more often than not.
I am new to the multithreading web server programming
Now I am writing a server program that:
Receive messages (in self-defined data format) from tcp socket
Process these messages (which takes time)
Send corresponding responses to the socket
Provide ACK mechanism for receiving messages and sending responses, that is every message contains a unique seq number and I should include the ack (same as seq) in the corresponding response. The other side also implements this mechanism. If I did not receive ACK from the other side for 5 min, I should re-send the message that I expected to receive corresponding ACK from.
My thought was to use a while loop to receive messages from the socket, then process the messages and send responses.
The problem is, processing messages takes time and I may receive multiple messages in a short period. So if I call the process_message() function in this while loop and wait for its finish, it will be blocking and I will definitely waste time. So I need non-blocking way.
I have done some research. I supposed I may use two common techs: thread pool and message queue.
For thread pool, my idea goes like the following pseudo code:
def process_message():
process_message // takes time
send_response(socket)
while True:
message = recv(socket)
thread = thread_pool.get_one()
thread.start(target=process_message)
For message queue, I am not sure, but my idea would be having producer thread and consumer thread:
def consumer:
// only one consumer thread?
message = queue.poll()
consumer_thread.process_message(message)
send_response(socket)
while True:
// only one producer thread?
message = recv(socket)
producer_thread.put_message_to_queue()
Hope my idea is clear. Can anyone provide some typical solution?
Then, the tricker part, any thoughts on how to implement the ACK mechanism?
Thank you!
This is rather broad because there is still too much to implement.
The general idea is indeed to implement:
a TCP server, that will receive incoming messages and write them (including the socket from which they were received) in a queue
a pool of worker threads that will get a message from the queue, process the message, and pass the response to an object in charge of sending the message and wait for the acknowledgement
an object that will send the responses, store the sequence number, the socket and the message until the response has been acknowledged. A thread would be handy to process the list of message waiting for acknowledgement and sent them again when the timeout is exhausted.
But each part requires a consequent amount of work, and can be implemented in different ways (select, TCPServer or threads processing accepted sockets for the first, which data structure to store the messages waiting for acknowledgement for the third, and which pool implementation for the second). I have done some tests and realized that a complete answer would be far beyond what is expected on this site. IMHO, you'd better break the question in smaller answerable pieces, keeping this one as the general context.
You should also say whether the incoming messages should be immediately acknowledged when received or will be implicitely acknowledged by the response.
while True:
data = resp.read(65536)
if not data:
break
yield data
Actually I'm not asking for code, but the principle of the entire http connection.
If I stop the program at one yield, for instance, debugging, where is the rest of my http response data? Are they still in the server, or in my client machine's memory?
If the former one, what does the program do in web server to prevent the data from being flushed to client all by once? Control the stream by TCP sequence?
First of all, it depends on your framework. Normally, for yielded responses the Chunked HTTP Transfer is used. So only data, that was read, is sent to the client. No data is buffered at the server side.
I think it depends on the length of your data,if your data is short, the client read once and get them all, if you stop the program, your data is in client's memory.Otherwise, if your data is too long to read once, it may still in the server side, at this time, you stop the client program, the rest data is not in your program memory.
I'm working on a really basic "image streaming" server as a school subject, and I've done most of the work but I'm still stuck on the separation between data and control related sockets:
My structure is : TCPServer (my server, used as control socket) contains a dataSocket (only used to send images and initialized within my TCPServer object, when I receive a certain query)
When I'm sending data (images) through my dataSocket, I still need to see if the client sent a PAUSE or STOP request, but if I use python's self.request.recv(1024) the server awaits a response instead of continuing to send data (which is quite logical).
What should I do to prevent this behavior ? Should I launch my recv(1024) on a separate thread and run it at each loop (and check if I get any relevant data in between two iterations) ?
Twisted should do the trick! It handles asynchronous sockets in Python
I wrote an audio broadcasting server with Python/Twisted. It works fine, but the usage of memory grows too fast! I think that's because some user's network might not be good enough to download the audio in time.
My audio server broadcast audio data to different listener's client, if some of them can't download the audio in time, that means, my server keep the audio data until listeners received. And what's more, my audio server is a broadcasting server, it receive audio data, and send them to different clients, I though Twisted copy those data in different buffer, even they are same audio piece.
I want to reduce the usage of memory usage, so I need to know when is the audio received by the client, so that I can decide when to discard some slow clients. But I have no idea how to achieve that with Twisted. Do anyone have idea?
And what else can I do to reduce usage of memory usage?
Thanks.
Victor Lin.
You didn't say, but I'm going to assume that you're using TCP. It would be hard to write a UDP-based system which had ever increasing memory because of clients who can't receive data as fast as you're trying to send it.
TCP has built-in flow control capabilities. If a receiver cannot read data as fast as you'd like to send it, this information will be made available to you and you can send more slowly. The way this works with the BSD socket API is that a send(2) call will block or will return 0 to indicate it cannot add any bytes to the send buffer. The way it works in Twisted is by a system called "producers and consumers". The gist of this system is that you register a producer with a consumer. The producer calls write on the consumer repeatedly. When the consumer cannot keep up, it calls pauseProducing on the producer. When the consumer is again ready for more data, it calls resumeProducing on the producer.
You can read about this system in more detail in the producer/consumer howto, part of Twisted's documentation.
Make sure you're using Python's garbage collector and then go through and delete variables you aren't using.