I was delving into socket tuneables, and I encountered the SO_RCVLOWAT option, so I created a test-case to determine whether the watermark eliminated a specific issue I've had in developing servers before (namely premature data truncation:)
def run_server(host: str, port: int, low_watermark: int):
sk_server = socket.socket()
sk_server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True)
sk_server.setsockopt(
socket.SOL_SOCKET, socket.SO_RCVLOWAT, low_watermark
)
sk_server.bind((host, port))
sk_server.listen(1)
cl_sockfd, cl_addr = sk_server.accept()
print(f"connected to client, receiving at least {low_watermark}"
" bytes of data")
assert len(cl_sockfd.recv(low_watermark)) == low_watermark
# *** never reached
print("all good!")
if __name__ == "__main__":
host, port = "localhost", 6969
watermark = 100
thd_server = threading.Thread(
target=run_server,
args=(host, port, watermark)
)
thd_server.start()
time.sleep(0.5)
client = socket.socket()
client.connect((host, port))
fragment_1 = b"a" * (n := watermark // 2)
fragment_2 = b"a" * (watermark - n)
print("sending first fragment, and waiting 1 second")
print("sent", client.send(fragment_1), "bytes")
time.sleep(1)
print("sending second fragment")
print("sent", client.send(fragment_2), "bytes")
print("done. waiting for server thread to finish")
thd_server.join()
In short, the client sends exactly watermark bytes to the server which is configured to receive at least watermark bytes, however the principal issue is that the server hangs on the cl_sockfd.recv(low_watermark).
As a sanity-check, I send a single fragment of size watermark, and of course this works as expected, and any single fragments below watermark are rejected. But, the curious observation I make is that with two fragments the only way the server will accept the send() is when the second fragment has size watermark, in which case the two fragments combine as expected, but the remainder length after watermark is discarded, but if I instead send three fragments, the first two of which combine to make length watermark, but the third fragment has size watermark // 2 then it all works, but now the third fragment is discarded.
This does not make sense at all. Is there any explanation for this?
Post-notes:
I experimented by enabling TCP_NODELAY so that send() buffers are immediately flushed, to question whether Linux was post-maturely flushing the buffer (after the third fragment triggered a flush,) but behaviour remained identical
In the scenario of three fragments, totalling 104 bytes of data, no extraneous data is left to recv(), which is expected considering the watermark enforces a minimum, but then after sending the remaining amount to total 2*watermark the recv() works perfectly as expected even though it's spread over two send()s, which again makes no sense considering it failed to receive the initial two fragments, despite them matching the same criteria.
Related
I'm a beginner in the field of sockets and lately trying ti create a terminal chat app with that.I still have trouble understanding setblocking and select functions
"This is the code i have taken from a website i'm reading from and in the code if there is nothing in data, how does it mean that the socket has been disconnected and please also do explain what affect the setblocking in the server or the client does.I have read somewhere that setblocking allows to move on if the data has been fully not recieved,i'm not quite satisfied with the explaination.Please explain in simple words "
import select
import socket
import sys
import Queue
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setblocking(0)
server_address = ('localhost', 10000)
server.bind(server_address)
server.listen(5)
inputs = [ server ]
outputs = [ ]
message_queues = {}
while inputs:
readable, writable, exceptional = select.select(inputs, outputs, inputs)
for s in readable:
if s is server:
connection, client_address = s.accept()
connection.setblocking(0)
inputs.append(connection)
message_queues[connection] = Queue.Queue()
else:
data = s.recv(1024)
if data:
message_queues[s].put(data)
if s not in outputs:
outputs.append(s)
else:
if s in outputs:
outputs.remove(s)
inputs.remove(s)
s.close()
if there is nothing in data, how does it mean that the socket has been disconnected
The POSIX specification of recv() says:
Upon successful completion, recv() shall return the length of the message in bytes. If no messages are available to be
received and the peer has performed an orderly shutdown, recv() shall return 0. …
In the Python interface, return value 0 corresponds to a returned buffer of length 0, i. e. nothing in data.
what affect the setblocking in the server or the client does.
The setblocking(0) sets the socket to non-blocking, i. e. if e. g. the accept() or recv() cannot be completed immediately, the operation fails rather than blocks until complete. In the given code, this can hardly happen, since the operations are not tried before they are possible (due to the use of select()). However, the example is bad in that it includes output in the select() arguments, resulting in a busy loop since output is writable most of the time.
Short description:
Client sends server data via TCP socket. Data varies in length and is strings broken up by the delimiter "~~~*~~~"
For the most part it works fine. For a while. After a few minutes data winds up all over the place. So I start tracking the problem and data is ending up in the wrong place because the full thing has not been passed.
Everything comes into the server script and is parsed by a different delimiter -NewData-* then placed into a Queue. This is the code:
Yes I know the buffer size is huge. No I don't send data that kind of size in one go but I was toying around with it.
class service(SocketServer.BaseRequestHandler):
def handle(self):
data = 'dummy'
#print "Client connected with ", self.client_address
while len(data):
data = self.request.recv(163840000)
#print data
BigSocketParse = []
BigSocketParse = data.split('*-New*Data-*')
print "Putting data in queue"
for eachmatch in BigSocketParse:
#print eachmatch
q.put(str(eachmatch))
#print data
#self.request.send(data)
#print "Client exited"
self.request.close()
class ThreadedTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
pass
t = ThreadedTCPServer(('',500), service)
t.serve_forever()
I then have a thread running on while not q.empty(): which parses the data by the other delimiter "~~~*~~~"
So this works for a while. An example of the kind of data I'm sending:
2016-02-23 18:01:24.140000~~~*~~~Snowboarding~~~*~~~Blue Hills~~~*~~~Powder 42
~~~*~~~Board Rental~~~*~~~15.0~~~*~~~1~~~*~~~http://bigshoes.com
~~~*~~~No Wax~~~*~~~50.00~~~*~~~No Ramps~~~*~~~2016-02-23 19:45:00.000000~~~*~~~-15
But things started to break. So I took some control data and sent it in a loop. Would work for a while then results started winding up in the wrong place. And this turned up in my queue:
2016-02-23 18:01:24.140000~~~*~~~Snowboarding~~~*~~~Blue Hills~~~*~~~Powder 42
~~~*~~~Board Rental~~~*~~~15.0~~~*~~~1~~~*~~~http://bigshoes.com
~~~*~~~No Wax~~~*~~~50.00~~~*~~~No Ramps~~~*~~~2016-02-23 19:45:00.000000~~~*~
Cutting out the last "~~-15".
So the exact same data works then later doesn't. That suggests some kind of overflow to me.
The client connects like this:
class Connect(object):
def connect(self):
host = socket.gethostname() # Get local machine name
#host = "127.0.0.1"
port = 500 # Reserve a port for your service.
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#print('connecting to host')
sock.connect((host, port))
return sock
def send(self, command):
sock = self.connect()
#recv_data = ""
#data = True
#print('sending: ' + command)
sock.sendall(command)
sock.close()
return
It doesn't wait for a response because I don't want it hanging around waiting for one. But it closes the socket and (as far as I understand) I don't need to flush the socket buffer or anything it should just be clearing itself when the connection closes.
Would really appreciate any help on this one. It's driving me a little spare at this point.
Updates:
I'm running this on both my local machine and a pretty beefy server and I'd be pushed to believe it's a hardware issue. The server/client both run locally and sockets are used as a way for them to communicate so I don't believe latency would be the cause.
I've been reading into the issues with TCP communication. An area where I feel I'll quickly be out of my depth but I'm starting to wonder if it's not an overflow but just some king of congestion.
If sendall on the client does not ensure everything is sent maybe some kind of timer/check on the server side to make sure nothing more is coming.
The basic issue is that your:
data = self.request.recv(163840000)
line is not guaranteed to receive all the data at once (regardless of how big you make the buffer).
In order to function properly, you have to handle the case where you don't get all the data at once (you need to track where you are, and append to it). See the relevant example in the Python docs on using a socket:
Now we come to the major stumbling block of sockets - send and recv operate on the network buffers. They do not necessarily handle all the bytes you hand them (or expect from them), because their major focus is handling the network buffers. In general, they return when the associated network buffers have been filled (send) or emptied (recv). They then tell you how many bytes they handled. It is your responsibility to call them again until your message has been completely dealt with.
As mentioned, you are not receiving the full message even though you have a large buffer size. You need to keep receiving until you get zero bytes. You can write your own generator that takes the request object and yields the parts. The nice side is that you can start processing messages while some are still coming in
def recvblocks(request):
buf = ''
while 1:
newdata = request.recv(10000)
if not newdata:
if buf:
yield buf
return
buf += newdata
parts = buf.split('*-New*Data-*')
buf = parts.pop()
for part in parts:
yield part
But you need a fix on your client also. You need to shutdown the socket before close to really close the TCP connection
sock.sendall(command)
sock.shutdown(request.SHUT_RDWR)
sock.close()
I am building a python based interface for pulling data over TCP from an instrument. The datastream comes as specific events, and the timing is not steady: I get bursts of data and then slow periods. They are small data packets, so for simplicity assume they come across as complete packets.
Here is the behavior I get from the socket:
Send Event #1: socket.recv returns event #1
Send Event #2: socket.recv returns event #2
Quickly Send Event #3-50: socket.recv returns only events #3-30 (returns 27 times)
Slowly send Event #51: socket returns.recv event #31
Slowly send Event #52: socket returns.recv event #32
No data is lost. But there is clearly a buffer somewhere that is filled, and the socket is now returning old data. But shouldn't recv just keep returning till that buffer is empty? Instead, it is only returning when it receives a new packet, despite having a buffer of packets built up. Weird!
Here is the essence of the code (this is for non-blocking, I've also done blocking with just recv - same result). For simplicity I stripped all the packet reassembly stuff. I've carefully traced it back to the socket, so I know that is not to blame.
class mysocket:
def __init__(self,ip,port):
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.socket.connect((ip,port))
self.keepConn = True
self.socket.setblocking(0)
threading.Thread(target = self.rcvThread).start()
threading.Thread(target = self.parseThread).start()
def rcvThread(self):
while self.keepConn:
readable,writable,inError = select([self.socket],[self.socket],[],.1)
if readable:
packet = self.socket.recv(4096)
self.recvqueue.put_nowait(packet)
try:
xmitmsg = self.sendqueue.get_nowait()
except Queue.Empty:
pass
else:
if writable:
self.socket.send(xmitmsg)
def parseThread(self,rest = .1):
while self.keepConn:
try:
output = self.recvqueue.get_nowait()
eventnumber = struct.unpack('<H',output[:2]
print eventnumber
except Queue.Empty:
sleep(rest)
Why can't I get the socket to dump all the data in it's buffer? I can never catch up! This one is too odd. Anybody have pointers?
I'm an amateur but I've really done my homework on this one and am completely baffled.
packet = self.socket.recv(4096)
self.recvqueue.put_nowait(packet)
TCP is a stream-based protocol, not a message-based one. It doesn't preserve message boundaries. Meaning you can't expect to have one recv() call per message. If you send data in a burst, Nagle's algorithm will combine the data into one TCP packet.
Your code assumes that each recv() call returns one "packet", and the parse thread prints the first number from each "packet". But recv() doesn't return packets, it returns chunks of data from the TCP stream. These chunks can contain one message or multiple messages or even partial messages. There's no guarantee that the first two bytes are always event numbers.
Typically, reading data from a TCP connection involves calling recv() multiple times and storing the data you get in a buffer. Once you've received an entire message then you remove the appropriate number of bytes from the buffer and process them.
If you have variable-length messages then you need to keep track of message boundaries yourself. TCP doesn't do it for you like UDP does. That means adding a header containing the message length to the front of each message.
try:
xmitmsg = self.sendqueue.get_nowait()
except Queue.Empty:
pass
else:
if writable:
self.socket.send(xmitmsg)
On another note, it looks like this code has a bug. It removes messages from the sendqueue whether or not the socket is writable. If the socket's not writable it'll silently throw away messages.
Hi i'm quite a newbie to Python. I' writting a simple LAN game (not simple for me) using a pygame module.
Here's the problem - I have two computers (one old intel Atom netbook, the other intel i5 NTB). I want to achieve at least 5 FPS (the netbook is slowering the NTB, but not so much, now i have around 1,5 FPS), but calling recv() function twice a main loop takes total around 0,5 seconds on each machine. The wifi signal is strong and the router is 300Mbit/s and it sends a short roughly 500-character string. As you can see for measuring time i use time.clock().
Here's the part of the "server" code, which i usually run on the i5 NTB:
while 1:
start = time.clock()
messagelen = c.recv(4) #length of the following message (fixed 4 character)
if " " in messagelen:
messagelen = messagelen.replace(" ","")
message = cPickle.loads(c.recv(int(messagelen))) #list of the arrows, other player position and changes in the game map
arrowsmod = message[0]
modtankposan = message[1]
removelistmod = message[2]
for i in removelistmod:
try:
randopos.remove(i)
except ValueError:
randopossv.remove(i)
print time.clock()-start
tosendlist=[]
if len(arrows) == 0: #if there are no arrows it appends only an empty list
tosendlist.append([])
else:
tosendlist.append(arrows)
tosendlist.append([zeltankpos, 360-angle])
if len(removelist) == 0: #if there are no changes of the map it appends only an empty list
tosendlist.append([])
else:
tosendlist.append(removelist)
removelist=[]
tosend=cPickle.dumps(tosendlist)
tosendlen = str(len(tosend))
while len(tosendlen)<4:
tosendlen+=" "
c.sendall(tosendlen) #sends the length to client
c.sendall(tosend) #sends the actual message(dumped list of lists) to client
...something else which takes <0,05 sec on the NTB
Here's the part of the "client" game code (just inverted the beginning - sending/receiving parts):
while 1:
tosendlist=[]
if len(arrows) == 0: #if there are no arrows it appends only an empty list
tosendlist.append([])
else:
tosendlist.append(arrows)
tosendlist.append([zeltankpos, 360-angle])
if len(removelist) == 0: #if there are no changes of the map it appends only an empty list
tosendlist.append([])
else:
tosendlist.append(removelist)
removelist=[]
tosend=cPickle.dumps(tosendlist)
tosendlen = str(len(tosend))
while len(tosendlen)<4:
tosendlen+=" "
s.sendall(tosendlen) #sends the length to server
s.sendall(tosend) #sends the actual message(dumped list of lists) to server
start = time.clock()
messagelen = s.recv(4) #length of the following message (fixed 4 character)
if " " in messagelen:
messagelen = messagelen.replace(" ","")
message = cPickle.loads(s.recv(int(messagelen))) #list of the arrows, other player position and changes in the game map
arrowsmod = message[0]
modtankposan = message[1]
removelistmod = message[2]
for i in removelistmod:
try:
randopos.remove(i)
except ValueError:
randopossv.remove(i)
print time.clock()-start
... rest which takes on the old netbook <0,17 sec
When I run let's say a single player version of the game on one machine (without the socket module) on the i5 NTB it has 50 FPS in the up left corner of the map and 25 FPS in the down right corner (the 1000x1000 pixel map contains 5x5 pixel squares, i think it's slower because of the bigger coordinates, but i can't believe that so much. BTW recv while ran as a LAN game in the down right corner of the map takes approx. the same time)
on the Atom netbook it has 4-8 FPS.
So could you please tell me, why it's so slow? The computers are not synchronized, one is faster, the other slower, but it can't be that they are waiting for each other, it would be max 0,17 secs delay, right? And plus the long recv calling would be only on the faster computer?
Also I don't exactly know how the send/recv function work. It's weird the sendall takes literally no time while receiving takes 0,5 secs. Maybe sendall
is trying to send in the background while the rest of the program continues forward.
As mentioned by Armin Rigo, recv will return after packets are received by the socket, but packets don't necessarily need to be transmitted immediately after calling send. While send returns immediately, OS caches the data internally and might wait some time for more data being written to the the socket before actually transmitting it; this is called Nagle's algorithm and avoids sending lots of small packets over the network. You can disable it and push packets quicker to the wire; try enabling TCP_NODELAY options on the sending socket (or both if your communication is bidirectional), by calling this:
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
This could potentially reduce amount of time recv is sleeping due to no data.
As the Wikipedia states:
This algorithm interacts badly with TCP delayed acknowledgments, a
feature introduced into TCP at roughly the same time in the early
1980s, but by a different group. With both algorithms enabled,
applications that do two successive writes to a TCP connection,
followed by a read that will not be fulfilled until after the data
from the second write has reached the destination, experience a
constant delay of up to 500 milliseconds, the "ACK delay". For this
reason, TCP implementations usually provide applications with an
interface to disable the Nagle algorithm. This is typically called the
TCP_NODELAY option.
There is a mention of 0.5s which you're seeing in your benchmark, so this might be a reason.
Yes, send() or sendall() will occur in the background (unless the connexion is saturated right now, i.e. there is already too much data waiting to be sent). By contrast, recv() will immediately get the data only if it arrived already, but if none did, it waits. Then it returns possibly a fraction of it. (I am assuming that c is a TCP socket, not a UDP one.) Note that you should not assume that recv(N) returns N bytes; you should write a function like this:
def recvall(c, n):
data = []
while n > 0:
s = c.recv(n)
if not s: raise EOFError
data.append(s)
n -= len(s)
return ''.join(data)
Anyway, to the point. The issue is not the speed of recv(). If I understood correctly, there are four operations:
the server renders (1/25th sec)
the server sends something on the socket, received by the client;
the client renters (1/4th sec);
the client send something back on the socket.
This takes almost (0.3 + 2 * network_delay) seconds. Nothing occurs in parallel. If you want more frames-per-second, you need to parallelize some of these four operations. For example, let's assume reasonably that the operation 3 is by far the slowest. Here's how we can make 3 run in parallel with the three other operations. You should change the client so that it receives data, process it, and immediately sends an answer to the server; and only then it proceeds to render it. This should be enough in this case, as it takes 1/4th seconds to do this rendering, which should be enough time for the answer to reach the server, the server to render, and the next packet to be sent again.
I ended up here when having same issue with it appearing that socket recv in python to be super slow. The fix for me (after days) was to do something along the lines:
recv_buffer = 2048 # ? guess & check
...
rx_buffer_temp = self._socket.recv(recv_buffer)
rx_buffer_temp_length = len(rx_buffer_temp)
recv_buffer = max(recv_buffer, rx_buffer_temp_length) # keep to the max needed/found
The gist of it is set to the amount of bytes trying to receive closest to the actual expected.
I have 2 UDP responses to a destination ip, one right after the other:
sendsock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
sendsock.bind(('%s' % ip_adr, 1036))
#send first packet
ok_response = "Reception Success, more to come"
str2bytes = bytes(ok_response,'utf-8')
sendsock.sendto(str2bytes, ("%s" % phone.ip_addr, int(phone.sip_port)))
#send second packet
ok_response = "Fun data here"
str2bytes = bytes(ok_response,'utf-8')
sendsock.sendto(str2bytes, ("%s" % phone.ip_addr, int(phone.sip_port)))
I can see with Wireshark the second packet gets sent. But the first seems be ignored.
Unless someone can see a hiccup in my code, is there a way to do an if statement on each sendsock.sendto() instance, to ensure the code doesn't continue until it's acknowledged as sent?
Also, should I be closing the sendsock?
There is no guarantee with UDP that the messages will arrive synchronously or that they will even arrive at all, so it's never actually acknowledged as sent unless you send an acknowledgement response back from the receiver program. That is the tradeoff that improves the speed of UDP versus TCP.
You could, however, check the return value of sendto (number of bytes sent) in a while loop and not exit the while loop until the bytes sent matches the bytes of the original message or a timeout value is reached.
Also, it might be easier to use the socketserver module to simplify the process of handling your sockets.