I have a rare bug that seems to occur reading a socket.
It seems, that during reading of data sometimes I get only 1-3 bytes of a data package that is bigger than this.
As I learned from pipe-programming, there I always get at least 512 bytes as long as the sender provides enough data.
Also my sender does at least transmit >= 4 Bytes anytime it does transmit anything -- so I was thinking that at least 4 bytes will be received at once in the beginning (!!) of the transmission.
In 99.9% of all cases, my assumption seems to hold ... but there are really rare cases, when less than 4 bytes are received. It seems to me ridiculous, why the networking system should do this?
Does anybody know more?
Here is the reading-code I use:
mySock, addr = masterSock.accept()
mySock.settimeout(10.0)
result = mySock.recv(BUFSIZE)
# 4 bytes are needed here ...
...
# read remainder of datagram
...
The sender sends the complete datagram with one call of send.
Edit: the whole thing is working on localhost -- so no complicated network applications (routers etc.) are involved. BUFSIZE is at least 512 and the sender sends at least 4 bytes.
I assume you're using TCP. TCP is a stream based protocol with no idea of packets or message boundaries.
This means when you do a read you may get less bytes than you request. If your data is 128k for example you may only get 24k on your first read requiring you to read again to get the rest of the data.
For an example in C:
int read_data(int sock, int size, unsigned char *buf) {
int bytes_read = 0, len = 0;
while (bytes_read < size &&
((len = recv(sock, buf + bytes_read,size-bytes_read, 0)) > 0)) {
bytes_read += len;
}
if (len == 0 || len < 0) doerror();
return bytes_read;
}
As far as I know, this behaviour is perfectly reasonable. Sockets may, and probably will fragment your data as they transmit it. You should be prepared to handle such cases by applying appropriate buffering techniques.
On other hand, if you are transmitting the data on the localhost and you are indeed getting only 4 bytes it probably means you have a bug somewhere else in your code.
EDIT: An idea - try to fire up a packet sniffer and see whenever the packet transmitted will be full or not; this might give you some insight whenever your bug is in your client or in your server.
The simple answer to your question, "Read from socket: Is it guaranteed to at least get x bytes?", is no. Look at the doc strings for these socket methods:
>>> import socket
>>> s = socket.socket()
>>> print s.recv.__doc__
recv(buffersize[, flags]) -> data
Receive up to buffersize bytes from the socket. For the optional flags
argument, see the Unix manual. When no data is available, block until
at least one byte is available or until the remote end is closed. When
the remote end is closed and all data is read, return the empty string.
>>>
>>> print s.settimeout.__doc__
settimeout(timeout)
Set a timeout on socket operations. 'timeout' can be a float,
giving in seconds, or None. Setting a timeout of None disables
the timeout feature and is equivalent to setblocking(1).
Setting a timeout of zero is the same as setblocking(0).
>>>
>>> print s.setblocking.__doc__
setblocking(flag)
Set the socket to blocking (flag is true) or non-blocking (false).
setblocking(True) is equivalent to settimeout(None);
setblocking(False) is equivalent to settimeout(0.0).
From this it is clear that recv() is not required to return as many bytes as you asked for. Also, because you are calling settimeout(10.0), it is possible that some, but not all, data is received near the expiration time for the recv(). In that case recv() will return what it has read - which will be less than you asked for (but consistenty < 4 bytes does seem unlikely).
You mention datagram in your question which implies that you are using (connectionless) UDP sockets (not TCP). The distinction is described here. The posted code does not show socket creation so we can only guess here, however, this detail can be important. It may help if you could post a more complete sample of your code.
If the problem is reproducible you could disable the timeout (which incidentally you do not seem to be handling) and see if that fixes the problem.
This is just the way TCP works. You aren't going to get all of your data at once. There are just too many timing issues between sender and receiver including the senders operating system, NIC, routers, switches, the wires themselves, the receivers NIC, OS, etc. There are buffers in the hardware, and in the OS.
You can't assume that the TCP network is the same as a OS pipe. With the pipe, it's all software so there's no cost in delivering the whole message at once for most messages. With the network, you have to assume there will be timing issues, even in a simple network.
That's why recv() can't give you all the data at once, it may just not be available, even if everything is working right. Normally, you will call recv() and catch the output. That should tell you how many bytes you've received. If it's less than you expect, you need to keep calling recv() (as has been suggested) until you get the correct number of bytes. Be aware that in most cases, recv() returns -1 on error, so check for that and check your documentation for ERRNO values. EAGAIN in particular seems to cause people problems. You can read about it on the internet for details, but if I recall, it means that no data is available at the moment and you should try again.
Also, it sounds like from your post that you're sure the sender is sending the data you need sent, but just to be complete, check this:
http://beej.us/guide/bgnet/output/html/multipage/advanced.html#sendall
You should be doing something similar on the recv() end to handle partial receives. If you have a fixed packet size, you should read until you get the amount of data you expect. If you have a variable packet size, you should read until you have the header that tells you how much data you send(), then read that much more data.
From the Linux man page of recv http://linux.about.com/library/cmd/blcmdl2_recv.htm:
The receive calls normally return any
data available, up to the requested
amount, rather than waiting for
receipt of the full amount requested.
So, if your sender is still transmitting bytes, the call will only give what has been transmitted so far.
If the sender sends 515 bytes, and your BUFSIZE is 512, then the first recv will return 512 bytes, and the next will return 3 bytes... Could this be what's happening?
(This is just one case amongst many which will result in a 3-byte recv from a larger send...)
If you are still interested, patterns like this :
# 4 bytes are needed here ......
# read remainder of datagram...
may create the silly window thing.
Check this out
Use recv_into(...) method from the socket module.
Robert S. Barnes written the example in C.
But you can use Python 2.x with standard python-libraries:
def readReliably(s,n):
buf = bytearray(n)
view = memoryview(buf)
sz = s.recv_into(view,n)
return sz,buf
while True:
sk,skfrom = s.accept()
sz,buf = io.readReliably(sk,4)
a = struct.unpack("4B",buf)
print repr(a)
...
Notice, that sz returned by readReliably() function may be greater than n.
Related
I have this short snippet of code here that works fine, but I have problem of getting rid of the hardcoded part.
ser = serial.Serial()
ser.baudrate = 38400
ser.port = '/dev/ttyUSB0'
ser.parity = serial.PARITY_EVEN
ser.timeout = 1
ser.open()
ser.flushInput()
ser.write(command) #command here is a simple request for data to my device
msg = ser.read(200)
ser.close()
While this works fine, the problem I'm having is this. The length of the returned message can vary from 8 byte to almost 200 bytes depending on what was registered. By using a timeout, I prevent my read command from stalling if it doesn't receive 200 bytes. I also don't know ahead the length of the returned message I therefore can't change dynamically the ser.read. Also, there is no constant endline or constant character at the end of the transmission to lock on in a while loop.
Is there a more stable/dynamic way to do this? I could run out of time if the request is too long or I could bust my read buffer without having the complete data transmission. On the other end, increasing the timer mean that my request rate will be slowed down (there is no problem in increasing the read buffer however).
If the reply had a header containing a length field, you could do a fixed-size read() to get the header, then a variable read() to get the rest...
If there is truly no way to tell how big the reply is, then a timeout is the only conceivable solution. However, you have apparently missed the detail that PySerial has two different timeout values: one that applies to the overall operation, and one that applies to gaps between characters. You could set timeout to multiple seconds, so that you never prematurely end a valid reply, and set inter_byte_timeout (was interCharTimeout in older versions) to perhaps 0.1 second, so that your read() will end almost immediately once the device stops sending data. (This assumes that the device never inserts pauses in the middle of sending a reply.)
If the responses (from 8 bytes to 200 bytes) are contiguous, then you could have a loop which concatenates bytes received from calls to ser.read(200), but having set the timeout to something like 1/100th second. Then when you have two timeouts in succession where no bytes were received, then you know you are at the end of the message.
exit = 0
while exit < 2:
more = ser.read(200)
msg += more
if len(more) == 0:
exit += 1
else:
exit = 0
At the moment I'm trying to write a quick Python program that reads in a .pcap file and writes out data about the various sessions that are stored in there.
The information I write out includes srcip, dstip, srcport and dstport etc.
However, even for a fairly small pcap this takes a lot of memory and ends up running for a very long time. We're talking 8GB+ of memory used for a pcap of a size of 212MB.
As usual I guess there might be a more efficient way of doing this that I'm just unaware of.
Here is a quick skeleton of my code - no vital parts missing.
import socket
from scapy.all import *
edges_file = "edges.csv"
pcap_file = "tcpdump.pcap"
try:
print '[+] Reading and parsing pcap file: %s' % pcap_file
a = rdpcap(pcap_file)
except Exception as e:
print 'Something went wrong while opening/reading the pcap file.' \
'\n\nThe error message is: %s' % e
exit(0)
sessions = a.sessions()
print '[+] Writing to edges.csv'
f1 = open(edges_file, 'w')
f1.write('source,target,protocol,sourceport,destinationport,'
'num_of_packets\n')
for k, v in sessions.iteritems():
tot_packets = len(v)
if "UDP" in k:
proto, source, flurp, target = k.split()
srcip, srcport = source.split(":")
dstip, dstport = target.split(":")
f1.write('%s,%s,%s,%s,%s,%s\n' % (srcip, dstip, proto, srcport,
dstport, tot_packets))
continue
elif "TCP" in k:
proto, source, flurp, target = k.split()
srcip, srcport = source.split(":")
dstip, dstport = target.split(":")
f1.write('%s,%s,%s,%s,%s,%s\n' % (srcip, dstip, proto, srcport,
dstport, tot_packets))
continue
elif "ICMP" in k:
continue # Not bothered about ICMP right now
else:
continue # Or any other 'weird' pacakges for that matter ;)
print '[+] Closing the edges file'
f1.close()
As always - grateful for any assistance.
I know I'm late to the party, but hopefully this will be useful to future visitors.
rdpcap() dissects the entire pcap file and retains an in-memory representation of each and every packet, which explains why it eats up a lot of memory.
As far as I am aware (I am a novice Scapy user myself), the only two ways you can invoke Scapy's session reassembly are:
By calling scapy.plist.PacketList.sessions(). This is what you're currently doing (rdpcap(pcap_file) returns a scapy.plist.PacketList).
By reading the pcap using sniff() in offline mode while also providing the function with a session decoder implementation. For example, for TCP reassembly you'd do sniff(offline='stackoverflow.pcap', session=TCPSession). (This was added in Scapy 2.4.3).
Option 1 is obviously a dead end (as it requires that we keep all packets of all sessions in memory at one time), so let's explore option 2...
Let's launch Scapy in interactive mode to access the documentation for sniff():
$ scapy
>>> help(sniff)
Help on function sniff in module scapy.sendrecv:
sniff(*args, **kwargs)
Sniff packets and return a list of packets.
Args:
count: number of packets to capture. 0 means infinity.
store: whether to store sniffed packets or discard them
prn: function to apply to each packet. If something is returned, it
is displayed.
--Ex: prn = lambda x: x.summary()
session: a session = a flow decoder used to handle stream of packets.
e.g: IPSession (to defragment on-the-flow) or NetflowSession
filter: BPF filter to apply.
lfilter: Python function applied to each packet to determine if
further action may be done.
--Ex: lfilter = lambda x: x.haslayer(Padding)
offline: PCAP file (or list of PCAP files) to read packets from,
instead of sniffing them
timeout: stop sniffing after a given time (default: None).
L2socket: use the provided L2socket (default: use conf.L2listen).
opened_socket: provide an object (or a list of objects) ready to use
.recv() on.
stop_filter: Python function applied to each packet to determine if
we have to stop the capture after this packet.
--Ex: stop_filter = lambda x: x.haslayer(TCP)
iface: interface or list of interfaces (default: None for sniffing
on all interfaces).
monitor: use monitor mode. May not be available on all OS
started_callback: called as soon as the sniffer starts sniffing
(default: None).
The iface, offline and opened_socket parameters can be either an
element, a list of elements, or a dict object mapping an element to a
label (see examples below).
Notice the store parameter. We can set this to False to make sniff() operate in a streamed fashion (read a single packet, process it, then release it from memory):
sniff(offline='stackoverflow.pcap', session=TCPSession, store=False)
I just tested this with a 193 MB pcap. For store=True (default value), this eats up about 1.7 GB of memory on my system (macOS), but only approximately 47 MB when store=False.
Processing the reassembled TCP sessions (open question)
So we managed to reduce our memory footprint - great! But how do we process the (supposedly) reassembled TCP sessions? The usage instructions indicates that we should use the prn parameter of sniff() to specify a callback function that will then be invoked with the reassembled TCP session (emphasis mine):
sniff() also provides Sessions, that allows to dissect a flow of
packets seamlessly. For instance, you may want your sniff(prn=...)
function to automatically defragment IP packets, before executing the
prn.
The example is in the context of IP fragmentation, but I'd expect the TCP analog to be to group all packets of a session and then invoke prn once for each session. Unfortunately, that's not how it works: I tried this on my example pcap, and the callback is invoked once for every packet---exactly as indicated in sniff()'s documentation shown above.
The usage instructions linked above also states the following about using session=TCPSession in sniff():
TCPSession -> defragment certain TCP protocols*. Only HTTP 1.0 currently uses this functionality.
With the output of the experiment above in mind, I now interpret this as that whenever Scapy finds an HTTP (1.0) request/response that spans across multiple TCP segments, it'll create a single packet in which the payload is the merged payload of those TCP segments (which in total is the full HTTP request/response). I'd appreciate it if anyone can help clarify the meaning of the above quote on TCPSession---or even better: clarify if TCP reassembly is indeed possible this way and that I'm just misunderstanding the API.
I'm confused about socket.send() and socket.sendall() functions in Python. As I understand from the documentation send() function uses TCP protocol and sendall() function uses UDP protocol for sending data. I know that TCP is more reliable for most of the Web Applications because we can check which packets are sent and which packets are not. That's why, I think use of send() function can be more reliable rather than sendall() function.
At this point, I want to ask what is the exact difference between these two functions and which one is more reliable for web applications?
Thank you.
socket.send is a low-level method and basically just the C/syscall method send(3) / send(2). It can send less bytes than you requested, but returns the number of bytes sent.
socket.sendall is a high-level Python-only method that sends the entire buffer you pass or throws an exception. It does that by calling socket.send until everything has been sent or an error occurs.
If you're using TCP with blocking sockets and don't want to be bothered
by internals (this is the case for most simple network applications),
use sendall.
And python docs:
Unlike send(), this method continues to send data from string until
either all data has been sent or an error occurs. None is returned on
success. On error, an exception is raised, and there is no way to
determine how much data, if any, was successfully sent
Credits to Philipp Hagemeister for brief description I got in the past.
edit
sendall use under the hood send - take a look on cpython implementation. Here is sample function acting (more or less) like sendall :
def sendall(sock, data, flags=0):
ret = sock.send(data, flags)
if ret > 0:
return sendall(sock, data[ret:], flags)
else:
return None
or from rpython (pypy source):
def sendall(self, data, flags=0, signal_checker=None):
"""Send a data string to the socket. For the optional flags
argument, see the Unix manual. This calls send() repeatedly
until all data is sent. If an error occurs, it's impossible
to tell how much data has been sent."""
with rffi.scoped_nonmovingbuffer(data) as dataptr:
remaining = len(data)
p = dataptr
while remaining > 0:
try:
res = self.send_raw(p, remaining, flags)
p = rffi.ptradd(p, res)
remaining -= res
except CSocketError, e:
if e.errno != _c.EINTR:
raise
if signal_checker is not None:
signal_checker()
Hi i'm quite a newbie to Python. I' writting a simple LAN game (not simple for me) using a pygame module.
Here's the problem - I have two computers (one old intel Atom netbook, the other intel i5 NTB). I want to achieve at least 5 FPS (the netbook is slowering the NTB, but not so much, now i have around 1,5 FPS), but calling recv() function twice a main loop takes total around 0,5 seconds on each machine. The wifi signal is strong and the router is 300Mbit/s and it sends a short roughly 500-character string. As you can see for measuring time i use time.clock().
Here's the part of the "server" code, which i usually run on the i5 NTB:
while 1:
start = time.clock()
messagelen = c.recv(4) #length of the following message (fixed 4 character)
if " " in messagelen:
messagelen = messagelen.replace(" ","")
message = cPickle.loads(c.recv(int(messagelen))) #list of the arrows, other player position and changes in the game map
arrowsmod = message[0]
modtankposan = message[1]
removelistmod = message[2]
for i in removelistmod:
try:
randopos.remove(i)
except ValueError:
randopossv.remove(i)
print time.clock()-start
tosendlist=[]
if len(arrows) == 0: #if there are no arrows it appends only an empty list
tosendlist.append([])
else:
tosendlist.append(arrows)
tosendlist.append([zeltankpos, 360-angle])
if len(removelist) == 0: #if there are no changes of the map it appends only an empty list
tosendlist.append([])
else:
tosendlist.append(removelist)
removelist=[]
tosend=cPickle.dumps(tosendlist)
tosendlen = str(len(tosend))
while len(tosendlen)<4:
tosendlen+=" "
c.sendall(tosendlen) #sends the length to client
c.sendall(tosend) #sends the actual message(dumped list of lists) to client
...something else which takes <0,05 sec on the NTB
Here's the part of the "client" game code (just inverted the beginning - sending/receiving parts):
while 1:
tosendlist=[]
if len(arrows) == 0: #if there are no arrows it appends only an empty list
tosendlist.append([])
else:
tosendlist.append(arrows)
tosendlist.append([zeltankpos, 360-angle])
if len(removelist) == 0: #if there are no changes of the map it appends only an empty list
tosendlist.append([])
else:
tosendlist.append(removelist)
removelist=[]
tosend=cPickle.dumps(tosendlist)
tosendlen = str(len(tosend))
while len(tosendlen)<4:
tosendlen+=" "
s.sendall(tosendlen) #sends the length to server
s.sendall(tosend) #sends the actual message(dumped list of lists) to server
start = time.clock()
messagelen = s.recv(4) #length of the following message (fixed 4 character)
if " " in messagelen:
messagelen = messagelen.replace(" ","")
message = cPickle.loads(s.recv(int(messagelen))) #list of the arrows, other player position and changes in the game map
arrowsmod = message[0]
modtankposan = message[1]
removelistmod = message[2]
for i in removelistmod:
try:
randopos.remove(i)
except ValueError:
randopossv.remove(i)
print time.clock()-start
... rest which takes on the old netbook <0,17 sec
When I run let's say a single player version of the game on one machine (without the socket module) on the i5 NTB it has 50 FPS in the up left corner of the map and 25 FPS in the down right corner (the 1000x1000 pixel map contains 5x5 pixel squares, i think it's slower because of the bigger coordinates, but i can't believe that so much. BTW recv while ran as a LAN game in the down right corner of the map takes approx. the same time)
on the Atom netbook it has 4-8 FPS.
So could you please tell me, why it's so slow? The computers are not synchronized, one is faster, the other slower, but it can't be that they are waiting for each other, it would be max 0,17 secs delay, right? And plus the long recv calling would be only on the faster computer?
Also I don't exactly know how the send/recv function work. It's weird the sendall takes literally no time while receiving takes 0,5 secs. Maybe sendall
is trying to send in the background while the rest of the program continues forward.
As mentioned by Armin Rigo, recv will return after packets are received by the socket, but packets don't necessarily need to be transmitted immediately after calling send. While send returns immediately, OS caches the data internally and might wait some time for more data being written to the the socket before actually transmitting it; this is called Nagle's algorithm and avoids sending lots of small packets over the network. You can disable it and push packets quicker to the wire; try enabling TCP_NODELAY options on the sending socket (or both if your communication is bidirectional), by calling this:
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
This could potentially reduce amount of time recv is sleeping due to no data.
As the Wikipedia states:
This algorithm interacts badly with TCP delayed acknowledgments, a
feature introduced into TCP at roughly the same time in the early
1980s, but by a different group. With both algorithms enabled,
applications that do two successive writes to a TCP connection,
followed by a read that will not be fulfilled until after the data
from the second write has reached the destination, experience a
constant delay of up to 500 milliseconds, the "ACK delay". For this
reason, TCP implementations usually provide applications with an
interface to disable the Nagle algorithm. This is typically called the
TCP_NODELAY option.
There is a mention of 0.5s which you're seeing in your benchmark, so this might be a reason.
Yes, send() or sendall() will occur in the background (unless the connexion is saturated right now, i.e. there is already too much data waiting to be sent). By contrast, recv() will immediately get the data only if it arrived already, but if none did, it waits. Then it returns possibly a fraction of it. (I am assuming that c is a TCP socket, not a UDP one.) Note that you should not assume that recv(N) returns N bytes; you should write a function like this:
def recvall(c, n):
data = []
while n > 0:
s = c.recv(n)
if not s: raise EOFError
data.append(s)
n -= len(s)
return ''.join(data)
Anyway, to the point. The issue is not the speed of recv(). If I understood correctly, there are four operations:
the server renders (1/25th sec)
the server sends something on the socket, received by the client;
the client renters (1/4th sec);
the client send something back on the socket.
This takes almost (0.3 + 2 * network_delay) seconds. Nothing occurs in parallel. If you want more frames-per-second, you need to parallelize some of these four operations. For example, let's assume reasonably that the operation 3 is by far the slowest. Here's how we can make 3 run in parallel with the three other operations. You should change the client so that it receives data, process it, and immediately sends an answer to the server; and only then it proceeds to render it. This should be enough in this case, as it takes 1/4th seconds to do this rendering, which should be enough time for the answer to reach the server, the server to render, and the next packet to be sent again.
I ended up here when having same issue with it appearing that socket recv in python to be super slow. The fix for me (after days) was to do something along the lines:
recv_buffer = 2048 # ? guess & check
...
rx_buffer_temp = self._socket.recv(recv_buffer)
rx_buffer_temp_length = len(rx_buffer_temp)
recv_buffer = max(recv_buffer, rx_buffer_temp_length) # keep to the max needed/found
The gist of it is set to the amount of bytes trying to receive closest to the actual expected.
Is it me, or can I not find a good tutorial on non-blocking sockets in python?
I'm not sure how to exactly work the .recv and the .send in it. According to the python docs, (my understanding of it, at least) the recv'ed or send'ed data might be only partial data. So does that mean I have to somehow concatenate the data while recv and make sure all data sends through in send. If so, how? An example would be much appreciated.
It doesn't really matter if your socket is in non-blocking mode or not, recv/send work pretty much the same; the only difference is that non-blocking socket throws 'Resource temporarily unavailable' error instead of waiting for data/socket.
recv method returns numbers of bytes received, which is told to be less or equal to the passed bufsize. If you want to receive exactly size bytes, you should do something similar to the following code:
def recvall(sock, size):
data = ''
while len(data) < size:
d = sock.recv(size - len(data))
if not d:
# Connection closed by remote host, do what best for you
return None
data += d
return data
This is important to remember, that in blocking mode you have to do exactly the same. (The number of bytes passed to application layer is for example limited by recv buffer size in the OS.)
send method returns number of bytes sent, which is told to be less or equal to the length of passed string. If you want to ensure the whole message was sent, you should do something similar to the following code:
def sendall(sock, data):
while data:
sent = sock.send(data)
data = data[sent:]
You can use sock.sendall directly, but (according to the documentation) on error, an exception is raised, and there is no way to determine how much data, if any, was successfully sent.
The sockets in Python follow the BSD socket API and behave in the similar way to c-style sockets (the difference is, for example, they throw exception instead of returning error code). You should be happy with any socket tutorial on the web and manpages.
Keep bytes you want to send in a buffer. (A list of byte-strings would be best, since you don't have to concatenate them.) Use the fcntl.fcntl function to set the socket in non-blocking mode:
import fcntl, os
fcntl.fcntl(mysocket, fcntl.F_SETFL, os.O_NONBLOCK)
Then select.select will tell you when it is OK to read and write to the socket. (Writing when it is not OK will give you the EAGAIN error in non-blocking mode.) When you write, check the return value to see how many bytes were actually written. Eliminate that many bytes from your buffer. If you use the list-of-strings approach, you only need to try writing the first string each time.
If you read the empty string, your socket has closed.