Develop Reference

not able to receive complete data from serial port

This is the link to my github code. I want to receive serial data from com port which is of size 8196 characters..
but every time I get different data and different data size and not exactly 8196 chars..
I have tried to adjust time delays and I also included flushinput() but didn't get the complete data properly
import serial
import time
ser=serial.Serial(port="COM5",baudrate=57600)
ser.write(serial.to_bytes(cmd))
time.sleep(0.5)
data=ser.read()
time.sleep(1)
data_left=ser.inWaiting()
data+=ser.read(data_left)
ser.flushInput()
print("Data: ",hex(int.from_bytes(data,byteorder="big")))
I expect to receive complete data on my console window...

You need to keep reading on the RX buffer until it's empty. For that you can use a loop:
import serial
import time
ser=serial.Serial(port="COM5",baudrate=57600, timeout=1.0)
ser.write(serial.to_bytes(cmd))
time.sleep(1)
data=b""
timeout = time.time() + 3.0
while ser.inWaiting() or time.time()-timeout < 0.0: #keep reading until the RX buffer is empty and wait for 3 seconds to make sure no more data is coming
if ser.inWaiting() > 0:
data+=ser.read(ser.inWaiting())
timeout = time.time() + 3.0
else:
print("waiting...")
ser.flushInput()
print("Data: ",hex(int.from_bytes(data,byteorder="big")))
The timer on the loop is intended to avoid the loop to finish when the receiver is trying to read faster than the speed at which the data is arriving.
EDIT: After looking a bit deeper I realized why the code above was still not working.
My (wrong) understanding was that ser.read() would read the number of bytes indicated by ser.inWaiting() but as it turns out (you just have to look at the code!) ser.read() is exactly the same as ser.read(1).
As discussed, reading 1 byte at a time result in so much overhead that the RX buffer overflows. To fix that, you can just add the number of bytes available as an argument to the reading function:
ser.read(ser.inWaiting())
Apologies for the confusion on this.

recv() function too slow

Hi i'm quite a newbie to Python. I' writting a simple LAN game (not simple for me) using a pygame module.
Here's the problem - I have two computers (one old intel Atom netbook, the other intel i5 NTB). I want to achieve at least 5 FPS (the netbook is slowering the NTB, but not so much, now i have around 1,5 FPS), but calling recv() function twice a main loop takes total around 0,5 seconds on each machine. The wifi signal is strong and the router is 300Mbit/s and it sends a short roughly 500-character string. As you can see for measuring time i use time.clock().
Here's the part of the "server" code, which i usually run on the i5 NTB:
while 1:
start = time.clock()
messagelen = c.recv(4) #length of the following message (fixed 4 character)
if " " in messagelen:
messagelen = messagelen.replace(" ","")
message = cPickle.loads(c.recv(int(messagelen))) #list of the arrows, other player position and changes in the game map
arrowsmod = message[0]
modtankposan = message[1]
removelistmod = message[2]
for i in removelistmod:
try:
randopos.remove(i)
except ValueError:
randopossv.remove(i)
print time.clock()-start
tosendlist=[]
if len(arrows) == 0: #if there are no arrows it appends only an empty list
tosendlist.append([])
else:
tosendlist.append(arrows)
tosendlist.append([zeltankpos, 360-angle])
if len(removelist) == 0: #if there are no changes of the map it appends only an empty list
tosendlist.append([])
else:
tosendlist.append(removelist)
removelist=[]
tosend=cPickle.dumps(tosendlist)
tosendlen = str(len(tosend))
while len(tosendlen)<4:
tosendlen+=" "
c.sendall(tosendlen) #sends the length to client
c.sendall(tosend) #sends the actual message(dumped list of lists) to client
...something else which takes <0,05 sec on the NTB
Here's the part of the "client" game code (just inverted the beginning - sending/receiving parts):
while 1:
tosendlist=[]
if len(arrows) == 0: #if there are no arrows it appends only an empty list
tosendlist.append([])
else:
tosendlist.append(arrows)
tosendlist.append([zeltankpos, 360-angle])
if len(removelist) == 0: #if there are no changes of the map it appends only an empty list
tosendlist.append([])
else:
tosendlist.append(removelist)
removelist=[]
tosend=cPickle.dumps(tosendlist)
tosendlen = str(len(tosend))
while len(tosendlen)<4:
tosendlen+=" "
s.sendall(tosendlen) #sends the length to server
s.sendall(tosend) #sends the actual message(dumped list of lists) to server
start = time.clock()
messagelen = s.recv(4) #length of the following message (fixed 4 character)
if " " in messagelen:
messagelen = messagelen.replace(" ","")
message = cPickle.loads(s.recv(int(messagelen))) #list of the arrows, other player position and changes in the game map
arrowsmod = message[0]
modtankposan = message[1]
removelistmod = message[2]
for i in removelistmod:
try:
randopos.remove(i)
except ValueError:
randopossv.remove(i)
print time.clock()-start
... rest which takes on the old netbook <0,17 sec
When I run let's say a single player version of the game on one machine (without the socket module) on the i5 NTB it has 50 FPS in the up left corner of the map and 25 FPS in the down right corner (the 1000x1000 pixel map contains 5x5 pixel squares, i think it's slower because of the bigger coordinates, but i can't believe that so much. BTW recv while ran as a LAN game in the down right corner of the map takes approx. the same time)
on the Atom netbook it has 4-8 FPS.
So could you please tell me, why it's so slow? The computers are not synchronized, one is faster, the other slower, but it can't be that they are waiting for each other, it would be max 0,17 secs delay, right? And plus the long recv calling would be only on the faster computer?
Also I don't exactly know how the send/recv function work. It's weird the sendall takes literally no time while receiving takes 0,5 secs. Maybe sendall
is trying to send in the background while the rest of the program continues forward.

As mentioned by Armin Rigo, recv will return after packets are received by the socket, but packets don't necessarily need to be transmitted immediately after calling send. While send returns immediately, OS caches the data internally and might wait some time for more data being written to the the socket before actually transmitting it; this is called Nagle's algorithm and avoids sending lots of small packets over the network. You can disable it and push packets quicker to the wire; try enabling TCP_NODELAY options on the sending socket (or both if your communication is bidirectional), by calling this:
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
This could potentially reduce amount of time recv is sleeping due to no data.
As the Wikipedia states:
This algorithm interacts badly with TCP delayed acknowledgments, a
feature introduced into TCP at roughly the same time in the early
1980s, but by a different group. With both algorithms enabled,
applications that do two successive writes to a TCP connection,
followed by a read that will not be fulfilled until after the data
from the second write has reached the destination, experience a
constant delay of up to 500 milliseconds, the "ACK delay". For this
reason, TCP implementations usually provide applications with an
interface to disable the Nagle algorithm. This is typically called the
TCP_NODELAY option.
There is a mention of 0.5s which you're seeing in your benchmark, so this might be a reason.

Yes, send() or sendall() will occur in the background (unless the connexion is saturated right now, i.e. there is already too much data waiting to be sent). By contrast, recv() will immediately get the data only if it arrived already, but if none did, it waits. Then it returns possibly a fraction of it. (I am assuming that c is a TCP socket, not a UDP one.) Note that you should not assume that recv(N) returns N bytes; you should write a function like this:
def recvall(c, n):
data = []
while n > 0:
s = c.recv(n)
if not s: raise EOFError
data.append(s)
n -= len(s)
return ''.join(data)
Anyway, to the point. The issue is not the speed of recv(). If I understood correctly, there are four operations:
the server renders (1/25th sec)
the server sends something on the socket, received by the client;
the client renters (1/4th sec);
the client send something back on the socket.
This takes almost (0.3 + 2 * network_delay) seconds. Nothing occurs in parallel. If you want more frames-per-second, you need to parallelize some of these four operations. For example, let's assume reasonably that the operation 3 is by far the slowest. Here's how we can make 3 run in parallel with the three other operations. You should change the client so that it receives data, process it, and immediately sends an answer to the server; and only then it proceeds to render it. This should be enough in this case, as it takes 1/4th seconds to do this rendering, which should be enough time for the answer to reach the server, the server to render, and the next packet to be sent again.

I ended up here when having same issue with it appearing that socket recv in python to be super slow. The fix for me (after days) was to do something along the lines:
recv_buffer = 2048 # ? guess & check
...
rx_buffer_temp = self._socket.recv(recv_buffer)
rx_buffer_temp_length = len(rx_buffer_temp)
recv_buffer = max(recv_buffer, rx_buffer_temp_length) # keep to the max needed/found
The gist of it is set to the amount of bytes trying to receive closest to the actual expected.

Reading serial data in realtime in Python

I am using a script in Python to collect data from a PIC microcontroller via serial port at 2Mbps.
The PIC works with perfect timing at 2Mbps, also the FTDI usb-serial port works great at 2Mbps (both verified with oscilloscope)
Im sending messages (size of about 15 chars) about 100-150x times a second and the number there increments (to check if i have messages being lost and so on)
On my laptop I have Xubuntu running as virtual machine, I can read the serial port via Putty and via my script (python 2.7 and pySerial)
The problem:
When opening the serial port via Putty I see all messages (the counter in the message increments 1 by 1). Perfect!
When opening the serial port via pySerial I see all messages but instead of receiving 100-150x per second i receive them at about 5 per second (still the message increments 1 by 1) but they are probably stored in some buffer as when I power off the PIC, i can go to the kitchen and come back and im still receiving messages.
Here is the code (I omitted most part of the code, but the loop is the same):
ser = serial.Serial('/dev/ttyUSB0', 2000000, timeout=2, xonxoff=False, rtscts=False, dsrdtr=False) #Tried with and without the last 3 parameters, and also at 1Mbps, same happens.
ser.flushInput()
ser.flushOutput()
While True:
data_raw = ser.readline()
print(data_raw)
Anyone knows why pySerial takes so much time to read from the serial port till the end of the line?
Any help?
I want to have this in real time.
Thank you

You can use inWaiting() to get the amount of bytes available at the input queue.
Then you can use read() to read the bytes, something like that:
While True:
bytesToRead = ser.inWaiting()
ser.read(bytesToRead)
Why not to use readline() at this case from Docs:
Read a line which is terminated with end-of-line (eol) character (\n by default) or until timeout.
You are waiting for the timeout at each reading since it waits for eol. the serial input Q remains the same it just a lot of time to get to the "end" of the buffer, To understand it better: you are writing to the input Q like a race car, and reading like an old car :)

A very good solution to this can be found here:
Here's a class that serves as a wrapper to a pyserial object. It
allows you to read lines without 100% CPU. It does not contain any
timeout logic. If a timeout occurs, self.s.read(i) returns an empty
string and you might want to throw an exception to indicate the
timeout.
It is also supposed to be fast according to the author:
The code below gives me 790 kB/sec while replacing the code with
pyserial's readline method gives me just 170kB/sec.
class ReadLine:
def __init__(self, s):
self.buf = bytearray()
self.s = s
def readline(self):
i = self.buf.find(b"\n")
if i >= 0:
r = self.buf[:i+1]
self.buf = self.buf[i+1:]
return r
while True:
i = max(1, min(2048, self.s.in_waiting))
data = self.s.read(i)
i = data.find(b"\n")
if i >= 0:
r = self.buf + data[:i+1]
self.buf[0:] = data[i+1:]
return r
else:
self.buf.extend(data)
ser = serial.Serial('COM7', 9600)
rl = ReadLine(ser)
while True:
print(rl.readline())

You need to set the timeout to "None" when you open the serial port:
ser = serial.Serial(**bco_port**, timeout=None, baudrate=115000, xonxoff=False, rtscts=False, dsrdtr=False)
This is a blocking command, so you are waiting until you receive data that has newline (\n or \r\n) at the end:
line = ser.readline()
Once you have the data, it will return ASAP.

From the manual:
Possible values for the parameter timeout:
…
x set timeout to x seconds
and
readlines(sizehint=None, eol='\n') Read a list of lines,
until timeout. sizehint is ignored and only present for API
compatibility with built-in File objects.
Note that this function only returns on a timeout.
So your readlines will return at most every 2 seconds. Use read() as Tim suggested.

Using PySerial is it possible to wait for data?

I've got a Python program which is reading data from a serial port via the PySerial module. The two conditions I need to keep in mind are: I don't know how much data will arrive, and I don't know when to expect data.
Based on this I have came up with the follow code snippets:
#Code from main loop, spawning thread and waiting for data
s = serial.Serial(5, timeout=5) # Open COM5, 5 second timeout
s.baudrate = 19200
#Code from thread reading serial data
while 1:
tdata = s.read(500) # Read 500 characters or 5 seconds
if(tdata.__len__() > 0): #If we got data
if(self.flag_got_data is 0): #If it's the first data we recieved, store it
self.data = tdata
else: #if it's not the first, append the data
self.data += tdata
self.flag_got_data = 1
So this code will loop forever getting data off the serial port. We'll get up to 500 characters store the data, then alert the main loop by setting a flag. If no data is present we'll just go back to sleep and wait.
The code is working, but I don't like the 5s timeout. I need it because I don't know how much data to expect, but I don't like that it's waking up every 5 seconds even when no data is present.
Is there any way to check when data becomes available before doing the read? I'm thinking something like the select command in Linux.
Note: I found the inWaiting() method, but really that seems it just change my "sleep" to a poll, so that's not what I want here. I just want to sleep until data comes in, then go get it.

Ok, I actually got something together that I like for this. Using a combination of read() with no timeout and the inWaiting() method:
#Modified code from main loop:
s = serial.Serial(5)
#Modified code from thread reading the serial port
while 1:
tdata = s.read() # Wait forever for anything
time.sleep(1) # Sleep (or inWaiting() doesn't give the correct value)
data_left = s.inWaiting() # Get the number of characters ready to be read
tdata += s.read(data_left) # Do the read and combine it with the first character
... #Rest of the code
This seems to give the results I wanted, I guess this type of functionality doesn't exist as a single method in Python

You can set timeout = None, then the read call will block until the requested number of bytes are there. If you want to wait until data arrives, just do a read(1) with timeout None. If you want to check data without blocking, do a read(1) with timeout zero, and check if it returns any data.
(see documentation https://pyserial.readthedocs.io/en/latest/)

def cmd(cmd,serial):
out='';prev='101001011'
serial.flushInput();serial.flushOutput()
serial.write(cmd+'\r');
while True:
out+= str(serial.read(1))
if prev == out: return out
prev=out
return out
call it like this:
cmd('ATZ',serial.Serial('/dev/ttyUSB0', timeout=1, baudrate=115000))

Read from socket: Is it guaranteed to at least get x bytes?

I have a rare bug that seems to occur reading a socket.
It seems, that during reading of data sometimes I get only 1-3 bytes of a data package that is bigger than this.
As I learned from pipe-programming, there I always get at least 512 bytes as long as the sender provides enough data.
Also my sender does at least transmit >= 4 Bytes anytime it does transmit anything -- so I was thinking that at least 4 bytes will be received at once in the beginning (!!) of the transmission.
In 99.9% of all cases, my assumption seems to hold ... but there are really rare cases, when less than 4 bytes are received. It seems to me ridiculous, why the networking system should do this?
Does anybody know more?
Here is the reading-code I use:
mySock, addr = masterSock.accept()
mySock.settimeout(10.0)
result = mySock.recv(BUFSIZE)
# 4 bytes are needed here ...
...
# read remainder of datagram
...
The sender sends the complete datagram with one call of send.
Edit: the whole thing is working on localhost -- so no complicated network applications (routers etc.) are involved. BUFSIZE is at least 512 and the sender sends at least 4 bytes.

I assume you're using TCP. TCP is a stream based protocol with no idea of packets or message boundaries.
This means when you do a read you may get less bytes than you request. If your data is 128k for example you may only get 24k on your first read requiring you to read again to get the rest of the data.
For an example in C:
int read_data(int sock, int size, unsigned char *buf) {
int bytes_read = 0, len = 0;
while (bytes_read < size &&
((len = recv(sock, buf + bytes_read,size-bytes_read, 0)) > 0)) {
bytes_read += len;
}
if (len == 0 || len < 0) doerror();
return bytes_read;
}

As far as I know, this behaviour is perfectly reasonable. Sockets may, and probably will fragment your data as they transmit it. You should be prepared to handle such cases by applying appropriate buffering techniques.
On other hand, if you are transmitting the data on the localhost and you are indeed getting only 4 bytes it probably means you have a bug somewhere else in your code.
EDIT: An idea - try to fire up a packet sniffer and see whenever the packet transmitted will be full or not; this might give you some insight whenever your bug is in your client or in your server.

The simple answer to your question, "Read from socket: Is it guaranteed to at least get x bytes?", is no. Look at the doc strings for these socket methods:
>>> import socket
>>> s = socket.socket()
>>> print s.recv.__doc__
recv(buffersize[, flags]) -> data
Receive up to buffersize bytes from the socket. For the optional flags
argument, see the Unix manual. When no data is available, block until
at least one byte is available or until the remote end is closed. When
the remote end is closed and all data is read, return the empty string.
>>>
>>> print s.settimeout.__doc__
settimeout(timeout)
Set a timeout on socket operations. 'timeout' can be a float,
giving in seconds, or None. Setting a timeout of None disables
the timeout feature and is equivalent to setblocking(1).
Setting a timeout of zero is the same as setblocking(0).
>>>
>>> print s.setblocking.__doc__
setblocking(flag)
Set the socket to blocking (flag is true) or non-blocking (false).
setblocking(True) is equivalent to settimeout(None);
setblocking(False) is equivalent to settimeout(0.0).
From this it is clear that recv() is not required to return as many bytes as you asked for. Also, because you are calling settimeout(10.0), it is possible that some, but not all, data is received near the expiration time for the recv(). In that case recv() will return what it has read - which will be less than you asked for (but consistenty < 4 bytes does seem unlikely).
You mention datagram in your question which implies that you are using (connectionless) UDP sockets (not TCP). The distinction is described here. The posted code does not show socket creation so we can only guess here, however, this detail can be important. It may help if you could post a more complete sample of your code.
If the problem is reproducible you could disable the timeout (which incidentally you do not seem to be handling) and see if that fixes the problem.

This is just the way TCP works. You aren't going to get all of your data at once. There are just too many timing issues between sender and receiver including the senders operating system, NIC, routers, switches, the wires themselves, the receivers NIC, OS, etc. There are buffers in the hardware, and in the OS.
You can't assume that the TCP network is the same as a OS pipe. With the pipe, it's all software so there's no cost in delivering the whole message at once for most messages. With the network, you have to assume there will be timing issues, even in a simple network.
That's why recv() can't give you all the data at once, it may just not be available, even if everything is working right. Normally, you will call recv() and catch the output. That should tell you how many bytes you've received. If it's less than you expect, you need to keep calling recv() (as has been suggested) until you get the correct number of bytes. Be aware that in most cases, recv() returns -1 on error, so check for that and check your documentation for ERRNO values. EAGAIN in particular seems to cause people problems. You can read about it on the internet for details, but if I recall, it means that no data is available at the moment and you should try again.
Also, it sounds like from your post that you're sure the sender is sending the data you need sent, but just to be complete, check this:
http://beej.us/guide/bgnet/output/html/multipage/advanced.html#sendall
You should be doing something similar on the recv() end to handle partial receives. If you have a fixed packet size, you should read until you get the amount of data you expect. If you have a variable packet size, you should read until you have the header that tells you how much data you send(), then read that much more data.

From the Linux man page of recv http://linux.about.com/library/cmd/blcmdl2_recv.htm:
The receive calls normally return any
data available, up to the requested
amount, rather than waiting for
receipt of the full amount requested.
So, if your sender is still transmitting bytes, the call will only give what has been transmitted so far.

If the sender sends 515 bytes, and your BUFSIZE is 512, then the first recv will return 512 bytes, and the next will return 3 bytes... Could this be what's happening?
(This is just one case amongst many which will result in a 3-byte recv from a larger send...)

If you are still interested, patterns like this :
# 4 bytes are needed here ......
# read remainder of datagram...
may create the silly window thing.
Check this out

Use recv_into(...) method from the socket module.
Robert S. Barnes written the example in C.
But you can use Python 2.x with standard python-libraries:
def readReliably(s,n):
buf = bytearray(n)
view = memoryview(buf)
sz = s.recv_into(view,n)
return sz,buf
while True:
sk,skfrom = s.accept()
sz,buf = io.readReliably(sk,4)
a = struct.unpack("4B",buf)
print repr(a)
...
Notice, that sz returned by readReliably() function may be greater than n.