I am using a script in Python to collect data from a PIC microcontroller via serial port at 2Mbps.
The PIC works with perfect timing at 2Mbps, also the FTDI usb-serial port works great at 2Mbps (both verified with oscilloscope)
Im sending messages (size of about 15 chars) about 100-150x times a second and the number there increments (to check if i have messages being lost and so on)
On my laptop I have Xubuntu running as virtual machine, I can read the serial port via Putty and via my script (python 2.7 and pySerial)
The problem:
When opening the serial port via Putty I see all messages (the counter in the message increments 1 by 1). Perfect!
When opening the serial port via pySerial I see all messages but instead of receiving 100-150x per second i receive them at about 5 per second (still the message increments 1 by 1) but they are probably stored in some buffer as when I power off the PIC, i can go to the kitchen and come back and im still receiving messages.
Here is the code (I omitted most part of the code, but the loop is the same):
ser = serial.Serial('/dev/ttyUSB0', 2000000, timeout=2, xonxoff=False, rtscts=False, dsrdtr=False) #Tried with and without the last 3 parameters, and also at 1Mbps, same happens.
ser.flushInput()
ser.flushOutput()
While True:
data_raw = ser.readline()
print(data_raw)
Anyone knows why pySerial takes so much time to read from the serial port till the end of the line?
Any help?
I want to have this in real time.
Thank you
You can use inWaiting() to get the amount of bytes available at the input queue.
Then you can use read() to read the bytes, something like that:
While True:
bytesToRead = ser.inWaiting()
ser.read(bytesToRead)
Why not to use readline() at this case from Docs:
Read a line which is terminated with end-of-line (eol) character (\n by default) or until timeout.
You are waiting for the timeout at each reading since it waits for eol. the serial input Q remains the same it just a lot of time to get to the "end" of the buffer, To understand it better: you are writing to the input Q like a race car, and reading like an old car :)
A very good solution to this can be found here:
Here's a class that serves as a wrapper to a pyserial object. It
allows you to read lines without 100% CPU. It does not contain any
timeout logic. If a timeout occurs, self.s.read(i) returns an empty
string and you might want to throw an exception to indicate the
timeout.
It is also supposed to be fast according to the author:
The code below gives me 790 kB/sec while replacing the code with
pyserial's readline method gives me just 170kB/sec.
class ReadLine:
def __init__(self, s):
self.buf = bytearray()
self.s = s
def readline(self):
i = self.buf.find(b"\n")
if i >= 0:
r = self.buf[:i+1]
self.buf = self.buf[i+1:]
return r
while True:
i = max(1, min(2048, self.s.in_waiting))
data = self.s.read(i)
i = data.find(b"\n")
if i >= 0:
r = self.buf + data[:i+1]
self.buf[0:] = data[i+1:]
return r
else:
self.buf.extend(data)
ser = serial.Serial('COM7', 9600)
rl = ReadLine(ser)
while True:
print(rl.readline())
You need to set the timeout to "None" when you open the serial port:
ser = serial.Serial(**bco_port**, timeout=None, baudrate=115000, xonxoff=False, rtscts=False, dsrdtr=False)
This is a blocking command, so you are waiting until you receive data that has newline (\n or \r\n) at the end:
line = ser.readline()
Once you have the data, it will return ASAP.
From the manual:
Possible values for the parameter timeout:
…
x set timeout to x seconds
and
readlines(sizehint=None, eol='\n') Read a list of lines,
until timeout. sizehint is ignored and only present for API
compatibility with built-in File objects.
Note that this function only returns on a timeout.
So your readlines will return at most every 2 seconds. Use read() as Tim suggested.
Related
I am trying to code the following using python3 in a raspberry pi:
1) wait for a 14 digits bar code (barcode scanner connected through usb port and input received as keyboard data)
2) after a barcode is read, wait for serial communication (device connected to usb port and sends serial commands. could be one, or more....) the idea is that all commands received are going to be associated with the scanned barcode
3) the process of waiting for serial commands has to stop when a new barcode is read. THIS IS THE PART I HAVE NOT FIGURED OUT HOW TO DO IT
After some research, I decided to use the "readchar" library for the barcode scanner and the "serial" library for the serial communication received. Both of them work by themselves but the problem is when I try to detect both things at the same time.
In the following code, I managed to read a barcode and then wait for 5 lines of serial communication to finally repeat the process and read a barcode again. The program works as it is right now BUT the problem is that I don't know how many lines of serial communication I will receive so I need to somehow detect a new barcode while also waiting to receive the serial communication.
import readchar
import time
import serial
ser = serial.Serial(
port='/dev/ttyUSB0',
baudrate = 115200,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
bytesize=serial.EIGHTBITS,
timeout=1
)
print("Waiting for barcode...")
while 1:
inputStr = ""
while len(inputStr) != 14: #detect only 14 digit barcodes
inputStr += str(readchar.readchar())
inputStr = ''.join(e for e in inputStr if e.isalnum()) #had to add this to strip non alphanumeric characters
currentCode = inputStr
inputStr = ""
print(currentCode)
ser.flushInput()
time.sleep(.1)
# Wait for 5 lines of serial communication
# BUT it should break the while loop when a new barcode is read!
count = 0
while count < 5:
dataRead=ser.readline()
if len(dataRead) > 0:
print(dataRead)
count+=1
print("Waiting for barcode...")
If I add a condition to the while loop that reading the serial communication using (ser.readline()) so that if a character is read from the scanner (readchar.readchar()) then it messes thing up. It is like if readline and reacher can not be in the same while loop.
Doing some research I think I need to use Asynchronous IO, or threads or something like that, but I have no clue. Also I don't know if I could keep using the same libraries (serial and readchar). Please help
I cannot be sure (I don't have your barcode reader and serial port device) but based on what you say I don't think you need threads, you just have to rely on the buffers to keep your data stored until you have time to read them.
Simply change the condition on your second while loop to:
while serial.inWaiting() != 0:
This way you will make sure the RX buffer on your serial port will empty. This approach might or might not work depending on the speed and timing of your devices.
You could also try to add a short delay after the buffer is emptied:
import serial
import time
ser=serial.Serial(port="/dev/ttyUSB0",baudrate=115200, timeout=1.0)
time.sleep(1)
data=b""
timeout = time.time() + 1.0
while ser.inWaiting() or time.time()-timeout < 0.0: #keep reading until the RX buffer is empty and wait for 1 seconds to make sure no more data is coming
if ser.inWaiting() > 0:
data+=ser.read(ser.inWaiting())
timeout = time.time() + 1.0
else:
print("waiting...")
This keeps trying to read from the port for 1 second after the last byte is received, to make sure nothing else is coming. You might want to play with the duration of the delay depending, again, on the speed and timing of your devices.
Again, I don't have your devices, so I'm in no position to judge, but the way you read characters from the barcode/keyboard looks far from optimum. I doubt readchar is the best approach. At the end of the day, your barcode reader is probably a serial port. You might want to dig into that and/or find a more efficient way to read several keyboard strokes in one go.
I found this answer in another question:
How to read keyboard-input?
I have tried it and it works! I´ll also give a try to the method proposed by Marcos G.
I have this short snippet of code here that works fine, but I have problem of getting rid of the hardcoded part.
ser = serial.Serial()
ser.baudrate = 38400
ser.port = '/dev/ttyUSB0'
ser.parity = serial.PARITY_EVEN
ser.timeout = 1
ser.open()
ser.flushInput()
ser.write(command) #command here is a simple request for data to my device
msg = ser.read(200)
ser.close()
While this works fine, the problem I'm having is this. The length of the returned message can vary from 8 byte to almost 200 bytes depending on what was registered. By using a timeout, I prevent my read command from stalling if it doesn't receive 200 bytes. I also don't know ahead the length of the returned message I therefore can't change dynamically the ser.read. Also, there is no constant endline or constant character at the end of the transmission to lock on in a while loop.
Is there a more stable/dynamic way to do this? I could run out of time if the request is too long or I could bust my read buffer without having the complete data transmission. On the other end, increasing the timer mean that my request rate will be slowed down (there is no problem in increasing the read buffer however).
If the reply had a header containing a length field, you could do a fixed-size read() to get the header, then a variable read() to get the rest...
If there is truly no way to tell how big the reply is, then a timeout is the only conceivable solution. However, you have apparently missed the detail that PySerial has two different timeout values: one that applies to the overall operation, and one that applies to gaps between characters. You could set timeout to multiple seconds, so that you never prematurely end a valid reply, and set inter_byte_timeout (was interCharTimeout in older versions) to perhaps 0.1 second, so that your read() will end almost immediately once the device stops sending data. (This assumes that the device never inserts pauses in the middle of sending a reply.)
If the responses (from 8 bytes to 200 bytes) are contiguous, then you could have a loop which concatenates bytes received from calls to ser.read(200), but having set the timeout to something like 1/100th second. Then when you have two timeouts in succession where no bytes were received, then you know you are at the end of the message.
exit = 0
while exit < 2:
more = ser.read(200)
msg += more
if len(more) == 0:
exit += 1
else:
exit = 0
I am trying to read frames sent to a ZigBee module plugged in the USB. Every frame gets discarded by the Python xBee package because the delimiter is 0x00 when it should be 0x7E. Actually it seems that every byte is also zero.
XCTU receives the frames perfectly.
I work with OS X, PyCharm, Python 3.4 and borrowed this code from Internet:
# Open serial port
ser = serial.Serial(PORT, BAUD_RATE)
# Create API object
xbee = ZigBee(ser,escaped=True)
# Continuously read and print packets
while True:
try:
response = xbee.wait_read_frame()
sa = hex(response['source_addr_long'][4:])
rf = hex(response['rf_data'])
datalength=len(rf)
# if datalength is compatible with two floats
# then unpack the 4 byte chunks into floats
if datalength==16:
h=struct.unpack('f',response['rf_data'][0:4])[0]
t=struct.unpack('f',response['rf_data'][4:])[0]
print (sa,' ',rf,' t=',t,'h=',h)
# if it is not two floats show me what I received
else:
print (sa,' ',rf)
except KeyboardInterrupt:
ser.close()
break
ser.close()
The program executes the xbee.wait_read_frame() call and waits there forever because no frame arrives.
I have tracked the call to "base.py" from the xBee package:
while True:
if self._callback and not self._thread_continue:
raise ThreadQuitException
if self.serial.inWaiting() == 0:
time.sleep(.01)
continue
byte = self.serial.read()
if byte != APIFrame.START_BYTE:
continue
The call to serial.read() always returns a zero.
I can't see anything wrong in code fragments you have provided. If you are reading just zeroes (are you?) from that serial port - there most likely something is wrong with serial port settings (e.g. you are reading at 115200 while data is being transmitted at 9600). What is the BAUD_RATE that you are using?
It's also could be a worth to test if you can access device with just dumb terminal.
I am reading string from serial in a loop and realize that the processor is at 100% (RaspberryPI) while waiting for the next serial.read().
I found recommendation to add a few sleeps here and there, but doing this might cause missing serial data. In theorie I am getting a string from serial every 5 seconds, but could be a bit more or less and not in my control.
Is there a way to solve this in python better and with less processor use?
#!/usr/bin/env python
import serial
ser = serial.Serial("/dev/ttyUSB0", 57600, timeout=0)
def sr():
while True:
for line in ser.read():
try:
response = ser.readlines(None)
response = str(response)
print response
except:
print datetime.datetime.now(), " No data from serial connection."
if __name__ == '__main__':
sr
ser.close()
from what i remember (been a while since i used pyserial) i am sure that serial uses buffers, so as long as your message doesn't fill the buffer you shouldn't lose any data.
assuming i'm looking at the docs for the right module the following page:
[Pyserial docs][1]http://pyserial.sourceforge.net/pyserial_api.html
make mention about buffers both on the input and output.
so you should have no problems with putting sleeps into your program as the buffers will collect the data until you read it. (assuming your messages are not big enough to cause an overflow)
James
I am reading serial data like this:
connected = False
port = 'COM4'
baud = 9600
ser = serial.Serial(port, baud, timeout=0)
while not connected:
#serin = ser.read()
connected = True
while True:
print("test")
reading = ser.readline().decode()
The problem is that it prevents anything else from executing including bottle py web framework. Adding sleep() won't help.
Changing "while True"" to "while ser.readline():" doesn't print "test", which is strange since it worked in Python 2.7. Any ideas what could be wrong?
Ideally I should be able to read serial data only when it's available. Data is being sent every 1,000 ms.
Using a separate thread is totally unnecessary. Just follow the example below for your infinite while loop instead.
I use this technique in my eRCaGuy_PyTerm serial terminal program here (search the code for inWaiting() or in_waiting).
Notes:
To check your python3 version, run this:
python3 --version
My output when I first wrote and tested this answer was Python 3.2.3.
To check your pyserial library (serial module) version, run this--I first learned this here:
python3 -c 'import serial; \
print("serial.__version__ = {}".format(serial.__version__))'
This simply imports the serial module and prints its serial.__version__ attribute.
My output as of Oct. 2022 is: serial.__version__ = 3.5.
If your pyserial version is 3.0 or later, use property in_waiting in the code below. If your pyserial version is < 3.0, use function inWaiting() in the code below. See the official pyserial documentation here: https://pyserial.readthedocs.io/en/latest/pyserial_api.html#serial.Serial.in_waiting.
Non-blocking, single-threaded serial read example
import serial
import time # Optional (required if using time.sleep() below)
ser = serial.Serial(port='COM4', baudrate=9600)
while (True):
# Check if incoming bytes are waiting to be read from the serial input
# buffer.
# NB: for PySerial v3.0 or later, use property `in_waiting` instead of
# function `inWaiting()` below!
if (ser.inWaiting() > 0):
# read the bytes and convert from binary array to ASCII
data_str = ser.read(ser.inWaiting()).decode('ascii')
# print the incoming string without putting a new-line
# ('\n') automatically after every print()
print(data_str, end='')
# Put the rest of your code you want here
# Optional, but recommended: sleep 10 ms (0.01 sec) once per loop to let
# other threads on your PC run during this time.
time.sleep(0.01)
This way you only read and print if something is there. You said, "Ideally I should be able to read serial data only when it's available." This is exactly what the code above does. If nothing is available to read, it skips on to the rest of your code in the while loop. Totally non-blocking.
(This answer originally posted & debugged here: Python 3 non-blocking read with pySerial (Cannot get pySerial's "in_waiting" property to work))
pySerial documentation: http://pyserial.readthedocs.io/en/latest/pyserial_api.html
UPDATE:
27 Dec. 2018: added comment about in_waiting vs inWaiting(). Thanks to #FurkanTürkal for pointing that out in the comments below. See documentation here: https://pyserial.readthedocs.io/en/latest/pyserial_api.html#serial.Serial.in_waiting.
27 Oct. 2018: Add sleep to let other threads run.
Documentation: https://docs.python.org/3/library/time.html#time.sleep
Thanks to #RufusV2 for bringing this point up in the comments.
Note on multi-threading:
Even though reading serial data, as shown above, does not require using multiple threads, reading keyboard input in a non-blocking manner does. Therefore, to accomplish non-blocking keyboard input reading, I've written this answer: How to read keyboard input?.
References:
Official pySerial serial.Serial() class API - https://pyserial.readthedocs.io/en/latest/pyserial_api.html
Put it in a separate thread, for example:
import threading
import serial
connected = False
port = 'COM4'
baud = 9600
serial_port = serial.Serial(port, baud, timeout=0)
def handle_data(data):
print(data)
def read_from_port(ser):
while not connected:
#serin = ser.read()
connected = True
while True:
print("test")
reading = ser.readline().decode()
handle_data(reading)
thread = threading.Thread(target=read_from_port, args=(serial_port,))
thread.start()
http://docs.python.org/3/library/threading
I would warn against using blocking IO in a thread. Remember Python has a GIL and at one time only one thread can execute. Now please note that pyserial module is a wrapper over an OS implementation of accessing the serial port. That means it calls code external to the Python. If that code blocks, then the interpreter also get blocked and nothing will execute in the Python program, even the main thread.
This can even happen when using non-blocking IO or timeout based polling if the underlying device driver does not implement timeout well.
A more robust approach is to use multiprocessing module with a queue. Run serial read code in a separate process. This will make sure main and other threads don't block and the program can exit in clean way.
Use a timer driven event to test and read the serial port.
Untested example:
import threading
class serialreading():
def __init__(self):
self.active = True
self.test()
def test(self):
n_in =comport.in_waiting()
if n_in> 0:
self.data = self.data + comport.read(size=n_in)
if len(self.data) > 0:
print(self.data)
self.data=""
if self.active:
threading.Timer(1, test).start() # start new timer of 1 second
def stop(self):
self.active = False