I've got a Python program which is reading data from a serial port via the PySerial module. The two conditions I need to keep in mind are: I don't know how much data will arrive, and I don't know when to expect data.
Based on this I have came up with the follow code snippets:
#Code from main loop, spawning thread and waiting for data
s = serial.Serial(5, timeout=5) # Open COM5, 5 second timeout
s.baudrate = 19200
#Code from thread reading serial data
while 1:
tdata = s.read(500) # Read 500 characters or 5 seconds
if(tdata.__len__() > 0): #If we got data
if(self.flag_got_data is 0): #If it's the first data we recieved, store it
self.data = tdata
else: #if it's not the first, append the data
self.data += tdata
self.flag_got_data = 1
So this code will loop forever getting data off the serial port. We'll get up to 500 characters store the data, then alert the main loop by setting a flag. If no data is present we'll just go back to sleep and wait.
The code is working, but I don't like the 5s timeout. I need it because I don't know how much data to expect, but I don't like that it's waking up every 5 seconds even when no data is present.
Is there any way to check when data becomes available before doing the read? I'm thinking something like the select command in Linux.
Note: I found the inWaiting() method, but really that seems it just change my "sleep" to a poll, so that's not what I want here. I just want to sleep until data comes in, then go get it.
Ok, I actually got something together that I like for this. Using a combination of read() with no timeout and the inWaiting() method:
#Modified code from main loop:
s = serial.Serial(5)
#Modified code from thread reading the serial port
while 1:
tdata = s.read() # Wait forever for anything
time.sleep(1) # Sleep (or inWaiting() doesn't give the correct value)
data_left = s.inWaiting() # Get the number of characters ready to be read
tdata += s.read(data_left) # Do the read and combine it with the first character
... #Rest of the code
This seems to give the results I wanted, I guess this type of functionality doesn't exist as a single method in Python
You can set timeout = None, then the read call will block until the requested number of bytes are there. If you want to wait until data arrives, just do a read(1) with timeout None. If you want to check data without blocking, do a read(1) with timeout zero, and check if it returns any data.
(see documentation https://pyserial.readthedocs.io/en/latest/)
def cmd(cmd,serial):
out='';prev='101001011'
serial.flushInput();serial.flushOutput()
serial.write(cmd+'\r');
while True:
out+= str(serial.read(1))
if prev == out: return out
prev=out
return out
call it like this:
cmd('ATZ',serial.Serial('/dev/ttyUSB0', timeout=1, baudrate=115000))
Related
This is the link to my github code. I want to receive serial data from com port which is of size 8196 characters..
but every time I get different data and different data size and not exactly 8196 chars..
I have tried to adjust time delays and I also included flushinput() but didn't get the complete data properly
import serial
import time
ser=serial.Serial(port="COM5",baudrate=57600)
ser.write(serial.to_bytes(cmd))
time.sleep(0.5)
data=ser.read()
time.sleep(1)
data_left=ser.inWaiting()
data+=ser.read(data_left)
ser.flushInput()
print("Data: ",hex(int.from_bytes(data,byteorder="big")))
I expect to receive complete data on my console window...
You need to keep reading on the RX buffer until it's empty. For that you can use a loop:
import serial
import time
ser=serial.Serial(port="COM5",baudrate=57600, timeout=1.0)
ser.write(serial.to_bytes(cmd))
time.sleep(1)
data=b""
timeout = time.time() + 3.0
while ser.inWaiting() or time.time()-timeout < 0.0: #keep reading until the RX buffer is empty and wait for 3 seconds to make sure no more data is coming
if ser.inWaiting() > 0:
data+=ser.read(ser.inWaiting())
timeout = time.time() + 3.0
else:
print("waiting...")
ser.flushInput()
print("Data: ",hex(int.from_bytes(data,byteorder="big")))
The timer on the loop is intended to avoid the loop to finish when the receiver is trying to read faster than the speed at which the data is arriving.
EDIT: After looking a bit deeper I realized why the code above was still not working.
My (wrong) understanding was that ser.read() would read the number of bytes indicated by ser.inWaiting() but as it turns out (you just have to look at the code!) ser.read() is exactly the same as ser.read(1).
As discussed, reading 1 byte at a time result in so much overhead that the RX buffer overflows. To fix that, you can just add the number of bytes available as an argument to the reading function:
ser.read(ser.inWaiting())
Apologies for the confusion on this.
I have this short snippet of code here that works fine, but I have problem of getting rid of the hardcoded part.
ser = serial.Serial()
ser.baudrate = 38400
ser.port = '/dev/ttyUSB0'
ser.parity = serial.PARITY_EVEN
ser.timeout = 1
ser.open()
ser.flushInput()
ser.write(command) #command here is a simple request for data to my device
msg = ser.read(200)
ser.close()
While this works fine, the problem I'm having is this. The length of the returned message can vary from 8 byte to almost 200 bytes depending on what was registered. By using a timeout, I prevent my read command from stalling if it doesn't receive 200 bytes. I also don't know ahead the length of the returned message I therefore can't change dynamically the ser.read. Also, there is no constant endline or constant character at the end of the transmission to lock on in a while loop.
Is there a more stable/dynamic way to do this? I could run out of time if the request is too long or I could bust my read buffer without having the complete data transmission. On the other end, increasing the timer mean that my request rate will be slowed down (there is no problem in increasing the read buffer however).
If the reply had a header containing a length field, you could do a fixed-size read() to get the header, then a variable read() to get the rest...
If there is truly no way to tell how big the reply is, then a timeout is the only conceivable solution. However, you have apparently missed the detail that PySerial has two different timeout values: one that applies to the overall operation, and one that applies to gaps between characters. You could set timeout to multiple seconds, so that you never prematurely end a valid reply, and set inter_byte_timeout (was interCharTimeout in older versions) to perhaps 0.1 second, so that your read() will end almost immediately once the device stops sending data. (This assumes that the device never inserts pauses in the middle of sending a reply.)
If the responses (from 8 bytes to 200 bytes) are contiguous, then you could have a loop which concatenates bytes received from calls to ser.read(200), but having set the timeout to something like 1/100th second. Then when you have two timeouts in succession where no bytes were received, then you know you are at the end of the message.
exit = 0
while exit < 2:
more = ser.read(200)
msg += more
if len(more) == 0:
exit += 1
else:
exit = 0
I have a python script which will parse xml file for serial numbers and will write them to a text file. The problem with the below code is, It is going on infinite loop. If I am adding a break statement some where after logging to a file, It is writing only one serial number. How do I increase the counter, so that the program will exit after writing all the serial numbers.
try:
while True:
data, addr = s.recvfrom(65507)
mylist=data.split('\r')
url = re.findall('http?://(?:[a-zA-Z]|[0-9]|[$-_#.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', data)
print url[0]
response = urllib2.urlopen(url[0])
the_page = response.read()
tree = ET.XML(the_page)
with open("temp.xml", "w") as f:
f.write(ET.tostring(tree))
document = parse('temp.xml')
actors = document.getElementsByTagName("ns0:serialNumber")
for act in actors:
for node in act.childNodes:
if node.nodeType == node.TEXT_NODE:
r = "{}".format(node.data)
print r
logToFile(str(r))
time.sleep(10)
s.sendto(msg, ('239.255.255.250', 1900) )
except socket.timeout:
pass
I would normally create a flag so that the while would be
while working == True:
Then reset the flag at the appropriate time.
This allows you to use the else statement to close the text file and output the final results after the while loop is complete. Else clause on Python while statement.
Note that it is always better to explicitly close open files when finished rather than relying on garbage collection. You should also close the file and output a timeout message in the except logic.
For debugging, you can output a statement at each write to the text file.
If your s.recvfrom(65507) is working correctly it should be an easy fix. Write this code just below your data, addr = s.recvfrom(65507)
if not data:
break
You open a UDP socket and you use recvfrom to get data from the socket.
You set a high timeout which makes this function a blocking function. It means when you start listening on the socket, if no data have been sent from the sender your program will be blocked on that line until either the sender sends something or the timeout reaches. In case of timeout and no data the function will raise an Exception.
I see two options:
Send something from the sender that indicates the end of stream (the serial numbers in your case).
Set a small timeout then catch the Exception and use it to break the loop.
Also, take a look at this question: socket python : recvfrom
Hope it helps.
I am writing python code that reads and writes to a serial device. The device is basically an Arduino Mega running the Marlin 3D printer firmware.
My python code is sending a series of GCode commands (ASCII strings terminated by newlines, including checksums and line numbers). Marlin responds to each successfully received line with an "ok\n". Marlin only has a limited line buffer size so if it is full Marlin will hold off on sending the "ok\n" response until space is freed up.
If the checksum fails then Marlin requests the line to be sent again with a "Resend: 143\n" response. Another possible response is "ok T:{temp value}\n" if the current temperature is requested.
My code uses three threads. The main thread, a read thread and a write thread. Here is a stripped down version of the code:
class Printer:
def connect(self):
self.s = serial.Serial(self.port, self.baudrate, timeout=3)
self.ok_received.set()
def _start_print_thread(self):
self.print_thread = Thread(target=self._empty_buffer, name='Print')
self.print_thread.setDaemon(True)
self.print_thread.start()
def _start_read_thread(self):
self.read_thread = Thread(target=self._continous_read, name='Read')
self.read_thread.setDaemon(True)
self.read_thread.start()
def _empty_buffer(self):
while not self.stop_printing:
if self.current_line_idx < len(self.buffer):
while not self.ok_received.is_set() and not self.stop_printing:
logger.debug('waiting on ok_received')
self.ok_received.wait(2)
line = self._next_line()
self.s.write(line)
self.current_line_idx += 1
self.ok_received.clear()
else:
break
def _continous_read(self):
while not self.stop_reading:
if self.s is not None:
line = self.s.readline()
if line == 'ok\n':
self.ok_received.set()
continue # if we got an OK then we need to do nothing else.
if 'Resend:' in line: # example line: "Resend: 143"
self.current_line_idx = int(line.split()[1]) - 1
if line: # if we received _anything_ then set the flag
self.ok_received.set()
else: # if no printer is attached, wait 10ms to check again.
sleep(0.01)
In the above code, self.ok_received is a threading.Event. This mostly works ok. Once every couple of hours however it gets stuck in the while not self.ok_received.is_set() and not self.stop_printing: loop inside of _empty_buffer(). This kills the print by locking up the machine.
When stuck inside the loop, I can get the print to continue by sending any command manually. This allows the read thread to set the ok_recieved flag.
Since Marlin does not respond with checksums, I guess it is possible the "ok\n" gets garbled. The third if statement in the read thread is supposed to handle this by setting the flag if anything is received from Marlin.
So my question is: Do I have a possible race condition somewhere? Before I add locks all over the place or combine the two threads into one I would really like to understand how this is failing. Any advice would be greatly appreciated.
It looks like the read thread could get some data in the window where the write thread has broken out of the is_set loop, but has not yet called self.ok_received.clear(). So, the read thread ends up calling self.ok_received.set() while the write thread is still processing the previous line, and then the write thread unknowingly calls clear() once its done processing the previous message, and never knows that another line should be written.
def _empty_buffer(self):
while not self.stop_printing:
if self.current_line_idx < len(self.buffer):
while not self.ok_received.is_set() and not self.stop_printing:
logger.debug('waiting on ok_received')
self.ok_received.wait(2)
# START OF RACE WINDOW
line = self._next_line()
self.s.write(line)
self.current_line_idx += 1
# END OF RACE WINDOW
self.ok_received.clear()
else:
break
A Queue might be a good way to handle this - you want to write one line in the write thread every time the read thread receives a line. If you replaced self.ok_received.set() with self.recv_queue.put("line"), then the write thread could just write one line every time it pulls something from the Queue:
def _empty_buffer(self):
while not self.stop_printing:
if self.current_line_idx < len(self.buffer):
while not self.stop_printing:
logger.debug('waiting on ok_received')
try:
val = self.recv_queue.get(timeout=2)
except Queue.Empty:
pass
else:
break
line = self._next_line()
self.s.write(line)
self.current_line_idx += 1
else:
break
You could also shrink the window to the point you probably won't hit it in practice by moving the call to self.ok_received.clear() up immediately after exiting the inner while loop, but technically there will still be a race.
I am reading string from serial in a loop and realize that the processor is at 100% (RaspberryPI) while waiting for the next serial.read().
I found recommendation to add a few sleeps here and there, but doing this might cause missing serial data. In theorie I am getting a string from serial every 5 seconds, but could be a bit more or less and not in my control.
Is there a way to solve this in python better and with less processor use?
#!/usr/bin/env python
import serial
ser = serial.Serial("/dev/ttyUSB0", 57600, timeout=0)
def sr():
while True:
for line in ser.read():
try:
response = ser.readlines(None)
response = str(response)
print response
except:
print datetime.datetime.now(), " No data from serial connection."
if __name__ == '__main__':
sr
ser.close()
from what i remember (been a while since i used pyserial) i am sure that serial uses buffers, so as long as your message doesn't fill the buffer you shouldn't lose any data.
assuming i'm looking at the docs for the right module the following page:
[Pyserial docs][1]http://pyserial.sourceforge.net/pyserial_api.html
make mention about buffers both on the input and output.
so you should have no problems with putting sleeps into your program as the buffers will collect the data until you read it. (assuming your messages are not big enough to cause an overflow)
James