Develop Reference

pyserial read variable byte array

I am writing a code that can flexibly read a mutable array (4 byte array and 7 byte array).
I don't think it's a good idea to read 7 bytes continuously and judge it by the number of bytes sent in the 2nd byte.
Reading an array that is sent in 4 bytes as 7 bytes results in slow reception.
serial_port = serial.Serial(
port="/dev/ttyUSB0",
baudrate=9600,
bytesize=serial.EIGHTBITS,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
)
if __name__ == '__main__':
while True :
time.sleep(1)
if serial_port.readable():
serial_port.flushInput()
serial_port.timeout = None
data =serial_port.read(7)
l_data=list(data)
print("Header : " , str.join("", ("0x%02X " % i for i in l_data)))
Results received from connected equipment
ouput:
Header : 0x2B 0x04 0x53 0x7C 0x2B 0x04 0x53 #0x04 is the actual total number of byte
Header : 0x2b 0x07 0x51 0x06 0x07 0x46 0x3a #0x07 is the actual total number of byte
If I replace read() with readline(), nothing comes up.
So did read_until().
(Actually, it is because of my lack of skills.)
I tried to find a solution to find the end of the array (\n or \r) and truncate it, but it didn't work.
Is there a way to flexibly receive 4 bytes and 7 bytes?

Problem with the delimiter of data packet

The following example reads data from the UART. In my case, the delimiter where packet starts with b '\x02' everything works, but the problem appears when b '\x02' appears in the packet and it is not the beginning of the packet because uart data often comes in one string. I will add that the packet that needs to be read always starts with b '\x02\x84' the only question is how to check if the first byte b '\x02’ followed by b '\x84' and if so then do a split.
Sample package consisting of several and with the problem:
\x02\x84"\x00\x19\x03\x00l\xe0\x02D\x00\x02\x84"\x00\x19\x03\x00l
I want to get two as handle_packet:
b'\x84"\x00\x19\x03\x00l\xe0\x02D\x00'
b'\x84"\x00\x19\x03\x00l'
rx_buff = bytes()
while True:
recv = reader.read(-1)
if not recv:
continue
rx_buff += recv
packets = rx_buff.split(bytes(b'\x02'))
for packet in packets:
if not packet:
continue
msg = mod.handle_packet(packet)
if (msg):
get_response(msg)
rx_buff = (bytes() if mod.handle_packet(packet) else packet)

I spent a little more time reading. If you do it this way, your problem is going to be the first packet you split, because, I am assuming it doesn't follow the splitting pattern b'\x02\x84". If that is the case, you can just take the first of split_bytes and remove the first byte.
my_bytes = b'\x02\x84"\x00\x19\x03\x00l\xe0\x02D\x00\x02\x84"\x00\x19\x03\x00l'
split_bytes = my_bytes.split(b'\x02\x84"')
packets = []
for e in split_bytes:
if e != b'':
packets.append(b'\x84"' + e)
print(packets)
Yields:
[b'\x84"\x00\x19\x03\x00l\xe0\x02D\x00', b'\x84"\x00\x19\x03\x00l']
Your desired packets:
[b'\x84"\x00\x19\x03\x00l\xe0\x02D\x00', b'\x84"\x00\x19\x03\x00l']
If I am missing something, please point it out.

How to wait for a large number packet erminator pyserial

I am debugging packets sent from a device with pyserial. The packets have a specific terminator pattern to differentiate between packets of data. The pattern is a large number (0x3423fe67). I am able to receive bytes and concatenate them as they come in with the code below.
ser = serial.Serial(
ports="/dev/ttyUSB0",
baudrate=115200,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
bytesize=serial.EIGHTBITS
)
y = b''
while True:
bytes = ser.inWaiting()
if bytes > 0:
x = ser.read(bytes)
y += binascii.hexify(x)
print(y)
I can see the pattern I'm looking for but how do I grab the packet and terminator once I see the terminator. I tried indexing the bytes object but that returns strange values. Everything received before the terminator is part of the packet.

Here is code that accumulates packets in a list. The code can be used as a guide.
The serialData is data that would be received over the serial port as an example (i.e. ser.read() which would replace the for piece in x: loop). Note that an arbitrary number of bytes can be received, and the test variable n is only used to test the algorithm at arbitrary incoming received character lengths.
The term variable needed hexlify to match the hexlify on the incoming data.
import binascii
import re
term = binascii.hexlify(b'\x34\x23\xfe\x67')
serialData = b'packet0\x34\x23\xfe\x67packet1\x34\x23\xfe\x67'
n = 5
x = [serialData[i:i+n] for i in range(0, len(serialData), n)]
y = b''
packets = []
for piece in x:
y += binascii.hexlify(piece)
for i in re.finditer(term, y):
packets.append(y[:i.start()])
y = y[i.end():]
print('packets = ', packets)

on-the-fly parsing of binary serial data in python

I'm new to using Python3 for data acquisition. I'm trying to find a way to parse binary data from a serial port on Linux.
import serial
ser = serial.Serial(
port='/dev/ttyS0',
baudrate = 9600,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
bytesize=serial.EIGHTBITS,
timeout=1)
counter = 0
while 1:
x = ser.read(31)
print (x)
This gives me a string which I'm not sure about the format of:
x='\x00\x00\x91\x00\x02\x88BM\x00\x1c\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\xe1\x00K\x00\x1a\x00\x02\x00\x00'
using
x.encode('hex')
gives a string of hex values
x='000091000288**424d**001c00000001000100000001000100e1004b001a00020000'
where 0x42 is the end of message and 0x4d is start of message.
I can convert it into a base 10 list using
y = map(ord,x)
print(y)
Then I have a way to re-order the message using the indexes but surely there is a neater way? How do I create a list which starts at 0x4d to parse with?

If you are using python3, this is likely already bytes:
x='\x00\x00\x91\x00\x02\x88BM\x00\x1c\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\xe1\x00K\x00\x1a\x00\x02\x00\x00'
It likely looks this way because Python printed it for you, and all of the non-ascii characters are shown in hex. Your start of message is in 0x42, 0x4d which is BM in ascii and can be seen in the data above between 0x88 and 0x00 as \x88BM\x00.
I would suggest just iterating over the byte array in x to do your parsing. The encoding and mapping should not be needed.
for b in x:
if b == 0x4d:
found_byte1 = True
... # etc

Error reading serial data transmission

Using a raspberry pi I am having issues reading data that is being transmitted serially. My code was working when I tested it on a different machine but isn't working now.
The baud rate is 9600 w/ no parity, 8 data bits, 1 stop bit and I want the program to handle a variable length of characters (Sometimes 100K+). The reading portion of the code is as follows:
ser = serial.Serial('/dev/ttyAMA0', 9600, parity = serial.PARITY_NONE, timeout=1)
While True:
data = ser.read(1)
bytesToRead = ser.inWaiting()
if bytesToRead:
data = data + ser.read(bytesToRead)
encodedData = data.encode('hex')
With this code, Shouldn't I be able to read all the characters as Hex as long as the baud/parity/etc match up with the transmitting system?

While True:
data = ser.read(1)
This waits for a single character to be read (times out after 1s due to the timeout specified in the Serial constructor) and stores it in data
bytesToRead = ser.inWaiting()
if bytesToRead:
data = data + ser.read(bytesToRead)
encodedData = data.encode('hex')
Now instantly check for any other characters in buffer - this will usually be zero. Due to the fact that you're running at 9600 baud, Python will usually see the characters come in one at a time. So your if bytesToRead statement will mostly be false as each incoming character is consumed by the above ser.read(1).
If you just want to process each character individually, you can do:
While True:
data = ser.read(1)
if data:
encodedData = data.encode('hex')
Or if you want to keep adding it to a buffer, use something like:
data = ''
While True:
bytesToRead = ser.inWaiting()
if bytesToRead:
data += ser.read(bytesToRead)
encodedData = data.encode('hex')
if encodedData.startswith('1234deadb33f`):
data = data[6:] # strip 6 chars from start of data