I'm receiving packages where the content has its start not at the beggining of the buffer but at byte 2 or 3 of the buffer. That just happens sometimes. That means the received data are all shifted then.
I set the buffer size to 4096 bytes. I'm using the recv method from the Python sockets and am receving data from a Siemens PLC (if that's a possible source of issues).
Am I missing something out or has the error to be somewhere else than in my program?
I'm trying to send a large stream of data via serial, and using the PySerial library for my purposes. However, the program freezes whenever I need to read more than somewhere between 10,000 and 15,000 bytes.
s = ser.read(10000) works, however s = ser.read(15000) does not.
I've tried flushing the buffer, I've tried reading in one byte at a time with a loop, I've even tried opening and closing the port after calls of less bytes to try and receive all the data I'm sending - but no luck.
Any idea how to receive more data without the program freezing?
I am trying to create a communication between an STM32 and a laptop.
I am trying to receive data from the serial, sent thanks to an STM32. Actual code that I am sending is 0x08 0x09 0x0A 0x0B
I checked on the oscilloscope and I am indeed sending the correct values in the correct order.
What I receive is actually :
b'\n\x0b\x08\t'
I assume that Python is not reading an input that is greater than a 3 bit size, but can not figure out why
Please find my code below :
import serial
ser = serial.Serial('COM3', 115200, bytesize=8)
while 1 :
if(ser.inWaiting() != 0) :
print(ser.read(4))
If someone could help, it would be nice ! :)
check your uart rate, keep the python serial rate the same for stm32
What comes to my mind when looking at pySerial library is that while You initialize Your COM port:
You are not providing read timeout parameter.
You are awaiting 4 bytes of data from serial port.
The problem with such approach is that Your code will wait forever until it gets these 4 bytes and if You are sending the same array from STM32 for me it looks like You received 0x0A,0x0B from older packet and 0x08,0x09 from newer packet, so python code printed these 4 bytes and in fact received into buffer also 0x0A,0x0B of newer packet but it waits until it will receive 2 more bytes before it will be allowed to return with data to print.
Putting here a timeout on read and limiting read argument to single byte might solve Your problem.
Also for further development if You would like to create regular communication between microcontroller and computer I would suggest to move these received bytes into separate buffer and recognize single packets in separate thread parser. In Python it will be painful to create more complex serial communication as even with separate thread it will be quite slow.
I'm working on an embedded system that sends commands via Uart.
Uart works at 115200 baud
On PC side I want to read these commands, parse them and execute the related action.
I choose python as language to build a script.
This is a typical command received from the embedded system:
S;SEND;40;{"ID":"asg01","T":1,"P":{"T":180}};E
Each message starts with S and ends with E.
The command associated to the message is "SEND" and the payload length is 40.
My idea is read the bytes coming from the UART and:
check if the message starts with S
check if the message ends with E
if the above assumptions are true, split the message in order to find the command and the payload.
Which is the best way to parse the all bytes coming from an asynchronous uart?
My concern regards the lost of message due to wrong (or slow) parsing.
Thanks for the help!
BR,
Federico
In my day job, I wrote the software for an embedded system and a PC communicating with each other by a USB cable, using the UART protocol at 115,200 baud.
I see that you tagged your post with PySerial, so you already know about Python's most popular package for serial port communication. I will add that if you are using PyQt, there's a serial module included in that package as well.
115,200 baud is not fast for a modern desktop PC. I doubt that any parsing you do on the PC side will fail to keep up. I parse data streams and plot graphs of my data in real time using PyQt.
What I have noticed in my work with communication between an embedded system and a PC over a UART is that some data gets corrupted occasionally. A byte can be garbled, repeated, or dropped. Also, even if no bytes are added or dropped, you can occasionally perform a read while only part of a packet is in the buffer, and the read will terminate early. If you use a fixed read length of 40 bytes and trust that each read will always line up exactly with a data packet as you show above, you will frequently be wrong.
To solve these kinds of problems, I wrote a FIFO class in Python which consumes serial port data at the head of the FIFO, yields valid data packets at the tail, and discards invalid data. My FIFO holds 3 times as many bytes as my data packets, so if I am looking for packet boundaries using specific sequences, I have plenty of signposts.
A few more recommendations: work in Python 3 if you have the choice, it's cleaner. Use bytes and bytearray objects. Don't use str, because you will find yourself converting back and forth between Unicode and ASCII.
This format is almost parseable as a csv, but not quite, because the fourth field is JSON, and you may not be able to guarantee that the JSON doesn't contain any strings with embedded semicolons. So, I think you probably want to just use string (or, rather, bytes) manipulation functions:
def parsemsg(buf):
s, cmd, length, rest = buf.split(b';', 3)
j, _, e = rest.rpartition(b';')
if s != b'S' or e != b'E':
raise ValueError('must start with S and end with E')
return cmd.decode('utf-8'), int(length), json.loads(j)
Then:
>>> parsemsg(b'S,SEND,40,{"ID":"asg01","T":1,"P":{"T":180}},E')
('SEND', 40, {'ID': 'asg01', 'T': 1, 'P': {'T': 180}})
The actual semicolon-parsing part takes 602ns on my laptop, The decode and int raise that to 902ns. The json.loads, on the other hand, takes 10us. So, if you're worried about performance, the JSON part is really the only part that matters (trying third-party JSON libs I happen to have installed, the fastest one is still 8.1us, which isn't much better). You might as well keep everything else simple and robust.
Also, considering that you're reading this at 115000 baud, you can't get these messages any faster than about 6ms, so spending 11us parsing them is not even close to a problem in the first place.
I'm using Voice Recognition Module -- Arduino Compatible. It says there to use AccessPort, but since I'm on Mac OS X it won't work. I don't know of any other solution but to write my own in Python while I'm open to any other language that works.
I found some solutions over here about pySerial but as of now it doesn't seem to work.
import serial
import struct
def C(val):
return struct.pack('H', val)
ser = serial.Serial(port='/dev/tty.PL2303-00001004', baudrate=9600, timeout=1)
ser.nonblocking()
ser.write(C(0xaa11))
print ser.read().__repr__()
ser.close()
It opens, write two bytes and doesn't read anything. 0xaa11 is used to start recording. My device seems to be in waiting mode since the red LEDs flash rapidly. Even if I do 0xaa00, it doesn't do much either. I even tried to change the byte order to 0x00aa, 0x11aa and so on, but I never receive my 0xcc or 0xe0 error code.
In the documentation it says to send it as hex, so my guess is that accessport converts hex to a Short which use two bytes. 0xaa00 is a two bytes number. I even tried to change the baud rate to 38400 in case someone before me did set that to this speed.
I'd be happy to read anything from that serial port because as of now I can write, it does nothing and reads nothing...
As of now, I'll try to install AccessPort with crossover. If it still doesn't work, I'll have to find a Windows box. I could even write my own bridge with my Arduino board since the driver I'm using might be buggy.
In case it matters, I'm using a USB to Serial PL2303.
Edit
I plugged my output and input pin to test my USB-to-serial. I do read what I write so the driver is working fine.
I'll try to set it up directly with Arduino and test directly from there.
Sending 0xaa11 is different than sending 0xaa and then 0x11. The serial seems to reverse the byte order.
On the other hand, the documentation doesn't tell about bit endianess, so it might be the problem too.
I kind of solved my problem while it was pretty dumb after all.
Here is my final script:
import serial
import struct
from time import sleep
def C(val):
return struct.pack('!H', val)
ser = serial.Serial(port='/dev/tty.PL2303-00002006', baudrate=9600)
sleep(2)
ser.write(C(0xaa00))
sleep(1)
ser.write(C(0xaa36))
sleep(1)
ser.write(C(0xaa11))
sleep(1)
while True:
print ser.readline()
If someone ever want to use that module, it works. It will switch to a mode where it puts to output words instead of bytes so you can read it using readline...
I used in struct.pack "!H" to use network endianess it change 0xaa00 to 0xaa 0x00 like it should be.
But the real problem here wasn't in the script. Since the usb to serial did work I thought that it was strange that the module didn't respond anything... not even a byte for errors. Every command in this module must return something. That's why I looked back at my cable and checked the pins... gnd to gnd, power supply to power supply and then rxd to txd, txd to rxd... While everything seems ok... I should receive something... And then I really thought... what if it's not supposed to use a cross cable and txd should be plugged to txd and rxd to rxd... Apparently it does.
I switch the cable and now data is sent and received.
So anyone using that module should check that it's correctly plugged in. If it doesn't work, switch the data pins.