Develop Reference

Python - serial does not read correct avlue

I am trying to create a communication between an STM32 and a laptop.
I am trying to receive data from the serial, sent thanks to an STM32. Actual code that I am sending is 0x08 0x09 0x0A 0x0B
I checked on the oscilloscope and I am indeed sending the correct values in the correct order.
What I receive is actually :
b'\n\x0b\x08\t'
I assume that Python is not reading an input that is greater than a 3 bit size, but can not figure out why
Please find my code below :
import serial
ser = serial.Serial('COM3', 115200, bytesize=8)
while 1 :
if(ser.inWaiting() != 0) :
print(ser.read(4))
If someone could help, it would be nice ! :)

check your uart rate, keep the python serial rate the same for stm32

What comes to my mind when looking at pySerial library is that while You initialize Your COM port:
You are not providing read timeout parameter.
You are awaiting 4 bytes of data from serial port.
The problem with such approach is that Your code will wait forever until it gets these 4 bytes and if You are sending the same array from STM32 for me it looks like You received 0x0A,0x0B from older packet and 0x08,0x09 from newer packet, so python code printed these 4 bytes and in fact received into buffer also 0x0A,0x0B of newer packet but it waits until it will receive 2 more bytes before it will be allowed to return with data to print.
Putting here a timeout on read and limiting read argument to single byte might solve Your problem.
Also for further development if You would like to create regular communication between microcontroller and computer I would suggest to move these received bytes into separate buffer and recognize single packets in separate thread parser. In Python it will be painful to create more complex serial communication as even with separate thread it will be quite slow.

Best approach to parse buffer string in python

I'm working on an embedded system that sends commands via Uart.
Uart works at 115200 baud
On PC side I want to read these commands, parse them and execute the related action.
I choose python as language to build a script.
This is a typical command received from the embedded system:
S;SEND;40;{"ID":"asg01","T":1,"P":{"T":180}};E
Each message starts with S and ends with E.
The command associated to the message is "SEND" and the payload length is 40.
My idea is read the bytes coming from the UART and:
check if the message starts with S
check if the message ends with E
if the above assumptions are true, split the message in order to find the command and the payload.
Which is the best way to parse the all bytes coming from an asynchronous uart?
My concern regards the lost of message due to wrong (or slow) parsing.
Thanks for the help!
BR,
Federico

In my day job, I wrote the software for an embedded system and a PC communicating with each other by a USB cable, using the UART protocol at 115,200 baud.
I see that you tagged your post with PySerial, so you already know about Python's most popular package for serial port communication. I will add that if you are using PyQt, there's a serial module included in that package as well.
115,200 baud is not fast for a modern desktop PC. I doubt that any parsing you do on the PC side will fail to keep up. I parse data streams and plot graphs of my data in real time using PyQt.
What I have noticed in my work with communication between an embedded system and a PC over a UART is that some data gets corrupted occasionally. A byte can be garbled, repeated, or dropped. Also, even if no bytes are added or dropped, you can occasionally perform a read while only part of a packet is in the buffer, and the read will terminate early. If you use a fixed read length of 40 bytes and trust that each read will always line up exactly with a data packet as you show above, you will frequently be wrong.
To solve these kinds of problems, I wrote a FIFO class in Python which consumes serial port data at the head of the FIFO, yields valid data packets at the tail, and discards invalid data. My FIFO holds 3 times as many bytes as my data packets, so if I am looking for packet boundaries using specific sequences, I have plenty of signposts.
A few more recommendations: work in Python 3 if you have the choice, it's cleaner. Use bytes and bytearray objects. Don't use str, because you will find yourself converting back and forth between Unicode and ASCII.

This format is almost parseable as a csv, but not quite, because the fourth field is JSON, and you may not be able to guarantee that the JSON doesn't contain any strings with embedded semicolons. So, I think you probably want to just use string (or, rather, bytes) manipulation functions:
def parsemsg(buf):
s, cmd, length, rest = buf.split(b';', 3)
j, _, e = rest.rpartition(b';')
if s != b'S' or e != b'E':
raise ValueError('must start with S and end with E')
return cmd.decode('utf-8'), int(length), json.loads(j)
Then:
>>> parsemsg(b'S,SEND,40,{"ID":"asg01","T":1,"P":{"T":180}},E')
('SEND', 40, {'ID': 'asg01', 'T': 1, 'P': {'T': 180}})
The actual semicolon-parsing part takes 602ns on my laptop, The decode and int raise that to 902ns. The json.loads, on the other hand, takes 10us. So, if you're worried about performance, the JSON part is really the only part that matters (trying third-party JSON libs I happen to have installed, the fastest one is still 8.1us, which isn't much better). You might as well keep everything else simple and robust.
Also, considering that you're reading this at 115000 baud, you can't get these messages any faster than about 6ms, so spending 11us parsing them is not even close to a problem in the first place.

PySerial does not receive data correctly

I have a little problem receiving data correctly via pySerial: it often does not read the full data or too much of it. Sometimes and sometimes more often, there are additional characters or some characters/parts of the sended data are missing. It seems, PC and the emitter of the data are not synchronised correctly.
In the current example I use a arduino, sending 'Hello World' to the serial port of my PC (os is Ubuntu 14.04), with the following simple code:
void setup(){
Serial.begin(9600);
Serial.print("Programme initiated\n");
}
I use the following python3 code to receive the data:
import serial
import time
arduino = serial.Serial(port, baudrate=9600, timeout=2)
print(arduino.isOpen)
print(arduino)
time.sleep(1)
while True:
print(arduino.readline())
This is pretty much a simple tutorial example, and here is what I receive (apart from the correct stuff):
b'PrProgramme initiated\n'
or
b'PProgramme initiated\n'
or
b'ProgProgramme initiated\n'
or
b'ogramme initiated\n'
I moved on with more complex problems in my code, but still I didn't solved that problem. When sending a message in a loop from the arduino (the standard hello world code), it often needs time to stabilise (while that, it again does only show the middle fragment of the data) and after that running quite stable, but even then it sometimes breaks down single lines.
I faced the same difficulties when communicating with a multimeter device. There, it often does not read the first characters or mixes up with previous data.
Did anyone faced that problem before? I think it is a question of synchronisation, but I don't know how to solve it. And what puzzles me, is that I really only used tutorial stuff and it doesn't seem to work properly. Or is it a configuration problem of my PC?

What you are looking at is happening because some different things are going on.
First of all every time you open the serial port, you are causing what is called and "autoreset" and the Arduino reboot. That can be avoided in hardware or even in software by explicitly disabling RST signal on open. How to do that may vary and is out of scope of the question.
Then we have to understand that serial does NOT wait for the other part to be listening to send data; so if you disable the autoreset and connecting to the Arduino you should see random part of output of the program, depending of its current state.
Finally we have some buffer capability on pc (and sometimes even on the UART to USB side), so its not true that if you are not listening that data get lost, it may be still in the buffer.
We could say the first 3 artifact may be given by buffered data + reboot (this happen a lot when you send a lot of data, and that break the autoupload of code and you have to do a manual procedure), while the last one may be something that prevented the buffer to fill, maybe it was disabled by you, maybe some weird timing opening the serial, maybe you disabled the autoreset, maybe time that the arduino got enumerated part of the message was gone.

Sending and reading ASCII values in pySerial

I am trying to communicating with a scale that does communication in ASCII format using python and pySerial. I have no experience how ever using ASCII format. So I have basic questions.
How would I send a a character T for example using pySerial and terminate it with CRLF using ASCII format?
I tried
myserialport.write('TCRLF')
myserialport.write('T\r\n')
myserialport.write('T\n\r')
I am also trying to read data from the scale which I would expect to be in a form of '208.01 g' for example. But when I use
myserialport.read(10)
or
myserialport.readline(10)
I get this from the scale
]ëýýÿ]W
ÿ]u_u]ÿ]uÕ
ýWýWë]uÝõW
ÿ½õÿ½WW]Ýý
WýW]Wÿ½ÿ×ë
From googling it seems as pySerial should receive data in ASCII format by default, and send it as well...but I am lost as to why its not working. Any help would be appreciated.

This is the right way to send a character with CRLF to a serial port:
myserialport.write('T\r\n')
Regarding messy response - make sure that you set the baudrate, the number of data bits, stop bits and parity bits correctly. You can find the required values in the scale's datasheet.
For example:
from serial import Serial, SEVENBITS, STOPBITS_ONE, PARITY_EVEN
myserialport = Serial('/dev/ttyS0', baudrate=9600, bytesize=SEVENBITS, parity=PARITY_EVEN, stopbits=STOPBITS_ONE)

The issue was ground connection on USB to Serial Converter (Pin 7 needed to be grounded). If you have same issue, check your pin out. (Fisher Scintific scales use Pin 7 as ground, which is not normal as pin 5 is ground...odd...) Thanks all.

Sending hex data over RS-232 between an Arduino and a Mac OS X system

I'm using Voice Recognition Module -- Arduino Compatible. It says there to use AccessPort, but since I'm on Mac OS X it won't work. I don't know of any other solution but to write my own in Python while I'm open to any other language that works.
I found some solutions over here about pySerial but as of now it doesn't seem to work.
import serial
import struct
def C(val):
return struct.pack('H', val)
ser = serial.Serial(port='/dev/tty.PL2303-00001004', baudrate=9600, timeout=1)
ser.nonblocking()
ser.write(C(0xaa11))
print ser.read().__repr__()
ser.close()
It opens, write two bytes and doesn't read anything. 0xaa11 is used to start recording. My device seems to be in waiting mode since the red LEDs flash rapidly. Even if I do 0xaa00, it doesn't do much either. I even tried to change the byte order to 0x00aa, 0x11aa and so on, but I never receive my 0xcc or 0xe0 error code.
In the documentation it says to send it as hex, so my guess is that accessport converts hex to a Short which use two bytes. 0xaa00 is a two bytes number. I even tried to change the baud rate to 38400 in case someone before me did set that to this speed.
I'd be happy to read anything from that serial port because as of now I can write, it does nothing and reads nothing...
As of now, I'll try to install AccessPort with crossover. If it still doesn't work, I'll have to find a Windows box. I could even write my own bridge with my Arduino board since the driver I'm using might be buggy.
In case it matters, I'm using a USB to Serial PL2303.
Edit
I plugged my output and input pin to test my USB-to-serial. I do read what I write so the driver is working fine.
I'll try to set it up directly with Arduino and test directly from there.
Sending 0xaa11 is different than sending 0xaa and then 0x11. The serial seems to reverse the byte order.
On the other hand, the documentation doesn't tell about bit endianess, so it might be the problem too.

I kind of solved my problem while it was pretty dumb after all.
Here is my final script:
import serial
import struct
from time import sleep
def C(val):
return struct.pack('!H', val)
ser = serial.Serial(port='/dev/tty.PL2303-00002006', baudrate=9600)
sleep(2)
ser.write(C(0xaa00))
sleep(1)
ser.write(C(0xaa36))
sleep(1)
ser.write(C(0xaa11))
sleep(1)
while True:
print ser.readline()
If someone ever want to use that module, it works. It will switch to a mode where it puts to output words instead of bytes so you can read it using readline...
I used in struct.pack "!H" to use network endianess it change 0xaa00 to 0xaa 0x00 like it should be.
But the real problem here wasn't in the script. Since the usb to serial did work I thought that it was strange that the module didn't respond anything... not even a byte for errors. Every command in this module must return something. That's why I looked back at my cable and checked the pins... gnd to gnd, power supply to power supply and then rxd to txd, txd to rxd... While everything seems ok... I should receive something... And then I really thought... what if it's not supposed to use a cross cable and txd should be plugged to txd and rxd to rxd... Apparently it does.
I switch the cable and now data is sent and received.
So anyone using that module should check that it's correctly plugged in. If it doesn't work, switch the data pins.