Develop Reference

Python 3 garbage data into the struct.unpack

I am trying to send and receive messages via socket using Python 3.
BUFFER_SIZE = 1024
# Create message
MLI = struct.pack("!I", len(MESSAGE))
MLI_MESSAGE = MLI + str.encode(MESSAGE)
When the message receive:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
s.send(MLI_MESSAGE)
print ("Sent data: ‘", MESSAGE, "’")
# Receive MLI from response (you might want to add some timeout handling as well
resp = s.recv(BUFFER_SIZE)
resp = struct.unpack("!I", resp)[0]
print(resp)
resp:
b'\x00\t\xeb\x07\xdf\x01\x00\xdf\x02\x010'
I am getting that error:
struct.error: unpack requires a buffer of 4 bytes
I think it is related with \t char into the resp but I am not sure. How can I remove that \t char and how to solve that issue?

You are basically trying to do the following (sockets removed):
1 import struct
2
3 msg = "foobar"
4 mli = struct.pack("!I", len(msg))
5 mli_msg = mli + str.encode(msg)
6
7 len = struct.unpack("!I", mli_msg)[0]
8 print(len)
The extraction of the length in line 7 will fail since you put the whole mli_msg as argument to unpack, not only the expected 4 bytes for the len. Instead you should do:
7 len = struct.unpack("!I", mli_msg[:4])[0]
Apart from that it is wrong to first take the length of the message and then convert the message to bytes. The first takes the number of characters while the latter takes the number of bytes, which will differ when non-ASCII characters are involved - check len("ü") vs. len(str.encode("ü")). You need to first convert the message to bytes thus and then take the length to provide the correct byte length for what you send.
4 encoded_msg = str.encode(msg)
5 mli_msg = struct.pack("!I", len(encoded_msg)) + encoded_msg

Explanation of !I:
! indicates big endian alignment
I indicates unsigned integer type, occupying 4 bytes.
The variable resp value is b'\x00\t\xeb\x07\xdf\x01\x00\xdf\x02\x010', and the length exceeds 4.
You can intercept 4 bytes for parsing, like below.
import struct
resp = b'\x00\t\xeb\x07\xdf\x01\x00\xdf\x02\x010'
print(struct.unpack("!I", resp[:4])[0])
# 649991

How to wait for a large number packet erminator pyserial

I am debugging packets sent from a device with pyserial. The packets have a specific terminator pattern to differentiate between packets of data. The pattern is a large number (0x3423fe67). I am able to receive bytes and concatenate them as they come in with the code below.
ser = serial.Serial(
ports="/dev/ttyUSB0",
baudrate=115200,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
bytesize=serial.EIGHTBITS
)
y = b''
while True:
bytes = ser.inWaiting()
if bytes > 0:
x = ser.read(bytes)
y += binascii.hexify(x)
print(y)
I can see the pattern I'm looking for but how do I grab the packet and terminator once I see the terminator. I tried indexing the bytes object but that returns strange values. Everything received before the terminator is part of the packet.

Here is code that accumulates packets in a list. The code can be used as a guide.
The serialData is data that would be received over the serial port as an example (i.e. ser.read() which would replace the for piece in x: loop). Note that an arbitrary number of bytes can be received, and the test variable n is only used to test the algorithm at arbitrary incoming received character lengths.
The term variable needed hexlify to match the hexlify on the incoming data.
import binascii
import re
term = binascii.hexlify(b'\x34\x23\xfe\x67')
serialData = b'packet0\x34\x23\xfe\x67packet1\x34\x23\xfe\x67'
n = 5
x = [serialData[i:i+n] for i in range(0, len(serialData), n)]
y = b''
packets = []
for piece in x:
y += binascii.hexlify(piece)
for i in re.finditer(term, y):
packets.append(y[:i.start()])
y = y[i.end():]
print('packets = ', packets)

Python bytes convert

I have a confused in processing byte.
My code background is about rs485 communication with current meter.
hex_string is command sent to meter.
I want to bytes convert to string deal recv,
trying to use recv.decode("utf-8") this method,but is invalid.
This is variable of recv content:
b'\x05\x03\x03\x00\x15\x88K' (bytes) ->
0503020015884b (hex)
I want to deal with this paragraph: 0503020015884b (hex) .
Remove 050302 and 884b, keep the middle 0015 and convert 0015 to decimal.
How should i do ?
#!/usr/bin/env python3.7
import serial,time
import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BOARD)
GPIO.setup(40,GPIO.OUT)
GPIO.output(40,GPIO.LOW)
ser = serial.Serial(port='/dev/ttyS0',baurdate=9600,parity='N',bytesize=8,stopbits=2,timeout=1)
hex_string = '0503010000018472'
print ('\nType-hex_string',type(hex_string))
ser.reset_input_buffer()
ser.reset_output_buffer()
GPIO.output(40,GPIO.HIGH)
ser.write(bytes.fromehex(hex_string))
time.sleep(0.01)
GPIO.output(40,GPIO.LOW)
recv = ser.read(11)
print ('\nOrigin:',recv)
print ('\nType-Origin:,'type(recv))
print ('\nHex:',recv.hex())
print ('\nType-recv.hex():',type(recv))

You can use struct.unpack():
from struct import unpack
src = b'\x05\x03\x03\x00\x15\x88K'
res = unpack('>H', src[3: 5])) # remove '>' if byte order is little endian
Also you can use struct.unpack_from():
res = unpack_from('>H', src, 3)
You can use int.from_bytes():
res = int.from_bytes(src[3: 5], byteorder='big')

Once you have the bytes array as recv you can convert the required parts to integer by:
int(recv.hex()[6:10],16)
where 6 and 10 is the range where you have the required value.

Saving Converted Unicode from Serial Port in Python

I am struggling to read in data from an Arduino and save this data as a csv file I could meddle with in Python later. Right now my code reads.
import serial
serial_port = '/dev/ttyUSB0'
baud_rate = 9600
file_path = "output.csv"
ser = serial.Serial(serial_port,baud_rate)
done = False
data = []
while done == False:
raw_bytes = ser.readline()
decoded_bytes = float(raw_bytes.decode("utf-8"))
data.append(decoded_bytes)
if (len(data) > 10) :
done = True
import numpy as np
np.savetxt(file_path, data, delimiter = ',', fmt='%s')
but I'm running into the error
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 1: invalid continuation byte
I want to decode into UTF-8 don't I? What is going wrong? I have checked the Serial Monitor on the Arduino IDE and I am getting correct outputs there. Thanks in advance.

If there's no other way to find out which coding your Arduino IDE uses, you can check/guess the coding on the Arduino side by returning the codes for characters in question via SerialMonitor
void loop () {
int c = Serial.read();
if ( c == -1 ) return; // nothing available
Serial.println (c, HEX); // return the character code in hex notation
}
However the characters you use to convert text into a float number should be plain ASCII, so your
float(raw_bytes.decode("utf-8"))
would probably fail anyway.

on-the-fly parsing of binary serial data in python

I'm new to using Python3 for data acquisition. I'm trying to find a way to parse binary data from a serial port on Linux.
import serial
ser = serial.Serial(
port='/dev/ttyS0',
baudrate = 9600,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
bytesize=serial.EIGHTBITS,
timeout=1)
counter = 0
while 1:
x = ser.read(31)
print (x)
This gives me a string which I'm not sure about the format of:
x='\x00\x00\x91\x00\x02\x88BM\x00\x1c\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\xe1\x00K\x00\x1a\x00\x02\x00\x00'
using
x.encode('hex')
gives a string of hex values
x='000091000288**424d**001c00000001000100000001000100e1004b001a00020000'
where 0x42 is the end of message and 0x4d is start of message.
I can convert it into a base 10 list using
y = map(ord,x)
print(y)
Then I have a way to re-order the message using the indexes but surely there is a neater way? How do I create a list which starts at 0x4d to parse with?

If you are using python3, this is likely already bytes:
x='\x00\x00\x91\x00\x02\x88BM\x00\x1c\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\xe1\x00K\x00\x1a\x00\x02\x00\x00'
It likely looks this way because Python printed it for you, and all of the non-ascii characters are shown in hex. Your start of message is in 0x42, 0x4d which is BM in ascii and can be seen in the data above between 0x88 and 0x00 as \x88BM\x00.
I would suggest just iterating over the byte array in x to do your parsing. The encoding and mapping should not be needed.
for b in x:
if b == 0x4d:
found_byte1 = True
... # etc