I send some data from an arduino using pySerial.
My Data looks like
bytearray(DST, SRC, STATUS, TYPE, CHANNEL, DATA..., SIZEOFDATA)
where sizeofData is a test that all bytes are received.
The problem is, every time when a byte is zero, my python program just stops reading there:
serial_port = serial.Serial("/dev/ttyUSB0")
while serial_port.isOpen():
response_header_str = serial_port.readline()
format = '>';
format += ('B'*len(response_header_str));
response_header = struct.unpack(format, response_header_str)
pprint(response_header)
serial_port.close()
For example, when I send bytearray(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15) everything is fine. But when I send something like bytearray(1,2,3,4,0,1,2,3,4) I don't see everything beginning with the zero.
The problem is that I cannot avoid sending zeros as I am just sending the "memory dump" e.g. when I send a float value, there might be zero bytes.
how can I tell pyserial not to ignore zero bytes.
I've looked through the source of PySerial and the problem is in PySerial's implementation of FileLike.readline (in http://svn.code.sf.net/p/pyserial/code/trunk/pyserial/serial/serialutil.py). The offending function is:
def readline(self, size=None, eol=LF):
"""\
Read a line which is terminated with end-of-line (eol) character
('\n' by default) or until timeout.
"""
leneol = len(eol)
line = bytearray()
while True:
c = self.read(1)
if c:
line += c
if line[-leneol:] == eol:
break
if size is not None and len(line) >= size:
break
else:
break
return bytes(line)
With the obvious problem being the if c: line. When c == b'\x00' this evaluates to false, and the routine breaks out of the read loop. The easiest thing to do would be to reimplement this yourself as something like:
def readline(port, size=None, eol="\n"):
"""\
Read a line which is terminated with end-of-line (eol) character
('\n' by default) or until timeout.
"""
leneol = len(eol)
line = bytearray()
while True:
line += port.read(1)
if line[-leneol:] == eol:
break
if size is not None and len(line) >= size:
break
return bytes(line)
To clarify from your comments, this is a replacement for the Serial.readline method that will consume null-bytes and add them to the returned string until it hits the eol character, which we define here as "\n".
An example of using the new method, with a file-object substituted for the socket:
>>> # Create some example data terminated by a newline containing nulls.
>>> handle = open("test.dat", "wb")
>>> handle.write(b"hell\x00o, w\x00rld\n")
>>> handle.close()
>>>
>>> # Use our readline method to read it back in.
>>> handle = open("test.dat", "rb")
>>> readline(handle)
'hell\x00o, w\x00rld\n'
Hopefully this makes a little more sense.
Related
I'm very new to python and I'm trying to read pressure data from a Honeywell differential pressure sensor using the LibMPSSE library. I'm reading it from a Adafruit FT232H chip and I'm using python 2.7.6 on ubuntu Linux.
#!/usr/bin/env python
from mpsse import *
SIZE = 2 # 2 bytes MSB first
WCMD = "\x50" # Write start address command
RCMD = "\x51" # Read command
FOUT = "eeprom.txt" # Output file
try:
eeprom = MPSSE(I2C)
# print "%s initialized at %dHz (I2C)" % (eeprom.GetDescription(), eeprom.GetClock())
eeprom.Start()
eeprom.Write(WCMD)
# Send Start condition to i2c-slave (Pressure Sensor)
if eeprom.GetAck() == ACK:
# ACK received,resend START condition and set R/W bit to 1 for read
eeprom.Start()
eeprom.Write(RCMD)
if eeprom.GetAck() == ACK:
# ACK recieved, continue supply the clock to slave
data = eeprom.Read(SIZE)
eeprom.SendNacks()
eeprom.Read(1)
else:
raise Exception("Received read command NACK2!")
else:
raise Exception("Received write command NACK1!")
eeprom.Stop()
print(data)
eeprom.Close()
except Exception, e:
print "MPSSE failure:", e
According to the library, Read returns a string of size bytes and whenever I want to print the data, the only output I see is �����2��T_`ʋ�Q}�*/�eE�
. I've tried encoding with utf-8 and still no luck.
Python may print "weird" output when bytes/characters in the string (data in your case) contain non-printable characters. To "view" the content of the individual bytes (as integers or hex), do the following:
print(','.join(['{:d}'.format(x) for x in map(ord, data)])) # decimal
or
print(','.join(['{:02X}'.format(x) for x in map(ord, data)]))
Since the length of your data buffer is set by SIZE=2, to extract each byte from this buffer as an integer you can do the following:
hi, lo = map(ord, data)
# or:
hi = ord(data[0])
lo = ord(data[1])
To read more about ord and what it does - see https://docs.python.org/2/library/functions.html#ord
I'd like to test the content of a variable containing a byte in a way like this:
line = []
while True:
for c in self.ser.read(): # read() from pySerial
line.append(c)
if c == binascii.unhexlify('0A').decode('utf8'):
print("Line: " + line)
line = []
break
But this does not work...
I'd like also to test, if a byte is empty:
In this case
print(self.ser.read())
prints: b'' (with two single quotes)
I do not until now succeed to test this
if self.ser.read() == b''
or what ever always shows a syntax error...
I know, very basic, but I don't get it...
Thank you for your help. The first part of the question was answerd by #sisanared:
if self.ser.read():
does the test for an empty byte
The second part of the question (the end-of-line with the hex-value 0A) stil doesn't work, but I think it is whise to close this question since the answer to the title is given.
Thank you all
If you want to verify the contents of your variable or string which you want to read from pySerial, use the repr() function, something like:
import serial
import repr as reprlib
from binascii import unhexlify
self.ser = serial.Serial(self.port_name, self.baudrate, self.bytesize, self.parity, self.stopbits, self.timeout, self.xonxoff, self.rtscts)
line = []
while 1:
for c in self.ser.read(): # read() from pySerial
line.append(c)
if if c == b'\x0A':
print("Line: " + line)
print repr(unhexlify(''.join('0A'.split())).decode('utf8'))
line = []
break
I open my file like so :
f = open("filename.ext", "rb") # ensure binary reading with b
My first line of data looks like this (when using f.readline()):
'\x04\x00\x00\x00\x12\x00\x00\x00\x04\x00\x00\x00\xb4\x00\x00\x00\x01\x00\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x18\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00\x05\x00\x00\x00\x06\x00\x00\x00:\x00\x00\x00;\x00\x00\x00<\x00\x00\x007\x00\x00\x008\x00\x00\x009\x00\x00\x00\x07\x00\x00\x00\x08\x00\x00\x00\t\x00\x00\x00\n'
Thing is, I want to read this data byte by byte (f.read(4)). While debugging, I realized that when it gets to the end of the first line, it still takes in the newline character \n and it is used as the first byte of the following int I read. I don't want to simply use .splitlines()because some data could have an n inside and I don't want to corrupt it. I'm using Python 2.7.10, by the way. I also read that opening a binary file with the b parameter "takes care" of the new line/end of line characters; why is not the case with me?
This is what happens in the console as the file's position is right before the newline character:
>>> d = f.read(4)
>>> d
'\n\x00\x00\x00'
>>> s = struct.unpack("i", d)
>>> s
(10,)
(Followed from discussion with OP in chat)
Seems like the file is in binary format and the newlines are just mis-interpreted values. This can happen when writing 10 to the file for example.
This doesn't mean that newline was intended, and it is probably not. You can just ignore it being printed as \n and just use it as data.
You should just be able to replace the bytes that indicate it is a newline.
>>> d = f.read(4).replace(b'\x0d\x0a', b'') #\r\n should be bytes b'\x0d\x0a'
>>> diff = 4 - len(d)
>>> while diff > 0: # You can probably make this more sophisticated
... d += f.read(diff).replace(b'\x0d\x0a', b'') #\r\n should be bytes b'\x0d\x0a'
... diff = 4 - len(d)
>>>
>>> s = struct.unpack("i", d)
This should give you an idea of how it will work. This approach could mess with your data's byte alignment.
If you really are seeing "\n" in your print of d then try .replace(b"\n", b"")
I am a beginner in python, and I am trying to read the data from several sensors (humidity, temperature, pressure sensors...) that I connect with a usb hub to my computer. My main goal is to record every five minutes the different values of those sensors and then store it to analyse it.
I have got all the data sheets and manuals of my sensors (which are from Hygrosens Instruments), I know how they work and what kind of data they are sending. But I do not know how to read them. Below is what I tried, using pyserial.
import serial #import the serial library
from time import sleep #import the sleep command from the time library
import binascii
output_file = open('hygro.txt', 'w') #create a file and allow you to write in it only. The name of this file is hygro.txt
ser = serial.Serial("/dev/tty.usbserial-A400DUTI", 9600) #load into a variable 'ser' the information about the usb you are listening. /dev/tty.usbserial.... is the port after plugging in the hygrometer, 9600 is for bauds, it can be diminished
count = 0
while 1:
read_byte = ser.read(size=1)
So now I want to find the end of the line of the data as the measurement informations that I need are in a line that begins with 'V', and if the data sheet of my sensor, it said that a line ends by , so I want to read one byte at a time and look for '<', then 'c', then 'r', then '>'. So I wanted to do this:
while 1:
read_byte = ser.read(size=8) #read a byte
read_byte_hexa =binascii.hexlify(read_byte) #convert the byte into hexadecimal
trad_hexa = int(read_byte_hexa , 16) #convert the hexadecimal into an int in purpose to compare it with another int
trad_firstcrchar = int('3c' , 16) #convert the hexadecimal of the '<' into a int to compare it with the first byte
if (trad_hexa == trad_firstcrchar ): #compare the first byte with the '<'
read_byte = ser.read(size=1) #read the next byte (I am not sure if that really works)
read_byte_hexa =binascii.hexlify(read_byte)# from now I am doing the same thing as before
trad_hexa = int(read_byte_hexa , 16)
trad_scdcrchar = int('63' , 16)
print(trad_hexa, end='/')# this just show me if it gets in the condition
print(trad_scdcrchar)
if (trad_hexa == trad_scdcrchar ):
read_byte = ser.read(size=1) #read the next byte
read_byte_hexa =binascii.hexlify(read_byte)
trad_hexa = int(read_byte_hexa , 16)
trad_thirdcrchar = int('72' , 16)
print(trad_hexa, end='///')
print(trad_thirdcrchar)
if (trad_hexa == trad_thirdcrchar ):
read_byte = ser.read(size=1) #read the next byte
read_byte_hexa =binascii.hexlify(read_byte)
trad_hexa = int(read_byte_hexa , 16)
trad_fourthcrchar = int('3e' , 16)
print(trad_hexa, end='////')
print(trad_fourthcrchar)
if (trad_hexa == trad_fourthcrchar ):
print ('end of the line')
But I am not sure that it works, I mean I think it does not have the time to read the second one, the second byte I am reading, it's not exactly the second one. So that's why I want to use a buffer, but I don't really get how I can do that. I am going to look for it, but if someone knows an easier way to do what I want, I am ready to try it!
Thank you
You seem to be under the impression that the end-of-line character for that sensor's communication protocol is 4 different characters: <, c, r and >. However, what is being referred to is the carriage return, often denoted by <cr> and in many programming languages just by \r (even though it looks like 2 characters, it represents just one character).
You could simplify your code greatly by reading in the data from the sensors line by line, as the protocol is structured. Here's something to help you get started:
import time
def parse_info_line(line):
# implement to your own liking
logical_channel, physical_probe, hardware_id, crc = [line[index:index+2] for index in (1, 3, 5, 19)]
serialno = line[7:19]
return physical_probe
def parse_value_line(line):
channel, crc = [line[ind:ind+2] for ind in (1,7)]
encoded_temp = line[3:7]
return twos_comp(int(encoded_temp, 16), 16)/100.
def twos_comp(val, bits):
"""compute the 2's compliment of int value `val`"""
if (val & (1 << (bits - 1))) != 0: # if sign bit is set e.g., 8bit: 128-255
val = val - (1 << bits) # compute negative value
return val # return positive value as is
def listen_on_serial(ser):
ser.readline() # do nothing with the first line: you have no idea when you start listening to the data broadcast from the sensor
while True:
line = ser.readline()
try:
first_char = line[0]
except IndexError: # got no data from sensor
break
else:
if first_char == '#': # begins a new sensor record
in_record = True
elif first_char == '$':
in_record = False
elif first_char == 'I':
parse_info_line(line)
elif first_char == 'V':
print(parse_value_line(line))
else:
print("Unexpected character at the start of the line:\n{}".format(line))
time.sleep(2)
The twos_comp function was written by travc and you are encouraged to upvote his answer when you have enough reputation and if you intend to use his code (and even if you won't, it's still a good answer, I upvoted it just now). The listen_on_serial could be improved as well (many Python programmers will recognize the switch-structure and implement it with a dictionary rather than if... elif... elif...), but this is only intended to get you started.
As a test, the following code extract simulates the sensor sending some data (which is line-delimited, using the carriage return as the end-of-line marker), which I copied from the pdf you linked to (FAQ_terminalfenster_E.pdf).
>>> import serial
>>> import io
>>>
>>> ser = serial.serial_for_url('loop://', timeout=1)
>>> serio = io.TextIOWrapper(io.BufferedRWPair(ser, ser), newline='\r', line_buffering=True)
>>> serio.write(u'A1A0\r' # simulation of starting to listen halfway between 2 records
... '$\r' # marks the end of the previous record
... '#\r' # marks the start of a new sensor record
... 'I0101010000000000001B\r' # info about a sensor's probe
... 'V0109470D\r' # data matching that probe
... 'I0202010000000000002B\r' # other probe, same sensor
... 'V021BB55C\r') # data corresponding with 2nd probe
73L
>>>
>>> listen_on_serial(serio)
23.75
70.93
>>>
Note that it is recommended by the pyserial docs to be using TextIOWrapper when the end-of-line character is not \n (the linefeed character), as was also answered here.
I am trying to read a line from serial connection and convert it to int:
print arduino.readline()
length = int(arduino.readline())
but getting this error:
ValueError: invalid literal for int() with base 10: ''
I looked up this error and means that it is not possible to convert an empty string to int, but the thing is, my readline is not empty, because it prints it out.
The print statement prints it out and the next call reads the next line. You should probably do.
num = arduino.readline()
length = int(num)
Since you mentioned that the Arduino is returning C style strings, you should strip the NULL character.
num = arduino.readline()
length = int(num.strip('\0'))
Every call to readline() reads a new line, so your first statement has read a line already, next time you call readline() data is not available anymore.
Try this:
s = arduino.readline()
if len(s) != 0:
print s
length = int(s)
When you say
print arduino.readline()
you have already read the currently available line. So, the next readline might not be getting any data. You might want to store this in a variable like this
data = arduino.readline()
print data
length = int(data)
As the data seems to have null character (\0) in it, you might want to strip that like this
data = arduino.readline().rstrip('\0')
The problem is when the arduino starts to send serial data it starts by sending empty strings initially, so the pyserial picks up an empty string '' which cannot be converted to an integer. You can add a delay above serial.readline(), like this:
while True:
time.sleep(1.5)
pos = arduino.readline().rstrip().decode()
print(pos)