Develop Reference

consistency issue(missing data and sequence) while reading RS485 Data using Pyserial

First time ever I am doing python coding with Pyserial and RS485.
My function for reading serial data is here:
#RS485 read function
from threading import Thread
from time import sleep
def v_read_left():
b = 0
packet = bytearray()
packet.append(0x01) #
packet.append(0x03)
packet.append(0x20)
packet.append(0xab)
packet.append(0x00)
packet.append(0x02)
a = crc_find(packet)
packet += a
ser.flush()
while True:
ser.write(bytes(packet)) # write request to motor driver
sleep(0.1) # for frequency control
b = ser.read(9) # read RS485 received
#print(b)
if len(b) > 0:
print(b)
thread = Thread(target = v_read_left, args = ( ))
thread.start()
thread.join()
and the output of this function is here:
b'\x00fj\x06\x01\x03\x04\x00c'
b'\x00a\xcb\xc5\x01\x03\x04\x00h'
b'\x00b\xfa\x06\x01\x03\x04\x00`'
b'\x00`\xfa\x05\x01\x03\x04\x00^'
b'\x00c\xdb\xc8\x01\x03\x04\x00l'
b'\x00e\xfa\x05\x01\x03\x04\x00b'
b'\x00_\x1b\xd5\x01\x03\x04\x00]'
b'\x00e\xab\xca\x01\x03\x04\x00o'
b'\x00m\x0b\xc3\x01\x03\x04\x00h'
b'\x00_;\xd7\x01\x03\x04\x00`'
I am not able to identify what the problem is?
This is the BLDC motor driver RS485 command reference, I am trying to read the speed of the motor via the given example in this image.
Image

Trying to send serial data over bluetooth from arduino sensors to python

***Python code:***
import serial
import pandas as pd
import time
import re
import xlrd
from msvcrt import getch
import numpy as np
i = 0
x = 0
y = 0
df = pd.read_excel(r'C:\Users\lynchfamily\Desktop\mlglovesdata.xls')
# Read COM9
# Read from COM10 as well
# Readline() only works with a timeout (IMPORTANT)
serHC = serial.Serial('COM9', 115200,timeout=.250,parity=serial.PARITY_NONE,rtscts=1) # This is the JY
serRN = serial.Serial('COM10', 115200,timeout=.250,parity=serial.PARITY_NONE,rtscts=1) # This is the silvermate
def serialin():
# Sensor lists
sensor_names = list()
sensor_values = list()
global i
# Read a certain amount of bytes from serial and then continue
# Regular expressions for finding the proper data
while i < 6:
# print(i) for debugging
global serHC
global serRN
#searchObj = re.search(r'(A\d?\d*)?(\d*)?',serHC.read(4).decode(),re.I)
#searchObjRN = re.search(r'(A\d?\d*)?(\d*)?',serRN.read(4).decode(),re.I)
# Serial data stops while in loop
# The if statements keep the false values out of the program
#if searchObj.group(1):
sensor_names.append(serHC.read(2))
#if searchObj.group(2):
sensor_values.append(serHC.read(2))
#if searchObjRN.group(1):
sensor_names.append(serRN.read(2))
#if searchObjRN.group(2):
sensor_values.append(serRN.read(2))
i = i + 1
while 1:
# Get the key from the msvcrt module
key = getch().decode('ASCII')
# If key is pressed, do something
if key:
print(key)
# Zip them together
# Final 2D list
final_2d_list = zip(sensor_names,sensor_values)
print(list(sorted(final_2d_list)))
#vals = df.Dataframe([
#df.append(vals)
#print(sorted_array_1stdim[r])
#sensor_values = [0] * 10
# Thread for reading definition
break
# Fancy recursion
sensor_values.clear()
sensor_names.clear()
i = 0
serialin()
serialin()
Arduino Code:
// The device with green colored wires
void setup() {
Serial.begin(115200);
}
void loop() {
// It won't work with the I2C while loop for some reason. Perhaps it is getting stuck up on it
Serial.print("A4");
Serial.print(analogRead(0)); // Read the local analog signal
delay(5);
Serial.print("A5");
Serial.print(analogRead(1)); // Read the local analog signal
delay(5);
Serial.print("A6");
Serial.print(analogRead(2)); // Read the local analog signal
delay(5);
Serial.print("A7");
Serial.print(analogRead(3)); // Read the local analog signal
}
I'm trying to send analog data from sensors over through bluetooth silver mate from sparkfun, and HC-06 modules to python.
I have to read the analog data at a delay of 5 seconds between each, so that the readings aren't conflicted.
The data comes through serial ports COM9 and COM10. I know that serial in python can be blocking, that's why I attempted to read it first, and then put it in a list.
I also knows that once serial has been read through, it appears to be non-blocking. When I was using serHC.readline() and serRN.readline(), I was getting something like what I'd expect to see.
However, the data in the list were not updating according to the change in the sensors. I have to admit python is not my main programming language, so that is why I'm asking for help.
I thought maybe using multiple threads might work, but I wasn't able to get the serHC and serRN variables in the main thread.
Any help will be appreciated!!

As you have discovered it is not possible to read sequentially from serial ports: a blocking read over one port implies a loss of data simultaneous sent over the other port.
Use a thread based approach.
The following sketch should be enough to get started:
import serial
import time
import re
import threading
BYTES_TO_READ = 6
# read from serial port
def read_from_serial(board, port):
print("reading from {}: port {}".format(board, port))
payload = b''
ser = serial.Serial(port, 115200,timeout=.250, parity=serial.PARITY_NONE, rtscts=1)
bytes_count = 0
while bytes_count < BYTES_TO_READ:
read_bytes = ser.read(2)
# sum number of bytes returned (not 2), you have set the timeout on serial port
# see https://pythonhosted.org/pyserial/pyserial_api.html#serial.Serial.read
bytes_count = bytes_count + len(read_bytes)
payload = payload + read_bytes
# here you have the bytes, do your logic
# ...
print("READ from {}: [{}]".format(board, payload))
return
def main():
board = {
'JY': 'COM9',
'SILVER': 'COM10'
}
threads = []
for b in board:
t = threading.Thread(target=read_from_serial, args=(b, board[b],))
threads.append(t)
t.start()
# wait for all threads termination
for t in threads:
t.join()
main()
For learning about threading: https://pymotw.com/3/threading/
Periodic read from serials
Below a sketch for reading each TIME_PERIOD seconds.
A parte the infinite while loop around the read there is a "thread" loop with a nested try/catch block
for catching serials communication problems and retrying to connect after TIME_PERIOD.
Take it just as a starting example!
import serial
import time
import re
import threading
BYTES_TO_READ = 6
TIME_PERIOD = 5
def read_message(board, port, handle):
payload = b''
bytes_count = 0
while bytes_count < BYTES_TO_READ:
read_bytes = handle.read(2)
bytes_count = bytes_count + len(read_bytes)
payload = payload + read_bytes
# here you have the bytes, do your logic
# ...
print("READ from {}: [{}]".format(board, payload))
def serial_thread(board, port):
print("reading from {}: port {}".format(board, port))
while True:
try:
handle = serial.Serial(port, 115200,timeout=.250, parity=serial.PARITY_NONE, rtscts=1)
while True:
read_message(board, port, handle)
time.sleep(TIME_PERIOD)
except Exception as e:
print("ERROR: {}".format(e))
print("retrying in {} seconds".format(TIME_PERIOD))
handle.close()
time.sleep(TIME_PERIOD)
def main():
board = {
'JY': '/dev/ttyUSB0',
'SILVER': '/dev/ttyACM0'
}
threads = []
for b in board:
t = threading.Thread(target=serial_thread, args=(b, board[b],))
threads.append(t)
t.start()
# wait for all threads termination
for t in threads:
t.join()
main()

Attempting to read from two serial ports at once

I am trying to read from two serial ports at once. Each connected device spits out a line of data. I read the data from each port as a list and then concatenate the list and print it out as one line.
If I read each port individually, the data updates fine. But the second I attempt to read from both, it lags up and the data stops changing in the print output. The timestamp updates fine, but the data itself is what starts to lag.
Below is my code, should I be doing some sort of threading? I am reading from an Arduino and a Teensy.
import serial
import time
serA = serial.Serial('/dev/arduino', 230400)
serT = serial.Serial('/dev/teensy', 9600)
while 1 :
timestamp = "%f" % time.time()
print(timestamp)
arduino = serA.readline().rstrip('\n')
data_listA = arduino.split('$')
teensy = serT.readline().rstrip('\n')
data_listT = teensy.split('$')
data_list = data_listA + data_listT
print(data_list)

just check to see if your serial port has bytes to read before you try to read it ...
while 1 :
timestamp = "%f" % time.time()
print(timestamp)
if serA.inWaiting(): # only read if there is something waiting to be read
arduino = serA.readline().rstrip('\n')
data_listA = arduino.split('$')
print("GOT ARDUINO:",data_listA)
if serB.inWaiting():
teensy = serT.readline().rstrip('\n')
data_listT = teensy.split('$')
print("GOT TEENSY:",data_listT)

Using inwaiting() unfortunately did not work for me. I ended up having to use threading. A basic example for people who might encounter my problem is shown below.
import serial
import Queue
import threading
queue = Queue.Queue(1000)
serA = serial.Serial('/dev/arduino', 230400)
serT = serial.Serial('/dev/teensy', 9600)
def serial_read(s):
while 1:
line = s.readline()
queue.put(line)
threadA = threading.Thread(target=serial_read, args=(serA,),).start()
threadT = threading.Thread(target=serial_read, args=(serT,),).start()
while 1:
line = queue.get(True, 1)
print line
I based my code on the last answer from this question.

Opening 2 serial ports simultaneously in python (one tx one for rx)

I am making a throughput test for a bluetooth link, and I need to send data through a serial port to one bluetooth device which will then transport that data wirelessly to another bluetooth device. The other device will then complete the circuit by sending the data back to the host PC via a different serial port.
The problem seems to be when I attempt to open up 2 different instances of PySerial, the program simply hangs. I have isolated it down to running vs. hanging when I comment out one of the two serial port instantiations. Anyone see a problem with how I'm doing this? If so, what is the proper way to do this? See code below:
#/usr/bin/python
import serial
import time
import sys
DEFAULT_BAUD = 115200
SEND_SIZE = 100
def addPath(file):
pth, fl = os.path.split(__file__)
return os.path.join(pth, file)
def is_number(s):
try:
int(s, 16)
return True
except:
return False
class SerialReader():
def __init__(self, portRx, portTx):
self.portTx = portTx
self.portRx = portRx
self.start_time__sec = time.time()
self.interval__sec = 0
self.buffer = []
self.sendtext = ''.join([str(i) for i in range(SEND_SIZE)])
# send first batch of data
self.portTx.write(self.sendtext)
def didDataArrive(self):
# Read port
self.buffer.extend(list(self.portRx.read(1024)))
# Step through the buffer byte and byte and see if the tick text
# is at the front.
while len(self.buffer) >= len(self.sendtext):
if self.buffer[:len(self.sendtext)] == self.sendtext:
# Discard the tick text
self.buffer = self.buffer[len(self.sendtext):]
# Record time
snapshot__sec = time.time()
self.interval__sec = snapshot__sec - self.start_time__sec
self.start_time__sec = snapshot__sec
# send more data
self.portTx.write(self.sendtext)
return True
else:
self.buffer.pop(0)
return False
def main(port1, port2, baudrate1 = DEFAULT_BAUD, baudrate2 = DEFAULT_BAUD):
try:
import serial
except:
traceback.print_exc()
print "="*60
print "You need to install PySerial"
print "Windows: easy_install pyserial"
print "Mac/Linux: sudo easy_install pyserial"
try:
s1 = serial.Serial(port1, baudrate1, timeout = 0.1)
s2 = serial.Serial(port2, baudrate2, timeout = 0.1)
print "Loading serial ports"
except:
print "Serial port error"
exit()
plot_stop = False
dataread = SerialReader(s2, s1)
try:
while plot_stop == False:
if dataread.didDataArrive():
print dataread.interval__sec
except KeyboardInterrupt:
print "Keyboard Interrupt"
plot_stop = True
finally:
print "Closing"
s1.close()
s2.close()
if __name__ == '__main__':
if (len(sys.argv) < 3):
print "Usage: python extract_data.py phonelink_serialport phonelinkclient_serialport [baudrate1] [baudrate2]"
else:
main(*sys.argv[1:])
If I remove one of the following lines (doesn't matter which one), the python script runs (although it eventually crashes because in the code it eventually tries to reference both ports). If I leave these lines in, the program seems to just hang (it just seems to sit there and run indefinitely):
s1 = serial.Serial(port1, baudrate1, timeout = 0.1)
s2 = serial.Serial(port2, baudrate2, timeout = 0.1)

Calling a thread out of scope in python

I have a thread that is defined as in a program that continuously reads serial data along with running a UI in wxpython.
dat = Thread(target=receiving, args=(self.ser,))
The method it calls "receiving" runs in an infinite loop
def receiving(ser):
global last_received
buffer = ''
while True:
date = datetime.date.today().strftime('%d%m%Y')
filename1 = str(date) + ".csv"
while date == datetime.date.today().strftime('%d%m%Y'):
buffer = buffer + ser.read(ser.inWaiting())
if '\n' in buffer:
lines = buffer.split('\n')
if lines[-2]:
last_received = lines[-2]
buffer = lines[-1]
print_data =[time.strftime( "%H:%M:%S"), last_received]
try:
with open(filename1, 'a') as fob:
writ = csv.writer(fob, delimiter = ',')
writ.writerow(print_data)
fob.flush()
except ValueError:
with open('errors.log','a') as log:
log.write('CSV file writing failed ' + time.strftime("%H:%M:%S")+' on '+datetime.date.today().strftime('%d/%m/%Y')+'\n')
log.close()
The argument is defined as
class SerialData(object):
def __init__(self, init=50):
try:
serial_list = serialenum.enumerate()
self.ser = ser = serial.Serial(
port=serial_list[0],
baudrate=9600,
bytesize=serial.EIGHTBITS,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
timeout=None,
xonxoff=0,
rtscts=0,
interCharTimeout=None
)
except serial.serialutil.SerialException:
# no serial connection
self.ser = None
else:
dat = Thread(target=receiving, args=(self.ser,))
if not dat.is_alive:
dat.start()
def next(self):
if not self.ser:
# return anything so we can test when Serial Device isn't connected
return 'NoC'
# return a float value or try a few times until we get one
for i in range(40):
raw_line = last_received
try:
return float(raw_line.strip())
time.sleep(0.1)
except ValueError:
# print 'Not Connected',raw_line
time.sleep(0.1)
return 0
Due to a bug in Ubuntu 14.04 the thread hangs after a while. I wanted to periodically check if the thread is alive and start it again if it is not. So I did something like
def on_timer(self):
self.text.SetLabel(str(mul_factor*self.datagen.next()))
if not dat.is_alive():
dat.start()
wx.CallLater(1, self.on_timer)
This runs every second to update the data in UI but also needs to check if the thread is not stopped. But this gives me an error saying "NameError: global name 'dat' is not defined". I also tried referring to the thread using the object name path. But didn't work either.
Can someone help me as to how I can start the thread out of scope?

It seems like you want to replace dat with self.dat. dat only exists in the scope of the __init__ method. I suggest reading up on Python scoping rules.