Develop Reference

consistency issue(missing data and sequence) while reading RS485 Data using Pyserial

First time ever I am doing python coding with Pyserial and RS485.
My function for reading serial data is here:
#RS485 read function
from threading import Thread
from time import sleep
def v_read_left():
b = 0
packet = bytearray()
packet.append(0x01) #
packet.append(0x03)
packet.append(0x20)
packet.append(0xab)
packet.append(0x00)
packet.append(0x02)
a = crc_find(packet)
packet += a
ser.flush()
while True:
ser.write(bytes(packet)) # write request to motor driver
sleep(0.1) # for frequency control
b = ser.read(9) # read RS485 received
#print(b)
if len(b) > 0:
print(b)
thread = Thread(target = v_read_left, args = ( ))
thread.start()
thread.join()
and the output of this function is here:
b'\x00fj\x06\x01\x03\x04\x00c'
b'\x00a\xcb\xc5\x01\x03\x04\x00h'
b'\x00b\xfa\x06\x01\x03\x04\x00`'
b'\x00`\xfa\x05\x01\x03\x04\x00^'
b'\x00c\xdb\xc8\x01\x03\x04\x00l'
b'\x00e\xfa\x05\x01\x03\x04\x00b'
b'\x00_\x1b\xd5\x01\x03\x04\x00]'
b'\x00e\xab\xca\x01\x03\x04\x00o'
b'\x00m\x0b\xc3\x01\x03\x04\x00h'
b'\x00_;\xd7\x01\x03\x04\x00`'
I am not able to identify what the problem is?
This is the BLDC motor driver RS485 command reference, I am trying to read the speed of the motor via the given example in this image.
Image

Python socket data-rate 10x slower than throughput on NIC

I've been sketching some test code for a file transfer using a TCP socket in python (3.9 - Win10 x64). When I run the code below the NIC throughput in task manager increases by about 100Mb/s (goes back down when the socket is broken). The data rates on the hard drives and the measured rate in the code seem to indicate that the actual transfer rate of the intended data is about 11Mb/s. First the code and next some more info about what I have been trying:
Client
import socket, os, sys
from threading import Thread
from time import monotonic
file_send = r'PATH TO FILE'
stats = os.stat(file_send)
print(stats.st_size)
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as client_soc:
client_soc.connect(('192.168.1.88', 6325))
client_soc.sendall(str(stats.st_size).encode('utf-8'))
client_soc.recv(10)
buffer_size = 1024 * 32
#client_soc.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, buffer_size)
with open(file_send, 'rb') as f:
bytes_read = 0
s_time = monotonic()
check_in = s_time
while bytes_read < stats.st_size:
read = f.read(1024*1024*10)
mono = monotonic()
t_diff = mono - s_time
if mono - check_in > 10:
print('{:,.0f} Kb/s'.format((bytes_read / 1024) / t_diff))
check_in = mono
client_soc.send(read)
bytes_read += len(read)
print('Done sending')
Server
import socket
from threading import Thread
class DLThread(Thread):
def __init__(self, bind_address, port, file_recv):
super(DLThread, self).__init__()
self.life_switch = True
self.bind_address = bind_address
self.port = port
self.file_recv = file_recv
def run(self) -> None:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listen_doc:
listen_doc.bind((self.bind_address, self.port))
listen_doc.listen()
print('listening')
client_conn, addr = listen_doc.accept()
with client_conn:
print('Connected')
data = client_conn.recv(1024)
size = int(data)
print('File size: {}'.format(size))
client_conn.sendall(b'k')
gotten = 0
percent_inc = 0.1
percent_chunk = percent_inc * size
thresh = percent_inc * size
with open(self.file_recv, 'wb') as f:
while gotten < size:
data = client_conn.recv(1024*1024*5)
f.write(data)
gotten += len(data)
if gotten > thresh:
print('{:.0%} Complete'.format(gotten / size))
thresh = int(gotten // percent_chunk) + percent_chunk
def pull_the_plug(self):
self.life_switch = False
self.join()
bind_addr = input('Bind Address: ')
port = int(input('port: '))
file_location = input('fileloc: ')
dl = DLThread(bind_addr, port, file_location)
dl.start()
print('Started')
dl.join()
I tried changing the buffer sizes ad using send / sendall. I also tried to have the client wait for a (redundant - I think) acknowledgement from the server at each chunk.
I took a look at the traffic in wireshark and I am seeing messages that appear to be padded with a bunch of 0s appended. I am not sure if this is a clue or not.
I can't think of why there would be so much overhead on the network interface. It feels weird just throwing all the data into the socket and letting the API do everything else but this is what many other people have done in their examples.
Sometimes with posts like this people recommend using a different tool. I just want to make it clear that I am not really trying to get this file transfer to work, I just want to understand why this is happening so I can learn something.

Although this is a very specific example of a common mistake, I'll confirm the answer here. Joachim Isaksson is correct. Task Manager reports network traffic in orders of bits/second not bytes/second. This is almost an order of magnitude different.
what learn? Pay attention to units and when things don't line up look for how the data could be related.

Trying to send serial data over bluetooth from arduino sensors to python

***Python code:***
import serial
import pandas as pd
import time
import re
import xlrd
from msvcrt import getch
import numpy as np
i = 0
x = 0
y = 0
df = pd.read_excel(r'C:\Users\lynchfamily\Desktop\mlglovesdata.xls')
# Read COM9
# Read from COM10 as well
# Readline() only works with a timeout (IMPORTANT)
serHC = serial.Serial('COM9', 115200,timeout=.250,parity=serial.PARITY_NONE,rtscts=1) # This is the JY
serRN = serial.Serial('COM10', 115200,timeout=.250,parity=serial.PARITY_NONE,rtscts=1) # This is the silvermate
def serialin():
# Sensor lists
sensor_names = list()
sensor_values = list()
global i
# Read a certain amount of bytes from serial and then continue
# Regular expressions for finding the proper data
while i < 6:
# print(i) for debugging
global serHC
global serRN
#searchObj = re.search(r'(A\d?\d*)?(\d*)?',serHC.read(4).decode(),re.I)
#searchObjRN = re.search(r'(A\d?\d*)?(\d*)?',serRN.read(4).decode(),re.I)
# Serial data stops while in loop
# The if statements keep the false values out of the program
#if searchObj.group(1):
sensor_names.append(serHC.read(2))
#if searchObj.group(2):
sensor_values.append(serHC.read(2))
#if searchObjRN.group(1):
sensor_names.append(serRN.read(2))
#if searchObjRN.group(2):
sensor_values.append(serRN.read(2))
i = i + 1
while 1:
# Get the key from the msvcrt module
key = getch().decode('ASCII')
# If key is pressed, do something
if key:
print(key)
# Zip them together
# Final 2D list
final_2d_list = zip(sensor_names,sensor_values)
print(list(sorted(final_2d_list)))
#vals = df.Dataframe([
#df.append(vals)
#print(sorted_array_1stdim[r])
#sensor_values = [0] * 10
# Thread for reading definition
break
# Fancy recursion
sensor_values.clear()
sensor_names.clear()
i = 0
serialin()
serialin()
Arduino Code:
// The device with green colored wires
void setup() {
Serial.begin(115200);
}
void loop() {
// It won't work with the I2C while loop for some reason. Perhaps it is getting stuck up on it
Serial.print("A4");
Serial.print(analogRead(0)); // Read the local analog signal
delay(5);
Serial.print("A5");
Serial.print(analogRead(1)); // Read the local analog signal
delay(5);
Serial.print("A6");
Serial.print(analogRead(2)); // Read the local analog signal
delay(5);
Serial.print("A7");
Serial.print(analogRead(3)); // Read the local analog signal
}
I'm trying to send analog data from sensors over through bluetooth silver mate from sparkfun, and HC-06 modules to python.
I have to read the analog data at a delay of 5 seconds between each, so that the readings aren't conflicted.
The data comes through serial ports COM9 and COM10. I know that serial in python can be blocking, that's why I attempted to read it first, and then put it in a list.
I also knows that once serial has been read through, it appears to be non-blocking. When I was using serHC.readline() and serRN.readline(), I was getting something like what I'd expect to see.
However, the data in the list were not updating according to the change in the sensors. I have to admit python is not my main programming language, so that is why I'm asking for help.
I thought maybe using multiple threads might work, but I wasn't able to get the serHC and serRN variables in the main thread.
Any help will be appreciated!!

As you have discovered it is not possible to read sequentially from serial ports: a blocking read over one port implies a loss of data simultaneous sent over the other port.
Use a thread based approach.
The following sketch should be enough to get started:
import serial
import time
import re
import threading
BYTES_TO_READ = 6
# read from serial port
def read_from_serial(board, port):
print("reading from {}: port {}".format(board, port))
payload = b''
ser = serial.Serial(port, 115200,timeout=.250, parity=serial.PARITY_NONE, rtscts=1)
bytes_count = 0
while bytes_count < BYTES_TO_READ:
read_bytes = ser.read(2)
# sum number of bytes returned (not 2), you have set the timeout on serial port
# see https://pythonhosted.org/pyserial/pyserial_api.html#serial.Serial.read
bytes_count = bytes_count + len(read_bytes)
payload = payload + read_bytes
# here you have the bytes, do your logic
# ...
print("READ from {}: [{}]".format(board, payload))
return
def main():
board = {
'JY': 'COM9',
'SILVER': 'COM10'
}
threads = []
for b in board:
t = threading.Thread(target=read_from_serial, args=(b, board[b],))
threads.append(t)
t.start()
# wait for all threads termination
for t in threads:
t.join()
main()
For learning about threading: https://pymotw.com/3/threading/
Periodic read from serials
Below a sketch for reading each TIME_PERIOD seconds.
A parte the infinite while loop around the read there is a "thread" loop with a nested try/catch block
for catching serials communication problems and retrying to connect after TIME_PERIOD.
Take it just as a starting example!
import serial
import time
import re
import threading
BYTES_TO_READ = 6
TIME_PERIOD = 5
def read_message(board, port, handle):
payload = b''
bytes_count = 0
while bytes_count < BYTES_TO_READ:
read_bytes = handle.read(2)
bytes_count = bytes_count + len(read_bytes)
payload = payload + read_bytes
# here you have the bytes, do your logic
# ...
print("READ from {}: [{}]".format(board, payload))
def serial_thread(board, port):
print("reading from {}: port {}".format(board, port))
while True:
try:
handle = serial.Serial(port, 115200,timeout=.250, parity=serial.PARITY_NONE, rtscts=1)
while True:
read_message(board, port, handle)
time.sleep(TIME_PERIOD)
except Exception as e:
print("ERROR: {}".format(e))
print("retrying in {} seconds".format(TIME_PERIOD))
handle.close()
time.sleep(TIME_PERIOD)
def main():
board = {
'JY': '/dev/ttyUSB0',
'SILVER': '/dev/ttyACM0'
}
threads = []
for b in board:
t = threading.Thread(target=serial_thread, args=(b, board[b],))
threads.append(t)
t.start()
# wait for all threads termination
for t in threads:
t.join()
main()

Read Modbus register and log to csv

I am newbie to python. I have an application where I would like to read a Modbus input register and log the data to a csv file every 10 seconds, I guess the time period is irrelevant. I can read a register, I can log to a csv, I can log to a csv every 10 seconds, but I am struggling to do all of this in the right order.
How would I rearrange my code below to open the port and log the latest register value? At the moment the code opens the port once, reads the value and then keeps logging the same value every 10 seconds.
# Code
import socket
import struct
import time
import csv
# Create a TCP/IP socket
TCP_IP = '169.254.84.141'
TCP_PORT = 502
BUFFER_SIZE = 0
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((TCP_IP, TCP_PORT))
try:
# Ask user for Modbus options
unitId = 5
functionCode = 4
startRegister = 00
numRegister = 16
# Construct request packet
req = struct.pack('>3H 2B 2H', 0, 0, 6, int(unitId), int(functionCode), int(startRegister), int(numRegister))
sock.send(req)
# Calculate receipt packet buffer and structure
BUFFER_SIZE = (3*2) + (3*1) + (int(numRegister)*2)
rec = sock.recv(BUFFER_SIZE)
def setB():
global BH
BH = 'B' #1
def setH():
global BH
BH = 'H' #2
functionLookup = {
1 : setB, # Read Coils (1 byte)
2 : setB, # Read Input Discrete Registers (1 byte)
3 : setH, # Read Holding Registers (2 byte)
4 : setH # Read Input Registers (2 byte)
}
functionLookup[int(functionCode)]()
s = struct.Struct('>3H 3B %s%s' %(numRegister, BH))
data = s.unpack(rec)
# Wait a couple of seconds before disconnecting
time.sleep(2);
Tank1Vol = data[6]
Tank1 = "Tank 1 Volume"
Units = "Litres"
Tank1csv = Tank1, (Tank1Vol),(Units)
starttime=time.time()
while True:
print ("tick")
print (Tank1csv)
time.sleep(10.0 - ((time.time() - starttime) % 10.0))
with open("Tank Level.csv", "a") as TanksFile:
TanksFileWriter = csv.writer(TanksFile)
TanksFileWriter.writerow(Tank1csv)
TanksFile.close()
finally:
sock.close()
Units = "Litres"
Tank1 = "Tank 1 Volume"
print (Tank1,(Tank1Vol),(Units))`

you can install modbus module using pip3 command. It has a client that will do your job.

There are several python libraries for modbus : pymodbus3, minimalmodbus, modbus-tk. I recommend to pick one and use it for your project.
I am maintaining modbus-tk, so it would be my choice. :-)
You can have a look at this example to read registers : https://github.com/ljean/modbus-tk/blob/master/examples/tcpmaster_example.py
I hope it helps

Attempting to read from two serial ports at once

I am trying to read from two serial ports at once. Each connected device spits out a line of data. I read the data from each port as a list and then concatenate the list and print it out as one line.
If I read each port individually, the data updates fine. But the second I attempt to read from both, it lags up and the data stops changing in the print output. The timestamp updates fine, but the data itself is what starts to lag.
Below is my code, should I be doing some sort of threading? I am reading from an Arduino and a Teensy.
import serial
import time
serA = serial.Serial('/dev/arduino', 230400)
serT = serial.Serial('/dev/teensy', 9600)
while 1 :
timestamp = "%f" % time.time()
print(timestamp)
arduino = serA.readline().rstrip('\n')
data_listA = arduino.split('$')
teensy = serT.readline().rstrip('\n')
data_listT = teensy.split('$')
data_list = data_listA + data_listT
print(data_list)

just check to see if your serial port has bytes to read before you try to read it ...
while 1 :
timestamp = "%f" % time.time()
print(timestamp)
if serA.inWaiting(): # only read if there is something waiting to be read
arduino = serA.readline().rstrip('\n')
data_listA = arduino.split('$')
print("GOT ARDUINO:",data_listA)
if serB.inWaiting():
teensy = serT.readline().rstrip('\n')
data_listT = teensy.split('$')
print("GOT TEENSY:",data_listT)

Using inwaiting() unfortunately did not work for me. I ended up having to use threading. A basic example for people who might encounter my problem is shown below.
import serial
import Queue
import threading
queue = Queue.Queue(1000)
serA = serial.Serial('/dev/arduino', 230400)
serT = serial.Serial('/dev/teensy', 9600)
def serial_read(s):
while 1:
line = s.readline()
queue.put(line)
threadA = threading.Thread(target=serial_read, args=(serA,),).start()
threadT = threading.Thread(target=serial_read, args=(serT,),).start()
while 1:
line = queue.get(True, 1)
print line
I based my code on the last answer from this question.