Develop Reference

pyserial not replenish to input buffer immediately after received data?

I want to use pyserial module to read serial data sent by Arduino, the data update rate is about 250Hz, one data consists of 29 bytes and the baud rate is 230400.
When my code is running, I found that the pyserial input buffer does not automatically replenish data from arduino until about 0.5 seconds has passed. Is there any way to replenish the input buffer immediately after data received from Arduino?
Many thanks!
Adam Shiau
import serial
import time
ser = serial.Serial()
old = 0
while True
if ser.in_waiting > 29:
print('buf: ', ser.readInputBuffer())
new = time.perf_counter_ns()
print(ser.readBinaryList(29))
print(''%.1f\n %((new - old)*1e-3))
old = new
# updated function definition_07/23
def readBinaryList(mum):
data_r = ser.read(mum)
data_r = [i for i in data_r]
return data_r
def readInputBuffer(self):
return ser.in_waiting
result1
result2

ser.readInputBuffer() and ser.readBinaryList() look weird. I cannot find those functions in the docs. If I was writing this code, I would just use ser.read(29)
import serial
import time
ser = serial.Serial()
old = 0
while True
if ser.in_waiting > 29:
new = time.perf_counter_ns()
print(ser.read(29))
print(''%.1f\n %((new - old)*1e-3))
old = new

Trying to send serial data over bluetooth from arduino sensors to python

***Python code:***
import serial
import pandas as pd
import time
import re
import xlrd
from msvcrt import getch
import numpy as np
i = 0
x = 0
y = 0
df = pd.read_excel(r'C:\Users\lynchfamily\Desktop\mlglovesdata.xls')
# Read COM9
# Read from COM10 as well
# Readline() only works with a timeout (IMPORTANT)
serHC = serial.Serial('COM9', 115200,timeout=.250,parity=serial.PARITY_NONE,rtscts=1) # This is the JY
serRN = serial.Serial('COM10', 115200,timeout=.250,parity=serial.PARITY_NONE,rtscts=1) # This is the silvermate
def serialin():
# Sensor lists
sensor_names = list()
sensor_values = list()
global i
# Read a certain amount of bytes from serial and then continue
# Regular expressions for finding the proper data
while i < 6:
# print(i) for debugging
global serHC
global serRN
#searchObj = re.search(r'(A\d?\d*)?(\d*)?',serHC.read(4).decode(),re.I)
#searchObjRN = re.search(r'(A\d?\d*)?(\d*)?',serRN.read(4).decode(),re.I)
# Serial data stops while in loop
# The if statements keep the false values out of the program
#if searchObj.group(1):
sensor_names.append(serHC.read(2))
#if searchObj.group(2):
sensor_values.append(serHC.read(2))
#if searchObjRN.group(1):
sensor_names.append(serRN.read(2))
#if searchObjRN.group(2):
sensor_values.append(serRN.read(2))
i = i + 1
while 1:
# Get the key from the msvcrt module
key = getch().decode('ASCII')
# If key is pressed, do something
if key:
print(key)
# Zip them together
# Final 2D list
final_2d_list = zip(sensor_names,sensor_values)
print(list(sorted(final_2d_list)))
#vals = df.Dataframe([
#df.append(vals)
#print(sorted_array_1stdim[r])
#sensor_values = [0] * 10
# Thread for reading definition
break
# Fancy recursion
sensor_values.clear()
sensor_names.clear()
i = 0
serialin()
serialin()
Arduino Code:
// The device with green colored wires
void setup() {
Serial.begin(115200);
}
void loop() {
// It won't work with the I2C while loop for some reason. Perhaps it is getting stuck up on it
Serial.print("A4");
Serial.print(analogRead(0)); // Read the local analog signal
delay(5);
Serial.print("A5");
Serial.print(analogRead(1)); // Read the local analog signal
delay(5);
Serial.print("A6");
Serial.print(analogRead(2)); // Read the local analog signal
delay(5);
Serial.print("A7");
Serial.print(analogRead(3)); // Read the local analog signal
}
I'm trying to send analog data from sensors over through bluetooth silver mate from sparkfun, and HC-06 modules to python.
I have to read the analog data at a delay of 5 seconds between each, so that the readings aren't conflicted.
The data comes through serial ports COM9 and COM10. I know that serial in python can be blocking, that's why I attempted to read it first, and then put it in a list.
I also knows that once serial has been read through, it appears to be non-blocking. When I was using serHC.readline() and serRN.readline(), I was getting something like what I'd expect to see.
However, the data in the list were not updating according to the change in the sensors. I have to admit python is not my main programming language, so that is why I'm asking for help.
I thought maybe using multiple threads might work, but I wasn't able to get the serHC and serRN variables in the main thread.
Any help will be appreciated!!

As you have discovered it is not possible to read sequentially from serial ports: a blocking read over one port implies a loss of data simultaneous sent over the other port.
Use a thread based approach.
The following sketch should be enough to get started:
import serial
import time
import re
import threading
BYTES_TO_READ = 6
# read from serial port
def read_from_serial(board, port):
print("reading from {}: port {}".format(board, port))
payload = b''
ser = serial.Serial(port, 115200,timeout=.250, parity=serial.PARITY_NONE, rtscts=1)
bytes_count = 0
while bytes_count < BYTES_TO_READ:
read_bytes = ser.read(2)
# sum number of bytes returned (not 2), you have set the timeout on serial port
# see https://pythonhosted.org/pyserial/pyserial_api.html#serial.Serial.read
bytes_count = bytes_count + len(read_bytes)
payload = payload + read_bytes
# here you have the bytes, do your logic
# ...
print("READ from {}: [{}]".format(board, payload))
return
def main():
board = {
'JY': 'COM9',
'SILVER': 'COM10'
}
threads = []
for b in board:
t = threading.Thread(target=read_from_serial, args=(b, board[b],))
threads.append(t)
t.start()
# wait for all threads termination
for t in threads:
t.join()
main()
For learning about threading: https://pymotw.com/3/threading/
Periodic read from serials
Below a sketch for reading each TIME_PERIOD seconds.
A parte the infinite while loop around the read there is a "thread" loop with a nested try/catch block
for catching serials communication problems and retrying to connect after TIME_PERIOD.
Take it just as a starting example!
import serial
import time
import re
import threading
BYTES_TO_READ = 6
TIME_PERIOD = 5
def read_message(board, port, handle):
payload = b''
bytes_count = 0
while bytes_count < BYTES_TO_READ:
read_bytes = handle.read(2)
bytes_count = bytes_count + len(read_bytes)
payload = payload + read_bytes
# here you have the bytes, do your logic
# ...
print("READ from {}: [{}]".format(board, payload))
def serial_thread(board, port):
print("reading from {}: port {}".format(board, port))
while True:
try:
handle = serial.Serial(port, 115200,timeout=.250, parity=serial.PARITY_NONE, rtscts=1)
while True:
read_message(board, port, handle)
time.sleep(TIME_PERIOD)
except Exception as e:
print("ERROR: {}".format(e))
print("retrying in {} seconds".format(TIME_PERIOD))
handle.close()
time.sleep(TIME_PERIOD)
def main():
board = {
'JY': '/dev/ttyUSB0',
'SILVER': '/dev/ttyACM0'
}
threads = []
for b in board:
t = threading.Thread(target=serial_thread, args=(b, board[b],))
threads.append(t)
t.start()
# wait for all threads termination
for t in threads:
t.join()
main()

PyAudio callback being called only once

I'm trying to create a simple app that loads wav files (one for each note of a keyboard) and plays specific ones when a midi note is pressed (or played). So far, I've created a midi input stream using mido and an audio stream using pyaudio in two separate threads. the goal is for the midi stream to update the currently playing notes, and the callback of the pyaudio stream to check for active notes and play those that are. The midi stream works fine, but my audio stream only seems to call the callback once, right when the script is started (print(notes)). Any idea how I can get the audio stream callback to update constantly?
import wave
from io import BytesIO
import os
from mido import MidiFile
import pyaudio
from time import sleep
from threading import Thread
import numpy
# Pipe: active, released
# Rank: many pipes
# Stop: one or more ranks
# Manual: multiple ranks
# Organ: multiple manuals
pipes = []
notes = []
p = pyaudio.PyAudio()
def mapRange(num, inMin, inMax, outMin, outMax):
return int((num - inMin) * (outMax - outMin) / (inMax - inMin) + outMin)
def callback(in_data, frame_count, time_info, status):
data = bytes(frame_count)
print(notes)
for note in notes:
pipedata = bytes()
if len(data) != 0:
data1 = numpy.fromstring(data, numpy.int16)
data2 = numpy.fromstring(note['sample'].readframes(frame_count), numpy.int16)
pipedata = (data1 * 0.5 + data2 * 0.5).astype(numpy.int16)
else:
data2 = numpy.fromstring(note['sample'].readframes(frame_count), numpy.int16)
pipedata = data2.astype(numpy.int16)
data = pipedata.tostring()
return (data, pyaudio.paContinue)
stream = p.open(format=pyaudio.paInt24,
channels=2,
rate=48000,
output=True,
stream_callback=callback,
start=True)
# start the stream (4)
stream.start_stream()
for root, dirs, files in os.walk("samples"):
for filename in files:
file_on_disk = open(os.path.join(root, filename), 'rb')
pipes.append(
{"sample": wave.open(BytesIO(file_on_disk.read()), 'rb')})
for msg in MidiFile('test.mid').play():
if msg.type == "note_on":
notes.append(pipes[mapRange(msg.note, 36, 96, 0, 56)])
print("on")
if msg.type == "note_off":
#notes[mapRange(msg.note, 36, 96, 0, 56)] = False
print("off")
# wait for stream to finish (5)
while stream.is_active():
sleep(0.1)
# stop stream (6)
stream.stop_stream()
stream.close()
# close PyAudio (7)
p.terminate()

I too faced this issue and found this question in hopes of finding an answer, ended up figuring it out myself.
The data returned on the callback must match the number of frames (frames_per_buffer parameter in p.open). I see you didn't specify one so I think the default is 1024.
The thing is frames_per_buffer does not represent bytes but acrual frames. So since you specify the format as being pyaudio.paInt24 that means that one frames is represented by 3 bytes (24 / 8). So in your callback you should be returning 3072 bytes or the callback will not be called again for some reason.
If you were using blocking mode and not writing those 3072 bytes in stream.write() it would result in a weird effect of slow and crackling audio.

Attempting to read from two serial ports at once

I am trying to read from two serial ports at once. Each connected device spits out a line of data. I read the data from each port as a list and then concatenate the list and print it out as one line.
If I read each port individually, the data updates fine. But the second I attempt to read from both, it lags up and the data stops changing in the print output. The timestamp updates fine, but the data itself is what starts to lag.
Below is my code, should I be doing some sort of threading? I am reading from an Arduino and a Teensy.
import serial
import time
serA = serial.Serial('/dev/arduino', 230400)
serT = serial.Serial('/dev/teensy', 9600)
while 1 :
timestamp = "%f" % time.time()
print(timestamp)
arduino = serA.readline().rstrip('\n')
data_listA = arduino.split('$')
teensy = serT.readline().rstrip('\n')
data_listT = teensy.split('$')
data_list = data_listA + data_listT
print(data_list)

just check to see if your serial port has bytes to read before you try to read it ...
while 1 :
timestamp = "%f" % time.time()
print(timestamp)
if serA.inWaiting(): # only read if there is something waiting to be read
arduino = serA.readline().rstrip('\n')
data_listA = arduino.split('$')
print("GOT ARDUINO:",data_listA)
if serB.inWaiting():
teensy = serT.readline().rstrip('\n')
data_listT = teensy.split('$')
print("GOT TEENSY:",data_listT)

Using inwaiting() unfortunately did not work for me. I ended up having to use threading. A basic example for people who might encounter my problem is shown below.
import serial
import Queue
import threading
queue = Queue.Queue(1000)
serA = serial.Serial('/dev/arduino', 230400)
serT = serial.Serial('/dev/teensy', 9600)
def serial_read(s):
while 1:
line = s.readline()
queue.put(line)
threadA = threading.Thread(target=serial_read, args=(serA,),).start()
threadT = threading.Thread(target=serial_read, args=(serT,),).start()
while 1:
line = queue.get(True, 1)
print line
I based my code on the last answer from this question.

How do I make the pyserial loopback work?

I'm trying to test a serial connection, prior to hooking up the external device that will actually source the data. I'm trying to use pySerial's "loop://" device, but I'm not receiving data correctly. I've started with a very toy program, just be sure I understood how/if it would work. Clearly I don't. :)
Here is my data "Source"
def serialDataPump():
ser = serial.serial_for_url('loop://', timeout=1)
testCtr = 0;
while not bbq.closing and testCtr<10:
ser.write(bytes("Test\n", encoding='ascii'))
time.sleep(1)
testCtr += 1
Here is my data "sink":
def serialDataTestRcv():
ser = serial.serial_for_url('loop://', timeout=1)
while not bbq.closing:
line = ser.readline()
sys.stdout.write('received' + str(line))
And here is my test Function - I use two threads:
def testSerMain():
thread1 = Thread(target = serialDataPump)
thread2 = Thread(target = serialDataTestRcv)
thread1.start()
thread2.start()
thread1.join()
bbq.closing = True
time.sleep(2)
exit()
And finally, here is the output - I am receiving the EOLs at a minimum, because readline() unblocks, and loops exactly 11 times, prior to terminating, which indicates that both the pump and the receive are looping and terminate properly. However, as you can see, it receives just empty data + the EOL:
>>>
receivedb''receivedb''receivedb''receivedb''receivedb''receivedb''receivedb''receivedb''receivedb''receivedb''receivedb''
>>>
Win 7, x64m py3.3
Incidentally, I know about com0com - I just can't run it on the machine I'm on.

I discovered the problem - you must use the same instance of ser = serial.serial_for_url('loop://', timeout=1) that you created for both receive and Xmt.