Develop Reference

Microcontroller serial stops working when mu serial is closed

I am trying to control some NeoPixel lights, and because the Raspberry Pi can only control one light at any given time, I decided I would just get a microcontroller to do it, specifically the Adafruit ItsyBitsy M4 Express, and just send serial data through the USB connecting the two to control them. I got that working, using PySerial to send data to the microcontroller. However, after the RPi script sends data 8 times, it stops working and I have to reload the script. Additonally, by stop working, I mean that the script gets stuck sending data and never ends. The strange part is that if I open the MU serial after my script has started running, I can A: see the data being sent in, and B: it never stops. It goes past the 8 issue and keeps going.
RPi Code:
import serial
import time
from random import randint
a = time.time() #Unimportant, was looking to see how quickly Serial could be re-defined
try:
ser = serial.Serial(port='/dev/ttyACM1', baudrate=115200, write_timeout=0.05)
except:
ser = serial.Serial(port='/dev/ttyACM0', baudrate=115200, write_timeout=0.05)
print(time.time() - a)
EnterKey = "\x1A\r\n"
e = 0
while True:
e+=1
d = []
for number in range(0, 30):
d.append([randint(0, 255), randint(0, 255), randint(0, 255)])
d = ((str(d)) + EnterKey).encode()
ser.write(d)
time.sleep(1)
print(e)
Microcontroller code:
import board
import supervisor
import neopixel
import time
a = neopixel.NeoPixel(board.D5, 30, auto_write=False)
supervisor.diable_autoreload()
while True:
b = input("automated")
RST = []
c = ""
R3T = []
started = False
for value in b:
if started == True:
if value == '[':
started2 = True
ending = False
elif value == ']':
if ending == True:
started=False
else:
RST.append(int(c))
c = ""
ending = True
started2 = False
R3T.append(RST)
RST = []
elif started2 == True:
if value.isdigit():
c += value
elif value == ",":
RST.append(int(c))
c = ""
elif value == '[':
started = True
for value in range(0, 30):
a[value] = R3T[value]
a.show()
print(a)
I tried being lazy and just re-defining the serial object after a write timeout, but that does not work. (Errno 16] Device or resource busy). I then went around looking for Raspberry Pi settings seeing if anything was getting in my way, but found nothing. I ultimately gave up and came here. If you have any ideas, please tell me!

Is there a way to sync sending information across sockets on a network?

I am programming ada fruit ring LED lights in python, I have setup a local network for a midi controller to read data into my Raspberry Pi 3 and send to the 3 Pi Zero W through web sockets. I am running into the issue where some of the functions I have that run on threads will get out of sync with each other. In other words, one of the LED lights will go at a slightly faster or slower rate and eventually look not synced. I was curious if anyone has run into an issue like this or if I am just using the threads and events the incorrect way.
Here is some of my server side code:
import socket
from socket import *
from threading import Thread
import threading
import board
import neopixel
import sys
import time
import random
import os
from adafruit_led_animation.animation.comet import Comet
def handle_client(client):
p = None
connected = True
lights_on_default()
while True:
global thread_running
data=client.recv(1024).decode()
if not data:
connected = False
thread_running = False
break
if 'Pad' in data:
thread_running = False
thread = threading.Event()
if data == 'Pad 1':
thread_running = False
if p:
p.join()
p = Thread(target=slow_circles, args=(thread,))
p.daemon = True
p.start()
def slow_circles(event):
global thread_running
thread_running = True
while not event.isSet():
if not thread_running:
event.set()
break
for i in range(num_of_pixels):
// goes through the lighting of pixels
if not thread_running:
event.set()
break
pixels.show()
Here is some of my client side code:
import rtmidi.midiutil as midiutil
import time
import socket
buttons = {
36: 'Pad 1',
}
try:
s1 = socket.socket()
s2 = socket.socket()
except:
print('Error making socket')
port1 = 12346
port2 = 12347
serverOne = '192.168.1.18' // 1st Pi Zero W
serverTwo = '192.168.1.17' // 2nd Pi Zero W
def start():
global s1
global s2
notConnected = True
while True:
while notConnected:
connected = False
try:
s1.connect((serverOne, port1))
s2.connect((serverTwo, port2))
connected = True
break
except:
s1.close()
s2.close()
s1.socket.socket()
s2.socket.socket()
if connected:
midiin, port = midiutil.open_midiinput(1)
midiin.set_callback(midiCallback) // gets pad from midi controller
while True:
retry = False
try:
s1.sendall('red connected'.encode('utf8'))
s2.sendall('yellow connected'.encode('utf8'))
except:
retry = True
if retry:
s1.close()
s2.close()
midiin.delete()
break
else:
time.sleep(15)
def midiCallback(message, data):
control = message[0][1]
try:
value = message[0][2]
if (control in button.keys()):
name = buttons[control]
if (value > 0):
return buttonDown(name)
except IndexError:
pass
def buttonDown(button):
s1.sendall(f'{button}'.encode('utf8'))
s2.sendall(f'{button}'.encode('utf8'))
start()
And when I hit the pad and do the function I send data to the two PIs and they start at the same time but over time get out of sync. All help would be appreciated thanks.

Opening 2 serial ports simultaneously in python (one tx one for rx)

I am making a throughput test for a bluetooth link, and I need to send data through a serial port to one bluetooth device which will then transport that data wirelessly to another bluetooth device. The other device will then complete the circuit by sending the data back to the host PC via a different serial port.
The problem seems to be when I attempt to open up 2 different instances of PySerial, the program simply hangs. I have isolated it down to running vs. hanging when I comment out one of the two serial port instantiations. Anyone see a problem with how I'm doing this? If so, what is the proper way to do this? See code below:
#/usr/bin/python
import serial
import time
import sys
DEFAULT_BAUD = 115200
SEND_SIZE = 100
def addPath(file):
pth, fl = os.path.split(__file__)
return os.path.join(pth, file)
def is_number(s):
try:
int(s, 16)
return True
except:
return False
class SerialReader():
def __init__(self, portRx, portTx):
self.portTx = portTx
self.portRx = portRx
self.start_time__sec = time.time()
self.interval__sec = 0
self.buffer = []
self.sendtext = ''.join([str(i) for i in range(SEND_SIZE)])
# send first batch of data
self.portTx.write(self.sendtext)
def didDataArrive(self):
# Read port
self.buffer.extend(list(self.portRx.read(1024)))
# Step through the buffer byte and byte and see if the tick text
# is at the front.
while len(self.buffer) >= len(self.sendtext):
if self.buffer[:len(self.sendtext)] == self.sendtext:
# Discard the tick text
self.buffer = self.buffer[len(self.sendtext):]
# Record time
snapshot__sec = time.time()
self.interval__sec = snapshot__sec - self.start_time__sec
self.start_time__sec = snapshot__sec
# send more data
self.portTx.write(self.sendtext)
return True
else:
self.buffer.pop(0)
return False
def main(port1, port2, baudrate1 = DEFAULT_BAUD, baudrate2 = DEFAULT_BAUD):
try:
import serial
except:
traceback.print_exc()
print "="*60
print "You need to install PySerial"
print "Windows: easy_install pyserial"
print "Mac/Linux: sudo easy_install pyserial"
try:
s1 = serial.Serial(port1, baudrate1, timeout = 0.1)
s2 = serial.Serial(port2, baudrate2, timeout = 0.1)
print "Loading serial ports"
except:
print "Serial port error"
exit()
plot_stop = False
dataread = SerialReader(s2, s1)
try:
while plot_stop == False:
if dataread.didDataArrive():
print dataread.interval__sec
except KeyboardInterrupt:
print "Keyboard Interrupt"
plot_stop = True
finally:
print "Closing"
s1.close()
s2.close()
if __name__ == '__main__':
if (len(sys.argv) < 3):
print "Usage: python extract_data.py phonelink_serialport phonelinkclient_serialport [baudrate1] [baudrate2]"
else:
main(*sys.argv[1:])
If I remove one of the following lines (doesn't matter which one), the python script runs (although it eventually crashes because in the code it eventually tries to reference both ports). If I leave these lines in, the program seems to just hang (it just seems to sit there and run indefinitely):
s1 = serial.Serial(port1, baudrate1, timeout = 0.1)
s2 = serial.Serial(port2, baudrate2, timeout = 0.1)

python-requests with multithreading

I am working on creating a HTTP client which can generate hundreds of connections each second and send up to 10 requests on each of those connections. I am using threading so concurrency can be achieved.
Here is my code:
def generate_req(reqSession):
requestCounter = 0
while requestCounter < requestRate:
try:
response1 = reqSession.get('http://20.20.1.2/tempurl.html')
if response1.status_code == 200:
client_notify('r')
except(exceptions.ConnectionError, exceptions.HTTPError, exceptions.Timeout) as Err:
client_notify('F')
break
requestCounter += 1
def main():
for q in range(connectionPerSec):
s1 = requests.session()
t1 = threading.Thread(target=generate_req, args=(s1,))
t1.start()
Issues:
It is not scaling above 200 connections/sec with requestRate = 1. I ran other available HTTP clients on the same client machine and against the server, test runs fine and it is able to scale.
When requestRate = 10, connections/sec drops to 30.
Reason: Not able to create targeted number of threads every second.
For issue #2, client machine is not able to create enough request sessions and start new threads. As soon as requestRate is set to more than 1, things start to fall apart.
I am suspecting it has something to do with HTTP connection pooling which requests uses.
Please suggest what am I doing wrong here.

I wasn't able to get things to fall apart, however the following code has some new features:
1) extended logging, including specific per-thread information
2) all threads join()ed at the end to make sure the parent process doesntt leave them hanging
3) multithreaded print tends to interleave the messages, which can be unwieldy. This version uses yield so a future version can accept the messages and print them clearly.
source
import exceptions, requests, threading, time
requestRate = 1
connectionPerSec = 2
def client_notify(msg):
return time.time(), threading.current_thread().name, msg
def generate_req(reqSession):
requestCounter = 0
while requestCounter < requestRate:
try:
response1 = reqSession.get('http://127.0.0.1/')
if response1.status_code == 200:
print client_notify('r')
except (exceptions.ConnectionError, exceptions.HTTPError, exceptions.Timeout):
print client_notify('F')
break
requestCounter += 1
def main():
for cnum in range(connectionPerSec):
s1 = requests.session()
th = threading.Thread(
target=generate_req, args=(s1,),
name='thread-{:03d}'.format(cnum),
)
th.start()
for th in threading.enumerate():
if th != threading.current_thread():
th.join()
if __name__=='__main__':
main()
output
(1407275951.954147, 'thread-000', 'r')
(1407275951.95479, 'thread-001', 'r')

Python pyserial and runaway CPU usage

MacOS 10.7.3, python 2.5
I am using pyserial to open a connection to an external server. The connection is opened as:
HOST = '10.0.0.1'
PORT = '16010'
theURL = 'socket://' + HOST + ':' + PORT
conn = serial.serial_for_url(theURL, baudrate=115200)
conn.timeout = 2
and then the read looks like this:
try:
while len(rawData) == 0 and self.shutdown == False:
rawData = conn.readline()
except:
some error handling code...
The problem is, if I kill the server at 10.0.0.1:16010 the code keeps running, but the cpu usage goes to 100%. No error is thrown, so the except is never entered.
This looks like a problem in pyserial, but maybe someone here has run into this before and knows how to detect the lost connection so the situation can be handled gracefully.
Thanks.

If you're not depending on .readline() you could do it like this:
self.plugin.conn = Serial(..., timeout = 1)
...
if not self.shutdown:
rawData = self.plugin.conn.read(1)
if rawData:
rawData += self.plugin.conn.read(self.plugin.conn.inWaiting())
else:
raise Exception("timeout")
I'm not sure if I got your intent right so you might have to adjust...

The fact that your CPU usage is pegged probably indicates the the readline call is not blocking to timeout but returns instantly. So if your normal timeout
is 2, you could use:
from time import time
try:
while len(rawData) == 0 and self.shutdown == False:
before = time()
rawData = conn.readline()
if (len(rawData)==0) and ((time()-before)<2):
raise Exception("Early readline return.")
except:
some error handling code...

A very good solution to this can be found here:
class ReadLine:
def __init__(self, s):
self.buf = bytearray()
self.s = s
def readline(self):
i = self.buf.find(b"\n")
if i >= 0:
r = self.buf[:i+1]
self.buf = self.buf[i+1:]
return r
while True:
i = max(1, min(2048, self.s.in_waiting))
data = self.s.read(i)
i = data.find(b"\n")
if i >= 0:
r = self.buf + data[:i+1]
self.buf[0:] = data[i+1:]
return r
else:
self.buf.extend(data)
ser = serial.Serial('COM7', 9600)
rl = ReadLine(ser)
while True:
print(rl.readline())