Develop Reference

Memory Error Exception using PIL to Process Video Stream on Raspberry Pi

I have written a Python script that runs on a Raspberry Pi and utilizes the PiCamera library to capture video, the Python Image Library (PIL) to extract individual frames, and then does some image processing on it using DIPLib and OpenCV. The goal of the program is to continuously capture frames of 3D printer filament and return the diameter value. The actual image processing portion works just fine- it's the frame capture that is causing me issues.
I am following the PiCamera Rapid Capture and processing tutorial from PiCamera and using the Python Threading library as they have done to ideally utilize more of th Pi's processor for processing and not getting bottlenecked and falling behind.
The implementation of this code is built to "Drop Frames" when there are not any threads available for processing. As I understand it, this should prevent the Pi from storing any extra frames in the buffer for processing, thus preventing a memory overflow (not sure if that's the correct terminology) from happening. Unfortunately this is exactly what is happening.
I am running the PiCamera at about 3 frames-per-second which gives ~10 threads the ability to process all the incoming images, that is until the memory starts to overflow. However, if I leave the script running for 5-10 minutes, the memory (as shown using htop) slowly compounds until it reaches maximum capacity- at which point the script basically drops all incoming frames.
UPDATE: here is the error it shows:
Exception has occurred: MemoryError
exception: no description
File "/home/pi/Desktop/FilamentPuller_01/pi_camera_threading_03.py", line 45, in run
img = np.array(Image.open(self.stream)
My theory is that the video recording functionality of PiCamera is holding a buffer of some sort, but I am not sure how to see it or how to stop it from doing that. I've been using VSCode on the Pi to debug, and each thread doesn't seem to holding any more data at a time than they should- essentially there should be no reason for them to compound more data from one cycle to the next as all the variables are reused.
I have included my code below, please let me know what other information I can provide to help with solving this issue. Thank you for any insight you might have
import io
import sys
import time
import threading
import cv2
import numpy as np
import os
import picamera
from PIL import Image
import csv
from diameter_detection import diameter_detection
from moving_average import MovingAverageFilter
from serial_write import serial_write
##### CAMERA SETTINGS ######
focalValue = 40 # focus
cameraResolution = (1080, 1000)
cameraZoom = (0.3,0,0.3,0.8)
cameraFrameRate = 1
# create an array for storing filtered diameter values
filtered_dia_meas = []
# create moving average filter
ma5_filter = MovingAverageFilter(2)
class ImageProcessor(threading.Thread):
def __init__(self, owner):
super(ImageProcessor, self).__init__()
self.stream = io.BytesIO()
self.event = threading.Event()
self.terminated = False
self.owner = owner
self.start()
def run(self):
# This method runs in a separate thread
while not self.terminated:
# Wait for an image to be written to the stream
if self.event.wait(1):
try:
self.stream.seek(0)
# Read the image and do some processing on it
img = np.array(Image.open(self.stream))
try:
diameter = diameter_detection(img)
except:
serial_write(0)
print('Could not read diameter, pausing and retrying...')
time.sleep(0.1)
# add the diameter to the filter
ma5_filter.step(diameter)
#filtered_dia_meas.append(ma5_filter.current_state())
# display the current filtered diameter to the Terminal
print(ma5_filter.current_state())
try:
# attempt to send the diameter to the connected serial device
serial_write(ma5_filter.current_state())
except:
print('Serial write failed!')
# Set done to True if you want the script to terminate
# at some point
#self.owner.done=True
finally:
# Reset the stream and event
self.stream.seek(0)
self.stream.truncate()
self.event.clear()
# Return ourselves to the available pool
with self.owner.lock:
self.owner.pool.append(self)
class ProcessOutput(object):
def __init__(self):
self.done = False
# Construct a pool of 10 image processors along with a lock
# to control access between threads
self.lock = threading.Lock()
self.pool = [ImageProcessor(self) for i in range(10)]
print('Threaded processes created')
self.processor = None
def write(self, buf):
if buf.startswith(b'\xff\xd8'):
# New frame; set the current processor going and grab
# a spare one
if self.processor:
self.processor.event.set()
with self.lock:
if self.pool:
self.processor = self.pool.pop()
else:
# No processor's available, we'll have to skip this frame
print('Frame Skipped!')
self.processor = None
if self.processor:
self.processor.stream.write(buf)
def flush(self):
# When told to flush (end of recording), shut
# down in an orderly fashion. First, add the current processor
# back to the pool
if self.processor:
with self.lock:
self.pool.append(self.processor)
self.processor = None
# Now, empty the pool, joining each thread as we go
while True:
with self.lock:
try:
proc = self.pool.pop()
except IndexError:
pass # pool is empty
proc.terminated = True
proc.join()
with picamera.PiCamera(resolution=cameraResolution) as camera:
print('Succesfully created camera object')
camera.framerate = cameraFrameRate
camera.zoom = (0.3,0,0.3,0.8)
# set focus motor
os.system("i2cset -y 0 0x0c %d %d" % (focalValue,0))
print('Camera focus set')
time.sleep(2)
print('Starting recording...')
output = ProcessOutput()
camera.start_recording(output, format='mjpeg')
while not output.done:
camera.wait_recording(1)
camera.stop_recording()

Threading in python on a raspberry pi issues

I am trying to run a GUI that spawns threads that perform very basic and not computationally complicated tasks on a raspberry pi 1 and I cannot seem to get the threads to work.
I developed the code on a x86 intel computer and it works perfectly well. The threading commands basically just allow for button presses and listening for serial data concurrently.
def extra_thread_disable():
# Disables buttons that would interfere with data that is currently being sent
while threading.active_count() == 3:
run_file_butt.config(state = 'disabled')
run_butt.config(state = 'disabled')
serial_butt.config(state = 'disabled')
popup_butt.config(state = 'disabled')
homing_butt.config(state = 'disabled')
level_butt.config(state = 'disabled')
zero_button1.config(state = 'disabled')
zero_button2.config(state = 'disabled')
zero_button3.config(state = 'disabled')
else:
run_file_butt.config(state = 'normal')
run_butt.config(state = 'normal')
serial_butt.config(state = 'normal')
popup_butt.config(state = 'normal')
homing_butt.config(state = 'normal')
level_butt.config(state = 'normal')
zero_button1.config(state = 'normal')
zero_button2.config(state = 'normal')
zero_button3.config(state = 'normal')
pass
def thread_data():
# Starts a thread to send data while allowing stop button to be pressed
try:
global t2
t2 = threading.Thread(name='send_line', target = send_data, daemon = True)
t_disable = threading.Thread(name='disable', target = extra_thread_disable, daemon = True)
t2.start()
t_disable.start()
except:
update_textbox("Threading Error: data thread not properly created")
def send_data():
# Sends single motion commands and waits for response to continue
global save_path
global motor_param
vals = get_vals()
try:
data = struct.pack("!llllhhhhhhhh", vals['dist1'], vals['dist2'], vals['dist34'], vals['dist34'], vals['speed1'], vals['speed2'], vals['speed34'], vals['speed34'], vals['accel1'], vals['accel2'], vals['accel34'], vals['accel34'])
try:
ser.write(data)
update_textbox("Running...")
except:
update_textbox("Error: Data not sent")
try:
motor1pos = int(ser.readline())
motor2pos = int(ser.readline())
motor3pos = int(ser.readline())
motor4pos = int(ser.readline())
ready = ser.read(1)
update_textbox("Movement complete")
axis1_current.set(str(reverse_convert(motor1pos, 1)))
axis2_current.set(str(reverse_convert(motor2pos, 2)))
axis3_current.set(str(reverse_convert(motor3pos, 3)))
writetofile()
except:
update_textbox("Error: reading data improperly")
except:
update_textbox("Error: data not sent properly")
pass
The code basically just allows the main GUI thread to allow for a stop button to be pressed and disable all the buttons that could interfere with the sent data. That thread then just waits for the response from an arduino it is connected to. Again this all works flawlessly on a normal computer. I get no errors or warnings in the terminal when run on a raspberry pi but it seems to be blocking. I thought maybe it was just such a slow computer or the infamous GIL. It seems like that might be the reason. If so, should I switch to the multiprocessing library in python? is there a way to get around this? It doesn't work when run in terminal calling python3 and it doesn't work when it was compiled to a static binary using pyinstaller.

The answer is to get better hardware. I changed nothing in my code but instead bought a newer raspberry pi 4 instead of the original raspberry pi and the threading worked as it originally did on my other pc running arch.

Python Queues memory leaks when called inside thread

I have python TCP client and need to send media(.mpg) file in a loop to a 'C' TCP server.
I have following code, where in separate thread I am reading the 10K blocks of file and sending it and doing it all over again in loop, I think it is because of my implementation of thread module, or tcp send. I am using Queues to print the logs on my GUI ( Tkinter ) but after some times it goes out of memory..
UPDATE 1 - Added more code as requested
Thread class "Sendmpgthread" used to create thread to send data
.
.
def __init__ ( self, otherparams,MainGUI):
.
.
self.MainGUI = MainGUI
self.lock = threading.Lock()
Thread.__init__(self)
#This is the one causing leak, this is called inside loop
def pushlog(self,msg):
self.MainGUI.queuelog.put(msg)
def send(self, mysocket, block):
size = len(block)
pos = 0;
while size > 0:
try:
curpos = mysocket.send(block[pos:])
except socket.timeout, msg:
if self.over:
self.pushlog(Exit Send)
return False
except socket.error, msg:
print 'Exception'
return False
pos = pos + curpos
size = size - curpos
return True
def run(self):
media_file = None
mysocket = None
try:
mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysocket.connect((self.ip, string.atoi(self.port)))
media_file = open(self.file, 'rb')
while not self.over:
chunk = media_file.read(10000)
if not chunk: # EOF Reset it
print 'resetting stream'
media_file.seek(0, 0)
continue
if not self.send(mysocket, chunk): # If some error or thread is killed
break;
#disabling this solves the issue
self.pushlog('print how much data sent')
except socket.error, msg:
print 'print exception'
except Exception, msg:
print 'print exception'
try:
if media_file is not None:
media_file.close()
media_file = None
if mysocket is not None:
mysocket.close()
mysocket = None
finally:
print 'some cleaning'
def kill(self):
self.over = True
I figured out that it is because of wrong implementation of Queue as commenting that piece resolves the issue
UPDATE 2 - MainGUI class which is called from above Thread class
class MainGUI(Frame):
def __init__(self, other args):
#some code
.
.
#from the above thread class used to send data
self.send_mpg_status = Sendmpgthread(params)
self.send_mpg_status.start()
self.after(100, self.updatelog)
self.queuelog = Queue.Queue()
def updatelog(self):
try:
msg = self.queuelog.get_nowait()
while msg is not None:
self.printlog(msg)
msg = self.queuelog.get_nowait()
except Queue.Empty:
pass
if self.send_mpg_status: # only continue when sending
self.after(100, self.updatelog)
def printlog(self,msg):
#print in GUI

Since printlog is adding to a tkinter text control, the memory occupied by that control will grow with each message (it has to store all the log messages in order to display them).
Unless storing all the logs is critical, a common solution is to limit the maximum number of log lines displayed.
A naive implementation is to eliminate extra lines from the begining after the control reaches a maximum number of messages. Add a function to get the number of lines in the control and then, in printlog something similar to:
while getnumlines(self.edit) > self.maxloglines:
self.edit.delete('1.0', '1.end')
(above code not tested)
update: some general guidelines
Keep in mind that what might look like a memory leak does not always mean that a function is wrong, or that the memory is no longer accessible. Many times there is missing cleanup code for a container that is accumulating elements.
A basic general approach for this kind of problems:
form an opinion on what part of the code might be causing the problem
check it by commenting that code out (or keep commenting code until you find a candidate)
look for containers in the responsible code, add code to print their size
decide what elements can be safely removed from that container, and when to do it
test the result

I can't see anything obviously wrong with your code snippet.
To reduce memory usage a bit under Python 2.7, I'd use buffer(block, pos) instead of block[pos:]. Also I'd use mysocket.sendall(block) instead of your send method.
If the ideas above don't solve your problem, then the bug is most probably elsewhere in your code. Could you please post the shortest possible version of the full Python script which still grows out-of-memory (http://sscce.org/)? That increases your change of getting useful help.

Out of memory errors are indicative of data being generated but not consumed or released. Looking through your code I would guess these two areas:
Messages are being pushed onto a Queue.Queue() instance in the pushlog method. Are they being consumed?
The MainGui printlog method may be writing text somewhere. eg. Is it continually writing to some kind of GUI widget without any pruning of messages?
From the code you've posted, here's what I would try:
Put a print statement in updatelog. If this is not being continually called for some reason such as a failed after() call, then the queuelog will continue to grow without bound.
If updatelog is continually being called, then turn your focus to printlog. Comment the contents of this function to see if out of memory errors still occur. If they don't, then something in printlog may be holding on to the logged data, you'll need to dig deeper to find out what.
Apart from this, the code could be cleaned up a bit. self.queuelog is not created until after the thread is started which gives rise to a race condition where the thread may try to write into the queue before it has been created. Creation of queuelog should be moved to somewhere before the thread is started.
updatelog could also be refactored to remove redundancy:
def updatelog(self):
try:
while True:
msg = self.queuelog.get_nowait()
self.printlog(msg)
except Queue.Empty:
pass
And I assume the the kill function is called from the GUI thread. To avoid thread race conditions, the self.over should be a thread safe variable such as a threading.Event object.
def __init__(...):
self.over = threading.Event()
def kill(self):
self.over.set()

There is no data piling up in your TCP sending loop.
Memory error is probably caused by logging queue, as you have not posted complete code try using following class for logging:
from threading import Thread, Event, Lock
from time import sleep, time as now
class LogRecord(object):
__slots__ = ["txt", "params"]
def __init__(self, txt, params):
self.txt, self.params = txt, params
class AsyncLog(Thread):
DEBUGGING_EMULATE_SLOW_IO = True
def __init__(self, queue_max_size=15, queue_min_size=5):
Thread.__init__(self)
self.queue_max_size, self.queue_min_size = queue_max_size, queue_min_size
self._queuelock = Lock()
self._queue = [] # protected by _queuelock
self._discarded_count = 0 # protected by _queuelock
self._pushed_event = Event()
self.setDaemon(True)
self.start()
def log(self, message, **params):
with self._queuelock:
self._queue.append(LogRecord(message, params))
if len(self._queue) > self.queue_max_size:
# empty the queue:
self._discarded_count += len(self._queue) - self.queue_min_size
del self._queue[self.queue_min_size:] # empty the queue instead of creating new list (= [])
self._pushed_event.set()
def run(self):
while 1: # no reason for exit condition here
logs, discarded_count = None, 0
with self._queuelock:
if len(self._queue) > 0:
# select buffered messages for printing, releasing lock ASAP
logs = self._queue[:]
del self._queue[:]
self._pushed_event.clear()
discarded_count = self._discarded_count
self._discarded_count = 0
if not logs:
self._pushed_event.wait()
self._pushed_event.clear()
continue
else:
# print logs
if discarded_count:
print ".. {0} log records missing ..".format(discarded_count)
for log_record in logs:
self.write_line(log_record)
if self.DEBUGGING_EMULATE_SLOW_IO:
sleep(0.5)
def write_line(self, log_record):
print log_record.txt, " ".join(["{0}={1}".format(name, value) for name, value in log_record.params.items()])
if __name__ == "__main__":
class MainGUI:
def __init__(self):
self._async_log = AsyncLog()
self.log = self._async_log.log # stored as bound method
def do_this_test(self):
print "I am about to log 100 times per sec, while text output frequency is 2Hz (twice per second)"
def log_100_records_in_one_second(itteration_index):
for i in xrange(100):
self.log("something happened", timestamp=now(), session=3.1415, itteration=itteration_index)
sleep(0.01)
for iter_index in range(3):
log_100_records_in_one_second(iter_index)
test = MainGUI()
test.do_this_test()
I have noticed that you do not sleep() anywhere in the sending loop, this means data is read as fast as it can and is sent as fast as it can. Note that this is not desirable behavior when playing media files - container time-stamps are there to dictate data-rate.

Python Thread communication not working

I have two threads, Reader and Writer.
The Writer gets data from the network and sends it then over a socket to some executable. When this is done the writer should block up to 70 seconds which I specify with a Event.wait(askrate).
This should give the executable enough time to compute the result and then submit the output. If the computation is finished I used Event.set() to release the lock on the Writer
thread so that it can read the next data that is forwared to the executeable and so on.
The problem that I have is, that the Writer thread still keeps reading data while the Reader thread is waiting for the result coming through the serial interface.
Anyone an idea why this blocking meachnism is not proberly working between these two threads?
askrate = 70
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect("/tmp/demo_socket")
class Reader(Thread):
def __init__(self):
Thread.__init__(self)
self.daemon = True
def run(self):
while True:
nonce = s.recv(4)
if len(nonce) == 4:
submitter = Submitter(writer.block, nonce)
#submit result and release thread lock in Writer class
golden.set()
class Writer(Thread):
def __init__(self):
Thread.__init__(self)
self.daemon = True
def run(self):
while True:
work = bc.getwork()
self.block = work['data']
self.midstate = work['midstate']
payload = self.midstate.decode('hex') + self.block.decode('hex')
s.send(payload)
result = golden.wait(askrate)
if result:
golden.clear()
golden = Event()
reader = Reader()
writer = Writer()
reader.start()
writer.start()

I'm pretty sure that it's not how you are supposed to use AF_UNIX sockets. You are supposed to open the pseudo-file twice (from the same of different processes); then writes to one side appear as reads on the other side, and vice-versa. In your code, you open the pseudo-file only once. Any write is probably blocking, waiting for another process to open the pseudo-file a second time.
In your case, you should use socket.socketpair(), which returns you two sockets at once, playing the role of the two ends. Use one end in each thread.

Dynamically updating Tkinter window based on serial data

I'm trying to write a program that gets data from a serial port connection and automatically updates the Tkinter window in real time based on that data.
I tried to create a separate thread for the window that periodically gets the current data from the main thread and updates the window, like this:
serialdata = []
data = True
class SensorThread(threading.Thread):
def run(self):
serial = serial.Serial('dev/tty.usbmodem1d11', 9600)
try:
while True:
serialdata.append(serial.readline())
except KeyboardInterrupt:
serial.close()
exit()
class GuiThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.root = Tk()
self.lbl = Label(self.root, text="")
def run(self):
self.lbl(pack)
self.lbl.after(1000, self.updateGUI)
self.root.mainloop()
def updateGUI(self):
msg = "Data is True" if data else "Data is False"
self.lbl["text"] = msg
self.root.update()
self.lbl.after(1000, self.updateGUI)
if __name == "__main__":
SensorThread().start()
GuiThread().start()
try:
while True:
# A bunch of analysis that sets either data = True or data = False based on serialdata
except KeyboardInterrupt:
exit()
Running it gives me this error:
Exception in thread Thread-2:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 522, in __bootstrap_inner
self.run()
File "analysis.py", line 52, in run
self.lbl1.pack()
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk/Tkinter.py", line 1764, in pack_configure
+ self._options(cnf, kw))
RuntimeError: main thread is not in main loop
When I google this error, I mostly get posts where people are trying to interact with the window from two different threads, but I don't think I'm doing that. Any ideas? Thanks so much!

Don't run the TK gui from a thread - run it from the main process. I mashed your example into something that demonstrates the principle
from time import sleep
import threading
from Tkinter import *
serialdata = []
data = True
class SensorThread(threading.Thread):
def run(self):
try:
i = 0
while True:
serialdata.append("Hello %d" % i)
i += 1
sleep(1)
except KeyboardInterrupt:
exit()
class Gui(object):
def __init__(self):
self.root = Tk()
self.lbl = Label(self.root, text="")
self.updateGUI()
self.readSensor()
def run(self):
self.lbl.pack()
self.lbl.after(1000, self.updateGUI)
self.root.mainloop()
def updateGUI(self):
msg = "Data is True" if data else "Data is False"
self.lbl["text"] = msg
self.root.update()
self.lbl.after(1000, self.updateGUI)
def readSensor(self):
self.lbl["text"] = serialdata[-1]
self.root.update()
self.root.after(527, self.readSensor)
if __name__ == "__main__":
SensorThread().start()
Gui().run()

You need to put the GUI in the main thread, and use a separate thread to poll the serial port. When you read data off of the serial port you can push it onto a Queue object.
In the main GUI thread you can set up polling to check the queue periodically, by using after to schedule the polling. Call a function which drains the queue and then calls itself with after to effectively emulate an infinite loop.
If the data that comes from the sensor comes at a fairly slow rate, and you can poll the serial port without blocking, you can do that all in the main thread -- instead of pushing and pulling from the queue, your main thread can just see if there's data available and read it if there is. You can only do this if it's possible to read without blocking, otherwise your GUI will freeze while it waits for data.
For example, you could make it work like this:
def poll_serial_port(self):
if serial.has_data():
data = serial.readline()
self.lbl.configure(text=data)
self.after(100, self.poll_serial_port)
The above will check the serial port 10 times per second, pulling one item off at a time. You'll have to adjust that for your actual data conditions of course. This assumes that you have some method like has_data that can return True if and only if a read won't block.