I'm writing a program that looks for objects in a video stream using a Google Coral device.
I have a function that continuously detects objects in the video.
I want to 'stream' the data from that function to two other functions/classes:
VideoViewer: This is a class that manages a debug window, showing the bounding boxes, etc.
An API endpoint: This part of the application has not been implemented yet, but I want to expose the data from the object detection function, via a websocket, later on.
The following is what I have so far. It yields the results
from PIL import Image
from cv2 import VideoCapture
def detect_with_video(self, video: VideoCapture):
"""Detect objects in a video"""
streaming = True
frame_id = 0
results = []
detection_delay = 4
while streaming:
_, frame = video.read()
if frame_id % detection_delay == 0:
results = self._detect_image(Image.fromarray(frame))
for result in results:
yield result
In my app.py I would like to be able to do something like this:
class App:
"""The application"""
def __init__(self):
self.video: VideoCapture = cv2.VideoCapture(0)
def run(self):
# Initialize video
VideoViewer().new_window()
generator = Detector().detect_with_video(self.video)
VideoViewer().set_stream(generator)
HttpApi().set_stream(generator)
class VideoViewer:
"""VideoViewer launches video windows"""
def set_stream(self, generator):
for item in generator:
print(item)
My guess is that I have to make the set_stream methods async, but yeah.
How can I make both consumers non blocking?
I'm new to python and aren't sure how to do this. Any help is highly appreciated.
Related
I have written a Python script that runs on a Raspberry Pi and utilizes the PiCamera library to capture video, the Python Image Library (PIL) to extract individual frames, and then does some image processing on it using DIPLib and OpenCV. The goal of the program is to continuously capture frames of 3D printer filament and return the diameter value. The actual image processing portion works just fine- it's the frame capture that is causing me issues.
I am following the PiCamera Rapid Capture and processing tutorial from PiCamera and using the Python Threading library as they have done to ideally utilize more of th Pi's processor for processing and not getting bottlenecked and falling behind.
The implementation of this code is built to "Drop Frames" when there are not any threads available for processing. As I understand it, this should prevent the Pi from storing any extra frames in the buffer for processing, thus preventing a memory overflow (not sure if that's the correct terminology) from happening. Unfortunately this is exactly what is happening.
I am running the PiCamera at about 3 frames-per-second which gives ~10 threads the ability to process all the incoming images, that is until the memory starts to overflow. However, if I leave the script running for 5-10 minutes, the memory (as shown using htop) slowly compounds until it reaches maximum capacity- at which point the script basically drops all incoming frames.
UPDATE: here is the error it shows:
Exception has occurred: MemoryError
exception: no description
File "/home/pi/Desktop/FilamentPuller_01/pi_camera_threading_03.py", line 45, in run
img = np.array(Image.open(self.stream)
My theory is that the video recording functionality of PiCamera is holding a buffer of some sort, but I am not sure how to see it or how to stop it from doing that. I've been using VSCode on the Pi to debug, and each thread doesn't seem to holding any more data at a time than they should- essentially there should be no reason for them to compound more data from one cycle to the next as all the variables are reused.
I have included my code below, please let me know what other information I can provide to help with solving this issue. Thank you for any insight you might have
import io
import sys
import time
import threading
import cv2
import numpy as np
import os
import picamera
from PIL import Image
import csv
from diameter_detection import diameter_detection
from moving_average import MovingAverageFilter
from serial_write import serial_write
##### CAMERA SETTINGS ######
focalValue = 40 # focus
cameraResolution = (1080, 1000)
cameraZoom = (0.3,0,0.3,0.8)
cameraFrameRate = 1
# create an array for storing filtered diameter values
filtered_dia_meas = []
# create moving average filter
ma5_filter = MovingAverageFilter(2)
class ImageProcessor(threading.Thread):
def __init__(self, owner):
super(ImageProcessor, self).__init__()
self.stream = io.BytesIO()
self.event = threading.Event()
self.terminated = False
self.owner = owner
self.start()
def run(self):
# This method runs in a separate thread
while not self.terminated:
# Wait for an image to be written to the stream
if self.event.wait(1):
try:
self.stream.seek(0)
# Read the image and do some processing on it
img = np.array(Image.open(self.stream))
try:
diameter = diameter_detection(img)
except:
serial_write(0)
print('Could not read diameter, pausing and retrying...')
time.sleep(0.1)
# add the diameter to the filter
ma5_filter.step(diameter)
#filtered_dia_meas.append(ma5_filter.current_state())
# display the current filtered diameter to the Terminal
print(ma5_filter.current_state())
try:
# attempt to send the diameter to the connected serial device
serial_write(ma5_filter.current_state())
except:
print('Serial write failed!')
# Set done to True if you want the script to terminate
# at some point
#self.owner.done=True
finally:
# Reset the stream and event
self.stream.seek(0)
self.stream.truncate()
self.event.clear()
# Return ourselves to the available pool
with self.owner.lock:
self.owner.pool.append(self)
class ProcessOutput(object):
def __init__(self):
self.done = False
# Construct a pool of 10 image processors along with a lock
# to control access between threads
self.lock = threading.Lock()
self.pool = [ImageProcessor(self) for i in range(10)]
print('Threaded processes created')
self.processor = None
def write(self, buf):
if buf.startswith(b'\xff\xd8'):
# New frame; set the current processor going and grab
# a spare one
if self.processor:
self.processor.event.set()
with self.lock:
if self.pool:
self.processor = self.pool.pop()
else:
# No processor's available, we'll have to skip this frame
print('Frame Skipped!')
self.processor = None
if self.processor:
self.processor.stream.write(buf)
def flush(self):
# When told to flush (end of recording), shut
# down in an orderly fashion. First, add the current processor
# back to the pool
if self.processor:
with self.lock:
self.pool.append(self.processor)
self.processor = None
# Now, empty the pool, joining each thread as we go
while True:
with self.lock:
try:
proc = self.pool.pop()
except IndexError:
pass # pool is empty
proc.terminated = True
proc.join()
with picamera.PiCamera(resolution=cameraResolution) as camera:
print('Succesfully created camera object')
camera.framerate = cameraFrameRate
camera.zoom = (0.3,0,0.3,0.8)
# set focus motor
os.system("i2cset -y 0 0x0c %d %d" % (focalValue,0))
print('Camera focus set')
time.sleep(2)
print('Starting recording...')
output = ProcessOutput()
camera.start_recording(output, format='mjpeg')
while not output.done:
camera.wait_recording(1)
camera.stop_recording()
I want to load a gif in ursina engine using the Animation class, it works flawlessly but when I want to reassign the Animation to another gif the memory stacks up like if I opened another gif. I tried to delete the instance and then recreate the gif but the memory keeps stacking up, also I tried to force a garbage collection but it doesn't work either.
What can I do?
The code is something like this:
from ursina import *
from ursina.prefabs.animation import Animation
import gc
app = Ursina()
FC={"FC":80}
monitor = Animation("monitor\gifs\monitor-%d.gif" % (FC["FC"]),loop=False)
monitor.enable()
def recreate_monitor():
global monitor
monitor = Animation("monitor\gifs\monitor-%d.gif" % (FC["FC"]),loop=False)
monitor.enabled = True
def refresh_monitor():
global monitor
monitor.sequence.start()
def update():
global monitor
if not monitor:
create_monitor()
else:
if monitor.sequence.finished:
if FC["FC"] != 80:
monitor.remove_node()
del monitor
print("DATA RECOLLECTED: ", gc.collect())
create_monitor()
else:
refresh_monitor()
def input(key):
if key == "space":
FC["FC"] = 100
app.run()
I'm trying to understand why some functions fail with threading in python.
For example, I'm trying to use VideoCapture to grab an image from a webcam.
This example works fine:
from VideoCapture import Device
cam = Device()
cam.saveSnapshot('image.jpg')
But when I put it on a thread I get an error.
import threading
from VideoCapture import Device
def grab():
cam = Device()
cam.saveSnapshot('image.jpg')
thr = threading.Thread(target=grab)
thr.start()
thr.join()
File "C:\Program
Files\Python36\lib\site-packages\VideoCapture__init__.py", line 60,
in init
self.dev = vidcap.new_Dev(devnum, showVideoWindow) vidcap.Error: Creation of the filter graph failed.
According to this reference, this function is not thread-safe. So is there any workaround to bypass similar issues like this? I tried to use threading.lock but got the same error. If I need to change the code, which part should I check?
I couldn't find a direct way to fix it, but importing the module inside the thread fixed that, I'm not sure if this applies to other modules.
import threading
def grab():
from VideoCapture import Device
cam = Device()
cam.saveSnapshot('image.jpg')
del cam
thr = threading.Thread(target=grab)
thr.start()
thr.join()
I have 4 video files (different scenes of a movie).
There's a starting scene that will be played when I run the player.
And before that scene ends, let's say the video player reads an int value (0-100) from external file (all happens at runtime), and depending on that int value, it has to determine which scene to play next.
pseudo example:
if (x > 0 && x < 30)
videoSource = scene2
else if (x >= 30 && x < 60)
videoSource = scene3
else if (x >= 60 && x <= 100)
videoSource = scene 4
How can I make it change video sources at runtime, depending on that variable?
I don't care about the format of the video file, (Avi, mp4...) whatever works will be fine.
I don't know how to approach this problem. I've searched for something that has the potential to accomplish this, like pyglet or GStreamer, but I didn't find a clear solution.
EDIT: I have the basic player and video player with pyglet, and I was able to play the video without depending on a variable using this code:
import pyglet
vidPath="sample.mpg"
window = pyglet.window.Window()
player = pyglet.media.Player()
source = pyglet.media.StreamingSource()
MediaLoad = pyglet.media.load(vidPath)
player.queue(MediaLoad)
player.play()
#window.event
def on_draw():
window.clear()
if player.source and player.source.video_format:
player.get_texture().blit(0,0)
pyglet.app.run()
How would I go about this? Guidance in the right direction and/or some sample code would be highly appreciated.
Thanks in advance.
Answer revised based on comments
If your goal is to constantly read a file that is receiving writes from the output of another process, you have a couple aspects that need to be solved...
You either need to read a file periodically that is constantly being overwritten, or you need to tail the output of a file that is being appended to with new values.
Your script currently blocks when you start the pyglet event loop, so this file check will have to be in a different thread, and then you would have to communicate the update event.
I can't fully comment on step 2 because I have never used pyglet and I am not familiar with how it uses events or signals. But I can at least suggest half of it with a thread.
Here is a super basic example of using a thread to read a file and report when a line is found:
import time
from threading import Thread
class Monitor(object):
def __init__(self):
self._stop = False
def run(self, inputFile, secs=3):
self._stop = False
with open(inputFile) as monitor:
while True:
line = monitor.readline().strip()
if line.isdigit():
# this is where you would notify somehow
print int(line)
time.sleep(secs)
if self._stop:
return
def stop(self):
self._stop = True
if __name__ == "__main__":
inputFile = "write.txt"
monitor = Monitor()
monitorThread = Thread(target=monitor.run, args=(inputFile, 1))
monitorThread.start()
try:
while True:
time.sleep(.25)
except:
monitor.stop()
The sleep loop at the end of the code is just a way to emulate your event loop and block.
Here is a test to show how it would work. First I open a python shell and open a new file:
>>> f = open("write.txt", 'w+', 10)
Then you can start this script. And back in the shell you can start writing lines:
>>> f.write('50\n'); f.flush()
In your script terminal you will see it read and print the lines.
The other way would be if your process that is writing to this file is constantly overwriting it, you would instead just reread the file by setting monitor.seek(0) and calling readline().
Again this is a really simple example to get you started. There are more advanced ways of solving this I am sure. The next step would be to figure out how you can signal the pyglet event loop to call a method that will change your video source.
Update
You should review this section of the pyglet docs on how to create your own event dispatcher: http://pyglet.org/doc/programming_guide/creating_your_own_event_dispatcher.html
Again, without much knowledge of pyglet, here is what it might look like:
class VideoNotifier(pyglet.event.EventDispatcher):
def updateIndex(self, value):
self.dispatch_events('on_update_index', value)
VideoNotifier.register_event('on_update_index')
videoNotifier = VideoNotifier()
#videoNotifier.event
def on_update_index(newIndex):
# thread has notified of an update
# Change the video here
pass
And for your thread class, you would pass in the dispatcher instance, and use the updateIndex() event to notify:
class Monitor(object):
def __init__(self, dispatcher):
self._stop = False
self._dispatcher = dispatcher
def run(self, inputFile, secs=3):
...
...
# should notify video of new value
line = int(line_from_file)
self._dispatcher.updateIndex(line)
...
...
Hope that gets you started!
I am trying to capture a single frame from the Apple iSight camera built into a Macbook Pro using Python (version 2.7 or 2.6) and the PyObjC (version 2.2).
As a starting point, I used this old StackOverflow question. To verify that it makes sense, I cross-referenced against Apple's MyRecorder example that it seems to be based on. Unfortunately, my script does not work.
My big questions are:
Am I initializing the camera correctly?
Am I starting the event loop correctly?
Was there any other setup I was supposed to do?
In the example script pasted below, the intended operation is that after calling startImageCapture(), I should start printing "Got a frame..." messages from the CaptureDelegate. However, the camera's light never turns on and the delegate's callback is never executed.
Also, there are no failures during startImageCapture(), all functions claim to succeed, and it successfully finds the iSight device. Analyzing the session object in pdb shows that it has valid input and output objects, the output has a delegate assigned, the device is not in use by another processes, and the session is marked as running after startRunning() is called.
Here's the code:
#!/usr/bin/env python2.7
import sys
import os
import time
import objc
import QTKit
import AppKit
from Foundation import NSObject
from Foundation import NSTimer
from PyObjCTools import AppHelper
objc.setVerbose(True)
class CaptureDelegate(NSObject):
def captureOutput_didOutputVideoFrame_withSampleBuffer_fromConnection_(self, captureOutput,
videoFrame, sampleBuffer,
connection):
# This should get called for every captured frame
print "Got a frame: %s" % videoFrame
class QuitClass(NSObject):
def quitMainLoop_(self, aTimer):
# Just stop the main loop.
print "Quitting main loop."
AppHelper.stopEventLoop()
def startImageCapture():
error = None
# Create a QT Capture session
session = QTKit.QTCaptureSession.alloc().init()
# Find iSight device and open it
dev = QTKit.QTCaptureDevice.defaultInputDeviceWithMediaType_(QTKit.QTMediaTypeVideo)
print "Device: %s" % dev
if not dev.open_(error):
print "Couldn't open capture device."
return
# Create an input instance with the device we found and add to session
input = QTKit.QTCaptureDeviceInput.alloc().initWithDevice_(dev)
if not session.addInput_error_(input, error):
print "Couldn't add input device."
return
# Create an output instance with a delegate for callbacks and add to session
output = QTKit.QTCaptureDecompressedVideoOutput.alloc().init()
delegate = CaptureDelegate.alloc().init()
output.setDelegate_(delegate)
if not session.addOutput_error_(output, error):
print "Failed to add output delegate."
return
# Start the capture
print "Initiating capture..."
session.startRunning()
def main():
# Open camera and start capturing frames
startImageCapture()
# Setup a timer to quit in 10 seconds (hack for now)
quitInst = QuitClass.alloc().init()
NSTimer.scheduledTimerWithTimeInterval_target_selector_userInfo_repeats_(10.0,
quitInst,
'quitMainLoop:',
None,
False)
# Start Cocoa's main event loop
AppHelper.runConsoleEventLoop(installInterrupt=True)
print "After event loop"
if __name__ == "__main__":
main()
Thanks for any help you can provide!
OK, I spent a day diving through the depths of PyObjC and got it working.
For future record, the reason the code in the question did not work: variable scope and garbage collection. The session variable was deleted when it fell out of scope, which happened before the event processor ran. Something must be done to retain it so it is not freed before it has time to run.
Moving everything into a class and making session a class variable made the callbacks start working. Additionally, the code below demonstrates getting the frame's pixel data into bitmap format and saving it via Cocoa calls, and also how to copy it back into Python's world-view as a buffer or string.
The script below will capture a single frame
#!/usr/bin/env python2.7
#
# camera.py -- by Trevor Bentley (02/04/2011)
#
# This work is licensed under a Creative Commons Attribution 3.0 Unported License.
#
# Run from the command line on an Apple laptop running OS X 10.6, this script will
# take a single frame capture using the built-in iSight camera and save it to disk
# using three methods.
#
import sys
import os
import time
import objc
import QTKit
from AppKit import *
from Foundation import NSObject
from Foundation import NSTimer
from PyObjCTools import AppHelper
class NSImageTest(NSObject):
def init(self):
self = super(NSImageTest, self).init()
if self is None:
return None
self.session = None
self.running = True
return self
def captureOutput_didOutputVideoFrame_withSampleBuffer_fromConnection_(self, captureOutput,
videoFrame, sampleBuffer,
connection):
self.session.stopRunning() # I just want one frame
# Get a bitmap representation of the frame using CoreImage and Cocoa calls
ciimage = CIImage.imageWithCVImageBuffer_(videoFrame)
rep = NSCIImageRep.imageRepWithCIImage_(ciimage)
bitrep = NSBitmapImageRep.alloc().initWithCIImage_(ciimage)
bitdata = bitrep.representationUsingType_properties_(NSBMPFileType, objc.NULL)
# Save image to disk using Cocoa
t0 = time.time()
bitdata.writeToFile_atomically_("grab.bmp", False)
t1 = time.time()
print "Cocoa saved in %.5f seconds" % (t1-t0)
# Save a read-only buffer of image to disk using Python
t0 = time.time()
bitbuf = bitdata.bytes()
f = open("python.bmp", "w")
f.write(bitbuf)
f.close()
t1 = time.time()
print "Python saved buffer in %.5f seconds" % (t1-t0)
# Save a string-copy of the buffer to disk using Python
t0 = time.time()
bitbufstr = str(bitbuf)
f = open("python2.bmp", "w")
f.write(bitbufstr)
f.close()
t1 = time.time()
print "Python saved string in %.5f seconds" % (t1-t0)
# Will exit on next execution of quitMainLoop_()
self.running = False
def quitMainLoop_(self, aTimer):
# Stop the main loop after one frame is captured. Call rapidly from timer.
if not self.running:
AppHelper.stopEventLoop()
def startImageCapture(self, aTimer):
error = None
print "Finding camera"
# Create a QT Capture session
self.session = QTKit.QTCaptureSession.alloc().init()
# Find iSight device and open it
dev = QTKit.QTCaptureDevice.defaultInputDeviceWithMediaType_(QTKit.QTMediaTypeVideo)
print "Device: %s" % dev
if not dev.open_(error):
print "Couldn't open capture device."
return
# Create an input instance with the device we found and add to session
input = QTKit.QTCaptureDeviceInput.alloc().initWithDevice_(dev)
if not self.session.addInput_error_(input, error):
print "Couldn't add input device."
return
# Create an output instance with a delegate for callbacks and add to session
output = QTKit.QTCaptureDecompressedVideoOutput.alloc().init()
output.setDelegate_(self)
if not self.session.addOutput_error_(output, error):
print "Failed to add output delegate."
return
# Start the capture
print "Initiating capture..."
self.session.startRunning()
def main(self):
# Callback that quits after a frame is captured
NSTimer.scheduledTimerWithTimeInterval_target_selector_userInfo_repeats_(0.1,
self,
'quitMainLoop:',
None,
True)
# Turn on the camera and start the capture
self.startImageCapture(None)
# Start Cocoa's main event loop
AppHelper.runConsoleEventLoop(installInterrupt=True)
print "Frame capture completed."
if __name__ == "__main__":
test = NSImageTest.alloc().init()
test.main()
QTKit is deprecated and PyObjC is a big dependency (and seems to be tricky to build if you want it in HomeBrew). Plus PyObjC did not have most of AVFoundation so I created a simple camera extension for Python that uses AVFoundation to record a video or snap a picture, it requires no dependencies (Cython intermediate files are committed to avoid the need to have Cython for most users).
It should be possible to build it like this:
pip install -e git+https://github.com/dashesy/pyavfcam.git
Then we can use it to take a picture:
import pyavfcam
# Open the default video source
cam = pyavfcam.AVFCam(sinks='image')
frame = cam.snap_picture('test.jpg') # frame is a memory buffer np.asarray(frame) can retrieve
Not related to this question, but if the AVFCam class is sub-classed, the overridden methods will be called with the result.