I am trying to create automatic attendance system with opencv2 in which i need to get rtsp stream from IP camera, find faces from it and recognize face.
I created different threads from frame catching and drawing because face recognition function needs some time to recognize face.
But just creating 2 threads, one for frame reading and other for drawing uses around 70% CPU.
and creating pytorch_facenet model increase usage 80-90% CPU.
does anyone know how to reduce CPU usage ?
my program:
import cv2
import threading
from facenet_pytorch import InceptionResnetV1
cap = cv2.VideoCapture("rtsp://test:Test12345#125.0.0.1")
resnet = InceptionResnetV1(pretrained='vggface2').eval()
ret, frame = cap.read()
exit = False
def th1():
global ret, frame, exit
while True:
ret, frame = cap.read()
if exit:
break
def th2():
global ret, frame, exit
while True:
cv2.imshow('frame', frame)
cv2.waitKey(1)
if cv2.getWindowProperty('frame',cv2.WND_PROP_VISIBLE) < 1:
exit = True
break
t1 = threading.Thread(target=th1)
t1.start()
t2 = threading.Thread(target=th2)
t2.start()
Update:
I used time.sleep(0.2) in my all threads except frame reading.
and it worked, my cpu usage is 30% now.
Two issues.
th2 runs in an almost-tight-loop. It won't consume a whole core of CPU because waitKey(1) sleeps for some time.
No synchronization at all between threads, but you need it. You need a threading.Event to notify the consumer thread of a fresh frame. The consumer thread must wait until a fresh frame is available, because it's pointless to display the same old frame again and again. You can be lazy and use waitKey(30) instead. For the displaying thread, that's good enough.
VideoCapture. You don't do any error checking at all! You must check:
cap = cv2.VideoCapture("rtsp://test:Test12345#125.0.0.1")
assert cap.isOpened()
...
and
while True:
ret, frame = cap.read()
if not ret:
break
...
This code works.
This first loop (thread) will be trying to read frames as fast as it can.
The frame can be updated 100 times per second or more, but it's too fast. Try to add time.sleep(0.03).
And in the second loop, you can change the waitKey() param to 30.
import time
def th1():
global ret, frame, exit
while True:
ret, frame = cap.read()
time.sleep(0.03)
if exit:
break
def th2():
global ret, frame, exit
while True:
cv2.imshow('frame', frame)
cv2.waitKey(30)
if cv2.getWindowProperty('frame',cv2.WND_PROP_VISIBLE) < 1:
exit = True
break
Related
I need a main while() loop, which updates the screenshot frame all the time, but when I get to a part of the code because the sync needs to be very precise, what I need is to create 2 threads or subprocess (I think using subprocess is better in this case).
One that keeps updating the frames and the other thread or subprocess that makes a delay of 3 seconds, only then to start working with the last frame that was updated (because of this delay it is so important to wait for the frames to be updated).
This is my code:
import multiprocessing
import time
import cv2
import numpy as np
#library for Optical Character Recognition (OCR)
import pytesseract #pip install pytesseract
pytesseract.pytesseract.tesseract_cmd = r'C:\Program Files\Tesseract-OCR\tesseract'
# Create a VideoCapture object
cap = cv2.VideoCapture(1)
# Check if camera opened successfully
if (cap.isOpened() == False):
print("Unable to read camera feed")
.
# We convert the resolutions from float to integer.
frame_width = int(cap.get(3))
frame_height = int(cap.get(4))
while(True):
ret, frame = cap.read()
if ret == True:
#HERE SHOULD BE THE FORK IN 2 INDEPENDENT PROCESSES
text = pytesseract.image_to_string(frame) #OCR in subprocess_2
cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
else:
break #Close main loop
This is the flowchart of how the program should work. Showing how the subprocess_1 repeats updating the value of the variable frame until subprocess_2 finishes executing (in this particular case, in principle I plan to try 3 seconds delay).
I thought of using a separate function but I'm really having trouble implementing it. I would also like to know if it is possible to implement all frame updates in a single loop while() .
def handle_frame_requests(conn1):
try:
while True:
request = conn1.recv()
conn1.send(frame) # The frame must be pickle-able
except EOFError:
pass
def capture_cam(conn1):
global frame
frame = None
Thread(target=handle_frame_requests, args=(conn1,), daemon=True).start()
cap = cv2.VideoCapture(1) #the same webcam
if (cap.isOpened() == False):
print("Unable to read camera!")
frame_width = int(cap.get(3))
frame_height = int(cap.get(4))
while(True):
ret, frame = cap.read() #here load the frame variable
if ret == True:
cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xFF == ord('q'): break
else:
break
But regardless of how many nested while loops I use, the problem is that I can't get one process to keep updating the webcam, while another process keeps a timer, so that when the timer indicates it, both processes will exit and return to the webcam. main line, where the main while loop can continue to receive data from the webcam via the first loop while(True):
I'm writing a program to control the video playback speed as custom rate.
Is there is anyway to achieve that?
What code should be added to control the playback speed?
import cv2
cap = cv2.VideoCapture('video.mp4')
while(cap.isOpened()):
ret, frame = cap.read()
cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
In the docs it is stated:
Note
This function should be followed by waitKey function which displays
the image for specified milliseconds. Otherwise, it won’t display the
image. For example, waitKey(0) will display the window infinitely
until any keypress (it is suitable for image display). waitKey(25)
will display a frame for 25 ms, after which display will be
automatically closed. (If you put it in a loop to read videos, it will
display the video frame-by-frame)
In cv2.waitKey(X) function X means the number of milliseconds for an image to be displayed on the screen. In your case it is set to 1, so theoretically you are able to achieve 1000 fps (frames per seconds). But frame decoding takes time in VideoCapture object and limits your framerate. To change the playback speed you need to declare variable and use it as a parameter in waitKey function.
import cv2
cap = cv2.VideoCapture('video.mp4')
frameTime = 10 # time of each frame in ms, you can add logic to change this value.
while(cap.isOpened()):
ret, frame = cap.read()
cv2.imshow('frame',frame)
if cv2.waitKey(frameTime) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
Alternatively, as frame decoding is the most time consuming task you can move it to the second thread and use a queue of decoded frames. See this link for details.
The third approach is to separate grabbing and decoding process and simply decode every nth frame. That will result in displaying only a subset of frames from the source video but from the user perspective the video will be played faster.
import cv2
cap = cv2.VideoCapture('video.mp4')
i=0 #frame counter
frameTime = 1 # time of each frame in ms, you can add logic to change this value.
while(cap.isOpened()):
ret = cap.grab() #grab frame
i=i+1 #increment counter
if i % 3 == 0: # display only one third of the frames, you can change this parameter according to your needs
ret, frame = cap.retrieve() #decode frame
cv2.imshow('frame',frame)
if cv2.waitKey(frameTime) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
You can use ffmpeg to speed up (or slow down) your video like fast forwarding by using "presentation time stamps".
For speed up an example would be:
ffmpeg -i YOUR_INPUT_MOVIE.mp4 -vf "setpts=0.20*PTS" YOUR_OUTPUT_MOVIE.mp4
which will speed up your movie by 5x.
For slow down an example would be:
ffmpeg -i YOUR_INPUT_MOVIE.mp4 -vf "setpts=5*PTS" YOUR_OUTPUT_MOVIE.mp4
which will slow down your movie by 5x.
Note: this method will drop frames.
I am using python 3.5 with opencv.
I want to use 2 threads:
Thread 1: Save the video to a file
Thread 2: Display the video to the user
To view/capture the video from webcam i am using snippets of code from the following website: opencv video docs
I can capture and save the video using the following code:
# Define the codec and create VideoWriter object
fourcc = cv2.VideoWriter_fourcc(*'XVID')
out = cv2.VideoWriter('output.avi',fourcc, 20.0, (640,480))
while(True):
ret, frame = cap.read()
if ret==True:
frame = cv2.flip(frame,0)
# write the flipped frame
out.write(frame)
else:
break
out.release()
cv2.destroyAllWindows()
I can view the video using the following code:
while(True):
# Capture frame-by-frame
ret, frame = cap.read()
cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cv2.destroyAllWindows()
Each of these pieces of code are in their own functions called capture and display. I then call them in separate threads with pythons threading library as follows:
cap = cv2.VideoCapture(0)
Thread(target=capture).start()
Thread(target=display).start()
cap.release()
I get an error I assume is related to both threads wanting to access the video buffer at the same time.
I understand this can be done without threads but there are other things I would like to do further than can only be done in separate threads.
How can I access the cap video capture from both threads?
My flask/django experience is increadibly limited, so I am not sure how to do it for that exactly, but I will answer the question posted directly.
First you need to create a thread-safe object to avoid calling at the same time the read function in different threads.
import cv2
import threading
class VideoCamera(object):
# filename can be 0 to access the webcam
def __init__(self, filename):
self.lock = threading.Lock()
self.openVideo(filename)
def openVideo(self, filename):
self.lock.acquire()
self.videoCap = cv2.VideoCapture(filename)
self.lock.release()
With this, you should be able to create an object with a lock and to open safely a video (in case that you want to open another video with the same object).
Now you have 2 options, either you create a thread that updates the frame and stores the current one internally or update do in a thread safe manner the get next frame function. I will do the second one here to show you:
def getNextFrame(self):
self.lock.acquire()
img = None
# if no video opened return None
if self.videoCap.isOpened():
ret, img = self.videoCap.read()
self.lock.release()
return img
This way you should be able to access the video cap with 2 frames... however, the frames will be different every time the function is called.
I hope this helps you.
I want to connect to a camera, and only capture a frame when an event happens (e.g. keypress). A simplified version of what I'd like to do is this:
cap = cv2.VideoCapture(device_id)
while True:
if event:
img = cap.read()
preprocess(img)
process(img)
cv.Waitkey(10)
However, cap.read seems to only capture the next frame in the queue, and not the latest. I did a lot of searching online, and there seems to be a lot of questions on this but no definitive answer. Only some dirty hacks which involve opening and closing the capture device just before and after grabbing (which won't work for me as my event might be triggered multiple times per second); or assuming a fixed framerate and reading a fixed-n times on each event (which won't work for me as my event is unpredictable and could happen at any interval).
A nice solution would be:
while True:
if event:
while capture_has_frames:
img = cap.read()
preprocess(img)
process(img)
cv.Waitkey(10)
But what is capture_has_frames? Is it possible to get that info? I tried looking into CV_CAP_PROP_POS_FRAMES but it's always -1.
For now I have a separate thread where the capture is running at full fps, and on my event I'm grabbing the latest image from that thread, but this seems overkill.
(I'm on Ubuntu 16.04 btw, but I guess it shouldn't matter. I'm also using pyqtgraph for display)
I think the solution mentioned in the question, namely having a separate thread that clears the buffer, is the easiest non-brittle solution for this. Here reasonably nice (I think) code for this:
import cv2, queue, threading, time
# bufferless VideoCapture
class VideoCapture:
def __init__(self, name):
self.cap = cv2.VideoCapture(name)
self.q = queue.Queue()
t = threading.Thread(target=self._reader)
t.daemon = True
t.start()
# read frames as soon as they are available, keeping only most recent one
def _reader(self):
while True:
ret, frame = self.cap.read()
if not ret:
break
if not self.q.empty():
try:
self.q.get_nowait() # discard previous (unprocessed) frame
except queue.Empty:
pass
self.q.put(frame)
def read(self):
return self.q.get()
cap = VideoCapture(0)
while True:
time.sleep(.5) # simulate time between events
frame = cap.read()
cv2.imshow("frame", frame)
if chr(cv2.waitKey(1)&255) == 'q':
break
The frame reader thread is encapsulated inside the custom VideoCapture class, and communication with the main thread is via a queue.
I posted very similar code for a node.js question, where a JavaScript solution would have been better. My comments on another answer to that question give details why a non-brittle solution without separate thread seems difficult.
An alternative solution that is easier but supported only for some OpenCV backends is using CAP_PROP_BUFFERSIZE. The 2.4 docs state it is "only supported by DC1394 [Firewire] v 2.x backend currently." For Linux backend V4L, according to a comment in the 3.4.5 code, support was added on 9 Mar 2018, but I got VIDEOIO ERROR: V4L: Property <unknown property string>(38) not supported by device for exactly this backend. It may be worth a try first; the code is as easy as this:
cap.set(cv2.CAP_PROP_BUFFERSIZE, 0)
Here's a simplified version of Ulrich's solution.
OpenCV's read() function combines grab() and retrieve() in one call, where grab() just loads the next frame in memory, and retrieve decodes the latest grabbed frame (demosaicing & motion jpeg decompression).
We're only interested in decoding the frame we're actually reading, so this solution saves some CPU, and removes the need for a queue
import cv2
import threading
# bufferless VideoCapture
class VideoCapture:
def __init__(self, name):
self.cap = cv2.VideoCapture(name)
self.lock = threading.Lock()
self.t = threading.Thread(target=self._reader)
self.t.daemon = True
self.t.start()
# grab frames as soon as they are available
def _reader(self):
while True:
with self.lock:
ret = self.cap.grab()
if not ret:
break
# retrieve latest frame
def read(self):
with self.lock:
_, frame = self.cap.retrieve()
return frame
EDIT: Following Arthur Tacca's comment, added a lock to avoid simultaneous grab & retrieve, which could lead to a crash as OpenCV isn't thread-safe.
Its also possible to always get the latest frame by using cv2.CAP_GSTREAMER backend. If you have gstreamer support enabled in cv2.getBuildInformation(), you can initialize your video capture with the appsink parameters sync=false and drop=true
Example:
cv2.VideoCapture("rtspsrc location=rtsp://... ! decodebin ! videoconvert ! video/x-raw,framerate=30/1 ! appsink drop=true sync=false", cv2.CAP_GSTREAMER)
On my Raspberry Pi 4,
cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)
does work and was all that I needed for my pi camera to give me the latest frame, with a consistent 3+ second delay between the scene in front of the camera and displaying that scene in the preview image. My code takes 1.3 seconds to process an image, so I'm not sure why the other 2 seconds of delay are present, but it's consistent and works.
Side note: since my code takes over a second to process an image, I also added
cap.set( cv2.CAP_PROP_FPS, 2 )
in case it reduces any unneeded activity, since I can't quite get a frame a second. When I put cv2.CAP_PROP_FPS to 1, though, I got a strange output of all my frames being almost entirely dark, so setting FPS too low can cause an issue
If you don't want to capture the frame when there is no event happening, why are you preprocessing/processing your frame? If you do not process your frame, you can simply discard it unless the event occur. Your program should be able to capture, evaluate your condition and discard at a sufficient speed, i.e. fast enough compared to your camera FPS capture rate, to always get the last frame in the queue.
If not proficient in python because I do my OpenCV in C++, but it should look similar to this:
vidcap = cv.VideoCapture( filename )
while True:
success, frame = vidcap.read()
If Not success:
break
If cv.waitKey(1):
process(frame)
As per OpenCV reference, vidcap.read() returns a bool. If frame is read correctly, it will be True. Then, the captured frame is store in variable frame. If there is no key press, the loop keeps on going. When a key is pressed, you process your last captured frame.
I am using cv2.VideoCapture to read the frames of an RTSP video link in a python script. The .read() function is in a while loop which runs once every second, However, I do not get the most current frame from the stream. I get older frames and in this way my lag builds up. Is there anyway that I can get the most current frame and not older frames which have piped into the VideoCapture object?
I also faced the same problem. Seems that once the VideoCapture object is initialized it keeps storing the frames in some buffer of sort and returns a frame from that for every read operation. What I did is I initialized the VideoCapture object every time I wanted to read a frame and then released the stream. Following code captures 10 images at an interval of 10 seconds and stores them. Same can be done using while(True) in a loop.
for x in range(0,10):
cap = cv2.VideoCapture(0)
ret, frame = cap.read()
cv2.imwrite('test'+str(x)+'.png',frame)
cap.release()
time.sleep(10)
I've encountered the same problem and found a git repository of Azure samples for their computer vision service.
The relevant part is the Camera Capture module, specifically the Video Stream class.
You can see they've implemented a Queue that is being updated to keep only the latest frame:
def update(self):
try:
while True:
if self.stopped:
return
if not self.Q.full():
(grabbed, frame) = self.stream.read()
# if the `grabbed` boolean is `False`, then we have
# reached the end of the video file
if not grabbed:
self.stop()
return
self.Q.put(frame)
# Clean the queue to keep only the latest frame
while self.Q.qsize() > 1:
self.Q.get()
I'm working with a friend in a hack doing the same. We don't want to use all the frames. So far we found that very same thing: grab() (or read) tries to get you all the frames, and I guess with rtp: it will maintain a buffer and drop if you're not responsive enough.
Instead of read you can also use grab() and receive(). First one ask for the frame. Receives reads it into memory. So if you call grab several times it will effectively skip those.
We got away with doing this:
#show some initial image
while True:
cv2.grab()
if cv2.waitKey(10):
im = cv2.receive()
# process
cv2.imshow...
Not production code but...
Inside the 'while' you can use:
while True:
cap = cv2.VideoCapture()
urlDir = 'rtsp://ip:port/h264_ulaw.sdp'
cap.open(urlDir)
# get the current frame
_,frame = cap.read()
cap.release() #releasing camera
image = frame
Using the following was causing a lot of issues for me. The frames being passed to the function were not sequention.
cap = cv2.VideoCapture(0)
while True:
ret, frame = cap.read()
function_that_uses_frame(frame)
time.sleep(0.5)
The following also didn't work for me as suggested by other comments. I was STILL getting issues with taking the most recent frame.
cap = cv2.VideoCapture(0)
while True:
ret = capture.grab()
ret, frame = videocapture.retrieve()
function_that_uses_frame(frame)
time.sleep(0.5)
Finally, this worked but it's bloody filthy. I only need to grab a few frames per second, so it will do for the time being. For context, I was using the camera to generate some data for an ML model and my labels compared to what was being captured was out of sync.
while True:
ret = capture.grab()
ret, frame = videocapture.retrieve()
ret = capture.grab()
ret, frame = videocapture.retrieve()
function_that_uses_frame(frame)
time.sleep(0.5)
I made an adaptive system as the ones the others on here posted here still resulted in somewhat inaccurate frame representation and have completely variable results depending on the hardware.
from time import time
#...
cap = cv2.VideoCapture(url)
cap_fps = cap.get(cv2.CAP_PROP_FPS)
time_start = time()
time_end = time_start
while True:
time_difference = int((((end_time-start_time))*cap_fps)+1) #Note that the 1 might be changed to fit script bandwidth
for i in range(0, time_difference):
a = cap.grab()
_, frame = cap.read()
time_start = time()
#Put your code here
variable = function(frame)
#...
time_end = time()
This way the skipped frames adapt to the amount of frames missed in the video stream - allowing for a much smoother transition and a relatively real-time frame representation.