I wrote some image analysis in OpenCV's Python API. Images are acquired in real-time from a webcam with the read() function of a cv2.VideoCapture object within a while True loop.
Processing a frame takes about 100ms. My camera would be capable of providing 30 fps. But if I even try to set its FPS to 15 my slow processing will lead to increasing lag. The processing happens on frames that get older and older in relation to "now". I can only run in real-time if I set the FPS to 5 which is a bit low. I assume incoming frames are buffered and once my loop returns to the start, the next frame is read from that buffer instead of straight from the camera.
I read elsewhere that running the frame grabbing and the processing in seperate threads would be the solution but I never used threading. Maybe I could get the most recent frame from the buffer instead?
I am using Python 3. I would prefer a OpenCV 3 answer if that is relevant, will accept a OpenCV 2 solution happily too.
Related
I am using Python 3.9 and Open-CV (cv2) to read frames from a video stream and save them as JPGs.
My program seems to run OK. It captures the video stream fine, obtains frames, and saves them as JPGs.
However, the frames it is obtaining from the stream are out-of-date - sometimes by several minutes. The clock in the video stream is running accurately, but the clock displays in the JPGs are all identical (to the second - but one or more minutes prior to the datetime in the program's "print()" output (and the saved JPG file time), and moving objects that were in view at the time they were saved are missing completely.
Strangely:
The JPG images are not identical in size. They grow by 10K - 20K as the sequence progresses. Even though they look identical to the eye, they show significant difference when compared using CV2 - but no difference if compared using PIL (which is about 10 - 15 times slower for image comparisons).
The camera can be configured to send a snapshot by email when it detects motion. These snapshots are up-to-date, and show moving objects that were in frame at the time (but no clock display). Enabling or disabling this facility has no effect on the out-of-date issue with JPGs extracted from the video stream. And, sadly, the snapshots are only about 60K, and too low resolution for our purposes (which is an AI application that needs images to be 600K or more).
The camera itself is ONVIF - and things like PTZ work nicely from Python code. Synology Surveillance Station works really well with it in every aspect. This model has reasonably good specs - zoom and good LPR anti-glare functionality. It is made in China - but I don't want to be 'a poor workman who blames his tools'.
Can anyone spot something in the program code that may be causing this?
Has anyone encountered this issue, and can suggest a work-around or different library / methodology?
(And if it is indeed an issue with this brand / model of camera, you are welcome to put in a plug for a mid-range LPR camera that works well for you in an application like this.)
Here is the current program code:
import datetime
from time import sleep
import cv2
goCapturedStream = None
# gcCameraLogin, gcCameraURL, & gcPhotoFolder are defined in the program, but omitted for simplicity / obfuscation.
def CaptureVideoStream():
global goCapturedStream
print(f"CaptureVideoStream({datetime.datetime.now()}): Capturing video stream...")
goCapturedStream = cv2.VideoCapture(f"rtsp://{gcCameraLogin}#{gcCameraURL}:554/stream0")
if not goCapturedStream.isOpened(): print(f"Error: Video Capture Stream was not opened.")
return
def TakePhotoFromVideoStream(pcPhotoName):
llResult = False ; laFrame = None
llResult, laFrame = goCapturedStream.read()
print(f"TakePhotoFromVideoStream({datetime.datetime.now()}): Result is {llResult}, Frame data type is {type(laFrame)}, Frame length is {len(laFrame)}")
if not ".jpg" in pcPhotoName.lower(): pcPhotoName += ".jpg"
lcFullPathName = f"{gcPhotoFolder}/{pcPhotoName}"
cv2.imwrite(lcFullPathName, laFrame)
def ReleaseVideoStream():
global goCapturedStream
goCapturedStream.release()
goCapturedStream = None
# Main Program: Obtain sequence of JPG images from captured video stream
CaptureVideoStream()
for N in range(1,7):
TakePhotoFromVideoStream(f"Test{N}.jpg")
sleep(2) # 2 seconds
ReleaseVideoStream()
Dan Masek's suggestions were very valuable.
The program (now enhanced significantly) saves up-to-date images correctly, when triggered by the camera's inbuilt motion detection (running in a separate thread and communicating through global variables).
The key tricks were:
A much faster loop reading the frames (and discarding most of them). I reduced the sleep to 0.1 (and even further to 0.01), and saved relatively few frames to JPG files only when required
Slowing down the frame rate on the camera (from 25 to 10 fps - even tried 5 at one point). This meant that the camera didn't get ahead of the software and send unpredictable frames.
I'm working on a code, which reads incoming videos from Raspberry Pi, performs face detection on the frames, places frames around the faces, and then write backs the frames into an MP4 file with the same FPS. I use OpenCV to open and read from the PiCam.
When I looked into the saved video, it looks like it's moving too fast. I let my code to run for around 2 minutes, but my video has a length of 30 second. When I disable all post-processings (face detection), I can observe stable speed on the output video.
I can understand that Raspberry Pi has a small processor for heavy computations, but cannot understand why the video length is shorter? Is it possible that my face detection pipeline running much slower than the camera FPS, so the camera buffer should drop frames that are not going to be grabbed by the pipeline in a timely-fashion?
Any help here is highly appreciated!
I am trying to change the Frame rate i.e., FPS of an existing video using openCV library in python. Below is the code that I am trying to execute. Even after setting the FPS property using cv2.CAP_PROP_FPS the video is not playing faster in the cv2.imshow() method. Even After setting the FPS property the getter returns the older FPS value. So how do I set the FPS value higher and make the video play faster?
Used version:
python = 3.7.4 and
opencv-python - 4.1.0.25
import cv2
video = cv2.VideoCapture("yourVideoPath.mp4");
video.set(cv2.CAP_PROP_FPS, int(60))
if __name__ == '__main__':
print("Frame rate : {0}".format(video.get(cv2.CAP_PROP_FPS)))
while video.isOpened():
ret1, frame2 = video.read()
cv2.imshow("Changed", frame2)
if cv2.waitKey(10) & 0xFF == ord('q'): # press q to quit
break
video.release()
cv2.destroyAllWindows()
If you're only trying to play the video in the displayed window, the limiting factor is not the fps of the video but the time spent waiting with the code waitKey(10) which makes the program wait for 10ms between each frame.
The read() method of the VideoCapture class simply returns the next frame with no concept of waiting or frame rate. The only thing preventing this code running as fast as it can is the waitKey(10) section, which is thus the main factor determining speed. To change the frame rate as seen through the imshow() method, you'd need to edit the time spent waiting. This is likely the dominant factor, but not the only one as the reading of a frame does take time.
If you're actually trying to change the playback rate of an existing file and have that saved to that file, I am unsure if OpenCV actually supports this, and I imagine it would be dependent on what back end you're using - OpenCV implements the VideoCapture class using different 3rd party backends.. As per the documentation of VideoCapture.set() I'd investigate the return value of video.set(cv2.CAP_PROP_FPS, int(60)) as the documentation suggests it will return true if this has changed something.
As an alternative you could investigate using something like FFMPEG which supports this relatively easily. If you want to stick with OpenCV I know from personal experience you can do this with the VideoWriter class. In this method you would read in the video frame by frame using the VideoCapture class, and then save it at the desired frame rate with VideoWriter. I suspect FFMPEG will likely meet your needs however!
I am using flask-python framework , opencv and face_recognition library to do face recognition. I'm trying to run the webcam using python -opencv and if a face is detected in the frame, it is sent to the backend where the frame is processed and recognition of the person is performed but this api call with response is taking around 0.3 seconds , this delay is causing a lag in the smooth flow of webcam.
what I would like to have is the webcam run continuously and when the face is detected in the frame it should not block the free flow of webcam streaming, instead the processing of this frame should be handled separately in the background while the main thread is running webcam.
Any help in this regard is much appreciated
Theres a number of things that you can do to resolve this issue,
Dont pass every frame into detection, chances are that you are using a good webcam, which gives you 30FPS or higher, you dont need such a high FPS for real time detection, what you could do is that, only send every 3rd or 4th frame for detection, and for the other frames, just draw the old boxes to the stream, this will give you accurate enough predictions
Shift your model to GPU, face_recognition gives you the choice between using a CPU and GPU(cnn only), shifting it onto any available GPU's will speed up the process
Resize your input frame, you really don't need a 1080p HD frame to detect frames, anything smaller like 480p will also do (though this is entirely dependent on your problem), resize your frame and make it smaller, the smaller the input matrix the quicker it is
Vectorize your operations, instead of using the default face_recognition API for identity comparisons, i suggest you code a vectorized version of comparisons in numpy that calculates the L2 distance in large matrices, instead of having to do it in a loop, this will save alot of computation cost.
Limit the number of comparisons per person. You could have 10 embeddings of person A and 10 of person B, finding the distance of the unknown embeddings with each of these 20 embeddings, can be computationally expensive, resort to doing something like, taking average embeddings, or the centroids of these embeddings, by clustering them
I am trying to rapidly select and process different frames from a video using OpenCV Python. To select a frame, I have used the 'CAP_PROP_POS_FRAMES' (or cap.set(2, frame_no)). However when using this I noticed a delay of about 200 ms to decode the selected frame. My script will be jumping in between frames a lot (not necessarily chronological) which means this will cause a big delay between each iteration.
I suspected OpenCV is buffering the upcoming frames after I set the frame number. Therefore I tried pre-decoding of the video by basically putting the entire video as a list so it can be accessed from RAM. This worked fantastic except bigger videos completely eat up my memory.
I was hoping someone knows a way to either set the frame number without this 200ms delay or to decode the video without using all of my memory space. Any suggestions are also welcome!
I don't know how to avoid that 200ms delay, but I have a suggestion on how you could decode the video first even if its size is greater than your RAM. You could use numpy's memmap:
https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.memmap.html.
In practice you could have a function that initializes this memory-mapped matrix and then iterate over each frame of the video using VideoCapture and then store each frame in this matrix. After that you will be able to jump between each frame just by accessing this memory-mapped matrix.