I am using Python 3.9 and Open-CV (cv2) to read frames from a video stream and save them as JPGs.
My program seems to run OK. It captures the video stream fine, obtains frames, and saves them as JPGs.
However, the frames it is obtaining from the stream are out-of-date - sometimes by several minutes. The clock in the video stream is running accurately, but the clock displays in the JPGs are all identical (to the second - but one or more minutes prior to the datetime in the program's "print()" output (and the saved JPG file time), and moving objects that were in view at the time they were saved are missing completely.
Strangely:
The JPG images are not identical in size. They grow by 10K - 20K as the sequence progresses. Even though they look identical to the eye, they show significant difference when compared using CV2 - but no difference if compared using PIL (which is about 10 - 15 times slower for image comparisons).
The camera can be configured to send a snapshot by email when it detects motion. These snapshots are up-to-date, and show moving objects that were in frame at the time (but no clock display). Enabling or disabling this facility has no effect on the out-of-date issue with JPGs extracted from the video stream. And, sadly, the snapshots are only about 60K, and too low resolution for our purposes (which is an AI application that needs images to be 600K or more).
The camera itself is ONVIF - and things like PTZ work nicely from Python code. Synology Surveillance Station works really well with it in every aspect. This model has reasonably good specs - zoom and good LPR anti-glare functionality. It is made in China - but I don't want to be 'a poor workman who blames his tools'.
Can anyone spot something in the program code that may be causing this?
Has anyone encountered this issue, and can suggest a work-around or different library / methodology?
(And if it is indeed an issue with this brand / model of camera, you are welcome to put in a plug for a mid-range LPR camera that works well for you in an application like this.)
Here is the current program code:
import datetime
from time import sleep
import cv2
goCapturedStream = None
# gcCameraLogin, gcCameraURL, & gcPhotoFolder are defined in the program, but omitted for simplicity / obfuscation.
def CaptureVideoStream():
global goCapturedStream
print(f"CaptureVideoStream({datetime.datetime.now()}): Capturing video stream...")
goCapturedStream = cv2.VideoCapture(f"rtsp://{gcCameraLogin}#{gcCameraURL}:554/stream0")
if not goCapturedStream.isOpened(): print(f"Error: Video Capture Stream was not opened.")
return
def TakePhotoFromVideoStream(pcPhotoName):
llResult = False ; laFrame = None
llResult, laFrame = goCapturedStream.read()
print(f"TakePhotoFromVideoStream({datetime.datetime.now()}): Result is {llResult}, Frame data type is {type(laFrame)}, Frame length is {len(laFrame)}")
if not ".jpg" in pcPhotoName.lower(): pcPhotoName += ".jpg"
lcFullPathName = f"{gcPhotoFolder}/{pcPhotoName}"
cv2.imwrite(lcFullPathName, laFrame)
def ReleaseVideoStream():
global goCapturedStream
goCapturedStream.release()
goCapturedStream = None
# Main Program: Obtain sequence of JPG images from captured video stream
CaptureVideoStream()
for N in range(1,7):
TakePhotoFromVideoStream(f"Test{N}.jpg")
sleep(2) # 2 seconds
ReleaseVideoStream()
Dan Masek's suggestions were very valuable.
The program (now enhanced significantly) saves up-to-date images correctly, when triggered by the camera's inbuilt motion detection (running in a separate thread and communicating through global variables).
The key tricks were:
A much faster loop reading the frames (and discarding most of them). I reduced the sleep to 0.1 (and even further to 0.01), and saved relatively few frames to JPG files only when required
Slowing down the frame rate on the camera (from 25 to 10 fps - even tried 5 at one point). This meant that the camera didn't get ahead of the software and send unpredictable frames.
Related
I'm working on a code, which reads incoming videos from Raspberry Pi, performs face detection on the frames, places frames around the faces, and then write backs the frames into an MP4 file with the same FPS. I use OpenCV to open and read from the PiCam.
When I looked into the saved video, it looks like it's moving too fast. I let my code to run for around 2 minutes, but my video has a length of 30 second. When I disable all post-processings (face detection), I can observe stable speed on the output video.
I can understand that Raspberry Pi has a small processor for heavy computations, but cannot understand why the video length is shorter? Is it possible that my face detection pipeline running much slower than the camera FPS, so the camera buffer should drop frames that are not going to be grabbed by the pipeline in a timely-fashion?
Any help here is highly appreciated!
I am working on Ubuntu 18.04 with Python 2.7 and OpenCV 3.2. My application is the front-end of a video pipeline and entails extracting video frames from a webcam, possibly cropping and/or rotating them (90, 180, 270 deg), and then distributing them to one or more other pieces of code for further processing. The overall system tries to maximize efficiency at every step to e.g., improve options for adding functionality later on (compute power and bandwidth wise).
Functionally, I have the front-end working, but I want to improve its efficiency by processing JPEG frames extracted from the camera's MJPEG stream. This would allow efficient, lossless cropping and rotation in the JPEG domain, e.g. using jpegtran-cffi, and distribution of compressed frames that are smaller than the corresponding decoded ones. JPEG decoding will take place if/when/where necessary, with an overall expected gain. As an extra benefit, this approach allows efficient saving of the webcam video without loss of image quality due to decoding + re-coding.
The problem I run into is that OpenCV's VideoCapture class does not seem to allow access to the MJPEG stream:
import cv2
cam = cv2.VideoCapture()
cam.open(0)
if not cam.isOpened():
print("Cannot open camera")
else:
enabled = True
while enabled:
enabled, frame = cam.read()
# do stuff
Here, frame is always in component (i.e., decoded) format. I looked at using cam.grab() + cam.retrieve() instead of cam.read() with the same result (in line with the OpenCV documentation). I also tried cam.set(cv2.CAP_PROP_CONVERT_RGB, False) but that only converts the decoded video to RGB (if it is in another component format) and does not prevent decoding. BTW I verified that the camera uses the MJPEG codec (via cam.get(cv2.CAP_PROP_FOURCC)).
So my questions are: am I missing something or will this approach not work? If the latter, is there an alternative?
A final point: the application has to be able to control the webcam within its capabilities; e.g., frame size, frame rate, exposure, gain, ... This is nicely supported by cv2.VideoCapture.
Thanks!
===
Follow-up: in absence of the solution I was looking for, I added explicit JPEG encoding:
jpeg_frame = cv2.imencode('.jpg', frame, [int(cv2.IMWRITE_JPEG_QUALITY), _JPEG_QUALITY])[1]
with _JPEG_QUALITY set to 90 (out of 100). While this adds computation and reduces image quality, both in principle redundant, it allows me to experiment with trade-offs. --KvZ
I'm currently using the sony QX1 for wireless transfers for large images. The camera is being triggered over the USB port. Pictures from the camera are being transferred with URLLib to a raspberry pi. (I can't use the api to trigger the camera. It has to be from this external source.)
The camera is triggered around every 2.5 seconds. Through timing testing it seems like I'm able to get the larger picture back to the pi at ~ 3.2 seconds per image.
I've noticed that when the camera is triggered my transfer is terminated. I'm assuming this has to do with the embedded design of the camera itself and there isn't a way to get around this but please correct me if I'm wrong!
Does the camera support the range header? Basically I grab the image size from the header. I'm trying to grab the beginning X bytes until the camera triggers again then grab the next X bytes until I get the entire image.
Thanks for the help and let me know if I need to give a deeper explanation of what is going on here.
I don't know about the range header, but it will still not allow you to take more pictures than your downloadspeed allows (unless you have some larger than 2.5 seconds intervals now and then).
Maybe you can reduce the image resolution to a size that fits into the 2.5 sec interval? Or (just some thinking outside of the box:-) use 2 QX1's switching, so you get a 5 second interval for each...
I wrote some image analysis in OpenCV's Python API. Images are acquired in real-time from a webcam with the read() function of a cv2.VideoCapture object within a while True loop.
Processing a frame takes about 100ms. My camera would be capable of providing 30 fps. But if I even try to set its FPS to 15 my slow processing will lead to increasing lag. The processing happens on frames that get older and older in relation to "now". I can only run in real-time if I set the FPS to 5 which is a bit low. I assume incoming frames are buffered and once my loop returns to the start, the next frame is read from that buffer instead of straight from the camera.
I read elsewhere that running the frame grabbing and the processing in seperate threads would be the solution but I never used threading. Maybe I could get the most recent frame from the buffer instead?
I am using Python 3. I would prefer a OpenCV 3 answer if that is relevant, will accept a OpenCV 2 solution happily too.
I'm recording video with a Raspberry Pi 2 and camera module in a Python script, using the picamera package. See minimal example below:
import picamera
import time
with picamera.PiCamera(resolution=(730, 1296), framerate=49) as camera:
camera.rotation=270
camera.start_preview()
time.sleep(0.5)
camera.start_recording('test.h264')
time.sleep(3)
camera.stop_recording()
camera.stop_preview()
Results
The result is a video with bad encoding:
first frame is ok
in the next 59 frames the scene is barely visible, almost all green or purple (not clear what's making change between the two colors)
frame number 61 is ok
Basically only the I-frames are correctly encoded. This was clarified experimenting different values of the intra_period parameter of the start_recording function.
Notes and attempts already made
First and foremost, I was using the same code to correctly record video in the past on the same Raspberry Pi and camera. It's not clear to me if the problem appeared when reinstalling the complete image, during updates, installation of other packages...
Also:
if I don't set the resolution parameter and rotation, the camera works fine
several video players have been tested on the same and on other machines, and processing frame by frame with OpenCV, it is really a problem in the video file
mjpeg format works fine
the same problem happens setting sensor_mode=5
Questions
The main question is how to correctly record video at a set resolution, by correction of the code above or a workaround.
Secondary question: I'm curious to know what could cause such behaviour.