Develop Reference

tqdm progressbar for opencv videocapture going beyong 100%

I am processing a video using Opencv in python and using tqdm to show the progressbar. However, the progress goes beyond 100%. Not really sure why is this happening.
I am new to opencv ,so it is possible I am passing wrong params to do what I is intended.
I have tried out a few ways. Listing them down.
cam = cv2.VideoCapture("path")
fps = cam.get(cv2.CAP_PROP_FPS)
total_frame_count = int(cam.get(cv2.CAP_PROP_FRAME_COUNT))
length = total_frame_count/fps
pbar = tqdm(total = total_frame_count)
count = 0
while(True):
ret,frame = cam.read()
pbar.update(count)
# process(frame)
count += fps*5
cam.set(cv2.CAP_PROP_POS_FRAMES, count)
I have a counter called count which is basically to skip the video 5 secs.

When using tqdm with manual updates you specify the step with which you increment not where you are currently.
From the manual
with tqdm(total=100) as pbar:
for i in range(10):
sleep(0.1)
pbar.update(10)
So what you should do is simply to have pbar.update(fps*5).

Parallelize slow functions that needs to be run only every X iterations in order to not slow the loop

The project
I am conducting a project where I need to both detect faces (bounding boxes and landmarks) and perform face recognition (identify a face). The detection is really fast (it takes not even a few milliseconds on my laptop) but the recognition can be really slow (about 0.4 seconds on my laptop). I am using the face_recognition Python library to do so. After a few tests, I discovered that it is the embedding of the image that is slow.
Here is an example code to try it out for yourself :
# Source : https://pypi.org/project/face-recognition/
import face_recognition
known_image = face_recognition.load_image_file("biden.jpg")
biden_encoding = face_recognition.face_encodings(known_image)[0]
image = face_recognition.load_image_file("your_file.jpg")
face_locations = face_recognition.face_locations(image)
face_landmarks_list = face_recognition.face_landmarks(image)
unknown_encoding = face_recognition.face_encodings(image)[0]
results = face_recognition.compare_faces([biden_encoding], unknown_encoding)
The problem
What I need to do is to process a video (30 FPS), therefore 0.4s of computation is unacceptable. The idea that I have is that the recognition will only need to be run a few times and not every frame since from one frame to another, if there are no cuts in the video, a given head will be close to its previous position. Therefore, the first time the head appears, we run the recognition which is very slow but then for the next X frames, we won't have to since we'll detect that the position is close to the previous one, therefore it must be the same person that moved. Of course, this approach is not perfect but seems to be a good compromise and I would like to try it.
The only problem is that by doing so the video is smooth until a head appears, then the video freezes because of the recognition and then becomes smooth again. This is where I would like to introduce multiprocessing, I would like to be able to compute the recognition in parallel of looping through the frame of the video. If I manage to do so, I will then only have to process a few frames in advance so that when a face shows up it already computed its recognition a few seconds ago during several frames so that we did not see a reduced frame rate.
Simple formulation
Therefore here is what I have (in python pseudo code so that it is clearer):
def slow_function(image):
# This function takes a lot of time to compute and would normally slow down the loop
return Recognize(image)
# Loop that we need to maintain at a given speed
person_name = "unknown"
frame_index = -1
while True:
frame_index += 1
frame = new_frame() # this is not important and therefore not detailes
# Every ten frames, we run a heavy function
if frame_index % 10 == 0:
person_name = slow_function(image)
# each frame we use the person_name even if we only compute it every so often
frame.drawText(person_name)
And I would like to do something like this :
def slow_function(image):
# This function takes a lot of time to compute and would normally slow down the loop
return Recognize(image)
# Loop that we need to maintain at a given speed
person_name = "unknown"
frame_index = -1
while True:
frame_index += 1
frame = new_frame() # this is not important and therefore not detailes
# Every ten frames, we run a heavy function
if frame_index % 10 == 0:
DO slow_function(image) IN parallel WITH CALLBACK(person_name = result)
# each frame we use the person_name even if we only compute it every so often
frame.drawText(person_name)
The goal is to compute a slow function over several iterations of a loop.
What I have tried
I looked up multiprocessing and Ray but I did not find examples of what I wanted to do. Most of the time I found people using multiprocessing to compute at the same time the result of a function for different inputs. This is not what I want. I want to have in parallel a loop and a process that accepts data from the loop (a frame), do some computation, and returns a value to the loop without interrupting or slowing down the loop (or at least, spreading the slow down rather than having one really slow iteration and 9 fast ones).

I think I found pretty much how to do what I want. Here is an example:
from multiprocessing import Pool
import time
# This seems to me more precise than time.sleep()
def sleep(duration, get_now=time.perf_counter):
now = get_now()
end = now + duration
while now < end:
now = get_now()
def myfunc(x):
time.sleep(1)
return x
def mycallback(x):
print('Callback for i = {}'.format(x))
if __name__ == '__main__':
pool=Pool()
# Approx of 5s in total
# Without parallelization, this should take 15s
t0 = time.time()
titer = time.time()
for i in range(100):
if i% 10 == 0: pool.apply_async(myfunc, (i,), callback=mycallback)
sleep(0.05) # 50ms
print("- i =", i, "/ Time iteration:", 1000*(time.time()-titer), "ms")
titer = time.time()
print("\n\nTotal time:", (time.time()-t0), "s")
t0 = time.time()
for i in range(100):
sleep(0.05)
print("\n\nBenchmark sleep time time:", 10*(time.time()-t0), "ms")
Of course, I will need to add flags so that I do not write a value with the callback at the same time that I read it in the loop.

Video record from an IP camera with variable frame rate with opencv and python

First of all, I want to comment on what I'm trying to do.
I have an IP camera connected to my network(FOSCAM 9800p) through a router with ethernet cable and from it I am trying to record a video with the RTSP protocol. My intention in the future is to add a small video processing in the middle with opencv but at the moment I want to do the tests to simply record it.
The main problem is that the camera is delivering a variable rate of frames per second, that is, sometimes it does it to 18, others to 22 and so on. When recording the video with a fixed rate of frames per second what ends up happening is that the video plays faster than it should
Something weird that when I run with opencv get (CAP_PROP_FPS) it is returning a big value like 180000.0
To try to solve this problem, what we do is read the frames and place them in a queue. From another process commanded by a timer.Event () we read them and try to write in our video at regular intervals of time in order to obtain a fixed frame rate.
The code is the following:
video_capture = cv2.VideoCapture("rtsp://"+user+":"+password+"#"+ip+":"+str(port)+"/videoMain")
if (video_capture.isOpened() == False):
print("Unable to read camera feed")
sys.exit()
frame_width = int(video_capture.get(3))
frame_height = int(video_capture.get(4))
video_writer =cv2.VideoWriter(output_filename,cv2.VideoWriter_fourcc(*'MP4V'), fps_to_save, (frame_width,frame_height))
input_buffer = queue.Queue(20)
finished = False
read_frames = 0
def readFile():
global finished
global read_frames
while not finished:
ret, frame = video_capture.read()
if not ret:
finished = True
while not finished:
try:
input_buffer.put_nowait(frame)
read_frames+=1
break
except queue.Full:
print("queue.Full")
pass
def processingFile():
global finished
written_frames = 0
repeated_frames = 0
time_per_frame_elapsed = 0.0
start_time=time.time()
ticker = threading.Event()
while True:
ticker.wait(time_per_frame-time_per_frame_elapsed)
time_per_frame_start=time.time()
try:
frame = input_buffer.get_nowait()
video_writer.write(frame)
writing_time = time.time()
if written_frames is 0:
start_time = writing_time
written_frames += 1
except queue.Empty:
if written_frames is not 0:
video_writer.write(frame)
writing_time = time.time()
written_frames += 1
repeated_frames += 1
except:
pass
total_elapsed_time = time.time() - start_time
print("total_elapsed_time:{:f}".format(total_elapsed_time))
if total_elapsed_time>time_to_save_seconds:
finished = True
ticker.clear()
print ("Playback terminated.")
break
time_per_frame_elapsed=time.time()-time_per_frame_start
print("Total readed frames:{:f}".format(read_frames))
print("Total frames repated:{:f}".format(repeated_frames))
print("Total frames writed:{:f}".format(written_frames))
tReadFile = threading.Thread(target=readFile)
tProcessingFile = threading.Thread(target=processingFile)
tReadFile.start()
tProcessingFile.start()
tProcessingFile.join()
tReadFile.join()
The result is close to what we want, but sometimes we have significant differences in the times. We are doing tests with short videos of about 10 seconds and we get 9.8 seconds of recording.
At first it would not seem a serious problem but the error is cumulative, that is if we increase the time increases so that to record videos of a longer time we have more serious problems.
We would like to know how to solve this type of video recording problems with cameras that deliver frames at variable rates. Is it a good idea to do so?
What can be generating the cumulative error in the times?
From already thank you very much!
Greetings to all!

I can say only one thing. In my own experience OpenCV VideoCapture class works with FFMPEG (OpenCV uses it to decode video) in on-line mode very badly. There was a lot of image artifacts and ffmpeg internal errors. But VideoCapture perfectly works with USB cameras. I solved problem with on-line capture from IP-camera with XSplit Broadcaster. This package is able to emulate a USB-camera via physical IP-camera. The only limitation is resizing of camera frame to 640*480 size. Basic license of XSplit Broadcaster is absolutely free

Parse video at lower frame rate

I currently am working on a project where I need to parse a video and pass it through several models. The videos come in at 60fps. I do not need to run every frame through the models. I am running into some issue when trying to skip the unneeded frames. I have tried two methods which are both fairly slow.
Method 1 code snippet:
The issue here is that I am still reading every frame of the video. Only every 4th frames is run through my model.
cap = cv2.VideoCapture(self.video)
while cap.isOpened():
success, frame = cap.read()
if count % 4 !=0:
count += 1
continue
if success:
''Run frame through models''
else:
break
Method 2 code snippet:
This method is slower. I am trying to avoid reading unnecessary frames in this case.
cap = cv2.VideoCapture(video)
count=0
while True:
if count % 4 != 0:
cap.set(cv2.CV_CAP_PROP_POS_FRAMES, count*15)
count+=1
success, frame = cap.read()
Any advice on how to achieve this in the most efficient way would be greatly appreciated.

Getting and setting frames by changing CV_CAP_PROP_POS_FRAMES is not accurate (and slow) due to how video compression works: by using keyframes.
It might help to not use the read() function at all. Instead use grab() and only retrieve() the needed frames. From the documentation: The (read) methods/functions combine VideoCapture::grab() and VideoCapture::retrieve() in one call.
grab() gets the frame data, and retrieve() decodes it (the computationally heavy part). What you might want to do is only grab frames you want to skip but not retrieve them.
Depending on your system and opencv build, you could also have ffmpeg decode video using hardware acceleration.

As I get it you are trying to process every fourth frame. You are using the condition:
if count % 4 != 0
which is triggered 3 out of 4 frames instead (you are processing frames 1, 2, 3, 5, 6, 7 etc)! Use the opposite:
if count % 4 == 0
Also although code snippets the two method does not seem to process the same frames. Although in both cases your counter seems to increase by 1 in each frame you actually point to the 15xframe of that counter in the second case (cap.set(cv2.CV_CAP_PROP_POS_FRAMES, count*15).
Some comments on your code also (maybe I misunderstood something):
Case 1:
while cap.isOpened():
success, frame = cap.read()
if count % 4 !=0:
count += 1
continue
here you seem to count only some frames (3 out of 4 as mentioned) since frames which are multiples of 4 are skipped: condition count % 4 !=0 is not met in that case and your counter is not updated although you read a frame. So, you have an inaccurate counter here. It's not shown how and where do you process you frames to judge that part though.
Case 2:
while True:
if count % 4 != 0:
cap.set(cv2.CV_CAP_PROP_POS_FRAMES, count*15)
count+=1
success, frame = cap.read()
here you read frames only if condition is met so in this code snippet you don't actually read any frame since frame 0 does not trigger the condition! If you update the counter outside the if scope it's not clear here. But if you do you should also read a frame there. Anyway more code should be revealed to tell.
As a general advice you should update the counter everytime you read a frame.

Instead putting a threshold on number of frames, which would make opencv process ALL the frames (which you rightly pointed out, slows the video processing), it's better to use CAP_PROP_POS_MSEC link and offload that processing to cv2. By using this option you can configure cv2 to sample 1 frame every nth Millisecond. So setting subsample_rate=1000 in vidcap.set(cv2.CAP_PROP_POS_MSEC, (frame_count * subsample_rate)) would sample 1 frame every 1 second (as 1000 milliseconds equals 1 second). Hopefully this improves your video processing speed.
def extractImagesFromVideo(path_in, subsample_rate, path_out, saveImage, resize=(), debug=False):
vidcap = cv2.VideoCapture(path_in)
if not vidcap.isOpened():
raise IOError
if debug:
length = int(vidcap.get(cv2.cv2.CAP_PROP_FRAME_COUNT))
width = int(vidcap.get(cv2.cv2.CAP_PROP_FRAME_WIDTH))
height = int(vidcap.get(cv2.cv2.CAP_PROP_FRAME_HEIGHT))
fps = vidcap.get(cv2.cv2.CAP_PROP_FPS)
print 'Length: %.2f | Width: %.2f | Height: %.2f | Fps: %.2f' % (length, width, height, fps)
success, image = vidcap.read() #extract first frame.
frame_count = 0
while success:
vidcap.set(cv2.CAP_PROP_POS_MSEC, (frame_count*subsample_rate))
success, image = vidcap.read()
if saveImage and np.any(image):
cv2.imwrite(os.path.join(path_out, "%s_%d.png" % (frame_count)), image)
frame_count = frame_count + 1
return frame_count

OpenCV frame count wrong for interlaced videos, workaround?

Working with the code sample from How does QueryFrame work? I noticed that the program used a lot of time to exit if it ran to the end of the video. I wanted to exit quickly on the last frame, and I verified that it's a lot quicker if I don't try to play past the end of the video, but there are some details that don't make sense to me. Here's my code:
import cv2
# create a window
winname = "myWindow"
win = cv2.namedWindow(winname, cv2.CV_WINDOW_AUTOSIZE)
# load video file
invideo = cv2.VideoCapture("video.mts")
frames = invideo.get(cv2.cv.CV_CAP_PROP_FRAME_COUNT)
print "frame count:", frames
# interval between frame in ms.
fps = invideo.get(cv2.cv.CV_CAP_PROP_FPS)
interval = int(1000.0 / fps)
# play video
while invideo.get(cv2.cv.CV_CAP_PROP_POS_FRAMES) < frames:
print "Showing frame number:", invideo.get(cv2.cv.CV_CAP_PROP_POS_FRAMES)
(ret, im) = invideo.read()
if not ret:
break
cv2.imshow(winname, im)
if cv2.waitKey(interval) == 27: # ASCII 27 is the ESC key
break
del invideo
cv2.destroyWindow(winname)
The only thing is that the frame count returned is 744, while the last played frame number is 371 (counting from 0, so that's 372 frames). I assume this is because the video is interlaced, and I guess I need to account for that and divide interval by 2 and frames by 2. But the question is, how do I figure out that I need to do this? There doesn't seem to be a property to check this:
http://docs.opencv.org/modules/highgui/doc/reading_and_writing_images_and_video.html#videocapture-get