I use the library PyAV because it's one of the fatest available with Python.
Here is a simple example of code I would like to use:
import av
video = av.open("My_Super_Video.mp4")
total_frames = # ????
for i, frame in enumerate(video.decode(video=0)):
img = frame.to_image() # PIL image
print("Frame: %d/%d ..." % (i, total_frames))
I could obviously use other libraries to load the library, however I would prefer using PyAV if possible due to its processing speed.
Question 1: Is it possible to obtain the number of frames with PyAV ? If yes, how ?
Question 2: In the case, I would consider using another library to load and process the video frame by frame. Which library would allow me to do the above with the highest speed as possible. I know the followings, but don't know how they compare:
PIMS On top of PyAV, could add some interesting feature ?
MoviePy (limited to videos which fit in RAM), but what about perf ?
Imageio (probably same limitation as above), but what about perf ?
OpenCV (probably same limitation as above), but what about perf ?
Others ?
To get the frames of the first video stream do:
container = av.open("My_Super_Video.mp4")
total_frames = container.streams.video[0].frames
You can get the number of frames in stream with Stream.frames attribute.
Source: http://docs.mikeboers.com/pyav/develop/api/stream.html#av.stream.Stream
Old question, but only partly answered. Let me answer the second question as well.
Question 1: Is it possible to obtain the number of frames with PyAV ? If yes, how ?
import av
with av.open("My_Super_Video.mp4") as container:
total_frames = container.streams.video[0].frames
Question 2: In the case, I would consider using another library to load and process the video frame by frame. Which library would allow me to do the above with the highest speed as possible. I know the followings, but don't know how they compare: [...]
ImageIO timings: 0.497
PyAV timings: 0.908
MoviePy timings: 0.766
OpenCV timings: 0.766
OpenCV timings: 0.569 (no conversion to RGB)
ImageIO is the fastest; hands down. OpenCV comes close (14% slower), but only if you can do your processing in BGR. If you have to work in RGB then the conversion costs you dearly (54% slower 🥵).
That said, it is highly workload-dependent and you should always benchmark with your specific setup. In practice, the difference is often negligible compared to how much time you spend processing each frame.
Here is the benchmark code for those interested:
import cv2
import av
import imageio.v3 as iio
from moviepy.editor import VideoFileClip
from PIL import Image
from timeit import Timer
# create a test video (roughly 11 sec and sane encoding)
frames = iio.imread("imageio:cockatoo.mp4", plugin="pyav")
iio.imwrite("test_video.mp4", frames, plugin="pyav", codec="h264")
def iio_read():
total_frames = iio.improps("test_video.mp4", plugin="pyav").shape[0]
for idx, frame in enumerate(iio.imiter("test_video.mp4", plugin="pyav")):
foo = Image.fromarray(frame)
# Note: I will not print in the benchmark. This will skew the result
# print("Frame: %d/%d ..." % (idx, total_frames))
def av_read():
with av.open("test_video.mp4") as container:
total_frames = container.streams.video[0].frames
for frame in container.decode(video=0):
foo = frame.to_image()
def moviepy_read():
# Can not read frame_count
for frame in VideoFileClip("test_video.mp4").iter_frames():
foo = Image.fromarray(frame)
def cv2_read():
cap = cv2.VideoCapture("test_video.mp4")
total_frames = cap.get(cv2.CAP_PROP_FRAME_COUNT)
success, frame = cap.read()
idx = 0
while success:
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
foo = Image.fromarray(frame)
success, frame = cap.read()
idx += 1
def cv2_read2():
cap = cv2.VideoCapture("test_video.mp4")
total_frames = cap.get(cv2.CAP_PROP_FRAME_COUNT)
success, frame = cap.read()
idx = 0
while success:
foo = Image.fromarray(frame)
success, frame = cap.read()
idx += 1
repeats = 10
time_moviepy = min(Timer("moviepy_read()", globals=globals()).repeat(repeats, number=1))
time_cv2 = min(Timer("cv2_read()", globals=globals()).repeat(repeats, number=1))
time_cv2_no_convert = min(Timer("cv2_read2()", globals=globals()).repeat(repeats, number=1))
time_iio = min(Timer("iio_read()", globals=globals()).repeat(repeats, number=1))
time_av = min(Timer("av_read()", globals=globals()).repeat(repeats, number=1))
print(
f"""
ImageIO timings: {time_iio:.3f}
PyAV timings: {time_av:.3f}
MoviePy timings: {time_moviepy:.3f}
OpenCV timings: {time_cv2:.3f}
OpenCV timings: {time_cv2_no_convert:.3f} (no conversion to RGB)
"""
)
Package Versions:
av==10.0.0
moviepy==1.0.3
Pillow==9.4.0
opencv-python==4.7.0.68
imageio==2.25.0
Related
so I want to create a program in which I will compress a video using this method:
Save a frame for a video each second, and for the rest of the frames in that second, save only the changes from that main frame. And then combine all that info to create a smaller-sized video.
So my idea is to iterate through each frame, save that frame in an array, and when I reach the main frame, find the differences and rewrite all the other frames. However I'm not sure how I can create a video using differences and a main frame afterward... Should I use cv2.VideoWriter or that only works with frames that haven't been altered?
This is what I have for now (I haven't saved the frames yet because I'm not sure of the format that I need):
import cv2
import time
imcap = cv2.VideoCapture('test2.mp4') # test 1
sample_rate = 30
success, img = imcap.read() # get the next frame
frame_no = 0
fps = imcap.get(cv2.CAP_PROP_FPS)
print("Frames per second: {0}".format(fps))
while success:
frame_no += 1
if frame_no % sample_rate == 0:
cv2.imshow('frame', img)
cv2.waitKey()
print(frame_no)
# read next frame
success, img = imcap.read()
imcap.release()
I want to save the images extracted from the video without losing any information, resolution and quality. I have saved using OpenCV in four formats png, bmp, jpg, tif The code is as below
file = "video.MP4"
video = cv2.VideoCapture(file)
# Find OpenCV version
(major_ver, minor_ver, subminor_ver) = (cv2.__version__).split('.')
# With webcam get(CV_CAP_PROP_FPS) does not work.
# Let's see for ourselves.
if int(major_ver) < 3 :
fps = video.get(cv2.cv.CV_CAP_PROP_FPS)
print ("Frames per second using video.get(cv2.cv.CV_CAP_PROP_FPS): {0}".format(fps))
else :
fps = video.get(cv2.CAP_PROP_FPS)
print ("Frames per second using video.get(cv2.CAP_PROP_FPS) : {0}".format(fps))
# Number of frames to capture
num_frames = 120;
print ("Capturing {0} frames".format(num_frames))
# Start time
start = time.time()
data = []
# Grab a few frames
for i in range(0, num_frames) :
ret, frame = video.read()
data.append(frame)
print("Shape",frame.shape)
# End time
end = time.time()
# Time elapsed
seconds = end - start
print ("Time taken : {0} seconds".format(seconds))
# Calculate frames per second
fps = num_frames / seconds;
print ("Estimated frames per second : {0}".format(fps))
for i,img in enumerate(data):
fileName = 'Frames\img_'+ str(i) + '.tif'
cv2.imwrite(fileName,img)
# Release video
video.release()
My Questions are
In which of the format (png, bmp, jpg, tif) I should save so that it should not lose the resolution and quality of video.
Just putting extension like .tif in cv2.imwrite() saves it to that particular format?
Which library is the best to save OpenCV, scikit-image or Pillow
Any help or suggestion will be highly appreciated.
Regards
well .jpg is a lossy format so you can loose some information. So it'd be better to use .png format to save the video frames. Because in this case you can also set the IMWRITE_PNG_COMPRESSION level. In cv2 its value is from 0 to 9. A higher value means higher compression level and longer time. This value defaults to 1 in cv2 so you can set it to 0 if you don't want to compress it at all. A example code is as follow,
# lets say you stored a single video frame in variable named 'frame' then
# save it as follows to avoid any compression or loss of information
png_compression_level = 0 # for no compression what so ever (means larger file size)
cv2.imwrite(dest_dir_path, frame,
[int(cv2.IMWRITE_PNG_COMPRESSION), png_compression_level])
you can then also verify if the stored image lost any information or not by using a pip package called sewar
img = cv2.imread('read the frame you saved as iamge')
# then
from sewar.full_ref import rmse, psnr, uqi, ssim
print("RMSE: ", rmse(img, frame)) # 0 is ideal value
print("PSNR: ", psnr(img, frame)) # inf is ideal value
print("SSIM: ", ssim(img, frame)) # (1, 1) is ideal value
print("UQI: ", uqi(img, frame)) # 1 is the ideal value
I'm using python OpenCV to read frames from a video file that has a variable frame rate.
I need to understand how the wall time changes between frames (i.e. I want to be able to write the timestamp on every frame). My understanding is that the underlying video format stores how long each frame lasts i.e. the video file contains the information to tell the video player how long it should display the given frame before moving to the next one.
I would like to programmatically access this data using the python OpenCV interface. An example (that does not work) of what this might look like is:
import numpy as np
import cv2 as cv
cap = cv.VideoCapture('my_variable_frame_rate_video.mp4')
while cap.isOpened():
ret, frame = cap.read()
frame_length_ms = ret.frame_length() # obviously doesn't work
print("The length of the frame in ms is {}".format(frame_length_ms))
# Should print: The length of the frame in ms is 23
Things that don't work
Reading the source frame rate from the capture device:
fps = cap.get(cv2.cv.CV_CAP_PROP_FPS)
Counting the total number of frames:
fps = cap.get(cv2.cv.CV_CAP_PROP_FRAME_COUNT)/total_length
Both of these don't work because we are dealing with a variable frame rate
An idea could be extract the amount of time and write it in each frame
import cv2
import time
path="video.mp4"
video = cv2.VideoCapture(path);
start = time.time()
ret, frame = video.read()
font = cv2.FONT_HERSHEY_SIMPLEX
i = 0
while ret:
start = time.time()
ret, frame = video.read()
end = time.time()
seconds = end - start
cv2.putText(frame,str(seconds),(10,500), font, 4,(255,255,255),2,cv2.LINE_AA)
cv2.imwrite(str(i)+".jpg", frame)
i+=1
video.release()
I would like to do video processing on neighboring frames. More specific, I would like to compute the mean square error between neighboring frames:
mean_squared_error(prev_frame,frame)
I know how to compute this in a linear straightforward way: I use the imutils package to utilize a queue to decouple loading the frames and processing them. By storing them in a queue, I don't need to wait for them before I can process them. ... but I want to be even faster...
# import the necessary packages to read the video
import imutils
from imutils.video import FileVideoStream
# package to compute mean squared errror
from skimage.metrics import mean_squared_error
if __name__ == '__main__':
# SPECIFY PATH TO VIDEO FILE
file = "VIDEO_PATH.mp4"
# START IMUTILS VIDEO STREAM
print("[INFO] starting video file thread...")
fvs = FileVideoStream(path_video, transform=transform_image).start()
# INITALIZE LIST to store the results
mean_square_error_list = []
# READ PREVIOUS FRAME
prev_frame = fvs.read()
# LOOP over frames from the video file stream
while fvs.more():
# GRAP THE NEXT FRAME from the threaded video file stream
frame = fvs.read()
# COMPUTE the metric
metric_val = mean_squared_error(prev_frame,frame)
mean_square_error_list.append(1-metric_val) # Append to list
# UPDATE previous frame variable
prev_frame = frame
Now my question is: How can I mutliprocess the computation of the metric to increase speed and save time ?
My operating system is Windows 10 and I am using python 3.8.0
There are too many aspects of making things faster, I'll only focus on the multiprocessing part.
As you don't want to read the whole video at a time, we have to read the video frame by frame.
I'll be using opencv (cv2), numpy for reading the frames, calculating mse, and saving the mse to disk.
First, we can start without any multiprocessing so we can benchmark our results. I'm using a video of 1920 by 1080 dimension, 60 FPS, duration: 1:29, size: 100 MB.
import cv2
import sys
import time
import numpy as np
import subprocess as sp
import multiprocessing as mp
filename = '2.mp4'
def process_video():
cap = cv2.VideoCapture(filename)
proc_frames = 0
mse = []
prev_frame = None
ret = True
while ret:
ret, frame = cap.read() # reading frames sequentially
if ret == False:
break
if not (prev_frame is None):
c_mse = np.mean(np.square(prev_frame-frame))
mse.append(c_mse)
prev_frame = frame
proc_frames += 1
np.save('data/' + 'sp' + '.npy', np.array(mse))
cap.release()
return
if __name__ == "__main__":
t1 = time.time()
process_video()
t2 = time.time()
print(t2-t1)
In my system, it runs for 142 secs.
Now, we can take the multiprocessing approach. The idea can be summarized in the following illustration.
GIF credit: Google
We make some segments (based on how many cpu cores we have) and process those segmented frames in parallel.
import cv2
import sys
import time
import numpy as np
import subprocess as sp
import multiprocessing as mp
filename = '2.mp4'
def process_video(group_number):
cap = cv2.VideoCapture(filename)
num_processes = mp.cpu_count()
frame_jump_unit = cap.get(cv2.CAP_PROP_FRAME_COUNT) // num_processes
cap.set(cv2.CAP_PROP_POS_FRAMES, frame_jump_unit * group_number)
proc_frames = 0
mse = []
prev_frame = None
while proc_frames < frame_jump_unit:
ret, frame = cap.read()
if ret == False:
break
if not (prev_frame is None):
c_mse = np.mean(np.square(prev_frame-frame))
mse.append(c_mse)
prev_frame = frame
proc_frames += 1
np.save('data/' + str(group_number) + '.npy', np.array(mse))
cap.release()
return
if __name__ == "__main__":
t1 = time.time()
num_processes = mp.cpu_count()
print(f'CPU: {num_processes}')
# only meta-data
cap = cv2.VideoCapture(filename)
width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
fps = cap.get(cv2.CAP_PROP_FPS)
frame_jump_unit = cap.get(cv2.CAP_PROP_FRAME_COUNT) // num_processes
cap.release()
p = mp.Pool(num_processes)
p.map(process_video, range(num_processes))
# merging
# the missing mse will be
final_mse = []
for i in range(num_processes):
na = np.load(f'data/{i}.npy')
final_mse.extend(na)
try:
cap = cv2.VideoCapture(filename) # you could also take it outside the loop to reduce some overhead
frame_no = (frame_jump_unit) * (i+1) - 1
print(frame_no)
cap.set(1, frame_no)
_, frame1 = cap.read()
#cap.set(1, ((frame_jump_unit) * (i+1)))
_, frame2 = cap.read()
c_mse = np.mean(np.square(frame1-frame2))
final_mse.append(c_mse)
cap.release()
except:
print('failed in 1 case')
# in the last few frames, nothing left
pass
t2 = time.time()
print(t2-t1)
np.save(f'data/final_mse.npy', np.array(final_mse))
I'm using just numpy save to save the partial results, you can try something better.
This one runs for 49.56 secs with my cpu_count = 12. There are definitely some bottlenecks that can be avoided to make it run faster.
The only issue with my implementation is, it's missing the mse for regions where the video was segmented, it's pretty easy to add. As we can index individual frames at any location with OpenCV in O(1), we can just go to those locations and calculate mse separately and merge to the final solution. [Check the updated code it fixes the merging part]
You can write a simple sanity check to ensure, both provide the same result.
import numpy as np
a = np.load('data/sp.npy')
b = np.load('data/final_mse.npy')
print(a.shape)
print(b.shape)
print(a[:10])
print(b[:10])
for i in range(len(a)):
if a[i] != b[i]:
print(i)
Now, some additional speedups can come from using a CUDA-compiled opencv, ffmpeg, adding queuing mechanism plus multiprocessing, etc.
import matplotlib.pyploy as plt
import cv2
def getFrame(cap):
frameRate = 25
frameId = cap.get(cv2.CAP_PROP_POS_FRAMES)
ret, frame = cap.read()
if not ret:
return None
print frameId
if frameId < 0:
return None
if not (frameId % int(frameRate)):
cap.set(cv2.CAP_PROP_POS_FRAMES, frameId + frameRate)
return frame
return None
videoFile = 'filename.webm'
cap = cv2.VideoCapture(videoFile,cv2.CAP_FFMPEG)
image = getFrame(cap)
plt.imshow(image)
cap.set(cv2.CAP_PROP_POS_FRAMES,4000.0)
image = getFrame(cap)
while(image is not None):
plt.imshow(image)
image = getFrame(cap)
cap.release()
I'm running the above code in a while loop and displaying the image. It seems that after around 4250 frames, the frame returned will be the same as the frame at the start. This loop continue after another 4250 frames.
The file I'm reading is mp4 and webm file. The behavior is the same for both types of file. videoCapture is using FFMPEG backend. Opencv version = 3.4.1, python 2.7 .
Another notable behaviour is the reading of the frame slows down as the frame increases and then went back to fast again after 4250 frames.
Edit:
I edited the code. I think you can try running this with mp4/webm file longer than 5 minutes. I can't upload the video due to privacy reason because it is recorded in office.
When I modified the code with only cap.read(), the error disappears. I suspect this is something to do with cap.set()
Use cap.set(cv2.cv.CV_CAP_PROP_FPS, 25) instead of skipping frames and setting the frame number in code.
Your code does not check the size of the video before setting cv2.CAP_PROP_POS_FRAMES, frameId + frameRate.
You can check the size in frames with cap.get(cv2.CAP_PROP_FRAME_COUNT).
def getNextFrame(cap):
frameRate = 25
for i in range(frameRate-1):
cap.grab()
ret, frame = cap.read()
return ret, frame
I've found a workaround. I use grab() for the 24 frames to be skipped and then only read() for the frame I need.
I'm not sure if this is the best approach but I'll put it out here. I'll still be accepting other answers.