Develop Reference

Concatenate a video, image and audio using ffmpeg

I am trying to concatenate a group of images with associated audio with a video clip at the start and front of the video. Whenever I concatenate the image with the associated audio it dosen't playback correctly in VLC media player and only displays the image for a frame before cutting to black and continually playing audio. I came across this github issue: https://github.com/kkroening/ffmpeg-python/issues/274 where the accepted solution was the one I implemented but one of the comments mentioned this issue of incorrect playback and error on youtube.
'''
Generates a clip from an image and a wav file, helper function for export_video
'''
def generate_clip(img):
transition_cond = os.path.exists("static/transitions/" + img + ".mp4")
chart_path = os.path.exists("charts/" + img + ".png")
if transition_cond:
clip = ffmpeg.input("static/transitions/" + img + ".mp4")
elif chart_path:
clip = ffmpeg.input("charts/" + img + ".png")
else:
clip = ffmpeg.input("static/transitions/Transition.jpg")
audio_clip = ffmpeg.input("audio/" + img + ".wav")
clip = ffmpeg.concat(clip, audio_clip, v=1, a=1)
clip = ffmpeg.filter(clip, "setdar","16/9")
return clip
'''
Combines the charts from charts/ and the audio from audio/ to generate one final video that will be uploaded to Youtube
'''
def export_video(CHARTS):
clips = []
intro = generate_clip("Intro")
clips.append(intro)
for key in CHARTS.keys():
value = CHARTS.get(key)
value.insert(0, key)
subclip = []
for img in value:
subclip.append(generate_clip(img))
concat_clip = ffmpeg.concat(*subclip)
clips.append(concat_clip)
outro = generate_clip("Outro")
clips.append(outro)
concat_clip = ffmpeg.concat(*clips)
concat_clip.output("export/export.mp4").run(overwrite_output=True)

It is unfortunate concat filter does not offer the shortest option like overlay. Anyway, the issue here is that image2 demuxer uses 25 fps by default, so a video stream with one image only lasts for 1/25 seconds long. There are a several ways to address this, but you first need to get the duration of the paired audio files. To incorporate the duration information to the ffmpeg command, you can:
Use tpad filter for each video (in series with setdar) to make the video duration to match the audio. Padded amount should be 1/25 seconds less than the audio duration.
Specify -loop 1 input option so the image will loop (indefinitely) and then specify an additional -t {duration} input option to limit the number of loops. Caution that the video duration may not be exact.
Specify -r {1/duration} so the image will last as long as the audio and use fps filter on each input to the output frame rate.
I'm not familiar with ffmpeg-python so I cannot provide its solution, but if you're interested, I'd be happy to post an equivalent code with my ffmpegio package.
[edit]
ffmpegio Solution
Here is how I'd code the 3rd solution with ffmpegio:
import ffmpegio
def generate_clip(img):
"""
Generates a clip from an image and a wav file,
helper function for export_video
"""
transition_cond = path.exists("static/transitions/" + img + ".mp4")
chart_path = path.exists("charts/" + img + ".png")
if transition_cond:
video_file = "static/transitions/" + img + ".mp4"
elif chart_path:
video_file = "charts/" + img + ".png"
else:
video_file = "static/transitions/Transition.jpg"
audio_file = "audio/" + img + ".wav"
video_opts = {}
if not transition_cond:
# audio_streams_basic() returns audio duration in seconds as Fraction
# set the "framerate" of the video to be the reciprocal
info = ffmpegio.probe.audio_streams_basic(audio_file)
video_opts["r"] = 1 / info[0]["duration"]
return [(video_file, video_opts), (audio_file, None)]
def export_video(CHARTS):
"""
Combines the charts from charts/ and the audio from audio/
to generate one final video that will be uploaded to Youtube
"""
# get all input files (video/audio pairs)
clips = [
generate_clip("Intro"),
*(generate_clip(img) for key, value in CHARTS.items() for img in value),
generate_clip("Outro"),
]
# number of clips
nclips = len(clips)
# filter chains to set DAR and fps of all video streams
vfilters = (f"[{2*n}:v]setdar=16/9,fps=30[v{n}]" for n in range(nclips))
# concatenation filter input: [v0][1:a][v1][3:a][v2][5:a]...
concatfilter = "".join((f"[v{n}][{2*n+1}:a]" for n in range(nclips))) + f"concat=n={nclips}:v=1:a=1[vout][aout]"
# form the full filtergraph
fg = ";".join((*vfilters, concatfilter))
# set output file and options
output = ("export/export.mp4", {"map": ["[vout]", "[aout]"]})
# run ffmpeg
ffmpegio.ffmpegprocess.run(
{
"inputs": [input for pair in clips for input in pair],
"outputs": [output],
"global_options": {"filter_complex": fg},
},
overwrite=True,
)
Since this code does not use the read/write features, ffmpegio-core package suffices:
pip install ffmpegio-core
Make sure that FFmpeg binary can be found by ffmpegio. See the installation doc.
Here are the direct links to the documentations of the functions used:
ffmpegprocess.run
ffmpeg_args dict argument
probe.audio_streams_basic (Ignore the documentation error both duration and start_time are both of Fraction type.
The code has not been fully validated. If you encounter a problem, it might be the easiest to post it on the GitHub Discussions to proceed.

Python New n frame video is heavier than the input video

I managed to write a code to decimate my video and take only 1 frame out of 10, in order to make my neural network more efficient in the future for character recognition.
The new video exit_video is well decimated because it's way faster than the previous one.
1: When I print the fps of the new video, I have 30 again despite the decimation
2: Why is my new video heavier ? 50.000 ko and it was 42.000 ko for the firts one
Thanks for your help
import cv2
#import os
import sys
video = cv2.VideoCapture("./video/inputvideo.mp4")
frameWidth = int(video.get(cv2.CAP_PROP_FRAME_WIDTH))
frameHeight = int(video.get(cv2.CAP_PROP_FRAME_HEIGHT))
frameFourcc = int(video.get(cv2.CAP_PROP_FOURCC))
success,image = video.read()
if not success:
print('impossible de prendre une frame')
sys.exit()
fps = video.get(cv2.CAP_PROP_FPS)
print("fps de base " + str(fps))
print(frameFourcc)
count = 0
exit_file = 'decimated_v1.mp4'
exit_video = cv2.VideoWriter(exit_file, frameFourcc, fps, (frameWidth, frameHeight))
while True:
if ((count % 10 ) ==0):
exit_video.write(image)
success,image = video.read()
if not success:
break
count +=1
exit_video.release()
exit_video_info = cv2.VideoCapture("decimated_v1.mp4")
fps_sortie = exit_video_info.get(cv2.CAP_PROP_FPS)
print("fps de sortie " + str(fps_sortie))

Decimating a video file that's not all Intra frames will require re-encoding. Unless your input file is e.g. ProRes or MJPEG, that's likely going to be the case.
Since you're not setting encoding parameters, OpenCV likely end up using some defaults that end up with a higher bitrate than your input file.
You'll probably have a better time using the FFmpeg tool than OpenCV, and its select filter.
ffmpeg -i ./video/inputvideo.mp4 -vf select='not(mod(n\,10))' ./decimated_v1.mp4
would be the basic syntax to use every tenth frame from the input; you can then add your desired encoding parameters such as -crf to adjust the H.264 rate factor – or, of course, you can change to a different codec altogether.

Unable to find headers of jpg while reading a raw disk image (dd)

I am reading a raw disk image using python 3. My task is to retrieve (carve) jpgs as individual files from the disk image. As I know header pattern (\xd8\xff\xe0 or \xd8\xff\xe1) of jpg. I want to know where I get this while reading file.
fobj = open('carve.dd', 'rb')
data = fobj.read(32)
while data != '':
head_loc = findheader(data)
print(head_loc)
data = fobj.read(32)
def findheader(data) : # to find header in each 32 bytes of data of raw disk image
for i in range(0, len(data) - 3) :
if data[i] == b'\xff' :
if data[i+1:i+4] == b'\xd8\xff\xe0' or data[i+1:i+4] == b'\xd8\xff\xe1' :
return i
return -1
The same code is working fine in Python 2. In Python 2, I am able to get headers in just a few seconds from image. Can someone help me out, what is the problem in Python 3?
This code snippet is actually from this https://github.com/darth-cheney/JPEG-Recover/blob/master/jpegrecover2.py
This runs fine in Python 2 but not in Python 3. Please forget about inconsistent tab error when you run the code in link. I again retyped in VS code.

Like the old saying goes, I've got some bad news and some good news. The bad is I can't figure out why your code doesn't work the same in both version 2 and version 3 of Python.
The good is that I was able to reproduce the problem using the sample data you provided, but—more importantly—able to devise something that not only works consistently in both versions, it's likely much faster because it doesn't use a for loop to search through each chunk of data looking for the .jpg header patterns.
from __future__ import print_function
LIMIT = 100000 # Number of chunks (for testing).
CHUNKSIZE = 32 # Bytes.
HDRS = b'\xff\xd8\xff\xe0', b'\xff\xd8\xff\xe1'
IMG_PATH = r'C:\vols\Files\Temp\carve.dd.002'
print('Searching...')
with open(IMG_PATH, 'rb') as file:
chunk_index = 0
found = 0
while True:
data = file.read(CHUNKSIZE)
if not data:
break
# Search for each of the headers in each chunk.
for hdr in HDRS:
offset = 0
while offset < (CHUNKSIZE - len(hdr)):
try:
head_loc = data[offset:].index(hdr)
except ValueError: # Not found.
break
found += 1
file_offset = chunk_index*CHUNKSIZE + head_loc
print('found: #{} at {:,}'.format(found, file_offset))
offset += (head_loc + len(hdr))
chunk_index += 1
if LIMIT and (chunk_index == LIMIT): break # Stop after this many chunks.
print('total found {}'.format(found))

Alsaaudio record and playback

I was just playing around with sound input and output on a raspberry pi using python.
My plan was to read the input of a microphone, manipulate it and playback the manipulated audio. At the moment I tried to read and playback the audio.
The reading seems to work, since i wrote the read data into a wave file in the last step, and the wave file seemed fine.
But the playback is noise sounds only.
Playing the wave file worked as well, so the headset is fine.
I think maybe I got some problem in my settings or the output format.
The code:
import alsaaudio as audio
import time
import audioop
#Input & Output Settings
periodsize = 1024
audioformat = audio.PCM_FORMAT_FLOAT_LE
channels = 16
framerate=8000
#Input Device
inp = audio.PCM(audio.PCM_CAPTURE,audio.PCM_NONBLOCK,device='hw:1,0')
inp.setchannels(channels)
inp.setrate(framerate)
inp.setformat(audioformat)
inp.setperiodsize(periodsize)
#Output Device
out = audio.PCM(audio.PCM_PLAYBACK,device='hw:0,0')
out.setchannels(channels)
out.setrate(framerate)
out.setformat(audioformat)
out.setperiodsize(periodsize)
#Reading the Input
allData = bytearray()
count = 0
while True:
#reading the input into one long bytearray
l,data = inp.read()
for b in data:
allData.append(b)
#Just an ending condition
count += 1
if count == 4000:
break
time.sleep(.001)
#splitting the bytearray into period sized chunks
list1 = [allData[i:i+periodsize] for i in range(0, len(allData), periodsize)]
#Writing the output
for arr in list1:
# I tested writing the arr's to a wave file at this point
# and the wave file was fine
out.write(arr)
Edit: Maybe I should mention, that I am using python 3

I just found the answer. audioformat = audio.PCM_FORMAT_FLOAT_LE this format isn't the one used by my Headset (just copied and pasted it without a second thought).
I found out about my microphones format (and additional information) by running speaker-test in the console.
Since my speakers format is S16_LE the code works fine with audioformat = audio.PCM_FORMAT_S16_LE

consider using plughw (alsa subsystem supporting resampling/conversion) for the sink part of the chain at least:
#Output Device
out = audio.PCM(audio.PCM_PLAYBACK,device='plughw:0,0')
this should help to negotiate sampling rate as well as the data format.
periodsize is better to estimate based on 1/times of the sample rate like:
periodsize = framerate / 8 (8 = times for 8000 KHz sampling rate)
and sleeptime is better to estimate as a half of the time necessary to play periodsize:
sleeptime = 1.0 / 16 (1.0 - is a second, 16 = 2*times for 8000 KHz sampling rate)

pyUSB get a continuous stream of data from sensor

I have a device that's connected through usb and I'm using pyUSB to interface with the data.
This is what my code currently looks like:
import usb.core
import usb.util
def main():
device = usb.core.find(idVendor=0x072F, idProduct=0x2200)
# use the first/default configuration
device.set_configuration()
# first endpoint
endpoint = device[0][(0,0)][0]
# read a data packet
data = None
while True:
try:
data = device.read(endpoint.bEndpointAddress,
endpoint.wMaxPacketSize)
print data
except usb.core.USBError as e:
data = None
if e.args == ('Operation timed out',):
continue
if __name__ == '__main__':
main()
It is based off the mouse reader, but the data that I'm getting isn't making sense to me:
array('B', [80, 3])
array('B', [80, 2])
array('B', [80, 3])
array('B', [80, 2])
My guess is that it's reading only a portion of what's actually being provided? I've tried settign the maxpacketsize to be bigger, but nothing.

pyUSB sends and receives data in string format. The data which you are receiving is ASCII codes. You need to add the following line to read the data properly in the code.
data = device.read(endpoint.bEndpointAddress,
endpoint.wMaxPacketSize)
RxData = ''.join([chr(x) for x in data])
print RxData
The function chr(x) converts ASCII codes to string. This should resolve your problem.

I'm only an occasional Python user so beware. If your python script cannot keep up with the amount of data being sampled, then this is what works for me. I'm sending from a uC to the PC blocks of 64 bytes. I use a buffer to hold my samples and later I save them in a file or plot them. I adjust the number multiplying 64 (10 in the example below) until I receive all the samples I was expecting.
# Initialization
rxBytes = array.array('B', [0]) * (64 * 10)
rxBuffer = array.array('B')
Within a loop, I get the new samples and store them in the buffer
# Get new samples
hid_dev.read(endpoint.bEndpointAddress, rxBytes)
rxBuffer.extend(rxBytes)
Hope this helps.