Develop Reference

Combining mp4 with wav in python

I am trying to combine a .mp4 file with a .wav file. I am rendering my mp4 with cv2 videowriter, and I don't think it has anyway of incorporating audio with it. I have tried moviepy.editor, and ffmpeg. moviepy.editor kept messing up the video file and ffmpeg repeatedly kept giving me an error that it couldn't edit existing files in-place. Combining .mp4 with another audio file type is also fine, but if so it would be nice to also answer how to convert midi files to the file type you answered with. Thanks for the help!
moviepy.editor workflow:
video = mpe.VideoFileClip(mp4_path)
os.system(f"timidity {midi_path} -Ow -o {wav_path)}") # Convert .mid to .wav
video = video.set_audio(mpe.AudioFileClip(wav_path))
video.write_videofile(mp4_path, fps=fps)
ffmpeg workflow:
video = ffmpeg.input(mp4_path)
os.system(f"timidity {midi_path} -Ow -o {wav_path)}") # Convert .mid to .wav
audio = ffmpeg.input(wav_path)
video = ffmpeg.output(video, audio, path, vcodec='copy', acodec='aac', strict='experimental')
ffmpeg.run(video)

I tested both modules and for moviepy I get correct output video with audio even if I use the same name as output. So I don't know what can mess with output.
For ffmpeg I had to use different name for output file to resolve problem with couldn't edit existing files in-place
I had to also use object.video and object.audio to replace audio in output file.
video = ffmpeg.input(video_path).video # get only video channel
audio = ffmpeg.input(audio_path).audio # get only audio channel
My testing code
def test_moviepy(video_path, audio_path, output_path='output-moviepy.mp4', fps=24):
import moviepy.editor as mpe
print('--- moviepy ---')
video = mpe.VideoFileClip(video_path)
video = video.set_audio(mpe.AudioFileClip(audio_path))
video.write_videofile(output_path, fps=fps)
def test_ffmpeg(video_path, audio_path, output_path='output-ffmpeg.mp4', fps=24):
import ffmpeg
print('--- ffmpeg ---')
video = ffmpeg.input(video_path).video # get only video channel
audio = ffmpeg.input(audio_path).audio # get only audio channel
output = ffmpeg.output(video, audio, output_path, vcodec='copy', acodec='aac', strict='experimental')
ffmpeg.run(output)
# --- main ---
video_path = 'movie.mp4'
audio_path = 'sound.wav'
output_path = 'output.mp4'
test_moviepy(video_path, audio_path)#, output_path)
test_ffmpeg(video_path, audio_path)#, output_path)
EDIT:
After installing python module graphviz and program graphviz I could run
ffmpeg.view(output, filename='output-ffmpeg.png')
to get image

how play mp4 video in google colab

I have a video.mp4 in content/video.mp4
if I wanted to play the video in google colab without downloading , ¿what code I should use to open a kind of video player in my jupyter notebook?

Here's the code
from IPython.display import HTML
from base64 import b64encode
mp4 = open('video.mp4','rb').read()
data_url = "data:video/mp4;base64," + b64encode(mp4).decode()
HTML("""
<video width=400 controls>
<source src="%s" type="video/mp4">
</video>
""" % data_url)
You can test it in a colab notebook here.
Update (Jun 2020)
To support a big vdo file, I write a library to upload to Google Drive and set it public. Then use the returned URL to display the video.
!pip install -U kora
from kora.drive import upload_public
url = upload_public('video.mp4')
# then display it
from IPython.display import HTML
HTML(f"""<video src={url} width=500 controls/>""")

Currently, we need to compress the video file to play it in google colaboratory, if the format is not supported.
from IPython.display import HTML
from base64 import b64encode
import os
# Input video path
save_path = "/content/videos/result.mp4"
# Compressed video path
compressed_path = "/content/videos/result_compressed.mp4"
os.system(f"ffmpeg -i {save_path} -vcodec libx264 {compressed_path}")
# Show video
mp4 = open(compressed_path,'rb').read()
data_url = "data:video/mp4;base64," + b64encode(mp4).decode()
HTML("""
<video width=400 controls>
<source src="%s" type="video/mp4">
</video>
""" % data_url)
Reference: https://towardsdatascience.com/yolov3-pytorch-on-google-colab-c4a79eeecdea

Just input the mp4 video path to that function and you're good to go.
from IPython.display import HTML
from base64 import b64encode
def show_video(video_path, video_width = 600):
video_file = open(video_path, "r+b").read()
video_url = f"data:video/mp4;base64,{b64encode(video_file).decode()}"
return HTML(f"""<video width={video_width} controls><source src="{video_url}"></video>""")
show_video(video_path)

To support a big vdo file ,getting link url for big videos
!pip install httplib2==0.15.0
!pip install google-api-python-client==1.7.11
#don't forget to restart the environment
from IPython.display import HTML
import IPython
from google.colab import output
!pip install -U kora
from kora.drive import upload_public
video_path='/content/drive/MyDrive/BigFileName.mkv'
video_url = upload_public(video_path) #for google disk to https://
if (video_url.startswith('https://drive.google.com/')):
video_url+='&confirm=t' # to bypass the window Google Drive - Virus scan warning
print('video_url',video_url)

This is all you need to define
import html
from IPython.display import display, Javascript, Image
from google.colab.output import eval_js
def preProcessVideo():
js = Javascript('''
const video = document.createElement('video');
const labelElement = document.createElement('span');
const videoUrl = 'https://rr2---sn-npoldn7z.c.drive.google.com/videoplayback?expire=1641882417&ei=8ercYbCiIuCKmvUPz5WB6Ac&ip=1.55.250.186&cp=QVRJU0lfUVRPSFhPOmpHU0F4ZW1JUnNobkNZVzY0MHlMYm44NDdNek45Nm5sSVQyTWota2J4MlE&id=8955091d9a3609fd&itag=18&source=webdrive&requiressl=yes&mh=yD&mm=32&mn=sn-npoldn7z&ms=su&mv=u&mvi=2&pl=27&ttl=transient&susc=dr&driveid=1S9PGt2CHDfuJSB1nIWebi4KVNRI7jEbf&app=explorer&mime=video/mp4&vprv=1&prv=1&dur=22.825&lmt=1641801389629564&mt=1641867503&txp=0011224&sparams=expire,ei,ip,cp,id,itag,source,requiressl,ttl,susc,driveid,app,mime,vprv,prv,dur,lmt&sig=AOq0QJ8wRgIhAJ8QuQoDRVLULTONbECJ9GyCqACa9Ci7i-4yK6vqgFdxAiEAoC-AMccHI239SCSOukNJEkXmqzKBIPqmb41I25Sjljs=&lsparams=mh,mm,mn,ms,mv,mvi,pl&lsig=AG3C_xAwRgIhAI650mDvui7WOdCTc-zfXSR_jXGCX0_marfJav3vEZDvAiEAz5-kvizrRBxJxmIZpO9LxDxkPQpcMTheY5Sq7pBMPQc=&cpn=BsF1Vhd4TGv91-3f&c=WEB_EMBEDDED_PLAYER&cver=1.20220109.00.00'
async function playVideo() {
const div = document.createElement('div');
video.style.width = 320;
video.style.height = 320;
document.body.appendChild(div);
div.appendChild(labelElement);
div.appendChild(video);
var source = document.createElement('source');
source.setAttribute('src', videoUrl);
source.setAttribute('type', 'video/mp4');
video.appendChild(source);
video.play();
// Resize the output to fit the video element.
google.colab.output.setIframeHeight(document.documentElement.scrollHeight, true);
}
''')
display(js)
eval_js('playVideo()'.format())
Then call it preProcessVideo()

Saving audio from mp4 as wav file using Moviepy Audiofile

I have a video file named 'video.mp4'. I am trying to seperate a section of audio from the video and save it as a wav file that can be used with other Python modules. I want to do this with MoviePy.
I send parameters to the write_audiofile function, specifying the filename, fps, nbyte, and codec.
Following the MoviePy AudioClip docs, I specified the codec as ‘pcm_s32le’ for a 32-bit wav file.
from moviepy.editor import *
sound = AudioFileClip("video.mp4")
newsound = sound.subclip("00:00:13","00:00:15") #audio from 13 to 15 seconds
newsound.write_audiofile("sound.wav", 44100, 2, 2000,"pcm_s32le")
This code generates a .wav file, named 'sound.wav'.
Opening the audio file in Audacity
The resulting file, sound.wav, can be opened in Audacity, however I run into problems when I try to use it as a wav file with other Python modules.
Playing the sound file in pygame
import pygame
pygame.mixer.init()
sound=pygame.mixer.Sound("sound.wav")
The third line gives the following error:
pygame.error: Unable to open file 'sound.wav'
Determining type of sound file using sndhdr.what()
import sndhdr
sndhdr.what("sound.wav")
The sndhdr method returned none
. According to the docs, when this happens, the method failed to determine the type of sound data stored in the file.
Reading the file with Google Speech Recognition
import speech_recognition as sr
r = sr.Recognizer()
audio = "sound.wav"
with sr.AudioFile(audio) as source:
audio = r.record(source)
text= r.recognize_google(audio)
print(text)
This code stops execution on the second to last line:
ValueError: Audio file could not be read as PCM WAV, AIFF/AIFF-C, or Native FLAC; check if file is corrupted or in another format
Why does the audio file open in Audacity, if sndhdr.what() can not recognize it as an audio file type?
How can I properly export a MoviePy AudioClip as a wav file?

I had the same issue with no codec specified or with codec = 'pcms32le', the one that worked for me was pcm_s16le.
Note that I am using "fr-FR" language, you should probably adapt to yur needs.
here is the entire code :
# Python code to convert video to audio
import moviepy.editor as mp
import speech_recognition as sr
# Insert Local Video File Path
clip = mp.VideoFileClip("/tmp/data/test.mp4")
# Insert Local Audio File Path
clip.audio.write_audiofile("/tmp/data/test.wav",codec='pcm_s16le')
# initialize the recognizer
r = sr.Recognizer()
# open the file
with sr.AudioFile("/tmp/data/test.wav") as source:
# listen for the data (load audio to memory)
audio_data = r.record(source)
# recognize (convert from speech to text)
text = r.recognize_google(audio_data, language = "fr-FR")
print(text)

I had the same issue. I was trying to get a mp4 file from URL, then convert It into wav file and call Google Speech Recognition over It. Instead I used pydub to handle conversion and it worked! Here's a sample of the code:
import requests
import io
import speech_recognition as sr
from pydub import AudioSegment
# This function translate speech to text
def speech_to_text(file):
recognizer = sr.Recognizer()
audio = sr.AudioFile(file)
with audio as source:
speech = recognizer.record(source)
try:
# Call recognizer with audio and language
text = recognizer.recognize_google(speech, language='pt-BR')
print("Você disse: " + text)
return text
# If recognizer don't understand
except:
print("Não entendi")
def mp4_to_wav(file):
audio = AudioSegment.from_file(file, format="mp4")
audio.export("audio.wav", format="wav")
return audio
def mp4_to_wav_mem(file):
audio = AudioSegment.from_file_using_temporary_files(file, 'mp4')
file = io.BytesIO()
file = audio.export(file, format="wav")
file.seek(0)
return file
url = ''
r = requests.get(url, stream=True)
file = io.BytesIO(r.content)
file = mp4_to_wav_mem(file)
speech_to_text(file)
Note that I wrote two functions: mp4_to_wav and mp4_to_wav_mem. The only difference is mp4_to_wav_mem handle all files in memory and mp4_to_wav generates .wav file.

I read the docs of MoviePy and found that the parameter nbyte should be consistent with codec. nbyte is for the Sample width (set to 2 for 16-bit sound, 4 for 32-bit sound). Hence, it better set nbyte=4, when you set codec=pcm_s32le.

i think this is the right method:
import os
from moviepy.editor import AudioFileClip
PATH= "files/"
fileName = "nameOfYourFile.mp4"
newFileName = "nameOfTheNewFile"
Ext = "wav"
AudioFileClip(os.path.join(PATH, f"{fileName}")).write_audiofile(os.path.join(PATH, f"{newFileName}.{Ext}"))

I think this approach is very easy to understand.
from moviepy.editor import *
input_file = "../Database/myvoice.mp4"
output_file = "../Database/myvoice.wav"
sound = AudioFileClip(input_file)
sound.write_audiofile(output_file, 44100, 2, 2000,"pcm_s32le")

IPython.display: Align Image with Text

I'm trying to display a base64 encoded image aligned to some HTML text in Jupyter notebooks. The code below is an initial attempt - the image (downloaded from https://stackoverflow.com/company/logos) is show belown the text.
import base64
from io import BytesIO
from IPython.display import Image
from IPython.core.display import display, HTML
with open(r'H:\se-icon.png', 'rb') as f:
data = BytesIO(f.read())
dataEncoded = base64.b64encode(data.getvalue())
display(HTML('<h3>Some Text</h3>'))
display(Image(dataEncoded, format='png', width=100))
The desired output is something like below. Ideally I could use just HTML code, with the img tag, however this does not work for my particular case, as the output of the notebook is emailed on. Use of the img tag does not guarantee that the image is displayed.
Is there any way to display the image aligned to the HTML text?

Python: editing a video file using opencv (and optionally moviepy, ipython)?

I need to read in a video using opencv, do some processing to it and export it. The code i have written is below. Right now the process is converting video to grayscale.
import cv2
# Define the codec and create VideoWriter object
fourcc = cv2.VideoWriter_fourcc(*'XVID')
out = cv2.VideoWriter('output.avi',fourcc, 25.0, (960,540))
cap = cv2.VideoCapture('input.mp4')
while(cap.isOpened()):
ret, frame = cap.read()
if ret == True:
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
out.write(gray)
else:
break
cap.release()
out.release()
The input file 'input.mp4' has these details. I am new to this so i don't know which attributes are necessarily needed, so i will post most of them. Right now the video is only writing 5.55 kb, and doesn't play the file if you click on it, says "Windows Media Player encountered a problem while playing this file".
File
Item type: MP4 Video
Size: 2.51 MB
Video
Length: 00:00:08
Frame width: 960
Frame height: 540
Data rate: 2634kbps
Total bitrate: 2637kbps
Frame rate: 25 frames/second
Audio
Bit rate: 2kbps
Channels: 2(stereo)
Audio sample rate: 48 kHz
Here is the more specific code of what i'm supposed to be doing, but i'm not sure if this is hacked up/pseudocode so i need to understand what i'm doing before i move to this step. If you understand on how to make this work, please clarify on this code as well. I just need to find a way to open, edit and save a working video file, it seems simple but a week has already passed...
# Import everything needed to edit/save/watch video clips
from moviepy.editor import VideoFileClip
from IPython.display import HTML
white_output = 'white.mp4'
clip1 = VideoFileClip("solidWhiteRight.mp4")
white_clip = clip1.fl_image(process_image)
%time white_clip.write_videofile(white_output, audio=False)
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format(white_output))