Develop Reference

grep bash command into Python script

There is a bash command, I am trying to convert the logic into python. But I don't know what to do, I need some help with that.
bash command is this :
ls
ls *
TODAY=`date +%Y-%m-%d`
cd xx/xx/autotests/
grep -R "TEST_F(" | sed s/"TEST_F("/"#"/g | cut -f2 -d "#" | while read LINE
The logic is inside a directory, reads the filename one by one, includes all sub-folders, then lists the file matches. Any help here will be much appreciated
I tried something like follows, but it is not what I would like to have. And there are some subfolders inside, which the code is not reading the file inside them
import fnmatch
import os
from datetime import datetime
time = datetime.now()
dir_path = "/xx/xx/autotests"
dirs = os.listdir(dir_path)
TODAY = time.strftime("%Y-%m-%d")
filesOfDirectory = os.listdir(dir_path)
print(filesOfDirectory)
pattern = "TEST_F("
for file in filesOfDirectory:
if fnmatch.fnmatch(file, pattern):
print(file)

Use os.walk() to scan the directory recursively.
Open each file, loop through the lines of the file looking for lines with "TEST_F(". Then extract the part of the line after that (that's what sed and cut are doing).
for root, dirs, files in os.walk(dir_path):
for file in files:
with open(os.path.join(root, file)) as f:
for line in f:
if "TEST_F(" in line:
data = line.split("TEST_F(")[1]
print(data)

Does any one has a python code to convert all mp4 files in folder to mp3?

For instance, i have five .mp4 in a folder. How do i loop and convert them to .mp3 and output them in a different folder?
Thanks
I have tried this but it is not working.
**import moviepy.editor as mp
import re
tgt_folder = "D:/Music/video"
for file in [n for n in os.listdir(tgt_folder) if re.search('mp4',n)]:
full_path = os.path.join(tgt_folder, file)
output_path = os.path.join(tgt_folder, os.path.splitext(file)[0] + '.mp3')
clip = mp.AudioFileClip(full_path).subclip(10,) # disable if do not want any clipping
clip.write_audiofile('D:/Music/mp3')**

you can do this by using ffmpeg and python subprocces
import subprocess
subprocess.run(["ffmpeg", "-i video.mp4 -b:a 192K -vn music.mp3"])

Uncompyle6 convert pyc to py file python 3 (Whole directory)

I have 200 pyc files I need to convert in a folder. I am aware of converting pyc to py files through uncompyle6 -o . 31.pyc however as I have so many pyc files, this would take a long period of time. I've founds lots of documentation but not much in bulk converting to py files. uncompyle6 -o . *.pyc was not supported.
Any idea on how I can achieve this?

Might not be perfect but it worked great for me.
import os
import uncompyle6
your_directory = ''
for dirpath, b, filenames in os.walk(your_directory):
for filename in filenames:
if not filename.endswith('.pyc'):
continue
filepath = dirpath + '/' + filename
original_filename = filename.split('.')[0]
original_filepath = dirpath + '/' + original_filename + '.py'
with open(original_filepath, 'w') as f:
uncompyle6.decompile_file(filepath, f)

This is natively supported by uncompyle6
uncompyle6 -ro <output_directory> <python_directory>
-r tells the tool to recurse into sub directories.
-o tells the tool to output to the given directory.

In operating systems with shell filename expansion, you might be able to use the shell's file expansion ability. For example:
uncompyle6 -o /tmp/unc6 myfiles/*.pyc
If you need something fancier or more control, you could always write some code that does the fancier expansion. Here is the above done in POSIX shell filtering out the single file myfiles/huge.pyc:
cd myfiles
for pyc in *.pyc; do
if [[ $pyc != huge.pyc ]] ; then
uncompyle -o /tmp/unc $pyc
fi
done
Note: It seems this question was also asked in Issue on output directory while executing commands with windows batch command "FOR /R"

thank you for the code, extending it to recursively call, nested sub directories, save as uncompile.py, in the directory to be converted, to run in command prompt type "python uncomple.py" would convert pyc to py in current working directory, with error handling and if rerun skips (recovery) files checking existing py extension match
import os
import uncompyle6
#Use current working directory
your_directory = os.getcwd()
#function processing current dir
def uncompilepath(mydir):
for dirpath, b, filenames in os.walk(mydir):
for d in b:
folderpath = dirpath + '/' + d
print(folderpath)
#recursive sub dir call
uncompilepath(folderpath)
for filename in filenames:
if not filename.endswith('.pyc'):
continue
filepath = dirpath + '/' + filename
original_filename = filename.split('.')[0]
original_filepath = dirpath + '/' + original_filename + '.py'
#ignore if already uncompiled
if os.path.exists(original_filepath):
continue
with open(original_filepath, 'w') as f:
print(filepath)
#error handling
try:
uncompyle6.decompile_file(filepath, f)
except Exception:
print("Error")
uncompilepath(your_directory)

ffmpeg in python script

I would like to run the following command in a python script, I also want to make it loop over several videos in a folder. This is the command I want to run.
ffmpeg -i mymovie.avi -f image2 -vf fps=fps=1 output%d.png
I want to fit it in something like this:
import ffmpy
import os
path = './Videos/MyVideos/'
for filename in os.listdir(path):
name = filename.replace('.avi','')
os.mkdir(os.path.join(path,name))
*ffmpeg command here*
I found a wrapper for ffmpeg called ffmpy, could this be a solution?

From a brief look at FFMPY, you could do this using ffmpy.FFmpeg, as that allows any and all FFMPEG command line options, including -f. -- Click the link for documentation.
You could do the FFMPEG command with os.system. You'll need to import OS anyway to iterate through the files.
You would need to iterate through all the files in a directory though. This would be the more challenging bit, it's quite easy with a for loop though.
for filename in os.listdir(path):
if (filename.endswith(".mp4")): #or .avi, .mpeg, whatever.
os.system("ffmpeg -i {0} -f image2 -vf fps=fps=1 output%d.png".format(filename))
else:
continue
The above code iterates through the directory at path, and uses command prompt to execute your given FFMPEG command, using the filename (if it's a video file) in place of mymovie.avi

Dont have reputation to comment, hence adding another response.
Another version of ocelot's answer with the more readable f-string syntax of python -
for filename in os.listdir(path):
if (filename.endswith(".mp4")): #or .avi, .mpeg, whatever.
os.system(f'ffmpeg -i {filename} -f image2 -vf fps=fps=1 output%d.png')
else:
continue

Try pydemux in https://github.com/Tee0125/pydemux. Pydemux module can extract video frames as in Pillow Image format
from PyDemux import Video
v = Video.open('video.mov')
i = 0
while True:
im = v.get_frame()
if im is None:
break
im.save('output%d.png'%i)
i = i + 1

This is a way to use ffmpeg in a python script, e.g. for extracting the last 10 seconds of every video:
ffmpeg -sseof -10 -i input.mp4 output.mp4
To apply this to a whole folder of mp4 files:
from pathlib import Path
import os
suffix = ".mp4"
input_path= Path.home() / "Desktop/foo"
file_paths= [subp for subp in input_path.rglob('*') if suffix == subp.suffix]
file_paths.sort()
output_path = Path.home() / "Desktop/foo/new"
output_path.mkdir(parents=True, exist_ok=True)
for file_p in file_paths:
input = str(file_p)
output = str( output_path / file_p.name )
command = f"ffmpeg -sseof -10 -i {input} {output}"
print(command)
os.system(command)

Faster way to Unzip jsons files with python

I have a program that downloads directories and then unzip these directories and finally decompress all json's files inside each directory.
I have to download 1260 directories and each directory has 1000 files with 300 MB.
So looks like this:
di1.zip
|_file1.json.gz
...
|_file1000.json.gz
di2.zip
|_file1.json.gz
...
|_file1000.json.gz
....
dir1260.zip
|_file1.json.gz
...
|_file1000.json.gz
This is my code:
def ProcesssDir(dirs_links_file):
with open(dirs_links_file, 'r') as inputFile:
lines = inputFile.readlines()
for line in lines:
#Download
directory = subprocess.Popen("wget -c " + line, shell=True).wait()
#Unzip:
for nameDirZip in glob.glob('*.zip'):
UnzipDir = zipfile.ZipFile(nameDirZip)
UnzipDir.extractall()
nameDir = nameDirZip[:-4] + "/" #This is just to get the name of the new dir.
subprocess.Popen("gunzip -d " + nameDir + "*.gz", shell=True).wait()
This works but very very ..very slow. Took like 20 min for each directory.
How can i do this faster?