Develop Reference

Automated Concatenate with FFMPEG Using Re-Encode

I would like to concaternate a fairly large list of videos using FFMPEG.
The following answer from another question is highly relevant:
https://stackoverflow.com/a/11175851/5065462.
I explain below why it isn't sufficient for me.
I'm using MP4 files.
I am using Windows 10. I'd like to write a bat file with arguments. I'll then call this from either Command Prompt or PowerShell.
I want to automate Option 1 https://stackoverflow.com/a/11175851/5065462 to take as input a txt (or similar) file containing the filepaths for the videos to be concatenated.
I'm happy with all the default [#:v] [#:a] options.
An alternative option is just to write a small program, either in Command Prompt or python3 is fine, which outputs a text string that I just copy+paste into cmd/PS.
Unfortunately, I'm not sure how to use python to get filenames.
Option 2 in https://stackoverflow.com/a/11175851/5065462 looks great. Unfortunatley, the stream-encoding has issues with my mp4 files. I found they are fixed by using Option 1 in the linked answer. However, I don't want to type every filename each time.

The following python script will generate the command described in option 1.
To run, use python3 script_file.py video_directory output.mkv.
video_directory should contain only video files.
import argparse
import os
parser = argparse.ArgumentParser()
parser.add_argument("dir", help="the directory to the videos")
parser.add_argument("output", help="the name of the output file")
args = parser.parse_args()
files = os.listdir(args.dir)
cmd = "ffmpeg "
for file in files:
cmd += f"-i {file} "
cmd += '-filter_complex "'
for i in range(len(files)):
cmd += f"[{i}:v] [{i}:a] "
cmd += f'concat=n={len(files)}:v=1:a=1 [v] [a]" '
cmd += f'-map "[v]" -map "[a]" {args.output}'
print(cmd)
NB: replacing the final line by os.system(cmd) will run the command directly from python.

How do i make my awk command into a python command

I have an awk command that works in bash, but im now trying to put it into a python script
I have tried both os.system, and subprocess.call both return the same error. sh: 1: Syntax error: "(" unexpected
os.system('awk \'FNR<=27{print;next} ++count%10==0{print;count}\' \'{0} > {1}\'.format(inputfile, outpufile)')
So this awk command will take the large inputfile and create an output file that leaves the first 27 lines of header, but then starting on line 28 it only takes every 10th line and puts it into the output file
Im using the .format() because it is within a python script where the input file will be different every times its run.
ive also tried
subprocess.call('awk \'FNR<=27{print;next} ++count%10==0{print;count}\' \'{0} > {1}\'.format(inputfile, outpufile)')
both come up with the same error above. What am I missing?

As per the comment above, probably more pythonic (and more manageable) to directly use python.
But, if you want to use awk then one way is to format your command with your variable filenames separately.
This works using a basic test text file:
import os
def awk_runner(inputfile, outputfile):
cmd = "awk 'FNR<=27{print;next} ++count%10==0{print;count}' " + inputfile + " > " + outputfile
os.system(cmd)
awk_runner('test1.txt', 'testout1.txt')

There are two main issues with your Python code:
format() is a python method call, it should not be put into the string of awk_cmd to execute under the shell
when calling format() method, braces {} are used to identify substitution target in the format string objects, they need to be escaped using {{ ... }}
See below a modified version of your code:
awk_cmd = "awk 'FNR<=7{{print;next}} ++count%10==0{{print;count}}' {0} > {1}".format(inputfile, outpufile)
os.system(awk_cmd)

python subprocess.popen redirect to create a file

I've been searching for how to do this without any success. I've inherited a python script for performing an hourly backup on our database. The original script is too slow, so I'm trying a faster utility. My new command would look like this if typed into a shell:
pg_basebackup -h 127.0.0.1 -F t -X f -c fast -z -D - --username=postgres > db.backup.tgz
The problem is that the original script uses call(cmd) and it fails if the above is the cmd string. I've been looking for how to modify this to use popen but cannot find any examples where a file create redirect is used as in
>. The pg_basebackup as shown will output to stdout. The only way I've succeeded so far is to change -D - to -D some.file.tgz and then move the file to the archive, but I'd rather do this in one step.
Any ideas?
Jay

May be like this ?
with open("db.backup.tgz","a") as stdout:
p = subprocess.Popen(cmd_without_redirector, stdout=stdout, stderr=stdout, shell=True)
p.wait()

Hmmm... The pg_basebackup executable must be able to attach to that file. If I open the file in the manner you suggest, I don't know the correct syntax in python to be able to do that. If I try putting either " > " or " >> " in the string to call with cmd(), python pukes on it. That's my real problem that I'm not finding any guidance on.

Parsing inline python command via linux using pdsh

So, I am trying to issue this command from a python script that collects cpu information across a predetermined number of nodes in a cluster. Here I use a fanout of 2 and only run it on nodes b127 through b129 for testing purposes.
pdsh -f2 -w b[127-129] 'python -c "import multiprocessing
num_cpu = multiprocessing.cpu_count()
stat_fd = open('/proc/stat')
stat_fd.close()"'
I printed the command and this is what it shows on the terminal. Thus, telling me that the quotes and commands are properly formatted. I get this string by executing the following code:
python_command = "'python -c "\
+ "\"import multiprocessing\n"\
+ "num_cpu = multiprocessing.cpu_count()\n"\
+ "stat_fd = open(\'/proc/stat\')\n"\
+ "stat_fd.close()\"'"
command = "pdsh -f2 -w b[127-129] " + python_command
print command
Unfortunately, the line with open(\'/proc/stat\') seems to be the problem as that is the only line that causes the parser to fail due to the nested single quotes. I've tried numerous combinations of quoting and escaping in order to make it work to no avail. I've omitted some code between the opening and closing of the file to minimize the posted code.
I searched around and found this link, but found it was too simple of an example because I could replicate those commands. And yes, I know I can use bash commands to get what I want done and I may end up doing so, but this one has me beating my head on the wall. I also have scripts that gather data using top and ps so I don't need an explanation using them. I greatly appreciate the help in advance.

Try this:
python_command = """pdsh -f2 -w b[127-129] 'python -c "import multiprocessing
num_cpu = multiprocessing.cpu_count()
stat_fd = open(\\"/proc/stat\\")
stat_fd.close()"'"""
In Python, you can use triple quotes ("""...""" or '''...''') for strings containing new lines and single/double quotes.
The last level of quotes (on the open() line) will need to be escaped so that they don't conflict with outer quotes. You also need to escape the backslashes so they aren't immediately consumed by Python when interpreting the string: \\".

Taking the results of a bash command and using it in python

I am trying to write a code in python that will take some information from top and put it into a file.
I want to just write the name of the application and generate the file. The problem i am having is that i can't get the output of the pidof command so i can use it in python. My code looks like this :
import os
a = input('Name of the application')
val=os.system('pidof ' + str(a))
os.system('top -d 30 | grep' + str(val) + '> test.txt')
os.system('awk '{print $10, $11}' test.txt > test2.txt')
The problem is that val always has 0 but the command is returning the pid i want. Any input would be great.

First up, the use of input() is discouraged as it expects the user to type in valid Python expressions. Use raw_input() instead:
app = raw_input('Name of the application: ')
Next up, the return value from system('pidof') isn't the PID, it's the exit code from the pidof command, i.e. zero on success, non-zero on failure. You want to capture the output of pidof.
import subprocess
# Python 2.7 only
pid = int(subprocess.check_output(['pidof', app]))
# Python 2.4+
pid = int(subprocess.Popen(['pidof', app], stdout=subprocess.PIPE).communicate()[0])
# Older (deprecated)
pid = int(os.popen('pidof ' + app).read())
The next line is missing a space after the grep and would have resulted in a command like grep1234. Using the string formatting operator % will make this a little easier to spot:
os.system('top -d 30 | grep %d > test.txt' % (pid))
The third line is badly quoted and should have caused a syntax error. Watch out for the single quotes inside of single quotes.
os.system("awk '{print $10, $11}' test.txt > test2.txt")

Instead of os.system, I recommend you to use the subprocess module: http://docs.python.org/library/subprocess.html#module-subprocess
With that module, you can communicate (input and output) with a shell. The documentation explains the details of how to use it.
Hope this helps!