Develop Reference

How to run this tsduck shell command containing quotes with subprocess.run in Python

Here is the command as written in the tsduck manual:
tsp -I dvb -a 1 #ts1028.txt \
-P svremove -s AlJazeeraEnglish \
-P merge "tsp -I dvb -a 0 #ts1022.txt -P zap TV5MondeEurope" \
-P analyze -i 30 -o merged.txt \
-O dektec #modulation.txt
Here is my version:
import sys
import subprocess
mod_values = { "bandwidth": "8-mhz",
"convolutional_rate": "7/8",
"frequency": "578000000",
"guard_interval": "1/4",
"dmb_constellation": "64-QAM",
"modulation": "DVB-T"}
tsterinfo_rate = subprocess.run(['tsterinfo',
"-h", mod_values["convolutional_rate"],
"-g", mod_values["guard_interval"],
"-c", mod_values["dmb_constellation"],
"-s"], stdout=subprocess.PIPE, universal_newlines=True)
mod_values["dvb_bitrate"] = tsterinfo_rate.stdout
infile=sys.argv[1]
run_tsp = subprocess.run(['tsp',
'--verbose',
'-b', mod_values["dvb_bitrate"],
'-I', 'null',
'-P', 'merge',
f'"tsp -I File {infile} --infinite"',
'-P', 'pcrbitrate',
'-P', 'regulate',
'-O', 'dektec',
'--frequency', mod_values["frequency"],
'--modulation', mod_values["modulation"],
'--guard-interval', mod_values["guard_interval"],
'--convolutional-rate', mod_values["convolutional_rate"],
'--dmb-constellation', mod_values["dmb_constellation"],
'-s'])
The quoted part in the command returns this error if I try keeping it as full string with spaces in double quotes surround my single quotes:
/bin/sh: 1: tsp -I File ../Videos/myts.ts --infinite: not found
without the quotes at all it errors saying too many inputs the same as it would straight into the terminal without quotes
python 3.8.5, ubuntu 20.04

I found a few things wrong with my tsp command when working through this. The answer to the question about passing quotes through to the sub-process seems to be solved by using
shell=True
In the sub-process options. Then you can pass the command line argument as one big string rather than as a list.
My final script for taking a transport stream as an argument and creating a CBR output ready for modulating through Dektec DTU-215 is this:
import sys
import subprocess
import json
# set modulation parameters in dict in order to reference once for bitrate calc and use again for modulator setup
mod_values = { "bandwidth": "8-mhz",
"convolutional_rate": "7/8",
"frequency": "578000000",
"guard_interval": "1/4",
"dmb_constellation": "64-QAM",
"modulation": "DVB-T"}
# calculate modulated bitrate and add to dict
tsterinfo_rate = subprocess.run(['tsterinfo',
"-h", mod_values["convolutional_rate"],
"-g", mod_values["guard_interval"],
"-c", mod_values["dmb_constellation"],
"-s"], stdout=subprocess.PIPE, universal_newlines=True)
mod_values["dvb_bitrate"] = tsterinfo_rate.stdout.strip()
# first argument is input file transport stream
infile=sys.argv[1]
# use mediainfo to calculate bitrate of input ts (must be CBR)
infile_mediainfo = subprocess.run(["mediainfo",
"--Output=JSON",
infile],
capture_output=True)
print(infile_mediainfo)
media_data = json.loads(infile_mediainfo.stdout)
ts_bitrate = int(media_data["media"]["track"][0]["OverallBitRate"])
print(f'ts_bitrate is: {ts_bitrate}')
# without -t option we don't have a PAT to even merge our stream with
# packet burst seems to make big difference to how smooth final playback is, default (16 according to docs) was jerky but 14 seems smooth
run_tsp = subprocess.run(f'tsp \
--verbose \
-d \
-b {mod_values["dvb_bitrate"]} \
-I null \
-P regulate --packet-burst 14 \
-P merge \
-t \
"tsp -I file {infile} \
--infinite \
-P regulate -b {ts_bitrate}" \
-O dektec \
-f {mod_values["frequency"]} \
-m {mod_values["modulation"]} \
-g {mod_values["guard_interval"]} \
-r {mod_values["convolutional_rate"]} \
--dmb-constellation {mod_values["dmb_constellation"]} \
-s', shell=True)

Python: Using FFMPEG to create a video using a series of images

I am fairly new to Python and this is my first time using ffmpeg (basing this code on a colleagues).
I have some code that takes a snapshot of the screen every X seconds and then creates a video out of these files however I am having issues when creating the video. This is my code for generating the video:
def create_video(screen_resolution, image_location, subfolder, count, preset, qaulity, duplicated_frame, video_path, video_filename):
video_path = path + video_path
{1}\\%d.jpg -vcodec libx264 -preset {2} -crf {3} -r {4} -pix_fmt yuv420p {5}\\{6}.mp4"
proc = subprocess.Popen("C:\\ffmpeg\\ffmpeg.exe -r 1 -f image2 -s %s \
-pattern_type sequence \
-start_number 1 \
-i %s%s\\%s%d.png \
-vcodec libx264 \
-preset %s \
-crf %s \
-r %s \
-pix_fmt yuv420p \
%s\\%s.mp4" %
(screen_resolution, image_location, subfolder, count, preset[2], qaulity, duplicated_frame, video_path, video_filename), shell = True)
and the error I am getting is:
TypeError: %d format: a number is required, not str
I have tried searching for a solution but not getting anywhere fast. I know the %d is looking for an input from the list at the end but from what I can see this is also the way to tell the FFMPEG to use all files in the image location.
Hopefully someone can help with this, apologies if this is a duplicate question but as mentioned, I have tried searching, a lot.

Don't try to manually construct shell strings. Let subprocess do that for you.
You can pass an array of command line arguments, and Popen will take care of properly escaping everything:
proc = subprocess.Popen(
[
"C:/ffmpeg/ffmpeg.exe",
"-r", "1",
"-f", "image2",
"-s", screen_resolution,
"-i", os.path.join(path, image_location, subfolder, "%d" + extension),
"-preset", preset[1],
"-crf", str(qaulity),
"-r", str(duplicated_frame),
"-pix_fmt", "yuv420p",
os.path.join(video_path, video_filename)
]
)

Python. How to pass parameters to Subprocess.Popen in a wrapper function (Linux)

I have a working BASH script with the process but I would like to migrate it to Python for flexibility, power and learning.
So In short I would like to write a function to wrap the Subprocess.open() function, but for different cases:
The main process to launch is the 'ffmpeg' programme.
The final objective it's to get to lunch lines like this:
ffmpeg -y -ss 0 -t 5 -i "MP3 sourceName" -codec:a copy "MP3 Output filename"
The params to pass are changing, following the audio process chain.
Till now i stuck in the following code:
#The 'Wrapping' function:
def proces(*args):
arg = ''.join(args)
print ("DEBUG- ffmpeg ",arg)
#these lines are for debug purpose only
try :
p = subprocess.Popen(["ffmpeg", arg], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
response, err = p.communicate()
return response, err
except:
pass
#Here the initially call in the main programme
def A_Process (files):
proces( " -y -ss 0 -t 3 -i ","\"%s\" "%(files), "%s"%(Param['ACOPY']), " \"%s\""%(fCH) ), " \"%s\""%(fCH) ] )
#-y param to indicate yes to overwrite
#-ss = param to indicate start time: decimal
#-t = param to indicate time length: decimal
#files is the MP3 sourceName as a string (pure string so i need the "")
#Param['ACOPY']= "copy" string i think?
#fCH = "Output MP3 filename" also string
#### ... more irrelevant code....
#### Launching the code gives me a print:
DEBUG- ffmpeg -y -ss 0 -t 3 -i "/media/temps/scp/fs/1.mp3" -codec:a copy "1_output.mp3"
##and stderr gives:
b'ffmpeg version 3.3.3-static http://johnvansickle.com/ffmpeg/ Copyright (c)(...etc...)...
... \n[NULL # 0x32861c0] Unable to find a suitable output
format for \' -y -ss 0 -t 3 -i "/media/temps/scp/fs/1.mp3" -codec:a copy "1_output.mp3"\'\n
-y -ss 0 -t 3 -i "/media/temps/scp/fs/1.mp3" -codec:a copy ""1_output.mp3": **Invalid argument**\n'
When i copy the DEBUG- line printed in the output and launch in Terminal, i get no errors.
I think the problem comes from the true params (-y -ss -t ) are been passed as a string but i don't know how to pass the params and its values
So someone have any ideas?

error "sh: 1: read: Illegal option -d" when using here-document in system command run from Python

I'm running a Bash system command from within Python and have encountered this problem when using read to define a here-document in the command:
import os
text = "~|||-:this is text:-|||~"
fileName = "sound.wav"
command =\
"IFS= read -d \'\' text <<EOF\n" +\
text + "\n" +\
"EOF\n" +\
"echo \"${text}\" | text2wave -scale 1 -o " + fileName
os.system(command)
Could you help me figure out how to fix this?
Here is a slightly simplified version:
import os
text = "~|||-:this is text:-|||~"
command =\
"IFS= read -d \'\' text <<EOF\n" +\
text + "\n" +\
"EOF\n" +\
"echo \"${text}\""
os.system(command)
I wanted to make clear in the one above that I'd be using pipes. When I run this, I get the following error:
sh: 1: read: Illegal option -d

There's no reason to do all that in shell. Python can write directly to the standard input of the process that will run text2wave. Here's an example using the subprocess module.
p = subprocess.Popen(["text2wave", "-scale", "1", "-o", filename], stdin=subprocess.PIPE)
p.stdin.write(text + "\n")
p.wait()

A more pythonic way to do this would be using subprocess, either by specifying stdin to check_call or by using Popen.communicate.
Also, -d might not be a valid option in your version of read. What does help [r]ead tell you?

bash commands from within python

I'm looking for the best way to use bash commands from within python. What ways are there? I know of os.system and subprocess.Popen.
I have tried these:
bootfile = os.system("ls -l /jffs2/a.bin | cut -d '/' -f 4")
print bootfile
This returns a.bin as expected but also it retuns 0 afterwards and so prints:
a.bin
0
with bootfile now being set to 0. The next time I print bootfile it just shows up as 0. Which is the exit value I guess, how do i stop this value interfering?
I have also tried:
bootfile = subprocess.Popen("ls -l /jffs2/a.bin | cut -d '/' -f 4")
print bootfile
but it seems to break the script, as in I get nothing returned at all, have I done that right?
Also which of these is better and why? Are there other ways and what is the preferred way?

Using os.readlink (proposed by #kojiro) and os.path.basename for getting only the namefile:
os.path.basename(os.readlink('/jffs2/a.bin'))

kojiro's comment about os.readlink is probably what you want.
I am explaining what you were trying to implement.
os.system would return you exit status of the command run.
subprocess.Popen will create a pipe, so that you can capture the output of the command run.
Below line will capture output of the command run:
bootfile = subprocess.Popen(["bash","-c","ls -l /jffs2/a.bin | cut -d '/' -f 4"], stdout=subprocess.PIPE).communicate()[0]
More details at http://docs.python.org/library/subprocess.html

The right answer, as #kojiro says, is:
os.readlink('/jffs2/a.bin')
But if you really wanted to do this the complicated way, then in Python 2.7:
cmd = "ls -l /jffs2/a.bin | cut -d '/' -f 4"
bootfile = subprocess.check_output(cmd, shell=True)
Or on older Pythons:
cmd = "ls -l /jffs2/a.bin | cut -d '/' -f 4"
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
bootfile = p.communicate()[0]
if p.returncode != 0:
raise Exception('It failed')