Develop Reference

Issues in python based ffmpeg operation

I am trying to write a script that would automatically take duration values and chop an audio file into smaller splits. To do this I have saved all my start times and durations in a list. And this is the code I am trying to run.
for k in range(0,len(start_time)):
s=start_time[k]
e=Duration[k]
filename = "output%d.mp3" % (k)
!ffmpeg -i Audio.mp3 -ss s -t e -acodec copy filename
k=k+1
On running this I get the following error
Invalid duration specification for ss: s
I suspect that this error arises because of the fact that since I am using lists to call elements, the time stamp comes with quotes on both sides. Secondly, I am not sure how to specify the filename, such that each created split has the name of this format Output_1.mp3 and so on. The integer will be an identifier of the split. What could be possible fixes for this piece of code? Please note that I am running this on Google Colab.

Google Colab is running Jupyter notebook powered by IPython in its cloud, IPython uses special syntax for shell invocation (commands starting from ! exclamation mark), i.e. they are executed in (temporary) shell session. In case of Google Colab it's bash:
res = !echo $SHELL
print(res)
> ['/bin/bash']
As I checked, ffmpeg is indeed available on Google Colab:
res = !which ffmpeg
print(res)
> ['/usr/bin/ffmpeg']
, so you have legitimate error message printed by ffmpeg: "Invalid duration specification ... " (I mean not by shell or python) and it just means that variable you're passing is not substituted with its value the way you do. And it's so because this above mentioned special syntax is not followed for passing variable to the shell, check this; variables should be curly-braced to be passed:
!ffmpeg -i Audio.mp3 -ss {s} -t {e} -acodec copy {filename}

How to get data from web in python using curl?

In bash when I used
myscript.sh
file="/tmp/vipin/kk.txt"
curl -L "myabcurlx=10&id-11.com" > $file
cat $file
./myscript.sh gives me below output
1,2,33abc
2,54fdd,fddg3
3,fffff,gfr54
When I tried to fetch it using python and tried below code -
mypython.py
command = curl + ' -L ' + 'myabcurlx=10&id-11.com'
output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE).stdout.read().decode('ascii')
print(output)
python mypython.py throw error, Can you please point out what is wrong with my code.
Error :
/bin/sh: line 1: &id=11: command not found
Wrong Parameter

command = curl + ' -L ' + 'myabcurlx=10&id-11.com'
Print out what this string is, or just think about it. Assuming that curl is the string 'curl' or '/usr/bin/curl' or something, you get:
curl -L myabcurlx=10&id-11.com
That’s obviously not the same thing you typed at the shell. Most importantly, that last argument is not quoted, and it has a & in the middle of it, which means that what you’re actually asking it to do is to run curl in the background and then run some other program that doesn’t exist, as if you’d done this:
curl -L myabcurlx=10 &
id-11.com
Obviously you could manually include quotes in the string:
command = curl + ' -L ' + '"myabcurlx=10&id-11.com"'
… but that won’t work if the string is, say, a variable rather than a literal in your source—especially if that variable might have quote characters within it.
The shlex module has helpers to quoting things properly.
But the easiest thing to do is just not try to build a command line in the first place. You aren’t using any shell features here, so why add the extra headaches, performance costs, problems with the shell getting in the way of your output and retcode, and possible security issues for no benefit?
Make the arguments a list rather than a string:
command = [curl, '-L', 'myabcurlx=10&id-11.com']
… and leave off the shell=True
And it just works. No need to get spaces and quotes and escapes right.
Well, it still won’t work, because Popen doesn’t return output, it’s a constructor for a Popen object. But that’s a whole separate problem—which should be easy to solve if you read the docs.
But for this case, an even better solution is to use the Python bindings to libcurl instead of calling the command-line tool. Or, even better, since you’re not using any of the complicated features of curl in the first place, just use requests to make the same request. Either way, you get a response object as a Python object with useful attributes like text and headers and request.headers that you can’t get from a command line tool except by parsing its output as a giant string.

import subprocess
fileName="/tmp/vipin/kk.txt"
with open(fileName,"w") as f:
subprocess.read(["curl","-L","myabcurlx=10&id-11.com"],stdout=f)
print(fileName)
recommended approaches:
https://docs.python.org/3.7/library/urllib.request.html#examples
http://docs.python-requests.org/en/master/user/install/

Python script doesn't delete file from archive -- printed command via terminal works fine

I'm creating an archive in Python using this code:
#Creates archive using string like [proxy_16-08-15_08.57.07.tar]
proxyArchiveLabel = 'proxy_%s' % EXECUTION_START_TIME + '.tar'
log.info('Packaging %s ...' % proxyArchiveLabel)
#Removes .tar from label during creation
shutil.make_archive(proxyArchiveLabel.rsplit('.',1)[0], 'tar', verbose=True)
So this creates an archive fine in the local directory. The problem is, there's a specific directory in my archive I want to remove, due to it's size and lack of necessity for this task.
ExecWithLogging('tar -vf %s --delete ./roles/jobs/*' % proxyArchiveLabel)
# ------------
def ExecWithLogging(cmd):
print cmd
p = subprocess.Popen(cmd.split(' '), env=os.environ, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while(True):
log.info(p.stdout.readline().strip())
if(p.poll() is not None):
break
However, this seems to do basically nothing. The size remains the same. If I print cmd inside of the ExecWithLogging, and copy/past that command to a terminal in the working directory of the script, it works fine. Just to be sure, I also tried hard-coding the full path to where the archive is created as part of the tar -vf %s --delete command, but still nothing seemed to happen.
I do get this output in my INFO log: tar: Pattern matching characters used in file names, so I'm kind of thinking Popen is interpreting my command incorrectly somehow... (or rather, I'm more likely passing in something incorrectly).
Am I doing something wrong? What else can I try?

You may have to use the --wildcards option in the tar command, which enables pattern matching. This may well be what you are seeing in your log, be it somewhat cryptically.
Edit:
In answer to your question Why? I suspect that the shell is performing the wildcard expansion whilst the command proffered through Popen is not. The --wildcard option for tar, forces tar to perform the wildcard expansion.
For a more detailed explanation see here:
Tar and wildcards

cupsfilter Does Not Work With Python subprocess.call()

After reading this question, I've been experimenting and using subprocess.call() in Python to run a command as if from the terminal. I've found it works quite well and I prefer it over os.system. I've run into a problem with cupsfilter, though. I'm trying to run this command on OS X, from Python:
cupsfilter -o cpi=10 PDFTestIn.txt 2>PDFTestError.txt >PDFTestOut.pdf
(I tried it, originally, with more than one option, but am limiting it to one option until I can get it working.)
I can run the command fine from the command line and from os.system, but I'm having trouble with this command in subprocess.call(). I've been experimenting in IDLE and cupsfilter does not work with this as other programs do. If I provide options in the form:
-o optionname=optionvalue
I always get error messages. I tried this:
import subprocess
import os
cmd = '/usr/sbin/cupsfilter'
args = ['-o cpi=10']
ifile='PDFTestIn.txt'
ofile='PDFTestOut.pdf'
efile='PDFTestError.txt'
cmdargs = [cmd] + args + [ifile]
with open(ofile, 'w') as pdfout, open(efile, 'w') as errout:
subprocess.call(cmdargs, stdout=pdfout, stderr=errout)
When I do that, I get a return value of 1 from the last command. I check PDFTestError.txt for output and it reads:
cupsfilter: Unknown option " ".
I experimented by changing the 4th line to this:
args = ['-ocpi=10']
and I get this error:
cupsfilter: Unknown option "c".
Whatever character comes after the "-o" is seen as an option, and only that one letter is acknowledged, not the word (like "cpi"). Even though there are other options I can use besides "-o," I thought I'd try it without the "-o" just in case. When I do that, I get this error:
cupsfilter: Only one filename can be specified.
(And if I use only the command as an argument in the list passed to subprocess.call(), and still specify stdout and stderr, it works okay.)
Summary: When I use "-o" to provide an option for cupsfilter, in subprocess.call(), cupsfilter looks only at the next character, not the next word. If I have a space after "-o" as I would on the command line, it expects that space to be an option. If I leave the space out, it looks at the next character and not the next word. If I leave out "-o" it sees the option as another file name (as I'd expect).
But if I use the command line, above, from a terminal, or from os.system(), there's no problem at all.
Why won't this work with subprocess.call() and is there a way to correct that?

You need to separate each arg, '-o cpi=10' -> '-o', 'cpi=10':
subprocess.call([cmd,'-o','cpi=10',infile], stdout=pdfout, stderr=errout)

Python sys.argv[1:] not picking up command line options

Update/Solution: the answer is below, from Zack. The problem was, indeed, DOS line endings on the script file itself, clenotes.cmd. Since I futzed with the various files so much, I deleted the whole directory and then re-downloaded a fresh copy from HERE. I ran Zack's perl script on the file just like so:
perl -pi.bak -e 's/[ \t\r]+$//' clenotes.cmd
I then edited the command execution just slightly so that the final script became:
CWD=`dirname $0`
JYTHON_HOME="$CWD"
LIB_DIR="$JYTHON_HOME/lib"
NOTES_HOME="/opt/ibm/lotus/notes/"
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$NOTES_HOME
java -cp "$LIB_DIR" -jar "$LIB_DIR/jython.jar" -Djython.home="$CWD/" -Dpython.path="$LIB_DIR:$CWD/ext" -Djava.library.path="$NOTES_HOME" "$LIB_DIR/clenotes/cletes/clenotes.py" "$#"
That was it -- everything else worked. No edits needed to clenotes.py or clenotes.cfg. Many thanks for sticking with the question, which I guess ended up being quite simple.
Update: I'm cutting down on some of the code to make this more readable and remove unnecessary information from the post.
I'm trying to get Lotus Notes command line to work on Linux and am having an issue with something related to sys.argv[1:] in the python file. The windows script is here:
#echo off
#setlocal
set CWD=%~dp0
set JYTHON_HOME=%CWD%
set LIB_DIR=%JYTHON_HOME%/lib
java -cp %LIB_DIR% -jar %LIB_DIR%/jython.jar -Djython.home=%CWD% -python.path=%LIB_DIR%;%CWD%/ext %LIB_DIR%/clenotes/clenotes.py %*
#endlocal
I was having a tough time with variables, so for Linux, it simply looks like this:
java -cp ./lib/ -jar ./lib/jython.jar -Djython.home=./ -Dpython.path=./lib:./ext -Djava.library.path=/opt/ibm/lotus/notes/ ./lib/clenotes/clenotes.py $*
I run it from within the directory. In any case, what puzzles me is that it's not picking up any options I pass from the command line. clenotes.cmd --help results in
No commands specified. Use --help option for usage.
Here is the section where the command line arguments are supposed to be parsed:
def main():
Output.log("Entering %s v%s" % (PROGRAM_NAME,VERSION),Output.LOGTYPE_DEBUG)
cliOptions2=[]
for opt in cliOptions:
opt2=opt.replace('--','')
opt2=opt2.replace('!','=')
cliOptions2.append(opt2)
opts=[]
args=[]
try:
opts, args = getopt.getopt(sys.argv[1:], '', cliOptions2)
I'm using Python 3.1.3 on Arch Linux 64bit in a 32bit chroot environment. Can I provide anything else?
Just in case it's needed... HERE is the whole clenotes.py file.
Also, as requested in the comments, the config file (which contains the help message and viable options/arguments, is HERE
Update
After a lot of fiddling, the best progress I have made has been to examine what it's setting as opts and args in the (main) method. Most surprising was that when passing an argument and then looking at it's parsed result using print sys.argv, the option would come up with a trailing \r in it. For example:
clenotes.cmd appointments
args is ['appointments\r']
On Windows I did the same and args was reported as ['appointments']. Furthermore, manually setting args=['appointments'] and then commenting out the section where getopt.getopt is assigning a value worked.
Lastly, I've found that when using multiple arguments, n-1 of them get interpreted and used while the nth one gets ignored. This is kind of a workaround since I can actually use the script... but obviously it's not preferred. If I want to look at today's appointments, I can execute clenotes.cmd appointments --today --today and it will work. sys.argv will spit out: ['appointments', '--today', '--today\r'].
So... what's causing the trailing \r? I'm thinking it has to do with the actual script. Note it again:
java -cp ./lib/ -jar ./lib/jython.jar -Djython.home=./ -Dpython.path=./lib:./ext -Djava.library.path=/opt/ibm/lotus/notes/ ./lib/clenotes/clenotes.py $*
So... bunch of path stuff and then the actual python file: clenotes.py $*
I got the $* from HERE
Is it picking up the carriage return??

I think your problem is that clenotes.cfg has DOS line endings, which Python is misinterpreting. Try changing this line of clenotes.py
config.readfp(open('%sconfig/clenotes.cfg' % System.getProperty('jython.home')))
to read
config.readfp(open('%sconfig/clenotes.cfg' % System.getProperty('jython.home'), "rU"))
The "rU" tells Python that even though it's running on a Unix system it should be prepared to cope with a file containing DOS line endings. See http://docs.python.org/library/functions.html#open -- scroll down to the paragraph that begins "In addition to the standard fopen() modes...".
(Or you could run this command: perl -pi.bak -e 's/[ \t\r]+$// clenotes.cfg -- that will convert it to Unix line endings. In your shoes I would probably do both.)
(If neither of the above suggestions helps, the next thing I would try is hitting clenotes.py itself with the above perl command. I don't see how that could be the problem, but if the \r characters are not coming from clenotes.cfg, the .py file is the only plausible remaining source.)
(EDIT: Based on your comments on the question itself, I now think it's clenotes.cmd, the shell script wrapper, that needs to be converted from DOS to Unix line endings.)

I'll have to keep looking to figure out where that \r is coming from. But in the meanwhile, this problem has become much simpler. Once the args are parsed, do this:
args = [arg.strip() for arg in args]
That will get rid of the \r
EDIT: But wait -- is this only a partial solution? Is it still not parsing options correctly?
EDIT2: Seems like the \r needs to be stripped earlier. When there's no command, the /r never gets stripped, because the above only strips \r after getopt is done. This should have been obvious to me before -- instead of passing sys.argv[1:] here
opts, args = getopt.getopt(sys.argv[1:], '', cliOptions2)
modify it first
argv = [arg.strip() for arg in sys.argv[1:]]
opts, args = getopt.getopt(argv, '', cliOptions2)
You could also just do sys.argv[-1] = sys.argv[-1].strip()... but the c programmer in me starts to feel a bit queasy looking at that. Probably irrational, I know.
Or just do what Zack said and convert clenotes.cmd to linux format -- however, note that stripping here will ensure that other people will not have to solve the same problem over again. (On the other hand, it's a little ugly, or at least mysterious to people not expecting such problems.)