I would like to be able to call some executables that take in parameters and then dump the output to a file. I've attempted to use both os.system and subprocess calls to no avail. Here is a sample of what I'd like python to execute for me...
c:\directory\executable_program.exe -f w:\directory\input_file.txt > Z\directory\output_file.txt
Notice the absolute paths as I will be traversing hundreds of various directories to act on files etc..
Many thanks ahead of time!
Some examples that I've tried:
subprocess.run(['c:\directory\executable_program.exe -f w:\directory\input_file.txt > Z\directory\output_file.txt']
subprocess.call(r'"c:\directory\executable_program.exe -f w:\directory\input_file.txt > Z\directory\output_file.txt"']
subprocess.call(r'"c:\directory\executable_program.exe" -f "w:\directory\input_file.txt > Z\directory\output_file.txt"']
Your attempts contain various amounts of quoting errors.
subprocess.run(r'c:\directory\executable_program.exe -f w:\directory\input_file.txt > Z\directory\output_file.txt', shell=True)
should work, where the r prefix protects the backslashes from being interpreted and removed by Python before the subprocess runs, and the absence of [...] around the value passes it verbatim to the shell (hence, shell=True).
On Windows you could get away with putting the command in square brackets even though it's not a list, and omitting shell=True in some circumstances.
If you wanted to avoid the shell, try
with open(r'Z\directory\output_file.txt', 'wb') as dest:
subprocess.run(
[r'c:\directory\executable_program.exe', '-f', r'w:\directory\input_file.txt'],
stdout=dest)
which also illustrates how to properly pass a list of strings in square brackets as the first argument to subprocess.run.
Related
I would like to retrieve output from a shell command that contains spaces and quotes. It looks like this:
import subprocess
cmd = "docker logs nc1 2>&1 |grep mortality| awk '{print $1}'|sort|uniq"
subprocess.check_output(cmd)
This fails with "No such file or directory". What is the best/easiest way to pass commands such as these to subprocess?
The absolutely best solution here is to refactor the code to replace the entire tail of the pipeline with native Python code.
import subprocess
from collections import Counter
s = subprocess.run(
["docker", "logs", "nc1"],
text=True, capture_output=True, check=True)
count = Counter()
for line in s.stdout.splitlines():
if "mortality" in line:
count[line.split()[0]] += 1
for count, word in count.most_common():
print(count, word)
There are minor differences in how Counter objects resolve ties (if two words have the same count, the one which was seen first is returned first, rather than by sort order), but I'm guessing that's unimportant here.
I am also ignoring standard output from the subprocess; if you genuinely want to include output from error messages, too, just include s.stderr in the loop driver too.
However, my hunch is that you don't realize your code was doing that, which drives home the point nicely: Mixing shell script and Python raises the mainainability burden, because now you have to understand both shell script and Python to understand the code.
(And in terms of shell script style, I would definitely get rid of the useless grep by refactoring it into the Awk script, and probably also fold in the sort | uniq which has a trivial and more efficient replacement in Awk. But here, we are replacing all of that with Python code anyway.)
If you really wanted to stick to a pipeline, then you need to add shell=True to use shell features like redirection, pipes, and quoting. Without shell=True, Python looks for a command whose file name is the entire string you were passing in, which of course doesn't exist.
I am trying to execute the following awk command from a python script
awk 'BEGIN {FS="\t"}; {print $1"\t"$2}' file_a > file_b
For this, I tried to use subprocess as follows:
subprocess.check_output(["awk", 'BEGIN {FS="\t"}; {print $1"\t"$2}',
file_a, ">",
file_b])
where file_a and file_b are strings pointing to the path of the files.
From this, I am getting the error
awk: cannot open > (No such file or directory)
I'm sure I'm inputing the arguments to subprocess in a wrong way, but I can't figure out what's wrong.
While it may look like it in your shell of choice, >, <, and | are not actually passed as arguments to the program you run. Rather, they're a special part of the shell that the program never gets to see.
Since they're part of the shell, and not part of the OS or program, you have to emulate their effects yourself with the normal facilities the language gives you. In your case, since you're trying to pipe to a file, simply use Python's open() as you would normally. The subprocess API supports arguments to specify stdout, stdin, and stderr, and you can supply any file object for those.
Check it out:
with open(file_b, 'wb') as f:
subprocess.call(["awk", 'BEGIN {FS="\t"}; {print $1"\t"$2}', file_a], stdout=f)
Since subprocess.check_output redirects output already, it doesn't take the stdout argument. Using subprocess.call avoids this. If you also need the output later in the script, you can instead assign the return value of check_output to a variable, and then save that to file_b.
If you use a lot of shell commands, you might also want to check out Plumbum, which gives you a large set of fairly silly shell-like operator overloads.
I'm trying to read the duration of video files using mediainfo. This shell command works
mediainfo --Inform="Video;%Duration/String3%" file
and produces an output like
00:00:33.600
But when I try to run it in python with this line
subprocess.check_output(['mediainfo', '--Inform="Video;%Duration/String3%"', file])
the whole --Inform thing is ignored and I get the full mediainfo output instead.
Is there a way to see the command constructed by subprocess to see what's wrong?
Or can anybody just tell what's wrong?
Try:
subprocess.check_output(['mediainfo', '--Inform=Video;%Duration/String3%', file])
The " in your python string are likely passed on to mediainfo, which can't parse them and will ignore the option.
These kind of problems are often caused by shell commands requiring/swallowing various special characters. Quotes such as " are often removed by bash due to shell magic. In contrast, python does not require them for magic, and will thus replicate them the way you used them. Why would you use them if you wouldn't need them? (Well, d'uh, because bash makes you believe you need them).
For example, in bash I can do
$ dd of="foobar"
and it will write to a file named foobar, swallowing the quotes.
In python, if I do
subprocess.check_output(["dd", 'of="barfoo"', 'if=foobar'])
it will write to a file named "barfoo", keeping the quotes.
I have a command which I use for deployment from windows command line. Now I need to run the same from an external python3.4 script.
The command is C:\Program Files (x86)\MSBuild\12.0\Bin\msbuild "D:\WebService\WebService.sln" /p:DeployOnBuild=true /p:PublishProfile="D:\WebService\Properties\PublishProfiles\MyDeployment.pubxml" /p:AllowUntrustedCertificate=true /p:UserName=name /p:Password=PASSWORD.
How can I achieve this. I tried subprocess . But it's not working.Please help me out.
Your problems appear to be the \ and " characters, so use raw strings. Also, it is safer to use a list:
proc = subprocess.Popen(
[r"C:\Program Files (x86)\MSBuild\12.0\Bin\msbuild",
r"D:\WebService\WebService.sln",
r"/p:DeployOnBuild=true",
r"/p:PublishProfile=D:\WebService\Properties\PublishProfiles\MyDeployment.pubxml",
r"/p:AllowUntrustedCertificate=true",
r"/p:UserName=name",
r"/p:Password=PASSWORD"])
proc.wait()
Strictly speaking you don't need raw strings for all those parameters, but it is safer to do so with Windows paths. You only need the internal double quotes if you have embedded whitespace (as the first parameter). Here we are not using a shell, set shell=True as a parameter if you need one. A reason to use a shell on Windows is for filename association, but you don't appear to be using that here.
Can you post some code with what you have tried so far?
The subprocess module should be able to handle that, with something like
theproc = subprocess.Popen(["COMMAND HERE"])
theproc.communicate()
or you could try with the shell flag
theproc = subprocess.Popen(["COMMAND HERE"], shell=True)
Update/Solution: the answer is below, from Zack. The problem was, indeed, DOS line endings on the script file itself, clenotes.cmd. Since I futzed with the various files so much, I deleted the whole directory and then re-downloaded a fresh copy from HERE. I ran Zack's perl script on the file just like so:
perl -pi.bak -e 's/[ \t\r]+$//' clenotes.cmd
I then edited the command execution just slightly so that the final script became:
CWD=`dirname $0`
JYTHON_HOME="$CWD"
LIB_DIR="$JYTHON_HOME/lib"
NOTES_HOME="/opt/ibm/lotus/notes/"
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$NOTES_HOME
java -cp "$LIB_DIR" -jar "$LIB_DIR/jython.jar" -Djython.home="$CWD/" -Dpython.path="$LIB_DIR:$CWD/ext" -Djava.library.path="$NOTES_HOME" "$LIB_DIR/clenotes/cletes/clenotes.py" "$#"
That was it -- everything else worked. No edits needed to clenotes.py or clenotes.cfg. Many thanks for sticking with the question, which I guess ended up being quite simple.
Update: I'm cutting down on some of the code to make this more readable and remove unnecessary information from the post.
I'm trying to get Lotus Notes command line to work on Linux and am having an issue with something related to sys.argv[1:] in the python file. The windows script is here:
#echo off
#setlocal
set CWD=%~dp0
set JYTHON_HOME=%CWD%
set LIB_DIR=%JYTHON_HOME%/lib
java -cp %LIB_DIR% -jar %LIB_DIR%/jython.jar -Djython.home=%CWD% -python.path=%LIB_DIR%;%CWD%/ext %LIB_DIR%/clenotes/clenotes.py %*
#endlocal
I was having a tough time with variables, so for Linux, it simply looks like this:
java -cp ./lib/ -jar ./lib/jython.jar -Djython.home=./ -Dpython.path=./lib:./ext -Djava.library.path=/opt/ibm/lotus/notes/ ./lib/clenotes/clenotes.py $*
I run it from within the directory. In any case, what puzzles me is that it's not picking up any options I pass from the command line. clenotes.cmd --help results in
No commands specified. Use --help option for usage.
Here is the section where the command line arguments are supposed to be parsed:
def main():
Output.log("Entering %s v%s" % (PROGRAM_NAME,VERSION),Output.LOGTYPE_DEBUG)
cliOptions2=[]
for opt in cliOptions:
opt2=opt.replace('--','')
opt2=opt2.replace('!','=')
cliOptions2.append(opt2)
opts=[]
args=[]
try:
opts, args = getopt.getopt(sys.argv[1:], '', cliOptions2)
I'm using Python 3.1.3 on Arch Linux 64bit in a 32bit chroot environment. Can I provide anything else?
Just in case it's needed... HERE is the whole clenotes.py file.
Also, as requested in the comments, the config file (which contains the help message and viable options/arguments, is HERE
Update
After a lot of fiddling, the best progress I have made has been to examine what it's setting as opts and args in the (main) method. Most surprising was that when passing an argument and then looking at it's parsed result using print sys.argv, the option would come up with a trailing \r in it. For example:
clenotes.cmd appointments
args is ['appointments\r']
On Windows I did the same and args was reported as ['appointments']. Furthermore, manually setting args=['appointments'] and then commenting out the section where getopt.getopt is assigning a value worked.
Lastly, I've found that when using multiple arguments, n-1 of them get interpreted and used while the nth one gets ignored. This is kind of a workaround since I can actually use the script... but obviously it's not preferred. If I want to look at today's appointments, I can execute clenotes.cmd appointments --today --today and it will work. sys.argv will spit out: ['appointments', '--today', '--today\r'].
So... what's causing the trailing \r? I'm thinking it has to do with the actual script. Note it again:
java -cp ./lib/ -jar ./lib/jython.jar -Djython.home=./ -Dpython.path=./lib:./ext -Djava.library.path=/opt/ibm/lotus/notes/ ./lib/clenotes/clenotes.py $*
So... bunch of path stuff and then the actual python file: clenotes.py $*
I got the $* from HERE
Is it picking up the carriage return??
I think your problem is that clenotes.cfg has DOS line endings, which Python is misinterpreting. Try changing this line of clenotes.py
config.readfp(open('%sconfig/clenotes.cfg' % System.getProperty('jython.home')))
to read
config.readfp(open('%sconfig/clenotes.cfg' % System.getProperty('jython.home'), "rU"))
The "rU" tells Python that even though it's running on a Unix system it should be prepared to cope with a file containing DOS line endings. See http://docs.python.org/library/functions.html#open -- scroll down to the paragraph that begins "In addition to the standard fopen() modes...".
(Or you could run this command: perl -pi.bak -e 's/[ \t\r]+$// clenotes.cfg -- that will convert it to Unix line endings. In your shoes I would probably do both.)
(If neither of the above suggestions helps, the next thing I would try is hitting clenotes.py itself with the above perl command. I don't see how that could be the problem, but if the \r characters are not coming from clenotes.cfg, the .py file is the only plausible remaining source.)
(EDIT: Based on your comments on the question itself, I now think it's clenotes.cmd, the shell script wrapper, that needs to be converted from DOS to Unix line endings.)
I'll have to keep looking to figure out where that \r is coming from. But in the meanwhile, this problem has become much simpler. Once the args are parsed, do this:
args = [arg.strip() for arg in args]
That will get rid of the \r
EDIT: But wait -- is this only a partial solution? Is it still not parsing options correctly?
EDIT2: Seems like the \r needs to be stripped earlier. When there's no command, the /r never gets stripped, because the above only strips \r after getopt is done. This should have been obvious to me before -- instead of passing sys.argv[1:] here
opts, args = getopt.getopt(sys.argv[1:], '', cliOptions2)
modify it first
argv = [arg.strip() for arg in sys.argv[1:]]
opts, args = getopt.getopt(argv, '', cliOptions2)
You could also just do sys.argv[-1] = sys.argv[-1].strip()... but the c programmer in me starts to feel a bit queasy looking at that. Probably irrational, I know.
Or just do what Zack said and convert clenotes.cmd to linux format -- however, note that stripping here will ensure that other people will not have to solve the same problem over again. (On the other hand, it's a little ugly, or at least mysterious to people not expecting such problems.)