I have a Python script (2.7) which I use to invoke an external process.Till recently it worked fine.
But now when I run it I see it doesn't pass over process arguments.I have also debugged the invoked process and it receives only the single argument (the path of the process executable).
p = subprocess.Popen(["./myapp","-p","s"],shell=True)
p.communicate()
Execution of the above code passes only "myapp" as the command argument.Why could that happen?
When using shell=True, just pass a string (not a list);
p = subprocess.Popen('./myapp -p s', shell=True)
p.communicate()
Update
Always prefer;
shell=False (the default) to shell=True and pass an array of strings; and
an absolute path to the executable, not a relative path.
I.e.;
with subprocess.Popen(['/path/to/binary', '-p', 's']) as proc:
stdout, stderr = proc.communicate()
If you're just interested in the stdout (and not the stderr), prefer this to the above solution (it's safer and shorter):
stdout = subprocess.check_output(['/path/to/binary', '-p', 's'])
Don't use shell=True:
p = subprocess.Popen(["./myapp","-p","s"])
p.communicate()
Related
I want to run this git command through a Python script and get the output of it:
git diff --name-only mybranch `git merge-base mybranch develop`
The purpose of the command is to see what changes have been made on mybranch since the last merge with develop.
To achieve this I'm using subprocess.Popen:
output = subprocess.Popen(["git", "diff", "--name-only", "mybranch", "`git merge-base mybranch develop`"], stdout=subprocess.PIPE, shell=True)
However, this does not work. The variable output.communicate()[0] simply gives me a printout of git usage -- essentially telling me the input command is wrong.
I saw that a similar question exists here, but it only told me to use shell=True which didn't solve my problem.
I also attempted to run the two commands in succession, but that gave me the same output as before. It is possible that I am missing something in this step, though.
Any help or tips are appreciated.
Backticks and subprocess
The backtick being a shell feature, you may not have a choice but to use shell=True, however pass in a shell command string, not a list of args
So for your particular command (assuming it works in the first place)
process = subprocess.Popen("git diff --name-only mybranch `git merge-base mybranch develop`", stdout=subprocess.PIPE, shell=True)
Notice when you call Popen() you get a process, shouldn't be called output IMO
Here's a simple example that works with backticks
>>> process = subprocess.Popen('echo `pwd`', stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'/Users/bakkal\n'
Or you can use the $(cmd) syntax
>>> process = subprocess.Popen('echo $(pwd)', stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'/Users/bakkal\n'
Here's what did NOT work (for backticks)
>>> process = subprocess.Popen(['echo', '`pwd`'], stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'\n'
>>> process = subprocess.Popen(['echo', '`pwd`'], stdout=subprocess.PIPE, shell=False)
>>> out, err = process.communicate()
>>> out
'`pwd`\n'
On POSIX, the argument list is passed to /bin/sh -c i.e., only the first argument is recognized as a shell command i.e., the shell runs git without any arguments that is why you see the usage info. You should pass the command as a string if you want to use shell=True. From the subprocess docs:
On POSIX with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
You don't need shell=True in this case.
#!/usr/bin/env python
from subprocess import check_output
merge_base_output = check_output('git merge-base mybranch develop'.split(),
universal_newlines=True).strip()
diff_output = check_output('git diff --name-only mybranch'.split() +
[merge_base_output])
When i am running a program in the console, i get some text output.
When i am running the same program in Popen(..), with the same parameters, stdout and stderr are empty.
I tried everything i could imagine like shell=False and shell=True, set stdout=subprocess.PIPE, did a os.chdir() to change into the directory of this program, try p.wait() and p.communicate(), set the command as a list and as a string, but nothing works.
example:
p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
out, err = p.communicate()
--> out and err are empty strings, but if i ran this command in console i get a real output. Command is with fullpath, so its regardless where the command will be started.
My question is, are there mechanisms for programms to detect they weren't run in a real console? If so, how can i cheat.
Or miss i something?
(Python 2.7.8. x32 in Win7 x64)
from subprocess import Popen, STDOUT, PIPE
p = Popen(command, shell=True, stdout=PIPE, stderr=STDOUT, stdin=PIPE)
while p.poll() is None:
print(p.stdout.read())
p.stdout.close()
p.stdin.close()
Try this and see if it makes any difference. Also make sure command is a string and not a list/touple, shell=True for whatever reason works better or only with strings.
Also note that shell=True will get you hanged because it's insecure etc.
Also skipping .communicate() you'll need to tap off stdout otherwise the buffer will get full and you might hang both yours and the child process.
If this doesn't work, please provide more information. Such as the command used and the expected output (at least first few lines)
Submitting a complex cmd string made of a full file path to an executable, the multiple flags, arguments, parameters, inputs and outputs seems to require me to set shell=True otherwise subprocess.Popen is not able understand anything more complex than just a simple path to executable (with no spaces in a filepath).
In my example I have quite a long cmd:
cmd = " '/Application/MyApp.app/Contents/MacOS/my_executable' '/Path/to/input/files' -some -flags -here -could -be -a -lot '/full/path/to/output/files' "
Submitting this cmd to subprocess.Popen " results to an error that complains on something about the path and not being able to find it.
So instead of using :
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
check_call seems workings quite well:
proc = subprocess.check_call(cmd, shell=True)
Interesting, only after shell is set to True
shell=True
the subprocess.check_call works with a supplied cmd.
The side effect is that the rest of the code seems proceeds running without waiting for subprocess.check_call(cmd, shell=True) to finish first.
The code is designed the way that the rest of the execution is dependent on a result of subprocess.check_call(cmd, shell=True).
I wonder if there is anyway to enforce the code execution to wait till subprocess.check_call(cmd, shell=True) is finished. Thanks in advance!
As #mikkas suggest just use it as a list here is a working example:
mainProcess = subprocess.Popen(['python', pyfile, param1, param2], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# get the return value from the method
communicateRes = mainProcess.communicate()
stdOutValue, stdErrValue = communicateRes
You are calling python.exe pyfile param1 param2
By using communicate() you can get the stdout and stderr as a Tuple
You can use python method split() to split your string to a list for example:
cmd = "python.exe myfile.py arg1 arg2"
cmd.split(" ")
Output:
['python.exe', 'myfile.py', 'arg1', 'arg2']
I think the check_call function should wait for the command to finish.
See the docs here
http://docs.python.org/2/library/subprocess.html
Check call does not wait. You need to do a process.wait() and check the return code explicitly to get the functionaly you want.
Process = subprocess.Popen('%s' %command_string,stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
Process.wait()
if Process1.returncode!=0:
print Process1.returncode
sendMail()
return
else:
sendMail()
I'm trying to run a shell command from within a python script which needs to do several things
1. The shell command is 'hspice tran.deck >! tran.lis'
2. The script should wait for the shell command to complete before proceeding
3. I need to check the return code from the command and
4. Capture STDOUT if it completed successfully else capture STDERR
I went through the subprocess module and tried out a couple of things but couldn't find a way to do all of the above.
- with subprocess.call() I could check the return code but not capture the output.
- with subprocess.check_output() I could capture the output but not the code.
- with subprocess.Popen() and Popen.communicate(), I could capture STDOUT and STDERR but not the return code.
I'm not sure how to use Popen.wait() or the returncode attribute. I also couldn't get Popen to accept '>!' or '|' as arguments.
Can someone please point me in the right direction? I'm using Python 2.7.1
EDIT: Got things working with the following code
process = subprocess.Popen('ls | tee out.txt', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = process.communicate()
if(process.returncode==0):
print out
else:
print err
Also, should I use a process.wait() after the process = line or does it wait by default?
Just use .returncode after .communicate(). Also, tell Popen that what you're trying to run is a shell command, rather than a raw command line:
p = subprocess.Popen('ls | tee out.txt', shell=True, ...)
p.communicate()
print p.returncode
From the docs:
Popen.returncode
The child return code, set by poll() and wait() (and indirectly by communicate()). A None value indicates that the process hasn’t terminated yet.
A negative value -N indicates that the child was terminated by signal N (Unix only).
Here is example how to interact with shell:
>>> process = subprocess.Popen(['/bin/bash'], shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>> process.stdin.write('echo it works!\n')
>>> process.stdout.readline()
'it works!\n'
>>> process.stdin.write('date\n')
>>> process.stdout.readline()
'wto, 13 mar 2012, 17:25:35 CET\n'
>>>
I want to create a subprocess with popen.
The special demand is to use shell direction in the command.
args = [
'/bin/cat',
'<',
'textfile',
]
process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
env={'LANG':'de_DE#euro'})
processpid = process.pid
output = process.communicate()
I do not want to use the shell=True option, therefore here my question to you, how to implent it.
Regards
Stefan
Impossible unless the program you're calling implements redirection itself (which cat doesn't). To use shell features, you have to pass shell=True or invoke a shell explicitly.
OTOH, if you just want to read from textfile, you could pass it as the subprocess's stdin:
subprocess.Popen(args,
stdin=open("textfile"),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
env={'LANG':'de_DE#euro'})