The subprocess.Popen() lets you pass the shell of your choice via the "executable" parameter.
I have chosen to pass "/bin/tcsh", and I do not want the tcsh to read my ~/.cshrc.
The tcsh manual says that I need to pass -f to /bin/tcsh to do that.
How do I ask Popen to execute /bin/tcsh with a -f option?
import subprocess
cmd = ["echo hi"]
print cmd
proc = subprocess.Popen(cmd, shell=False, executable="/bin/tcsh", stderr=subprocess.PIPE, stdout=subprocess.PIPE)
return_code = proc.wait()
for line in proc.stdout:
print("stdout: " + line.rstrip())
for line in proc.stderr:
print("stderr: " + line.rstrip())
print return_code
Make your life easier:
subprocess.Popen(['/bin/tcsh', '-f', '-c', 'echo hi'],
shell=False, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
I do not understand what the title of your question "Passing arguments to subprocess executable" has to do with the rest of it, especially "I want the tcsh to not to read my ~/.cshrc."
However - I do know that you are not using your Popen correctly.
Your cmd should either be a list or a string, not a list of 1 string.
So cmd = ["echo hi"] should be either cmd = "echo hi" or cmd = ["echo", "hi"]
Then, depending on if it is a string or list you need to set the shell value to True or False. True if it is a string, False if it is a list.
"passing" an argument is a term for functions, using Popen, or subprocess module is not the same as a function, though they are functions, you are actually running a command with them, not passing arguments to them in the traditional sense, so if you want to run a process with '-f' you simply add '-f' to the string or list that you want to run the command with.
To put the whole thing together, you should run something like:
proc = subprocess.Popen('/bin/tcsh -f -c "echo hi"', shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
Related
I want to write a function that will execute multiple shell commands one at a time and print what the shell returns in real time.
I currently have the following code which does not print the shell (I am using Windows 10 and python 3.6.2):
commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", shell=True, stdin=subprocess.PIPE, \
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for command in commands:
p.stdin.write((command + "\n").encode("utf-8"))
p.stdin.close()
p.stdout.read()
How can I see what the shell returns in real time ?
Edit : This question is not a duplicate of the two first links in the comments, they do not help printing in real time.
It is possible to handle stdin and stdout in different threads. That way one thread can be handling printing the output from stdout and another one writing new commands on stdin. However, since stdin and stdout are independent streams, I do not think this can guarantee the order between the streams. For the current example it seems to work as intended, though.
import subprocess
import threading
def stdout_printer(p):
for line in p.stdout:
print(line.rstrip())
commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
universal_newlines=True)
t = threading.Thread(target=stdout_printer, args=(p,))
t.start()
for command in commands:
p.stdin.write((command + "\n"))
p.stdin.flush()
p.stdin.close()
t.join()
Also, note that I am writing stdout line by line, which is normally OK, since it tends to be buffered and being generated a line (or more) at a time. I guess it is possible to handle an unbuffered stdout stream (or e.g. stderr) character-by-character instead, if that is preferable.
I believe you need something like this
commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", shell=True, stdin=subprocess.PIPE, \
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for command in commands:
p.stdin.write((command + "\n").encode("utf-8"))
out, err = p.communicate()
print("{}".format(out))
print("{}".format(err))
Assuming you want control of the output in your python code you might need to do something like this
import subprocess
def run_process(exe):
'Define a function for running commands and capturing stdout line by line'
p = subprocess.Popen(exe.split(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
return iter(p.stdout.readline, b'')
if __name__ == '__main__':
commands = ["foo", "foofoo"]
for command in commands:
for line in run_process(command):
print(line)
I want to execute bash command
'/bin/echo </verbosegc> >> /tmp/jruby.log'
in python using Popen. The code does not raise any exception, but none change is made on the jruby.log after execution. The python code is shown below.
>>> command='/bin/echo </verbosegc> >> '+fullpath
>>> command
'/bin/echo </verbosegc> >> /tmp/jruby.log'
>>process = subprocess.Popen(command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
>>> output= process.communicate()[0]
>>> output
'</verbosegc> >> /tmp/jruby.log\n
I also print out the process.pid and then check the pid using ps -ef | grep pid. The result shows that the process pid has been finished.
Just use pass file object if you want to append the output to a file, you cannot redirect to a file unless you set shell=True:
command = ['/bin/echo', '</verbosegc>']
with open('/tmp/jruby.log',"a") as f:
subprocess.check_call(command, stdout=f,stderr=subprocess.STDOUT)
The first argument to subprocess.Popen is the array ['/bin/echo', '</verbosegc>', '>>', '/tmp/jruby.log']. When the first argument to subprocess.Popen is an array, it does not launch a shell to run the command, and the shell is what's responsible for interpreting >> /tmp/jruby.log to mean "write output to jruby.log".
In order to make the >> redirection work in this command, you'll need to pass command directly to subprocess.Popen() without splitting it into a list. You'll also need to quote the first argument (or else the shell will interpret the "<" and ">" characters in ways you don't want):
command = '/bin/echo "</verbosegc>" >> /tmp/jruby.log'
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
Consider the following:
command = [ 'printf "%s\n" "$1" >>"$2"', # shell script to execute
'', # $0 in shell
'</verbosegc>', # $1
'/tmp/jruby.log' ] # $2
subprocess.Popen(command, shell=True)
The first argument is a shell script referring to $1 and $2, which are in turn passed as separate arguments. Keeping data separate from code, rather than trying to substitute the former into the latter, is a precaution against shell injection (think of this as an analog to SQL injection).
Of course, don't actually do anything like this in Python -- the native primitives for file IO are far more appropriate.
Have you tried without splitting the command and using shell=True? My usual format is:
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
output = process.stdout.read() # or .readlines()
I want to run this git command through a Python script and get the output of it:
git diff --name-only mybranch `git merge-base mybranch develop`
The purpose of the command is to see what changes have been made on mybranch since the last merge with develop.
To achieve this I'm using subprocess.Popen:
output = subprocess.Popen(["git", "diff", "--name-only", "mybranch", "`git merge-base mybranch develop`"], stdout=subprocess.PIPE, shell=True)
However, this does not work. The variable output.communicate()[0] simply gives me a printout of git usage -- essentially telling me the input command is wrong.
I saw that a similar question exists here, but it only told me to use shell=True which didn't solve my problem.
I also attempted to run the two commands in succession, but that gave me the same output as before. It is possible that I am missing something in this step, though.
Any help or tips are appreciated.
Backticks and subprocess
The backtick being a shell feature, you may not have a choice but to use shell=True, however pass in a shell command string, not a list of args
So for your particular command (assuming it works in the first place)
process = subprocess.Popen("git diff --name-only mybranch `git merge-base mybranch develop`", stdout=subprocess.PIPE, shell=True)
Notice when you call Popen() you get a process, shouldn't be called output IMO
Here's a simple example that works with backticks
>>> process = subprocess.Popen('echo `pwd`', stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'/Users/bakkal\n'
Or you can use the $(cmd) syntax
>>> process = subprocess.Popen('echo $(pwd)', stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'/Users/bakkal\n'
Here's what did NOT work (for backticks)
>>> process = subprocess.Popen(['echo', '`pwd`'], stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'\n'
>>> process = subprocess.Popen(['echo', '`pwd`'], stdout=subprocess.PIPE, shell=False)
>>> out, err = process.communicate()
>>> out
'`pwd`\n'
On POSIX, the argument list is passed to /bin/sh -c i.e., only the first argument is recognized as a shell command i.e., the shell runs git without any arguments that is why you see the usage info. You should pass the command as a string if you want to use shell=True. From the subprocess docs:
On POSIX with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
You don't need shell=True in this case.
#!/usr/bin/env python
from subprocess import check_output
merge_base_output = check_output('git merge-base mybranch develop'.split(),
universal_newlines=True).strip()
diff_output = check_output('git diff --name-only mybranch'.split() +
[merge_base_output])
Submitting a complex cmd string made of a full file path to an executable, the multiple flags, arguments, parameters, inputs and outputs seems to require me to set shell=True otherwise subprocess.Popen is not able understand anything more complex than just a simple path to executable (with no spaces in a filepath).
In my example I have quite a long cmd:
cmd = " '/Application/MyApp.app/Contents/MacOS/my_executable' '/Path/to/input/files' -some -flags -here -could -be -a -lot '/full/path/to/output/files' "
Submitting this cmd to subprocess.Popen " results to an error that complains on something about the path and not being able to find it.
So instead of using :
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
check_call seems workings quite well:
proc = subprocess.check_call(cmd, shell=True)
Interesting, only after shell is set to True
shell=True
the subprocess.check_call works with a supplied cmd.
The side effect is that the rest of the code seems proceeds running without waiting for subprocess.check_call(cmd, shell=True) to finish first.
The code is designed the way that the rest of the execution is dependent on a result of subprocess.check_call(cmd, shell=True).
I wonder if there is anyway to enforce the code execution to wait till subprocess.check_call(cmd, shell=True) is finished. Thanks in advance!
As #mikkas suggest just use it as a list here is a working example:
mainProcess = subprocess.Popen(['python', pyfile, param1, param2], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# get the return value from the method
communicateRes = mainProcess.communicate()
stdOutValue, stdErrValue = communicateRes
You are calling python.exe pyfile param1 param2
By using communicate() you can get the stdout and stderr as a Tuple
You can use python method split() to split your string to a list for example:
cmd = "python.exe myfile.py arg1 arg2"
cmd.split(" ")
Output:
['python.exe', 'myfile.py', 'arg1', 'arg2']
I think the check_call function should wait for the command to finish.
See the docs here
http://docs.python.org/2/library/subprocess.html
Check call does not wait. You need to do a process.wait() and check the return code explicitly to get the functionaly you want.
Process = subprocess.Popen('%s' %command_string,stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
Process.wait()
if Process1.returncode!=0:
print Process1.returncode
sendMail()
return
else:
sendMail()
I'm trying to perform simple echo operation using subprocess:
import subprocess
import shlex
cmd = 'echo $HOME'
proc = subprocess.Popen(shlex.split(cmd), shell=True, stdout=subprocess.PIPE)
print proc.communicate()[0]
But it prints nothing. Even if I change the command to echo "hello, world" it still prints nothing. Any help is appreciated.
On Unix shell=True implies that 2nd and following arguments are for the shell itself, use a string to pass a command to the shell:
import subprocess
cmd = 'echo $HOME'
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
print proc.communicate()[0],
You could also write it as:
import subprocess
cmd = 'echo $HOME'
print subprocess.check_output(cmd, shell=True),
From the subprocess' docs:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
You are confusing the two different invocations of Popen. Either of these will work:
proc=subprocess.Popen(['/bin/echo', 'hello', 'world'], shell=False, stdout=subprocess.PIPE)
or
proc=subprocess.Popen('echo hello world', shell=True, stdout=subprocess.PIPE)
When passing shell=True, the first argument is a string--the shell command line. When not using the shell, the first argument is a list. Both produce this:
print proc.communicate()
('hello world\n', None)