I want to run this git command through a Python script and get the output of it:
git diff --name-only mybranch `git merge-base mybranch develop`
The purpose of the command is to see what changes have been made on mybranch since the last merge with develop.
To achieve this I'm using subprocess.Popen:
output = subprocess.Popen(["git", "diff", "--name-only", "mybranch", "`git merge-base mybranch develop`"], stdout=subprocess.PIPE, shell=True)
However, this does not work. The variable output.communicate()[0] simply gives me a printout of git usage -- essentially telling me the input command is wrong.
I saw that a similar question exists here, but it only told me to use shell=True which didn't solve my problem.
I also attempted to run the two commands in succession, but that gave me the same output as before. It is possible that I am missing something in this step, though.
Any help or tips are appreciated.
Backticks and subprocess
The backtick being a shell feature, you may not have a choice but to use shell=True, however pass in a shell command string, not a list of args
So for your particular command (assuming it works in the first place)
process = subprocess.Popen("git diff --name-only mybranch `git merge-base mybranch develop`", stdout=subprocess.PIPE, shell=True)
Notice when you call Popen() you get a process, shouldn't be called output IMO
Here's a simple example that works with backticks
>>> process = subprocess.Popen('echo `pwd`', stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'/Users/bakkal\n'
Or you can use the $(cmd) syntax
>>> process = subprocess.Popen('echo $(pwd)', stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'/Users/bakkal\n'
Here's what did NOT work (for backticks)
>>> process = subprocess.Popen(['echo', '`pwd`'], stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'\n'
>>> process = subprocess.Popen(['echo', '`pwd`'], stdout=subprocess.PIPE, shell=False)
>>> out, err = process.communicate()
>>> out
'`pwd`\n'
On POSIX, the argument list is passed to /bin/sh -c i.e., only the first argument is recognized as a shell command i.e., the shell runs git without any arguments that is why you see the usage info. You should pass the command as a string if you want to use shell=True. From the subprocess docs:
On POSIX with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
You don't need shell=True in this case.
#!/usr/bin/env python
from subprocess import check_output
merge_base_output = check_output('git merge-base mybranch develop'.split(),
universal_newlines=True).strip()
diff_output = check_output('git diff --name-only mybranch'.split() +
[merge_base_output])
Related
I want to execute bash command
'/bin/echo </verbosegc> >> /tmp/jruby.log'
in python using Popen. The code does not raise any exception, but none change is made on the jruby.log after execution. The python code is shown below.
>>> command='/bin/echo </verbosegc> >> '+fullpath
>>> command
'/bin/echo </verbosegc> >> /tmp/jruby.log'
>>process = subprocess.Popen(command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
>>> output= process.communicate()[0]
>>> output
'</verbosegc> >> /tmp/jruby.log\n
I also print out the process.pid and then check the pid using ps -ef | grep pid. The result shows that the process pid has been finished.
Just use pass file object if you want to append the output to a file, you cannot redirect to a file unless you set shell=True:
command = ['/bin/echo', '</verbosegc>']
with open('/tmp/jruby.log',"a") as f:
subprocess.check_call(command, stdout=f,stderr=subprocess.STDOUT)
The first argument to subprocess.Popen is the array ['/bin/echo', '</verbosegc>', '>>', '/tmp/jruby.log']. When the first argument to subprocess.Popen is an array, it does not launch a shell to run the command, and the shell is what's responsible for interpreting >> /tmp/jruby.log to mean "write output to jruby.log".
In order to make the >> redirection work in this command, you'll need to pass command directly to subprocess.Popen() without splitting it into a list. You'll also need to quote the first argument (or else the shell will interpret the "<" and ">" characters in ways you don't want):
command = '/bin/echo "</verbosegc>" >> /tmp/jruby.log'
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
Consider the following:
command = [ 'printf "%s\n" "$1" >>"$2"', # shell script to execute
'', # $0 in shell
'</verbosegc>', # $1
'/tmp/jruby.log' ] # $2
subprocess.Popen(command, shell=True)
The first argument is a shell script referring to $1 and $2, which are in turn passed as separate arguments. Keeping data separate from code, rather than trying to substitute the former into the latter, is a precaution against shell injection (think of this as an analog to SQL injection).
Of course, don't actually do anything like this in Python -- the native primitives for file IO are far more appropriate.
Have you tried without splitting the command and using shell=True? My usual format is:
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
output = process.stdout.read() # or .readlines()
I'm trying to issue a command over ssh and get its return code via subprocess. I have some code that looks like this:
cmd = 'ssh user#ip_addr "some_command"'
res = subprocess.check_output(
cmd,
shell=True,
stdout=subprocess.PIPE)
Now this works great if the cmd only generates an exit code (for instance, setting cmd to "exit 1", and then doing a try/catch to see if it exits with a non-zero. However, the following hangs indefinitely:
cmd = 'ssh user#ip_addr "ls -la && exit 0;"'
res = subprocess.check_output(
cmd,
shell=True,
stdout=subprocess.PIPE)
I saw two questions that looked similar, and I did RTFM, but I'm still not sure what to do. I don't really care whether or not the command generates output; I'm more concerned with the exit code. If anyone knows what the best way of doing this is, or whether or not I'm using subprocess inappropriately, I would appreciate it.
Remove the stdout=subprocess.PIPE, and it should work; check_output itself captures the output, so redirecting it using stdout=subprocess.PIPE will cause problems. If you don't care about the output at all, just use subprocess.check_call (and again, don't use stdout=subprocess.PIPE).
DO NOT USE std{out,err}=PIPE UNLESS YOU READ FROM THE PIPE!!!
To get the return code while discarding the output of a command issued over ssh using subprocess module:
from subprocess import call, DEVNULL, STDOUT
returncode = call(['ssh', 'user#ip', 'ls -la && exit 0;'],
stdin=DEVNULL, stdout=DEVNULL, stderr=STDOUT)
See also, How to hide output of subprocess in Python 2.7.
Note: shell=True is not used.
I'm trying to run repo command using subprocess.check_call. I don't see any error but it's not running.
Here is my code.
def repo(*args):
return subprocess.check_call(['repo'] + list(args), shell = True, stdout = subprocess.PIPE, stderr = subprocess.PIPE)
repo('forall','-pc','"','git','merge','--strategy=ours','\${REPO_REMOTE}/branch_name','"','>log.log','2>&1')
Am I missing something?
Please help.
Thanks.
I'm going on a hunch, I guess you don't see anything because the error messages are stuck in your stderr pipe. Try this;
import subprocess
def repo(command):
subprocess.check_call('repo ' + command, shell=True)
repo('forall -pc "git merge --strategy=ours \${REPO_REMOTE}/branch_name" > log.log 2>&1')
Does that look more like you imagined? Also;
when using shell=True (although I don't recommend that your do) you can just pass a str (as opposed to a list); and
there is not pressing need for the return because check_call() either raise an exception or returns 0.
If you have shell=True and the first argument a sequence, as you have, then the first element in the sequence will be passed as option -c to the shell, and the rest of the elements will be additional arguments to the shell. Example
subprocess.check_call(['ls', '-l'], shell=True)
means the following is run:
sh -c "ls" -l
Note that ls doesn't get the option -l, but the shell sh does.
So, you should not use shell=True. If you have to, use a string instead of a list as args.
Also, the fine manual warns not to use stdout=PIPE and stderr=PIPE with check_call().
I'm trying to perform simple echo operation using subprocess:
import subprocess
import shlex
cmd = 'echo $HOME'
proc = subprocess.Popen(shlex.split(cmd), shell=True, stdout=subprocess.PIPE)
print proc.communicate()[0]
But it prints nothing. Even if I change the command to echo "hello, world" it still prints nothing. Any help is appreciated.
On Unix shell=True implies that 2nd and following arguments are for the shell itself, use a string to pass a command to the shell:
import subprocess
cmd = 'echo $HOME'
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
print proc.communicate()[0],
You could also write it as:
import subprocess
cmd = 'echo $HOME'
print subprocess.check_output(cmd, shell=True),
From the subprocess' docs:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
You are confusing the two different invocations of Popen. Either of these will work:
proc=subprocess.Popen(['/bin/echo', 'hello', 'world'], shell=False, stdout=subprocess.PIPE)
or
proc=subprocess.Popen('echo hello world', shell=True, stdout=subprocess.PIPE)
When passing shell=True, the first argument is a string--the shell command line. When not using the shell, the first argument is a list. Both produce this:
print proc.communicate()
('hello world\n', None)
The subprocess.Popen() lets you pass the shell of your choice via the "executable" parameter.
I have chosen to pass "/bin/tcsh", and I do not want the tcsh to read my ~/.cshrc.
The tcsh manual says that I need to pass -f to /bin/tcsh to do that.
How do I ask Popen to execute /bin/tcsh with a -f option?
import subprocess
cmd = ["echo hi"]
print cmd
proc = subprocess.Popen(cmd, shell=False, executable="/bin/tcsh", stderr=subprocess.PIPE, stdout=subprocess.PIPE)
return_code = proc.wait()
for line in proc.stdout:
print("stdout: " + line.rstrip())
for line in proc.stderr:
print("stderr: " + line.rstrip())
print return_code
Make your life easier:
subprocess.Popen(['/bin/tcsh', '-f', '-c', 'echo hi'],
shell=False, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
I do not understand what the title of your question "Passing arguments to subprocess executable" has to do with the rest of it, especially "I want the tcsh to not to read my ~/.cshrc."
However - I do know that you are not using your Popen correctly.
Your cmd should either be a list or a string, not a list of 1 string.
So cmd = ["echo hi"] should be either cmd = "echo hi" or cmd = ["echo", "hi"]
Then, depending on if it is a string or list you need to set the shell value to True or False. True if it is a string, False if it is a list.
"passing" an argument is a term for functions, using Popen, or subprocess module is not the same as a function, though they are functions, you are actually running a command with them, not passing arguments to them in the traditional sense, so if you want to run a process with '-f' you simply add '-f' to the string or list that you want to run the command with.
To put the whole thing together, you should run something like:
proc = subprocess.Popen('/bin/tcsh -f -c "echo hi"', shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)