When executing this command from the terminal:
\grep -inH -r --exclude-dir={node_modules,.meteor,.git} -e test -- /Users/nelsyeung/Sites/foo
It outputs the correct results, where it excludes the --exclude-dir directories.
The following Python script should theoretically perform the exact same operation:
#!/usr/bin/env python3
from subprocess import Popen, PIPE
cmd = 'grep -inH -r --exclude-dir={node_modules,.meteor,.git} -e test -- /Users/nelsyeung/Sites/foo'
with Popen(cmd.split(), stdout=PIPE, bufsize=1, universal_newlines=True) as p:
for l in p.stdout:
print(l)
But the --exclude-dir flag seems to be ignored, that is, it also grep from node_modules, .meteor and .git.
Question: Why is the above code outputting a different result from just executing the command?
Note that this is more of a theory question than me looking for an alternative fix, since the Python code is basically from a plugin that I'm using where I can only supply flags for the command, and just that, to my surprise, passing in --exclude-dir does nothing. If there's something wrong with the code above, do point it out though.
System info:
OS: macOS 10.13.6
Python: 3.7.0
grep: (BSD grep) 2.5.1-FreeBSD (with --exclude-dir support)
--exclude-dir={dir1,dir2} is expanded to
--exclude-dir=dir1 --exclude-dir=dir2
by the shell, not by grep. Popen uses shell=False by default.
So instead use
from subprocess import Popen, PIPE
cmd = '''grep -inH -r --exclude-dir=node_modules --exclude-dir=.meteor
--exclude-dir=.git -e test -- /Users/nelsyeung/Sites/foo'''
with Popen(cmd.split(), stdout=PIPE, bufsize=1, universal_newlines=True) as p:
for l in p.stdout:
print(l)
(Note that while using shell=True might be another option, it is not preferred because of security issues.)
Related
I'm struggling to understand why this fails with a wget: missing URL error:
import shlex
import subprocess
copy_command = "wget -O - 'http://example.com/somepath/somefile.txt?someparam=test' | sshpass -p pass ssh user#localhost -p 2222 \"cat - > /upload/somefile.txt\""
cmd = shlex.split(copy_command, posix=False)
with subprocess.Popen(
cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True
) as proc:
output, error = proc.communicate()
What am I missing here? If I just give subprocess the copy_command string directly, then it works without issues.
To set up a pipeline requires the parent process to spawn all the programs involved and the connect (pipe) the stdio of one to another.
The Python documentation for subprocess explains how to do this.
It works with string argument andshell=True because then it just hands off the command line to a sub shell, and that shell handles all those details.
On my Mac, I am using the following command to call g++ in python:
subprocess.run(['g++', './rmc-output/*.cpp', '-w', '-o', 'executable'], check=True,
stderr=PIPE, stdout=PIPE, shell=True)
however, I get the following error while the rmc-output folder is not empty.
clang: error: no such file or directory: './rmc-output/*.cpp'
Am I missing something?
shell=True won't expand wildcards when arguments are put in a list. Quickfix: use a string:
subprocess.run('g++ ./rmc-output/*.cpp -w -o executable', check=True,
stderr=PIPE, stdout=PIPE, shell=True)
quick but dirty, so a better solution:
drop shell=True (avoid whenever possible, security issues, lazy command lines...)
use glob to compute files using python, not the shell
like this:
import glob
subprocess.run(['g++'] + glob.glob('./rmc-output/*.cpp') +['-w', '-o', 'executable'], check=True,
stderr=PIPE, stdout=PIPE)
note that Windows versions of g++ have internal wildcard expansion to make up for the fact that Windows "shell" hasn't. Would have worked on Windows probably.
*.cpp is a shell wildcard pattern. Since you're specifying the command arguments directly, wildcard filename expansion never happens, and it's using *.cpp as a literal filename.
See this answer https://stackoverflow.com/a/9997093/494134 for more details.
I have a python script which captures repo command.
import subprocess
processing(commandforrepo)
def processing(repocmd):
process = None
process = subprocess.Popen(repocmd,
stdout=subprocess.PIPE, stderr=None, shell=True)
process.communicate()
In the particular command, I am trying to parse a list of repocmd to compare two branches and print out the differences
"repo forall $(repo forall -c 'echo $REPO_PROJECT')\
-c 'git log --abbrev-commit --pretty=oneline --no-merges \
--cherry-pick --left-only HEAD...$REPO_RREV'"
Attempted to run the script on the terminal but the command did not get executed. However, when this command is issued on the terminal, it produces a list of differences between the two branches.
Any clue as to what is missing?
Warning: most of the standard output for Git command are done on... stderr.
See here for why: informative messages are on stderr only.
So make sure to parse stderr, not stdout.
I'm trying to use python Popen to achieve what looks like this using the command line.
echo "hello" | docker exec -i $3 sh -c 'cat >/text.txt'
The goal is to pipe the "hello" text into the docker exec command and have it written to the docker container.
I've tried this but can't seem to get it to work.
import subprocess
from subprocess import Popen, PIPE, STDOUT
p = Popen(('docker', 'exec', '-i', 'nginx-ssl', 'sh', '-c', 'cat >/text.txt'), stdin=subprocess.PIPE)
p.stdin.write('Hello')
p.stdin.close()
You need to give stdin the new line also:
p.stdin.write('Hello\n')
That is the same thing even with sys.stdout. You don't need to give print a new line because it does that for you, but any writing to a file that you do manually, you need to include it. You should use p.communicate('Hello') instead, though. It's made for that.
I am writing a small python script that needs to execute git commands from inside a given directory
The code is as follows:
import subprocess, os
pr = subprocess.Popen(['/usr/bin/git', 'status'],
cwd=os.path.dirname('/path/to/dir/'),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True)
(out, error) = pr.communicate()
print out
But it shows git usage as the output.
If the command doesn't involve git for eg. ['ls'] then it shows the correct output.
Is there anything I am missing ?
python version - 2.6.6
Thanks.
subprocess.Popen:
On Unix, with shell=True: […] If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself.
You don't want shell=True and also a list of arguments. Set shell=False.