I use Python 3.10.7 and I am trying to get the Python interpreter to run this command:
rg mysearchterm /home/user/stuff
This command, when I run it in bash directly successfully runs ripgrep and searches the directory (recursively) /home/user/stuff for the term mysearchterm. However, I'm trying to do this programmatically with Python's subprocess.Popen() and I am running into issues:
from subprocess import Popen, PIPE
proc1 = Popen(["rg", "term", "/home/user/stuff", "--no-filename"],stdout=PIPE,shell=True)
proc2 = Popen(["wc","-l"],stdin=proc1.stdin,stdout=PIPE,shell=True)
#Note: I've also tried it like below:
proc1 = Popen(f"rg term /home/user/stuff --no-filename",stdout=PIPE,shell=True)
proc2 = Popen("wc -l",stdin=proc1.stdin,stdout=PIPE,shell=True)
result, _ = proc2.communicate()
print(result.decode())
What happens here was bizarre to me; I get an error (from rg itself) which says:
error: The following required arguments were not provided:
<PATTERN>
So, using my debugging/tracing skills, I looked at the process chain and I see that the python interpreter itself is performing:
python3 1921496 953810 0 /usr/bin/python3 ./debug_script.py
sh 1921497 1921496 0 /bin/sh -c rg term /home/user/stuff --no-filename
sh 1921498 1921496 0 /bin/sh -c wc -l
So my next thought is just trying to run that manually in bash, leading to the same error. However, in bash, when I run /bin/sh -c "rg term /home/user/stuff --no-filename" with double quotations, the command works in bash but when I try to do this programmatically in Popen() it again doesn't work even when I try to escape them with \. This time, I get errors about unexpected EOF.
As for the behavior when shell=True is specified,
the python document tells:
If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
Then your command invocation is equivalent to:
/bin/sh -c "rg" "term" "/home/tshiono/stackoverflow/221215" ...
where no arguments are fed to rg.
You need to pass the command as a string (not a list) or just drop shell=True.
Related
I am calling this piece of code, but it produces some output in the console where I ran the python script (due to tee command):
os.system("echo 3 | sudo tee /proc/sys/vm/drop_caches")
This version does not produce console output but is there another way?
os.system('sudo bash -c "echo 3 > /proc/sys/vm/drop_caches"')
To answer the question based on its title in the most generic form:
To suppress all output from os.system(), append >/dev/null 2>&1 to the shell command, which silences both stdout and stderr; e.g.:
import os
os.system('echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null 2>&1')
Note that os.system() by design passes output from the calling process' stdout and stderr streams through to the console (terminal) - your Python code never sees them.
Also, os.system() does not raise an exception if the shell command fails and instead returns an exit code; note that it takes additional work to extract the shell command's true exit code: you need to extract the high byte from the 16-bit value returned, by applying >> 8 (although you can rely on a return value other than 0 implying an error condition).
Given the above limitations of os.system(), it is generally worthwhile to use the functions in the subprocess module instead:
For instance, subprocess.check_output() could be used as follows:
import subprocess
subprocess.check_output('echo 3 | sudo tee /proc/sys/vm/drop_caches', shell=True)
The above will:
capture stdout output and return it (with the return value being ignored in the example above)
pass stderr output through; passing stderr=subprocess.STDOUT as an additional argument would also capture stderr.
raise an error, if the shell command fails.
Note: Python 3.5 introduced subprocess.run(), a more flexible successor to both os.system() and subprocess.check_output() - see https://docs.python.org/3.5/library/subprocess.html#using-the-subprocess-module
Note:
The reason that the OP is employing tee in the first place - despite not being interested in stdout output - is that a naïve attempt to use > ... instead would be interpreted before sudo is invoked, and thus fail, because the required privileges to write to /proc/sys/... haven't been granted yet.
Whether you're using os.system() or a subprocess function, stdin is not affected by default, so if you're invoking your script from a terminal, you'll get an interactive password prompt when the sudo command is encountered (unless the credentials have been cached).
Write directly to the proc pseudo file instead via Python i/o lib.
This will require your script to run as root (via sudo), which means you should limit its scope to being an admin only tool. This also allows the script to run on boxes where sudo requires a password.
Example:
with open("/proc/sys/vm/drop_caches", "w") as drop_caches:
drop_caches.write("3")
subprocess.check_call(command,stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
you forgot to add stderr.
You also can use this simple method of subprocess module.
command = 'echo 3 | sudo tee /proc/sys/vm/drop_caches'
subprocess.check_call(shlex.split(command),stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL)
All outputs will be passed to DEVNULL. Any issues with the command will be reported by an exception. No issues means no output.
I believe, the easiest way to hide the console output when it's not possible with os.system is using os.popen:
os.popen("echo 3 | sudo tee /proc/sys/vm/drop_caches")
If I run this command in ubuntu shell:
debconf-set-selections <<< 'postfix postfix/mailname string server.exmaple.com'
It runs successfully, but if I run it via python:
>>> from subprocess import run
>>> run("debconf-set-selections <<< 'postfix postfix/mailname string server.exmaple.com'", shell=True)
/bin/sh: 1: Syntax error: redirection unexpected
CompletedProcess(args="debconf-set-selections <<< 'postfix postfix/mailname string server.exmaple.com'", returncode=2)
I don't understand why python is trying to interpret whether there is redirection etc. How does one make the command successfully run so one can script installation of an application, e.g. postfix in this case via python (not a normal bash script)?
I have tried various forms with double and single quotes (as recommended in other posts), with no success.
subprocess uses /bin/sh as shell, and presumably your system's one does not support here-string (<<<), hence the error.
From subprocess source:
if shell:
# On Android the default shell is at '/system/bin/sh'.
unix_shell = ('/system/bin/sh' if
hasattr(sys, 'getandroidapilevel') else '/bin/sh')
You can run the command as an argument to any shell that supports here string e.g. bash:
run('bash -c "debconf-set-selections <<< \"postfix postfix/mailname string server.exmaple.com\""', shell=True)
Be careful with the quoting.
Or better you can stay POSIX and use echo and pipe to pass via STDIN:
run("echo 'postfix postfix/mailname string server.exmaple.com' | debconf-set-selections", shell=True)
I am calling this piece of code, but it produces some output in the console where I ran the python script (due to tee command):
os.system("echo 3 | sudo tee /proc/sys/vm/drop_caches")
This version does not produce console output but is there another way?
os.system('sudo bash -c "echo 3 > /proc/sys/vm/drop_caches"')
To answer the question based on its title in the most generic form:
To suppress all output from os.system(), append >/dev/null 2>&1 to the shell command, which silences both stdout and stderr; e.g.:
import os
os.system('echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null 2>&1')
Note that os.system() by design passes output from the calling process' stdout and stderr streams through to the console (terminal) - your Python code never sees them.
Also, os.system() does not raise an exception if the shell command fails and instead returns an exit code; note that it takes additional work to extract the shell command's true exit code: you need to extract the high byte from the 16-bit value returned, by applying >> 8 (although you can rely on a return value other than 0 implying an error condition).
Given the above limitations of os.system(), it is generally worthwhile to use the functions in the subprocess module instead:
For instance, subprocess.check_output() could be used as follows:
import subprocess
subprocess.check_output('echo 3 | sudo tee /proc/sys/vm/drop_caches', shell=True)
The above will:
capture stdout output and return it (with the return value being ignored in the example above)
pass stderr output through; passing stderr=subprocess.STDOUT as an additional argument would also capture stderr.
raise an error, if the shell command fails.
Note: Python 3.5 introduced subprocess.run(), a more flexible successor to both os.system() and subprocess.check_output() - see https://docs.python.org/3.5/library/subprocess.html#using-the-subprocess-module
Note:
The reason that the OP is employing tee in the first place - despite not being interested in stdout output - is that a naïve attempt to use > ... instead would be interpreted before sudo is invoked, and thus fail, because the required privileges to write to /proc/sys/... haven't been granted yet.
Whether you're using os.system() or a subprocess function, stdin is not affected by default, so if you're invoking your script from a terminal, you'll get an interactive password prompt when the sudo command is encountered (unless the credentials have been cached).
Write directly to the proc pseudo file instead via Python i/o lib.
This will require your script to run as root (via sudo), which means you should limit its scope to being an admin only tool. This also allows the script to run on boxes where sudo requires a password.
Example:
with open("/proc/sys/vm/drop_caches", "w") as drop_caches:
drop_caches.write("3")
subprocess.check_call(command,stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
you forgot to add stderr.
You also can use this simple method of subprocess module.
command = 'echo 3 | sudo tee /proc/sys/vm/drop_caches'
subprocess.check_call(shlex.split(command),stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL)
All outputs will be passed to DEVNULL. Any issues with the command will be reported by an exception. No issues means no output.
I believe, the easiest way to hide the console output when it's not possible with os.system is using os.popen:
os.popen("echo 3 | sudo tee /proc/sys/vm/drop_caches")
I'm trying to execute a rsync command via subrocess & popen. Everything's ok until I don't put the rsh subcommand where things go wrong.
from subprocess import Popen
args = ['-avz', '--rsh="ssh -C -p 22 -i /home/bond/.ssh/test"', 'bond#localhost:/home/bond/Bureau', '/home/bond/data/user/bond/backups/']
p = Popen(['rsync'] + args, shell=False)
print p.wait()
#just printing generated command:
print ' '.join(['rsync']+args)
I've tried to escape the '--rsh="ssh -C -p 22 -i /home/bond/.ssh/test"' in many ways, but it seems that it's not the problem.
I'm getting the error
rsync: Failed to exec ssh -C -p 22 -i /home/bond/.ssh/test: No such file or directory (2)
If I copy/paste the same args that I output at the time, I'm getting a correct execution of the command.
Thanks.
What happens if you use '--rsh=ssh -C -p 22 -i /home/bond/.ssh/test' instead (I removed the double quotes).
I suspect that this should work. What happens when you cut/paste your line into the commandline is that your shell sees the double quotes and removes them but uses them to prevent -C -p etc. from being interpreted as separate arguments. when you call subprocess.Popen with a list, you've already partitioned the arguments without the help of the shell, so you no longer need the quotes to preserve where the arguments should be split.
Having the same problem, I googled this issue extensively. It would seem you simply cannot pass arguments to ssh with subprocess. Ultimately, I wrote a shell script to run the rsync command, which I could pass arguments to via subprocess.call(['rsyncscript', src, dest, sshkey]). The shell script was: /usr/bin/rsync -az -e "ssh -i $3" $1 $2
This fixed the problem.
I am writing a small python script that needs to execute git commands from inside a given directory
The code is as follows:
import subprocess, os
pr = subprocess.Popen(['/usr/bin/git', 'status'],
cwd=os.path.dirname('/path/to/dir/'),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True)
(out, error) = pr.communicate()
print out
But it shows git usage as the output.
If the command doesn't involve git for eg. ['ls'] then it shows the correct output.
Is there anything I am missing ?
python version - 2.6.6
Thanks.
subprocess.Popen:
On Unix, with shell=True: […] If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself.
You don't want shell=True and also a list of arguments. Set shell=False.