I have subprocess command to check md5 checksum as
subprocess.check_output('md5 Downloads/test.txt', stderr=subprocess.STDOUT, shell=True)
It works fine.
But I read try to avoid shell=True
but when I run
subprocess.check_output('md5 Downloads/test.txt', stderr=subprocess.STDOUT, shell=False)
I get error OSError: [Errno 2] No such file or directory
Can I run above command or workaround with shell=False or it's ok to keep shell=True?
Just pass the arguments to check_output() as a list:
subprocess.check_output(["md5", "Downloads/test.txt"], stderr=subprocess.STDOUT)
From the docs:
args is required for all calls and should be a string, or a sequence
of program arguments. Providing a sequence of arguments is generally
preferred, as it allows the module to take care of any required
escaping and quoting of arguments (e.g. to permit spaces in file
names). If passing a single string, either shell must be True (see
below) or else the string must simply name the program to be executed
without specifying any arguments.
in case of complex commands , you can use shlex to pass the commands as a list to Check_Output or any other subprocess classes
from the document
shlex.split() can be useful when determining the correct tokenization for args, especially in complex cases:
https://docs.python.org/3.6/library/subprocess.html#subprocess.check_output
coming to above example
import shlex
inp="md5 Downloads/test.txt"
command=shlex.split(inp)
subprocess.check_output(command, stderr=subprocess.STDOUT)
Related
How can I pass multiple arguments within my subprocess call in python while running my shell script?
import subprocess
subprocess.check_call('./test_bash.sh '+arg1 +arg2, shell=True)
This prints out the arg1 and arg2 concatenated as one argument. I would need to pass 3 arguments to my shell script.
of course it concatenates because you're not inserting a space between them. Quickfix would be (with format, and can fail if some arguments contain spaces)
subprocess.check_call('./test_bash.sh {} {}'.format(arg1,arg2), shell=True)
can you try (more robust, no need to quote spaces, automatic command line generation):
check_call(['./test_bash.sh',arg1,arg2],shell=True)`
(not sure it works on all systems because of shell=True and argument list used together)
or drop shell=True and call the shell explicitly (may fail because shebang is not considered, unlike with shell=True, but it's worth giving up shell=True, liable to code injection among other issues):
check_call(['sh','-c','./test_bash.sh',arg1,arg2])
import os
import subprocess
proc = subprocess.Popen(['ls','*.bc'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out,err = proc.communicate()
print out
This script should print all the files with .bc suffix however it returns an empty list. If I do ls *.bc manually in the command line it works. Doing ['ls','test.bc'] inside the script works as well but for some reason the star symbol doesnt work.. Any ideas ?
You need to supply shell=True to execute the command through a shell interpreter.
If you do that however, you can no longer supply a list as the first argument, because the arguments will get quoted then. Instead, specify the raw commandline as you want it to be passed to the shell:
proc = subprocess.Popen('ls *.bc', shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
Expanding the * glob is part of the shell, but by default subprocess does not send your commands via a shell, so the command (first argument, ls) is executed, then a literal * is used as an argument.
This is a good thing, see the warning block in the "Frequently Used Arguments" section, of the subprocess docs. It mainly discusses security implications, but can also helps avoid silly programming errors (as there are no magic shell characters to worry about)
My main complaint with shell=True is it usually implies there is a better way to go about the problem - with your example, you should use the glob module:
import glob
files = glob.glob("*.bc")
print files # ['file1.bc', 'file2.bc']
This will be quicker (no process startup overhead), more reliable and cross platform (not dependent on the platform having an ls command)
Besides doing shell=True, also make sure that your path is not quoted. Otherwise it will not be expanded by shell.
If your path may have special characters, you will have to escape them manually.
In the documentaiton for Popen I read:
class subprocess.Popen(args, bufsize=0, ...)
args should be a sequence of program arguments or else a single
string. [...] Unless otherwise stated, it is recommended to pass args
as a sequence.
Why is it recommended to use a sequence for args? What are the cases when I must use a single string?
On Unix a single string argument to Popen only works if you're not passing arguments to the program. Otherwise you would need shell=True. That's because the string is interpreted as the name of the program to execute.
Using a sequence also tends to be more secure. If you get program arguments from the user, you must fully sanitize them before appending to the command string. Otherwise the user will be able to pass arbitrary commands for execution. Consider the following example:
>>> def ping(host):
... cmd = "ping -c 1 {}".format(host)
... Popen(cmd, shell=True)
...
>>> ping(input())
8.8.8.8; cat /etc/passwd
Using a sequence for args helps to avoid such vulnerabilities.
Inside a subprocess call, I want to use shell=True so that it does globbing on pathnames (code below), however this has the annoying side-effect of making subprocess spawn a child process (which must then be `communicate()d/ poll()ed/ wait()ed/ terminate()d/ kill()ed/ whatevah).
(Yes I am aware the globbing can also be done with fnmatch/glob, but please show me the 'correct' use of subprocess on this, i.e. the minimal incantation to both get the stdout and stop the child process.)
This works fine (returns output):
subprocess.check_output(['/usr/bin/wc','-l','[A-Z]*/[A-Z]*.F*'], shell=False)
but this hangs
subprocess.check_output(['/usr/bin/wc','-l','[A-Z]*/[A-Z]*.F*'], shell=True)
(PS: It's seriously aggravating that you can't tell subprocess you want some but not all shell functionality e.g. globbing but not spawning. I think there's a worthy PEP in that, if anyone cares to comment, i.e. pass in a tuple of Boolean, or an or of binary flags)
(PPS: the idiom of whether you pass subprocess...(cmdstring.split() or [...]) is just a trivial idiomatic difference. I say tomato, you say tomay-to. In my case, the motivation is the command is fixed but I may want to call it more than once with a difference filespec.)
First off -- there's very little point to passing an array to:
subprocess.check_output(['/usr/bin/wc','-l','A-Z*/A-Z*.F*'], shell=True)
...as this simply runs wc with no arguments, in a shell also passed arguments -l and A-Z*/A-Z*.F* as arguments (to the shell, not to wc). Instead, you want:
subprocess.check_output('/usr/bin/wc -l A-Z*/A-Z*.F*', shell=True)
Before being corrected, this would hang because wc had no arguments and was reading from stdin. I would suggest ensuring that stdin is passed in closed, rather than passing along your Python program's stdin (as is the default behavior).
An easy way to do this, since you have shell=True:
subprocess.check_output(
'/usr/bin/wc -l A-Z*/A-Z*.F* </dev/null',
shell=True)
...alternately:
p = subprocess.Popen('/usr/bin/wc -l A-Z*/A-Z*.F*', shell=True,
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=None)
(output, _) = p.communicate(input='')
...which will ensure an empty stdin from Python code rather than relying on the shell.
For example:
import subprocess
p=subprocess.Popen("imp -help",stdout=subprocess.PIPE,stdin=subprocess.PIPE)
out,err=p.communicate
the out is null
but other oracle command like "sqlplus -help","rman -help" works fine
There could be two problems why you are not getting any output in stdout:
The process is dumping all it's output to stderr.
The system does not know how to execute "imp -help".
The solution for the first problem is easy: capture stderr using the argument stderr = subprocess.PIPE.
The solution to the second is also easy, but the explanation is a bit longer: Subprocess does not guess much, it will just try to execute the whole string as one command. That means, in your case, it will try to execute "imp -help" as one command. It does not try to execute the command "imp" with the argument "-help". You have to explicitly tell subprocess the command and the arguments separately.
From the python documentation on subprocess:
args should be a string, or a sequence
of program arguments. The program to
execute is normally the first item in
the args sequence or the string if a
string is given, ...
That means you have to separate the command and the arguments and pack them together in a sequence. This: "imp -help" should look like this: ["imp", "-help"]. Read the documentation on subprocess for more details on the intricacies of spliting the command and arguments.
Here is how the code should look like:
import subprocess
p=subprocess.Popen(["imp", "-help"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
out,err=p.communicate()
Note: you also typed p.communicate instead of p.communicate(). I assume that was a typo in your question, not in your code.