I wrote a method that is defined as below and works
def cmd_exec(cmd_tokens = []):
p = subprocess.Popen(cmd_tokens,
stdout=subprocess.PIPE,stderr=subprocess.PIPE)
out, err = p.communicate()
return (out, err)
I have a constant as LOAD_IMAGES=['docker', 'load', '-i', 'my_img_file_101']
When I execute the above method with LOAD_IMAGES as arguments, it works fine. However, the filename number might change for me and when I try to use a wildcard, I get the error. Say when I have LOAD_IMAGES=['docker', 'load', '-i', 'my_img_file*'], I get an error from the Py/Bash as open my_img_file*: no such file or directory
How do I make the wild card work. Executing the command directly on bash works.I mean when I say this on bash, it works docker load -i my_img_file*
Wildcard expansion is something bash takes care of while you're in the shell. It's not something built into Linux/Unix to be able to expand wildcards or any of that syntax. So you need to be explicit about it and do the expansion by hand.
There is an alternative, which is actually letting the shell do all the work, via shell=True. It has its drawbacks, as documented in the question. Quoting:
This is a good thing, see the warning block in the "Frequently Used Arguments" section, of the subprocess docs. It mainly discusses security implications, but can also helps avoid silly programming errors (as there are no magic shell characters to worry about)
My main complaint with shell=True is it usually implies there is a better way to go about the problem - with your example, you should use the glob module...
Related
I am writing a script to open notepad.exe using subprocess.Popen()
import subprocess
command = '%windir%\system32\\notepad.exe'
process = subprocess.Popen(command)
output = process.communicate()
print(output[0])
This throws a FileNotFoundError
Is it possible to change/add to the above code to make it work with relative paths?
I did try to run the script from C:\Windows> after moving it there, which again failed. Also set the shell=True, but failed as well.
Writing a similar script using os.popen() works ok with relative paths, regardless which directory the script is run from, but as far as I understand popen is not the way forward..
Early steps in the world of programming/Python. Any input much appreciated.
Use os.path.expandvars to expand %windir%:
command = os.path.expandvars('%windir%\\system32\\notepad.exe')
The result is a path that then can be passed to subprocess.Popen.
subprocess.Popen does not expand environment variables such as %windir%. The shell might but you really should not depend on shell=True to do that.
Pro tip: whenever you get an error asking the system to execute a command, print the command (and, if applicable, the current working directory). The results will often surprise you.
In your case, I suspect you're just missing a backslash. Use this instead:
command = '%windir%\\system32\\notepad.exe'
Before you make that change, try printing the value of command immediately after assignment. I think you'll find the leading "s" in "system" is missing, and that the mistake is obvious.
HTH.
You could use raw strings to avoid having to double-up your backslashes.
command = r'%windir%\system32\notepad.exe'
I'm doing this simple thing
import subprocess
with cd("/home/myuserid"):
subprocess.call("ls ")
where cd is taken from here and it just does not work (same with any other path):
OSError: [Errno 2] No such file or directory
You have an extra space after ls, which is causing your issue. Remove that and it should work fine.
import subprocess
with cd("/home/myuserid"):
subprocess.call("ls")
When you use subprocess without shell=True, it interprets the entire string you pass as the command to execute. So it looks for a program literally called "ls " when you provide that extra space, which of course doesn't exist.
If you were to use shell=True, it would work fine even with the extra space, because a /bin/sh shell would be used to run the command, and the shell wouldn't care about the extra space. In general it's safer to use the default of shell=False, though, so I'd stick with that.
I am trying to call a shell (Bash) script from python. The script is in my /home/user/bin directory with execute permission for group & user, i.e., -rwxr-xr--. I am using subprocess.check_call(["/home/user/bin/script.sh %s %s" % (subj,-6)],shell=True) and this is generating an exit status 127 code. Adding stderr=subprocess.STDOUT to the command does nothing to elucidate. Here is the exact output:
CalledProcessError: Command
'['/home/.../bin/MNE_setup_source_space.sh kubi_td104 -6']'
returned non-zero exit status 127`
I believe this might be a PATH related issue, is that correct? I don't know how to resolve this. If I am already passing in the absolute path to the executable how can there be a PATH issue?
Thanks in advance
Do not use shell=True. Do not pass arguments as part of argv[0]. Pass your argument vector as a vector -- which is to say, in Python, a list:
subprocess.check_call(["/home/user/bin/script.sh", str(subj), "-6"])
If you were going to use shell=True, you would do it like so:
subprocess.check_call("/home/user/bin/script.sh %s %s" % (subj,-6), shell=True)
...which is to say, you wouldn't use a list form at all.
To clarify why what you're currently trying is failing -- because you're using shell=True, it's trying to pass only the first list element as a script, and additional arguments as extra argv elements which would only be read or interpreted if the script passed in the first argument chose to look at them (as by referring to "$1", "$2", or the like).
shell=True is only needed in very rare circumstances where you need a shell to perform redirections or logic before starting the program you're trying to run, and comes with serious security concerns if any unvetted input is incorporated into the command being run. Do not use it unless you're very, very sure you need to.
According to the documentation for the subprocess module, its default shell is /bin/sh, but I have an ingrained, and probably irrational, aversion to hard-coding such constants.
Therefore, I much prefer to refer to some constant defined in subprocess. I have not been able to find any way to interrogate subprocess directly for this constant. The best I've managed is
def _getshpath():
return subprocess.check_output('which sh', shell=True).strip()
or
def _getshpath():
return subprocess.check_output('echo "$0"', shell=True).strip()
...both of which look pathetically fragile, since their validity ultimately depend on precisely the specific value I'm trying to determine in the first place. (I.e., if the value of this executable is not "/bin/sh", either definition could easily be nonsensical.)
What's best-practice for getting this path (without hard-coding it as "/bin/sh")?
Thanks!
Hard-coding it as /bin/sh is perfectly valid. If you look at the documentation for C's popen() you'll find it does this too. /bin/sh is, by construction, the system's standard shell.
You could try
>>> import os
>>> shell = os.environ['SHELL']
>>> print shell
'/bin/bash'
You can use this to set the executable argument of subprocess.Popen.
Thanks in advance for any help. I am new to python, but not particularly new to scripting. I am trying to run a simple, automated email program, but the email module seems to be installed incorrectly on our system (I don't have 75% of the functions described in the python examples, only "message_from_string" and "message_from_file") and smtplib is overly complicated for what I need.
In fact, in simple bash terms, all I need is:
/bin/email -s "blah" "recipients" < file.with.body.info.txt
or,
echo "my body details" | /bin/email -s "blah" "recipients"
so that I can avoid having to write to a file just to send a message.
I tried using subprocess, either call or Popen, and the only way I could eventually get things to work is if I used:
subprocess.call('/bin/mail -s "blah" "recipients" < file.with.body.info.txt', shell=True)
A few things I specifically don't like about this method:
(1) I couldn't break things into a list or tuple, as it is supposed to work, so that I lost the whole advantage of subprocess, as I understand it, in keeping things secure. If I tried:
subprocess.call(['/bin/mail', '-s', subjVariable, recipVariable, '<', 'file.with.body.info.txt'], shell=True)
it would fail. Similarly, if I tried to use the pipe, '|', instead of reading from a file, it would fail. It was also failing if I used '-cmd' instead of a pipe. The "fail" was usually that it would read '<' and 'file.with.body.info.txt' as if they were further recipients. In other words, whether I said "shell = True" or not, subprocess was not able to interpret the special characters in the call as the special characters that they are. '<' wasn't recognized as an input from a file, etc., unless I kept everything in one large call.
What I would ideally like to be able to do, because it seems more secure, as well as more flexible, is something like this:
subprocess.call(['/bin/echo', varWithBody, '|', '/bin/mail', '-s', subjVariable, recipVariable,])
but it seems that pipes are not understood at all with subprocess and I cannot figure out how to pipe things together while stuck behind python.
Any suggestions? All help is welcome, except attempts to explain how to use the 'email' or 'smtplib' modules. Regardless of this particular application, I really want to learn how to use subprocess better, so that I can tie together disparate programs. My understanding is that python should be fairly decent at that.
Thanks! Mike
The Python docs seem to cover this situation.
What I'd probably do is something like the following
from subprocess import *
readBody = Popen(["/bin/echo", varWithBody], stdout=PIPE)
mail = Popen(["/bin/mail", "-s", subjVariable, recipVariable], stdin=readBody.stdout, stdout=PIPE)
output = mail.communicate()[0]
| and < are not arguments; they are shell redirections. To replace the | in your code, see these instructions.
To replace <, use:
subprocess.Popen(["command", "args"], stdin=open("file.txt", 'r'))
eg.
subprocess.Popen(["cat"], stdin=open("file.txt", 'r')) is the same as cat < file.txt
<, | etc are features of the shell, not the operating system. Therefore something like subprocess won't know anything about them - internally it's just passing the list of arguments to the equivalent OS functions. The way to do input/output redirection using subprocess is using the stdin, stdout and strerr parameters. You can pass in a file object (it has to contain a file descriptor, though, but normally opened files always do) or a naked file descriptor. Or a pipe object.
The manual has an example for replacing a pipeline, just replace the pipe with a file object and you should be all set.
You need to run the command through the shell using the shell argument:
>>> import subprocess
>>> subprocess.call('ls -a | cat', shell=True)
.
..
.git
.gitignore
doc
generate_rands.py
infile1
infile2
infile3
matrix.pyc
matrix.py~
median.py
problems
simple_median.py
test
test_matrix.py
test_matrix.py~
test_median.py