I'm trying to copy files from one location to another using subprocess library and Popen method. When runing following script I'm getting the error cp: cannot stat /some/dev_path/*. I was told that the * is not expanded to the file names and that's where the problem is. Also in some other posts people were suggesting to use call instead of Popen, but call will not return stderr as far as I know.
devPath = '/some/dev_path/'
productionPath = '/some/prod_path/'
p = subprocess.Popen(['cp', '-r', devPath + '*', productionPath], stdout = subprocess.PIPE, stderr = subprocess.PIPE)
pout, perr = p.communicate()
if perr != '':
sys.exit('Error: ' + perr)
Expanding the * (globbing) is a function of your shell, bash for example. Therefore you'd have to use the keyword argument shell=True in your subprocess.Popen call.
However, for this case I'd strongly suggest to use shutil.copytree instead.
(First of all, because it's much simpler (see Zen of Python) and less error-prone. Dealing with errors is much cleaner, you get nice exceptions including a list of errors (for multi file operations like yours), and you don't have to deal with spawning a subprocess and communicating with it. Second, it's an unnecessary waste of resources to fork a child process if you don't need to. Other issues include quoting / escaping and possibly introducing security vulnerabilities into your code if you fail to properly sanitize user input.)
For example:
from shutil import copytree
from shutil import Error
try:
copytree('dir_a', 'dir_b')
except (Error, OSError), e:
print "Attempt to copy failed: %s" % e
Also, you shouldn't build filesystem paths by concatenating strings together, but instead use os.path.join(). That will use the correct directory separator (os.sep) for the current OS and allow you to easily write portable code.
Example:
>>> import os
>>> os.path.join('/usr/lib', 'python2.7')
'/usr/lib/python2.7'
Note: os.path.join still only does (smart) string manipulation - it doesn't care if that path is accessible or even exists.
Related
I wrote a method that is defined as below and works
def cmd_exec(cmd_tokens = []):
p = subprocess.Popen(cmd_tokens,
stdout=subprocess.PIPE,stderr=subprocess.PIPE)
out, err = p.communicate()
return (out, err)
I have a constant as LOAD_IMAGES=['docker', 'load', '-i', 'my_img_file_101']
When I execute the above method with LOAD_IMAGES as arguments, it works fine. However, the filename number might change for me and when I try to use a wildcard, I get the error. Say when I have LOAD_IMAGES=['docker', 'load', '-i', 'my_img_file*'], I get an error from the Py/Bash as open my_img_file*: no such file or directory
How do I make the wild card work. Executing the command directly on bash works.I mean when I say this on bash, it works docker load -i my_img_file*
Wildcard expansion is something bash takes care of while you're in the shell. It's not something built into Linux/Unix to be able to expand wildcards or any of that syntax. So you need to be explicit about it and do the expansion by hand.
There is an alternative, which is actually letting the shell do all the work, via shell=True. It has its drawbacks, as documented in the question. Quoting:
This is a good thing, see the warning block in the "Frequently Used Arguments" section, of the subprocess docs. It mainly discusses security implications, but can also helps avoid silly programming errors (as there are no magic shell characters to worry about)
My main complaint with shell=True is it usually implies there is a better way to go about the problem - with your example, you should use the glob module...
I need to do this:
paste file1 file2 file3 > result
I have the following in my python script:
from subprocess import call
// other code here.
// Here is how I call the shell command
call ["paste", "file1", "file2", "file3", ">", "result"])
Unfortunately I get this error:
paste: >: No such file or directory.
Any help with this will be great!
You need to implement the redirection yourself, if you're wisely deciding not to use a shell.
The docs at https://docs.python.org/2/library/subprocess.html warn you not to use a pipe -- but, you don't need to:
import subprocess
with open('result', 'w') as out:
subprocess.call(["paste", "file1", "file2", "file3"], stdout=out)
should be just fine.
There are two approaches to this.
Use shell=True:
call("paste file1 file2 file3 >result", shell=True)
Redirection, >, is a shell feature. Consequently, you can only access it when using a shell: shell=True.
Keep shell=False and use python to perform the redirection:
with open('results', 'w') as f:
subprocess.call(["paste", "file1", "file2", "file3"], stdout=f)
The second is normally preferred as it avoids the vagaries of the shell.
Discussion
When the shell is not used, > is just another character on the command line. Thus, consider the error message:
paste: >: No such file or directory.
This indicates that paste had received > as an argument and was trying to open a file by that name. No such file exists. Therefore the message.
As the shell command line, one can create a file by that name:
touch '>'
If such a file had existed, paste, when called by subprocess with shell=False, would have used that file for input.
If you don't mind adding an additional dependency in your code base you might consider installing the sh Python module (from PyPI:sh using pip, of course).
This is a rather clever wrapper around Python's subprocess module's functionality. Using sh your code would look something like:
#!/usr/bin/python
from sh import paste
paste('file1', 'file2', 'file3', _out='result')
... although I think you'd want some exception handling around that so you could use something like:
#!/usr/bin/python
from __future__ import print_function
import sys
from sh import paste
from tempfile import TemporaryFile
with tempfile.TemporaryFile() as err:
try:
paste('file1', 'file2', 'file3', _out='result', _err=err)
except (EnvironmentError, sh.ErrorReturnCode) as e:
err.seek(0)
print("Caught Error: %s" % err.read(), file=sys.stderr)
sh makes such things almost trivially easy although there are some tricks as you get more advanced. You also have to note the difference between _out= and other keyword arguments of that form, vs. sh's magic for most other keyword arguments.
All that sh magic make confuse anyone else who ever reads your code. You might also find that using Python modules with sh code interlaced into it makes you complacent about portability issues. Python code is generally fairly portable while Unix command line utilities can vary considerably from one OS to another and even from one Linux distribution or version to another. Having lots of shell utilities interlaced with your Python code in such a transparent way may make that problem less visible.
I am a bit confused as to how to get this done.
What I need to do is call an external command, from within a Python script, that takes as input several arguments, and a file name.
Let's call the executable that I am calling "prog", the input file "file", so the command line (in Bash terminal) looks like this:
$ prog --{arg1} {arg2} < {file}
In the above {arg1} is a string, and {arg2} is an integer.
If I use the following:
#!/usr/bin/python
import subprocess as sbp
sbp.call(["prog","--{arg1}","{arg2}","<","{file}"])
The result is an error output from "prog", where it claims that the input is missing {arg2}
The following produces an interesting error:
#!/usr/bin/python
import subprocess as sbp
sbp.call(["prog","--{arg1} {arg2} < {file}"])
all the spaces seem to have been removed from the second string, and equal sign appended at the very end:
command not found --{arg1}{arg2}<{file}=
None of this behavior seems to make any sense to me, and there isn't much that one can go by from the Python man pages found online. Please note that replacing sbp.call with sbp.Popen does not fix the problem.
The issue is that < {file} isn’t actually an argument to the program, but is syntax for the shell to set up redirection. You can tell Python to use the shell, or you can setup the redirection yourself.
from subprocess import *
# have shell interpret redirection
check_call('wc -l < /etc/hosts', shell=True)
# set up redirection in Python
with open('/etc/hosts', 'r') as f:
check_call(['wc', '-l'], stdin=f.fileno())
The advantage of the first method is that it’s faster and easier to type. There are a lot of disadvantages, though: it’s potentially slower since you’re launching a shell; it’s potentially non-portable because it depends on the operating system shell’s syntax; and it can easily break when there are spaces or other special characters in filenames.
So the second method is preferred.
I am trying to delete a file, then rename a file to the deleted file in python.
import sys
import subprocess
fileInput = sys.argv[1]
|
|
#code to create fileInput.tmp
|
|
ret=subprocess.Popen("rm "+fileInput,shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
print ret
ret1=subprocess.Popen("mv "+ fileInput+".tmp "+fileInput,shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
print ret1
Whats happening is sometimes (not always) both fileInput and fileInput.tmp are being deleted in "ret=" step and "ret1=" step doesn't execute.
Can someone suggest why is it happeing. This code is run on MacOSx
Ok, not answering the question directly but providing a better alternative.
Using subprocess here is a very bad idea. It is very easy here to end up doing horrible things; you'll at least need to escape the arguments passed to shell.
Instead, use what Python gives you:
os.rename(fileInput + '.tmp', fileInput)
This is the atomic move/rename operation. And you don't have to rm file before replacing it. Moreover, you usually don't do that because between the rm and mv calls there will be no file with that name. If you just use mv, the replacement will be atomic and some file will always be there.
That's all the short story, I think.
As a side note, os.rename() doesn't work across filesystems. So while using it for renaming a file is ok (the source and destination are in the same directory), if you're moving files across different directories you'd rather use:
import shutil
shutil.move(src, dst)
The first subprocess is not completing, do this:
p = subprocess.Popen("...")
p.communicate() # waits until the subprocess completes
Thanks in advance for any help. I am new to python, but not particularly new to scripting. I am trying to run a simple, automated email program, but the email module seems to be installed incorrectly on our system (I don't have 75% of the functions described in the python examples, only "message_from_string" and "message_from_file") and smtplib is overly complicated for what I need.
In fact, in simple bash terms, all I need is:
/bin/email -s "blah" "recipients" < file.with.body.info.txt
or,
echo "my body details" | /bin/email -s "blah" "recipients"
so that I can avoid having to write to a file just to send a message.
I tried using subprocess, either call or Popen, and the only way I could eventually get things to work is if I used:
subprocess.call('/bin/mail -s "blah" "recipients" < file.with.body.info.txt', shell=True)
A few things I specifically don't like about this method:
(1) I couldn't break things into a list or tuple, as it is supposed to work, so that I lost the whole advantage of subprocess, as I understand it, in keeping things secure. If I tried:
subprocess.call(['/bin/mail', '-s', subjVariable, recipVariable, '<', 'file.with.body.info.txt'], shell=True)
it would fail. Similarly, if I tried to use the pipe, '|', instead of reading from a file, it would fail. It was also failing if I used '-cmd' instead of a pipe. The "fail" was usually that it would read '<' and 'file.with.body.info.txt' as if they were further recipients. In other words, whether I said "shell = True" or not, subprocess was not able to interpret the special characters in the call as the special characters that they are. '<' wasn't recognized as an input from a file, etc., unless I kept everything in one large call.
What I would ideally like to be able to do, because it seems more secure, as well as more flexible, is something like this:
subprocess.call(['/bin/echo', varWithBody, '|', '/bin/mail', '-s', subjVariable, recipVariable,])
but it seems that pipes are not understood at all with subprocess and I cannot figure out how to pipe things together while stuck behind python.
Any suggestions? All help is welcome, except attempts to explain how to use the 'email' or 'smtplib' modules. Regardless of this particular application, I really want to learn how to use subprocess better, so that I can tie together disparate programs. My understanding is that python should be fairly decent at that.
Thanks! Mike
The Python docs seem to cover this situation.
What I'd probably do is something like the following
from subprocess import *
readBody = Popen(["/bin/echo", varWithBody], stdout=PIPE)
mail = Popen(["/bin/mail", "-s", subjVariable, recipVariable], stdin=readBody.stdout, stdout=PIPE)
output = mail.communicate()[0]
| and < are not arguments; they are shell redirections. To replace the | in your code, see these instructions.
To replace <, use:
subprocess.Popen(["command", "args"], stdin=open("file.txt", 'r'))
eg.
subprocess.Popen(["cat"], stdin=open("file.txt", 'r')) is the same as cat < file.txt
<, | etc are features of the shell, not the operating system. Therefore something like subprocess won't know anything about them - internally it's just passing the list of arguments to the equivalent OS functions. The way to do input/output redirection using subprocess is using the stdin, stdout and strerr parameters. You can pass in a file object (it has to contain a file descriptor, though, but normally opened files always do) or a naked file descriptor. Or a pipe object.
The manual has an example for replacing a pipeline, just replace the pipe with a file object and you should be all set.
You need to run the command through the shell using the shell argument:
>>> import subprocess
>>> subprocess.call('ls -a | cat', shell=True)
.
..
.git
.gitignore
doc
generate_rands.py
infile1
infile2
infile3
matrix.pyc
matrix.py~
median.py
problems
simple_median.py
test
test_matrix.py
test_matrix.py~
test_median.py