I am trying to delete a file, then rename a file to the deleted file in python.
import sys
import subprocess
fileInput = sys.argv[1]
|
|
#code to create fileInput.tmp
|
|
ret=subprocess.Popen("rm "+fileInput,shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
print ret
ret1=subprocess.Popen("mv "+ fileInput+".tmp "+fileInput,shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
print ret1
Whats happening is sometimes (not always) both fileInput and fileInput.tmp are being deleted in "ret=" step and "ret1=" step doesn't execute.
Can someone suggest why is it happeing. This code is run on MacOSx
Ok, not answering the question directly but providing a better alternative.
Using subprocess here is a very bad idea. It is very easy here to end up doing horrible things; you'll at least need to escape the arguments passed to shell.
Instead, use what Python gives you:
os.rename(fileInput + '.tmp', fileInput)
This is the atomic move/rename operation. And you don't have to rm file before replacing it. Moreover, you usually don't do that because between the rm and mv calls there will be no file with that name. If you just use mv, the replacement will be atomic and some file will always be there.
That's all the short story, I think.
As a side note, os.rename() doesn't work across filesystems. So while using it for renaming a file is ok (the source and destination are in the same directory), if you're moving files across different directories you'd rather use:
import shutil
shutil.move(src, dst)
The first subprocess is not completing, do this:
p = subprocess.Popen("...")
p.communicate() # waits until the subprocess completes
Related
I want to call a command like
scp username#hostname:/dir/to/files/\{a,b,c\} /target/dir
from Python to copy many files in one command.
The command works perfectly if entered directly into the shell.
But if I use
import subprocess
p = subprocess.Popen(['scp', 'username#hostname:/dir/to/files/\{a,b,c\}',
'/target/dir'])
sts = os.waitpid(p.pid, 0)
I get the error
scp: /dir/to/files/{a,b,c}: No such file or directory
Obviously, the backslashes are missing. And if I use double backslashes in the Popen arguments like
\\{a,b,c,d\\}
I get the error
scp: /dir/to/target/\a: No such file or directory
scp: /dir/to/target/\b: No such file or directory
scp: /dir/to/target/\c\: No such file or directory
Nothing changes if I use raw strings like r'\{' + r'\}'
How can I call the scp command from Python with the correctly escaped curly braces '\\{a,b,c\\}'?
cannot test, but I would remove all blackslashes altogether since they're just here to protect the expansion from the shell on the local machine:
import subprocess
p = subprocess.Popen(['scp', 'username#hostname:/dir/to/files/{a,b,c}',
'/target/dir'])
sts = p.wait()
also note that p.wait() is way better than the wait command you performed (more portable!)
Jean-Francois Fabre got me on the right track:
import subprocess
p = subprocess.Popen('scp username#hostname:/dir/to/files/\{a,b,c\} /target/dir',
shell=True)
sts = p.wait()
The
shell=True
argument was the missing bit. It is a solution that is not recommended, but at least it's working.
Don't use a shell feature unnecessarily in a script; you have your text editor to make typing easier. Just pass the three file names individually:
p = subprocess.Popen(['scp',
'username#hostname:/dir/to/files/a',
'username#hostname:/dir/to/files/b',
'username#hostname:/dir/to/files/c',
'/target/dir'])
Alternatively, let Python build the list of files for you.
file_list = ['username#hostname:/dir/to/files/%s' % (s,)
for f in ['a', 'b', 'c']]
p = subprocess.Popen(['scp'] + file_list + ['/target/dir'])
If I'm getting it correctly, you need to have the "\" and the "{". Since you need to escape both of them, what about 'username#hostname:/dir/to/files/\\\{a,b,c\\\}'
I want to call a subprocess in python using subprocess.call(), with the 'cwd' argument so that this particular subprocess is executed in a different directory. I don't want to use os.chdir() because for future processes later in the program I want to remain in the original directory from where the program was run.
BUT, I also want to run this particular subprocess on a set of files matching a glob pattern. So for example, I might want to do
subprocess.call(['ls'] + glob('*.txt'), cwd="/my/other/dir/")
But of course the glob command doesn't know to look in /my/other/dir, so it fails. How can I do this without using shell=True?
You could use the CWD in the glob pattern as well. Like glob.glob("/my/other/dir/*.txt"). It will expand with full match, like /my/other/dir/aa.txt. In case you do not want to pass the full path to the executable, cut it off.
CWD = "/my/other/dir/"
files = map(lambda x: x[len(CWD):], glob.glob(CWD + "*.txt"))
subprocess.call(['ls'] + files, cwd=CWD)
Or you could just change the directory back after the subprocess has finished.
I need to do this:
paste file1 file2 file3 > result
I have the following in my python script:
from subprocess import call
// other code here.
// Here is how I call the shell command
call ["paste", "file1", "file2", "file3", ">", "result"])
Unfortunately I get this error:
paste: >: No such file or directory.
Any help with this will be great!
You need to implement the redirection yourself, if you're wisely deciding not to use a shell.
The docs at https://docs.python.org/2/library/subprocess.html warn you not to use a pipe -- but, you don't need to:
import subprocess
with open('result', 'w') as out:
subprocess.call(["paste", "file1", "file2", "file3"], stdout=out)
should be just fine.
There are two approaches to this.
Use shell=True:
call("paste file1 file2 file3 >result", shell=True)
Redirection, >, is a shell feature. Consequently, you can only access it when using a shell: shell=True.
Keep shell=False and use python to perform the redirection:
with open('results', 'w') as f:
subprocess.call(["paste", "file1", "file2", "file3"], stdout=f)
The second is normally preferred as it avoids the vagaries of the shell.
Discussion
When the shell is not used, > is just another character on the command line. Thus, consider the error message:
paste: >: No such file or directory.
This indicates that paste had received > as an argument and was trying to open a file by that name. No such file exists. Therefore the message.
As the shell command line, one can create a file by that name:
touch '>'
If such a file had existed, paste, when called by subprocess with shell=False, would have used that file for input.
If you don't mind adding an additional dependency in your code base you might consider installing the sh Python module (from PyPI:sh using pip, of course).
This is a rather clever wrapper around Python's subprocess module's functionality. Using sh your code would look something like:
#!/usr/bin/python
from sh import paste
paste('file1', 'file2', 'file3', _out='result')
... although I think you'd want some exception handling around that so you could use something like:
#!/usr/bin/python
from __future__ import print_function
import sys
from sh import paste
from tempfile import TemporaryFile
with tempfile.TemporaryFile() as err:
try:
paste('file1', 'file2', 'file3', _out='result', _err=err)
except (EnvironmentError, sh.ErrorReturnCode) as e:
err.seek(0)
print("Caught Error: %s" % err.read(), file=sys.stderr)
sh makes such things almost trivially easy although there are some tricks as you get more advanced. You also have to note the difference between _out= and other keyword arguments of that form, vs. sh's magic for most other keyword arguments.
All that sh magic make confuse anyone else who ever reads your code. You might also find that using Python modules with sh code interlaced into it makes you complacent about portability issues. Python code is generally fairly portable while Unix command line utilities can vary considerably from one OS to another and even from one Linux distribution or version to another. Having lots of shell utilities interlaced with your Python code in such a transparent way may make that problem less visible.
I'm doing this simple thing
import subprocess
with cd("/home/myuserid"):
subprocess.call("ls ")
where cd is taken from here and it just does not work (same with any other path):
OSError: [Errno 2] No such file or directory
You have an extra space after ls, which is causing your issue. Remove that and it should work fine.
import subprocess
with cd("/home/myuserid"):
subprocess.call("ls")
When you use subprocess without shell=True, it interprets the entire string you pass as the command to execute. So it looks for a program literally called "ls " when you provide that extra space, which of course doesn't exist.
If you were to use shell=True, it would work fine even with the extra space, because a /bin/sh shell would be used to run the command, and the shell wouldn't care about the extra space. In general it's safer to use the default of shell=False, though, so I'd stick with that.
I'm trying to copy files from one location to another using subprocess library and Popen method. When runing following script I'm getting the error cp: cannot stat /some/dev_path/*. I was told that the * is not expanded to the file names and that's where the problem is. Also in some other posts people were suggesting to use call instead of Popen, but call will not return stderr as far as I know.
devPath = '/some/dev_path/'
productionPath = '/some/prod_path/'
p = subprocess.Popen(['cp', '-r', devPath + '*', productionPath], stdout = subprocess.PIPE, stderr = subprocess.PIPE)
pout, perr = p.communicate()
if perr != '':
sys.exit('Error: ' + perr)
Expanding the * (globbing) is a function of your shell, bash for example. Therefore you'd have to use the keyword argument shell=True in your subprocess.Popen call.
However, for this case I'd strongly suggest to use shutil.copytree instead.
(First of all, because it's much simpler (see Zen of Python) and less error-prone. Dealing with errors is much cleaner, you get nice exceptions including a list of errors (for multi file operations like yours), and you don't have to deal with spawning a subprocess and communicating with it. Second, it's an unnecessary waste of resources to fork a child process if you don't need to. Other issues include quoting / escaping and possibly introducing security vulnerabilities into your code if you fail to properly sanitize user input.)
For example:
from shutil import copytree
from shutil import Error
try:
copytree('dir_a', 'dir_b')
except (Error, OSError), e:
print "Attempt to copy failed: %s" % e
Also, you shouldn't build filesystem paths by concatenating strings together, but instead use os.path.join(). That will use the correct directory separator (os.sep) for the current OS and allow you to easily write portable code.
Example:
>>> import os
>>> os.path.join('/usr/lib', 'python2.7')
'/usr/lib/python2.7'
Note: os.path.join still only does (smart) string manipulation - it doesn't care if that path is accessible or even exists.