I want to remove all the files and directories except for some of them by using
`subprocess.call(['rm','-r','!(new_models|creat_model.py|my_mos.tit)'])`
but it gives back information
rm: cannot remove `!(new_models|creat_model.py|my_mos.tit)': No such file or directory
how can I fix this? Thanks
If you use that rm command on the command line the !(…|…|…) pattern is expanded by the shell into all file names except those in the pattern before calling rm. Your code calls rm directly so rm gets the shell pattern as a file name and tries to delete a file with that name.
You have to add shell=True to the argument list of subprocess.call() or actually code this in Python instead of calling external commands. Downside: That would be more than one line. Upside: it can be done independently from external shells and system dependent external programs.
An alternative to shell=True could be the usage of glob and manual filtering:
import glob
files = [i for i in glob.glob("*") if i not in ('new_models', 'creat_model.py', 'my_mos.tit')]
subprocess.call(['rm','-r'] + files)
Edit 4 years later:
Without glob (of which I don't remember why I suggested it):
import os
files = [i for i in os.listdir() if i not in ('new_models', 'creat_model.py', 'my_mos.tit')]
subprocess.call(['rm','-r'] + files)
Code to remove all png
args = ('rm', '-rf', '/path/to/files/temp/*.png')
subprocess.call('%s %s %s' % args, shell=True)
Related
I'm trying to invoke tar command via subprocess call from Python.The challenge I have is there are a lot files that get passed on to tar which is causing the command to throw the error Argument list too long: '/bin/sh'
The command I'm running is below
subprocess.call(f"ulimit -s 99999999; tar -cz -f {output_file} {file_list}", cwd=source_dir, shell=True)
To try to overcome the error, I added ulimit which doesn't seem to help. The OS I am running this on is Ubuntu-20.04 & Pyhon version is 3.8
Please could I get help to solve this problem.
ulimit does nothing to lift the kernel constant ARG_MAX which is what you are bumping into here. In fact, the only way to increase it is typically to recompile your kernel.
If your tar supports --files-from -, use that.
subprocess.check_call(
['tar', '-cz', '-f', output_file, '--files-from', '-'],
input='\n'.join(file_list), cwd=source_dir)
I obviously made assumptions about the contents of file_list (in particular, this will break if you have files whose name contains a newline character). Notice also how I avoid shell=True by passing in the command as a list of strings.
Of course, a much better solution for this use case is to use the Python tarfile module to create the tar file; this entirely avoids the need to transmit the list of file names across a process boundary.
import tarfile
with tarfile.open(output_file, "x:gz") as tar:
for name in file_list:
tar.add(name)
The "x:gz" mode of creation triggers an exception if the file already exists (use "w:gz" to simply overwrite).
I'm trying to convert a file from .m4a to .mp3 using ffmpeg and I need to access to the music folder.
The path name of this folder is : C:\\Users\A B\Desktop\Music
I can't access it with subprocess.call() because only C:\\Users\A gets recognized. The white space is not processed.
Here's my python script :
import constants
import os
import subprocess
path = 'C:\\Users\A B\Desktop\Music'
def main():
files = sorted(os.listdir(path), key=lambda x: os.path.getctime(os.path.join(path, x)))
if "Thumbs.db" in files: files.remove("Thumbs.db")
for f in files:
if f.lower()[-3:] == "m4a":
process(f)
def process(f):
inFile = f
outFile = f[:-3] + "mp3"
subprocess.call('ffmpeg -i {} {} {}'.format('C:\\Users\A B\Desktop\Music', inFile, outFile))
main()
When I run it I get an error that states :
C:\Users\A: No such file or directory
I wonder if someones knows how to put my full path name (C:\Users\A B\Desktop\Music) in subprocess.call() ?
Beforehand edit: spaces or not, the following command line -i <directory> <infilename> <outfilename> is not correct for ffmpeg since it expects the -i option, then input file and output file, not a directory first. So you have more than one problem here (which explains the "permission denied" message you had, because ffmpeg was trying to open a directory as a file!)
I suppose that you want to:
read all files from directory
convert them all to a file located in the same directory
In that case, you could add quotes to your both input & output absolute files like this:
subprocess.call('ffmpeg -i "{0}\{1}" "{0}\{2}"'.format('C:\\Users\A B\Desktop\Music', inFile, outFile))
That would work, but that's not the best thing to do: not very performant, using format when you already have all the arguments already, you may not have knowledge of other characters to escape, etc... don't reinvent the wheel.
The best way to do it is to pass the arguments in a list so subprocess module handles the quoting/escaping when necessary:
path = r'C:\Users\A B\Desktop\Music' # use raw prefix to avoid backslash escaping
subprocess.call(['ffmpeg','-i',os.path.join(path,inFile), os.path.join(path,outFile)])
Aside: if you're the user in question, it's even better to do:
path = os.getenv("USERPROFILE"),'Desktop','Music'
and you could even run the process in the path directory with cwd option:
subprocess.call(['ffmpeg','-i',inFile, outFile],cwd=path)
and if you're not, be sure to run the script with elevated privileges or you won't get access to another user directory (read-protected)
I'm creating an archive in Python using this code:
#Creates archive using string like [proxy_16-08-15_08.57.07.tar]
proxyArchiveLabel = 'proxy_%s' % EXECUTION_START_TIME + '.tar'
log.info('Packaging %s ...' % proxyArchiveLabel)
#Removes .tar from label during creation
shutil.make_archive(proxyArchiveLabel.rsplit('.',1)[0], 'tar', verbose=True)
So this creates an archive fine in the local directory. The problem is, there's a specific directory in my archive I want to remove, due to it's size and lack of necessity for this task.
ExecWithLogging('tar -vf %s --delete ./roles/jobs/*' % proxyArchiveLabel)
# ------------
def ExecWithLogging(cmd):
print cmd
p = subprocess.Popen(cmd.split(' '), env=os.environ, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while(True):
log.info(p.stdout.readline().strip())
if(p.poll() is not None):
break
However, this seems to do basically nothing. The size remains the same. If I print cmd inside of the ExecWithLogging, and copy/past that command to a terminal in the working directory of the script, it works fine. Just to be sure, I also tried hard-coding the full path to where the archive is created as part of the tar -vf %s --delete command, but still nothing seemed to happen.
I do get this output in my INFO log: tar: Pattern matching characters used in file names, so I'm kind of thinking Popen is interpreting my command incorrectly somehow... (or rather, I'm more likely passing in something incorrectly).
Am I doing something wrong? What else can I try?
You may have to use the --wildcards option in the tar command, which enables pattern matching. This may well be what you are seeing in your log, be it somewhat cryptically.
Edit:
In answer to your question Why? I suspect that the shell is performing the wildcard expansion whilst the command proffered through Popen is not. The --wildcard option for tar, forces tar to perform the wildcard expansion.
For a more detailed explanation see here:
Tar and wildcards
I am attempting to write a script that utilises sys.argv to wrap the scp command. The idea, is that you will be able to run: pyscp folder/* host but if I run this script with those arguments:
import sys
for arg in sys.argv:
print arg
I get a list of all the folders inside folder:
pyscp.py
folder/0
folder/1
folder/2
folder/3
folder/4
folder/5
folder/67
folder/8
folder/9
host
Assuming a UNIXoid operating system: The shell is expanding the * into the matching files. Try to call your script like
pyscp "folder/*" host
The quotes keep the shell from interpreting the * character.
If you do not escape the asterisk, the shell is performing filename expansion for you. The pattern including the asterisk becomes replaced with an alphabetically sorted list of file names matching the pattern before your Python program becomes executed. You can prevent the shell from performing filename expansion using e.g. single quotes, i.e.
pyscp 'folder/*' hostname
You can then do this yourself within Python using the glob module and control things the way you want it.
The shell is expanding the file list for you. You can leverage this by allowing multiple parameters in the command.
import sys
files = sys.argv[1:-1]
host = sys.argv[-1]
Now you have a more flexible program that lets caller jump through whatever hoops he wants for the transfer, like maybe all text files in folder1 plus anything that's changed in the last day in folder2 (on a linux machine):
pyscp folder1/*.txt `find -mtime -1` example.com
I am trying to do $ mv <file> .. in a python script using subprocess.call(). I am able to do this on 'normal' filenames, but on certain filenames it does not work. I do not have control of the filenames that are given to the script. Here is an example:
M filename is "ITunes ES Film Metadata_10_LaunchTitles(4th Batch)_08_20_2010.XLS"
When I try and do the command directly into the python prompt and drag the file into it, this is what I get:
>>> /Users/David/Desktop/itunes_finalize/TheInventionOfLying_CSP/
ITunes\ ES\ Film\ Metadata_10_LaunchTitles\(4th\ Batch\)_08_20_2010.XLS
No such file or directory
How would I go about moving this file in a python script?
Update:
Thanks for the answers, this is how I ended up doing it:
for file in glob.glob(os.path.join(dir, '*.[xX][lL][sS]')):
shutil.move(file, os.path.join(os.path.dirname(file), os.path.pardir))
subprocess is not the best way to go here. For example, what if you're on an operating system that isn't POSIX compliant?
Check out the shutil module.
>>> import shutil
>>> shutil.move(src, dest)
If finding the actual string for the filename is hard you can use glob.glob to pattern match what you want. For example, if you're running the script/prompt from the directory with the .XLS file in question you could do the following.
>>> import glob
>>> glob.glob('*ITunes*.XLS')
You'll get a list back with all the file strings that fit that pattern.
Rather than using subprocess and spawning a new process, use shutil.move() to just do it in Python. That way, the names won't be reinterpreted and there will be little chance for error.
Spaces, parens, etc. are the shell's problem. They don't require escaping in Python provided you don't pass them to a shell.
open('*WOW!* Rock&Roll(uptempo).mp3')