I was looking for ways to loop over files in directory with python, and I found this question:
Loop through all CSV files in a folder
The point is that the files I have are binary files, with no file extension at the end.
What I want my program to do is to iterate through all the files that have no extension.
Anyway to apply this using wildcards? (Or any other way?)
You can use os.path.splitext to check if a file has an extension or not.
See this examples:
import os
os.path.splitext("foo.ext")
=> ('foo', '.ext')
os.path.splitext("foo")
=> ('foo', '')
So, you can do that:
import os
path = "path/to/files"
dirs = os.listdir(path)
for path in dirs:
if not os.path.splitext(path)[1]:
print(path)
But, beware of "hidden" files which name starts with a dot, ie.: ".bashrc".
You can also check for the existence of a dot in the filename:
for path in dirs:
if "." not in path:
print(path)
Sounds like what you are interested in is
[f for f in next(os.walk(folder))[2] if '.' not in f]
I would suggest using os.listdir(), and then check whether filename has an extension (check if there is a dot in a filename). Once You get all filenames without dots (that is, without extension), just be sure to check that the filename isn't actually directory name, and that's it.
You could use the glob module and filter out any files with extensions:
import glob
for filename in (filename for filename in glob.iglob('*') if '.' not in filename):
print(filename)
Related
I know how to delete files by extension but what if my files are looking like this:
update_24-08-2022_14-54.zip.001
Where last 3 digits can be between 001-029
Here is code that I'm using for standard zip files
files_in_directory = os.listdir(directory)
filtered_files = [file for file in files_in_directory if file.endswith(".zip")]
for file in filtered_files:
path_to_file = os.path.join(directory, file)
os.remove(path_to_file)
Assuming the double extensions are of the form .zip.xyz, with xyz being triple digits, you can use globbing:
import glob
import os
for path in glob.glob('*.zip.[0-9][0-9][0-9]'):
os.remove(path)
(As a usual precaution, check first, by replacing os.remove with print).
If you have a specific directory, its name stored in directory, you can use:
import glob
import os
for path in glob.glob(os.path.join(directory, '*.zip.[0-9][0-9][0-9]')):
os.remove(path)
There is no need to join the directory and path inside the for loop (as is the case in the question): path itself will already contain the directory name.
I recently recovered a folder that i had accidentally deleted. It has .jpg and .tar.gz files. However, all of the files now have some sort of hash extension appended to them and it is different for every file. There are more than 600 files in the folders. So example names would be:
IMG001.jpg.3454637876876978068
IMG002.jpg.2345447786787689769
IMG003.jpg.3454356457657757876
and
folder1.tar.gz.45645756765876
folder2.tar.gz.53464575678588
folder3.tar.gz.42345435647567
I would like to have a script that could go in turn (maybe i can specify extension or it can have two iterations, one through the .jpg files and the other through the .tar.gz) and clean up the last part of the file name starting from the . right before the number. So the final file names would end in .jpg and .tar.gz
What I have so far in python:
import os
def scandirs(path):
for root, dirs, files in os.walk(path):
for currentFile in files:
os.path.splitext(currentFile)
scandirs('C:\Users\ad\pics')
Obviously it doesn't work. I would appreciate any help. I would also consider using a bash script, but I do not know how to do that.
shutil.move(currentFile,os.path.splitext(currentFile)[0])
at least I think ...
Here is how I would do it, using regular expressions:
import os
import re
pattern = re.compile(r'^(.*)\.\d+$')
def scandirs(path):
for root, dirs, files in os.walk(path):
for currentFile in files:
match = pattern.match(currentFile)
if match:
os.rename(
os.path.join(root, currentFile),
os.path.join(root, match.groups(1)[0])
)
scandirs('C:/Users/ad/pics')
Since you tagged with bash I will give you an answer that will remove the last extension for all files/directories in a directory:
for f in *; do
mv "$f" "${f%.*}"
done
I use os.listdir and it works fine, but I get sub-directories in the list also, which is not what I want: I need only files.
What function do I need to use for that?
I looked also at os.walk and it seems to be what I want, but I'm not sure of how it works.
You need to filter out directories; os.listdir() lists all names in a given path. You can use os.path.isdir() for this:
basepath = '/path/to/directory'
for fname in os.listdir(basepath):
path = os.path.join(basepath, fname)
if os.path.isdir(path):
# skip directories
continue
Note that this only filters out directories after following symlinks. fname is not necessarily a regular file, it could also be a symlink to a file. If you need to filter out symlinks as well, you'd need to use not os.path.islink() first.
On a modern Python version (3.5 or newer), an even better option is to use the os.scandir() function; this produces DirEntry() instances. In the common case, this is faster as the direntry loaded already has cached enough information to determine if an entry is a directory or not:
basepath = '/path/to/directory'
for entry in os.scandir(basepath):
if entry.is_dir():
# skip directories
continue
# use entry.path to get the full path of this entry, or use
# entry.name for the base filename
You can use entry.is_file(follow_symlinks=False) if only regular files (and not symlinks) are needed.
os.walk() does the same work under the hood; unless you need to recurse down subdirectories, you don't need to use os.walk() here.
Here is a nice little one-liner in the form of a list comprehension:
[f for f in os.listdir(your_directory) if os.path.isfile(os.path.join(your_directory, f))]
This will return a list of filenames within the specified your_directory.
import os
directoryOfChoice = "C:\\" # Replace with a directory of choice!!!
filter(os.path.isfile, os.listdir(directoryOfChoice))
P.S: os.getcwd() returns the current directory.
for fname in os.listdir('.'):
if os.path.isdir(fname):
pass # do your stuff here for directory
else:
pass # do your stuff here for regular file
The solution with os.walk() would be:
for r, d, f in os.walk('path/to/dir'):
for files in f:
# This will list all files given in a particular directory
Even though this is an older post, let me please add the pathlib library introduced in 3.4 which provides an OOP style of handling directories and files for sakes of completeness. To get all files in a directory, you can use
def get_list_of_files_in_dir(directory: str, file_types: str ='*') -> list:
return [f for f in Path(directory).glob(file_types) if f.is_file()]
Following your example, you could use it like this:
mypath = '/path/to/directory'
files = get_list_of_files_in_dir(mypath)
If you only want a subset of files depending on the file extension (e.g. "only csv files"), you can use:
files = get_list_of_files_in_dir(mypath, '*.csv')
Note PEP 471 DirEntry object attributes is: is_dir(*, follow_symlinks=True)
so...
from os import scandir
folder = '/home/myfolder/'
for entry in scandir(folder):
if entry.is_dir():
# do code or skip
continue
myfile = folder + entry.name
#do something with myfile
I need to iterate through the subdirectories of a given directory and search for files. If I get a file I have to open it and change the content and replace it with my own lines.
I tried this:
import os
rootdir ='C:/Users/sid/Desktop/test'
for subdir, dirs, files in os.walk(rootdir):
for file in files:
f=open(file,'r')
lines=f.readlines()
f.close()
f=open(file,'w')
for line in lines:
newline = "No you are not"
f.write(newline)
f.close()
but I am getting an error. What am I doing wrong?
The actual walk through the directories works as you have coded it. If you replace the contents of the inner loop with a simple print statement you can see that each file is found:
import os
rootdir = 'C:/Users/sid/Desktop/test'
for subdir, dirs, files in os.walk(rootdir):
for file in files:
print(os.path.join(subdir, file))
If you still get errors when running the above, please provide the error message.
Another way of returning all files in subdirectories is to use the pathlib module, introduced in Python 3.4, which provides an object oriented approach to handling filesystem paths (Pathlib is also available on Python 2.7 via the pathlib2 module on PyPi):
from pathlib import Path
rootdir = Path('C:/Users/sid/Desktop/test')
# Return a list of regular files only, not directories
file_list = [f for f in rootdir.glob('**/*') if f.is_file()]
# For absolute paths instead of relative the current dir
file_list = [f for f in rootdir.resolve().glob('**/*') if f.is_file()]
Since Python 3.5, the glob module also supports recursive file finding:
import os
from glob import iglob
rootdir_glob = 'C:/Users/sid/Desktop/test/**/*' # Note the added asterisks
# This will return absolute paths
file_list = [f for f in iglob(rootdir_glob, recursive=True) if os.path.isfile(f)]
The file_list from either of the above approaches can be iterated over without the need for a nested loop:
for f in file_list:
print(f) # Replace with desired operations
From python >= 3.5 onward, you can use **, glob.iglob(path/**, recursive=True) and it seems the most pythonic solution, i.e.:
import glob, os
for filename in glob.iglob('/pardadox-music/**', recursive=True):
if os.path.isfile(filename): # filter dirs
print(filename)
Output:
/pardadox-music/modules/her1.mod
/pardadox-music/modules/her2.mod
...
Notes:
glob.iglob
glob.iglob(pathname, recursive=False)
Return an iterator which yields the same values as glob() without actually storing them all simultaneously.
If recursive is True, the pattern '**' will match any files and
zero or more directories and subdirectories.
If the directory contains files starting with . they won’t be matched by default. For example, consider a directory containing card.gif and .card.gif:
>>> import glob
>>> glob.glob('*.gif') ['card.gif']
>>> glob.glob('.c*')['.card.gif']
You can also use rglob(pattern),
which is the same as calling glob() with **/ added in front of the given relative pattern.
What is the simplest way to get the full recursive list of files inside a folder with python? I know about os.walk(), but it seems overkill for just getting the unfiltered list of all files. Is it really the only option?
There's nothing preventing you from creating your own function:
import os
def listfiles(folder):
for root, folders, files in os.walk(folder):
for filename in folders + files:
yield os.path.join(root, filename)
You can use it like so:
for filename in listfiles('/etc/'):
print filename
os.walk() is not overkill by any means. It can generate your list of files and directories in a jiffy:
files = [os.path.join(dirpath, filename)
for (dirpath, dirs, files) in os.walk('.')
for filename in (dirs + files)]
You can turn this into a generator, to only process one path at a time and safe on memory.
You could also use the find program itself from Python by using sh
import sh
text_files = sh.find(".", "-iname", "*.txt")
Either that or manually recursing with isdir() / isfile() and listdir() or you could use subprocess.check_output() and call find .. Bascially os.walk() is highest level, slightly lower level is semi-manual solution based on listdir() and if you want the same output find . would give you for some reason you can make a system call with subprocess.
pathlib.Path.rglob is pretty simple. It lists the entire directory tree
(The argument is a filepath search pattern. "*" means list everything)
import pathlib
for path in pathlib.Path("directory_to_list/").rglob("*"):
print(path)
os.walk() is hard to use, just kick it and use pathlib instead.
Here is a python function mimicking a similar function of list.files in R language.
def list_files(path,pattern,full_names=False,recursive=True):
if(recursive):
files=pathlib.Path(path).rglob(pattern)
else:
files=pathlib.Path(path).glob(pattern)
if full_names:
files=[str(f) for f in files]
else:
files=[f.name for f in files]
return(files)
import os
path = "path/to/your/dir"
for (path, dirs, files) in os.walk(path):
print files
Is this overkill, or am I missing something?