Develop Reference

Trying to pass files from a subfolder to a new folder after Filtering

I am trying to make a code that allows me to copy files from various subfolders to a new folder. The main problem and why it wasn't a straight forward thing to deal with since the beginning is that all the files start the same and end the same, just 1 letter changes in the file that I need.
I keep getting an error saying: "FileNotFoundError: [Errno 2] No such file or directory: '20210715 10.1_Plate_R_p00_0_B04f00d0.TIF'"
So far this is what I have:
import os,fnmatch, shutil
mainFolder = r"E:\Screening\202010715\20210715 Screening"
dst = r"C:\Users\**\Dropbox (**)\My PC\Desktop\Screening Data\20210715"
for subFolder in os.listdir(mainFolder):
sf = mainFolder+"/"+subFolder
id os.path.isdir(sf):
subset = fnmatch.filter(os.listdir(sf), "*_R_*")
for files in subset:
if '_R_' in files:
shutil.copy(files, dst)
PLEASE HELP!

The issue is that you aren't providing the full path to the file to shutil.copy, so it assumes the file exists in the folder you are running the script from. Another issue the code has, is that the subset variable gets replaced in each loop before you are able to copy the files you matched in the next loop.
import os, fnmatch, shutil
mainFolder = r"E:\Screening\202010715\20210715 Screening"
dst = r"C:\Users\**\Dropbox (**)\My PC\Desktop\Screening Data\20210715"
matches = [] # To store the matches
for subFolder in os.listdir(mainFolder):
sf = mainFolder + "/" + subFolder
if os.path.isdir(sf):
matches.extend([f'{sf}/{file}' for file in fnmatch.filter(os.listdir(sf), "*_R_*")]) # Append the full path to the file
for files in matches:
shutil.copy(files, dst)

How to get the files with the biggest size in the folders, change their name and save to a different folder

I need to get files with the biggest size in different folders, change their name to folder name that they belong to and save to a new folder. I have something like this and I got stuck:
import os
# Core settings
rootdir = 'C:\\Users\\X\\Desktop\\humps'
to_save = 'C:\\Users\\X\\Desktop\\new'
for root, dirs, files in os.walk(rootdir):
new_list = []
for file in files:
if file.endswith(".jpg"):
try:
print(file)
os.chdir(to_save)
add_id = root.split("humps\\")[1]
add_id = add_id.split("\\")[0]
file_name = os.path.join(root,file)
new_list.append(file_name)
bigfile = max(new_list, key=lambda x: x.stat().st_size)
except:
pass
To make it more clear: Let's say the name of the sub-folder is "elephant" and there are different elephant photos and subfolders in in this elephant folder. I want to go through those photos and subfolders and find the elephant foto with the biggest size, name it as elephant and save it to my target folder. Also repaet it for other sub folders such as lion, puma etc.
How I could achieve what I want ?

To find biggest file and save to another location
import os
import shutil
f_list = []
root = "path/to/directory"
root = os.path.abspath(root)
for folder, subfolders, files in os.walk(root):
for file in files:
filePath = os.path.join(folder, file)
f_list.append(filePath)
bigest_file = max(f_list,key=os.path.getsize)
new_path = "path/where/you/want/to/save"
shutil.copy(biggest_file,new_path)
if you want only images then add one more condition in loop
for folder, subfolders, files in os.walk(root):
for file in files:
if file.endswith(".jpg"):
filePath = os.path.join(folder, file)
f_list.append(filePath)
To get all folders biggest file
root = "demo"
root = os.path.abspath(root)
def test(path):
big_files = []
all_paths = [x[0] for x in os.walk(path)]
for paths in all_paths:
f_list = filter(os.path.isfile, os.listdir(paths))
if len(f_list) > 0:
big_files.append((paths,max(f_list,key=os.path.getsize)))
return big_files
print test(root)

How to get the files with the biggest size in the folders, change their name and save to a different folder
Basically you already have a good description of what you need to do. You just need to follow it step by step:
get all files in some search directory
filter for relevant files ("*.jpg")
get their sizes
find the maximum
copy to new directory with name of search directory
IMO it's an important skill to be able to break down a task into smaller tasks. Then, you just need to implement the smaller tasks and combine:
def iterate_files_recursively(directory="."):
for entry in os.scandir(directory):
if entry.is_dir():
for file in iterate_files_recursively(entry.path):
yield file
else:
yield entry
files = iterate_files_recursively(subfolder_name)
I'd use os.scandir because it avoids building up a (potentially) huge list of files in memory and instead allows me (via a generator) to work one file at a time. Note that starting with 3.6 you can use the result of os.scandir as a context manager (with syntax).
images = itertools.filterfalse(lambda f: not f.path.endswith('.jpg'), files)
Filtering is relatively straightforward except for the IMO strange choice of ìtertools.filterfalse to only keep elements for which its predicate returns False.
biggest = max(images, key=(lambda img: img.stat().st_size))
This is two steps in one: Get the maximum with the builtin max function, and use the file size as "key" to establish an order. Note that this raises a ValueError if you don't have any images ... so you might want to supply default=None or handle that exception.
shutil.copy(biggest.path, os.path.join(target_directory, subfolder_name + '.jpg')
shutil.copy copies the file and some metadata. Instead of hardcoding path separators, please use os.path.join!
Now all of this assumes that you know the subfolder_name. You can scan for those easily, too:
def iterate_directories(directory='.'):
for entry in os.scandir(directory):
if entry.is_dir():
yield entry

Here's some code that does what you want. Instead of using the old os.walk function, it uses modern pathlib functions.
The heart of this code is the recursive biggest function. It scans all the files and directories in folder, saving the matching file names to the files list, and recursively searching any directories it finds. It then returns the path of the largest file that it finds, or None if no matching files are found.
from pathlib import Path
import shutil
def filesize(path):
return path.stat().st_size
def biggest(folder, pattern):
''' Find the biggest file in folder that matches pattern
Search recursively in all subdirectories
'''
files = []
for f in folder.iterdir():
if f.is_file():
if f.match(pattern):
files.append(f)
elif f.is_dir():
found = biggest(f, pattern)
if found:
files.append(found)
if files:
return max(files, key=filesize)
def copy_biggest(src, dest, pattern):
''' Find the biggest file in each folder in src that matches pattern
and copy it to dest, using the folder's name as the new file name
'''
for path in src.iterdir():
if path.is_dir():
found = biggest(path, pattern)
if found:
newname = dest / path
print(path, ':', found, '->', newname)
shutil.copyfile(found, newname)
You can call it like this:
rootdir = r'C:\Users\X\Desktop\humps'
to_save = r'C:\Users\X\Desktop\new'
copy_biggest(Path(rootdir), Path(to_save), '*.jpg')
Note that the copied files will have the same name as the top-level folder in rootdir that they were found in, with no file extension. If you want to give them a .jpg extension, you can change
newname = dest / path
to
newname = (dest / path).with_suffix('.jpg')
The shutil module on older versions of Python 3 doesn't understand pathlib paths. But that's easy enough to remedy. In the copy_biggest function, replace
shutil.copyfile(found, newname)
with
shutil.copyfile(str(found), str(newname))

Renaming multiple files in a directory using Python

I'm trying to rename multiple files in a directory using this Python script:
import os
path = '/Users/myName/Desktop/directory'
files = os.listdir(path)
i = 1
for file in files:
os.rename(file, str(i)+'.jpg')
i = i+1
When I run this script, I get the following error:
Traceback (most recent call last):
File "rename.py", line 7, in <module>
os.rename(file, str(i)+'.jpg')
OSError: [Errno 2] No such file or directory
Why is that? How can I solve this issue?
Thanks.

You are not giving the whole path while renaming, do it like this:
import os
path = '/Users/myName/Desktop/directory'
files = os.listdir(path)
for index, file in enumerate(files):
os.rename(os.path.join(path, file), os.path.join(path, ''.join([str(index), '.jpg'])))
Edit: Thanks to tavo, The first solution would move the file to the current directory, fixed that.

You have to make this path as a current working directory first.
simple enough.
rest of the code has no errors.
to make it current working directory:
os.chdir(path)

import os
from os import path
import shutil
Source_Path = 'E:\Binayak\deep_learning\Datasets\Class_2'
Destination = 'E:\Binayak\deep_learning\Datasets\Class_2_Dest'
#dst_folder = os.mkdir(Destination)
def main():
for count, filename in enumerate(os.listdir(Source_Path)):
dst = "Class_2_" + str(count) + ".jpg"
# rename all the files
os.rename(os.path.join(Source_Path, filename), os.path.join(Destination, dst))
# Driver Code
if __name__ == '__main__':
main()

As per #daniel's comment, os.listdir() returns just the filenames and not the full path of the file. Use os.path.join(path, file) to get the full path and rename that.
import os
path = 'C:\\Users\\Admin\\Desktop\\Jayesh'
files = os.listdir(path)
for file in files:
os.rename(os.path.join(path, file), os.path.join(path, 'xyz_' + file + '.csv'))

Just playing with the accepted answer define the path variable and list:
path = "/Your/path/to/folder/"
files = os.listdir(path)
and then loop over that list:
for index, file in enumerate(files):
#print (file)
os.rename(path+file, path +'file_' + str(index)+ '.jpg')
or loop over same way with one line as python list comprehension :
[os.rename(path+file, path +'jog_' + str(index)+ '.jpg') for index, file in enumerate(files)]
I think the first is more readable, in the second the first part of the loop is just the second part of the list comprehension

If your files are renaming in random manner then you have to sort the files in the directory first. The given code first sort then rename the files.
import os
import re
path = 'target_folder_directory'
files = os.listdir(path)
files.sort(key=lambda var:[int(x) if x.isdigit() else x for x in re.findall(r'[^0-9]|[0-9]+', var)])
for i, file in enumerate(files):
os.rename(path + file, path + "{}".format(i)+".jpg")

I wrote a quick and flexible script for renaming files, if you want a working solution without reinventing the wheel.
It renames files in the current directory by passing replacement functions.
Each function specifies a change you want done to all the matching file names. The code will determine the changes that will be done, and displays the differences it would generate using colors, and asks for confirmation to perform the changes.
You can find the source code here, and place it in the folder of which you want to rename files https://gist.github.com/aljgom/81e8e4ca9584b481523271b8725448b8
It works in pycharm, I haven't tested it in other consoles
The interaction will look something like this, after defining a few replacement functions
when it's running the first one, it would show all the differences from the files matching in the directory, and you can confirm to make the replacements or no, like this

This works for me and by increasing the index by 1 we can number the dataset.
import os
path = '/Users/myName/Desktop/directory'
files = os.listdir(path)
index=1
for index, file in enumerate(files):
os.rename(os.path.join(path, file),os.path.join(path,''.join([str(index),'.jpg'])))
index = index+1
But if your current image name start with a number this will not work.

Python - IOError when running through folder tree

I am trying to read in a series of DICOM files in a folder tree and I am using the below code to run through the tree, reading in each file as I go. The problem is I am getting IOErrors for files that definitely exist, I have checked file permissions and other SO threads such as Python: IOError: [Errno 2] No such file or directory but I haven't managed to get it working without these IOErrors yet. Does anyone have any ideas?
for root, dirs, files in os.walk(path):
for fname in files:
name = os.path.basename(os.path.abspath(fname))
if name.startswith('.') == True:
pass
else:
try:
plan=dicom.read_file(fname)
ds=dicom.read_file(fname, stop_before_pixels = True)
kVp = TagChecker([0x18,0x60]) #KVP
Target = TagChecker([0x18,0x1191]) #ANODE
Filter = TagChecker([0x18,0x7050]) #
write_results.writerow([Survey_Number, Patient_ID, View_Protocol, int(kVp), Target, Filter, Thickness, mAs_Exposure, LPad_Yes_No, autoorman, AECMode, AECDset, Patient_Age, Comment, Compression_Force])
#print(fname)
except IOError:
print "IOError: ", "//" + os.path.join(root, fname) + "//"
except InvalidDicomError:
# This exception line prints an error message to the command line, checks to see if an error log
# has been generated for this session, writes a new one if not and then writes the error to the log file
print "Invalid Dicom File: ", fname

Usually a method that takes a filename, like dicom.read_file(fname), will take an absolute filename (or assume that the filename is relative to the dir that your main python program is running in, the cwd()). Can I suggest that you put this line in front of the first read_file() call:
print "reading: %s" % os.path.abspath(fname)
Then you'll see the filename that you're actually trying to read. I'm guessing it's not the file (or droids) you think you're looking for.
In order to fix your problem.. join the dir and the fname before you read.. e.g.
full_fname = os.path.join(dir, fname)
dicom.read_file(full_fname)
In other words, I think you're reading files with relative paths and you want to be using absolute paths.

Python: Files stop being opened at a certain point

I've written the following program in Python:
import re
import os
import string
folder = 'C:\Users\Jodie\Documents\Uni\Final Year Project\All Data'
folderlisting = os.listdir(folder)
for eachfolder in folderlisting:
print eachfolder
if os.path.isdir(folder + '\\' + eachfolder):
filelisting = os.listdir('C:\Users\Jodie\Documents\Uni\Final Year Project\All Data\\' + eachfolder)
print filelisting
for eachfile in filelisting:
if re.search('.genbank.txt$', eachfile):
genbankfile = open(eachfile, 'r')
print genbankfile
if re.search('.alleles.txt$', eachfile):
allelesfile = open(eachfile, 'r')
print allelesfile
It looks through a lot of folders, and prints the following:
The name of each folder, without the path
A list of all files in each folder
Two specific files in each folder (Any files containing ".genbank.txt" and ".alleles.txt").
The code works until it reaches a certain directory, and then fails with the following error:
Traceback (most recent call last):
File "C:\Users\Jodie\Documents\Uni\Final Year Project\Altering Frequency Practice\Change_freq_data.py", line 16, in <module>
genbankfile = open(eachfile, 'r')
IOError: [Errno 2] No such file or directory: 'ABP1.genbank.txt'
The problem is:
That file most definitely exists, since the program lists it before it tries to open the file.
Even if I take that directory out of the original group of directories, the program throws up the same error for the next folder it iterates to. And the next, if that one is removed. And so on.
This makes me think that it's not the folder or any files in it, but some limitation of Python? I have no idea. It has stumped me.
Any help would be appreciated.

You should use os.walk() http://docs.python.org/library/os.html#os.walk
Also, you need to read the contents of the file, you don't want to print the file object. And you need to close the file when you're done or use a context manager to close it for you.
would look something like:
for root, dirs, files in os.walk(folder):
for file_name in files:
if re.search('.genbank.txt$', file_name) or \
re.search('.alleles.txt$', file_name):
with open(os.path.join(root, f), 'r') as f:
print f.read()
Keep in mind this is not 'exactly' what you're doing, this will walk the entire tree, you may just want to walk a single level like you are already doing.