Develop Reference

How to Iterate over several directory levels and move files based on condition

I would like some help to loop through some directories and subdirectories and extracting data. I have a directory with three levels, with the third level containing several .csv.gz files. The structure is like this
I need to access level 2 (where subfolders are) of each folder and check the existence of a specific folder (in my example, this will be subfolder 3; I left the other folders empty for this example, but in real cases they will have data). If checking returns True, then I want to change the name of files within the target subfolder3 and transfer all files to another folder.
Bellow is my code. It is quite cumbersome and there is probably better ways of doing it. I tried using os.walk() and this is the closest I got to a solution but it won't move the files.
import os
import shutil
def organizer(parent_dir, target_dir, destination_dir):
for root, dirs, files in os.walk(parent_dir):
if root.endswith(target_dir):
target = root
for files in os.listdir(target):
if not files.startswith("."):
# this is to change the name of the file
fullname = files.split(".")
just_name = fullname[0]
csv_extension = fullname[1]
gz_extension = fullname[2]
subject_id = target
#make a new name
origin = subject_id + "/"+ just_name + "." + csv_extension + "." + gz_extension
#make a path based on this new name
new_name = os.path.join(destination_dir, origin)
#move file from origin folder to destination folder and rename the file
shutil.move(origin, new_name)
Any suggestions on how to make this work and / or more eficient?

simply enough, you can use the built-in os module, with os.walk(path) returns you root directories and files found
import os
for root, _, files in os.walk(path):
#your code here
for your problem, do this
import os
for root, dirs, files in os.walk(parent_directory);
for file in files:
#exctract the data from the "file"
check this for more information os.walk()
and if you want to get the name of the file, you can use os.path.basename(path)
you can even check for just the gzipped csv files you're looking for using built-in fnmatch module
import fnmathch, os
def find_csv_files(path):
result = []
for root, _, files in os.walk(path):
for name in files:
if fnmatch.fnmatch(name, "*.csv.gz"): # find csv.gz using regex paterns
result.append(os.path.join(root, name))
return list(set(results)) #to get the unique paths if for some reason duplicated

Ok, guys, I was finally able to find a solution. Here it is. Not the cleanest one, but it works in my case. Thanks for the help.
def organizer(parent_dir, target_dir, destination_dir):
for root, dirs, files in os.walk(parent_dir):
if root.endswith(target_dir):
target = root
for files in os.listdir(target):
#this one because I have several .DS store files in the folder which I don't want to extract
if not files.startswith("."):
fullname = files.split(".")
just_name = fullname[0]
csv_extension = fullname[1]
gz_extension = fullname[2]
origin = target + "/" + files
full_folder_name = origin.split("/")
#make a new name
new_name = full_folder_name[5] + "_"+ just_name + "." + csv_extension + "." + gz_extension
#make a path based on this new name
new_path = os.path.join(destination_dir, new_name)
#move file from origin folder to destination folder and rename the file
shutil.move(origin, new_path)
The guess the problem was that was passing a variable that was a renamed file (in my example, I wrongly called this variable origin) as the origin path to shutil.move(). Since this path does not exist, then the files weren't moved.

How to copy a file two folders back by replacing file name with names of folders using python

I have several folders such as a1-b1, a1-b2, a1-b3. a2-b2 and so on. Within each folder there are subfolders e.g. c_1, c_2, c_3 and so on. Within each subfolder I have data files with same name e.g. abc.dat. I desire to copy abc.dat two folders back by replacing its name with subfolders e.g. a1-b1-c_1.dat, a1-b1-c_2.dat, a1-b3_c1.dat etc..
My present approach can only copy one folder back, but also changes the name of abc.dat files in their existing directories, which I would now like to avoid and prefer that these files are copied with their desired changed names in two folders back, but exist as abc.dat in their current directories. Thanks in advance for your support!
input_dir = "/user/my_data"
# Walk through all files in the directory that contains the files to copy
for root, dirs, files in os.walk(input_dir):
for filename in files:
if filename == 'abc.dat':
base = os.path.join(os.path.abspath(root))
#Get current name
old_name = os.path.join(base, filename)
#Get parent folder
parent_folder = os.path.basename(base)
#New name based on parent folder
new_file_name = parent_folder + ".dat" #assuming same extension
new_abs_name = os.path.join(base, new_file_name)
#Rename to new name
os.rename(old_name,new_abs_name)
#Copy to one level up
one_level_up = os.path.normpath(os.path.join(base, os.pardir))
one_level_up_name = os.path.join(one_level_up, new_file_name)
shutil.copy(new_abs_name,one_level_up_name)

So, I figured out its solution. Here it is!
import os
import shutil
current_dir = os.getcwd()
for dirpath, dirs, files in os.walk(current_dir):
for f in files:
if f.endswith('.dat'):
folder_1 = os.path.split(os.path.split(dirpath)[0])[1]
folder_2 = os.path.split(os.path.split(dirpath)[1])[1]
os.rename(os.path.join(dirpath, f),
os.path.join(dirpath, folder_1 + '-' + folder_2 + '.dat'))
totalCopyPath = os.path.join(dirpath, folder_1 + '-' + folder_2 + '.dat')
shutil.copy(totalCopyPath,current_dir)

Python: copying files group to subdirectories

I'm working with Python and I'd like to copy 3 files of a certain folder
/Users/jake/Desktop/exp to all the subfolders of other sub-directories belonging to the directory /toyspace:
/Users/jake/Desktop/toyspace/1A/AAA
/Users/jake/Desktop/toyspace/1A/BBB
/Users/jake/Desktop/toyspace/1A/CCC
/Users/jake/Desktop/toyspace/2B/AAA
/Users/jake/Desktop/toyspace/2B/BBB
/Users/jake/Desktop/toyspace/2B/CCC
So the subfolders names are the same for all the sub-directories. I wrote something like that:
from distutils.dir_util import copy_tree
def myfunc (source, destination):
fromDirectory = source
toDirectory = destination
copy_tree(fromDirectory, toDirectory)
for subfold in toDirectory:
myfunc(fromDirectory, subfold)
Where source =/Users/jake/Desktop/exp and destination =/Users/jake/Desktop/toyspace, but it returns me an error:
DistutilsFileError: could not create '/motif_list.txt': Read-only file system
Can you help me? Thanks in advance!

Unfortunately I have not used distutils but you can try to automate using the os command as below
import os
def copy_folders(source_dir, destination_dir):
files = os.listdir(source_dir)
for sub_folder1 in os.listdir(destination_dir):
for sub_folder in os.listdir(destination_dir + sub_folder1 + '/'):
for file in files:
os.system('cp ' + source_dir + file + ' ' + destination_dir + sub_folder1 + '/' + sub_folder + '/')
Let me know how it goes :)

Python shutil - copy file + file tree for specific file only

I am trying to copy files from list with shutil and place them in same folder structure in another directory:
import os
import shutil
src = "/sourcedoc/1/"
dest = "/destdoc/"
files_to_find = []
with open('filelist.txt') as fh:
for row in fh:
files_to_find.append(row.strip())
for root, dirs, files in os.walk(src):
for _file in files:
if _file in files_to_find:
print('Found file in: ' + str(root))
os.makedirs(os.path.dirname(dest), exist_ok = True)
shutil.copy(os.path.abspath(root + '/' + _file), dest + _file)
I want to create the folders with the same names as the ones containing the files that are to be copied in the new destination and then copy the files in them.
The final goal is to have structure like "/destdoc/1/" and having copied the files from the list. However, what I get is all the files in the destination directory without the folder structure. It seems that using shutil.copytree copies all of the files in the folders.

after run python script to run recursive rename my files has disappeared

I've just run this script to rename my files, adding the 15 first chars of files, but now all the files are disappearead, and i can't find them. i've just run this on a mac
import os
def replace(folder_path, old, new):
for path, subdirs, files in os.walk(folder_path):
for name in files:
file_path = os.path.join(path,name)
new_name = os.path.basename(path)[:15] + " - " + name
print path + "##" + new_name
os.rename(file_path, new_name)
replace('/Users/myuser/mp3/', '', '')

#DDave,
In the function call os.rename(src, dst): src is the fully qualified filepath of the source file, and dst is fully qualified filepath of the target file.
In your above program, new_name is just the new name of the file which does not have the directory information (and hence it is considering the directory as your current working directory). That is why you are not seeing the renamed files where you were looking to see them.
The program is rather trying to create the files under your current working directory. Run the following from your console of your IDE or from your program to figure out the current working directory:
print(os.getcwd())
Providing the full solution as requested in the comment below:
import os
def replace(folder_path, old, new):
for path, subdirs, files in os.walk(folder_path):
for name in files:
file_path = os.path.join(path,name)
# new_name = os.path.basename(path)[:15] + " - " + name
new_filepath = os.path.join(path, (name[:15] if len(name)>=15 else name) + " - " + name)
print ("new path: " + new_filepath)
os.rename(file_path, new_filepath)