I have multiple folders that look like folder.0 and folder.1.
Inside each folder there is one file ('junk') that I want to copy and rename to the .0 or .1 part of the folder name in which it currently resides.
Here is what I'm trying to do:
inDirec = '/foobar'
outDirec = '/complete/foobar'
for root, dirs,files in os.walk(inDirec):
for file in files:
if file =='junk'
d = os.path.split(root)[1]
filename, iterator = os.path.splitext(d) # folder no. captured
os.rename(file, iterator+file) # change name to include folder no.
fullpath = os.path.join(root,file)
shutil.copy(fullpath,outDirec)
This returns:
os.rename(file,iterator+file)
OSError: [Errno 2] No such file or directory
I'm not even sure I should be using os.rename. I just want to pull out files == 'junk' and copy them to one directory but they all have the exact same name. So I really just need to rename them so they can exist in the same directory. Any help?
Update
for root, dirs,files in os.walk(inDirec):
for file in files:
if file =='junk'
d = os.path.split(root)[1]
filename, iterator = os.path.splitext(d) # folder no. captured
it = iterator[-1] # iterator began with a '.'
shutil.copy(os.path.join(root,file),os.path.join(outDirec,it+file))
Your problem is that your program is using the working directory at launch for your rename operation. You need to provide full relative or absolute paths as arguments to os.rename().
Replace:
os.rename(file, iterator+file)
fullpath = os.path.join(root,file)
shutil.copy(fullpath,outDirec)
With (if you want to move):
os.rename(os.path.join(root, file), os.path.join(outDirec, iterator+file))
Or with (if you want to copy):
shutil.copy(os.path.join(root, file), os.path.join(outDirec, iterator+file))
NOTE: The destination directory should already exist or you will need code to create it.
Related
I would like some help to loop through some directories and subdirectories and extracting data. I have a directory with three levels, with the third level containing several .csv.gz files. The structure is like this
I need to access level 2 (where subfolders are) of each folder and check the existence of a specific folder (in my example, this will be subfolder 3; I left the other folders empty for this example, but in real cases they will have data). If checking returns True, then I want to change the name of files within the target subfolder3 and transfer all files to another folder.
Bellow is my code. It is quite cumbersome and there is probably better ways of doing it. I tried using os.walk() and this is the closest I got to a solution but it won't move the files.
import os
import shutil
def organizer(parent_dir, target_dir, destination_dir):
for root, dirs, files in os.walk(parent_dir):
if root.endswith(target_dir):
target = root
for files in os.listdir(target):
if not files.startswith("."):
# this is to change the name of the file
fullname = files.split(".")
just_name = fullname[0]
csv_extension = fullname[1]
gz_extension = fullname[2]
subject_id = target
#make a new name
origin = subject_id + "/"+ just_name + "." + csv_extension + "." + gz_extension
#make a path based on this new name
new_name = os.path.join(destination_dir, origin)
#move file from origin folder to destination folder and rename the file
shutil.move(origin, new_name)
Any suggestions on how to make this work and / or more eficient?
simply enough, you can use the built-in os module, with os.walk(path) returns you root directories and files found
import os
for root, _, files in os.walk(path):
#your code here
for your problem, do this
import os
for root, dirs, files in os.walk(parent_directory);
for file in files:
#exctract the data from the "file"
check this for more information os.walk()
and if you want to get the name of the file, you can use os.path.basename(path)
you can even check for just the gzipped csv files you're looking for using built-in fnmatch module
import fnmathch, os
def find_csv_files(path):
result = []
for root, _, files in os.walk(path):
for name in files:
if fnmatch.fnmatch(name, "*.csv.gz"): # find csv.gz using regex paterns
result.append(os.path.join(root, name))
return list(set(results)) #to get the unique paths if for some reason duplicated
Ok, guys, I was finally able to find a solution. Here it is. Not the cleanest one, but it works in my case. Thanks for the help.
def organizer(parent_dir, target_dir, destination_dir):
for root, dirs, files in os.walk(parent_dir):
if root.endswith(target_dir):
target = root
for files in os.listdir(target):
#this one because I have several .DS store files in the folder which I don't want to extract
if not files.startswith("."):
fullname = files.split(".")
just_name = fullname[0]
csv_extension = fullname[1]
gz_extension = fullname[2]
origin = target + "/" + files
full_folder_name = origin.split("/")
#make a new name
new_name = full_folder_name[5] + "_"+ just_name + "." + csv_extension + "." + gz_extension
#make a path based on this new name
new_path = os.path.join(destination_dir, new_name)
#move file from origin folder to destination folder and rename the file
shutil.move(origin, new_path)
The guess the problem was that was passing a variable that was a renamed file (in my example, I wrongly called this variable origin) as the origin path to shutil.move(). Since this path does not exist, then the files weren't moved.
Essentially what I'm trying to do is loop through a directory that contains multiple sub-directories and within those run code against each file in a for loop.
The only start I managed to make was listing the directories but as I've rarely ever used os I'm not sure if I could potentially loop through os.chdir and a bit of f string formatting to loop through each subdirectory.
The files I want to run code against are just txt files.
Here goes my code, up to the moment:
import os
for folders in os.listdir('../main_directory'):
for something in os.listdir(f'{folders}'):
# run some function of sorts
pass
Any help would be greatly appreciated.
I like using pure os:
import os
for fname in os.listdir(src):
# build the path to the folder
folder_path = os.path.join(src, fname)
if os.path.isdir(folder_path):
# we are sure this is a folder; now lets iterate it
for file_name in os.listdir(folder_path):
file_path = os.path.join(folder_path, file_name)
# now you can apply any function assuming it is a file
# or double check it if needed as `os.path.isfile(file_path)`
Note that this function just iterate over the folder given at src and one more level:
src/foo.txt # this file is ignored
src/foo/a.txt # this file is processed
src/foo/foo_2/b.txt # this file is ignored; too deep.
src/foo/foo_2/foo_3/c.txt # this file is ignored; too deep.
In case you need to go as deep as possible, you can write a recursive function and apply it to every single file, as follows:
import os
def function_over_files(path):
if os.path.isfile(path):
# do whatever you need with file at path
else:
# this is a dir: we will list all files on it and call recursively
for fname in os.listdir(path):
f_path = os.path.join(path, fname)
# here is the trick: recursive call to the same function
function_over_files(f_path)
src = "path/to/your/dir"
function_over_files(src)
This way you can apply the function to any file under path, don't care how deep it is in the folder, now:
src/foo.txt # this file is processed; as each file under src
src/foo/a.txt # this file is processed
src/foo/foo_2/b.txt # this file is processed
src/foo/foo_2/foo_3/c.txt # this file is processed
You could try something like this:
for subdir, dirs, files in os.walk(rootdir):
Now you have "access" to all subdirs, dirs, and files for your main folder.
Hope it helps
I have a series of files that are nested as shown in the attached image. For each "inner" folder (e.g. like the 001717528 one), I want to extract a row of data from each the FITS files and create a CSV file that contains all the rows, and name that CSV file after the name of the "inner" folder (e.g. 001717528.csv that has data from the 18 fits files). The data-extracting part is easy but I have trouble coding the iteration.
I don't really know how to iterate over both the outer folders such as the 0017 and inner folders, and name the csv files as I want.
My code is looking like this:
for subdir, dirs, files in os.walk('../kepler'):
for file in files:
filepath = subdir + os.sep + file
if filepath.endswith(".fits"):
extract data
write to csv file
Apparently this will iterate over all files in the kepler folder so it doesn't work.
If you need to keep track of how far you've walked into the directory structure, you can count the file path delimiter (os.sep). In your case it's / because you're on a Mac.
for path, dirs, _ in os.walk("../kepler"):
if path.count(os.sep) == 2:
# path should be ../kepler/0017
for dir in dirs:
filename = dir + ".csv"
data_files = os.listdir(path + os.sep + dir)
for file in data_files:
if file.endswith(".fits"):
# Extract data
# Write to CSV file
As far as I can tell this meets your requirements, but let me know if I've missed something.
Try this code it should print the file path of all your ".fits" files:
# !/usr/bin/python
import os
base_dir = './test'
for root, dirs, files in os.walk(base_dir, topdown=False):
for name in files:
if name.endswith(".fits"):
file_path = os.path.join(root, name) #path of files
print(file_path)
# do your treatment on file_path
All you have to do is add your specific treatment.
Lets say my python script is in a folder "/main". I have a bunch of text files inside subfolders in main. I want to be able to open a file just by specifying its name, not the subdirectory its in.
So open_file('test1.csv') should open test1.csv even if its full path is /main/test/test1.csv.
I don't have duplicated file names so it should no be a problem.
I using windows.
you could use os.walk to find your filename in a subfolder structure
import os
def find_and_open(filename):
for root_f, folders, files in os.walk('.'):
if filename in files:
# here you can either open the file
# or just return the full path and process file
# somewhere else
with open(root_f + '/' + filename) as f:
f.read()
# do something
if you have a very deep folder structure you might want to limit the depth of the search
import os
def get_file_path(file):
for (root, dirs, files) in os.walk('.'):
if file in files:
return os.path.join(root, file)
This should work. It'll return the path, so you should handle opening the file, in your code.
import os
def open_file(filename):
f = open(os.path.join('/path/to/main/', filename))
return f
I'm trying to write my first script.
I have been reading about python but I am stock.
I'm trying to write a script that will rename all the file names in a specific folder.
this is what I have so far:
import os
files = os.listdir('files_to_Change')
print (files)
Get all the file names from folder:
for i in files:
if i == ".DS_Store":
p = files.index(".DS_Store")
del files[p]
If mac invisible file exists delete from list (maybe a mistake here).
for i in files:
oldName = i
fileName, fileExtension = os.path.splitext(i)
print (oldName)
print (fileName)
os.rename(oldName,fileName)
This is where I am stock, I get this error:
Output:
FileNotFoundError: [Errno 2] No such file or directory: 'File.1'
On the above part I'm just removing the file extension, but that is only the beginning.
I'm also trying to substitute every point by a space and make the first letter of every word a capital.
Can anyone point me in the right direction?
Thanks so much
In your example, when you get a list of files in a files_to_Change directory, you get file names without the directory name:
>>> files = os.listdir('test_folder')
>>> print files[0]
.com.apple.timemachine.supported
So in order to get the full path to that file, from whereever you're in your directory tree, you should join the directory name (files_to_Change) with the file name:
import os
join = os.path.join
src = 'files_to_Change'
files = os.listdir( src )
for i in files:
old = i
new, ext = os.path.splitext ( old )
os.rename( join( src, old ), join( src, fileName ))