Develop Reference

How to use data files of sub-directories and perform iterative operation in python

I have my jupyter notebook (python script) in current directory. In current directory, I have two subfolders, namely a and b. In both directories a and b I have equal number of .dat files with same names. For example, directory a contains files, namely x1-x1-val_1, x1-x1-val_5, x1-x1-val_11...x1-x1-val_86 and x1-x2-val_1, x1-x2-val_5, x1-x2-val_11...x1-x2-val_86, i.e. values are in range(1,90,5). Likewise I have files in directory b.
I want to use my python script to access files in a and b to perform iterative operations on .dat files. My present code works only if I keep files of directory a or b in current directory. For example, my script uses following function.
def get_info(test):
my_dict = {'test':test}
c = []
for i in range(1,90,5):
x_val = 'x_val_'+test+'-val_'+str(i)
y_val = 'y_val_'+test+'-val_'+str(i)
my_dict[x_val],my_dict[y_val]= np.loadtxt(test+'-val_'+str(i)+'.dat'
,usecols= (1,2),unpack=True)
dw = compute_yy(my_dict[x_val],my_dict[y_val],test)
c.append(dw)
my_dict.update({test+'_c'+:np.array(c)})
return my_dict
I call get_info() by using following:
tests = ['x1-x1', 'x1-x2']
new_dict = {}
for i in tests:
new_dict.update({i:get_info(i)})
How can I use my code to access files in either directory a and/or b? I know its about providing correct path, but I am unsure how can I do so. One way I thought is following;
ext = '.dat'
for files in os.listdir(path_to_dir):
if files.endswith(ext):
print(files) # do operations
Alternative could be to make use of os.path.join(). However, I am unable to solve it such that I can use same python script (with minimum changes perhaps) that can use files and iterate on them which are in subfolders a and b. Thanks for your feedback in advance!

If you want to run get_info() on every folder separatelly then you have two methods:
First: described by #medium-dimensional in comment
You can use os.chdir(folder) to change Current Working Directory. And then code will run with files in this folder
You can see current working directory with print( os.getcwd() )
os.chdir("a")
get_info(i)
os.chdir("..") # move back to parent folder
os.chdir("b")
get_info(i)
os.chdir("..") # move back to parent folder
chdir() (similar to command cd in console) can use relative path (r"a") full path (r"C:\full\path\to\a") and .. to move to parent folder (r"a\..\b")
If files can be in nested folders then .. may not go back you can use getcwd()
cwd = os.getcwd()
os.chdir("folder1/folder2/a")
get_info(i)
os.chdir(cwd) # move back to previous folder
os.chdir("folder1/folder2/b")
get_info(i)
os.chdir(cwd) # move back to previous folder
(BTW: in console on Linux you can use cd - to move back to previous folder)
Second: use folder when you open file
Every command which gets filename can also get path with folder\filename (it can be relative path, full path, and path with ..) like
r"a\filename.dat"
r"C:\full\path\to\b\filename.dat"
r"a\..\b\filename.dat"
So you could define function with extra option folder
def get_info(text, folder):
and use this folder when you read file
loadtxt(folder + r'\' + test+'-val_'+str(i)+'.dat', ...)
or more readable with f-string
loadtxt(rf'{folder}\{test}-val_{i}.dat', ...)
And later you run it as
get_info(i, "a")
get_info(i, "b")

How can I use the pims.open() function under a for loop?

I am running a piece of code (using trackpy and pims libraries) on all images in a given folder. Now I need to make a loop which will access all the folders in a given folder and then sequentially run the same code over all the files in each folder one-by-one and save the output. I could access all the folders in a given folder using the os library like so:
path = '/file_path'
for root, directories, files in os.walk(path, topdown=False):
for name in directories:
print(os.path.join(root, subdirectory))
But if I try to add my code into this in which the first step is to open all the images in one specific folder using pims.open() it asks for the name of the folder to be specified that is it doesn't accept this code:
for name in directories:
variable = pims.open("name/*.jpg")
neither this:
for name in directories:
variable = pims.open("(os.path.join(name))/*.jpg")
The traceback I get is: This file does not exist.
Edit: As pointed out in the answers, I was referencing name as a string and not a variable but then when I followed
for name in directories:
variable =pims.open(name + '/*.jpg')
it is still unable to access all the .jpg files in the subfolders for my given folder. For example, right now I have just saved 5 .jpg images in one subfolder (subA) of a main folder (mainA). So the error I get now is: The file subA/*.jpg does not exist.

as mentioned by gshpychka you have a string "name/*.jpg". The name in there is not referencing the variable but just part of that string.
for name in directories:
variable = pims.open(name + '/*.jpg')
Or you can use f"{name}/*.jpg" to tell python that it should substitute {name} with the content of the variable name. Starting the string with f enables that kind of behaviour(E.g. f"here is {a} and {b}." would replace {a} with the contents of variable a and {b} b with the contents of variable b). This has the advantage, that it also works the same with non-string types while for the simple + you would need to explicitly convert others (like e.g. numeric types) into string types.
for name in directories:
variable = pims.open(f"{name}/*.jpg")
But you should probably use os.path.join as that reduces chances of errors a bit.
for name in directories:
variable = pims.open(os.path.join(name, '*.jpg'))
you can have a look there:
https://docs.python.org/3/library/os.path.html#os.path.join
or
https://www.geeksforgeeks.org/python-os-path-join-method/

You could try f-strings:
for name in directories:
variable = pims.open(f"{os.path.join(name)}/*.jpg")

Python: For each directory in the working directory go into that directory and get the name of the directory

I need to make a script that will iterate through all the directories inside a directory. It should go into each directory, get its name and save it to a variable and comes back out, and then loops.
for dir in os.walk(exDir):
path = dir
os.chdir(path)
source = #dir trimmed to anything after the last /
os.chdir("..")
loops
It needs to go into the directory to do other things not mentioned above. I've only just started Python and have been stuck on this problem for the last day or so.

For each iteration of your for loop, dir is a tuple of format (filepath, subdirectories, files). As such dir[0] will give you the filepath.
It sounds like you just want to os.chdir for each folder recursively in exDir in which case the following will work:
for dir in os.walk(exDir):
os.chdir(dir[0])
...

Renaming files in Python

I'm doing a Python course on Udacity. And there is a class called Rename Troubles, which presents the following incomplete code:
import os
def rename_files():
file_list = os.listdir("C:\Users\Nick\Desktop\temp")
print (file_list)
for file_name in file_list:
os.rename(file_name, file_name.translate(None, "0123456789"))
rename_files()
As the instructor explains it, this will return an error because Python is not attempting to rename files in the right folder. He then proceeds to check the "current working directory", and then goes on to specify to Python which directory to rename files in.
This makes no sense to me. We are using the for loop to specifically tell Python that we want to rename the contents of file_list, which we have just pointed to the directory we need, in rename_files(). So why does it not attempt to rename in that folder? Why do we still need to figure out cwd and then change it?? The code looks entirely logical without any of that.

Look closely at what os.listdir() gives you. It returns only a list of names, not full paths.
You'll then proceed to os.rename one of those names, which will be interpreted as a relative path, relative to whatever your current working directory is.
Instead of messing with the current working directory, you can os.path.join() the path that you're searching to the front of both arguments to os.rename().

I think your code needs some formatting help.
The basic issue is that os.listdir() returns names relative to the directory specified (in this case an absolute path). But your script can be running from any directory. Manipulate the file names passed to os.rename() to account for this.

Look into relative and absolute paths, listdir returns names relative to the path (in this case absolute path) provided to listdir. os.rename is then given this relative name and unless the app's current working directory (usually the directory you launched the app from) is the same as provided to listdir this will fail.
There are a couple of alternative ways of handling this, changing the current working directory:
os.chdir("C:\Users\Nick\Desktop\temp")
for file_name in os.listdir(os.getcwd()):
os.rename(file_name, file_name.translate(None, "0123456789"))
Or use absolute paths:
directory = "C:\Users\Nick\Desktop\temp"
for file_name in os.listdir(directory):
old_file_path = os.path.join(directory, file_name)
new_file_path = os.path.join(directory, file_name.translate(None, "0123456789"))
os.rename(old_file_path, new_file_path)

You can get a file list from ANY existing directory - i.e.
os.listdir("C:\Users\Nick\Desktop\temp")
or
os.listdir("C:\Users\Nick\Desktop")
or
os.listdir("C:\Users\Nick")
etc.
The instance of the Python interpreter that you're using to run your code is being executed in a directory that is independent of any directory for which you're trying to get information. So, in order to rename the correct file, you need to specify the full path to that file (or the relative path from wherever you're running your Python interpreter).

Change to a known directory name but unkown absolute path in Python

I would like to change the cwd to a specific folder.
The folder name is known; however, the path to it will vary.
I am attempting the following but cannot seem to get what I am looking for:
absolute_path = os.path.abspath(folder_name)
directory_path = os.path.dirname(absolute_path)
os.chdir(directory_path)
This does not do what I'm looking for because it is keeping the original cwd to where the .py file is run from. I've tried adding os.chdir(os.path.expanduser("~")) prior to the first code block; however, it just creates the absolute_path to /home/user/folder_name.
Of course if there is a simple import that I could use, I'll be open to anything.
What would be the correct way to get the paths of all folders with with a specific name?

def find_folders(start_path,needle):
for cwd, folders, files,in os.walk(start_path):
if needle in folders:
yield os.path.join(cwd,needle)
for path in find_folders("/","a_folder_named_x"):
print path
all this is doing is walking down your directory structure from a given start path and finding all occurances of a folder named needle
in the example it is starting at the root folder of the system and looking for a folder named "a_folder_named_x" ... be forwarned this could take a while to run if you need to search the whole system ...

You need to understand that abspath accepts a relative pathname (which might just be a filename), and gives you the equivalent absolute (full) pathname. A relative pathname is one that begins in your current directory; no searching is involved, and so it always points to one place (which may or may not exist).
What you actually need is to search down a directory tree, starting at ~ or whatever directory makes sense in your case, until you find a folder with the requested name. That's what #Joran's code does.