Develop Reference

How to use data files of sub-directories and perform iterative operation in python

I have my jupyter notebook (python script) in current directory. In current directory, I have two subfolders, namely a and b. In both directories a and b I have equal number of .dat files with same names. For example, directory a contains files, namely x1-x1-val_1, x1-x1-val_5, x1-x1-val_11...x1-x1-val_86 and x1-x2-val_1, x1-x2-val_5, x1-x2-val_11...x1-x2-val_86, i.e. values are in range(1,90,5). Likewise I have files in directory b.
I want to use my python script to access files in a and b to perform iterative operations on .dat files. My present code works only if I keep files of directory a or b in current directory. For example, my script uses following function.
def get_info(test):
my_dict = {'test':test}
c = []
for i in range(1,90,5):
x_val = 'x_val_'+test+'-val_'+str(i)
y_val = 'y_val_'+test+'-val_'+str(i)
my_dict[x_val],my_dict[y_val]= np.loadtxt(test+'-val_'+str(i)+'.dat'
,usecols= (1,2),unpack=True)
dw = compute_yy(my_dict[x_val],my_dict[y_val],test)
c.append(dw)
my_dict.update({test+'_c'+:np.array(c)})
return my_dict
I call get_info() by using following:
tests = ['x1-x1', 'x1-x2']
new_dict = {}
for i in tests:
new_dict.update({i:get_info(i)})
How can I use my code to access files in either directory a and/or b? I know its about providing correct path, but I am unsure how can I do so. One way I thought is following;
ext = '.dat'
for files in os.listdir(path_to_dir):
if files.endswith(ext):
print(files) # do operations
Alternative could be to make use of os.path.join(). However, I am unable to solve it such that I can use same python script (with minimum changes perhaps) that can use files and iterate on them which are in subfolders a and b. Thanks for your feedback in advance!

If you want to run get_info() on every folder separatelly then you have two methods:
First: described by #medium-dimensional in comment
You can use os.chdir(folder) to change Current Working Directory. And then code will run with files in this folder
You can see current working directory with print( os.getcwd() )
os.chdir("a")
get_info(i)
os.chdir("..") # move back to parent folder
os.chdir("b")
get_info(i)
os.chdir("..") # move back to parent folder
chdir() (similar to command cd in console) can use relative path (r"a") full path (r"C:\full\path\to\a") and .. to move to parent folder (r"a\..\b")
If files can be in nested folders then .. may not go back you can use getcwd()
cwd = os.getcwd()
os.chdir("folder1/folder2/a")
get_info(i)
os.chdir(cwd) # move back to previous folder
os.chdir("folder1/folder2/b")
get_info(i)
os.chdir(cwd) # move back to previous folder
(BTW: in console on Linux you can use cd - to move back to previous folder)
Second: use folder when you open file
Every command which gets filename can also get path with folder\filename (it can be relative path, full path, and path with ..) like
r"a\filename.dat"
r"C:\full\path\to\b\filename.dat"
r"a\..\b\filename.dat"
So you could define function with extra option folder
def get_info(text, folder):
and use this folder when you read file
loadtxt(folder + r'\' + test+'-val_'+str(i)+'.dat', ...)
or more readable with f-string
loadtxt(rf'{folder}\{test}-val_{i}.dat', ...)
And later you run it as
get_info(i, "a")
get_info(i, "b")

Why does a python program continue to write to the directory in which it was originally located, after I move the file to another directory?

I have 2 directories. I had a python program located in dir_1 writing to a .txt file also in dir_1. I meant to create them in dir_2, but when I move them both to dir_2, the python program, instead of writing to the existing .txt file that is now with it in dir_2, creates a new .txt file in dir_1 and writes to it. How do I fix this? I'm very new to programming and python and googling didn't help me out, probably because I didn't know what exactly to search.
with open('guest_book.txt', 'w') as file:
while True:
name = input('Please enter your name: ')
if name == 'q':
break
else:
print(f"Hello, {name.title()}!\nYou have been added to the guest"
f"book")
file.write(f"{name.title()}\n")

Python writes to the file location you supply it with. If this file location is a relative path, then it will create files relative to the directory of the script. I.e. when you move the script then the .txt file will be created relative to the new directiory.
On the other hand, if you provide an absolute path, then it does not matter where the script is located / where you execute it from. Instead, it will create the file at that location always.
From the sounds of it, you are using an absolute path when you want a relative path.
So change from something like /home/bob/file.txt (Linux) or C:\\Users\Bob\file.txt (Win) to simply file.txt or even ./file.txt.
Update: Since you were using a relative location all along, the problem will lie with the context that you are executing the script from. Your code is not the issue here, it is how you are executing it.
As vlovero suggests, maybe your IDE is not executing the new file in its new location?
One way you can test this robustly is to navigate to dir_2 in a terminal and run
python your_program_name.py
This will execute the script in the dir_2 location.

Since you have not specified an absolute path, your program is then specifying a directory relative to the current working directory (if instead, for example, you had specified a path such as '../guest_book.txt', you would have been specifying a directory one level above the current working directory). So let's imagine your OS is Linux and the Python program resides in /my_home/programs:
cd /my_home/data # this is the current working directory
python ../programs/your_program.py
The current working directory when the program is executed is /home/my_home/data even though the program being executed resides in /my_home/programs, and thus the output file will be created in the /my_home/data directory. os.getcwd() can be called to tell you what the current working directory is.

How to get higher directory folder in python project?

It drives me crazy so I decided to ask here. I need to save the generated txt file (from lower structures of the project), to folder which is the sibling of the parent directory. I have used variety of different code along with all possibiliteis given by os.path.join etc and still nothing.
My structure:
--reports
--parent folder
--another folder
--another folder
-file.py
My lates code (based on string):
abs_dir = str(Path(__file__))
i = abs_dir.index("master")
self.new_dir = os.path.join(abs_dir[:i]+f"reports//log({self.date}).txt")

Use double dots (..), that will go to the parent directory
If you want to get to a folder in a parent folder
f1
- file.py
f2
- a.txt
you could just do ../f2/a.txt
In your case, you would have to go 3 folders up: ../../../reports/log...

How do I create a string containing the filepath of my python program?

We are creating a python program that executes specific macros within Polyworks based on user input into the program. Right now the code is:
roto.command.CommandExecute('MACRO EXEC("C:\\RotoWorks\\Macros\\CentrifugalCompressor")')
However this assumes that our program is always installed in C:\RotoWorks. Ideally, our app is portable. I'm sure theres a way to retrieve the filepath that Rotoworks is stored in, then just concatenate the rest of the filepath to the end. How do I do this?

You can retrieve the path from the __file__ attribute of the file. Use os.path.abspath on that attribute to retrieve the absolute path of the file and then os.path.dirname to retrieve the containing directory:
import os
file_directory = os.path.dirname(os.path.abspath(__file__))
path = os.path.join(file_directory, other_path) # join directory to an inner path
roto.command.CommandExecute('MACRO EXEC({})'.format(path))
Use os.path.dirname recursively to move out as many directories as you want.

Change to a known directory name but unkown absolute path in Python

I would like to change the cwd to a specific folder.
The folder name is known; however, the path to it will vary.
I am attempting the following but cannot seem to get what I am looking for:
absolute_path = os.path.abspath(folder_name)
directory_path = os.path.dirname(absolute_path)
os.chdir(directory_path)
This does not do what I'm looking for because it is keeping the original cwd to where the .py file is run from. I've tried adding os.chdir(os.path.expanduser("~")) prior to the first code block; however, it just creates the absolute_path to /home/user/folder_name.
Of course if there is a simple import that I could use, I'll be open to anything.
What would be the correct way to get the paths of all folders with with a specific name?

def find_folders(start_path,needle):
for cwd, folders, files,in os.walk(start_path):
if needle in folders:
yield os.path.join(cwd,needle)
for path in find_folders("/","a_folder_named_x"):
print path
all this is doing is walking down your directory structure from a given start path and finding all occurances of a folder named needle
in the example it is starting at the root folder of the system and looking for a folder named "a_folder_named_x" ... be forwarned this could take a while to run if you need to search the whole system ...

You need to understand that abspath accepts a relative pathname (which might just be a filename), and gives you the equivalent absolute (full) pathname. A relative pathname is one that begins in your current directory; no searching is involved, and so it always points to one place (which may or may not exist).
What you actually need is to search down a directory tree, starting at ~ or whatever directory makes sense in your case, until you find a folder with the requested name. That's what #Joran's code does.