Develop Reference

Rename certain subfolders in a directory using python

I have a folder structure that looks something like this:
/Forecasting/as_of_date=20220201/type=full/export_country=Spain/import_country=France/000.parquet'
and there are approx 2500 such structures.
I am trying to rename only certain subfolders, mainly export_country as exp_cty and import_country as imp_cty.
So far I tried this but it doesn't seem to work. I never had to deal with such complex folder structures and I'm a bit unsure how to go about it. My script is below:
import os
from pathlib import Path
path = r"/datasets/local/Trade_Data" # Path to directory that will be searched
old = "import_country*"
new = "imp_cty"
for root, dirs, files in os.walk(path, topdown=False): # os.walk will return all files and folders in thedirectory
for name in dirs: # only concerned with dirs since I only want to change subfolders
directoryPath = os.path.join(root, name) # create a path to subfolder
if old in directoryPath: # if the 'export_country' is found in my path then
parentDirectory = Path(directoryPath).parent # save the parent directory path
os.chdir(parentDirectory) # set parent to working directory
os.rename(old, new)

The code you have proposed has 2 issues:
The first one: if old in directoryPath: checks if the string import_country* is inside the path.
From your question I have understood that you would like to rename all directories that start with "import_country"
so you can use the startswith for that.
The second problem is os.rename(old, new) you are trying to rename directory with name import_country* which doesn't exist, instead you should use the name variable.
Here is your code with slightly changes that is working, please note that you must use topdown=False as you are renaming directories while walking through them:
import os
from pathlib import Path
path = "/datasets/local/Trade_Data"
old_prefix = "import_country"
new_name = "imp_cty"
for root, dirs, files in os.walk(path, topdown=False):
for name in dirs:
if name.startswith(old_prefix):
directoryPath = os.path.join(root, name)
parentDirectory = Path(directoryPath).parent
os.chdir(parentDirectory)
os.rename(name, new_name)

I want to add all the names of the files in a specific folder to a list

I want to add the names of all the files in a specific folder to a list how can i do that? the pathway is from dropbox -> a folder called 'UMM' -> a folder called '2018' could someone help me with the code on this. I have tried using os.walk() but it doesn't seem to work

You can use os.walk and append only names which are in files.
from os import walk
file_names = list()
path = 'path/of/folder'
for root, dirc, files in walk(path):
for FileName in files:
file_names.append(FileName)
print(file_names)
This will append all the files name from all the directories and sub-directories of the specified path.

this will create a list of the files in a folder
from os import listdir
# the path
path = ''
fileList = listdir(path)

Skip a certain folder when using os.walk

Here is my code:
rootdir_path_without_slash = '/home/winpc/Downloads/Prageeth/backups/Final/node-wen-app'
rootdir_path_with_slash= '/home/winpc/Downloads/Prageeth/backups/Final/node-wen-app/'
dir_src = (rootdir_path_with_slash)
for subdir, dirs, files in os.walk(rootdir_path_without_slash):
for file in files:
file_name=os.path.join(subdir, file)
if file_name.endswith('.html'):
print file_name
Here this code navigate all the sub directories from the given source directory for searching .html file.I need to skip if node modules folder found.Please help me.

You'll need to put an if condition on the root directory, to avoid traversing node_modules or any of its descendants. You'll want:
for subdir, dirs, files in os.walk(rootdir_path_without_slash):
if 'node_modules' in subdir:
continue
... # rest of your code
Also, subdir here is a misnomer, the first argument os.walk returns is the root path.

In python, how to get the path to all the files in a directory, including files in subdirectories, but excluding path to subdirectories

I have a directory containing folders and subfolders. At the end of each path there are files. I want to make a txt file containing the path to all the files, but excluding the path to folders.
I tried this suggestion from Getting a list of all subdirectories in the current directory, and my code looks like this:
import os
myDir = '/path/somewhere'
print [x[0] for x in os.walk(myDir)]
And it gives the path of all elements (files AND folders), but I want only the paths to the files. Any ideas for it?

os.walk(path) returns three tuples parent folder, sub directories and files.
so you can do like this:
for dir, subdir, files in os.walk(path):
for file in files:
print os.path.join(dir, file)

The os.walk method gives you dirs, subdirs and files in each iteration, so when you are looping through os.walk, you will have to then iterate over the files and combine each file with "dir".
In order to perform this combination, what you want to do is do an os.path.join between the directory and the files.
Here is a simple example to help illustrate how traversing with os.walk works
from os import walk
from os.path import join
# specify in your loop in order dir, subdirectory, file for each level
for dir, subdir, files in walk('path'):
# iterate over each file
for file in files:
# join will put together the directory and the file
print(join(dir, file))

If you just want the paths, then add a filter to your list comprehension as follows:
import os
myDir = '/path/somewhere'
print [dirpath for dirpath, dirnames, filenames in os.walk(myDir) if filenames]
This would then only add the path for folders which contain files.

def get_paths(path, depth=None):
for name in os.listdir(path):
full_path = os.path.join(path, name)
if os.path.isfile(full_path):
yield full_path
else:
d = depth - 1 if depth is not None else None
if d is None or d >= 0:
for sub_path in get_paths(full_path):
yield sub_path

Read structure in files python

I need to populate a txt with the structure in certain directory, i read the files in the directory:
os.listdir(r)
But if one item is a directory i need to search into it, any idea?

Use os.walk. It will recursively traverse paths. ex:
import os
for root, dirs, files in os.walk(top, topdown=False):
for name in files:
print(os.path.join(root, name))

You can use os.walk.
for root,dirs,files in os.walk(directory):
...