Develop Reference

Loop through each subdirectory in a main directory and run code against each file using OS

Essentially what I'm trying to do is loop through a directory that contains multiple sub-directories and within those run code against each file in a for loop.
The only start I managed to make was listing the directories but as I've rarely ever used os I'm not sure if I could potentially loop through os.chdir and a bit of f string formatting to loop through each subdirectory.
The files I want to run code against are just txt files.
Here goes my code, up to the moment:
import os
for folders in os.listdir('../main_directory'):
for something in os.listdir(f'{folders}'):
# run some function of sorts
pass
Any help would be greatly appreciated.

I like using pure os:
import os
for fname in os.listdir(src):
# build the path to the folder
folder_path = os.path.join(src, fname)
if os.path.isdir(folder_path):
# we are sure this is a folder; now lets iterate it
for file_name in os.listdir(folder_path):
file_path = os.path.join(folder_path, file_name)
# now you can apply any function assuming it is a file
# or double check it if needed as `os.path.isfile(file_path)`
Note that this function just iterate over the folder given at src and one more level:
src/foo.txt # this file is ignored
src/foo/a.txt # this file is processed
src/foo/foo_2/b.txt # this file is ignored; too deep.
src/foo/foo_2/foo_3/c.txt # this file is ignored; too deep.
In case you need to go as deep as possible, you can write a recursive function and apply it to every single file, as follows:
import os
def function_over_files(path):
if os.path.isfile(path):
# do whatever you need with file at path
else:
# this is a dir: we will list all files on it and call recursively
for fname in os.listdir(path):
f_path = os.path.join(path, fname)
# here is the trick: recursive call to the same function
function_over_files(f_path)
src = "path/to/your/dir"
function_over_files(src)
This way you can apply the function to any file under path, don't care how deep it is in the folder, now:
src/foo.txt # this file is processed; as each file under src
src/foo/a.txt # this file is processed
src/foo/foo_2/b.txt # this file is processed
src/foo/foo_2/foo_3/c.txt # this file is processed

You could try something like this:
for subdir, dirs, files in os.walk(rootdir):
Now you have "access" to all subdirs, dirs, and files for your main folder.
Hope it helps

How do I move both files and folders to the specified directory?

There is a code that moves files from one directory to another, but it doesn't move folders.
import os,glob
import shutil
inpath = str5
outpath = str6
os.chdir(inpath)
for file in glob.glob("*.*"):
shutil.move(inpath+'/'+file,outpath)
How to make it move both files and folders to the specified directory?

*.* selects files that have an extension, so it omits sub-folders.
Use * to select files and folders.
Then you should see your desired result.
for file in glob.glob("*"):
shutil.move(inpath+'/'+file,outpath)

You can use os.listdir to get all the files and folders in a directory.
import os
import shutil
def move_file_and_folders(inpath, outpath):
for filename in os.listdir(inpath):
shutil.move(os.path.join(inpath, filename), os.path.join(outpath, filename))
In your case,
inpath = <specify the source>
outpath = <specify the destination>
move_file_and_folders(inpath, outpath)

Python OS - using 'os.listdir' to view a directory that returns a list of folders. How do I view the contents of the subfolders?

I have been set a challenge to browse a directory that contains mostly empty folders - the flag(answer) is in one of them. I have used os module to see all the names of the folders - they are all named 'folder-' plus a number between 1 and 200. How do I view what is inside them?

You should use os.walk() instead of litdir() like as
import os
import os.path
for dirpath, dirnames, filenames in os.walk("."):
for file in filenames:
print(file)

import os
def getListOfFiles(dirName):
# create a list of file and sub directories
# names in the given directory
File_list = os.listdir(dirName)
Files = list()
# Iterate over all the entries
for entry in File_list:
# Create full path
fullPath = os.path.join(dirName, entry)
# If entry is a directory then get the list of files in this directory
if os.path.isdir(fullPath):
Files = Files + getListOfFiles(fullPath)
else:
Files.append(fullPath)
return Files
#Call the above function to create a list of files in a directory tree i.e.
dirName = 'C:/Users/huzi95s/Desktop/Django';
# Get the list of all files in directory tree at given path
listOfFiles = getListOfFiles(dirName)

How can I extract all .zip extension in a folder without retaining directory using python?

Here is my code I don't know how can I loop every .zip in a folder, please help me: I want all contents of 5 zip files to extracted in one folder, not including its directory name
import os
import shutil
import zipfile
my_dir = r"C:\\Users\\Guest\\Desktop\\OJT\\scanner\\samples_raw"
my_zip = r"C:\\Users\\Guest\\Desktop\\OJT\\samples\\001-100.zip"
with zipfile.ZipFile(my_zip) as zip_file:
zip_file.setpassword(b"virus")
for member in zip_file.namelist():
filename = os.path.basename(member)
# skip directories
if not filename:
continue
# copy file (taken from zipfile's extract)
source = zip_file.open(member)
target = file(os.path.join(my_dir, filename), "wb")
with source, target:
shutil.copyfileobj(source, target)

repeated question, please refer below link.
How to extract zip file recursively in Pythonn

What you are looking for is glob. Which can be used like this:
#<snip>
import glob
#assuming all your zip files are in the directory below.
for my_zip in glob.glob(r"C:\\Users\\Guest\\Desktop\\OJT\\samples\\*.zip"):
with zipfile.ZipFile(my_zip) as zip_file:
zip_file.setpassword(b"virus")
for member in zip_file.namelist():
#<snip> rest of your code here.

Open a file without specifying the subdirectory python

Lets say my python script is in a folder "/main". I have a bunch of text files inside subfolders in main. I want to be able to open a file just by specifying its name, not the subdirectory its in.
So open_file('test1.csv') should open test1.csv even if its full path is /main/test/test1.csv.
I don't have duplicated file names so it should no be a problem.
I using windows.

you could use os.walk to find your filename in a subfolder structure
import os
def find_and_open(filename):
for root_f, folders, files in os.walk('.'):
if filename in files:
# here you can either open the file
# or just return the full path and process file
# somewhere else
with open(root_f + '/' + filename) as f:
f.read()
# do something
if you have a very deep folder structure you might want to limit the depth of the search

import os
def get_file_path(file):
for (root, dirs, files) in os.walk('.'):
if file in files:
return os.path.join(root, file)
This should work. It'll return the path, so you should handle opening the file, in your code.

import os
def open_file(filename):
f = open(os.path.join('/path/to/main/', filename))
return f