Develop Reference

Loop through each subdirectory in a main directory and run code against each file using OS

Essentially what I'm trying to do is loop through a directory that contains multiple sub-directories and within those run code against each file in a for loop.
The only start I managed to make was listing the directories but as I've rarely ever used os I'm not sure if I could potentially loop through os.chdir and a bit of f string formatting to loop through each subdirectory.
The files I want to run code against are just txt files.
Here goes my code, up to the moment:
import os
for folders in os.listdir('../main_directory'):
for something in os.listdir(f'{folders}'):
# run some function of sorts
pass
Any help would be greatly appreciated.

I like using pure os:
import os
for fname in os.listdir(src):
# build the path to the folder
folder_path = os.path.join(src, fname)
if os.path.isdir(folder_path):
# we are sure this is a folder; now lets iterate it
for file_name in os.listdir(folder_path):
file_path = os.path.join(folder_path, file_name)
# now you can apply any function assuming it is a file
# or double check it if needed as `os.path.isfile(file_path)`
Note that this function just iterate over the folder given at src and one more level:
src/foo.txt # this file is ignored
src/foo/a.txt # this file is processed
src/foo/foo_2/b.txt # this file is ignored; too deep.
src/foo/foo_2/foo_3/c.txt # this file is ignored; too deep.
In case you need to go as deep as possible, you can write a recursive function and apply it to every single file, as follows:
import os
def function_over_files(path):
if os.path.isfile(path):
# do whatever you need with file at path
else:
# this is a dir: we will list all files on it and call recursively
for fname in os.listdir(path):
f_path = os.path.join(path, fname)
# here is the trick: recursive call to the same function
function_over_files(f_path)
src = "path/to/your/dir"
function_over_files(src)
This way you can apply the function to any file under path, don't care how deep it is in the folder, now:
src/foo.txt # this file is processed; as each file under src
src/foo/a.txt # this file is processed
src/foo/foo_2/b.txt # this file is processed
src/foo/foo_2/foo_3/c.txt # this file is processed

You could try something like this:
for subdir, dirs, files in os.walk(rootdir):
Now you have "access" to all subdirs, dirs, and files for your main folder.
Hope it helps

Opening a file with python in the same directory from different locations

I am currently accessing a script that opens a file in the directory it's located in. I am accessing this file from both the main.py file located in the same directory, as well as a testfile which is located in a "Test" subdirectory. Trying to use a file from the Test subdirectory to call the function that opens the file causes the script to try and open it from the Test directory instead of the super directory, since I am opening the file simply by calling it as following:
with open(filename,"w") as f:
Is there a way to define the location of the file in a way that makes it possible for the script opening it to be called from anywhere?

Use __file__ to get the path to the current script file, then find the file relative to that:
# In main.py: find the file in the same directory as this script
import os.path
open(os.path.join(os.path.dirname(__file__), 'file.txt'))
# In Test/test.py: find the file one directory up from this script
import os.path
open(os.path.join(os.path.dirname(__file__), '..', 'file.txt'))

just give the absolute file path instead of giving a relative one
for eg
abs_path = '/home/user/project/file'
with open(abs_path, 'r') as f:
f.write(data)

Try specifying the path:
import os
path = 'Your path'
path = os.path.abspath(path)
with open(path, 'w') as f:
f.write(data)

From what I understood your file is in a directory parent_directory/file_name.txt
and in another folder parent_directory/sub_directory/file_name.txt. All you have to do is paste the below code in both parent and sub directories.
import os
file_name = 'your_file_name'
# if the file is in current directory set the path to file_name
if file_name in os.listdir(os.getcwd()):
path = file_name
# if the path is not in current directory go back to parent directory set the path to parent directory
else:
path = os.path.abspath(os.path.join(os.getcwd(), os.pardir))
print('from',os.getcwd())
with open(path, 'r') as filename:
print(filename.read())

How to open a file only using its extension?

I have a Python script which opens a specific text file located in a specific directory (working directory) and perform some actions.
(Assume that if there is a text file in the directory then it will always be no more than one such .txt file)
with open('TextFileName.txt', 'r') as f:
for line in f:
# perform some string manipulation and calculations
# write some results to a different text file
with open('results.txt', 'a') as r:
r.write(someResults)
My question is how I can have the script locate the text (.txt) file in the directory and open it without explicitly providing its name (i.e. without giving the 'TextFileName.txt'). So, no arguments for which text file to open would be required for this script to run.
Is there a way to achieve this in Python?

You could use os.listdir to get the files in the current directory, and filter them by their extension:
import os
txt_files = [f for f in os.listdir('.') if f.endswith('.txt')]
if len(txt_files) != 1:
raise ValueError('should be only one txt file in the current directory')
filename = txt_files[0]

You Can Also Use glob Which is easier than os
import glob
text_file = glob.glob('*.txt')
# wild card to catch all the files ending with txt and return as list of files
if len(text_file) != 1:
raise ValueError('should be only one txt file in the current directory')
filename = text_file[0]
glob searches the current directory set by os.curdir
You can change to the working directory by setting
os.chdir(r'cur_working_directory')

Since Python version 3.4, it is possible to use the great pathlib library. It offers a glob method which makes it easy to filter according to extensions:
from pathlib import Path
path = Path(".") # current directory
extension = ".txt"
file_with_extension = next(path.glob(f"*{extension}")) # returns the file with extension or None
if file_with_extension:
with open(file_with_extension):
...

self replicating file, retrieve name of executed script

Is there a way to change os.path.basename(__file__) to the name of a file executed within the current running script?
I've created a simple self replicating file. It starts by accessing the root directory. It then sorts through all writeable sub-directories and picks a random one out of the list. After that it writes a .py file with random string of numbers as it's name. It then uses os.path.basename(__file__) to retrieve the file name and copy the lines to the newly written file. After all that is said and done, it executes the newly created file which repeats the process again. However, on the second round through, it retains __file__ for the original file name and not the newly generated file.
This portion stores the original directory path and file name
import os
import random
orig_dir = os.getcwd()
orig_file = os.path.basename(__file__)
This portion is the file writing process and execution of the new file.
NOTE: the path variable is a randomly picked sub-directory of the root that contains the newly written file.
file_create = open(rand_name, "w")
file_create.close()
file_create = open(rand_name, "a")
os.chdir(orig_dir)
copy_file = open(orig_file, "r")
for line in copy_file.readlines():
file_create.write(line)
file_create.close()
copy_file.close()
os.chdir(path)
exec(open(rand_name).read())
UPDATE:
I'm posting the entire script below. There are a few brute and ugly approaches I too to get it working, but I'm more focused on the problem at hand.
import os
import random
orig_dir = os.getcwd()
orig_file = os.path.basename(__file__)
for loop in range(20):
os.chdir("..")
root = os.getcwd()
direct = []
for items in os.listdir():
if os.path.isdir(items) and os.access(items, os.W_OK) and os.access(items, os.R_OK):
direct.append(items)
path = os.path.join(os.getcwd(), random.choice(direct))
rand_name = str(random.randint(1, 10000000000)) + ".py"
os.chdir(path)
print(path)
# prevents the creation of a duplicate file name in the same location.
# If it is, loops the name generator until a valid name is generated
while True:
if os.path.isfile(rand_name):
rand_name = str(random.randint(1, 10000000000)) + ".py"
else:
break
file_create = open(rand_name, "w")
file_create.close()
file_create = open(rand_name, "a")
os.chdir(orig_dir)
copy_file = open(orig_file, "r")
for line in copy_file.readlines():
file_create.write(line)
file_create.close()
copy_file.close()
os.chdir(path)
os.execl(rand_name, ''),
os.exec(path, '') is the last solution I tried but it started to return PermissionError: [Errno 13] Permission denied. I tried to run it with sudo privileges as well and it returned the same permission error.

Renaming multiple files in a directory using Python

I'm trying to rename multiple files in a directory using this Python script:
import os
path = '/Users/myName/Desktop/directory'
files = os.listdir(path)
i = 1
for file in files:
os.rename(file, str(i)+'.jpg')
i = i+1
When I run this script, I get the following error:
Traceback (most recent call last):
File "rename.py", line 7, in <module>
os.rename(file, str(i)+'.jpg')
OSError: [Errno 2] No such file or directory
Why is that? How can I solve this issue?
Thanks.

You are not giving the whole path while renaming, do it like this:
import os
path = '/Users/myName/Desktop/directory'
files = os.listdir(path)
for index, file in enumerate(files):
os.rename(os.path.join(path, file), os.path.join(path, ''.join([str(index), '.jpg'])))
Edit: Thanks to tavo, The first solution would move the file to the current directory, fixed that.

You have to make this path as a current working directory first.
simple enough.
rest of the code has no errors.
to make it current working directory:
os.chdir(path)

import os
from os import path
import shutil
Source_Path = 'E:\Binayak\deep_learning\Datasets\Class_2'
Destination = 'E:\Binayak\deep_learning\Datasets\Class_2_Dest'
#dst_folder = os.mkdir(Destination)
def main():
for count, filename in enumerate(os.listdir(Source_Path)):
dst = "Class_2_" + str(count) + ".jpg"
# rename all the files
os.rename(os.path.join(Source_Path, filename), os.path.join(Destination, dst))
# Driver Code
if __name__ == '__main__':
main()

As per #daniel's comment, os.listdir() returns just the filenames and not the full path of the file. Use os.path.join(path, file) to get the full path and rename that.
import os
path = 'C:\\Users\\Admin\\Desktop\\Jayesh'
files = os.listdir(path)
for file in files:
os.rename(os.path.join(path, file), os.path.join(path, 'xyz_' + file + '.csv'))

Just playing with the accepted answer define the path variable and list:
path = "/Your/path/to/folder/"
files = os.listdir(path)
and then loop over that list:
for index, file in enumerate(files):
#print (file)
os.rename(path+file, path +'file_' + str(index)+ '.jpg')
or loop over same way with one line as python list comprehension :
[os.rename(path+file, path +'jog_' + str(index)+ '.jpg') for index, file in enumerate(files)]
I think the first is more readable, in the second the first part of the loop is just the second part of the list comprehension

If your files are renaming in random manner then you have to sort the files in the directory first. The given code first sort then rename the files.
import os
import re
path = 'target_folder_directory'
files = os.listdir(path)
files.sort(key=lambda var:[int(x) if x.isdigit() else x for x in re.findall(r'[^0-9]|[0-9]+', var)])
for i, file in enumerate(files):
os.rename(path + file, path + "{}".format(i)+".jpg")

I wrote a quick and flexible script for renaming files, if you want a working solution without reinventing the wheel.
It renames files in the current directory by passing replacement functions.
Each function specifies a change you want done to all the matching file names. The code will determine the changes that will be done, and displays the differences it would generate using colors, and asks for confirmation to perform the changes.
You can find the source code here, and place it in the folder of which you want to rename files https://gist.github.com/aljgom/81e8e4ca9584b481523271b8725448b8
It works in pycharm, I haven't tested it in other consoles
The interaction will look something like this, after defining a few replacement functions
when it's running the first one, it would show all the differences from the files matching in the directory, and you can confirm to make the replacements or no, like this

This works for me and by increasing the index by 1 we can number the dataset.
import os
path = '/Users/myName/Desktop/directory'
files = os.listdir(path)
index=1
for index, file in enumerate(files):
os.rename(os.path.join(path, file),os.path.join(path,''.join([str(index),'.jpg'])))
index = index+1
But if your current image name start with a number this will not work.