Develop Reference

Python execute all Python files in directory and subdirs [duplicate]

This question already has answers here:
Command line execution in different folder
(3 answers)
Closed 29 days ago.
I'm trying using this code for the city wise updated population data but the python file is not writing the output results in that folder
import os
import subprocess
import glob
root_dir = "path/to/root/directory"
for dirpath, dirnames, filenames in os.walk(root_dir):
for text_file in glob.glob(os.path.join(dirpath, '*.txt')):
os.remove(text_file)
for filename in filenames:
if filename.endswith(".py"):
filepath = os.path.join(dirpath, filename)
os.system("python" + filepath)
or
subprocess.run(["python", filepath])
It is deleting the old text files but python file is not generating the updated data and in the sub process it showing in the command prompt but didn't write the text files or even creating new text files
but when I manually go to the folder and run the Python file it works fine

The issue with the line os.system("python" + filepath) is that there is no space between "python" and the file, so it will attempt to run something like pythontest.py, an invalid command. The second issue is that your program might run itself, causing an infinite loop. To fix this, check if the file is the current file with the __file__ variable before you run it. Try this code:
import os
import subprocess
import glob
root_dir = os.getcwd()
for dirpath, dirnames, filenames in os.walk(root_dir):
for text_file in glob.glob(os.path.join(dirpath, '*.txt')):
os.remove(text_file)
for filename in filenames:
if filename.endswith(".py") and os.path.join(root_dir, filename) != __file__:
filepath = os.path.join(dirpath, filename)
subprocess.run(["python", filepath])
Note that I also changed root_dir to automatically get the current directory. You can change this if you want.
Also, thanks to MrChadMWoods for helping catch an edge case in this current file detection.

You need to change the current working directory for your script to work:
Try:
subprocess.run(["python", filepath], cwd=dirpath)

Find JSON files in a directory and open them [duplicate]

This question already has an answer here:
Python: existing file not found (IOError: [Errno 2]) when using os.walk
(1 answer)
Closed 1 year ago.
My goal is to search for specific JSON files in a directory and to read them so that I can export the content of them as an Excel file.
DIRECTORY LISTING: \Linkedin\linkedin_hb_ma\2021-260\1eb95ebb-d87d-XX1-XXX-XXX1cX
Details: (linkedin hb_ma): the folder contains several folders (year - day) // (year - day): contains several folders with (member ID) // (member ID): contains a Json file
My code:
import os
import json
from pprint import pprint
import win32com.client as win32 # pip install pywin32
rootDir = 'C:/Users/Adam/Desktop/Linkedin/linkedin_hb_ma'
for dirName, subdirList, fileList in os.walk(rootDir , topdown=False):
if dirName.endswith("1eb95ebb-d87d-4aac-XX-XX182"):
abs_path = os.path.join(dirName, file)
print('Found directory: %s' % dirName)
#print(fileList)
for file in fileList:
if file.endswith("activities.json"):
#print('\t%s' % file)
json_data = json.loads(open(abs_path).read())
pprint(json_data)
Error: FileNotFoundError: [Errno 2] No such file or directory: 'activities.json'
NB: my python file is in another working directory.
Does anyone have an idea?
Thank you so much

The issue is that os.walk will not return the absolute file paths for the fileList you will need to concatenate the parent directory and the filename like this:
abs_path = os.path.join(dirName, file)
json_data = json.loads(open(abs_path).read())

Scan directory and rename files in Python [duplicate]

This question already has answers here:
Bug in Python Renaming Program.....No such file or Directory (Fnmatch)
(2 answers)
Closed 6 years ago.
I have 7 files in my web directory like this :
I'm trying to scan through all my files in that folder and rename a certain files.
I have
import os
path = '/Users/username/Desktop/web'
for filename in os.listdir(path):
if filename.startswith("test"):
print 'found'
os.rename(filename, filename.replace("test_", " "))
else:
continue
After run it,
python scan_dir.py
I got
found
Traceback (most recent call last):
File "scan_dir.py", line 9, in <module>
os.rename(filename, filename.replace("test_", " "))
OSError: [Errno 2] No such file or directory
Any hints on what I did wrong ?

When you rename you should use the full path:
import os
path = '/Users/name/Desktop/web'
for filename in os.listdir(path):
if filename.startswith("test"):
print 'found'
os.rename(os.path.join(path, filename), os.path.join(path, filename.replace("test_", " ")))
else:
continue
In your current code what you do use try to rename the file test_3.jpg (from your example) in the current directory, which probably don't exists.
BTW, I would concider using the glob function instead.

command line arguments using python

I have this code that will let the user choose which file he wants to update by passing an argument in the command line, and then it do some more things but I have not included that here:
import sys
import os
from sys import argv
path = "/home/Desktop/python/test"
files = os.walk( path )
filename = argv[1]
if filename in files:
inputFile = open(filename, 'r')
else:
print "no match found"
sys.exit()
inputFile.close()
When I run the script it keeps giving me "no match found" but im pretty sure the file is there. I cant see what Im doing wrong

os.walk() returns a generator, one that produces tuples with (root, directories, files) values for each iteration.
You can't use that generator to test for a single file, not with a simple in membership test.
You'll also need to re-instate the whole path; you can't just open an unclassified filename without the directory it lives in. Just use a for loop here, and break once you found it. The else suite on a for loop only executes when you did not use break (e.g. the file was not found):
path = "/home/Desktop/python/test"
filename = argv[1]
for root, directories, files in os.walk(path):
if filename in files:
full_path = os.path.join(root, filename)
break
else:
print "no match found"
sys.exit()
with open(full_path) as input_file:
# do something with the file
I added a with statement to handle the lifetime of the file object; once the with block is exited the file is automatically closed for you.

Alternatively, you may use following code snippet.
import os.path
filename = argv[1]
path = "/home/Desktop/python/test/"
if os.path.isfile(path + filename):
inputFile = open(path + filename, "r")
else:
print "File Not Found"

Python tarfile - check if file in tar exists outside (i.e., already been extracted)

I'm new to stackoverflow. Sorry if this post is redundant but I haven't found the answer yet. Also, I'm fairly new to Python. I'd like to extract files from a tar file if they do not already exist in the root directory where the tar file exists. I've tried a number of versions. I think there is some redundancy in the code below, and it doesn't do what I need it to. It just keeps extracting and overwriting the existing file(s).
Files that need to be extracted will always end in "_B7.TIF". Code currently takes one argument - the full path of the directory that contains the tar file.
import os, shutil, sys, tarfile
directory = sys.argv[1]
tifFiles = []
for root, dirs, files in os.walk(directory):
for file in files:
if file.endswith(".TIF"):
# also tried tifFiles.append(file)
tifFiles.append(file.name)
elif file.endswith(".tar.gz"):
tar = tarfile.open(root + "/" + file)
for item in tar:
if str(item) in tifFiles:
print "{0} has already been unzipped.".format(str(item))
elif "_B7" in str(item):
tar.extract(item, path=root)
shutil.rmtree(root + "\gap_mask")
Here is another version that does not appear to be doing anything. I was trying to simplify...
import os, shutil, sys, tarfile
directory = sys.argv[1]
for root, dirs, files in os.walk(directory):
if file not in tarfile.getnames() and file.endswith("_B7.TIF"):
tar.extract(file, path=root)
else:
print "File: {0} has already been unzipped.".format(file)
shutil.rmtree(root + "\gap_mask")
Thank you both for your comments/suggestions. They both helped in some way. This code works for me.
import os, shutil, sys, tarfile
folder = sys.argv[1]
listFiles = os.listdir(folder)
try:
for file in listFiles:
if file.endswith(".tar.gz"):
sceneTIF = file[:-7] + "_B7.TIF"
if os.path.exists(os.path.join(folder,sceneTIF)):
print sceneTIF, "has already been extracted."
else:
tar = tarfile.open(os.path.join(folder,file))
for item in tar:
if "_B7" in str(item):
tar.extract(item, path=folder)
shutil.rmtree(os.path.join(folder,"gap_mask")
except WindowsError:
pass
Any thoughts on style/redundancy/ways to make it better? Thomas, your code was not working straight out of the box. I think it was the tarfile.open component. Probably needed tarfile.open(os.path.join(directory, archive)). I only thought of that after reworking the above though. Haven't tested. Thanks again.

os.walk iterates over directory trees, including sub-directories. From your description that is not what you want. Also, only files that are encountered earlier than your tarfiles will be considered for existence.
It is a lot easier to just check for the existence of files you encounter:
import sys
import os
import tarfile
directory = sys.argv[1]
def extract_nonexisting(archive):
for name in archive.getnames():
if os.path.exists(os.path.join(directory, name)):
print name, "already exists"
else:
archive.extract(name, path=directory)
archives = [name for name in os.listdir(directory) if name.endswith("tar.gz")]
for archive_name in archives:
with tarfile.open(archive_name) as archive:
extract_nonexisting(archive)