Develop Reference

Python execute all Python files in directory and subdirs [duplicate]

This question already has answers here:
Command line execution in different folder
(3 answers)
Closed 29 days ago.
I'm trying using this code for the city wise updated population data but the python file is not writing the output results in that folder
import os
import subprocess
import glob
root_dir = "path/to/root/directory"
for dirpath, dirnames, filenames in os.walk(root_dir):
for text_file in glob.glob(os.path.join(dirpath, '*.txt')):
os.remove(text_file)
for filename in filenames:
if filename.endswith(".py"):
filepath = os.path.join(dirpath, filename)
os.system("python" + filepath)
or
subprocess.run(["python", filepath])
It is deleting the old text files but python file is not generating the updated data and in the sub process it showing in the command prompt but didn't write the text files or even creating new text files
but when I manually go to the folder and run the Python file it works fine

The issue with the line os.system("python" + filepath) is that there is no space between "python" and the file, so it will attempt to run something like pythontest.py, an invalid command. The second issue is that your program might run itself, causing an infinite loop. To fix this, check if the file is the current file with the __file__ variable before you run it. Try this code:
import os
import subprocess
import glob
root_dir = os.getcwd()
for dirpath, dirnames, filenames in os.walk(root_dir):
for text_file in glob.glob(os.path.join(dirpath, '*.txt')):
os.remove(text_file)
for filename in filenames:
if filename.endswith(".py") and os.path.join(root_dir, filename) != __file__:
filepath = os.path.join(dirpath, filename)
subprocess.run(["python", filepath])
Note that I also changed root_dir to automatically get the current directory. You can change this if you want.
Also, thanks to MrChadMWoods for helping catch an edge case in this current file detection.

You need to change the current working directory for your script to work:
Try:
subprocess.run(["python", filepath], cwd=dirpath)

How to use os.system to convert all files in a folder at once using external python script

I've managed to find out the method to convert a file from one file extension to another (.evtx to .xml) using an external script. Below is what I am using:
os.system("file_converter.py file1.evtx > file1.xml")
This successfully converts a file from .txt to .xml using the external script I called (file_converter.py).
I am now trying to find out a method on how I can use 'os.system' or perhaps another method to convert more than one file at once, I would like for my program to dive into a folder and convert all of the 10 files I have at once to .xml format.
The questions I have are how is this possible as os.system only takes 1 argument and I'm not sure on how I could make it locate through a directory as unlike the first file I converted was on my standard home directory, but the folder I want to access with the 10 files is inside of another folder, I am trying to find out a way to address this argument and for the conversion to be done at once, I also want the file name to stay the same for each individual file with the only difference being the '.xml' being changed from '.evtx' at the end.
The file "file_converter.py" is downloadable from here

import threading
import os
def file_converter(file):
os.system("file_converter.py {0} > {1}".format(file, file.replace(".evtx", ".xml")))
base_dir = "C:\\Users\\carlo.zanocco\\Desktop\\test_dir\\"
for file in os.listdir(base_dir):
threading.Thread(target=file_converter, args=(file,)).start()
Here my sample code.
You can generate multiple thread to run the operation "concurrently". The program will check for all files in the directory and convert it.
EDIT python2.7 version
Now that we have more information about what you want I can help you.
This program can handle multiple file concurrently from one folder, it check also into the subfolders.
import subprocess
import os
base_dir = "C:\\Users\\carlo.zanocco\\Desktop\\test_dir\\"
commands_to_run = list()
#Search all files
def file_list(directory):
allFiles = list()
for entry in os.listdir(directory):
fullPath = os.path.join(directory, entry)
#if is directory search for more files
if os.path.isdir(fullPath):
allFiles = allFiles + file_list(fullPath)
else:
#check that the file have the right extension and append the command to execute later
if(entry.endswith(".evtx")):
commands_to_run.append("C:\\Python27\\python.exe file_converter.py {0} > {1}".format(fullPath, fullPath.replace(".evtx", ".xml")))
return allFiles
print "Searching for files"
file_list(base_dir)
print "Running conversion"
processes = [subprocess.Popen(command, shell=True) for command in commands_to_run]
print "Waiting for converted files"
for process in processes:
process.wait()
print "Conversion done"
The subprocess module can be used in two ways:
subprocess.Popen: it run the process and continue the execution
subprocess.call: it run the process and wait for it, this function return the exit status. This value if zero indicate that the process terminate succesfully
EDIT python3.7 version
if you want to solve all your problem just implement the code that you share from github in your program. You can easily implement it as function.
import threading
import os
import Evtx.Evtx as evtx
import Evtx.Views as e_views
base_dir = "C:\\Users\\carlo.zanocco\\Desktop\\test_dir\\"
def convert(file_in, file_out):
tmp_list = list()
with evtx.Evtx(file_in) as log:
tmp_list.append(e_views.XML_HEADER)
tmp_list.append("<Events>")
for record in log.records():
try:
tmp_list.append(record.xml())
except Exception as e:
print(e)
tmp_list.append("</Events>")
with open(file_out, 'w') as final:
final.writelines(tmp_list)
#Search all files
def file_list(directory):
allFiles = list()
for entry in os.listdir(directory):
fullPath = os.path.join(directory, entry)
#if is directory search for more files
if os.path.isdir(fullPath):
allFiles = allFiles + file_list(fullPath)
else:
#check that the file have the right extension and append the command to execute later
if(entry.endswith(".evtx")):
threading.Thread(target=convert, args=(fullPath, fullPath.replace(".evtx", ".xml"))).start()
return allFiles
print("Searching and converting files")
file_list(base_dir)
If you want to show your files generate, just edit as above:
def convert(file_in, file_out):
tmp_list = list()
with evtx.Evtx(file_in) as log:
with open(file_out, 'a') as final:
final.write(e_views.XML_HEADER)
final.write("<Events>")
for record in log.records():
try:
final.write(record.xml())
except Exception as e:
print(e)
final.write("</Events>")
UPDATE
If you want to delete the '.evtx' files after the conversion you can simply add the following rows at the end of the convert function:
try:
os.remove(file_in)
except(Exception, ex):
raise ex
Here you just need to use try .. except because you run the thread only if the input value is a file.
If the file doesn't exist, this function throws an exception, so it's necessary to check os.path.isfile() first.

import os, sys
DIR = "D:/Test"
# ...or as a command line argument
DIR = sys.argv[1]
for f in os.listdir(DIR):
path = os.path.join(DIR, f)
name, ext = os.path.splitext(f)
if ext == ".txt":
new_path = os.path.join(DIR, f"{name}.xml")
os.rename(path, new_path)
Iterates over a directory, and changes all text files to XML.

Python Script Saving/Copying into specific directory

I'm attempting to write a script that will save a file in the given directory, but I'm getting a NotADirecotryError[WinError 267] whenever I run it. Any ideas or tips on what I may have done incorrectly?
import shutil
import os
src = 'C:\\Users\\SpecificUsername\\Pictures\\test.txt\'
dest = 'C:\\Users\\SpecificUsername\\Desktop'
files = os.listdir(src)
for file in files:
shutil.copy(file, dest)
for file in files:
if os.path.isfile(file):
shutil.copy(file,dest) ```

There are a couple of things going on here:
You can just use forward slashes in the paths.
Your src is the test.txt file, and not a directory, so you cannot iterate over it using os.listdir().
You can also merge the two loops together since they are looping over the same set of data.
shutil.copy() takes a file path as input, while what you are passing is a filename.
The following code should work and it also copies directories as is:
import shutil
import os
basepath = "C:/Users/SpecificUsername/"
src = "Pictures/"
dest = "Desktop/"
files = os.listdir(os.path.join(basepath, src))
for filename in files:
filepath = os.path.join(basepath, src, filename)
if (os.path.isfile(filepath)):
print("File: " + filename)
shutil.copy(filepath,dest)
else:
print("Dir: " + filename)
shutil.copytree(filepath, os.path.join(dest, filename))
Hope it helps!

Rename .pgsql to .sql files recursively with Python

I'm attempting to rename multiple files in a github repo directory on windows 10 pro
The file extensions are ".pgsql" (old) and ".sql" (rename to)
I'm using vscode (latest) and python 3.7 (latest)
I can do it, one folder at a time, but whenever I have tried any recursive directory code I've looked up on here I cant get it to work.
Currently working single directory only
#!/usr/bin/env python3
import os
import sys
folder = 'C:/Users/YOURPATHHERE'
for filename in os.listdir(folder):
infilename = os.path.join(folder,filename)
if not os.path.isfile(infilename): continue
oldbase = os.path.splitext(filename)
newname = infilename.replace('.pgsql', '.sql')
output = os.rename(infilename, newname)
I'd like to have it recursively start in a directory and change only the file extensions specified to .sql in all sub directories as well on windows, for example
folder = 'C:/Users/username/github/POSTGRESQL-QUERY/'

You can use os.walk(),
import os
folder = 'C:/Users/YOURPATHHERE'
for root, dirs, files in os.walk(folder):
for filename in files:
infilename = os.path.join(root,filename)
newname = infilename.replace('.pgsql', '.sql')
output = os.rename(infilename, newname)

Renaming all file extensions with Python

So I have a group of folders with many subfolders in them, and each of those have subfolders have multiple .pyc files and .pyc.py inside them. I have made a script on deleting all of a certain extension. Now I'm trying to rename the extensions of the .pyc.py with this code:
import os
extensions = ('.py',)
print 'Changing to root directory...'
os.chdir('../')
print 'Current Directory:'
print os.getcwd()
os.chdir('TTOLD')
print "Current Directory:"
print os.getcwd()
def rename(filepath):
print "Renaming '%s'..." % filepath
os.rename(filepath, extension)
for root, folders, files in os.walk('.'):
for filename in files:
filepath = os.path.join(root, filename)
extension = os.path.splitext(filename)[1]
if os.path.splitext(filename)[1] in extensions:
rename(filepath)
print 'Finished'
This code isn't finished, and I don't exactly know how to finish it. Could anyone help?