Develop Reference

Python: Finding files in directory but ignoring folders and their contents

So my program search_file.py is trying to look for .log files in the directory it is currently placed in. I used the following code to do so:
import os
# This is to get the directory that the program is currently running in
dir_path = os.path.dirname(os.path.realpath(__file__))
# for loop is meant to scan through the current directory the program is in
for root, dirs, files in os.walk(dir_path):
for file in files:
# Check if file ends with .log, if so print file name
if file.endswith('.log')
print(file)
My current directory is as follows:
search_file.py
sample_1.log
sample_2.log
extra_file (this is a folder)
And within the extra_file folder we have:
extra_sample_1.log
extra_sample_2.log
Now, when the program runs and prints the files out it also takes into account the .log files in the extra_file folder. But I do not want this. I only want it to print out sample_1.log and sample_2.log. How would I approach this?

Try this:
import os
files = os.listdir()
for file in files:
if file.endswith('.log'):
print(file)
The problem in your code is os.walk traverses the whole directory tree and not just your current directory. os.listdir returns a list of all filenames in a directory with the default being your current directory which is what you are looking for.
os.walk documentation
os.listdir documentation

By default, os.walk does a root-first traversal of the tree, so you know the first emitted data is the good stuff. So, just ask for the first one. And since you don't really care about root or dirs, use _ as the "don't care" variable name
# get root files list.
_, _, files = next(os.walk(dir_path))
for file in files:
# Check if file ends with .log, if so print file name
if file.endswith('.log')
print(file)
Its also common to use glob:
from glob import glob
dir_path = os.path.dirname(os.path.realpath(__file__))
for file in glob(os.path.join(dir_path, "*.log")):
print(file)
This runs the risk that there is a directory that ends in ".log", so you could also add a testing using os.path.isfile(file).

Recursively find and copy files from many folders

I have some files in an array that I want to recursively search from many folders
An example of the filename array is ['A_010720_X.txt','B_120720_Y.txt']
Example of folder structure is as below which I can also provide as an array e.g ['A','B'] and ['2020-07-01','2020-07-12']. The "DL" remains the same for all.
C:\A\2020-07-01\DL
C:\B\2020-07-12\DL
etc
I have tried to use shutil but it doesn't seem to work effectively for my requirement as I can only pass in a full file name and not a wildcard. The code I have used with shutil which works but without wildcards and with absolute full file name and path e.g the code below will only give me A_010720_X.txt
I believe the way to go would be using glob or pathlib which i have not used before or cannot find some good examples similar to my use case
import shutil
filenames_i_want = ['A_010720_X.txt','B_120720_Y.txt']
RootDir1 = r'C:\A\2020-07-01\DL'
TargetFolder = r'C:\ELK\LOGS\ATH\DEST'
for root, dirs, files in os.walk((os.path.normpath(RootDir1)), topdown=False):
for name in files:
if name in filenames_i_want:
print ("Found")
SourceFolder = os.path.join(root,name)
shutil.copy2(SourceFolder, TargetFolder)

I think this should do what you need assuming they are all .txt files.
import glob
import shutil
filenames_i_want = ['A_010720_X.txt','B_120720_Y.txt']
TargetFolder = r'C:\ELK\LOGS\ATH\DEST'
all_files = []
for directory in ['A', 'B']:
files = glob.glob('C:\{}\*\DL\*.txt'.format(directory))
all_files.append(files)
for file in all_files:
if file in filenames_i_want:
shutil.copy2(file, TargetFolder)

I want to add all the names of the files in a specific folder to a list

I want to add the names of all the files in a specific folder to a list how can i do that? the pathway is from dropbox -> a folder called 'UMM' -> a folder called '2018' could someone help me with the code on this. I have tried using os.walk() but it doesn't seem to work

You can use os.walk and append only names which are in files.
from os import walk
file_names = list()
path = 'path/of/folder'
for root, dirc, files in walk(path):
for FileName in files:
file_names.append(FileName)
print(file_names)
This will append all the files name from all the directories and sub-directories of the specified path.

this will create a list of the files in a folder
from os import listdir
# the path
path = ''
fileList = listdir(path)

Iterate over files located in different folders

I’d like to write a function to iterate over excel files that are in different folders. Parts of the path of each file are the same, for instance:
C:\Main\Division\Reports\Year\Data.xls
The only part of each path that changes is ‘Year’. The files all have the same name.
Is there a way to do this with a placeholder for Year? If not, what approach should I take?

You can use os.listdir function
directory = "C:\Main\Division\Reports"
root_dir = os.path.dirname(directory)
for data in os.listdir(directory):
file_name = os.path.join(root_dir, data, 'Data.xls')
# do something

You could try os.walk
import os
parent = "C:\Main\Division\Reports"
for root, directory, files in os.walk(parent):
print root
print directory
print files

Python: search multiple directories and grab newest file, deleting others

New to python and would appreciate a little help.
I would like to go through 10 directories and copy the newest file from each directory back into a single folder. There may be multiple files in each directory.
I can pull a complete listing from each directory, not sure how to narrow this down. Any direction would be appreciated.
inside the STATES directory will be directories for each state (i.e. CA, NY, FL, MI, GA)
**Edited if it is helpful, the directory structure looks like this:
'/dat/users/states/CA/'
'/dat/users/states/NY/'
'/dat/users/states/MI/'
import glob
import os
data_dir = '/dat/users/states/*/'
file_dir_extension = os.path.join(data_dir, '*.csv')
for file_name in glob.glob(file_dir_extension):
if file_name.endswith('.csv'):
print (file_name)

You can use os.walk() instead of glob.glob() to traverse all of your folders. For each folder you get a list of the filename in it. This can be sorted by date using os.path.getmtime(). This will result in the newest file being at the start of the list.
Pop the first element off the list and copy this to your target folder. The remaining elements in the list could then be deleted using os.remove() as follows:
import os
import shutil
root = r'/src/folder/'
copy_to = r'/copy to/folder'
for dirpath, dirnames, filenames in os.walk(root):
# Filter only csv files
files = [file for file in filenames if os.path.splitext(file)[1].lower() == '.csv']
# Sort list by file date
files = sorted(files, key=lambda x: os.path.getmtime(os.path.join(dirpath, x)), reverse=True)
if files:
# Copy the newest file
copy_me = files.pop(0)
print("Copying '{}'".format(copy_me))
shutil.copyfile(os.path.join(dirpath, copy_me), os.path.join(copy_to, copy_me))
# Remove the remaining files
for file in files:
src = os.path.join(dirpath, file)
print("Removing '{}'".format(src))
#os.remove(src)
os.path.join() is used to safely join a path and filename together.
Note: If it is supported on your system, you might need to use something like:
os.stat(os.path.join(dirpath, x)).st_birthtime
to sort based on the creation date/time.