Develop Reference

How to find the last three files added in a directory

I'm using this code to find the last csv file added but I'm not able to find the last 3 files added. I can eliminate the last file and then find the max again but I think it'd be too long. Can you please help me find a solution? All I need is to find the last 3 csv files added in a directory.
import pandas as pd
import csv
import os
import zipfile
t=[]
j_csvs="path2"
#Find all csv files directories and collect them within t
d = os.path.join(j_csvs)
for root,dirs,files in os.walk(d):
for file in files:
if file.endswith(".csv"):
p=os.path.abspath(os.path.join(root, file))
t.append(p)
else: "DoNothing"
latest_f_j = max(t, key=os.path.getctime)
df=pd.read_csv(latest_f_j)
df

Use sorted with a callback function to infer the ordering relationship, some possibilities:
with os.path.getctime for system’s ctime (it is system dependent, see doc)
with os.path.getmtime for the time of last modification
with os.path.getatime for the time of last access.
Pass the reverse=True parameter for a result in descending order and then slice.
import os.path
def last_newest_files(path, ref_ext='csv', amount=3):
# return files ordered by newest to oldest
def f_conditions(path):
# check by file and extension
_, ext = os.path.splitext(path) # ext start with ".", ie ".csv"
return os.path.isfile(path) and ext.lstrip('.') == ref_ext
# apply conditions
filtered_files = filter(f_conditions, (os.path.join(path, basename) for basename in os.listdir(path)))
# get the newest
return sorted(filtered_files, key=os.path.getctime, reverse=True)[:amount]
path_dir = '.'
ext = 'csv'
last_n_files = 3
print(*last_newest_files(path_dir, ext, last_n_files), sep='\n')

You cannot determine what the last 3 files added are with any degree of certainty.
At the upper level, a system may put those file in order of date, file type, size, name - both case sensitive and without.
With date order, you have no way of knowing since date stamps can be manipulated, as can pre-dated files moved into a directory and thus preserving its original date and time details.
If you are looking at files at a lower level, as that seen by the file system, then they are generally unordered. The o/s on its own whim, will store the details as it sees fit.
You have no way whatsoever in determining which of 3 files were the last to added. Well, you have one way, run a watch on the directory which will fire when a file is added and keep a circular list of 3 replacing the current before moving onto the next and then waiting for the next trigger to fire.

Readstream path

I have say a/b/c/date/Xyz.txt how can I pick up the path dynamically, date is changing randomly. Other directory fields remains the same like a/b/c/dateIsChanging/Xyz.txt
Someone is adding file random date directory, how can I get the path, each time when it is added
a/b/c/dateIsChanging/Xyz.txt here in directory under random chosen by the user, I want to pick it up whatever date comes in
Date in YYYYMMDD formate

This is an example of what one would do:
import os
import datetime as dt
# create a date for folder name
date_part = dt.datetime.today().strftime('%Y-%m-%d')
# generate the folder path
the_path = os.path.join('C:\\','test',str(date_part),'file_name.txt')
print(the_path)
the output:
C:\test\2022-03-24\file_name.txt
Obviously, would need to adjust for your particular operating system and date format.

Automate appending yesterday's date to end of file names using Python?

I'm trying to solve a problem at work, I am not a developer but work in general IT operations and am trying to learn a bit here and there, so I may be way off right now with what I'm trying to do here. I've just been utilizing online resources, here and a little bit from the book Automate the Boring Stuff with Python. Here is my objective:
I have two files that are automatically placed in a folder on my computer every morning at the same time using a post processor, and I need to add yesterday's date to the end of the file names before I upload them to an FTP server which I have do each morning around the same time. I am trying to write a Python script that I can somehow schedule to run each morning right after the files are placed in the folder, which will append yesterday's date in MMDDYYYY format. For example, if the files are called "holdings.CSV" and "transactions.CSV" when they are placed in the folder, I need to rename them to "holdings01112022.CSV" and "transactions01112022.CSV". I only want to rename the new files in the folder, the files from previous days with the dates already appended will remain in the folder. Again, I'm a total beginner, so my code may not make sense and there may be superfluous or redundant lines, I'd love corrections... Am I going down the right path here, am I off altogether? Any suggestions?
import os, re
from datetime import date
from datetime import timedelta
directory = 'C:\Users\me\main folder\subfolder'
filePattern = re.compile('%s.CSV', re.VERBOSE)
for originalName in os.listdir('.'):
mo = filePattern.search(originalName)
if mo == None:
continue
today = date.today()
yesterday = today - timedelta(days = 1), '%M%D%Y'
for originalName in directory:
newName = originalName + yesterday
os.rename(os.path.join(directory, originalName), os.path.join(directory, newName))
Any help is appreciated. Thanks.

Here's a short example on how to code your algorithm.
import pathlib
from datetime import date, timedelta
if __name__ == '__main__':
directory = pathlib.Path('/Users/cesarv/Downloads/tmp')
yesterday = date.today() - timedelta(days=1)
for file in directory.glob('*[!0123456789].csv'):
new_path = file.with_stem(file.stem + yesterday.strftime('%m-%d-%Y'))
if not new_path.exists():
print(f'Renaming {file.name} to {new_path.name}')
file.rename(new_path)
else:
print(f'File {new_path.name} already exists.')
By using pathlib, you'll simplify the handling of filenames and paths and, if you run this program on Linux or macOS, it will work just the same.
Note that:
We're limiting the list of files to process with glob(), where the pattern *[!0123456789].csv means "all filenames that do not end with a digit before the suffix (the [!0123456789] represents one character, before the suffix, that should not - ! - equal any of the characters in the brackets)." This allows you to process only files that do not contain a date in their names.
The elements given by the for cycle, referenced with the file variable, are objects of class Path, which give us methods and properties we can work with, such as with_stem() and stem.
We create a new Path object, new_path, which will have its stem (the file's name part without the suffix .csv) renamed to the original file's name (file.name) plus yesterday in the format you require by using the method with_stem().
Since this new Path object contains the same path of the original file, we can use it to rename the original file.
As you can see, we can also check that a file with the new name does not exist before any renaming by using the exists() method.
You can also review the documentation on pathlib.
If you have any doubts, ask away!

Are you running to any issue? If all you want is to get the csv file you can simplify your code by using one if statement instead of using regex. You also need to convert your date to string before adding it to your file name. (i.e: newName = originalName[:-3] +str(yesterday)).
If this code didnt solve your problem please mention the error that you are receiving to make it easier for others to help you.
Here is my suggestion to eliminate possible errors without knowing what type of error you are getting.
import os, re
from datetime import date
from datetime import timedelta
directory = 'C:/Users/me/main folder/subfolder'
for originalName in os.listdir(directory):
print(f'original name is = {originalName}')
if originalName[-3:].lower() =='csv':
today = date.today()
yesterday = today - timedelta(days = 1)
yesterday_file_name = originalName[:-4]+yesterday.strftime("%m%d%Y")+'.csv'
os.rename(os.path.join(directory, originalName), os.path.join(directory, yesterday_file_name))
#print('formatted file names are')
#print(yesterday_file_name)
#here is the sample output
#original name is = holdings.csv
#formatted file names are
#holdings01112022.csv
#original name is = transactions.CSV
#formatted file names are
#transactions01112022.csv

Query Related to Python - Folders Read

I want to read folders in python and probably make a list of it. Now my main concern is that most recent folder should be at location that is known to me. It can be the first element or last element of list. I am attaching image suggesting folders name. I want folder with name 20181005 either first in the list or last in the list.
I have tried this task and used os.listdir, but I am not very much confident on the way this function reads folders and store in list form. Would it store first folder as element or will it use creation date or modification date. If I could sort on the basis of name (20181005 etc), it would be really good.
Kindly suggest suitable method for the same.
Regards

os.listdir returns directory contents in arbitrary order, but you can sort that yourself:
l = sorted(listdir())
Since it seems that your folder names are ISO dates, they should sort correctly and the most recent one should be the last element after sorting.

If you need to access creation & modification times you can do that with os.path functions. If you want to sort by that, I would probably choose to put it in something like a pandas DataFrame to make it easier to manipulate.
import os
from datetime import datetime
import pandas as pd
path = "."
objects = os.listdir(path)
dirs = list()
for o in objects:
opath = os.path.join(path, o)
if os.path.isdir(opath):
dirs.append(dict(path=opath,
mtime=datetime.fromtimestamp(os.path.getmtime(opath)),
ctime=datetime.fromtimestamp(os.path.getctime(opath))))
data = pd.DataFrame(dirs)
data.sort_values(by='mtime')

Assumed, your directories has YYYYMMDD format naming. Then you can use listdir and sort to get the latest directory in last index.
import os
from os import listdir
mypath = 'D:\\anil'
list_dirs = []
for f in listdir(mypath):
if os.path.isdir(os.path.join(mypath, f)):
list_dirs.append(f)
list_dirs.sort()
for current_dir in list_dirs:
print(current_dir)

python create directory structure based on the date

I used the following function to created dirctory based on today date ,
#!/usr/bin/python
import time, datetime, os
today = datetime.date.today()
todaystr = today.isoformat()
os.mkdir(todaystr)
so the out put will be
/2015-12-22/
what i'm looking to is adjust the structure which is create dirctories structure based on day date as following
/2015/12/22
/2015/12/23
etc
when ever i run the function it will check the date and make sure the folder is exist other wise will create it ..
any tips to follow here ?

Consider using strftime instead. Which you can use to defined a format to your liking. You will also need to use os.makedirs as described by #Valijon below.
os.makedirs(time.strftime("/%Y/%m/%d"), exist_ok=True)
You can also append a given time to create a time-stamp in the past or in the future.
time.strftime("/%Y/%m/%d", time.gmtime(time.time()-3600)) # -1 hour
Also note that your path is a bit dangerous, unless you want to create folders directly under the root partition.
Note that makedirs will raise an exception by default if the directory already exists, you can specify exist_ok=True to avoid this, read more about it in the docs for os.makedirs.
Since Python 3.4, the module pathlib was Introduced which offers some directory and file creation features.
import time
import pathlib
pathlib.Path(time.strftime("/%Y/%m/%d")).mkdir(parents=True, exist_ok=True)

Just change os.mkdir to os.makedirs
os.makedirs(today.strftime("%Y/%m/%d"))