Develop Reference

Apply python code to all csv files in a folder

I have 3000 csv files for machine learning and I need to treat each of these files separately, but the code I will apply is the same. File sizes and number of lines vary between 16 kb and 25 mb, and 60 lines and 330 thousand lines, respectively. Also, each csv file has 77 columns. I wrote only code inside the loop with the help of the previous post, but after applying the code, I cannot update within the same files. I just applied the code from previous post
and get error "No such file or directory: '101510EF'" (101510EF is my first csv file on my folder)
Loooking forward to your help. Thank you!

You don't need the line:
file_name=os.path.splitext(...)
Just this:
path = "absolute/path/to/your/folder"
os.chdir(path)
all_files = glob.glob('*.csv')
for file in all_files:
df = pd.read_csv(file)
df["new_column"] = df["seq"] + df["log_id"]
df.to_csv(file)

You need to provide absolute path WITH extension to pd.read_csv and df.to_csv methods.
e.g. c:/Users/kaanarik/Desktop/tez_deneme/ornel/101510EF.csv

operations on data of multiple txt. files in a folder

For some research I am analysing an area for precipitation and runoff. Everything works fine when I analyse one area (pd.read_csv), but I would like to analyse multiple (>10) areas at the same time without manually changing the file every time.
My goals in steps:
Operations on all txt files in folder A.
Operations on all txt files in folder B.
Operations on combination of matching files from folder A and B. For example, put the precipitation of area 5 and runoff of area 5 in a graph together.
What I tried
I figured that this can be done by using for files in os.scandir(foldername) and then start operations on all files in that folder, but my method is definitely wrong.
The first thing I like to do is to add a index in all of my files. I changed my path to "folderA" for this question. I tried it this way:
for files in os.scandir(r"C:folderA"):
with open (file) in files:
file.loc[:,'dt'] = file.to_datetime(file[['year', 'month', 'day']])
file.index = file['dt']
file.columns = ['year', 'month', 'day','Rain','dt']
file.head()
This results in the error: FileNotFoundError
Traceback (most recent call last)
<ipython-input-44-1641f38697a9> in <module>
1 for files in os.scandir(r"C:\FolderA"):
----> 2 with open (file) in files:
3 file.loc[:,'dt'] = file.to_datetime(file[['year', 'month', 'day']])
4 file.index = file['dt']
5 file.columns = ['year', 'month', 'day','Rain','dt']
FileNotFoundError: [Errno 2] No such file or directory: 'C'
As you can see I can really use some help to get me started. I think that when I understand the process better, that I will be able to do the operations myself. In conclusion, my questions is how can I do operations automatically on the data of all txt. files in a folder?

To loop over all files in a directory, do this:
for root, dirs, files in os.walk(path):
for filename in files:
file_path = os.path.join(root, filename)
with open(file_path) as file:
# logic
make sure the path you give is correct

Amending date in a file path using python

How do I make the filepath dynamic. I have to pick info from 7 spreadsheets and I have to just daily change the dates in these file paths. How can i declare the date and call the date in a string.
date = 20200607
filepath = HKTR Report\06. June**20200605**\Summary006T-CTRD2603-FX-20200605.csv"
The two dates in the file path have to be replaced with date declared making the filepath dynamic.
Kindly assist.

Welcome to Stack Overflow!
The path you have listed is peculiar in that it contains star characters which are not usually used in pathnames.
That said, try using this code:
from datetime import date
d = date.today()
date_str = d.strftime('%Y%m%d')
print(date_str)
filepath = f'HKTR Report\\06. June**{date_str}**\\Summary006T-CTRD2603-FX-{date_str}.csv'
print(filepath)
which generates this output:
20200608
HKTR Report\06. June**20200608**\Summary006T-CTRD2603-FX-20200608.csv

merging multiple CSV files in one with same header but different csv files name with python

I'm new in python ...I have tried to apply this code to merge multiple csv files but it doesn't work..basically, I have a files which contains stock prices with header: date,open,High,low,Close,Adj Close Volume... . but each csv file has a different name: Apl.csv,VIX.csv,FCHI.csv etc..
I would like to merge all these csv files in One.. but I would like to add a new columns which will disclose the name of the csv files example:
stock_id,date,open,High,low,Close,Adj Close Volume with stock_id = apl,Vix etc..
I used this code but I got stuck in line 4
here is the code:
files = os.listdir()
file_list = list()
for file in os.listdir():
if file.endswith(".csv")
df=pd.read_csv(file,sep=";")
df['filename'] = file
file_list.append(df)
all_days = pd.concat(file_list, axis=0, ignore_index=True)
all_days.to_csv("all.csv")
Someone could help me to sort out this ?

In Python, the indentation level matters, and you need a colon at the end of an if statement. I can't speak to the method you're trying, but you can clean up the synax with this:
files = os.listdir()
file_list = list()
for file in os.listdir():
if file.endswith(".csv"):
df=pd.read_csv(file,sep=";")
df['filename'] = file
file_list.append(df)
all_days = pd.concat(file_list, axis=0, ignore_index=True)
all_days.to_csv("all.csv")

I'm relatively new in python ..here is what I'd like to do..I got a folder with multiples csv files ( 2018.csv,2017.csv,2016.csv etc..)500 csv files to be precise.. each csv contains header "date","Code","Cur",Price etc..I'd like to concatenate all 500 csv files in one datafame...here is my code for one csv files but it's very slow , I want to do it for all 500 files and concantanate in one dataframe :
DB_2017 = pd.read_csv("C:/folder/2018.dat",sep=",", header =None).iloc[: 0,4,5,6]
DB_2017.columns =["date","Code","Cur",Price]
DB_2017['Code'] =DB_2017['Code'].map(lambdax:x.lstrip('#').rstrip('#'))
DB_2017['Cur'] =DB_2017['Cur'].map(lambdax:x.lstrip('#').rstrip('#'))
DB_2017['date'] =DB_2017['date'].apply(lambdax:pd.timestamp(str(x)[:10)
DB_2017['Price'] =pd.to_numeric(DB_2017.Price.replace(',',';')

Combining files based on a date range

I'm very new to scripting and as a result am not sure how best to merge a series of files. I'm attempting to create a Quality Control script that makes sure a nightly load was properly uploaded to the DB (we've noticed that if there's a lag for some reason, the sync will exclude any donations that came in during said lag).
I have a directory of daily synced files labeled as such:
20161031_donations.txt
20161030_donations.txt
20161029_donations.txt
20161028_donations.txt
etc etc
Every file has the same header.
I'd like to merge the last 7 days of files into one file with just 1 header row. I'm mostly struggling with understanding how to wildcard a date range. I've only ever done:
for i in a.txt b.txt c.txt d.txt
do this
done
which is fine for a static merge but not dynamic to integrate into a proper QC script.
I have a unix background but would like to do this in python. I'm new to python so please be explanatory in any suggestions.

Expanding on Alex Hall's answer, you can grab the header from one file and skip it for the remaining files to do the merge
from glob import glob
from shutil import copyfileobj
files = sorted(glob('*_donations.txt'))[-7:]
# if you want most recent file first do
# files.reverse()
with open("merged_file.txt", "w") as outfp:
for i, filename in enumerate(files):
with open(filename) as infile:
if i:
next(infile) # discard header
copyfileobj(infile, outfile) # write remaining

The advantage of your date format (assuming it has zero padding, e.g. 20160203 for 3rd Feb) is that it can be sorted alphabetically! So you can just do this:
from glob import glob
for path in sorted(glob('*_donations.txt'))[-7:]:
with open(path) as f:
# get the content for merging
This will get the 7 most recent files, starting with the oldest. This is why ISO 8601 is the best date format.