Develop Reference

How to make Python read multiple txt files

I need help with this Python Script I am a noob to Python, but I really need this to work
I have over 200.txt files all located in different folders and subs, and in each txt file I have 2 codes that need to be edited
This is what I have going now and this works great but I have to rename the script to match the file name
f1 = open('(1).txt', 'r')
f2 = open('(1A).txt', 'w')
for line in f1:
f2.write(line.replace('"aplid": 2147483648', '"aplid": -2147483648'))
f1.close()
f2.close()
This is my goal for the script to read anyfilename.txt in any folder
import glob
import os
for f in glob.glob("*.txt"):
f1 = open('f', 'r')
f2 = open('f', 'w')
for line in f1:
f2.write(line.replace('"aplid": 2147483648,', '"aplid": -2147483648,'))
f2.write(line.replace('"orlid": 2147483648,', '"orlid": -2147483648,'))
f1.close()
f2.close()
I made a batch script for this project and it works great, but it's super slow, it would be faster for me to edit the python script for each txt file lol..
I don't know if Python can read Folders and subfolders like batch dir /b /s /a-d
I'm sorry for bothering you'll, but I searched online and I can't find anything that helps with this and most of it I don't understand
I keep reading that using a path can help, but this script is going to be placed in several computers so using path is not best

You can use os.walk to traverse through a directory tree:
import os
for root, dirs, files in os.walk('.'):
for name in files:
if os.path.splitext(name)[1] != '.txt':
continue
nmin = os.path.join(root,name)
with open(nmin,"r") as fin:
data = fin.read()
data = data.replace('"aplid": 2147483648,', '"aplid": -2147483648,') \
.replace('"orlid": 2147483648,', '"orlid": -2147483648,')
with open(nmin,"w") as fout:
fout.write(data)

Edit multiple text files, and save as new files

My first post on StackOverflow, so please be nice. In other words, a super beginner to Python.
So I want to read multiple files from a folder, divide the text and save the output as a new file. I currently have figured out this part of the code, but it only works on one file at a time. I have tried googling but can't figure out a way to use this code on multiple text files in a folder and save it as "output" + a number, for each file in the folder. Is this something that's doable?
with open("file_path") as fReader:
corpus = fReader.read()
loc = corpus.find("\n\n")
print(corpus[:loc], file=open("output.txt","a"))

Possibly work with a list, like:
from pathlib import Path
source_dir = Path("./") # path to the directory
files = list(x for x in filePath.iterdir() if x.is_file())
for i in range(len(files)):
file = Path(files[i])
outfile = "output_" + str(i) + file.suffix
with open(file) as fReader, open(outfile, "w") as fOut:
corpus = fReader.read()
loc = corpus.find("\n\n")
fOut.write(corpus[:loc])
** sorry for multiple editting....

welcome to the site. Yes, what you are asking above is completely doable and you are on the right track. You will need to do a little research/practice with the os module which is highly useful when working with files. The two commands that you will want to research a bit are:
os.path.join()
os.listdir()
I would suggest you put two folders within your python file, one called data and the other called output to catch the results. Start and see if you can just make the code to list all the files in your data directory, and just keep building that loop. Something like this should list all the files:
# folder file lister/test writer
import os
source_folder_name = 'data' # the folder to be read that is in the SAME directory as this file
output_folder_name = 'output' # will be used later...
files = os.listdir(source_folder_name)
# get this working first
for f in files:
print(f)
# make output folder names and just write a 1-liner into each file...
for f in files:
output_filename = f.split('.')[0] # the part before the period
output_filename += '_output.csv'
output_path = os.path.join(output_folder_name, output_filename)
with open(output_path, 'w') as writer:
writer.write('some data')

How to open one folder at a time to acces files

I have multiple folders, in a common parent folder, say 'work'. Inside that, I have multiple sub-folders, named 'sub01', 'sub02', etc. All the folders have same files inside, for eg, mean.txt, sd.txt.
I have to add contents of all 'mean.txt' into a single file. I am stuck with, how to open subfolder one by one. Thanks.
getting all files as a list
g = open("new_file", "a+")
for files in list:
f = open(files, 'r')
g.write(f.read())
f.close()
g.close()
I am not getting how to get a list of all files in the subfolder, to make this work
************EDIT*********************
found a solution
os.walk() helped, but had a problem, it was random (it didn't iterate in alphabetical order)
had to use sort to make it in order
import os
p = r"/Users/xxxxx/desktop/bianca_test/" # main_folder
list1 = []
for root, dirs, files in os.walk(p):
if root[-12:] == 'native_space': #this was the sub_folder common in all parent folders
for file in files:
if file == "perfusion_calib_gm_mean.txt":
list1.append(os.path.join(root, file))
list1.sort() # os.walk() iterated folders randomly; this is to overcome that
f = open("gm_mean.txt", 'a+')
for item in list1:
g = open(item, 'r')
f.write(g.read())
print("writing", item)
g.close()
f.close()
Thanks to all who helped.

As i understand it you want to collate all 'mean.txt' files into one file. This should do the job but beware there is no ordering to which file goes where. Note also i'm using StringIO() to buffer all the data since strings are immutable in Python.
import os
from io import StringIO
def main():
buffer = StringIO()
for dirpath, dirnames, filenames in os.walk('.'):
if 'mean.txt' in filenames:
fp = os.path.join(dirpath, 'mean.txt')
with open(fp) as f:
buffer.write(f.read())
all_file_contents = buffer.getvalue()
print(all_file_contents)
if __name__ == '__main__':
main()

Here's a pseudocode to help you get started. Try to google, read and understand the solutions to get better as a programmer:
open mean_combined.txt to write mean.txt contents
open sd_combined.txt to write sd.txt contents
for every subdir inside my_dir:
for every file inside subdir:
if file.name is 'mean.txt':
content = read mean.txt
write content into mean_combined.txt
if file.name is 'sd.txt':
content = read sd.txt
write content into sd_combined.txt
close mean_combined.txt
close sd_combined.txt
You need to look up how to:
open a file to read its contents (hint: use open)
iterate files inside directory (hint: use pathlib)
write a string into a file (hint: read Input and Output)
use context managers for releasing resources (hint: read with statement)

Python script not combining csv files

I am trying to combine over 100,000 CSV files (all same formats) in a folder using below script. Each CSV file is on average 3-6KB of size. When I run this script, it only opens exact 47 .csv files and combines. When I re-run it only combines same .csv files, not all of them. I don't understand why it is doing that?
import os
import glob
os.chdir("D:\Users\Bop\csv")
want_header = True
out_filename = "combined.files.csv"
if os.path.exists(out_filename):
os.remove(out_filename)
read_files = glob.glob("*.csv")
with open(out_filename, "w") as outfile:
for filename in read_files:
with open(filename) as infile:
if want_header:
outfile.write('{},Filename\n'.format(next(infile).strip()))
want_header = False
else:
next(infile)
for line in infile:
outfile.write('{},{}\n'.format(line.strip(), filename))

Firstly check the length of read_files:
read_files = glob.glob("*.csv")
print(len(read_files))
Note that glob isn't necessarily recursive as described in this SO question.
Otherwise your code looks fine. You may want to consider using the CSV library but note that you need to adjust the field size limit with really large files.

Are you shure your all filenames ends with .csv? If all files in this directory contains what you need, then open all of them without filtering.
glob.glob('*')

Writing a user specified folder structure to csv in Python

I am trying to write a python script that will take in a user specified directory and then write the contents of that directory to a csv file. The challenge I am facing is to write the folder structure to the csv in a certain format. The format I need is shown below:
I've tried using os.walk(dir) to list the directories and files but I'm having trouble writing to the csv in the above format.
I've also found some code that creates a nested dictionary out of a given folder structure but I'm finding it very difficult to navigate through this nested structure and write the rows in the way that I need.
If anyone has an easier approach to accomplishing this task it would be much appreciated.

Ok, here is the code that does the job:
import os
import csv
data = []
for root, dirs, files in os.walk('.'):
for f in files:
l = root.split('/') + [f]
data.append(l)
with open('output.csv', 'wb') as f:
writer = csv.writer(f)
for d in data:
writer.writerow(d)
If you're on windows, replace split('/') with split('\\')