I have several files I want to open at the same time and pull data from and write them each to their own files. I'm currently doing something like this:
files = os.listdir(FilePath)
for file in files:
with open(os.path.join(FilePath, file), 'r') as LFILE:
LFILE.read()
Will I need to some how read all the files, put the file in a list and then have each thread read down the list and remove the file once its been read? Or is there a better way to open files and not read the same one more then once?
I don't have the reputation to ask for further clarification in a comment, but if you're not aggregating the data in such a way that info needs to be shared across files, I would just map the list of files to a pool, like so:
def analyze_file(filename: str):
with open(filename, 'r') as LFILE:
# analyze the file how you'd like
# store results in a string
with open(filename + 'analyzed.txt', 'w') as result_fh:
result_fh.write(results)
if __name__ == "__main__":
with multiprocessing.Pool(4) as p:
p.map(
analyze_file,
[os.path.join(FilePath, file) for file in os.listdir(FilePath)]
)
Related
I am processing files in a directory and after processing a file I want to save it using the original name but also add xx to the file name. My purpose is to identify which files have been processed.
Basic suggestions as to how to proceed are appreciated
If the only purpose is flag the file in order to know which files have been processed I would try another strategy (adding file metadata or something). But from your question, I infer the only thing you need is a rename of the file after being processed... You can use os.rename:
import os
filename = "example.txt"
flag_suffix = ".xx"
with open(filename, "wb+") as f:
# process file
...
os.rename(filename, f"{filename}{flag_suffix}")
My first post on StackOverflow, so please be nice. In other words, a super beginner to Python.
So I want to read multiple files from a folder, divide the text and save the output as a new file. I currently have figured out this part of the code, but it only works on one file at a time. I have tried googling but can't figure out a way to use this code on multiple text files in a folder and save it as "output" + a number, for each file in the folder. Is this something that's doable?
with open("file_path") as fReader:
corpus = fReader.read()
loc = corpus.find("\n\n")
print(corpus[:loc], file=open("output.txt","a"))
Possibly work with a list, like:
from pathlib import Path
source_dir = Path("./") # path to the directory
files = list(x for x in filePath.iterdir() if x.is_file())
for i in range(len(files)):
file = Path(files[i])
outfile = "output_" + str(i) + file.suffix
with open(file) as fReader, open(outfile, "w") as fOut:
corpus = fReader.read()
loc = corpus.find("\n\n")
fOut.write(corpus[:loc])
** sorry for multiple editting....
welcome to the site. Yes, what you are asking above is completely doable and you are on the right track. You will need to do a little research/practice with the os module which is highly useful when working with files. The two commands that you will want to research a bit are:
os.path.join()
os.listdir()
I would suggest you put two folders within your python file, one called data and the other called output to catch the results. Start and see if you can just make the code to list all the files in your data directory, and just keep building that loop. Something like this should list all the files:
# folder file lister/test writer
import os
source_folder_name = 'data' # the folder to be read that is in the SAME directory as this file
output_folder_name = 'output' # will be used later...
files = os.listdir(source_folder_name)
# get this working first
for f in files:
print(f)
# make output folder names and just write a 1-liner into each file...
for f in files:
output_filename = f.split('.')[0] # the part before the period
output_filename += '_output.csv'
output_path = os.path.join(output_folder_name, output_filename)
with open(output_path, 'w') as writer:
writer.write('some data')
I have multiple folders, in a common parent folder, say 'work'. Inside that, I have multiple sub-folders, named 'sub01', 'sub02', etc. All the folders have same files inside, for eg, mean.txt, sd.txt.
I have to add contents of all 'mean.txt' into a single file. I am stuck with, how to open subfolder one by one. Thanks.
getting all files as a list
g = open("new_file", "a+")
for files in list:
f = open(files, 'r')
g.write(f.read())
f.close()
g.close()
I am not getting how to get a list of all files in the subfolder, to make this work
************EDIT*********************
found a solution
os.walk() helped, but had a problem, it was random (it didn't iterate in alphabetical order)
had to use sort to make it in order
import os
p = r"/Users/xxxxx/desktop/bianca_test/" # main_folder
list1 = []
for root, dirs, files in os.walk(p):
if root[-12:] == 'native_space': #this was the sub_folder common in all parent folders
for file in files:
if file == "perfusion_calib_gm_mean.txt":
list1.append(os.path.join(root, file))
list1.sort() # os.walk() iterated folders randomly; this is to overcome that
f = open("gm_mean.txt", 'a+')
for item in list1:
g = open(item, 'r')
f.write(g.read())
print("writing", item)
g.close()
f.close()
Thanks to all who helped.
As i understand it you want to collate all 'mean.txt' files into one file. This should do the job but beware there is no ordering to which file goes where. Note also i'm using StringIO() to buffer all the data since strings are immutable in Python.
import os
from io import StringIO
def main():
buffer = StringIO()
for dirpath, dirnames, filenames in os.walk('.'):
if 'mean.txt' in filenames:
fp = os.path.join(dirpath, 'mean.txt')
with open(fp) as f:
buffer.write(f.read())
all_file_contents = buffer.getvalue()
print(all_file_contents)
if __name__ == '__main__':
main()
Here's a pseudocode to help you get started. Try to google, read and understand the solutions to get better as a programmer:
open mean_combined.txt to write mean.txt contents
open sd_combined.txt to write sd.txt contents
for every subdir inside my_dir:
for every file inside subdir:
if file.name is 'mean.txt':
content = read mean.txt
write content into mean_combined.txt
if file.name is 'sd.txt':
content = read sd.txt
write content into sd_combined.txt
close mean_combined.txt
close sd_combined.txt
You need to look up how to:
open a file to read its contents (hint: use open)
iterate files inside directory (hint: use pathlib)
write a string into a file (hint: read Input and Output)
use context managers for releasing resources (hint: read with statement)
In my system i have to list all the wav files from one folder and then i have to bind that files as one file. I have the code to bind two files but i have 8 or more sound file i have to bind those file. Anybody can you help me? Stackoverflow solution have merge only two audio file. I have to merge lot of audio file from the folder. I don't know how many audio file will come into that folder.
this code is bind two files:
print(glob.glob('upload/updated_audios/*.wav'))
file_data = glob.glob('upload/convertedAudio/*.wav')
outfile = "upload/output_files/output.wav"
data = []
for infile in file_data:
w = wave.open(infile, 'rb')
data.append([w.getparams(), w.readframes(w.getnframes())])
w.close()
output = wave.open(outfile, 'wb')
output.setparams(data[0][0])
output.writeframes(data[0][1])
output.writeframes(data[1][1])
output.close()
file_data = glob.glob('upload/convertedAudio/*.wav')
outfile = "upload/output_files/output.wav"
with wave.open(outfile, 'wb') as wav_out:
for wav_path in file_data:
with wave.open(wav_path, 'rb') as wav_in:
if not wav_out.getnframes():
wav_out.setparams(wav_in.getparams())
wav_out.writeframes(wav_in.readframes(wav_in.getnframes()))
this code is working but i can't merge orderly. this answer already in stackoverflow question. but that question is different so this answer is most suitable for this question.
note: order mean ascending and descending.
I need to make a script that executes a script one time in each folder of a directory.
Script in question:
f = open('OrderEXAMPLE.txt', 'r')
data = f.readlines()
mystr = ",".join([line.strip() for line in data])
with open('CSV.csv', 'w') as f2:
f2.write(mystr)
With this script, it changes a list of customer data into csv form.
Each order form has its own folder, so my intial thought was to put the same script into each folder. From there, write another script that executes each script simultaneously.
Folder structure is like so:
Order_forms
--Order_123
-----Order_form
--Order_124
-----Order_form
Amateur at python, so advice is needed and appreciated.
Just walk the directory structure with one script. This will write a separate CSV for each file with the name <original_filename>_CSV.csv. Without more clarity on the desired output nor knowing what the data looks like I can't help much more. You should be able to tweak this for whatever you need.
import os
parent_folder = 'Order_forms'
for root, dirs, files in os.walk(parent_folder):
for f in files:
with open(os.path.join(root, f), 'r') as f1:
data = f1.readlines()
mystr = ",".join([line.strip() for line in data])
with open(os.path.join(root, '{}_CSV.csv'.format(f)), 'w') as f2:
f2.write(mystr)