I am generating multiple .csv files using a script which is plugged in to my Flask app. Below is my route file section where required inputs are taken and passing in to the script.
#app.route('/summary_report', methods= ['GET', 'POST'])
def summary_report():
"""
Showing page for generating daily report
:return:
"""
form = frm.SummaryReportForm()
if form.validate_on_submit():
from_date = form.from_date.data
is_active = form.is_active.data
reports = current_summary_report.fetch_report(from_date=from_date, status=is_active) # Go the script
return send_file(reports, as_attachment=True, attachment_filename="reports.zip") # Download attachment
return render_template('pages/reports/current/summary.html', form=form)
And my script file is running a loop and creating multiple csv files.
def fetch_report(from_date=from_date, status=is_active):
for record in records:
....
f = open(file_name + '-' + str(time) + '.csv','w+')
f.write(csv)
f.close()
What changes I should do here to make all csv files in to a zip file and making it downloadable.
Python allows you to quickly create zip/tar archives.
Following command will zip entire directory
shutil.make_archive(output_filename, 'zip', dir_name)
Following command gives you control on the files you want to archive
ZipFile.write(filename)
Code Explanation
Import make_archive class from module shutil.
Use the split function to split out the directory and the file name from the path to the location of the text file.
Then we call the module "shutil.make_archive("guru99 archive, "zip", root_dir)" to create archive file, which will be in zip format
Related
I have been having some issues with my aws lambda that unzips a file inside of our s3 bucket. I created a script that would activate from a json payload that gets those passed through to it. The problem is it seems to be loosing the parent folder of the zip file and uploading the child folders underneath it. This is an issue for me as we also have another script I made to parse a log4j file inside of a folder to review for errors. That script is having problems because of the name lost that defines the farm the folder comes from.
To give an example of the issue ---
There's an s3 bucket on us-east, and inside that bucket is a key for "OriginalFolder.zip". When this lambda is activated it unzips and places the child file into the exact same bucket and place where the original zip file is but names it "Log.folder". I want it to keep the original name of the zip file so that when multiple farms are activating this lambda it doesn't overwrite that folder that's created or get confused on which one to read from with the second lambda.
I tried to append something at the end of the created file name to allow for params to be passed through for each farm that runs it but can't seem to make it work. I also contemplated having a separate action called in the script to copy and rename it using boto3 but I would rather not use that as my first choice. I feel there has to be an easier method but might be overlooking it.
Any thoughts would be helpful.
Edit: Here's a picture of the example. The green arrow is what I want it to stay named as. The red arrow is what the file is becoming named inside of our s3 environment. "on1" is the next folder inside "update-dc-logs-test".
import os
import tempfile
import zipfile
from concurrent import futures
from io import BytesIO
import boto3
s3 = boto3.client('s3')
def handler(event, context):
# Parse and prepare required items from event
global bucket, path, zipdata, rn_file
action = event.get("action", None)
if action == "create" or action == "update":
bucket = event['payload']['BucketName']
key = event['payload']['Key']
#rn_file = event['payload']['RenameFile']
path = os.path.dirname(key)
# Create temporary file
temp_file = tempfile.mktemp()
# Fetch and load target file
s3.download_file(bucket, key, temp_file)
zipdata = zipfile.ZipFile(temp_file)
# Call action method with using ThreadPool
with futures.ThreadPoolExecutor(max_workers=4) as executor:
future_list = [
executor.submit(extract, filename)
for filename in zipdata.namelist()
]
result = {'success': [], 'fail': []}
for future in future_list:
filename, status = future.result()
result[status].append(filename)
return result
def extract(filename):
# Extract zip and place it back in bucket
upload_status = 'success'
try:
s3.upload_fileobj(
BytesIO(zipdata.read(filename)),
bucket,
os.path.join(path, filename)
)
except Exception:
upload_status = 'fail'
finally:
return filename, upload_status
You are prefixing all uploaded files with path which is the path at which the ZIP file is found. If you want the uploaded files to be stored below a prefix which is the path and name of the ZIP file (minus the .zip extension), then change the value of path to this:
path = os.path.splitext(key)[0]
Now, instead of path holding the ZIP file's folder prefix it will contain the folder prefix plus the first part of the ZIP filename. For example, if an object is uploaded to folder1/myarchive.zip then path would previously contain folder1, but with this change it will now contain folder1/myarchive.
When that new path is combined in the extract function via os.path.join(path, filename), the object will now be uploaded to folder1/myarchive/on1/file.txt.
Hi is it possible for a user to upload a file using flask;
user would select it from there computer, select submit, which would be downloaded to a ZIP file folder on webserver(local host) and unzip that file, and search for a certain file within that unzip file directory
I have the functionality of the form down to upload it can’t figuire out how to unzip the file and save its content in a folder
This may work for your problem:
You'd used this first then,
response = make_response(log_file.text)
this for the second
handle.write(response.content)
.content is "Content of the response, in bytes." .text is "Content of the response, in unicode."
If you want a byte stream, use .content.
for further comprehension go to:
Can't unzip file retrieved with Requests in Flask app
or to:
Unzipping files in Python
One of them should work
its pure python unzip file
import zipfile
with zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:
zip_ref.extractall(directory_to_extract_to)
then you can use it in your flask app
import zipfile
#app.route("/",methods=["POST"])
def page_name_post():
file = request.files['data_zip_file']
file_like_object = file.stream._file
zipfile_ob = zipfile.ZipFile(file_like_object)
file_names = zipfile_ob.namelist()
# Filter names to only include the filetype that you want:
file_names = [file_name for file_name in file_names if file_name.endswith(".txt")]
files = [(zipfile_ob.open(name).read(),name) for name in file_names]
return str(files)
I'm building a web app on Django and I've implemented two functions, one to save csv files in a file directory on my PC, and the other is to read these files to view them in the website.
My problem is that I want to read csv files from the directory and display them but without the csv extension, I want their names to be the only thing visible, but I keep getting this error FileNotFoundError.
this is the function that saves the files to a directory
def openDataset(request):
if request.method == "GET":
return render(request, 'blog/upload_csv_ag.html')
if request.FILES.get("file2") is not None:
csv_file = request.FILES['file2']
if not csv_file.name.endswith('.csv'):
message='The uploaded file has to be CSV. Please try again.'
return render(request, 'blog/upload_csv_ag.html',{'message':message})
else:
save_path = 'C:/Users/user/Desktop/Fault Detection App/Uploaded_Datasets/'
file_name = csv_file.name
fs = FileSystemStorage(location=save_path)
file = fs.save(file_name, csv_file)
else:
message='no file is uploaded'
return render(request, 'blog/upload_csv_ag.html',{'message':message})
return render(request,'blog/upload_csv_ag.html',{'message':'Dataset Uploaded'})
and the function that reads the files from the directory
def read_datasets(request):
path = r"C:/Users/user/Desktop/Fault Detection App/Uploaded_Datasets/"
test = os.listdir(path)
path1, dirs, files = next(os.walk(path))
file_count = len(files)
print(file_count)
dataframes_list_html = []
file_names = []
index = []
for i in range(file_count):
temp_df = pd.read_csv(path+files[i])
print(files[i])
dataframes_list_html.append(temp_df.to_html(index=False))
index.append(i)
for item in test:
if item.endswith(".csv"):
os.remove(os.path.join(path, item))
file_names.append(files[i])
return render(request,'blog/view_datasets.html',{'files': file_names})
Extracting the names of the files
Step 1 : iterate through the directory
A simpler way would be to just do
for file in os.listdir(base_path)
Step 2 - remove the extension
You can use the method that evergreen suggested
Step 3 - store the processed string
Just append to your file_names list like you're doing and return the list in the response context
Reading and displaying the CSVs content
Actually reading and returning the content of the CSVs is slightly more involved, but your current approach by using pandas to read the files, and converting the dataframes to html tables should work just fine. Just remember to return the dataframes_list_html list in the context as well so that you can access it in the template
In the below code I am trying to zip a list list of files, I am trying to rename the files before zipping it. So the file name will be in a more readable format for the user.
It works for the first time, but when I do it again It fails with the error the file name already exists
Returning the response via Django Rest Framework via FileResponse.
Is there any more simplistic way to achieve this?
filenames_list=['10_TEST_Comments_12/03/2021','10_TEST_Posts_04/10/2020','10_TEST_Likes_04/09/2020']
with zipfile.ZipFile(fr"reports/downloads/reports.zip", 'w') as zipF:
for file in filenames_list:
friendly_name = get_friendly_name(file)
if friendly_name is not None:
os.rename(file,fr"/reports/downloads/{friendly_name}")
file = friendly_name
zipF.write(fr"reports/downloads/{file}", file, compress_type=zipfile.ZIP_DEFLATED)
zip_file = open(fr"reports/downloads/reports.zip", 'rb')
response = FileResponse(zip_file)
return response
ZipFile.write has a second parameter, arcname, which allows you to rename files without any copying. You don't need to move file to a separate folder, or actually rename it.
from os.path import basename
for file in filenames_list:
if (name := get_friendly_name(file)) is None:
name = basename(file)
zipF.write(file, name, compress_type=zipfile.ZIP_DEFLATED)
By stripping off the basename, you avoid the need to move to a common folder at all.
I currently have a program that collects data from .txt files in a folder and then saves that data to a csv file. Due to how I am planning on distributing this program, I need the Python file to live in the folder where these .txt files are located. However, I need the .csv files to be thrown to an absolute file path rather than being created in the same folder as the Python script and .txt documents. Here is what I have currently coded,
def write_to_csv(journal_list):
#writes a list of journal dictionaries to a csv file.
import csv
username = "Christian"
csv_name = username + ".csv"
myFile = open(csv_name, 'w')
with myFile:
myFields = ["filename", "username", "project_name", "file_path",
"date", "start_time", "end_time", "length_of_revit_session",
"os_version", "os_build", "revit_build", "revit_branch",
"cpu_name", "cpu_clockspeed", "gpu_name", "ram_max", "ram_avg", "ram_peak",
"sync_count", "sync_time_total", "sync_time_peak", "sync_time_avg",
"commands_total", "commands_hotkey_percentage", "commands_unique",
"commands_dynamo", "commands_escape_key", "commands_most_used"]
writer = csv.DictWriter(myFile, fieldnames=myFields)
writer.writeheader()
for item in journal_list:
try:
writer.writerow(item)
except:
print("error writing data to:", item)
I appreciate the help.
USing os.path.join() you can select your desire path for your file to be written. Here is an example:
import os
desier_path = '/home/foo/'
file_path = os.path.join(dest_dir, csv_name)
with open(file_path, 'w'):
...
Consider asking for the path from the script, and setting a default if not passed in. This would make your script a lot more flexible than having the path coded in it.
You can use the click package which simplifies this a bit.
import os
import click
def write_to_csv(path, journal_list):
# .. your normal code
file_path = os.path.join(path, '{}.csv'.format(username))
# .. rest of your code here
#click.command()
#click.option('--output_dir', default='/home/foo/bar/', help='Default path to save files')
def main(output_dir):
write_to_csv(output_dir)
if __name__ == '__main__':
main()