Develop Reference

How to zip an html file from a stream/rendered dictionary?

I am having trouble downloading an html file through the flask send_file.
Basically, to download an html file alone, it works perfectly. by giving the stream to the send_file function as a parameter
However; I need to put this file into a zip along with other unrelated files. There, in the write function, neither the stream nor the string (result_html) work. I need somehow to transform it directly to an html file and put in the zip file
I don't see how I could do this for the moment. I have the data (output) as a dict...
Thank you if you have any pointers
from flask import render_template, send_file
from io import BytesIO
result_html = render_template('myResult.html', **output)
result_stream = BytesIO(str(result_html).encode())
with ZipFile("zipped_result.zip", "w") as zf:
zf.write(result_html)
# zf.write(other_files)
send_file(zf, as_attachment=True, attachment_filename="myfile.zip")

If I understand you correctly, it is sufficient to write the zip file in a stream and add the result of the rendering as a character string to the zip file. The stream can then be transmitted via send_file.
from flask import render_template, send_file
from io import BytesIO
from zipfile import ZipFile
# ...
#app.route('/download')
def download():
output = { 'name': 'Unknown' }
result_html = render_template('result.html', **output)
stream = BytesIO()
with ZipFile(stream, 'w') as zf:
zf.writestr('result.html', result_html)
# ...
stream.seek(0)
return send_file(stream, as_attachment=True, attachment_filename='archive.zip')

"File not open for reading" Flask after PDF splitting

I'm trying to split a PDF file that gets as a post request to this Flask API. I am using PyPDF2 to work with the pdf files and I'm getting this error when trying to post the freshly split pdf: "IOError: File not open for reading". I tried opening it again before the post but that causes another type of error.
from flask import render_template, request
from app import app
from PyPDF2 import PdfFileWriter, PdfFileReader
import requests
#app.route('/')
def start():
return render_template('home.html')
#upload-route
#app.route('/upload', methods = ['POST'])
def upload():
pdf_data = None
split_pages = []
if 'pdf' in request.files:
incoming_pdf = request.files['pdf']
pdf_data = PdfFileReader(incoming_pdf, 'rb')
for i in range(pdf_data.numPages):
output = PdfFileWriter()
output.addPage(pdf_data.getPage(i))
with open("document-page%s.pdf" % i, "wb").read() as outputStream:
##new_page = output.write(outputStream)
try:
split_pages.append(output)
print('Created: {}'.format("document-page%s.pdf" % i))
files = {'userfile': outputStream }
upload_url ='*URL IM POSTING TO*'
auth_upload={
*AUTH KEYS*
}
r=requests.post(url=upload_url, files=files,data=auth_upload)
finally:
outputStream.close()
else:
return "please upload a file to process"
I know this may be a super basic question but I have spent an outrageous amount of time with this and I no longer think straight.

I just had look at your code, and what about that?
from PyPDF2 import PdfFileWriter, PdfFileReader
with open('./reklamacja.pdf', 'rb') as my_pdf:
pdf_data = PdfFileReader(my_pdf)
for i in range(pdf_data.numPages):
output = PdfFileWriter()
# output.addPage(pdf_data.getPage(i))
output.insertPage(pdf_data.getPage(i))
filename = "document-page%s.pdf" % i
with open(filename, "wb") as outputStream:
output.write(outputStream)
with open(filename, 'rb') as file_to_send:
r = requests.post('http://someurl.bin', files={'myname': file_to_send})
{ Some other code...}
Build that into your flask code, it should work.
For me you just overcomplicated a little it in 'try' block :)
Hope this solves you problem.

Python JSON import to MongoDB from ZIP files

I have a script that gets all of the .zip files from a folder, then one by one, opens the zip file, loads the content of the JSON file inside and imports this to MongoDB.
The error I am getting is the JSON object must be str, bytes or bytearray, not 'TextIOWrapper'
The code is:
import json
import logging
import logging.handlers
import os
from logging.config import fileConfig
from pymongo import MongoClient
def import_json():
try:
client = MongoClient('5.57.62.97', 27017)
db = client['vuln_sets']
coll = db['vulnerabilities']
basepath = os.path.dirname(__file__)
filepath = os.path.abspath(os.path.join(basepath, ".."))
archive_filepath = filepath + '/vuln_files/'
filedir = os.chdir(archive_filepath)
for item in os.listdir(filedir):
if item.endswith('.json'):
file_name = os.path.abspath(item)
fp = open(file_name, 'r')
json_data = json.loads(fp)
for vuln in json_data:
print(vuln)
coll.insert(vuln)
os.remove(file_name)
except Exception as e:
logging.exception(e)
I can get this working to use a single file but not multiple, i.e. to do one file I wrote:
from zipfile import ZipFile
import json
import pymongo
archive = ZipFile("vulners_collections/cve.zip")
archived_file = archive.open(archive.namelist()[0])
archive_content = archived_file.read()
archived_file.close()
connection = pymongo.MongoClient("mongodb://localhost")
db=connection.vulnerability
vuln1 = db.vulnerability_collection
vulners_objects = json.loads(archive_content)
for item in vulners_objects:
vuln1.insert(item)

From my comment above:
I have no experience with glob, but from skimming the doc I get the impression your archive_files is a simple list of file-paths as strings, correct? You can not perform actions like .open on string (thus your error), so try changing your code to this:
...
archive_filepath = filepath + '/vuln_files/'
archive_files = glob.glob(archive_filepath + "/*.zip")
for file in archive_files:
with open(file, "r") as currentFile:
file_content = currentFile.read()
vuln_content = json.loads(file_content)
for item in vuln_content:
coll.insert(item)
...
file is NOT a file object or anything but just a simple string. So you cant perform methods on it that are not supported by string.

You are redefining your iterator by setting it to the result of the namelist method. You need a for loop within the for to go through the contents of the zip file and of course a new iterator variable.

Isn't file.close wrong and the correct call is file.close().
U can use json.load() to load file directly, instead of json.loads()
fp = open(file_name, 'r')
json_data = json.load(fp)
fp.close()

Python Flask send_file StringIO blank files

I want to process a Pandas dataframe and send it to download as a CSV without a temp file. The best way to accomplish this I've seen is to use StringIO. Using the code below, a file downloads with the proper name, however the file is completely blank, and no error is shown. Why doesn't the file contain data?
#app.route('/test_download', methods = ['POST'])
def test_download():
buffer = StringIO()
buffer.write('Just some letters.')
buffer.seek(0)
return send_file(
buffer,
as_attachment=True,
download_name='a_file.txt',
mimetype='text/csv'
)

The issue here is that in Python 3 you need to use StringIO with csv.write and send_file requires BytesIO, so you have to do both.
#app.route('/test_download')
def test_download():
row = ['hello', 'world']
proxy = io.StringIO()
writer = csv.writer(proxy)
writer.writerow(row)
# Creating the byteIO object from the StringIO Object
mem = io.BytesIO()
mem.write(proxy.getvalue().encode())
# seeking was necessary. Python 3.5.2, Flask 0.12.2
mem.seek(0)
proxy.close()
return send_file(
mem,
as_attachment=True,
download_name='test.csv',
mimetype='text/csv'
)
Prior to Flask 2.0, download_name was called attachment_filename.

Use BytesIO to write bytes.
from io import BytesIO
from flask import Flask, send_file
app = Flask(__name__)
#app.route('/test_download', methods=['POST'])
def test_download():
# Use BytesIO instead of StringIO here.
buffer = BytesIO()
buffer.write(b'Just some letters.')
# Or you can encode it to bytes.
# buffer.write('Just some letters.'.encode('utf-8'))
buffer.seek(0)
return send_file(
buffer,
as_attachment=True,
download_name='a_file.txt',
mimetype='text/csv'
)
Prior to Flask 2.0, download_name was called attachment_filename.

make_response
To get Flask to download a csv file to the user, we pass a csv string to the make_response function, which returns a
Response object.
Then we add a Header which tells the browser to accept the file as a download.
The Mimetype also must be set to text/csv in order to get the web browser to save it in something other than an html document.
from flask import Flask, make_response
app = Flask(__name__)
#app.route('/test_download', methods=['POST'])
def test_download():
with StringIO() as buffer:
# forming a StringIO object
buffer = StringIO()
buffer.write('Just some letters.')
# forming a Response object with Headers to return from flask
response = make_response(buffer.getvalue())
response.headers['Content-Disposition'] = 'attachment; filename=namaste.csv'
response.mimetype = 'text/csv'
# return the Response object
return response
P.S. It is preferred to use python's built-in csv library to deal with csv files
References
https://matthewmoisen.com/blog/how-to-download-a-csv-file-in-flask/
https://www.geeksforgeeks.org/stringio-module-in-python/
https://docs.python.org/3/library/csv.html
Namaste 🙏

if someone use python 2.7 with Flask and got the error about the module StringIO by importing it. This post can help you to solve your problem.
If you are importing String IO module, you can just change the import syntax by using this : from io import StringIO instead from StringIO import StringIO.
You can Also use from io import BytesIO if you are using image or some others ressource.
Thank you

How to use pyramid.response.FileIter

I have the following view code that attempts to "stream" a zipfile to the client for download:
import os
import zipfile
import tempfile
from pyramid.response import FileIter
def zipper(request):
_temp_path = request.registry.settings['_temp']
tmpfile = tempfile.NamedTemporaryFile('w', dir=_temp_path, delete=True)
tmpfile_path = tmpfile.name
## creating zipfile and adding files
z = zipfile.ZipFile(tmpfile_path, "w")
z.write('somefile1.txt')
z.write('somefile2.txt')
z.close()
## renaming the zipfile
new_zip_path = _temp_path + '/somefilegroup.zip'
os.rename(tmpfile_path, new_zip_path)
## re-opening the zipfile with new name
z = zipfile.ZipFile(new_zip_path, 'r')
response = FileIter(z.fp)
return response
However, this is the Response I get in the browser:
Could not convert return value of the view callable function newsite.static.zipper into a response object. The value returned was .
I suppose I am not using FileIter correctly.
UPDATE:
Since updating with Michael Merickel's suggestions, the FileIter function is working correctly. However, still lingering is a MIME type error that appears on the client (browser):
Resource interpreted as Document but transferred with MIME type application/zip: "http://newsite.local:6543/zipper?data=%7B%22ids%22%3A%5B6%2C7%5D%7D"
To better illustrate the issue, I have included a tiny .py and .pt file on Github: https://github.com/thapar/zipper-fix

FileIter is not a response object, just like your error message says. It is an iterable that can be used for the response body, that's it. Also the ZipFile can accept a file object, which is more useful here than a file path. Let's try writing into the tmpfile, then rewinding that file pointer back to the start, and using it to write out without doing any fancy renaming.
import os
import zipfile
import tempfile
from pyramid.response import FileIter
def zipper(request):
_temp_path = request.registry.settings['_temp']
fp = tempfile.NamedTemporaryFile('w+b', dir=_temp_path, delete=True)
## creating zipfile and adding files
z = zipfile.ZipFile(fp, "w")
z.write('somefile1.txt')
z.write('somefile2.txt')
z.close()
# rewind fp back to start of the file
fp.seek(0)
response = request.response
response.content_type = 'application/zip'
response.app_iter = FileIter(fp)
return response
I changed the mode on NamedTemporaryFile to 'w+b' as per the docs to allow the file to be written to and read from.

current Pyramid version has 2 convenience classes for this use case- FileResponse, FileIter. The snippet below will serve a static file. I ran this code - the downloaded file is named "download" like the view name. To change the file name and more set the Content-Disposition header or have a look at the arguments of pyramid.response.Response.
from pyramid.response import FileResponse
#view_config(name="download")
def zipper(request):
path = 'path_to_file'
return FileResponse(path, request) #passing request is required
docs:
http://docs.pylonsproject.org/projects/pyramid/en/latest/api/response.html#
hint: extract the Zip logic from the view if possible