I am writing my first flask application. I am dealing with file uploads, and basically what I want is to read the data/content of the uploaded file without saving it and then print it on the resulting page. Yes, I am assuming that the user uploads a text file always.
Here is the simple upload function i am using:
#app.route('/upload/', methods=['GET', 'POST'])
def upload():
if request.method == 'POST':
file = request.files['file']
if file:
filename = secure_filename(file.filename)
file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
a = 'file uploaded'
return render_template('upload.html', data = a)
Right now, I am saving the file, but what I need is that 'a' variable to contain the content/data of the file .. any ideas?
FileStorage contains stream field. This object must extend IO or file object, so it must contain read and other similar methods. FileStorage also extend stream field object attributes, so you can just use file.read() instead file.stream.read(). Also you can use save argument with dst parameter as StringIO or other IO or file object to copy FileStorage.stream to another IO or file object.
See documentation: http://flask.pocoo.org/docs/api/#flask.Request.files and http://werkzeug.pocoo.org/docs/datastructures/#werkzeug.datastructures.FileStorage.
If you want to use standard Flask stuff - there's no way to avoid saving a temporary file if the uploaded file size is > 500kb. If it's smaller than 500kb - it will use "BytesIO", which stores the file content in memory, and if it's more than 500kb - it stores the contents in TemporaryFile() (as stated in the werkzeug documentation). In both cases your script will block until the entirety of uploaded file is received.
The easiest way to work around this that I have found is:
1) Create your own file-like IO class where you do all the processing of the incoming data
2) In your script, override Request class with your own:
class MyRequest( Request ):
def _get_file_stream( self, total_content_length, content_type, filename=None, content_length=None ):
return MyAwesomeIO( filename, 'w' )
3) Replace Flask's request_class with your own:
app.request_class = MyRequest
4) Go have some beer :)
I share my solution (assuming everything is already configured to connect to google bucket in flask)
from google.cloud import storage
#app.route('/upload/', methods=['POST'])
def upload():
if request.method == 'POST':
# FileStorage object wrapper
file = request.files["file"]
if file:
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = app.config['GOOGLE_APPLICATION_CREDENTIALS']
bucket_name = "bucket_name"
storage_client = storage.Client()
bucket = storage_client.bucket(bucket_name)
# Upload file to Google Bucket
blob = bucket.blob(file.filename)
blob.upload_from_string(file.read())
My post
Direct to Google Bucket in flask
I was trying to do the exact same thing, open a text file (a CSV for Pandas actually). Don't want to make a copy of it, just want to open it. The form-WTF has a nice file browser, but then it opens the file and makes a temporary file, which it presents as a memory stream. With a little work under the hood,
form = UploadForm()
if form.validate_on_submit():
filename = secure_filename(form.fileContents.data.filename)
filestream = form.fileContents.data
filestream.seek(0)
ef = pd.read_csv( filestream )
sr = pd.DataFrame(ef)
return render_template('dataframe.html',tables=[sr.to_html(justify='center, classes='table table-bordered table-hover')],titles = [filename], form=form)
I share my solution, using pandas
#app.route('/upload/', methods=['POST'])
def upload():
if request.method == 'POST':
# FileStorage object wrapper
file = request.files["file"]
if file:
df = pd.read_excel(files_excel["file"])
Building on a great answer by #tbicr the simplest form of that boils down to:
for line in request.files.get('file'):
print("Next line: " + line)
in function
def handleUpload():
if 'photo' in request.files:
photo = request.files['photo']
if photo.filename != '':
image = request.files['photo']
image_string = base64.b64encode(image.read())
image_string = image_string.decode('utf-8')
#use this to remove b'...' to get raw string
return render_template('handleUpload.html',filestring = image_string)
return render_template('upload.html')
in html file
<html>
<head>
<title>Simple file upload using Python Flask</title>
</head>
<body>
{% if filestring %}
<h1>Raw image:</h1>
<h1>{{filestring}}</h1>
<img src="data:image/png;base64, {{filestring}}" alt="alternate" />.
{% else %}
<h1></h1>
{% endif %}
</body>
In case we want to dump the in memory file to disk. This code can be used
if isinstanceof(obj,SpooledTemporaryFile):
obj.rollover()
Related
I am trying to give the user a "Save as" option when the user clicks the download button in my Django app. When the user clicks the button it kicks-off the following function. The function gets some CSVs from a blob container in Azure and adds them to a zip. That zip should then be offered to download and store in a location of the user's choice.
def create_downloadable_zip():
container_client = az.container_client(container_name=blob_generator.container_name)
blobs = container_client.list_blobs()
zip_file = zipfile.ZipFile(f'{models.AppRun.client_name}.zip', 'w')
for blob in blobs:
if blob.name.endswith(".csv"):
downloaded_blob = container_client.download_blob(blob)
blob_data = downloaded_blob.readall()
zip_file.writestr(blob.name, blob_data)
zip_file.close()
return zip_file
My views.py looks like follow:
def download_file(request):
if request.method == 'POST':
zip = create_downloadable_zip()
response = HttpResponse(zip, content_type='application/zip')
response['Content-Disposition'] = 'attachement;' f'filename={zip}.zip'
return response
#
# else:
# # return a 404 response if this is a POST request
# return HttpResponse(status=404)
return render(request, "download_file.html")
The functionality works, but it returns an empty non-zip file when the "Save as" window pop-ups. However, the actual zip file contains the files is being saved in the root folder of the Django project.
I really don't get why I doesn't return the zip file from memory, but rather directly stores that zip file in root and returns an empty non-zip file with the download functionality.
Someone knows what I am doing wrong?
zipfile is used to open a file, but it is not the actual file, simply a zipfile object as #b-remmelzwaal mentioned. You will need to create a file like object, and return that instead. This can be done using io.BytesIO.
from io import BytesIO
from zipfile import ZipFile
def create_zip():
container_client = az.container_client(container_name=blob_generator.container_name)
blobs = container_client.list_blobs()
buffer = BytesIO()
with ZipFile(buffer, 'w') as zip_file:
for blob in blobs:
if blob.name.endswith(".csv"):
downloaded_blob = container_client.download_blob(blob)
blob_data = downloaded_blob.readall()
zip_file.writestr(blob.name, blob_data)
return buffer.getvalue()
Note we are returning the file like object, not the zip file object. This is because buffer represents the actual file you've created.
You don't have to use a context manager, but I find them very useful.
Also, check your spelling for the line:
# attachment instead attachement
response['Content-Disposition'] = 'attachment;' f'filename={zip}.zip'
BytesIO Documentation
I'm struggling to understand how to pass an uploaded filename to the next #app.route in flask to retrieve a file saved on the server.
I have been able to create a form to upload the file to the server like this, and have been able to render the file data and the filename to the proceeding view.
#app.route('/file_upload', methods=['GET', 'POST'])
def upload():
form = UploadForm()
if form.validate_on_submit():
f=form.file.data
filename = secure_filename(f.filename)
file_path = os.path.join(UPLOAD_FOLDER, filename)
f.save(file_path)
Input_data_dataframe = pd.read_excel(f)
return render_template('Set_parameters.html',filename=filename,tables=[Input_data_dataframe.to_html(classes=["table table-dark table-hover"], header="true", index=False,)])
return render_template('file_upload.html',form=form)
The file successfully saves at the required server destination, and the filename is also rendered in the next view like so:
I now want this file to be retrieved from the server location for further processing. I understand that I can use the send_from_directory() function to retrieve the file, however I am not sure how to implement it with the correct filename from the previous #app.route('/file_upload')
#app.route('/Set_Parameters/', methods=['GET', 'POST'])
def processing(filename):
retrievedfile= send_from_directory(app.config['UPLOAD_FOLDER'],filename=filename, as_attachment=False)
Input_data_dataframe = pd.read_excel(retrievedfile) #converts uploaded file to a dataframe astype(int)
if request.method == 'POST':
Input_value1= request.form.get['Input_value1'] #get html form data value
Input_value= request.form.get['Input_value2'] #get html form data value
return render_template('Results.html',Input_value1=Input_value1, Input_value2=Input_value2, tables=[Input_data_dataframe.to_html(classes=["table table-dark table-hover"], header="true", index=False,)])
return render_template('Set_parameters.html')
Currently if I was to run as is, I would get this error
TypeError: processing() missing 1 required positional argument: 'filename' as I can understand I haven't passed the filename to the function.. I am just unsure how to do this without hardcoding the filename into the code, and then to use the same file for further processing.
Any help would be greatly appreciated.
Your processing function expects filename parameter from URL. You can see how it should be done here: https://flask.palletsprojects.com/en/1.1.x/quickstart/#variable-rules.
Route to this view should be: /Set_Parameters/<filename>/. It means that action attribute of the from should be, for example: action="/Set_Parameters/This_is_the_file.xlsx".
If you render that form in flask, using flask templates, it should be easy if you have a filename in a render context:
<form action="/Set_Parameters/{{ filename }}">
...
</form>
I know that there are some similar questions, although none of them has helped me to resolve my issue. The ultimate goal is to have an flask app, which takes an excel file, stores it inside of the azure blob storage, which is then used by my python function app to do some further transformations. What I struggle with, is how to encode/decode this file as I have to use
block_blob_service.create_blob_from_bytes() function.
The only thing that came to my mind was to use Pandas library to read this excel and then tobytes() function. This way, I am able to upload my excel to the blob as a CSV file. However I cannot really convert it to its previous form.
This is how it looks like after opening :
9]�0��j�9p/�j�� �`��/wj1=p/�j�� p�^�.wj2=p/�[...]
Trying to decode it with utf-8 gives me some never ending errors saying that 'utf-8' codec can't decode byte[...] I have tried many different encodings but it always ends up with this message at some byte. Excel contains numericals, strings and dates.
So, to the code:
#getting the file
file = request.files['file']
#reading into pandas df
data = pd.read_excel(file)
df_to_records = data.to_records(index=False)
records_to_bytes = df_to_records.tobytes()
block_blob_service = BlockBlobService(account_name='xxx', account_key="xxx")
block_blob_service.create_blob_from_bytes("test","mydata.csv",records_to_bytes)
Thank you for any advice!
Update: The full code working at my side.
import os
from flask import Flask, request, redirect, url_for
from azure.storage.blob import BlockBlobService
import string, random, requests
app = Flask(__name__, instance_relative_config=True)
account = "your_account name" # Azure account name
key = "account key" # Azure Storage account access key
container = "f22" # Container name
blob_service = BlockBlobService(account_name=account, account_key=key)
#app.route('/', methods=['GET', 'POST'])
def upload_file():
if request.method == 'POST':
file = request.files['file']
file.seek(0)
filename = "test1.csv" # just use a hardcoded filename for test
blob_service.create_blob_from_stream(container, filename, file)
ref = 'http://'+ account + '.blob.core.windows.net/' + container + '/' + filename
return '''
<!doctype html>
<title>File Link</title>
<h1>Uploaded File Link</h1>
<p>''' + ref + '''</p>
<img src="'''+ ref +'''">
'''
return '''
<!doctype html>
<title>Upload new File</title>
<h1>Upload new File</h1>
<form action="" method=post enctype=multipart/form-data>
<p><input type=file name=file>
<input type=submit value=Upload>
</form>
'''
if __name__ == '__main__':
app.run(debug=True)
After running:
After select a .csv file and click upload button, check the .csv file in azure portal:
I think you can take a try with method create_blob_from_stream instead of create_blob_from_bytes.
Here is the sample code:
def upload_file():
if request.method == 'POST':
file = request.files['file']
file.seek(0)
try:
blob_service = BlockBlobService(account_name='xxx', account_key="xxx")
blob_service.create_blob_from_stream(container, filename, file)
except Exception:
print 'Exception=' + Exception
pass
I am making a data visualization tool that takes input from the user (choosing a file on the computer); processes it in Python with Pandas, Numpy, etc; and displays the data in the browser on a local server.
I am having trouble accessing the data once the file is selected using an HTML input form.
HTML form:
<form action="getfile" method="POST" enctype="multipart/form-data">
Project file path: <input type="file" name="myfile"><br>
<input type="submit" value="Submit">
</form>
Flask routing:
#app.route("/")
def index():
return render_template('index.html')
#app.route('/getfile', methods=['GET','POST'])
def getfile():
if request.method == 'POST':
result = request.form['myfile']
else:
result = request.args.get['myfile']
return result
This returns a "Bad Request The browser (or proxy) sent a request that this server could not understand." error. I have tried a number of different ways of getting the data out of the file and simply printing it to the screen to start, and have received a range of errors including "TypeError: 'FileStorage' object is not callable" and "ImmutableMultiDict' object is not callable". Any pointers on how to approach this task correctly are appreciated.
Try this. I've been working on saving and unzipping files for the last few days. If you have any trouble with this code, let me know :)
I'd suggest saving the file on disk and then reading it. If you don't want to do that, you needn't.
from flask import Flask, render_template, request
from werkzeug import secure_filename
#app.route('/getfile', methods=['GET','POST'])
def getfile():
if request.method == 'POST':
# for secure filenames. Read the documentation.
file = request.files['myfile']
filename = secure_filename(file.filename)
# os.path.join is used so that paths work in every operating system
file.save(os.path.join("wherever","you","want",filename))
# You should use os.path.join here too.
with open("wherever/you/want/filename") as f:
file_content = f.read()
return file_content
else:
result = request.args.get['myfile']
return result
And as zvone suggested in the comments, I too would advise against using GET to upload files.
Uploading files
os.path by Effbot
Edit:-
You don't want to save the file.
Uploaded files are stored in memory or at a temporary location on the filesystem. You can access those files by looking at the files attribute on the request object. Each uploaded file is stored in that dictionary. It behaves just like a standard Python file object, but it also has a save() method that allows you to store that file on the filesystem of the server.
I got this from the Flask documentation. Since it's a Python file you can directly use file.read() on it without file.save().
Also if you need to save it for sometime and then delete it later, you can use os.path.remove to delete the file after saving it. Deleting a file in Python
A input type=file data isn't passed in as the form dictionary of a request object. It is passed in as request.files (files dictionary in the request object).
So simply change:
result = request.form['myfile']
to
result = request.files['myfile']
I am using Flask micro-framework 0.6 and Python 2.6
I need to get the mimetype from an uploaded file so I can store it.
Here is the relevent Python/Flask code:
#app.route('/upload_file', methods=['GET', 'POST'])
def upload_file():
if request.method == 'POST':
file = request.files['file']
mimetype = #FIXME
if file:
file.save(os.path.join(UPLOAD_FOLDER, 'File-Name')
return redirect(url_for('uploaded_file'))
else:
return redirect(url_for('upload'))
And here is the code for the webpage:
<form action="upload_file" method=post enctype=multipart/form-data>
Select file to upload: <input type=file name=file>
<input type=submit value=Upload>
</form>
The code works, but I need to be able to get the mimetype when it uploads. I've had a look at the Flask docs here: http://flask.pocoo.org/docs/api/#incoming-request-data
So I know it does get the mimetype, but I can't work out how to retrieve it - as a text string, e.g. 'txt/plain'.
Any ideas?
Thank you.
From the docs, file.content_type contains the full type with encoding, mimetype contains just the mime type.
#app.route('/upload_file', methods=['GET', 'POST'])
def upload_file():
if request.method == 'POST':
file = request.files.get('file')
if file:
mimetype = file.content_type
filename = werkzeug.secure_filename(file.filename)
file.save(os.path.join(UPLOAD_FOLDER, filename)
return redirect(url_for('uploaded_file'))
else:
return redirect(url_for('upload'))
You could in theory use request.files['YOUR_FILE_KEY'].content_type, but the implementation (included below, found in werkzeug.datastructures) either trust whatever the client is providing, or uses mimetypes.guess_type which only checks the file extension (see Python doc here).
class FileMultiDict(MultiDict):
"""A special :class:`MultiDict` that has convenience methods to add
files to it. This is used for :class:`EnvironBuilder` and generally
useful for unittesting.
.. versionadded:: 0.5
"""
def add_file(self, name, file, filename=None, content_type=None):
"""Adds a new file to the dict. `file` can be a file name or
a :class:`file`-like or a :class:`FileStorage` object.
:param name: the name of the field.
:param file: a filename or :class:`file`-like object
:param filename: an optional filename
:param content_type: an optional content type
"""
if isinstance(file, FileStorage):
value = file
else:
if isinstance(file, string_types):
if filename is None:
filename = file
file = open(file, 'rb')
if filename and content_type is None:
content_type = mimetypes.guess_type(filename)[0] or \
'application/octet-stream'
value = FileStorage(file, filename, name, content_type)
self.add(name, value)
Depending on your use case, you might want to use python-magic which will use the actual file to get the mimetype. It would something like that:
import magic
def get_mimetype(data: bytes) -> str:
"""Get the mimetype from file data."""
f = magic.Magic(mime=True)
return f.from_buffer(data)
get_mimetype(request.files['YOUR_FILE_KEY'].stream.read(MAX_LENGTH))