Develop Reference

Getting NULL data when uploading file to Sqlite DB [duplicate]

I am writing my first flask application. I am dealing with file uploads, and basically what I want is to read the data/content of the uploaded file without saving it and then print it on the resulting page. Yes, I am assuming that the user uploads a text file always.
Here is the simple upload function i am using:
#app.route('/upload/', methods=['GET', 'POST'])
def upload():
if request.method == 'POST':
file = request.files['file']
if file:
filename = secure_filename(file.filename)
file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
a = 'file uploaded'
return render_template('upload.html', data = a)
Right now, I am saving the file, but what I need is that 'a' variable to contain the content/data of the file .. any ideas?

FileStorage contains stream field. This object must extend IO or file object, so it must contain read and other similar methods. FileStorage also extend stream field object attributes, so you can just use file.read() instead file.stream.read(). Also you can use save argument with dst parameter as StringIO or other IO or file object to copy FileStorage.stream to another IO or file object.
See documentation: http://flask.pocoo.org/docs/api/#flask.Request.files and http://werkzeug.pocoo.org/docs/datastructures/#werkzeug.datastructures.FileStorage.

If you want to use standard Flask stuff - there's no way to avoid saving a temporary file if the uploaded file size is > 500kb. If it's smaller than 500kb - it will use "BytesIO", which stores the file content in memory, and if it's more than 500kb - it stores the contents in TemporaryFile() (as stated in the werkzeug documentation). In both cases your script will block until the entirety of uploaded file is received.
The easiest way to work around this that I have found is:
1) Create your own file-like IO class where you do all the processing of the incoming data
2) In your script, override Request class with your own:
class MyRequest( Request ):
def _get_file_stream( self, total_content_length, content_type, filename=None, content_length=None ):
return MyAwesomeIO( filename, 'w' )
3) Replace Flask's request_class with your own:
app.request_class = MyRequest
4) Go have some beer :)

I share my solution (assuming everything is already configured to connect to google bucket in flask)
from google.cloud import storage
#app.route('/upload/', methods=['POST'])
def upload():
if request.method == 'POST':
# FileStorage object wrapper
file = request.files["file"]
if file:
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = app.config['GOOGLE_APPLICATION_CREDENTIALS']
bucket_name = "bucket_name"
storage_client = storage.Client()
bucket = storage_client.bucket(bucket_name)
# Upload file to Google Bucket
blob = bucket.blob(file.filename)
blob.upload_from_string(file.read())
My post
Direct to Google Bucket in flask

I was trying to do the exact same thing, open a text file (a CSV for Pandas actually). Don't want to make a copy of it, just want to open it. The form-WTF has a nice file browser, but then it opens the file and makes a temporary file, which it presents as a memory stream. With a little work under the hood,
form = UploadForm()
if form.validate_on_submit():
filename = secure_filename(form.fileContents.data.filename)
filestream = form.fileContents.data
filestream.seek(0)
ef = pd.read_csv( filestream )
sr = pd.DataFrame(ef)
return render_template('dataframe.html',tables=[sr.to_html(justify='center, classes='table table-bordered table-hover')],titles = [filename], form=form)

I share my solution, using pandas
#app.route('/upload/', methods=['POST'])
def upload():
if request.method == 'POST':
# FileStorage object wrapper
file = request.files["file"]
if file:
df = pd.read_excel(files_excel["file"])

Building on a great answer by #tbicr the simplest form of that boils down to:
for line in request.files.get('file'):
print("Next line: " + line)

in function
def handleUpload():
if 'photo' in request.files:
photo = request.files['photo']
if photo.filename != '':
image = request.files['photo']
image_string = base64.b64encode(image.read())
image_string = image_string.decode('utf-8')
#use this to remove b'...' to get raw string
return render_template('handleUpload.html',filestring = image_string)
return render_template('upload.html')
in html file
<html>
<head>
<title>Simple file upload using Python Flask</title>
</head>
<body>
{% if filestring %}
<h1>Raw image:</h1>
<h1>{{filestring}}</h1>
<img src="data:image/png;base64, {{filestring}}" alt="alternate" />.
{% else %}
<h1></h1>
{% endif %}
</body>

In case we want to dump the in memory file to disk. This code can be used
if isinstanceof(obj,SpooledTemporaryFile):
obj.rollover()

Flask: How do I pass an uploaded file as input to a method? AttributeError: '_io.BytesIO' object has no attribute 'lower'

I'm completely new to flask (in fact I've just started this morning) and so far I've only managed to implement a basic upload function (taken from the flask doc). The file upload works but I just can't seem to figure out how to pass the file into another method (mining). When I do, I always get the error message: AttributeError: '_io.BytesIO' object has no attribute 'lower'. It would mean the world to me to receive any kind of help! Thank you all in advance.
def mining(xes):
log = xes_importer.apply(xes)
tracefilter_log_pos = attributes_filter.apply_events(log, ["complete"], parameters={attributes_filter.Parameters.ATTRIBUTE_KEY: "lifecycle:transition", attributes_filter.Parameters.POSITIVE: True})
variants = pm4py.get_variants_as_tuples(tracefilter_log_pos)
Alpha(variants.keys())
Heuristic(variants.keys(), 1, 0.5)
UPLOAD_FOLDER = '/Users/jenny/processmining_lab/uploads'
ALLOWED_EXTENSIONS = set(['xes'])
app = Flask(__name__)
app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER
def allowed_file(filename):
return '.' in filename and \
filename.rsplit('.', 1)[1].lower() in ALLOWED_EXTENSIONS
#app.route('/', methods=['GET', 'POST'])
def upload_file():
if request.method == 'POST':
# check if the post request has the file part
if 'file' not in request.files:
flash('No file part')
return redirect(request.url)
file = request.files['file']
# If the user does not select a file, the browser submits an
# empty file without a filename.
if file.filename == '':
flash('No selected file')
return redirect(request.url)
if file and allowed_file(file.filename):
filename = secure_filename(file.filename)
file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
return mining(file)
return '''
<!doctype html>
<title>Upload new File</title>
<h1>Upload new File</h1>
<form method=post enctype=multipart/form-data>
<input type=file name=file>
<input type=submit value=Upload>
</form>
'''
Here is the traceback

This depends on what you're mining function looks like.
file is a werkzeug.FileStorage object which has it's own various methods.
So (without seeing) your mining function it should probably be something like:
def mining(file_obj):
lower_fname = file_obj.filename.lower()
data = file_obj.read()
# Do something
The AttributeError you mention sounds like you're trying to run the lower method of something which doesn't return a string.
As a side-note: it's difficult to know where in your code this exception hasn't occured as you never posted the full traceback.
EDIT
My mining function is actually at the top of the code I've inserted. I've also now included the traceback.
Looks like you're using PM4Py for which this document states:
from pm4py.objects.log.importer.xes import importer as xes_importer
log = xes_importer.apply('<path_to_xes_file.xes>')
So you need to actually pass a file path to the mining function.
I suggest changing your main code to do:
if file and allowed_file(file.filename):
sfilename = secure_filename(file.filename)
output_path = os.path.join(app.config['UPLOAD_FOLDER'], sfilename)
# Now save to this `output_path` and pass that same var to your mining function
file.save(output_path)
return mining(output_path)

python flask render_template and return csv/excel to browser [duplicate]

This question already has answers here:
Return a download and rendered page in one Flask response
(2 answers)
Closed 1 year ago.
I am creating a flask app that takes a CSV as input. Parses the CSV and the content. Then returns an updated CSV to the browser to download/or instantly downloads. The program needs an HTML template for users to interact. I am attempting to use the render_template() method to return the HTML file and the CSV file via flask. What is the best way to achieve this?
#app.route("/", methods=["GET", "POST"])
def index():
if request.method == "POST":
print("POST request received!")
if "file" not in request.files:
return redirect(request.url)
list_keywords = request.files["file"]
if not list_keywords:
return "No file"
the_list = io.StringIO(list_keywords.stream.read().decode("UTF8"), newline=None)
csv_input = csv.reader(the_list)
...# rest of program.....
csv_file1 = pd.DataFrame(internal_linking_opportunities,
columns=["Keyword", "Text", "Source URL", "Target URL", "Link Presence", "Keyword Position"])
si = io.StringIO()
cw = csv.writer(csv_file1)
output = make_response(si.getvalue())
output.headers["Content-Disposition"] = "attachment; filename=export.csv"
output.headers["Content-type"] = "text/csv"
return Response(render_template('index.html', output=output))
I have attempted to use other methods to return the CSV with no luck. Starting to think it could be due to the render_template function that I am using. Your help is appreciated.

The second parameter of render_template is called context and can be used to provide the template with some variables for rendering. Adding a Response object to the templates context is not going to return that Response, but instead make that Response object accessible from within the template.
Flask has a dedicated method for sending files back to the browser:
https://flask.palletsprojects.com/en/1.1.x/api/#flask.send_file
Your return statement could look something like this:
return send_file(file_pointer, as_attachment=True, attachment_filename="export.csv")
Instead of providing the file pointer as the first argument, you can also use a file from your local disk by providing the path to that file. If you provide the file pointer, make sure to set it to the start of the file.

How to properly encode/decode an excel file using Python?

I know that there are some similar questions, although none of them has helped me to resolve my issue. The ultimate goal is to have an flask app, which takes an excel file, stores it inside of the azure blob storage, which is then used by my python function app to do some further transformations. What I struggle with, is how to encode/decode this file as I have to use
block_blob_service.create_blob_from_bytes() function.
The only thing that came to my mind was to use Pandas library to read this excel and then tobytes() function. This way, I am able to upload my excel to the blob as a CSV file. However I cannot really convert it to its previous form.
This is how it looks like after opening :
9]�0��j�9p/�j���`��/wj1=p/�j��p�^�.wj2=p/�[...]
Trying to decode it with utf-8 gives me some never ending errors saying that 'utf-8' codec can't decode byte[...] I have tried many different encodings but it always ends up with this message at some byte. Excel contains numericals, strings and dates.
So, to the code:
#getting the file
file = request.files['file']
#reading into pandas df
data = pd.read_excel(file)
df_to_records = data.to_records(index=False)
records_to_bytes = df_to_records.tobytes()
block_blob_service = BlockBlobService(account_name='xxx', account_key="xxx")
block_blob_service.create_blob_from_bytes("test","mydata.csv",records_to_bytes)
Thank you for any advice!

Update: The full code working at my side.
import os
from flask import Flask, request, redirect, url_for
from azure.storage.blob import BlockBlobService
import string, random, requests
app = Flask(__name__, instance_relative_config=True)
account = "your_account name" # Azure account name
key = "account key" # Azure Storage account access key
container = "f22" # Container name
blob_service = BlockBlobService(account_name=account, account_key=key)
#app.route('/', methods=['GET', 'POST'])
def upload_file():
if request.method == 'POST':
file = request.files['file']
file.seek(0)
filename = "test1.csv" # just use a hardcoded filename for test
blob_service.create_blob_from_stream(container, filename, file)
ref = 'http://'+ account + '.blob.core.windows.net/' + container + '/' + filename
return '''
<!doctype html>
<title>File Link</title>
<h1>Uploaded File Link</h1>
<p>''' + ref + '''</p>
<img src="'''+ ref +'''">
'''
return '''
<!doctype html>
<title>Upload new File</title>
<h1>Upload new File</h1>
<form action="" method=post enctype=multipart/form-data>
<p><input type=file name=file>
<input type=submit value=Upload>
</form>
'''
if __name__ == '__main__':
app.run(debug=True)
After running:
After select a .csv file and click upload button, check the .csv file in azure portal:
I think you can take a try with method create_blob_from_stream instead of create_blob_from_bytes.
Here is the sample code:
def upload_file():
if request.method == 'POST':
file = request.files['file']
file.seek(0)
try:
blob_service = BlockBlobService(account_name='xxx', account_key="xxx")
blob_service.create_blob_from_stream(container, filename, file)
except Exception:
print 'Exception=' + Exception
pass

Using Flask to load a txt file through the browser and access its data for processing

I am making a data visualization tool that takes input from the user (choosing a file on the computer); processes it in Python with Pandas, Numpy, etc; and displays the data in the browser on a local server.
I am having trouble accessing the data once the file is selected using an HTML input form.
HTML form:
<form action="getfile" method="POST" enctype="multipart/form-data">
Project file path: <input type="file" name="myfile"><br>
<input type="submit" value="Submit">
</form>
Flask routing:
#app.route("/")
def index():
return render_template('index.html')
#app.route('/getfile', methods=['GET','POST'])
def getfile():
if request.method == 'POST':
result = request.form['myfile']
else:
result = request.args.get['myfile']
return result
This returns a "Bad Request The browser (or proxy) sent a request that this server could not understand." error. I have tried a number of different ways of getting the data out of the file and simply printing it to the screen to start, and have received a range of errors including "TypeError: 'FileStorage' object is not callable" and "ImmutableMultiDict' object is not callable". Any pointers on how to approach this task correctly are appreciated.

Try this. I've been working on saving and unzipping files for the last few days. If you have any trouble with this code, let me know :)
I'd suggest saving the file on disk and then reading it. If you don't want to do that, you needn't.
from flask import Flask, render_template, request
from werkzeug import secure_filename
#app.route('/getfile', methods=['GET','POST'])
def getfile():
if request.method == 'POST':
# for secure filenames. Read the documentation.
file = request.files['myfile']
filename = secure_filename(file.filename)
# os.path.join is used so that paths work in every operating system
file.save(os.path.join("wherever","you","want",filename))
# You should use os.path.join here too.
with open("wherever/you/want/filename") as f:
file_content = f.read()
return file_content
else:
result = request.args.get['myfile']
return result
And as zvone suggested in the comments, I too would advise against using GET to upload files.
Uploading files
os.path by Effbot
Edit:-
You don't want to save the file.
Uploaded files are stored in memory or at a temporary location on the filesystem. You can access those files by looking at the files attribute on the request object. Each uploaded file is stored in that dictionary. It behaves just like a standard Python file object, but it also has a save() method that allows you to store that file on the filesystem of the server.
I got this from the Flask documentation. Since it's a Python file you can directly use file.read() on it without file.save().
Also if you need to save it for sometime and then delete it later, you can use os.path.remove to delete the file after saving it. Deleting a file in Python

A input type=file data isn't passed in as the form dictionary of a request object. It is passed in as request.files (files dictionary in the request object).
So simply change:
result = request.form['myfile']
to
result = request.files['myfile']