I am creating a tool where either
A new XLSX file is generated for the user to download
The user can upload an XLSX file they have, I will read the contents of that file, aand use them to generate a new file for the user to download.
I would like to make use of Pandas to read the XLSX file into a dataframe, so I can work with it easily. However, I can't get it working. Can you help me?
Example extract from CGI file:
import pandas as pd
import cgi
from mako.template import Template
from mako.lookup import TemplateLookup
import http.cookies as Cookie
import os
import tempfile
import shutil
import sys
cookie = Cookie.SimpleCookie(os.environ.get("HTTP_COOKIE"))
method = os.environ.get("REQUEST_METHOD", "GET")
templates = TemplateLookup(directories = ['templates'], output_encoding='utf-8')
if method == "GET": # This is for getting the page
template = templates.get_template("my.html")
sys.stdout.flush()
sys.stdout.buffer.write(b"Content-Type: text/html\n\n")
sys.stdout.buffer.write(
template.render())
if method == "POST":
form = cgi.FieldStorage()
print("Content-Type: application/vnd.ms-excel")
print("Content-Disposition: attachment; filename=NewFile.xlsx\n")
output_path = "/tmp/" + next(tempfile._get_candidate_names()) + '.xlsx'
data = *some pandas dataframe previously created*
if "editfile" in form:
myfilename = form['myfile'].filename
with open(myfilename, 'wb') as f:
f.write(form['myfile'].file.read())
data = pd.read_excel(myfilename)
data.to_excel(output_path)
with open(path, "rb") as f:
sys.stdout.flush()
shutil.copyfileobj(f, sys.stdout.buffer)
Example extract from HTML file:
<p>Press the button below to generate a new version of the xlsx file</p>
<form method=post>
<p><input type=submit value='Generate new version of file' name='newfile'>
<div class="wrapper">
</div>
</form>
<br>
<p>Or upload a file.</p>
<p>In this case, a new file will be created using the contents of this file.</p>
<form method="post" enctype="multipart/form-data">
<input id="fileupload" name="myfile" type="file" />
<input value="Upload and create new file" name='editfile' type="submit" />
</form>
This works without the if "editfile" in form: bit so I know something is going wrong when I am trying to access the file that the user has uploaded.
The problem is that whilst a file is created, the created file has a file size of 0 KB and will not open in Excel. Crucially, the file that the user has uploaded can not be found in the location that I have written it out.
You've passed myfilename to pandas; however that file doesn't exist on the server yet. You'll have to save the file somewhere locally first before using it.
The following will download the file to the current directory (same directory as the CGI script). Of course, you're welcome to save it to some more suitable directory, depending on your setup.
form = cgi.FieldStorage()
myfilename = form['myfile'].filename
with open(myfilename, 'wb') as f: # Save the file locally
f.write(form['myfile'].file.read())
data = pd.read_excel(myfilename)
Related
I'm using python pandas and flask for some postprocessing tasks (anlaysis and visualization). Until now I uploaded/read *.csv *.xlsx and *.xls via pd.read_csv, pd.read_xlsx. Everything worked quiet fine.
Now I have a *.xml file as datasource and tried according my habit pattern.
So i tried:
<form action="/input" method="POST" enctype="multipart/form-data">
<input class="form-control" type="file" name="file">
<input type="submit" class="btn btn-outline-secondary" name="Preview" value ="Preview Data" > </input>
from flask import Flask, render_template,request, render_template
import pandas as pd
import xml.etree.ElementTree as ET
#app.route("/input", methods=['POST', 'GET'])
def input():
if request.method == 'POST':
if request.form['Preview'] == "Preview Data":
file = request.files['file']
filename = file.filename
if '.xml' in filename:
content = pd.read_xml(file, parser='lxml')
But when I pass a .xml file to the app via the form. I get the error:
File "C:\ProgramData\MiniforgeEnvs\TestEnv\lib\site-packages\pandas\io\xml.py", line 627, in _parse_doc
with preprocess_data(handle_data) as xml_data:
AttributeError: __enter__
I tried check different options:
when I use the inbuild xml.etree package it works fine:
import xml.etree.ElementTree as ET
if '.xml' in filename:
tree = ET.parse(file)
root = tree.getroot()
print(root[1][0][1].attrib)
when I load the .xml direct from the app directory into pd.read_xml() it also works fine:
if '.xml' in filename:
content = pd.read_xml('SampleExport.xml', parser='lxml')
I tried different prasers: "lxml" and "etree"
But at the end when I pass the .xml via the Form/input and using pd.read_xml(file,parser='lxml') I got the error from above.
I just solved my issue even though I'm not quite sure why pd.read_xml() behaves different compared to pd.read_csv() or pd.read_xlsx().
pd.read_xml is not able to read a FileStorage object. The variable passed by request.file[] is a instance of the class: werkzeug.datastructures.FileStorage(stream=None, filename=None, name=None, content_type=None, content_length=None, headers=None).
Via the read function I extracted the file itsself.
filestorage = request.files['file']
file=filestorage.read()
with this passed to pd.read_xml it works fine.
Is there anybody who can explain why _parse_doc() funtion of pd.read_xml() is not able to read FileStotage type?
I have a system to load image files from html but once I select my file and load it from the front end I need a python api to read that image file I select. How can I connect python to my html?
So, to be specific it would be to load an image file with html and have my python api (I use flask) download it and read it to extract data.enter image description here
The html to load the file would be:
<form action="" method="POST" enctype=multipart/form-data>
<div class="col-md-6">
<input type="file" name="file">
<input class="button button-block button-primary" type="submit" value="Upload">
</div>
</form>
The python API to download and read the file would be:
import os
from flask import Flask, render_template, request
from werkzeug import secure_filename
from PIL import Image
# instance of the Flask object
app = Flask(__name__)
# Carpeta de subida
app.config['UPLOAD_FOLDER'] = './Archivos GTiff'
#app.route("/")
#app.route("imgeo/upload-img.html")
def upload_file():
# render the template "upload-img.html"
return render_template('upload-img.html')
#app.route('imgeo/upload-img.html')
def viewer():
"""Show a map to view the GeoJSON returned from a ImGeo endpoint"""
return render_template('imgeo/upload-img.html')
#app.route("imgeo/upload-img.html", methods=['POST'])
def uploader():
if request.method == 'POST':
# we get the file from the "file" input
f = request.files['archivo']
filename = secure_filename(f.filename)
# Save the file in the directory "tiff files".
f.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
# We return a satisfactory answer
return "<h1>File uploaded successfully</h1>"
if __name__ == '__main__':
# Start the application
app.run(debug=True)
You could try to encode your image in base64 with JavaScript, send the result with a POST request, and then (in Python) decode it. Check out these two threads:
How can I convert an image into Base64 string using JavaScript? (JavaScript)
Convert string in base64 to image and save on filesystem (Python)
Here is a similar question too:
Sending an image from html to python script through CGI
I know that there are some similar questions, although none of them has helped me to resolve my issue. The ultimate goal is to have an flask app, which takes an excel file, stores it inside of the azure blob storage, which is then used by my python function app to do some further transformations. What I struggle with, is how to encode/decode this file as I have to use
block_blob_service.create_blob_from_bytes() function.
The only thing that came to my mind was to use Pandas library to read this excel and then tobytes() function. This way, I am able to upload my excel to the blob as a CSV file. However I cannot really convert it to its previous form.
This is how it looks like after opening :
9]�0��j�9p/�j�� �`��/wj1=p/�j�� p�^�.wj2=p/�[...]
Trying to decode it with utf-8 gives me some never ending errors saying that 'utf-8' codec can't decode byte[...] I have tried many different encodings but it always ends up with this message at some byte. Excel contains numericals, strings and dates.
So, to the code:
#getting the file
file = request.files['file']
#reading into pandas df
data = pd.read_excel(file)
df_to_records = data.to_records(index=False)
records_to_bytes = df_to_records.tobytes()
block_blob_service = BlockBlobService(account_name='xxx', account_key="xxx")
block_blob_service.create_blob_from_bytes("test","mydata.csv",records_to_bytes)
Thank you for any advice!
Update: The full code working at my side.
import os
from flask import Flask, request, redirect, url_for
from azure.storage.blob import BlockBlobService
import string, random, requests
app = Flask(__name__, instance_relative_config=True)
account = "your_account name" # Azure account name
key = "account key" # Azure Storage account access key
container = "f22" # Container name
blob_service = BlockBlobService(account_name=account, account_key=key)
#app.route('/', methods=['GET', 'POST'])
def upload_file():
if request.method == 'POST':
file = request.files['file']
file.seek(0)
filename = "test1.csv" # just use a hardcoded filename for test
blob_service.create_blob_from_stream(container, filename, file)
ref = 'http://'+ account + '.blob.core.windows.net/' + container + '/' + filename
return '''
<!doctype html>
<title>File Link</title>
<h1>Uploaded File Link</h1>
<p>''' + ref + '''</p>
<img src="'''+ ref +'''">
'''
return '''
<!doctype html>
<title>Upload new File</title>
<h1>Upload new File</h1>
<form action="" method=post enctype=multipart/form-data>
<p><input type=file name=file>
<input type=submit value=Upload>
</form>
'''
if __name__ == '__main__':
app.run(debug=True)
After running:
After select a .csv file and click upload button, check the .csv file in azure portal:
I think you can take a try with method create_blob_from_stream instead of create_blob_from_bytes.
Here is the sample code:
def upload_file():
if request.method == 'POST':
file = request.files['file']
file.seek(0)
try:
blob_service = BlockBlobService(account_name='xxx', account_key="xxx")
blob_service.create_blob_from_stream(container, filename, file)
except Exception:
print 'Exception=' + Exception
pass
I want to write to file from request.form["file"] but I can't do it.
My contact.html is here.
Client side code is like this...
<form action="contact" method="POST" enctype="multipart/form-data">
<input type="file" name="file">
<input type="submit" value="submit">
</form>
Server side is like this
filestorage = request.files["file"]
print type(_file) #-> <FileStorage: u"__proto.png" ("img/png")>
# I tried below this , but it doesn't work.
f = open("tmp.png","wb")
f.write(filestorage)
I want to write this which is png file to uploaded file somewhere. Do you have any idea?
Thanks in advance.
You have the save() method of the FileStorage object, that lets you save the file content to disk:
file.save('/path/to/your/file')
Flask Documentation: http://werkzeug.pocoo.org/docs/datastructures/#werkzeug.datastructures.FileStorage.save
A useful tutorial: http://flask.pocoo.org/docs/patterns/fileuploads/
First, you have to configure your upload folder
app.config['UPLOAD_FOLDER'] = PATH_TO_UPLOAD_FOLDER
Then to save your file
f = request.files["file"]
f.save(os.path.join(app.config['UPLOAD_FOLDER'], 'tmp.png'))
First, I don't need any functions more than upload a file. No progress bar, no file type, or size check, no multiple files.
What I want is the most simple HTML webpage to handle the upload and save the file with the name I specified.
I tried to use this:
<form action="../cgi-bin/upload.py" method="post" enctype="multipart/form-data">
<input type="file" name="upload" />
<input type="submit" /></form>
In upload.py:
#!/usr/bin/python
import os
import commands
import cgi, cgitb
cgitb.enable()
print "Content-Type: text/html"
print
print 'start!'
form = cgi.FieldStorage()
filedata = form['upload']
But I don't know how to save this in file, like "Beautiful.mp3".
Can any body help?
Though, really, I don't want to use any scripts. I just want the most basic html pages. Python scripts will only exist when there must be some CGI handlers. Flash is not preferred.
The filedata object will wrap a file-like object that can be treated like a regular file. Basically you would do this:
if filedata.file: # field really is an upload
with file("Beautiful.mp3", 'w') as outfile:
outfile.write(filedata.file.read())
Or, you could do just about anything else with it, using read(), readlines() or readline()