I tried to upload blobs to Google App Engine's blobstore using the following HTML form:
<!DOCTYPE html>
<html>
<head>
<meta charset=utf-8>
</head>
<body>
<form id=upload action={{upload_url}} method=post enctype=multipart/form-data>
Name: <input type=text name=name>
Your photo: <input type=file name=image required=required><br><br>
<input type=submit value=submit>
</form>
</body>
</html>
The value of the template variable {{upload_url}} is obtained by upload_url = blobstore.create_upload_url('/upload') on the server side. The post-handling script is as follows:
class Test(ndb.Model):
name = StringProperty()
image = StringProperty()
test = Test()
test.name = self.request.get('name')
image = self.get_uploads('image')[0]
test.image = str(image.key())
test.put()
Usually, the name field will be filled with non-English characters (E.g., Chinese). The above programs works fine on my local SDK. However, the name is incorrectly coded when the program is run on Google App Engine. What's the problem then?
Don't you have to put quotations around the meta tag parameter: <meta charset="UTF-8">? Also, try: <meta http-equiv="content-type" content="text/html; charset=utf-8" />. And, make sure you are saving your template's text document in UTF-8 encoding.
Just found out that this is an old bug for years, see here. There are two solutions:
(1) Add the following statements into app.yaml:
libraries:
- name: webob
version: "1.2.3"
(2) Add the file appengine_config.yaml with the following content:
# -*- coding: utf-8 -*-
from webob import multidict
def from_fieldstorage(cls, fs):
"""Create a dict from a cgi.FieldStorage instance.
See this for more details:
http://code.google.com/p/googleappengine/issues/detail?id=2749
"""
import base64
import quopri
obj = cls()
if fs.list:
# fs.list can be None when there's nothing to parse
for field in fs.list:
if field.filename:
obj.add(field.name, field)
else:
# first, set a common charset to utf-8.
common_charset = 'utf-8'
# second, check Content-Transfer-Encoding and decode
# the value appropriately
field_value = field.value
transfer_encoding = field.headers.get('Content-Transfer-Encoding', None)
if transfer_encoding == 'base64':
field_value = base64.b64decode(field_value)
if transfer_encoding == 'quoted-printable':
field_value = quopri.decodestring(field_value)
if field.type_options.has_key('charset') and field.type_options['charset'] != common_charset:
# decode with a charset specified in each
# multipart, and then encode it again with a
# charset specified in top level FieldStorage
field_value = field_value.decode(field.type_options['charset']).encode(common_charset)
# TODO: Should we take care of field.name here?
obj.add(field.name, field_value)
return obj
multidict.MultiDict.from_fieldstorage = classmethod(from_fieldstorage)
Related
I am trying to create a Python web server that takes user input and makes a request to a third-party API. The user input is obtained via a simple form I created.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="ie=edge">
<title>Test Document</title>
</head>
<body>
<form action="server.py" method="post">
<label for="firstname">First Name</label>
<input type="text" name="firstname" id="firstname"><br>
<label for="lastname">Last Name</label>
<input type="text" name="lastname" id="lastname"><br>
<label for="url">URL</label>
<input type="text" name="url" id="application-url"><br>
<input type="submit" value="Submit">
</form>
</body>
</html>
I named this file index.html. My server.py file looks like this:
import cgi
from http.server import HTTPServer, CGIHTTPRequestHandler
class TestServerHandler(CGIHTTPRequestHandler):
def do_POST(self):
if self.path == '/':
self.path = './index.html'
try:
form = cgi.FieldStorage()
firstname = form.getvalue('firstname')
lastname = form.getvalue('lastname')
url = form.getvalue('url')
print(firstname + ' ' + lastname + ' ' + url)
output=firstname + lastname + url
except:
self.send_error(404, 'Bad request submitted.')
self.end_headers()
self.wfile.write(bytes(output, 'utf-8'))
test_server = HTTPServer(('localhost', 8080), TestServerHandler)
test_server.serve_forever()
I then run the server.py file on my terminal and go to the http://localhost:8080/ url in my browser. However, when I submit the form I get an error in the browser. The terminal output shows an error that says 'cannot concatenate a 'str' with a 'nonetype'. Based on this error, the form values aren't being passed to this page. Either that or the HTTP server is passing them as query parameters. In any case, would I be able to use the cgi.FieldStorage class inside my HTTP server to access the form field values?
After about a week of looking for a solution to this I found that the http.server Python module is not really compatible with the cgi module. The cgi module is used inside of Python scripts that are passed form values from some HTML document on the web server (i.e. a form on the index.html page of a web server with the "action" attribute set to the Python script in question). However, in order for the cgi module to have access to the form values passed to that script (via the cgi.FieldStorage() call), the script must be running inside a web server. The problem with my code example above is that the server.py script I created IS a web server itself. Specifically, it creates an instance of HTTPServer using a custom request handler class I create (TestServerHandler). My custom class subclasses the CGIHTTPRequestHandler class contained in the http.server module. This class contains the Do_POST method. When implementing this method, any form data passed to the Python script is contained inside the self.rfile instance variable. To access this variable and get to the form data I wrote code similar to this.
content_length = int(self.headers['Content-Length'])
data_input = bytes.decode(self.rfile.read(content_length))
After you have the form_data stored in the data_input variable, you can then use a URL parser to access the values you need from the form.
The FieldStorage class has a constructor defined on this page. It's important parameters are fp, headers, and environ. OP is right in that cgi.FieldStorage() is usually called without parameters in a CGI script, in which case the parameters are filled in for you. However, we can create a FieldStorage object with variables that CGIHTTPRequestHandler provides for us:
form = cgi.FieldStorage(
fp=self.rfile,
headers=self.headers,
environ={
'REQUEST_METHOD': 'POST',
'CONTENT_TYPE': self.headers['Content-Type'],
}
I have just installed FusionCharts Suite XT v3.13.4 to use in my (Python) Django application. I have done the Getting Started Guide (https://www.fusioncharts.com/dev/getting-started/django/your-first-chart-using-django#installation-2), but I can't seem to get it to work. I don't get an error, but my page remains completely empty. I don't know what I did wrong, I followed the tutorial exactly.
dash.html
<!-- Filename: app_name/templates/index.html -->
<!DOCTYPE html>
<html>
<head>
<title>FC-python wrapper</title>
{% load static %}
<script type="text/javascript" src="{% static "https://cdn.fusioncharts.com/fusioncharts/latest/fusioncharts.js" %}"></script>
<script type="text/javascript" src="{% static "https://cdn.fusioncharts.com/fusioncharts/latest/themes/fusioncharts.theme.fusion.js" %}"></script>
</head>
<body>
<div id="myFirstchart-container">{{ output|safe }}</div>
</body>
</html>
views.py
from django.shortcuts import render
from django.http import HttpResponse
from collections import OrderedDict
# Include the `fusioncharts.py` file that contains functions to embed the charts.
#from fusioncharts import FusionCharts
from vsdk.dashboard.fusioncharts import FusionCharts
def myFirstChart(request):
#Chart data is passed to the `dataSource` parameter, like a dictionary in the form of key-value pairs.
dataSource = OrderedDict()
# The `chartConfig` dict contains key-value pairs of data for chart attribute
chartConfig = OrderedDict()
chartConfig['caption'] = 'Countries With Most Oil Reserves [2017-18]'
chartConfig['subCaption'] = 'In MMbbl = One Million barrels'
chartConfig['xAxisName'] = 'Country'
chartConfig['yAxisName'] = 'Reserves (MMbbl)'
chartConfig['numberSuffix'] = 'K'
chartConfig['theme'] = 'fusion'
# The `chartData` dict contains key-value pairs of data
chartData = OrderedDict()
chartData['Venezuela'] = 290
chartData['Saudi'] = 260
chartData['Canada'] = 180
chartData['Iran'] = 140
chartData['Russia'] = 115
chartData['UAE'] = 100
chartData['US'] = 30
chartData['China'] = 30
dataSource['chart'] = chartConfig
dataSource['data'] = []
# Convert the data in the `chartData`array into a format that can be consumed by FusionCharts.
#The data for the chart should be in an array wherein each element of the array
#is a JSON object# having the `label` and `value` as keys.
#Iterate through the data in `chartData` and insert into the `dataSource['data']` list.
for key, value in chartData.items():
data = {}
data['label'] = key
data['value'] = value
dataSource['data'].append(data)
# Create an object for the column 2D chart using the FusionCharts class constructor
# The chart data is passed to the `dataSource` parameter.
column2D = FusionCharts("column2d", "ex1" , "600", "400", "chart-1", "json", dataSource)
return render(request, 'dash.html', {'output' : column2D.render(), 'chartTitle': 'Simple Chart Using Array'})
urls.py
from django.shortcuts import render
from django.urls import path
from vsdk.dashboard.fusioncharts import FusionCharts
from . import views
from django.conf.urls import url, include
urlpatterns = [
url(r'^$', views.myFirstChart, name = 'demo'),
]
The instructions in the "Getting Started Guide" are a bit confusion and there are some errors. Here is my working version of the Fusionchart example.
Fusionchart example
You can use FusionCharts to render charts in html. I have based the example on the Getting Started Guide and tweaked here and there to make it work. To replicate just do the following:
copy my github code, this will create an new project directory fusionchart_example.
git clone https://github.com/bvermeulen/fusionchart_example
The tree structure should look like:
go to this folder and create a virtual environment for Django, note I work with Python 3.6.8, but likely other python 3.6 or 3.7 would work ok as well.
python -m venv ./venv
activate the environment (Linux)
source ./venv/bin/activate
(or Windows)
./venv/scripts/activate
with the virtual environment enabled install Django
pip install django==2.2.3
You can now run the app
python manage.py runserver
and view the result in your browser at 127.0.0.1:8000 that should look like:
You can review the source code when you have cloned my github, especially settings.py, but I give urls.py, views.py and chart.html below as first reference.
urls.py:
from django.urls import path
from render_graph import views
urlpatterns = [
path('', views.chart, name='chart'),
]
views.py:
from django.shortcuts import render
from django.http import HttpResponse
from collections import OrderedDict
# Include the `fusioncharts.py` file that contains functions to embed the charts.
from fusioncharts import FusionCharts
from pprint import pprint
def chart(request):
#Chart data is passed to the `dataSource` parameter, like a dictionary in the form of key-value pairs.
dataSource = OrderedDict()
# The `chartConfig` dict contains key-value pairs of data for chart attribute
chartConfig = OrderedDict()
chartConfig["caption"] = "Countries With Most Oil Reserves [2017-18]"
chartConfig["subCaption"] = "In MMbbl = One Million barrels"
chartConfig["xAxisName"] = "Country"
chartConfig["yAxisName"] = "Reserves (MMbbl)"
chartConfig["numberSuffix"] = "K"
chartConfig["theme"] = "fusion"
# The `chartData` dict contains key-value pairs of data
chartData = OrderedDict()
chartData["Venezuela"] = 290
chartData["Saudi"] = 260
chartData["Canada"] = 180
chartData["Iran"] = 140
chartData["Russia"] = 115
chartData["UAE"] = 100
chartData["US"] = 30
chartData["China"] = 30
dataSource["chart"] = chartConfig
dataSource["data"] = []
# Convert the data in the `chartData`array into a format that can be consumed by FusionCharts.
#The data for the chart should be in an array wherein each element of the array
#is a JSON object# having the `label` and `value` as keys.
#Iterate through the data in `chartData` and insert into the `dataSource['data']` list.
for key, value in chartData.items():
dataSource["data"].append({'label':key, 'value': value})
# print the datasource to see what will be rendered
pprint(dataSource)
# Create an object for the column 2D chart using the FusionCharts class constructor
# The chart data is passed to the `dataSource` parameter.
column2D = FusionCharts("column2d", "Oil_Reserves", "600", "400", "Oil_Reserves-container", "json", dataSource)
context = {'output': column2D.render(), }
return render(request, 'chart.html', context)
chart.html:
<!DOCTYPE html>
<html>
<head>
<title>Oil Reserves</title>
{% load static %}
<script type="text/javascript" src="{% static 'fusioncharts/types/fusioncharts.js' %}"></script>
<script type="text/javascript" src="{% static 'fusioncharts/themes/fusioncharts.theme.fusion.js' %}"></script>
<link rel="icon" href="data:,">
</head>
<body>
<div id="Oil_Reserves-container">{{ output|safe }}</div>
</body>
</html>
Fusioncharts says this is a trial version, so better check if any costs are involved. Let me know if you need any more info. Good luck...
Bruno Vermeulen
bruno.vermeulen#hotmail.com
8 July 2019
I have a script which takes uploaded data, munges it together, turns it into a plot (using Bokeh) and then exports it to a directory as JSON.
At some point in the future, a user can hit the right URL and the appropriate plot should be displayed to the user as part of the HTML template.
I can generate the plot. I can save it as JSON. I can get the URL to retrieve it as JSON, but I cannot get the JSON plot to render within the template.
I've had a dig around the Bokeh documentation and examples, but they all seem to use a flask app to serve the pages.
I think I'm on the right track, using views.py to find and return JSON as part of a render() response, and then have Bokeh.embed.embed_items() do the work in the template to make it look right, but it's not working out - everything but the plot is displayed.
1) Create the plot and puts it in the directory for later use (app/results/1)
create plot.py
import os
import json
from django.conf import settings
from bokeh.embed import json_item
from bokeh.plotting import figure
x=[1,2,3,4,5]
y=[0,-1,-2,3,4]
p=figure(title="test_example")
p.line(x, y)
#json_export = json_item(p, "result")
json_export = json_item(p)
with open(os.path.join(settings.RESULTS_DIR,"1", "test.json"), 'w') as fp:
fp.write(json.dumps(json_export))
2) Set up the url
urls.py
urlpatterns=[
path('result/<int:pk>', views.resultdetailview, name='result-detail'),
]
3) Take the request, use the pk to find the plot json and render it all in the appropriate template.
views.py
def resultdetailview(request, pk):
results=str(pk)
with open(os.path.join(settings.RESULTS_DIR, results, "test.json"), 'r') as fp:
#data=json.load(fp)
data=json.loads(fp.read())
#context={'result':data}
#return render(request, 'app/sandbox_detail.html', context)
return render(request=request,
context={'json_object':data, 'resources':CDN.render()})
NB: If I instead use return JsonResponse(data, safe=False) then the url returns the json successfully ...
I think therefore that the issue is in the template.
4) Show the wonderous plot to the user
sandbox_detail.html
<header>
<link rel="stylesheet" href="http://cdn.bokeh.org./bokeh/release/bokeh-0.11.1.min.css" type="text/css" >
<script type="text/javascript" src="http://cdn.bokeh.org./bokeh/release/bokeh-0.11.1.min.js"> </script>
</header>
<h1> Title: Test </h1>
<div>Test</div>
<body>
<div id="result"></div>
<script>
Bokeh.embed.embed_item({{json_object}}, "result");
</script>
</body>
This template renders everything but the 'result' div.
What have I missed?
This is what I see so far:
FIRST: You are mixing 2 methods for injecting plot json data into the page.
According to documentation you can do it using either of these two methods:
1) specify the div directly:
Python: json_data = json.dumps(json_item(p, "myplot"))
JavaScript: Bokeh.embed.embed_item(item);
2) specify the div in embed_item function:
Python: json_data = json.dumps(json_item(p))
JavaScript: Bokeh.embed.embed_item(item, "myplot");
But not both of them at the same time. Could this be the problem?
SECOND: Preferably don't insert Bokeh resources by hand: rather use CDN.render() or INLINE.render() to automatically include all that your script needs:
import json
from bokeh.resources import CDN
return render(request = request,
template_name = 'app/sandbox_detail.html',
context = { json_object = json.loads(json_string),
resources = CDN.render() } )
sandbox_detail.html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
{{ resources }}
</head>
<body>
<div id="result"></div>
<script>
Bokeh.embed.embed_item({{ json_object }}, "result");
</script>
</body>
</html>
THIRD: Make sure what you embed in the page is json object not a string (see variable naming above)
It helps when you debug your rendered template in your browser's debug tool.
I tried a very similar approach and found one large flaw: My browser noted that it could not find the None object. The reason here is that python stores the empty value as None, while JavaScript expects a null object.
The solution? Python already translates None to null, when you run json.dumps. To keep it that way, read the json string as a string. So instead of your data=json.loads(fp.read()) use data=fp.read().
I'm using Google Books API to integrate with an E-commerce site that I'm developing. The idea is to let the users search for books using a search bar, and then calling the Google API to output the number of books corresponding to the search keyword.
However, I get a 403 Forbidden error after I click submit after entering my query in the form. This is strange, because this never happened when I was testing my application on the localhost. Here's the code for my application:
main.py
class SearchHandler(Handler):
def get(self):
self.render("search.html")
def post(self):
keey = self.request.get('keey')
finaal = "https://www.googleapis.com/books/v1/volumes?q=" + keey + "&key=MY_APP_KEY"
f = urllib2.urlopen(finaal).read()
self.render("jsony.html", finaal = finaal)
app = webapp2.WSGIApplication(('/search', SearchHandler)], debug=True)
search.html
<html>
<head>
<title>Web Mining</title>
</head>
<body>
<form method = "post">
Book Name:<input type = "text" name = "keey">
<input type = "submit">
</form>
</body>
</html>
jsony.html
<html>
<head>
<title>Web Mining</title>
</head>
<body>
<form method = "post">
{{finaal}}
</form>
</body>
Now, the jsony.html is still incomplete. All I'm doing now is displaying the URL which contains the outputted json in it's raw, unprocessed form.
What seems to be causing this 403 error to arise after I deploy my application ?
EDIT 1:
The problem resolves when I remove the following line from my main python file:
f = urllib2.urlopen(finaal).read()
However, I would be needing my API's URL in order to extract data from its source code. What's happening ?
Try adding &country=US to the URL query string.
I would like to somehow instrument a mako.lookup.TemplateLookup such that it applies certain preprocessors only for certain file extensions.
Specifically, I have a haml.preprocessor that I would like to apply to all templates whose file name ends with .haml.
Thanks!
You should be able to customize TemplateLookup to get the behavior you want.
customlookup.py
from mako.lookup import TemplateLookup
import haml
class Lookup(TemplateLookup):
def get_template(self, uri):
if uri.rsplit('.')[1] == 'haml':
# change preprocessor used for this template
default = self.template_args['preprocessor']
self.template_args['preprocessor'] = haml.preprocessor
template = super(Lookup, self).get_template(uri)
# change it back
self.template_args['preprocessor'] = default
else:
template = super(Lookup, self).get_template(uri)
return template
lookup = Lookup(['.'])
print lookup.get_template('index.haml').render()
index.haml
<%inherit file="base.html"/>
<%block name="content">
%h1 Hello
</%block>
base.html
<html>
<body>
<%block name="content"/>
</body>
</html>