I'm trying to host a website off a Beaglebone Black using lighttpd and web.py. Any post request creates Error 413 - Request Entity Too Large. I am very new to web development, so I apologize if I've used the wrong terminology. To be clear, I am not attempting to upload any file of any kind. I created a simple page, test, to illustrate the error. My code:
First, Monitor.py:
#!/usr/bin/env python
import web
import Model
from Account import Account
from Forgot import Forgot
from Login import Login
from Reset import Reset
from Subscribe import Subscribe
from Success import Success
from Status import Status
from Test import Test
urls = ('/', 'Status',
'/test', 'Test',
'/account', 'Account',
'/forgot', 'Forgot',
'/login', 'Login',
'/reset/(\d+)', 'Reset',
'/success', 'Success',
'/subscribe', 'Subscribe',
)
render = web.template.render('templates')
app = web.application(urls, globals())
web.config.debug = False
session = Model.create_session(app)
if __name__ == '__main__':
app.run()
Now, Test.py:
#!/usr/bin/env python
import web
import Model
render = web.template.render('templates')
class Test:
def GET(self):
return render.test()
def POST(self):
i = web.input()
raise web.seeother('/test')
Now, the html file, located in the templates folder:
$def with()
<html>
<head>
<link rel="stylesheet" href="static/stylesheet.css" type="text/css" media="screen" charset="utf-8"/>
<title>Account Page</title>
</head>
<body>
div>
<form action="/test" method="POST">
<table class="std-element">
<tr>
<th>
<input type="text" name="example_text">
</th>
</tr>
</table>
</form>
</div>
</body>
</html>
The relevant portion of the model file:
session = None
def create_session(app):
global session
session = web.session.Session(app, web.session.DiskStore('sessions'), initializer={'username':default_username})
session['username'] = default_username
return session
def get_session():
global session
return session
And, finally, the lighttpd.config file:
server.modules = (
"mod_access",
"mod_accesslog",
"mod_alias",
"mod_compress",
)
server.document-root = "/var/www/"
server.upload-dirs = ( "/var/cache/lighttpd/uploads" )
server.errorlog = "/var/log/lighttpd/error.log"
server.pid-file = "/var/run/lighttpd.pid"
server.username = "www-data"
server.groupname = "www-data"
server.port = 80
server.breakagelog = "/var/log/lighttpd/breakage.log"
server.max-request-size=100000000
server.uploads-dirs=("/mnt")
server.network-backend="write"
## Use ipv6 if available
#include_shell "/usr/share/lighttpd/use-ipv6.pl"
compress.cache-dir = "/var/cache/lighttpd/compress/"
compress.filetype = ( "application/x-javascript", "text/css", "text/html", "text/plain" )
include_shell "/usr/share/lighttpd/create-mime.assign.pl"
include_shell "/usr/share/lighttpd/include-conf-enabled.pl"
server.max-request-size=100000000 is way too large. It is measured in KB. Internally, lighttpd left-shifts that number 10 bits, which, in your version of lighttpd, results in any POST being compared with 0 and rejected because it is too large. This has been fixed in lighttpd 1.4.40, but you should probably reduce that limit anyway. Having a 100 TB (!) request size limit is probably not going to protect your Beaglebone. Try this: server.max-request-size=1024 will set a 1 MB limit on POST requests. (Again, that config directive is in KB)
Related
I am able to create signed URLs and just need to know what to do with them after they are created.
There are several examples using Javascript to upload via a signed URL, but I cannot find any in Python. I am trying to use signed URLs as a workaround for the 32 MB limit imposed by Google App Engine for my Flask application.
Here is my python app.py script (not full functionality of my app here, just trying to upload to my bucket successfully):
from flask import Flask, request, render_template
from google.cloud import storage
import pandas as pd
import os
import gcsfs
bucket_name = "my-bucket"
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = '/path/to/file.json'
app = Flask(__name__)
def upload_blob(bucket_name, source_file_name, destination_blob_name):
storage_client = storage.Client()
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(destination_blob_name)
blob.upload_from_file(source_file_name)
print("success")
#app.route('/')
def homepage():
return render_template('home.html')
#app.route('/', methods = ['GET', 'POST'])
def upload_file():
if request.method == 'POST':
file1 = request.files['file1']
file2 = request.files['file2']
upload_blob(bucket_name, file1, 'file-1')
upload_blob(bucket_name, file2, 'file-2')
df = pd.read_csv('gs://' + bucket_name + '/' + 'file-1')
print(df.shape)
return "done"
if __name__ == "__main__":
app.run(debug=True)
Here is the function I am using to create the signed URL:
def generate_upload_signed_url_v4(bucket_name, blob_name):
storage_client = storage.Client()
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(blob_name)
url = blob.generate_signed_url(
version="v4",
# This URL is valid for 15 minutes
expiration=datetime.timedelta(minutes=15),
# Allow GET requests using this URL.
method="PUT",
content_type="application/octet-stream",
)
print(url)
return url
generate_upload_signed_url_v4(bucket_name, 'file.csv')
And below is my home.html:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>test upload</title>
</head>
<body>
<h3> test upload </h3>
<form method="POST" action="/" enctype="multipart/form-data">
<p>Upload file1 below</p>
<input type="file" name="file1">
<br>
<br>
<p>Upload file2 below</p>
<input type="file" name="file2">
<br>
<br>
<input type="submit" value="upload">
</form>
</body>
</html>
Based on what I researched here is my CORS configuration for the bucket I am trying to upload to:
[
{"maxAgeSeconds": 3600,
"method": ["GET", "PUT", "POST"],
"origin": ["https://my-app.uc.r.appspot.com", "http://local.machine.XXXX/"],
"responseHeader": ["Content-Type"]}
]
Does the signed URL that is generated go in the html form? Does it need to go into my upload_file function?
Finally, when I paste the signed URL into my browser it shows this error:
<Error>
<Code>MalformedSecurityHeader</Code>
<Message>Your request has a malformed header.</Message>
<ParameterName>content-type</ParameterName>
<Details>Header was included in signedheaders, but not in the request.</Details>
</Error>
This is my first SO question so I apologize if it is poorly constructed. I am super lost and new to GCP. I have searched SO for a while now, and not found a use-case with Python/Flask where I can see how the signed URL is incorporated into the file upload process.
Again, I am building a webapp on Google App Engine flex, and need signed URLs to workaround the 32 MB file upload restriction.
UPDATE
I got the signed URL component figured out after realizing I needed to simply make a request to the signed URL.
Below is my new script that is loaded in App Engine (imports and "if name = main..." removed for snippet below).
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = '/path/to/file.json'
EXPIRATION = datetime.timedelta(minutes=15)
FILE_TYPE = 'text/csv'
BUCKET = 'my-bucket'
def upload_via_signed(bucket_name, blob_name, filename, expiration, file_type):
bucket = storage.Client().get_bucket(bucket_name)
blob = bucket.blob(blob_name)
signed_url = blob.generate_signed_url(method='PUT', expiration=expiration, content_type=file_type)
requests.put(signed_url, open(filename.filename, 'rb'), headers={'Content-Type': file_type})
app = Flask(__name__)
app.config['UPLOAD_FOLDER'] = '/tmp'
#app.route('/')
def homepage():
return render_template('home.html')
#app.route('/', methods = ['GET', 'POST'])
def upload_file():
if request.method == 'POST':
diag = request.files['file']
filename_1 = secure_filename(diag.filename)
filepath_1 = os.path.join(app.config['UPLOAD_FOLDER'], filename_1)
diag.save(filepath_1)
person = request.files['person']
filename_2 = secure_filename(person.filename)
filepath_2 = os.path.join(app.config['UPLOAD_FOLDER'], filename_2)
person.save(filepath_2)
upload_via_signed(BUCKET, 'diag.csv', diag, EXPIRATION, FILE_TYPE)
upload_via_signed(BUCKET, 'person.csv', person, EXPIRATION, FILE_TYPE)
df_diag = pd.read_csv('gs://' + BUCKET + '/' + 'diag.csv')
print(df_diag.shape)
return "done"
The code above is still throwing the 413 entity too large error. I think it's because I've got the 'POST' going through App Engine even though I am creating signed URLs. How do I need to re-arrange/what am I doing wrong? How does the code need to be structured to have the user upload directly to Google Cloud Storage via the signed URLs and avoid triggering the 413 entity too large error?
Once you have generated the signed url on the server, you just need to send it back to the client and use it to upload your files. you can for example send file data using normal fetch put request or as I prefer always using axios:
await axios.put(url, file);
the url here is the signed url. you may want to send your files as formData
I am using flask (python) to host a interface that allows the user to upload an image, and use a slider to change the value of a variable in a function that reduces noise in an image.
The slider works but the problem is I have to reload the page every time that I want to see the change in value on the updated image.
how can I add a slider that will update the image in real time so I won't have to continuously reload the page to see the changes?
(If the slider is at 0 I want it to to look how the image did when it first uploaded)
I'm doing some searching and it looks like I would use jquery or something but I don't know how to implement it
thanks for reading
app.py:
import os
from flask import Flask, render_template, request, send_from_directory,url_for, session, redirect
import cv2
import shutil
app = Flask(__name__)
APP_ROOT = os.path.dirname(os.path.abspath(__file__))
app.config['UPLOAD_FOLDER'] = os.path.join(APP_ROOT, 'images')
app.config['SEND_FILE_MAX_AGE_DEFAULT'] = 0
#app.route("/")
def index():
session.clear()
return render_template("upload.html")
#app.route('/images/<filename>')
def uploadfile(filename):
return send_from_directory(app.config['UPLOAD_FOLDER'],filename)
#app.route('/home')
def home():
return render_template("completed.html", imgSrc="images/" + session.get('imgSrc') , message=session.get('message'))
#app.route("/upload" , methods = ['POST'])
def upload():
target = os.path.join(APP_ROOT, 'images')
if request.method == 'POST':
if not os.path.isdir(target):
os.mkdir(target)
for file in request.files.getlist("file"):
filename = file.filename
destination = "/".join((target, filename))
file.save(destination)
filename = destination
org = open(filename, 'rb')
base = os.path.basename(filename)
dir = os.path.dirname(filename)
filename_cp = os.path.splitext(base)[0]
filename_cp = "cp_"+filename_cp+os.path.splitext(base)[1]
destination2 = dir+"/"+filename_cp
file.save(destination2)
cpy = open (destination2, 'wb')
shutil.copyfileobj(org, cpy)
session['image'] = filename
session['filename'] = filename
session['imgSrc'] = os.path.basename(destination)
session['cimgsrc'] = os.path.basename(destination2)
session['cpimg'] = destination2
print("session", session)
return render_template("completed.html",imgSrc="images/"+session.get('imgSrc'))
#app.route("/imgp/nr", methods=['post'])
def nr():
print(session)
img = cv2.imread(session.get('cpimg'), 0)
#threshold = 40
threshold = float(request.form['slider'])
cv2.threshold(img, threshold, 255, cv2.THRESH_BINARY, img)
print (session['cpimg'])
cv2.imwrite(session.get('cpimg'),img)
session['message'] = "NR is done!"
session['imgSrc'] = os.path.basename(session['cpimg'])
return redirect(url_for('home', op='nr'))
if __name__ =="__main__":
app.secret_key = "abcdefghijklmnopqrstuvwxyz"
app.run(port = 4555, debug = True)
upload.html:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Upload</title>
</head>
<form id ="upload-form" action="{{ url_for ('upload') }}" method = "POST" enctype="multipart/form-data">
<input type="file" name="file" accept = image/* multiple>
<p>Drag your files here or click in this area.</p>
<button type ="submit" value="Upload"> Upload</button>
</form>
<body>
</body>
</html>
completed.hmtl (where the slider and uploaded image is):
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Title</title>
</head>
<body>
<h1> file uploaded</h1>
<img src="{{imgSrc}}" style="height:200px;width:300px;"/>
<h2>{{message}}</h2>
<form action ="imgp/nr" method = "post">
<input type="range" min={{min}} max={{max}} value={{max}} class="slider" id="myRange" name="slider">
<input type="submit" value="Apply ga">
</form>
</body>
</html>
The short answer is you will need to do this with Javascript/jQuery. There's a similar question here covers two ways to do it:
Either load the (base64 converted) image via ajax; or
Change the src of the image on the page using jquery, to have it reload.
You will likely need to change a few parts of your code to get this all working, but it may be something along the lines of:
app.py:
Change your nr() function to GET, and return a base64 encoded image, see here for example (upload_file() function)
completed.html:
Add an ajax (the following is jquery) call to /imgp/nr that changes the display of the image on the page, when the slider is changed. For example:
...
<img id="yourImage" style="height:200px;width:300px;">
...
<script type="text/javascript">
$("#myRange").change(function(){
var sliderVal = $(this).val();
$.ajax({
medthod: 'POST',
url:'/imgp/nr/',
data: JSON.stringify({slider: sliderVal}),
success: function(data){
$("#yourImage").attr('src', 'data:image/png;base64, ' + data);
}
});
});
</script>
This code may need some fixing, I don't have much time to make it perfect sorry. Hopefully you get the idea!
#In my views.py file
pi1 = None
pis1 = None
def my_func():
#Essentially this function sets a random integer to pi1 and pis1
global pi1, pis1
pi1 = randint(0,9)
pis1 = randint(0,9)
return
def index(request):
my_func()
context = {
"pi1" : pi1,
"pis1" : pis1,
}
return render(request, "index.html", context)
#In the index.html file
<h1>{{ pi1 }}</h1>
<h1>{{ pis1 }}</h1>
I've removed a lot of my code for simplicity, but this is the gist of it. Despite the code that I've posted for my_func, it is a time consuming function that causes index.html to load for awhile when it is accessed. How would I run my_func in the backround using celery and redis so that index.html loads more quickly?
I've read the celery documentation, but I am still having trouble setting up celery and redis. Thank you.
As is said previous, you might not need celery. Here's an example derived from case 2 of this: https://zapier.com/blog/async-celery-example-why-and-how/. It's fully working for me:
from time import sleep
import json
from django.http import HttpResponse
from django.shortcuts import render
def main_view(request):
return render(request, 'index.html')
def ajax_view(request):
sleep(10) #This is whatever work you need
pi1 = "This is pi1" #I just made pi1/pis1 random values
pis1 = "This is pis1"
context = {
"pi1" : pi1,
"pis1" : pis1,
}
data = json.dumps(context)
return HttpResponse(data, content_type='application/json')
My index.html contains:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Main View</title>
<script src="//code.jquery.com/jquery-1.11.3.min.js"></script>
<script>
$(document).ready(function(){
$.ajax({
url: "/test_ajax/",
}).done(function( data) {
$("#pi1").text(data.pi1);
$("#pis1").text(data.pis1);
});
});
</script>
</head>
<body>
<h1 id = "pi1">Loading</h1>
<h1 id = "pis1">Loading</h1>
</body>
</html>
And my urls.py contains:
from django.conf.urls import include, url
from django.contrib import admin
from testDjango.test import main_view, ajax_view
urlpatterns = [
url(r'^admin/', include(admin.site.urls)),
url(r'^test/', main_view),
url(r'^test_ajax/', ajax_view)
]
What happens when I visit localhost:8000/test/ is that I instantly see:
After about 10 seconds, I then see:
The idea is that you return your page instantly and use jquery to fetch the result of the operation whenever that's finished and update your page accordingly. You can add more things like progress bars/loading image etc. For your example, you can do the processing for pi1 and pis in the background and load it into the HTML after that's finished.
You don't need celery here. You can load these values on page with AJAX request. You should create a separate view that will calculate this values and after the index.html is loaded call it with javascript.
I want to use a RecaptchaField() in my WTForms, such as:
class MyForm(Form):
name = StringField('Your name', [InputRequired(message='Please enter your name!')])
recaptcha = RecaptchaField() # RecaptchaField as provided by Flask-WTF
submit = SubmitField('Send')
Since I am using CherryPy, I am not sure whether or not I should use Flask-WTF, because Flask is a whole framework itself. I am wondering if I can use the Recaptcha functionality of Flask-WTF within my CherryPy solution. I tried the following:
from wtforms import StringField
from wtforms.validators import InputReqired
from flask.ext.wtf import Form
from flask.ext.wtf.recaptcha import RecaptchaField
# ...
form = MyForm() # Somewhere in my code
as seen in this Example here. I get the following Exception:
RuntimeError: working outside of application context
It means I have to properly set up a Flask app considering the right context... This is where I am starting to wonder if I am doing the right approach. Is there no other way than set up a separate Flask app inside my CherryPy app??
My answer is mostly relevant to CherryPy and reCaptcha parts of the question. In my opinion, usage of WTForms and other similar libraries leads to a design problem when, speaking in terms of MVC-like design, you scatter the view and the controller into your own code and WTForms code/configuration. Things are simple when you manage one thing in a single place. Thus I suggest to use template engine like Jinja2 for the view (you can create a macro for a repetitive form element) and use input validation library like voluptuous in the controller (you can use same schema for form and API validation).
If you can't avoid WTForms, just grab validateRecaptcha and turn it into WTForms custom validator.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import urllib
import json
import cherrypy
import voluptuous as volu
config = {
'global' : {
'server.socket_host' : '127.0.0.1',
'server.socket_port' : 8080,
'server.thread_pool' : 8
},
'/' : {
'recaptcha' : {
# By default, all keys work on localhost
'siteKey' : '6LeYIbsSAAAAACRPIllxA7wvXjIE411PfdB2gt2J',
'secret' : '6LeYIbsSAAAAAJezaIq3Ft_hSTo0YtyeFG-JgRtu'
}
}
}
def validateRecaptcha(value):
'''https://developers.google.com/recaptcha/docs/verify'''
if 'g-recaptcha-response' not in cherrypy.request.params:
raise volu.Invalid('Recaptcha response is missing')
payload = urllib.urlencode({
'secret' : cherrypy.request.config['recaptcha']['secret'],
'remoteip' : cherrypy.request.headers['remote-addr'],
'response' : cherrypy.request.params['g-recaptcha-response']
})
url = 'https://www.google.com/recaptcha/api/siteverify'
response = json.load(urllib.urlopen(url, payload))
if not response['success']:
cherrypy.log(str(response))
raise volu.Invalid(response['error-codes'])
class App:
#cherrypy.expose
def index(self, **kwargs):
form = dict(form = {'value': ''}, errors = '')
if cherrypy.request.method == 'POST':
schema = volu.Schema({
'value' : volu.All(unicode, volu.Length(min = 8, max = 16)),
'g-recaptcha-response' : validateRecaptcha,
}, required = True, extra = True)
try:
kwargs = schema(kwargs)
except volu.MultipleInvalid as ex:
form = dict(form = kwargs, errors = {e.path[0] for e in ex.errors})
else:
raise cherrypy.HTTPRedirect('#success')
return '''<!DOCTYPE html>
<html>
<head>
<title>reCAPTCHA demo</title>
<script src="https://www.google.com/recaptcha/api.js" type="text/javascript"></script>
</head>
<body>
<form action="/" method="POST">
<div style='border: 1px red solid'>{errors}</div>
<div>Name</div>
<input type="text" name="value" value="{form[value]}"/>
<br/>
<div class="g-recaptcha" data-sitekey="{0}"></div>
<br/>
<input type="submit" value="Submit"/>
</form>
</body>
</html>
'''.format(cherrypy.request.config['recaptcha']['siteKey'], **form)
if __name__ == '__main__':
cherrypy.quickstart(App(), '/', config)
i have a webservice that takes data from mongodb and returns it over a local web service.
I want to implement a query based control over here that would allow filtering the webservice data from a GUI.
Currently my html file is using the code:
<!DOCTYPE html>
<html lang="en">
<body>
<h1>Welcome to App</h1>
<h2>Enter your username</h2>
<form action="." method="POST">
<input type="text" name="text">
<input type="submit" name="my-form" value="Submit">
</form>
</body>
</html>
and my app.py which is running on flask looks like:
from flask import Flask
from flask import render_template
from flask import request
from pymongo import Connection
import json
from bson import json_util
from bson.json_util import dumps
app = Flask(__name__)
MONGODB_HOST = '172.16.1.95'
MONGODB_PORT = 27017
DBS_NAME = 'AllTables'
COLLECTION_NAME = 'SubjectViews'
#app.route('/')
def my_form():
return render_template("index.html")
#app.route('/testtag', methods=['POST'])
def my_form():
text = request.form['text']
processed_text = " + text + "
return processed_text
def donorschoose_projects():
connection = Connection(MONGODB_HOST, MONGODB_PORT)
collection = connection[DBS_NAME][COLLECTION_NAME]
projects = collection.find({"tags":"processed_text"})
json_projects = []
for project in projects:
json_projects.append(project)
json_projects = json.dumps(json_projects, default=json_util.default)
connection.disconnect()
return json_projects
if __name__ == "__main__":
app.run(host='0.0.0.0',port=5001,debug=True)
What i am trying to do is to get a input from the user in index.html, pass it into the collection.find parameter in app.py so that on my webservice URL i will get data filtered for that particular user. Unfortunately i am running into errors.
Any help would be much appreciated.
Thank you,
Anmol
EDIT: No luck so far!