Develop Reference

How can I write a route to receive Content Security Policy report with Flask without getting a 400 Bad Request error (flask_wtf.csrf.CSRFError)?

TL;DR: Apologies for the long post. In a nutshell I am trying to debug a CSP report-uri. If I am missing critical information please let me know.
CSP implementation: Flask-Talisman
The attribute that needs to be set: content_security_policy_report_uri
There does not seem to be a lot of information out there on how to capture this report
I can't find anything specific in the Flask-Talisman documentation
As Flask-Talisman only sets headers, including the report-uri, I imagine this is outside the scope of the extension anyway
The route
All resources I've found have roughly the same function:
https://www.merixstudio.com/blog/content-security-policy-flask-and-django-part-2/
http://csplite.com/csp260/
https://github.com/GoogleCloudPlatform/flask-talisman/issues/21
The only really detailed explanation I've found for this route is below (it is not related to Flask-Talisman however)
From https://www.merixstudio.com/blog/content-security-policy-flask-and-django-part-2/ (This is what I am currently using)
# app/routes/report.py
import json
import pprint
from flask import request, make_response, Blueprint
...
bp = Blueprint("report", __name__, url_prefix="report")
...
#bp.route('/csp-violations', methods=['POST'])
def report():
"""Receive a post request containing csp-resport.
This is the report-uri. Print report to console and
return Response object.
:return: Flask Response object.
"""
pprint.pprint(json.loads(str(request.data, 'utf-8')))
response = make_response()
response.status_code = 200
return response
...
This route only receives a 400 error and I am at a loss in finding out how to actually debug this
127.0.0.1 - - [04/Nov/2021 14:29:09] "POST /report/csp-violations HTTP/1.1" 400 -
I have tried with a GET request and could see I am receiving an empty request, which seems to mean that the CSP report isn't being delivered (response from https://127.0.0.1:5000/report/csp-violations)
# app/routes/report.py
...
#bp.route('/csp-violations', methods=['GET', 'POST'])
def report():
...
b''
b''
127.0.0.1 - - [04/Nov/2021 18:03:52] "GET /report/csp_violations HTTP/1.1" 200 -
Edit: Definitely receiving nothing
...
#bp.route("/csp_violations", methods=["GET", "POST"])
def report():
...
return str(request.args) # ImmutableMultiDict([ ])
Will not work with a GET or POST request (still 400)
...
#bp.route("/csp_violations", methods=["GET", "POST"])
def report():
content = request.get_json(force=True)
...
Without force=True
#bp.route("/csp_violations", methods=["GET", "POST"])
def report():
content = request.get_json() # json.decoder.JSONDecodeError
...
Chromium (same result on Chrome, Brave, and FireFox)
When I check this out on Chromium under CTRL-SHIFT-I > Network I see
Request URL: https://127.0.0.1:5000/report/csp_violations
Request Method: POST
Status Code: 400 BAD REQUEST
Remote Address: 127.0.0.1:5000
Referrer Policy: strict-origin-when-cross-origin
But apparently the payload does exist at the bottom under "Request Payload"...
{
"document-uri": "https://127.0.0.1:5000/",
"referrer": "",
"violated-directive": "script-src-elem",
"effective-directive": "script-src-elem",
"original-policy": "default-src 'self' https://cdnjs.cloudflare.com https://cdn.cloudflare.com https://cdn.jsdelivr.net https://gravatar.com jquery.js; report-uri /report/csp-violations",
"disposition": "enforce",
"blocked-uri": "inline",
"line-number": 319,
"source-file": "https://127.0.0.1:5000/",
"status-code": 200,
"script-sample": ""
}
Is the CSP blocking this request?
After reading Python Flask 400 Bad Request Error Every Request I set all requests to HTTPS with the help of https://stackoverflow.com/a/63708394/13316671
This fixed a few unittests but no change to the 400 error for this particular route
# app/config.py
from environs import Env
from flask import Flask
env = Env()
env.read_env()
class Config:
"""Load environment."""
...
#property
def PREFERRED_URL_SCHEME(self) -> str: # noqa
return env.str("PREFERRED_URL_SCHEME", default="https")
...
# app/__init__.py
from app.config import Config
def create_app() -> Flask:
app = Flask(__name__)
app.config.from_object(Config())
...
return app
flask run --cert="$HOME/.openssl/cert.pem" --key="$HOME/.openssl/key.pem"
400 Bad Request
Through what I've read a 400 Bad Request error usually happens from an empty request or form
https://stackoverflow.com/a/14113958/13316671
...the issue is that Flask raises an HTTP error when it fails to find a key in
the args and form dictionaries. What Flask assumes by default is that if you
are asking for a particular key and it's not there then something got
left out of the request and the entire request is invalid.
https://stackoverflow.com/a/37017020/13316671
99% of the time, this error is a key error caused by your requesting a key in
the request.form dictionary that does not exist. To debug it, run
print(request.form)
request.data in my case
Through trying to fix this I have gone down the rabbit-hole in regard to just seeing the cause of the 400 error
I have implemented the following
https://stackoverflow.com/a/34172382/13316671
import traceback
from flask import Flask
...
app = Flask(__name__)
...
#app.errorhandler(400)
def internal_error(exception): # noqa
print("400 error caught")
print(traceback.format_exc())
And added the following to my config
I haven't checked, but I think these values already get set alongside DEBUG
# app/config.py
from environs import Env
env = Env()
env.read_env()
class Config:
"""Load environment."""
...
#property
def TRAP_HTTP_EXCEPTIONS(self) -> bool: # noqa
"""Report traceback on error."""
return env.bool("TRAP_HTTP_EXCEPTIONS", default=self.DEBUG) # noqa
#property
def TRAP_BAD_REQUEST_ERRORS(self) -> bool: # noqa
"""Report 400 traceback on error."""
return env.bool("TRAP_BAD_REQUEST_ERRORS", default=self.DEBUG) # noqa
...
I still am not getting a traceback. The only time werkzeug shows a traceback is through a complete crash, for something like a syntax error
It seems, because I am not initializing this request that the app continues to run through the 400 code, no problem
Summary
Main conclusion I'm drawing is that for some reason the report-uri is not valid because I can see the payload exists, I am just not receiving it
I used the relative route as the subdomain would be different for a localhost and a remote. It looks as though the request is being made to the full URL in the Chromium snippet though.
Could I be receiving invalid headers https://stackoverflow.com/a/45682197? If so how can I debug that?
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/report-uri
Note: The report-uri is deprecated, but I don't think most browsers support the report-to parameter
EDIT:
I have taken the following out of my application factory
(this code renders my exceptions - which I posted as an answer here so it is available https://stackoverflow.com/a/69671506/13316671
...
app = Flask(__name__)
config.init_app(app)
...
# exceptions.init_app(app)
...
return app
After doing this (on FireFox Private mode), I made another POST (I've removed the GET method)
I can see the response - here I have managed to collect a Werkzeug traceback finally:
wtforms.validators.ValidationError: The CSRF token is missing.
Working out how to authenticate a csp-report coming from the browser
So far I've had a look at this: Add a new Http Header in Content-Security-Policy-Report-Only "report-uri" POST call

Try this piece of code:
#app.route('/report-csp-violations', methods=['POST'])
def report():
content = request.get_json(force=True) # that's where the shoe pinches
print(json.dumps(content, indent=4, sort_keys=True))
response = make_response()
response.status_code = 204
return response
I think that request.data tries automatically parse JSON data and for success parsing it expects application/json MIME type been sent.
But violation reports is sent with the application/csp-report MIME type, therefore Flask considers it as a wrong data from the client -> 404 Bad Request.
.get_json(force=True) means ignore the mimetype and always try to parse JSON.
Also I don't think you need to do a conversion to utf-8 because according to rfc4627 "3. Encoding":
JSON text SHALL be encoded in Unicode. The default encoding is UTF-8.

For now I have exempted the view from CRSFProtect.
I think that should be fine as this view does not return a template, so I haven't been able to add
<form method="post">
{{ form.csrf_token }}
</form>
And the below didn't work for me (Don't think it's a valid token)
# app/routes/report.py
...
def report():
response = make_response()
response.headers["X-CSRFToken"] = csrf.generate_csrf()
...
return response
...
With the instantiated CSRFProtect object...
# app/extensions.py
...
from flask_wtf.csrf import CSRFProtect
...
csrf_protect = CSRFProtect()
...
def init_app(app: Flask) -> None:
...
csrf_protect.init_app(app)
...
...
...decorate the view
# app/routes/report.py
from app.extensions import csrf_protect
...
#blueprint.route("/csp_violations", methods=["POST"])
#csrf_protect.exempt
def report():
....
127.0.0.1 - - [06/Nov/2021 21:30:46] "POST /report/csp_violations HTTP/1.1" 204 -
{
"csp-report": {
"blocked-uri": "inline",
"column-number": 8118,
"document-uri": "https://127.0.0.1:5000/",
"line-number": 3,
"original-policy": "default-src" ...
"referrer": "",
"source-file": ...
}
}
I had a system going where my errors were obfuscated, so that's my mistake. This is not the first time I've had trouble with CSRFProtect.
I've fixed the debugging problem with
# app/exceptions.py
...
def init_app(app: Flask) -> None:
...
def render_error(error: HTTPException) -> Tuple[str, int]:
# the below code is new
app.logger.error(error.description)
...
...
So, this is what I would see now if I was still getting this error
[2021-11-06 21:23:56,054] ERROR in exceptions: The CSRF token is missing.
127.0.0.1 - - [06/Nov/2021 21:23:56] "POST /report/csp_violations HTTP/1.1" 400 -

Why does Django trigger GET after POST?

I have the following view which takes either a URL or an uploaded text file, creates a Word Cloud and finally displays the generated image to the user.
def create(request):
"""
Displays the generated WordCloud from the
given URI or uploaded file
"""
response = HttpResponse(content_type="image/png")
# in order to avoid KeyError
myfile = request.FILES.get('myfile', None)
if request.POST['uri'] == '' and myfile is None:
return render(request, 'nube/index.html', {
'error_message': NOTHING_TO_PROCESS
})
try:
if myfile:
cloud = WordCloud(myfile, type="upload")
else:
cloud = WordCloud(request.POST['uri'], type="internet")
except (MissingSchema):
return render(request, 'nube/index.html', {
'error_message': URI_COULD_NOT_BE_PROCESSED
})
else:
# img is a PIL.Image instance
img = cloud.get_word_cloud_as_image()
img.save(response, 'PNG')
return response
The image is displayed with no problems; the POST request is processed properly, as can be seen from the log:
[16/Jan/2018 22:53:25] "POST /nube/create HTTP/1.1" 200 216961
However, even though the server didn't crash, I noticed an Exception was raised every time immediately after:
Internal Server Error: /nube/create
Traceback (most recent call last):
File "C:\repos\phuyu\venv\lib\site-packages\django\utils\datastructures.py", line 77, in __getitem__
list_ = super().__getitem__(key)
KeyError: 'uri'
After debugging the code I noticed that my create view was being called once again, but this time as a GET request, and of course, the parameters uri and myfile didn't exist this time, thus raising the exceptions.
To make sure, I changed create into a class based view and only defined its post method. As suspected, now I got the following line in the logs after the successful POST:
Method Not Allowed (GET): /nube/create
[16/Jan/2018 22:44:41] "GET /nube/create HTTP/1.1" 405 0
What would be the correct way to handle this? I'm fairly new to Django.

As #usman-maqbool suggested, the problem was actually in my WordCloud clode, especifically, in get_word_cloud_as_image(), a trailing semicolon:
def get_word_cloud_as_image(self):
return self.img;
After removing it, there's no more sneaky GET requests. I'm not sure why it had that effect. I'd appreciate if someone could clarify.

Flask test client using GET request instead of POST

I have a route for only POST request and it returns json response if conditions are met. It's something like this:
#app.route('/panel', methods=['POST'])
def post_panel():
# Check for conditions and database operations
return jsonify({"message": "Panel added to database!"
"success": 1})
I am using flask-sslify to force http requests to https.
I am testing this route with Flask test client and unittest. The test function is similar to following:
class TestAPI2_0(unittest.TestCase):
def setUp(self):
self.app = create_app('testing')
self.app_context = self.app.app_context()
self.app_context.push()
db.create_all()
create_fake_data(db)
self.client = self.app.test_client()
def tearDown(self):
....
def test_post_panel_with_good_data(self):
# data
r = self.client.post('/panel',
data=json.dumps(data),
follow_redirects=True)
print(r.data)
self.assertEqual(r.status_code, 200)
Output is exactly below:
test_post_panel_with_good_data (tests.test_api_2_0.TestAPI2_0) ... b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>405 Method Not Allowed</title>\n<h1>Method Not Allowed</h1>\n<p>The method is not allowed for the requested URL.</p>\n'
======================================================================
FAIL: test_post_panel_with_good_data (tests.test_api_2_0.TestAPI2_0)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tanjibpa/work/craftr-master/tests/test_api_2_0.py", line 110, in test_post_panel_with_good_data
self.assertEqual(r.status_code, 200)
AssertionError: 405 != 200
I am getting an error that Method is not allowed in that route.
If I specify GET as a method (methods=['GET', 'POST']) for the route test seems to work. But why test client is making a GET request? Is there any way around rather than specifying a GET request for the route?
Update:
If do it like this:
#app.route('/panel', methods=['GET', 'POST'])
def post_panel():
if request.method == 'POST':
# Check for conditions and database operations
return jsonify({"message": "Panel added to database!"
"success": 1})
return jsonify({"message": "GET request"})
I get output like this:
test_post_panel_with_good_data (tests.test_api_2_0.TestAPI2_0) ... b'{\n "message": "GET request"\n}\n'

I found out what was causing the GET request within flask test client.
I am using flask-sslify to force http requests to https.
Somehow flask-sslify is enforcing a GET request although test client is specified with other kind of requests (POST, PUT, DELETE...).
So, If I disable sslify during testing flask test client works as it should.

Returning error indication when flask POST Request fails

I'm trying to create a simple flask application that takes a string from an iOS application and stores it in a local data base. I'm a bit confused whats happening in the return portion of the submitPost() function. I'm trying to return a dictionary that contains a BOOL that indicates whether the Post request executed fully. However, i'm not sure how to variate between returning a 0 or a 1.
//Function that handles the Post request
#app.route('/submitPost/', methods=['POST'])
def submitPost():
post = Post.from_json(request.json)
db.session.add(post)
db.session.commit()
return jsonify(post.to_json), {'Success': 1}

Try the below. This way if an exception is thrown when trying to insert data it will be caught and the transaction will be rolled back as well as a response being send back.
you could also replace the zero and one with True & False depending on your preference
Below code hasn't been tested
#app.route('/submitPost/', methods=['POST'])
def submitPost():
try:
post = Post.from_json(request.json)
db.session.add(post)
db.session.commit()
return jsonify({'Success': 1})
except Exception: # replace with a more meaningful exception
db.session.rollback()
return jsonify{'Failure': 0}

How to catch all exceptions with CherryPy?

I use CherryPy to run a very simple web server. It is intended to process the GET parameters and, if they are correct, do something with them.
import cherrypy
class MainServer(object):
def index(self, **params):
# do things with correct parameters
if 'a' in params:
print params['a']
index.exposed = True
cherrypy.quickstart(MainServer())
For example,
http://127.0.0.1:8080/abcde:
404 Not Found
The path '/abcde' was not found.
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\cherrypy\_cprequest.py", line 656, in respond
response.body = self.handler()
File "C:\Python27\lib\site-packages\cherrypy\lib\encoding.py", line 188, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "C:\Python27\lib\site-packages\cherrypy\_cperror.py", line 386, in __call__
raise self
NotFound: (404, "The path '/abcde' was not found.")
Powered by CherryPy 3.2.4
I am trying to catch this exception and show a blank page because the clients do not care about it. Specifically, the result would be an empty body, no matter the url or query string that resulted in an exception.
I had a look at documentation on error handling cherrypy._cperror, but I did not find a way to actually use it.
Note: I gave up using CherryPy and found a simple solution using BaseHTTPServer (see my answer below)

Docs somehow seem to miss this section. This is what I found while looking for detailed explanation for custom error handling from the source code.
Custom Error Handling
Anticipated HTTP responses
The 'error_page' config namespace can be used to provide custom HTML output for
expected responses (like 404 Not Found). Supply a filename from which the
output will be read. The contents will be interpolated with the values
%(status)s, %(message)s, %(traceback)s, and %(version)s using plain old Python
string formatting.
_cp_config = {
'error_page.404': os.path.join(localDir, "static/index.html")
}
Beginning in version 3.1, you may also provide a function or other callable as
an error_page entry. It will be passed the same status, message, traceback and
version arguments that are interpolated into templates
def error_page_402(status, message, traceback, version):
return "Error %s - Well, I'm very sorry but you haven't paid!" % status
cherrypy.config.update({'error_page.402': error_page_402})
Also in 3.1, in addition to the numbered error codes, you may also supply
error_page.default to handle all codes which do not have their own error_page
entry.
Unanticipated errors
CherryPy also has a generic error handling mechanism: whenever an unanticipated
error occurs in your code, it will call
Request.error_response to
set the response status, headers, and body. By default, this is the same
output as
HTTPError(500). If you want to provide
some other behavior, you generally replace "request.error_response".
Here is some sample code that shows how to display a custom error message and
send an e-mail containing the error
from cherrypy import _cperror
def handle_error():
cherrypy.response.status = 500
cherrypy.response.body = [
"<html><body>Sorry, an error occurred</body></html>"
]
sendMail('error#domain.com',
'Error in your web app',
_cperror.format_exc())
#cherrypy.config(**{'request.error_response': handle_error})
class Root:
pass
Note that you have to explicitly set
response.body
and not simply return an error message as a result.

Choose what's most suitable for you: Default Methods, Custom Error Handling.
I don't think you should use BaseHTTPServer. If your app is that simple, just get a lightweight framework (e. g. Flask), even though it might be a bit overkill, OR stay low level but still within the WSGI standard and use a WSGI-compliant server.

CherryPy IS catching your exception. That's how it returns a valid page to the browser with the caught exception.
I suggest you read through all the documentation. I realize it isn't the best documentation or organized well, but if you at least skim through it the framework will make more sense. It is a small framework, but does almost everything you'd expect from a application server.
import cherrypy
def show_blank_page_on_error():
"""Instead of showing something useful to developers but
disturbing to clients we will show a blank page.
"""
cherrypy.response.status = 500
cherrypy.response.body = ''
class Root():
"""Root of the application"""
_cp_config = {'request.error_response': show_blank_page_on_error}
#cherrypy.expose
def index(self):
"""Root url handler"""
raise Exception
See this for the example in the documentation on the page mentioned above for further reference.

You can simply use a try/except clause:
try:
cherrypy.quickstart(MainServer())
except: #catches all errors, including basic python errors
print("Error!")
This will catch every single error. But if you want to catch only cherrypy._cperror:
from cherrypy import _cperror
try:
cherrypy.quickstart(MainServer())
except _cperror.CherryPyException: #catches only CherryPy errors.
print("CherryPy error!")
Hope this helps!

import cherrypy
from cherrypy import HTTPError
def handle_an_exception():
cherrypy.response.status = 500
cherrypy.response.headers['content-type'] = 'text/plain;charset=UTF-8'
cherrypy.response.body = b'Internal Server Error'
def handle_a_404(status=None, message=None, version=None, traceback=None):
cherrypy.response.headers['content-type'] = 'text/plain;charset=UTF-8'
return f'Error page for 404'.encode('UTF-8')
def handle_default(status=None, message=None, version=None, traceback=None):
cherrypy.response.headers['content-type'] = 'text/plain;charset=UTF-8'
return f'Default error page: {status}'.encode('UTF-8')
class Root:
"""Root of the application"""
_cp_config = {
# handler for an unhandled exception
'request.error_response': handle_an_exception,
# specific handler for HTTP 404 error
'error_page.404': handle_a_404,
# default handler for any other HTTP error
'error_page.default': handle_default
}
#cherrypy.expose
def index(self):
"""Root url handler"""
raise Exception("an exception")
#cherrypy.expose
def simulate400(self):
raise HTTPError(status=400, message="Bad Things Happened")
cherrypy.quickstart(Root())
Test with:
http://127.0.0.1:8080/
http://127.0.0.1:8080/simulate400
http://127.0.0.1:8080/missing

Though this was the one of the top results when I searched for cherrypy exception handling, accepted answer did not fully answered the question. Following is a working code against cherrypy 14.0.0
# Implement handler method
def exception_handler(status, message, traceback, version)
# Your logic goes here
class MyClass()
# Update configurations
_cp_config = {"error_page.default": exception_handler}
Note the method signature. Without this signature your method will not get invoked.Following are the contents of method parameters,
status : HTTP status and a description
message : Message attached to the exception
traceback : Formatted stack trace
version : Cherrypy version

Maybe you could use a 'before_error_response' handler from cherrypy.tools
#cherrypy.tools.register('before_error_response', priority=90)
def handleexception():
cherrypy.response.status = 500
cherrypy.response.body = ''
And don't forget to enable it:
tools.handleexception.on = True

I gave up using CherryPy and ended up using the follwing code, which solves the issue in a few lines with the standard BaseHTTPServer:
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
from urlparse import urlparse, parse_qs
class GetHandler(BaseHTTPRequestHandler):
def do_GET(self):
url = urlparse(self.path)
d = parse_qs(url[4])
if 'c' in d:
print d['c'][0]
self.send_response(200)
self.end_headers()
return
server = HTTPServer(('localhost', 8080), GetHandler)
server.serve_forever()