Develop Reference

How can I write a route to receive Content Security Policy report with Flask without getting a 400 Bad Request error (flask_wtf.csrf.CSRFError)?

TL;DR: Apologies for the long post. In a nutshell I am trying to debug a CSP report-uri. If I am missing critical information please let me know.
CSP implementation: Flask-Talisman
The attribute that needs to be set: content_security_policy_report_uri
There does not seem to be a lot of information out there on how to capture this report
I can't find anything specific in the Flask-Talisman documentation
As Flask-Talisman only sets headers, including the report-uri, I imagine this is outside the scope of the extension anyway
The route
All resources I've found have roughly the same function:
https://www.merixstudio.com/blog/content-security-policy-flask-and-django-part-2/
http://csplite.com/csp260/
https://github.com/GoogleCloudPlatform/flask-talisman/issues/21
The only really detailed explanation I've found for this route is below (it is not related to Flask-Talisman however)
From https://www.merixstudio.com/blog/content-security-policy-flask-and-django-part-2/ (This is what I am currently using)
# app/routes/report.py
import json
import pprint
from flask import request, make_response, Blueprint
...
bp = Blueprint("report", __name__, url_prefix="report")
...
#bp.route('/csp-violations', methods=['POST'])
def report():
"""Receive a post request containing csp-resport.
This is the report-uri. Print report to console and
return Response object.
:return: Flask Response object.
"""
pprint.pprint(json.loads(str(request.data, 'utf-8')))
response = make_response()
response.status_code = 200
return response
...
This route only receives a 400 error and I am at a loss in finding out how to actually debug this
127.0.0.1 - - [04/Nov/2021 14:29:09] "POST /report/csp-violations HTTP/1.1" 400 -
I have tried with a GET request and could see I am receiving an empty request, which seems to mean that the CSP report isn't being delivered (response from https://127.0.0.1:5000/report/csp-violations)
# app/routes/report.py
...
#bp.route('/csp-violations', methods=['GET', 'POST'])
def report():
...
b''
b''
127.0.0.1 - - [04/Nov/2021 18:03:52] "GET /report/csp_violations HTTP/1.1" 200 -
Edit: Definitely receiving nothing
...
#bp.route("/csp_violations", methods=["GET", "POST"])
def report():
...
return str(request.args) # ImmutableMultiDict([ ])
Will not work with a GET or POST request (still 400)
...
#bp.route("/csp_violations", methods=["GET", "POST"])
def report():
content = request.get_json(force=True)
...
Without force=True
#bp.route("/csp_violations", methods=["GET", "POST"])
def report():
content = request.get_json() # json.decoder.JSONDecodeError
...
Chromium (same result on Chrome, Brave, and FireFox)
When I check this out on Chromium under CTRL-SHIFT-I > Network I see
Request URL: https://127.0.0.1:5000/report/csp_violations
Request Method: POST
Status Code: 400 BAD REQUEST
Remote Address: 127.0.0.1:5000
Referrer Policy: strict-origin-when-cross-origin
But apparently the payload does exist at the bottom under "Request Payload"...
{
"document-uri": "https://127.0.0.1:5000/",
"referrer": "",
"violated-directive": "script-src-elem",
"effective-directive": "script-src-elem",
"original-policy": "default-src 'self' https://cdnjs.cloudflare.com https://cdn.cloudflare.com https://cdn.jsdelivr.net https://gravatar.com jquery.js; report-uri /report/csp-violations",
"disposition": "enforce",
"blocked-uri": "inline",
"line-number": 319,
"source-file": "https://127.0.0.1:5000/",
"status-code": 200,
"script-sample": ""
}
Is the CSP blocking this request?
After reading Python Flask 400 Bad Request Error Every Request I set all requests to HTTPS with the help of https://stackoverflow.com/a/63708394/13316671
This fixed a few unittests but no change to the 400 error for this particular route
# app/config.py
from environs import Env
from flask import Flask
env = Env()
env.read_env()
class Config:
"""Load environment."""
...
#property
def PREFERRED_URL_SCHEME(self) -> str: # noqa
return env.str("PREFERRED_URL_SCHEME", default="https")
...
# app/__init__.py
from app.config import Config
def create_app() -> Flask:
app = Flask(__name__)
app.config.from_object(Config())
...
return app
flask run --cert="$HOME/.openssl/cert.pem" --key="$HOME/.openssl/key.pem"
400 Bad Request
Through what I've read a 400 Bad Request error usually happens from an empty request or form
https://stackoverflow.com/a/14113958/13316671
...the issue is that Flask raises an HTTP error when it fails to find a key in
the args and form dictionaries. What Flask assumes by default is that if you
are asking for a particular key and it's not there then something got
left out of the request and the entire request is invalid.
https://stackoverflow.com/a/37017020/13316671
99% of the time, this error is a key error caused by your requesting a key in
the request.form dictionary that does not exist. To debug it, run
print(request.form)
request.data in my case
Through trying to fix this I have gone down the rabbit-hole in regard to just seeing the cause of the 400 error
I have implemented the following
https://stackoverflow.com/a/34172382/13316671
import traceback
from flask import Flask
...
app = Flask(__name__)
...
#app.errorhandler(400)
def internal_error(exception): # noqa
print("400 error caught")
print(traceback.format_exc())
And added the following to my config
I haven't checked, but I think these values already get set alongside DEBUG
# app/config.py
from environs import Env
env = Env()
env.read_env()
class Config:
"""Load environment."""
...
#property
def TRAP_HTTP_EXCEPTIONS(self) -> bool: # noqa
"""Report traceback on error."""
return env.bool("TRAP_HTTP_EXCEPTIONS", default=self.DEBUG) # noqa
#property
def TRAP_BAD_REQUEST_ERRORS(self) -> bool: # noqa
"""Report 400 traceback on error."""
return env.bool("TRAP_BAD_REQUEST_ERRORS", default=self.DEBUG) # noqa
...
I still am not getting a traceback. The only time werkzeug shows a traceback is through a complete crash, for something like a syntax error
It seems, because I am not initializing this request that the app continues to run through the 400 code, no problem
Summary
Main conclusion I'm drawing is that for some reason the report-uri is not valid because I can see the payload exists, I am just not receiving it
I used the relative route as the subdomain would be different for a localhost and a remote. It looks as though the request is being made to the full URL in the Chromium snippet though.
Could I be receiving invalid headers https://stackoverflow.com/a/45682197? If so how can I debug that?
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/report-uri
Note: The report-uri is deprecated, but I don't think most browsers support the report-to parameter
EDIT:
I have taken the following out of my application factory
(this code renders my exceptions - which I posted as an answer here so it is available https://stackoverflow.com/a/69671506/13316671
...
app = Flask(__name__)
config.init_app(app)
...
# exceptions.init_app(app)
...
return app
After doing this (on FireFox Private mode), I made another POST (I've removed the GET method)
I can see the response - here I have managed to collect a Werkzeug traceback finally:
wtforms.validators.ValidationError: The CSRF token is missing.
Working out how to authenticate a csp-report coming from the browser
So far I've had a look at this: Add a new Http Header in Content-Security-Policy-Report-Only "report-uri" POST call

Try this piece of code:
#app.route('/report-csp-violations', methods=['POST'])
def report():
content = request.get_json(force=True) # that's where the shoe pinches
print(json.dumps(content, indent=4, sort_keys=True))
response = make_response()
response.status_code = 204
return response
I think that request.data tries automatically parse JSON data and for success parsing it expects application/json MIME type been sent.
But violation reports is sent with the application/csp-report MIME type, therefore Flask considers it as a wrong data from the client -> 404 Bad Request.
.get_json(force=True) means ignore the mimetype and always try to parse JSON.
Also I don't think you need to do a conversion to utf-8 because according to rfc4627 "3. Encoding":
JSON text SHALL be encoded in Unicode. The default encoding is UTF-8.

For now I have exempted the view from CRSFProtect.
I think that should be fine as this view does not return a template, so I haven't been able to add
<form method="post">
{{ form.csrf_token }}
</form>
And the below didn't work for me (Don't think it's a valid token)
# app/routes/report.py
...
def report():
response = make_response()
response.headers["X-CSRFToken"] = csrf.generate_csrf()
...
return response
...
With the instantiated CSRFProtect object...
# app/extensions.py
...
from flask_wtf.csrf import CSRFProtect
...
csrf_protect = CSRFProtect()
...
def init_app(app: Flask) -> None:
...
csrf_protect.init_app(app)
...
...
...decorate the view
# app/routes/report.py
from app.extensions import csrf_protect
...
#blueprint.route("/csp_violations", methods=["POST"])
#csrf_protect.exempt
def report():
....
127.0.0.1 - - [06/Nov/2021 21:30:46] "POST /report/csp_violations HTTP/1.1" 204 -
{
"csp-report": {
"blocked-uri": "inline",
"column-number": 8118,
"document-uri": "https://127.0.0.1:5000/",
"line-number": 3,
"original-policy": "default-src" ...
"referrer": "",
"source-file": ...
}
}
I had a system going where my errors were obfuscated, so that's my mistake. This is not the first time I've had trouble with CSRFProtect.
I've fixed the debugging problem with
# app/exceptions.py
...
def init_app(app: Flask) -> None:
...
def render_error(error: HTTPException) -> Tuple[str, int]:
# the below code is new
app.logger.error(error.description)
...
...
So, this is what I would see now if I was still getting this error
[2021-11-06 21:23:56,054] ERROR in exceptions: The CSRF token is missing.
127.0.0.1 - - [06/Nov/2021 21:23:56] "POST /report/csp_violations HTTP/1.1" 400 -

How to return 400 (Bad Request) on Flask?

I have created a simple flask app that and I'm reading the response from python as:
response = requests.post(url,data=json.dumps(data), headers=headers )
data = json.loads(response.text)
Now my issue is that under certain conditions I want to return a 400 or 500 message response. So far I'm doing it like this:
abort(400, 'Record not found')
#or
abort(500, 'Some error...')
This does print the message on the terminal:
But in the API response I kept getting a 500 error response:
The structure of the code is as follows:
|--my_app
|--server.py
|--main.py
|--swagger.yml
Where server.py has this code:
from flask import render_template
import connexion
# Create the application instance
app = connexion.App(__name__, specification_dir="./")
# read the swagger.yml file to configure the endpoints
app.add_api("swagger.yml")
# Create a URL route in our application for "/"
#app.route("/")
def home():
"""
This function just responds to the browser URL
localhost:5000/
:return: the rendered template "home.html"
"""
return render_template("home.html")
if __name__ == "__main__":
app.run(host="0.0.0.0", port="33")
And main.py has all the function I'm using for the API endpoints.
E.G:
def my_funct():
abort(400, 'Record not found')
When my_funct is called, I get the Record not found printed on the terminal, but not in the response from the API itself, where I always get the 500 message error.

You have a variety of options:
The most basic:
#app.route('/')
def index():
return "Record not found", 400
If you want to access the headers, you can grab the response object:
#app.route('/')
def index():
resp = make_response("Record not found", 400)
resp.headers['X-Something'] = 'A value'
return resp
Or you can make it more explicit, and not just return a number, but return a status code object
from flask_api import status
#app.route('/')
def index():
return "Record not found", status.HTTP_400_BAD_REQUEST
Further reading:
You can read more about the first two here: About Responses (Flask quickstart)
And the third here: Status codes (Flask API Guide)

I like to use the flask.Response class:
from flask import Response
#app.route("/")
def index():
return Response(
"The response body goes here",
status=400,
)
flask.abort is a wrapper around werkzeug.exceptions.abort which is really just a helper method to make it easier to raise HTTP exceptions. That's fine in most cases, but for restful APIs, I think it may be better to be explicit with return responses.

Here's some snippets from a Flask app I wrote years ago. It has an example of a 400 response
import werkzeug
from flask import Flask, Response, json
from flask_restplus import reqparse, Api, Resource, abort
from flask_restful import request
from flask_cors import CORS
app = Flask(__name__)
CORS(app)
api = Api(app)
parser = reqparse.RequestParser()
parser.add_argument('address_to_score', type=werkzeug.datastructures.FileStorage, location='files')
class MissingColumnException(Exception):
pass
class InvalidDateFormatException(Exception):
pass
#api.route('/project')
class Project(Resource):
#api.expect(parser)
#api.response(200, 'Success')
#api.response(400, 'Validation Error')
def post(self):
"""
Takes in an excel file of addresses and outputs a JSON with scores and rankings.
"""
try:
df, input_trees, needed_zones = data.parse_incoming_file(request)
except MissingColumnException as e:
abort(400, 'Excel File Missing Mandatory Column(s):', columns=str(e))
except Exception as e:
abort(400, str(e))
project_trees = data.load_needed_trees(needed_zones, settings['directories']['current_tree_folder'])
df = data.multiprocess_query(df, input_trees, project_trees)
df = data.score_locations(df)
df = data.rank_locations(df)
df = data.replace_null(df)
output_file = df.to_dict('index')
resp = Response(json.dumps(output_file), mimetype='application/json')
resp.status_code = 200
return resp
#api.route('/project/health')
class ProjectHealth(Resource):
#api.response(200, 'Success')
def get(self):
"""
Returns the status of the server if it's still running.
"""
resp = Response(json.dumps('OK'), mimetype='application/json')
resp.status_code = 200
return resp

You can return a tuple with the second element being the status (either 400 or 500).
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello():
return "Record not found", 400
if __name__ == '__main__':
app.run()
Example of calling the API from python:
import requests
response = requests.get('http://127.0.0.1:5000/')
response.text
# 'This is a bad request!'
response.status_code
# 400

I think you're using the abort() function correctly. I suspect the issue here is that an error handler is that is catching the 400 error and then erroring out which causes the 500 error. See here for more info on flask error handling.
As an example, the following would change a 400 into a 500 error:
#app.errorhandler(400)
def handle_400_error(e):
raise Exception("Unhandled Exception")
If you're not doing any error handling, it could be coming from the connexion framework, although I'm not familiar with this framework.

You can simply use #app.errorhandler decorator.
example:
#app.errorhandler(400)
def your_function():
return 'your custom text', 400

flask/python creating multiple TestCase classes returns 404

While running python's unittest in my Flask application, I am returned a 404 status code when using multiple TestCase classes.
I've tried flask_testing and have gotten similar issues. I opted for unittest because of its greater popularity and availability to find documentation online.
test_global.py
from server import create_app
class Global(unittest.TestCase):
def setUp(self):
self.app = create_app(testing=True)
self.client = self.app.test_client()
self.client.testing = True
# tests different cookie redirect
def test_different_cookie_redirect(self):
self.client.set_cookie('127.0.0.1', 'lang', 'en')
response = self.client.get('/fr')
# this passes
self.assertEqual(response.status_code, 302)
The above works as intended. If the cookie is different, the page should redirect. The problem happens when I want to add another class.
class Index(unittest.TestCase):
def setUp(self):
self.app = create_app(testing=True)
self.client = self.app.test_client()
self.client.testing = True
# tests same cookie redirect
def test_same_cookie_redirect(self):
self.client.set_cookie('127.0.0.1', 'lang', 'fr')
response = self.client.get('/fr')
# this returns a 404 and fails the test
self.assertEqual(response.status_code, 200)
This is the error
Traceback (most recent call last):
File "/test_global.py", line 55, in test_same_cookie_redirect
self.assertEqual(response.status_code, 200)
AssertionError: 404 != 200
If I remove the Global class, the Index test then works and returns a status_code of 200. Why can't both work at the same time?
The reason I opted to have multiple classes is to be able to split my code in different files and run python -m unittest discover to handle them all.

Catch http-status code in Flask

I lately started using Flask in one of my projects to provide data via a simple route. So far I return a json file containing the data and some other information. When running my Flask app I see the status code of this request in terminal. I would like to return the status code as a part of my final json file. Is it possible to catch the same code I see in terminal?
Some simple might look like this
from flask import Flask
from flask import jsonify
app = Flask(__name__)
#app.route('/test/<int1>/<int2>/')
def test(int1,int2):
int_sum = int1 + int2
return jsonify({"result":int_sum})
if __name__ == '__main__':
app.run(port=8082)
And in terminal I get:

You are who set the response code (by default 200 on success response), you can't catch this value before the response is emited. But if you know the result of your operation you can put it on the final json.
#app.route('/test/<int1>/<int2>/')
def test(int1, int2):
int_sum = int1 + int2
response_data = {
"result": int_sum,
"sucess": True,
"status_code": 200
}
# make sure the status_code on your json and on the return match.
return jsonify(response_data), 200 # <- the status_code displayed code on console
By the way if you access this endpoint from a request library, on the response object you can find the status_code and all the http refered data plus the json you need.
Python requests library example
import requests
req = requests.get('your.domain/test/3/3')
print req.url # your.domain/test/3/3
print req.status_code # 200
print req.json() # {u'result': 6, u'status_code: 200, u'success': True}

You can send HTTP status code as follow:
#app.route('/test')
def test():
status_code = 200
return jsonify({'name': 'Nabin Khadka'}, status_code) # Notice second element of the return tuple(return)
This way you can control what status code to return to the client (typically to web browser.)

Flask and error handling BadRequest exception

I have simple view like that:
#blueprint.route('/')
def index():
'''CMS splash page!'''
print request.form['no-key-like-that']
return render_template('home/splash.html', title='Welcome')
The goal of that view is to cause BadRequest error on Flask. That happens and I got very generic error page in the process that says:
Bad Request
The browser (or proxy) sent a request that this server could not understand.
However I want to intercept all errors and serve them wrapped in our templates for better look and feel, I do this like that:
#app.errorhandler(TimeoutException)
def handle_timeout(error):
if utils.misc.request_is_xhr(request):
return jsonify({'api_timeout': True}), 503
return redirect(url_for('errors.api_timeout', path=request.path))
However the same cannot be done for BadRequest exception, I tried:
#app.errorhandler(werkzeug.exceptions.BadRequestError)
def handle_bad_request(error):
return render_template(
'errors/bad_request.html',
exception_message=unicode(error),
return_path=request.path), 400
And:
#app.errorhandler(werkzeug.exceptions.BadRequestKeyError)
def handle_bad_request(error):
return render_template(
'errors/bad_request.html',
exception_message=unicode(error),
return_path=request.path), 400
And:
#app.errorhandler(werkzeug.exceptions.HttpError)
def handle_bad_request(error):
return render_template(
'errors/bad_request.html',
exception_message=unicode(error),
return_path=request.path), 400
In each case, instead of my bad_request.html there is raw response I mentioned above.
Bad Request
The browser (or proxy) sent a request that this server could not understand.
What actually works for me:
# NOTE: for some reason we cannot intercpet BadRequestKeyError, didn't
# figure out why. Thus I have added check if given 400 is
# BadRequestKeyError if so display standard api error page. This happens
# often when dev tries to access request.form that does not exist.
#app.errorhandler(400)
def handle_bad_request(error):
if isinstance(error, werkzeug.exceptions.BadRequestKeyError):
if utils.misc.request_is_xhr(request):
return jsonify({
'api_internal_error': True,
'error': unicode(error)
}), 500
return render_template(
'errors/api_internal.html',
exception_message=unicode(error),
return_path=request.path), 500
However as you can see it's far from perfection as error 400 is not necessarily always BadRequestKeyError.
Because exception handling works for any other exception but not BadRequest family it keeps me wondering, is it a bug? Or perhaps I am doing something wrong.