Develop Reference

Is it possible to run custom code before Swagger validations in a python/flask server stub?

I'm using the swagger editor (OpenApi 2) for creating flask apis in python. When you define a model in swagger and use it as a schema for the body of a request, swagger validates the body before handing the control to you in the X_controller.py files.
I want to add some code before that validation happens (for printing logs for debugging purposes). Swagger just prints to stdout errors like the following and they are not useful when you have a lot of fields (I need the key that isn't valid).
https://host/path validation error: False is not of type 'string'
10.255.0.2 - - [20/May/2020:20:20:20 +0000] "POST /path HTTP/1.1" 400 116 "-" "GuzzleHttp/7"
I know tecnically you can remove the validations in swagger and do them manually in your code but I want to keep using this feature, when it works it's awesome.
Any ideas on how to do this or any alternative to be able to log the requests are welcome.

After some time studying the matter this is what I learnt.
First let's take a look at how a python-flask server made with Swagger Editor works.
Swagger Editor generates the server stub through Swagger Codegen using the definition written in Swagger Editor. This server stub returned by codegen uses the framework Connexion on top of flask to handle all the HTTP requests and responses, including the validation against the swagger definition (swagger.yaml).
Connexion is a framework that makes it easy to develop python-flask servers because it has a lot of functionality you'd have to make yourself already built in, like parameter validation. All we need to do is replace (in this case modify) these connexion validators.
There are three validators:
ParameterValidator
RequestBodyValidator
ResponseValidator
They get mapped to flask by default but we can replace them easily in the __main__.py file as we'll see.
Our goal is to replace the default logs and default error response to some custom ones. I'm using a custom Error model and a function called error_response() for preparing error responses, and Loguru for logging the errors (not mandatory, you can keep the original one).
To make the changes needed, looking at the connexion validators code, we can see that most of it can be reused, we only need to modify:
RequestBodyValidator: __call__() and validate_schema()
ParameterValidator: __call__()
So we only need to create two new classes that extend the original ones, and copy and modify those functions.
Be careful when copying and pasting. This code is based on connexion==1.1.15. If your are on a different version you should base your classes on it.
In a new file custom_validators.py we need:
import json
import functools
from flask import Flask
from loguru import logger
from requests import Response
from jsonschema import ValidationError
from connexion.utils import all_json, is_null
from connexion.exceptions import ExtraParameterProblem
from swagger_server.models import Error
from connexion.decorators.validation import ParameterValidator, RequestBodyValidator
app = Flask(__name__)
def error_response(response: Error) -> Response:
return app.response_class(
response=json.dumps(response.to_dict(), default=str),
status=response.status,
mimetype='application/json')
class CustomParameterValidator(ParameterValidator):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
def __call__(self, function):
"""
:type function: types.FunctionType
:rtype: types.FunctionType
"""
#functools.wraps(function)
def wrapper(request):
if self.strict_validation:
query_errors = self.validate_query_parameter_list(request)
formdata_errors = self.validate_formdata_parameter_list(request)
if formdata_errors or query_errors:
raise ExtraParameterProblem(formdata_errors, query_errors)
for param in self.parameters.get('query', []):
error = self.validate_query_parameter(param, request)
if error:
response = error_response(Error(status=400, description=f'Error: {error}'))
return self.api.get_response(response)
for param in self.parameters.get('path', []):
error = self.validate_path_parameter(param, request)
if error:
response = error_response(Error(status=400, description=f'Error: {error}'))
return self.api.get_response(response)
for param in self.parameters.get('header', []):
error = self.validate_header_parameter(param, request)
if error:
response = error_response(Error(status=400, description=f'Error: {error}'))
return self.api.get_response(response)
for param in self.parameters.get('formData', []):
error = self.validate_formdata_parameter(param, request)
if error:
response = error_response(Error(status=400, description=f'Error: {error}'))
return self.api.get_response(response)
return function(request)
return wrapper
class CustomRequestBodyValidator(RequestBodyValidator):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
def __call__(self, function):
"""
:type function: types.FunctionType
:rtype: types.FunctionType
"""
#functools.wraps(function)
def wrapper(request):
if all_json(self.consumes):
data = request.json
if data is None and len(request.body) > 0 and not self.is_null_value_valid:
# the body has contents that were not parsed as JSON
return error_response(Error(
status=415,
description="Invalid Content-type ({content_type}), JSON data was expected".format(content_type=request.headers.get("Content-Type", ""))
))
error = self.validate_schema(data, request.url)
if error and not self.has_default:
return error
response = function(request)
return response
return wrapper
def validate_schema(self, data, url):
if self.is_null_value_valid and is_null(data):
return None
try:
self.validator.validate(data)
except ValidationError as exception:
description = f'Validation error. Attribute "{exception.validator_value}" return this error: "{exception.message}"'
logger.error(description)
return error_response(Error(
status=400,
description=description
))
return None
Once we have our validators, we have to map them to the flask app (__main__.py) using validator_map:
validator_map = {
'parameter': CustomParameterValidator,
'body': CustomRequestBodyValidator,
'response': ResponseValidator,
}
app = connexion.App(__name__, specification_dir='./swagger/', validator_map=validator_map)
app.app.json_encoder = encoder.JSONEncoder
app.add_api(Path('swagger.yaml'), arguments={'title': 'MyApp'})
If you also need to replace the validator I didn't use in this example, just create a custom child class of ResponseValidator and replace it on the validator_map dictionary in __main__.py.
Connexion docs:
https://connexion.readthedocs.io/en/latest/request.html

Forgive me for repeating an answer first posted at https://stackoverflow.com/a/73051652/1630244
Have you tried the Connexion before_request feature? Here's an example that logs the headers and content before Connexion validates the body:
import connexion
import logging
from flask import request
logger = logging.getLogger(__name__)
conn_app = connexion.FlaskApp(__name__)
#conn_app.app.before_request
def before_request():
for h in request.headers:
logger.debug('header %s', h)
logger.debug('data %s', request.get_data())

Routing Cherrypy web apps(fail to road resource)

I have spent more than 4 days trying to figure out this, but my every attempt failed, so I thought I could really use some of your helps.
In the code below, Storm is an app with function submit_data() which fetches data from databases according to pagename parameter passed at instantiation and idx parameter passed from xmlhttp GET request in Javascript.
Basically I want to have an index page with Root class, and a bunch of similar Storm pages with different pagename parameters read from an XML file.
I am using RoutesDispatcher because I am parametrizing page name(address) using the XML file too.
When I used /storm as the address it didn't even work probably because the name clashes, so I set the address as /st/.
Now it displays the header, but it does not find submit_data function at all. The Chrome console throws the following error.
Failed to load resource: the server responded with a status of 404 (Not Found) http://localhost:8080/st/submit_data?idx=0
Failed to load resource: the server responded with a status of 404 (Not Found) http://localhost:8080/st/submit_data?idx=2
Failed to load resource: the server responded with a status of 404 (Not Found) http://localhost:8080/st/submit_data?idx=1
How can I structure this so that when I type localhost:8080/st on my browser, the html/javascript template storm.html looks up submit_data function and call it and display it on the webpage? Before I started routing, it worked totally fine.
class Storm(object):
pagename = ""
def __init__(self, pagename):
self.pagename = pagename
#cherrypy.expose
#cherrypy.tools.mako(filename="storm.html", directories=MEDIA_DIR)
def index(self, name=None):
return {'size': len(params["dashboards"][self.pagename])}
#cherrypy.expose
def submit_data(self, idx):
idx = int(idx)
chart = params["dashboards"][self.pagename][idx]
# Sample page that displays the number of records in "table"
# Open a cursor, using the DB connection for the current thread
c = cherrypy.thread_data.dbdict[chart["dbname"]].cursor()
print 'query being executed......'
c.execute(chart["command"])
print 'query being fetched......'
res = c.fetchall()
c.close()
# construct a JSON object from query result
jsres = []
jsres.append(chart["selected"])
.
.
.
return json.dumps(jsres)
class Root(object):
#storm = Storm("first")
#cherrypy.expose
#cherrypy.tools.mako(filename="index.html", directories=MEDIA_DIR)
def index(self, name=None):
return {}
config = {'/media':
{'tools.staticdir.on': True,
'tools.staticdir.dir': MEDIA_DIR,
}
}
root = Root()
storm = Storm("first")
mapper = cherrypy.dispatch.RoutesDispatcher()
mapper.connect('home','/',controller=root, action='index')
mapper.connect('st','/st/',controller=storm, action='index')
conf = {'/': {'request.dispatch': mapper}}
app=cherrypy.tree.mount(root=None,config=conf)
cherrypy.quickstart(app)

mapper.connect('st','/st/submit_data',controller=storm, action='submit_data')
Routes explicitly list all their handlers, and because of that also don't need to be exposed. Based on this example, you might not want the routes dispatcher. The default dispatcher with multiple roots mounted would work fine.

How to catch all exceptions with CherryPy?

I use CherryPy to run a very simple web server. It is intended to process the GET parameters and, if they are correct, do something with them.
import cherrypy
class MainServer(object):
def index(self, **params):
# do things with correct parameters
if 'a' in params:
print params['a']
index.exposed = True
cherrypy.quickstart(MainServer())
For example,
http://127.0.0.1:8080/abcde:
404 Not Found
The path '/abcde' was not found.
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\cherrypy\_cprequest.py", line 656, in respond
response.body = self.handler()
File "C:\Python27\lib\site-packages\cherrypy\lib\encoding.py", line 188, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "C:\Python27\lib\site-packages\cherrypy\_cperror.py", line 386, in __call__
raise self
NotFound: (404, "The path '/abcde' was not found.")
Powered by CherryPy 3.2.4
I am trying to catch this exception and show a blank page because the clients do not care about it. Specifically, the result would be an empty body, no matter the url or query string that resulted in an exception.
I had a look at documentation on error handling cherrypy._cperror, but I did not find a way to actually use it.
Note: I gave up using CherryPy and found a simple solution using BaseHTTPServer (see my answer below)

Docs somehow seem to miss this section. This is what I found while looking for detailed explanation for custom error handling from the source code.
Custom Error Handling
Anticipated HTTP responses
The 'error_page' config namespace can be used to provide custom HTML output for
expected responses (like 404 Not Found). Supply a filename from which the
output will be read. The contents will be interpolated with the values
%(status)s, %(message)s, %(traceback)s, and %(version)s using plain old Python
string formatting.
_cp_config = {
'error_page.404': os.path.join(localDir, "static/index.html")
}
Beginning in version 3.1, you may also provide a function or other callable as
an error_page entry. It will be passed the same status, message, traceback and
version arguments that are interpolated into templates
def error_page_402(status, message, traceback, version):
return "Error %s - Well, I'm very sorry but you haven't paid!" % status
cherrypy.config.update({'error_page.402': error_page_402})
Also in 3.1, in addition to the numbered error codes, you may also supply
error_page.default to handle all codes which do not have their own error_page
entry.
Unanticipated errors
CherryPy also has a generic error handling mechanism: whenever an unanticipated
error occurs in your code, it will call
Request.error_response to
set the response status, headers, and body. By default, this is the same
output as
HTTPError(500). If you want to provide
some other behavior, you generally replace "request.error_response".
Here is some sample code that shows how to display a custom error message and
send an e-mail containing the error
from cherrypy import _cperror
def handle_error():
cherrypy.response.status = 500
cherrypy.response.body = [
"<html><body>Sorry, an error occurred</body></html>"
]
sendMail('error#domain.com',
'Error in your web app',
_cperror.format_exc())
#cherrypy.config(**{'request.error_response': handle_error})
class Root:
pass
Note that you have to explicitly set
response.body
and not simply return an error message as a result.

Choose what's most suitable for you: Default Methods, Custom Error Handling.
I don't think you should use BaseHTTPServer. If your app is that simple, just get a lightweight framework (e. g. Flask), even though it might be a bit overkill, OR stay low level but still within the WSGI standard and use a WSGI-compliant server.

CherryPy IS catching your exception. That's how it returns a valid page to the browser with the caught exception.
I suggest you read through all the documentation. I realize it isn't the best documentation or organized well, but if you at least skim through it the framework will make more sense. It is a small framework, but does almost everything you'd expect from a application server.
import cherrypy
def show_blank_page_on_error():
"""Instead of showing something useful to developers but
disturbing to clients we will show a blank page.
"""
cherrypy.response.status = 500
cherrypy.response.body = ''
class Root():
"""Root of the application"""
_cp_config = {'request.error_response': show_blank_page_on_error}
#cherrypy.expose
def index(self):
"""Root url handler"""
raise Exception
See this for the example in the documentation on the page mentioned above for further reference.

You can simply use a try/except clause:
try:
cherrypy.quickstart(MainServer())
except: #catches all errors, including basic python errors
print("Error!")
This will catch every single error. But if you want to catch only cherrypy._cperror:
from cherrypy import _cperror
try:
cherrypy.quickstart(MainServer())
except _cperror.CherryPyException: #catches only CherryPy errors.
print("CherryPy error!")
Hope this helps!

import cherrypy
from cherrypy import HTTPError
def handle_an_exception():
cherrypy.response.status = 500
cherrypy.response.headers['content-type'] = 'text/plain;charset=UTF-8'
cherrypy.response.body = b'Internal Server Error'
def handle_a_404(status=None, message=None, version=None, traceback=None):
cherrypy.response.headers['content-type'] = 'text/plain;charset=UTF-8'
return f'Error page for 404'.encode('UTF-8')
def handle_default(status=None, message=None, version=None, traceback=None):
cherrypy.response.headers['content-type'] = 'text/plain;charset=UTF-8'
return f'Default error page: {status}'.encode('UTF-8')
class Root:
"""Root of the application"""
_cp_config = {
# handler for an unhandled exception
'request.error_response': handle_an_exception,
# specific handler for HTTP 404 error
'error_page.404': handle_a_404,
# default handler for any other HTTP error
'error_page.default': handle_default
}
#cherrypy.expose
def index(self):
"""Root url handler"""
raise Exception("an exception")
#cherrypy.expose
def simulate400(self):
raise HTTPError(status=400, message="Bad Things Happened")
cherrypy.quickstart(Root())
Test with:
http://127.0.0.1:8080/
http://127.0.0.1:8080/simulate400
http://127.0.0.1:8080/missing

Though this was the one of the top results when I searched for cherrypy exception handling, accepted answer did not fully answered the question. Following is a working code against cherrypy 14.0.0
# Implement handler method
def exception_handler(status, message, traceback, version)
# Your logic goes here
class MyClass()
# Update configurations
_cp_config = {"error_page.default": exception_handler}
Note the method signature. Without this signature your method will not get invoked.Following are the contents of method parameters,
status : HTTP status and a description
message : Message attached to the exception
traceback : Formatted stack trace
version : Cherrypy version

Maybe you could use a 'before_error_response' handler from cherrypy.tools
#cherrypy.tools.register('before_error_response', priority=90)
def handleexception():
cherrypy.response.status = 500
cherrypy.response.body = ''
And don't forget to enable it:
tools.handleexception.on = True

I gave up using CherryPy and ended up using the follwing code, which solves the issue in a few lines with the standard BaseHTTPServer:
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
from urlparse import urlparse, parse_qs
class GetHandler(BaseHTTPRequestHandler):
def do_GET(self):
url = urlparse(self.path)
d = parse_qs(url[4])
if 'c' in d:
print d['c'][0]
self.send_response(200)
self.end_headers()
return
server = HTTPServer(('localhost', 8080), GetHandler)
server.serve_forever()

Bottle.py error routing

Bottle.py ships with an import to handle throwing HTTPErrors and route to a function.
Firstly, the documentation claims I can (and so do several examples):
from bottle import error
#error(500)
def custom500(error):
return 'my custom message'
however, when importing this statement error is unresolved but on running the application ignores this and just directs me to the generic error page.
I found a way to get around this by:
from bottle import Bottle
main = Bottle()
#Bottle.error(main, 500)
def custom500(error):
return 'my custom message'
But this code prevents me from embedding my errors all in a separate module to control the nastiness that would ensue if I kept them in my main.py module because the first argument has to be a bottle instance.
So my questions:
Has anyone else experienced this?
why doesn't error seem to resolve in only my case (I installed from pip install bottle)?
Is there a seamless way to import my error routing from a separate python module into the main application?

If you want to embed your errors in another module, you could do something like this:
error.py
def custom500(error):
return 'my custom message'
handler = {
500: custom500,
}
app.py
from bottle import *
import error
app = Bottle()
app.error_handler = error.handler
#app.route('/')
def divzero():
return 1/0
run(app)

This works for me:
from bottle import error, run, route, abort
#error(500)
def custom500(error):
return 'my custom message'
#route("/")
def index():
abort("Boo!")
run()

In some cases I find it's better to subclass Bottle. Here's an example of doing that and adding a custom error handler.
#!/usr/bin/env python3
from bottle import Bottle, response, Route
class MyBottle(Bottle):
def __init__(self, *args, **kwargs):
Bottle.__init__(self, *args, **kwargs)
self.error_handler[404] = self.four04
self.add_route(Route(self, "/helloworld", "GET", self.helloworld))
def helloworld(self):
response.content_type = "text/plain"
yield "Hello, world."
def four04(self, httperror):
response.content_type = "text/plain"
yield "You're 404."
if __name__ == '__main__':
mybottle = MyBottle()
mybottle.run(host='localhost', port=8080, quiet=True, debug=True)

Why does Twisted think I'm calling request.finish() twice when I am not?

This is an annoying problem I am having with Twisted.web. Basically, I have a class that inherits from twisted.web.resource.Resource and adds some default stuff to Mako templates:
from twisted.web.resource import Resource
from mako.lookup import TemplateLookup
from project.session import SessionData
from project.security import make_nonce
class Page(Resource):
template = ""
def display(self, request, **kwargs):
session = SessionData(request.getSession())
if self.template:
templates = TemplateLookup(directories=['templates'])
template = templates.get_template(self.template)
return template.render(user=session.user,
info=session.info,
current_path=request.path,
nonce=make_nonce(session),
**kwargs)
else:
return ""
Then, and I have narrowed the problem down to this small class (which I tested), I write a resource which inherits from Page:
class Test(pages.Page):
def render_GET(self, request):
return "<form method='post'><input type='submit'></form>"
def render_POST(self, request):
request.redirect("/test")
request.finish()
I'd like to note that, in every other case, if request.finish() isn't the last line in a function, then I return immediately after it.
Anyways, I add this class to the site at /test and when I navigate there, I get a submit button. I click the submit button, and in the console I get:
C:\Python26\lib\site-packages\twisted\web\server.py:200: UserWarning: Warning! request.finish called twice.
self.finish()
But, I get this ONLY the first time I submit the page. Every other time, it's fine. I would just ignore this, but it's been nagging at me, and I can't for the life of me figure out why it's doing this at all, and why only the first time the page is submitted. I can't seem to find anything online, and even dropping print statements and tracebacks in the request.finish() code didn't reveal anything.
edit
This morning I tried adding a second request.finish() line to the resource, and it still only gave me the error one time. I suppose it will only warn about it in a resource once -- maybe per run of the program, or per session, I'm not sure. In any case, I changed it to:
class Test(pages.Page):
def render_GET(self, request):
return "<form method='post'><input type='submit'></form>"
def render_POST(self, request):
request.redirect("/test")
request.finish()
request.finish()
and just got two messages, one time. I still have no idea why I can't redirect the request without it saying I finished it twice (because I can't redirect without request.finish()).

Short Answer
It has to be:
request.redirect("/test")
request.finish()
return twisted.web.server.NOT_DONE_YET
Long Answer
I decided to go sifting through some Twisted source code. I first added a traceback to the area that prints the error if request.finish() is called twice:
def finish(self):
import traceback #here
"""
Indicate that all response data has been written to this L{Request}.
"""
if self._disconnected:
raise RuntimeError(
"Request.finish called on a request after its connection was lost; "
"use Request.notifyFinish to keep track of this.")
if self.finished:
warnings.warn("Warning! request.finish called twice.", stacklevel=2)
traceback.print_stack() #here
return
#....
...
File "C:\Python26\lib\site-packages\twisted\web\server.py", line 200, in render
self.finish()
File "C:\Python26\lib\site-packages\twisted\web\http.py", line 904, in finish
traceback.print_stack()
I went in and checked out render in twisted.web.server and found this:
if body == NOT_DONE_YET:
return
if type(body) is not types.StringType:
body = resource.ErrorPage(
http.INTERNAL_SERVER_ERROR,
"Request did not return a string",
"Request: " + html.PRE(reflect.safe_repr(self)) + "<br />" +
"Resource: " + html.PRE(reflect.safe_repr(resrc)) + "<br />" +
"Value: " + html.PRE(reflect.safe_repr(body))).render(self)
if self.method == "HEAD":
if len(body) > 0:
# This is a Bad Thing (RFC 2616, 9.4)
log.msg("Warning: HEAD request %s for resource %s is"
" returning a message body."
" I think I'll eat it."
% (self, resrc))
self.setHeader('content-length', str(len(body)))
self.write('')
else:
self.setHeader('content-length', str(len(body)))
self.write(body)
self.finish()
body is the result of rendering a resource, so once body is populated, in the example case given in my question, finish has already been called on this request object (since self is passed from this method to the resource's render method).
From looking at this code it becomes apparent that by returning NOT_DONE_YET I would avoid the warning.
I could have also changed the last line of that method to:
if not self.finished:
self.finish()
but, in the interest of not modifying the library, the short answer is:
after calling request.redirect() you must call request.finish() and then return twisted.web.server.NOT_DONE_YET
More
I found some documentation about this. It isn't related to redirecting a request, but instead rendering a resource, using request.write(). It says to call request.finish() and then return NOT_DONE_YET. From looking at the code in render() I can see why that is the case.