I already have a REST API with GAE python built using webapp2. I was looking at protorpc messages used in protorpc and Cloud Enpoints and really like how I can define the request and responses. Is there a way to incorporate that into my webapp2 handlers?
Firstly I use a decorator on the webapp2 method. I define the decorator as follows*:
# Takes webapp2 request (self.request on baseHandler) and converts to defined protoRpc object
def jsonMethod(requestType, responseType, http_method='GET'):
"""
NB: if both URL and POST parameters are used, do not use 'required=True' values in the protorpc Message definition
as this will fail on one of the sets of parms
"""
def jsonMethodHandler(handler):
def jsonMethodInner(self, **kwargs):
requestObject = getJsonRequestObject(self, requestType, http_method, kwargs)
logging.info(u'request={0}'.format(requestObject))
response = handler(self, requestObject) # Response object
if response:
# Convert response to Json
responseJson = protojson.encode_message(response)
else:
responseJson = '{}'
logging.info(u'response json={0}'.format(responseJson))
if self.response.headers:
self.response.headers['Content-Type'] = 'application/json'
if responseJson:
self.response.write(responseJson)
self.response.write('')
return jsonMethodInner
return jsonMethodHandler
The jsonMethod decorator uses a protorpc message for 'requestType' and 'responseType'.
I have constrained the http_method to be either GET, POST or DELETE for a method; you may wish to change this.
Note that this decorator must be applied to instance methods on a webapp2.RequestHandler class (see the example below) as it needs to access the webapp2 request and response objects.
The protorpc message is populated in getJsonRequestObject():
def getJsonRequestObject(self, requestType, http_method, kwargs):
"kwargs: URL keywords eg: /api/test/<key:\d+> => key"
"request.GET: used for 'GET' URL query string arguments"
"request.body: used for 'POST' or 'DELETE' form fields"
logging.info(u'URL parameters: {0}'.format(kwargs))
if http_method == 'POST' or http_method == 'DELETE':
requestJson = self.request.body
if requestJson == None:
requestJson = '' # Cater for no body (eg: IE10)
try:
logging.info("Content type = {}".format(self.request.content_type))
logRequest = requestJson if len(requestJson) < 1024 else requestJson[0:1024]
try:
logging.info(u'Request JSON: {0}'.format(logRequest))
except:
logging.info("Cannot log request JSON (invalid character?)")
postRequestObject = protojson.decode_message(requestType, requestJson)
except:
logError()
raise
if self.request.query_string:
# combine POST and GET parameters [GET query string overrides POST field]
getRequestObject = protourlencode.decode_message(requestType, self.request.query_string)
requestObject = combineRequestObjects(requestType, getRequestObject, postRequestObject)
else:
requestObject = postRequestObject
elif http_method == 'GET':
logging.info(u'Query strings: {0}'.format(self.request.query_string))
requestObject = protourlencode.decode_message(requestType, self.request.query_string)
logging.info(u'Request object: {0}'.format(requestObject))
else:
raise ValidationException(u'Invalid HTTP method: {0}'.format(http_method))
if len(kwargs) > 0:
#Combine URL keywords (kwargs) with requestObject
queryString = urllib.urlencode(kwargs)
keywordRequestObject = protourlencode.decode_message(requestType, queryString)
requestObject = combineRequestObjects(requestType, requestObject, keywordRequestObject)
return requestObject
getJsonRequestObject() handles GET, POST and webapp2 URL arguments (note: these are entered as kwargs).
combineRequestObjects() combines two objects of the requestType message:
def combineRequestObjects(requestType, request1, request2):
"""
Combines two objects of requestType; Note that request2 values will override request1 values if duplicated
"""
members = inspect.getmembers(requestType, lambda a:not(inspect.isroutine(a)))
members = [m for m in members if not m[0].startswith('_')]
for key, value in members:
val = getattr(request2, key)
if val:
setattr(request1, key, val)
return request1
Finally, a decorated webapp2 method example:
from protorpc import messages, message_types
class FileSearchRequest(messages.Message):
"""
A combination of file metadata and file information
"""
filename = messages.StringField(1)
startDateTime = message_types.DateTimeField(2)
endDateTime = message_types.DateTimeField(3)
class ListResponse(messages.Message):
"""
List of strings response
"""
items = messages.StringField(1, repeated=True)
...
class FileHandler(webapp2.RequestHandler):
#jsonMethod(FileSearchRequest, ListResponse, http_method='POST')
def searchFiles(self, request):
# Can now use request.filename etc
...
return ListResponse(items=items)
Hopefully this will give you some idea of how to go about implementing your own webapp2/protorpc framework.
You can also check and see how Cloud Endpoints is implementing their protorpc message handling. You may also need to dive into the protorpc code itself.
Please note that I have attempted to simplify my existing implementation, so you may come across various issues that you will need to address in your implementation.
In addition methods like 'logError()' and classes like 'ValidationException' are non-standard, so you will need to replace them as you see fit.
You may also wish to remove the logging at some point.
Related
Currently, I have a working API that uses Connexion and receives an OpenAPI specification:
connexion_app.add_api(
"openapi.yaml",
options={"swagger_ui": False},
validate_responses=True,
strict_validation=True, # Changing this also didn't help
)
A response gets validated in the following order:
Check if API-Key is valid
Validate if the request body contains all necessary parameters
Validate message-signature
Handle request and send response
The verification of the API-Key is done via the OpenAPI spec:
securitySchemes:
apiKeyAuth:
type: apiKey
in: header
name: API-Key
x-apikeyInfoFunc: server.security.check_api_key
security:
- apiKeyAuth: []
The validation is also done via the OpenAPI spec.
The signature gets verified in the endpoint:
if not verify_signature(kwargs):
abort(401, "Signature could not be verified")
Where verify_signature is basically this:
def verify_signature(request) -> bool:
"""Calculate the signature using the header and data."""
signature = re.findall(r'"([A-Za-z0-9+/=]+)"', connexion.request.headers.get("Message-Signature", ""))
created = re.findall(r"created=(\d+)", connexion.request.headers.get("Message-Signature", ""))
if len(signature) == 0:
abort(401, "No valid Signature found.")
if len(created) == 0:
abort(401, "No valid created timestamp found.")
signature = signature[0]
created = int(created[0])
method, path, host, api_key, content_type = _get_attributes_from_request()
message = create_signature_message(request["body"], created, method, path, host, api_key, content_type)
recreated_signature = _encode_message(message)
return recreated_signature == str(signature)
For security purposes I would like to swap 2. and 3.:
Check if API-Key is valid
Validate message-signature
Validate if the request body contains all necessary parameters
Handle request and send response
The problem is that Connexion validates the body before I get to my endpoint in which I execute my Python code such as verify_signature.
I tried adding the following to my OpenAPI.yaml:
signatureAuth:
type: http
scheme: basic
x-basicInfoFunc: server.security.verify_signature
security:
- apiKeyAuth: []
signatureAuth: []
But I think this is the wrong approach since I think this is only used as a simple verification method and I get the following error message:
No authorization token provided.
Now to my question:
Is there a way to execute a function which receives the whole request that gets executed before Connexion validates the body?
Yes you can use the Connexion before_request annotation so it runs a function on a new request before validating the body. Here's an example that logs the headers and content:
import connexion
import logging
from flask import request
logger = logging.getLogger(__name__)
conn_app = connexion.FlaskApp(__name__)
#conn_app.app.before_request
def before_request():
for h in request.headers:
logger.debug('header %s', h)
logger.debug('data %s', request.get_data())
I'm using the swagger editor (OpenApi 2) for creating flask apis in python. When you define a model in swagger and use it as a schema for the body of a request, swagger validates the body before handing the control to you in the X_controller.py files.
I want to add some code before that validation happens (for printing logs for debugging purposes). Swagger just prints to stdout errors like the following and they are not useful when you have a lot of fields (I need the key that isn't valid).
https://host/path validation error: False is not of type 'string'
10.255.0.2 - - [20/May/2020:20:20:20 +0000] "POST /path HTTP/1.1" 400 116 "-" "GuzzleHttp/7"
I know tecnically you can remove the validations in swagger and do them manually in your code but I want to keep using this feature, when it works it's awesome.
Any ideas on how to do this or any alternative to be able to log the requests are welcome.
After some time studying the matter this is what I learnt.
First let's take a look at how a python-flask server made with Swagger Editor works.
Swagger Editor generates the server stub through Swagger Codegen using the definition written in Swagger Editor. This server stub returned by codegen uses the framework Connexion on top of flask to handle all the HTTP requests and responses, including the validation against the swagger definition (swagger.yaml).
Connexion is a framework that makes it easy to develop python-flask servers because it has a lot of functionality you'd have to make yourself already built in, like parameter validation. All we need to do is replace (in this case modify) these connexion validators.
There are three validators:
ParameterValidator
RequestBodyValidator
ResponseValidator
They get mapped to flask by default but we can replace them easily in the __main__.py file as we'll see.
Our goal is to replace the default logs and default error response to some custom ones. I'm using a custom Error model and a function called error_response() for preparing error responses, and Loguru for logging the errors (not mandatory, you can keep the original one).
To make the changes needed, looking at the connexion validators code, we can see that most of it can be reused, we only need to modify:
RequestBodyValidator: __call__() and validate_schema()
ParameterValidator: __call__()
So we only need to create two new classes that extend the original ones, and copy and modify those functions.
Be careful when copying and pasting. This code is based on connexion==1.1.15. If your are on a different version you should base your classes on it.
In a new file custom_validators.py we need:
import json
import functools
from flask import Flask
from loguru import logger
from requests import Response
from jsonschema import ValidationError
from connexion.utils import all_json, is_null
from connexion.exceptions import ExtraParameterProblem
from swagger_server.models import Error
from connexion.decorators.validation import ParameterValidator, RequestBodyValidator
app = Flask(__name__)
def error_response(response: Error) -> Response:
return app.response_class(
response=json.dumps(response.to_dict(), default=str),
status=response.status,
mimetype='application/json')
class CustomParameterValidator(ParameterValidator):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
def __call__(self, function):
"""
:type function: types.FunctionType
:rtype: types.FunctionType
"""
#functools.wraps(function)
def wrapper(request):
if self.strict_validation:
query_errors = self.validate_query_parameter_list(request)
formdata_errors = self.validate_formdata_parameter_list(request)
if formdata_errors or query_errors:
raise ExtraParameterProblem(formdata_errors, query_errors)
for param in self.parameters.get('query', []):
error = self.validate_query_parameter(param, request)
if error:
response = error_response(Error(status=400, description=f'Error: {error}'))
return self.api.get_response(response)
for param in self.parameters.get('path', []):
error = self.validate_path_parameter(param, request)
if error:
response = error_response(Error(status=400, description=f'Error: {error}'))
return self.api.get_response(response)
for param in self.parameters.get('header', []):
error = self.validate_header_parameter(param, request)
if error:
response = error_response(Error(status=400, description=f'Error: {error}'))
return self.api.get_response(response)
for param in self.parameters.get('formData', []):
error = self.validate_formdata_parameter(param, request)
if error:
response = error_response(Error(status=400, description=f'Error: {error}'))
return self.api.get_response(response)
return function(request)
return wrapper
class CustomRequestBodyValidator(RequestBodyValidator):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
def __call__(self, function):
"""
:type function: types.FunctionType
:rtype: types.FunctionType
"""
#functools.wraps(function)
def wrapper(request):
if all_json(self.consumes):
data = request.json
if data is None and len(request.body) > 0 and not self.is_null_value_valid:
# the body has contents that were not parsed as JSON
return error_response(Error(
status=415,
description="Invalid Content-type ({content_type}), JSON data was expected".format(content_type=request.headers.get("Content-Type", ""))
))
error = self.validate_schema(data, request.url)
if error and not self.has_default:
return error
response = function(request)
return response
return wrapper
def validate_schema(self, data, url):
if self.is_null_value_valid and is_null(data):
return None
try:
self.validator.validate(data)
except ValidationError as exception:
description = f'Validation error. Attribute "{exception.validator_value}" return this error: "{exception.message}"'
logger.error(description)
return error_response(Error(
status=400,
description=description
))
return None
Once we have our validators, we have to map them to the flask app (__main__.py) using validator_map:
validator_map = {
'parameter': CustomParameterValidator,
'body': CustomRequestBodyValidator,
'response': ResponseValidator,
}
app = connexion.App(__name__, specification_dir='./swagger/', validator_map=validator_map)
app.app.json_encoder = encoder.JSONEncoder
app.add_api(Path('swagger.yaml'), arguments={'title': 'MyApp'})
If you also need to replace the validator I didn't use in this example, just create a custom child class of ResponseValidator and replace it on the validator_map dictionary in __main__.py.
Connexion docs:
https://connexion.readthedocs.io/en/latest/request.html
Forgive me for repeating an answer first posted at https://stackoverflow.com/a/73051652/1630244
Have you tried the Connexion before_request feature? Here's an example that logs the headers and content before Connexion validates the body:
import connexion
import logging
from flask import request
logger = logging.getLogger(__name__)
conn_app = connexion.FlaskApp(__name__)
#conn_app.app.before_request
def before_request():
for h in request.headers:
logger.debug('header %s', h)
logger.debug('data %s', request.get_data())
I'm writing unit tests for the Client class of client.py, which queries an API. Each test instantiates the client with c = client.Client("apikey"). Running one test at a time works fine, but running them all (e.g. with py.test) I get a 401: "Exception: Response 401: Unauthorized Access. Requests must contain a valid api-key."
I have a valid API key but this should not be included in the unit tests. I would appreciate an explanation of why "apikey" works for only one query. More specifically, how can I mock out the calls to the API? Below is an example unit test:
def testGetContextReturnFields(self):
c = client.Client("apikey")
contexts = c.getContext("foo")
assert(isinstance(contexts[0]["context_label"], str))
assert(contexts[0]["context_id"] == 0)
Separate out the tests for API calls and for the Client.getContext() method. For explicitly testing the API calls, patch a request object...
import client
import httpretty
import requests
from mock import Mock, patch
...
def testGetQueryToAPI(self):
"""
Tests the client can send a 'GET' query to the API, asserting we receive
an HTTP status code reflecting successful operation.
"""
# Arrange: patch the request in client.Client._queryAPI().
with patch.object(requests, 'get') as mock_get:
mock_get.return_value = mock_response = Mock()
mock_response.status_code = 200
# Act:
c = client.Client()
response = c._queryAPI("path", 'GET', {}, None, {})
# Assert:
self.assertEqual(response.status_code, 200)
# Repeat the same test for 'POST' queries.
And for testing getContext(), mock out the HTTP with httpretty...
#httpretty.activate
def testGetContextReturnFields(self):
"""
Tests client.getContext() for a sample term.
Asserts the returned object contains the corrcet fields and have contents as
expected.
"""
# Arrange: mock JSON response from API, mock out the API endpoint we expect
# to be called.
mockResponseString = getMockApiData("context_foo.json")
httpretty.register_uri(httpretty.GET,
"http://this.is.the.url/query",
body=mockResponseString,
content_type="application/json")
# Act: create the client object we'll be testing.
c = client.Client()
contexts = c.getContext("foo")
# Assert: check the result object.
self.assertTrue(isinstance(contexts, list),
"Returned object is not of type list as expected.")
self.assertTrue(("context_label" and "context_id") in contexts[0],
"Data structure returned by getContext() does not contain"
" the required fields.")
self.assertTrue(isinstance(contexts[0]["context_label"], str),
"The \'context_label\' field is not of type string.")
self.assertEqual(contexts[0]["context_id"], 0,
"The top context does not have ID of zero.")
This is the method in ReportRunner class in report_runner.py in my Flask-Restful app:
class ReportRunner(object):
def __init__(self):
pass
def setup_routes(self, app):
app.add_url_rule("/run_report", view_func=self.run_report)
def request_report(self, key):
# code #
def key_exists(self, key):
# code #
def run_report(self):
key = request.args.get("key", "")
if self.key_exists(key):
self.request_report(report_type, key)
return jsonify(message = "Success! Your report has been created.")
else:
response = jsonify({"message": "Error => report key not found on server."})
response.status_code = 404
return response
and the nose test calls the URL associated with that route
def setUp(self):
self.setup_flask()
self.controller = Controller()
self.report_runner = ReportRunner()
self.setup_route(self.report_runner)
def test_run_report(self):
rr = Report(key = "daily_report")
rr.save()
self.controller.override(self.report_runner, "request_report")
self.controller.expectAndReturn(self.report_runner.request_report("daily_report"), True )
self.controller.replay()
response = self.client.get("/run_report?key=daily_report")
assert_equals({"message": "Success! Your report has been created."}, response.json)
assert_equals(200, response.status_code)
and the test was failing with the following message:
AttributeError: 'Response' object has no attribute 'json'
but according to the docs it seems that this is how you do it. Do I change the return value from the method, or do I need to structure the test differently?
The test is now passing written like this:
json_response = json.loads(response.data)
assert_equals("Success! Your report has been created.", json_response["message"])
but I'm not clear on the difference between the two approaches.
According to Flask API Response object doesn't have attribute json (it's Request object that has it). So, that's why you get exception. Instead, it has generic method get_data() that returns the string representation of response body.
json_response = json.loads(response.get_data())
assert_equals("Success! Your report has been created.", json_response.get("message", "<no message>"))
So, it's close to what you have except:
get_data() is suggested instead of data as API says: This should not be used and will eventually get deprecated.
reading value from dictionary with get() to not generate exception if key is missing but get correct assert about missing message.
Check this Q&A also.
I adapted this sample code in order to get webapp2 sessions to work on Google App Engine.
What do I need to do to be able to return webapp2.Response objects from a handler that's inheriting from a BaseHandler that overrides the dispatch method?
Here's a demonstration of the kind of handler I want to write:
import webapp2
import logging
from webapp2_extras import sessions
class BaseHandler(webapp2.RequestHandler):
def dispatch(self):
# Get a session store for this request.
self.session_store = sessions.get_store(request=self.request)
try:
# Dispatch the request.
webapp2.RequestHandler.dispatch(self)
finally:
# Save all sessions.
self.session_store.save_sessions(self.response)
class HomeHandler(BaseHandler):
def get(self):
logging.debug('In homehandler')
response = webapp2.Response()
response.write('Foo')
return response
config = {}
config['webapp2_extras.sessions'] = {
'secret_key': 'some-secret-key',
}
app = webapp2.WSGIApplication([
('/test', HomeHandler),
], debug=True, config=config)
This code is obviously not working, since BaseHandler always calls dispatch with self. I've looked through the code of webapp2.RequestHandler, but it seriously eludes me how to modify my BaseHandler (or perhaps set a custom dispatcher) such that I can simply return response objects from inheriting handlers.
Curiously, the shortcut of assigning self.response = copy.deepcopy(response) does not work either.
You're mixing the two responses in one method. Use either
return webapp2.Response('Foo')
or
self.response.write('Foo')
...not both.
I took a look at webapp2.RequestHandler and noticed that returned values are just passed up the stack.
A solution which works for me is to use the returned Response when one is returned from the handler, or self.response when nothing is returned.
class BaseHandler(webapp2.RequestHandler):
def dispatch(self):
# Get a session store for this request.
self.session_store = sessions.get_store(request=self.request)
response = None
try:
# Dispatch the request.
response = webapp2.RequestHandler.dispatch(self)
return response
finally:
# Save all sessions.
if response is None:
response = self.response
self.session_store.save_sessions(response)
While I was playing I noticed that my session stored as a secure cookie was not getting updated when exceptions were raised in the handler.