Because of project specificity I have to write my own model validator for Flask-restplus API app. Simply speaking - when validation error occurs, its format and status code (400) is not proper. It should return JSON object with messages in particular format with status code 422.
What I do is more or less this:
ns = api.namespace('somenamespace', description='blabla')
class MyModel(MyBaseModel):
def __init__(self):
self.id = fields.Integer()
self.name = fields.String()
my_model = api.model('MyModel', MyModel())
#api.marshal_list_with(my_model, envelope='json')
#ns.route('/')
class SomeClass(Resource):
#api.expect(my_model, validate=False)
#api.doc(responses={
200: 'Success',
401: 'Authentication Error',
403: 'Requested resource unavailable',
409: 'Conflict, document already exists',
422: 'Validation Error'
})
def post(self):
"""
Save single document in the database.
:return:
"""
request_payload = json.loads(request.data)
validated_payload, payload_errors = some_validation(request_payload)
if payload_errors:
return jsonify(payload_errors), 422
else:
return jsonify({'response': 'ok})
Instance of `MyModel` behaves basically like a dict, so no problem in registration. Problem is that when I send data in `-d`, be it through `curl` from command line, or swagger, I constantly get `400` instead of `422`. I assume this is due to the default, built-in validation of input data based on `MyModel`. This is cool, but I have to omit it, and apply my own validation.
in the documentation and as #CloC says, one method is specifying the model as
your_model = ns.model('YourModel', {
'id': fields.Integer(
description='The user id'
)
'name': fields.String(
description='The user name'
)
})
... > profit
#ns.route('/', methods=["post"])
#ns.doc(params={
"id": "the user id (int)",
"name": "the user name (str)"
})
class SomeClass(Resource):
#ns.expect(your_model) # this defines the request
# #ns.marshal_list_with(your_model, envelope='json') # this defines the response
#ns.response(200, 'Success')
... > reponses
def post(self):
etc...
return <response with format as model in marshal>
you might want to re-define the response model though unless you will return something of the form that you put in.. also maybe specify marshal_with as you do not return a list?
Related
During the running test case of my application, I keep on seeing test failed even when the data is well formatted. The following are my code snippets:
I was able to create a new user through its application interface, but trying its behaviour, it was giving me an unexpected error status code, 422. I don't really know what was going wrong with the snippets. Here I have included all the following code for better look into the stated issue.
For the endpoint /users
#app.route('/users/, methods=['POST'])
def create_user():
try:
body = request.get_json((
new_user = body get('user_name', None)
if user_name is None:
about(405)
new_entry = User(new_user)
new_entry.insert()
return jsonify({
'success': True
)}
except:
abort(422)
Here is my model_class:
class User(db.Model):
__tablename__='users'
id = Column(Integer, primary_key=True)
name = Column(String, nullable=False)
score = Column(Integer, nullable=False)
def __init__(self, name, score=0):
self.name = name
self.score = score
def insert(self):
db.session.add(self)
db.session.commit()
def format(self):
return {
'id': self.id,
'name': self.name,
'score': self.score
And here is my test_case_file:
class TriviaTestCase(unit test.TestCase):
def setUp(self):
self.app = create_app()
self.client = self.app.test_client
self.database_path = "PostgreSQL://postgres:postgresspass#localhos:5432/user_test_db"
setup_db(self.app, self database_path)
self.new_user = {
"username":"P.Son",
"score": 0
}
def test_create_user(self):
res = self.client().post("/users", json=self.new_user)
data = json.loads(res.data)
self.assertEqual(res.status_code, 200)
self.assertEqual(data['success'], True)
if __name__=="__main__":
unittest.main()
Output of my test:
============================
FAIL: test_create_user (__main__.TriviaTestCase)
----------------------------
Traceback (most recent call last):
File "C:\...\test_flaskr.py", line 201, in test_create_user
self.assertEqual(res.status_code, 200)
AssertionError: 422 != 200
Note Other endpoints pass the test but the above endpoint has been failing test. I don't know what was wrong with the snippets I have written.
First of all, you should always be logging such errors when they happen in your endpoints.
Second, you are returning the 422 error yourself in case of "any" exception and there is some in your code. One of which is the fact that you are just passing user_name to the constructor of the User class but there must be a score variable present too.
Another problem is that you should pass key-value arguments to the User class constructor, not a dictionary.
is should be something like this:
body = request.get_json()
user_name = body.get('user_name', '')
score = body.get('score', '')
if not all([user_name, score]):
# return some customized error here
new_entry = User(user_name=user_name, score=score)
Also, you should consider some form of input validation rather than just checking the data yourself. Something like Marshmellow would be fine.
ive got an api that takes in an id
http://127.0.0.1:5000/api/v1/resources/books?id=u3qR4Ps4TbATrg97
looks like that
what im trying to do after that is add something to the end of the url, for example
http://127.0.0.1:5000/api/v1/resources/books?id=u3qR4Ps4TbATrg97uid=something
im not 100% sure if this is possible
# Create some test data for our catalog in the form of a list of dictionaries.
books = [
{'id': 'u3qR4Ps4TbATrg97',
'uid': 'what',
'title': 'A Fire Upon the Deep',
'author': 'Vernor Vinge',
'first_sentence': 'The coldsleep itself was dreamless.',
'year_published': '1992'}
]
#app.route('/api/v1/resources/books', methods=['GET'])
def api_id():
# Check if an ID was provided as part of the URL.
# If ID is provided, assign it to a variable.
# If no ID is provided, display an error in the browser.
if 'id' and 'uid' in request.args:
id = str(request.args['id'])
uid = str(request.args['uid'])
else:
return "Error: No id field provided. Please specify an id."
results = []
for book in books:
if book['id'] == id:
results.append(book)
if book['uid'] == uid:
results.append(book)
this is what i have so far, mostly copy pasted from here
thats no the whole file just the important bits i can think of
You can add two inputs inside the GET query like this
http://127.0.0.1:5000/api/v1/resources/books?id=u3qR4Ps4TbATrg97&uid=something
Just put an & in between!
Use request.args.get method to get parameters from your url. Also add & to your URL as a parameter separator.
from flask import Flask, request
app = Flask(__name__)
#app.route('/api/v1/resources/books')
def books():
id_ = request.args.get('id')
uid = request.args.get('uid')
return f'id: {id_}, uid: {uid}'
app.run()
Open http://127.0.0.1:5000/api/v1/resources/books?id=u3qR4Ps4TbATrg97&uid=something
in browser and you'll get:
id: u3qR4Ps4TbATrg97, uid: something
Multiple parameters|arguments are passed with & character. ?params1=5¶ms2=3. For your example: http://127.0.0.1:5000/api/v1/resources/books?id=u3qR4Ps4TbATrg97&uid=what. For the code, I would do:
from flask import Flask, request, jsonify, make_response
app = Flask(__name__)
# Create some test data for our catalog in the form of a list of dictionaries.
books = [
{
"id": "u3qR4Ps4TbATrg97",
"uid": "what",
"title": "A Fire Upon the Deep",
"author": "Vernor Vinge",
"first_sentence": "The coldsleep itself was dreamless.",
"year_published": "1992",
}
]
#app.route("/api/v1/resources/books", methods=["GET"])
def api_id():
# Check if an ID was provided as part of the URL.
# If ID is provided, assign it to a variable.
# If no ID is provided, display an error in the browser.
if set(["id","uid"]).intersection(set(request.args)):
id_ = str(request.args["id"])
uid = str(request.args["uid"])
else:
return make_response(
jsonify({"message": "Error: No id field provided. Please specify an id."}),
400,
)
results = []
for book in books:
if book["id"] == id_:
results.append(book)
if book["uid"] == uid:
results.append(book)
response = make_response(
jsonify({"message": results}),
200,
)
response.headers["Content-Type"] = "application/json"
return response
This would return status code 400 if no match and 200 when match
I'm developing a REST api with Django and REST-framework. I have endpoint which takes a POST request with this kind of json:
{
"pipeline": ["Bayes"],
"material": [
"Rakastan iloisuutta!",
"Autojen kanssa pitää olla varovainen.",
"Paska kesä taas. Kylmää ja sataa"
]
}
It is a machine-learning analysis api and the json tells to use Bayes classifier to provided strings and return results. This works fine when I test it manually by doing the post requests. However, it breaks down when I try to write an unit test. I have the following test:
class ClassifyTextAPITests(APITestCase):
fixtures = ['fixtures/analyzerfixtures.json'] #suboptimal fixture since requires bayes.pkl in /assets/classifiers folder
def test_classification(self):
""" Make sure that the API will respond correctly when required url params are supplied.
"""
response = self.client.post(reverse('analyzer_api:classifytext'), {
"pipeline": ["Bayes"],
"material": [
"Rakastan iloisuutta!",
"Autojen kanssa pitää olla varovainen.",
"Paska kesä taas. Kylmää ja sataa",
]
})
self.assertTrue(status.is_success(response.status_code))
self.assertEqual(response.data[0], 1)
test fails everytime because of the latter assert gives "AssertionError: 'P' != 1"
Here is my view code:
class ClassifyText(APIView):
"""
Takes text snippet as a parameter and returns analyzed result.
"""
authentication_classes = (authentication.TokenAuthentication,)
permission_classes = (permissions.AllowAny,)
parser_classes = (JSONParser,)
def post(self, request, format=None):
try:
self._validate_post_data(request.data)
print("juttu", request.data["material"])
#create pipeline from request
pipeline = Pipeline()
for component_name in request.data["pipeline"]:
pipeline.add_component(component_name)
response = pipeline.execute_pipeline(request.data['material'])
status_code = status.HTTP_200_OK
except Exception as e:
response = {"message": "Please provide a proper data.",
"error": str(e) }
status_code = status.HTTP_400_BAD_REQUEST
return Response(response, status=status_code)
def _validate_post_data(self, data):
if "pipeline" not in data:
raise InvalidRequest("Pipeline field is missing. Should be array of components used in analysis. Available components at /api/classifiers")
if len(data["pipeline"]) < 1:
raise InvalidRequest("Pipeline array is empty.")
if "material" not in data:
raise InvalidRequest("Material to be analyzed is missing. Please provide an array of strings.")
if len(data["material"]) < 1:
raise InvalidRequest("Material to be analyzed is missing, array is empty. Please provide an array of strings.")
The really interesting part was when I fired the debugger to check what happens here. Turns out that the line
request.data['material']
gives the last entry of the list in in my request, in this case
"Paska kesä taas. Kylmää ja sataa"
However, while I inspect the contents of the request.data, it shows a querydict with lists pipeline and material as they are in request. Why do I get string instead of material list when I call request.data["material"] ? Is there something I have forgotten and I have to specify some kind of serializer? And why it works during normal execution but not with tests?
I'm using Django 1.8 with Python 3. Also, I'm not tying the view to any specific model.
Finally here is what my debugger shows when I put break points into view:
request.data:
QueryDict: {'material': ['Rakastan iloisuutta!', 'Autojen kanssa pitää olla varovainen.', 'Paska kesä taas. Kylmää ja sataa'], 'pipeline': ['Bayes']}
asd = request.data["material"]:
'Paska kesä taas. Kylmää ja sataa'
This is because QueryDict returns the last value of a list in __getitem__:
QueryDict.getitem(key)
Returns the value for the given key. If the key has more than one value, getitem() returns the last value. Raises django.utils.datastructures.MultiValueDictKeyError if the key does not exist. (This is a subclass of Python’s standard KeyError, so you can stick to catching KeyError.)
https://docs.djangoproject.com/en/1.8/ref/request-response/#django.http.QueryDict.getitem
If you post a form, in which a key maps to a list:
d = {"a": 123, "b": [1,2,3]}
requests.post("http://127.0.0.1:6666", data=d)
this is what you get in the request body:
a=123&b=1&b=2&b=3
Since the test method post the data as a form, what you get from request.data is a QueryDict (the same as request.POST), hence you get the last value in the list when getting request.data.
To get expected behavior, post the data as JSON in the request body (as in #Vladir Parrado Cruz's answer).
By default the QueryDict will return a single item from the list when doing a getitem call (or access by square brackets, such as you do in request.data['material'])
You can instead use the getlist method to return all values for the key:
https://docs.djangoproject.com/en/1.8/ref/request-response/#django.http.QueryDict.getlist
class ClassifyText(APIView):
"""
Takes text snippet as a parameter and returns analyzed result.
"""
authentication_classes = (authentication.TokenAuthentication,)
permission_classes = (permissions.AllowAny,)
parser_classes = (JSONParser,)
def post(self, request, format=None):
try:
self._validate_post_data(request.data)
print("juttu", request.data["material"])
print("juttu", request.data.getlist("material"]))
#create pipeline from request
pipeline = Pipeline()
for component_name in request.data["pipeline"]:
pipeline.add_component(component_name)
response = pipeline.execute_pipeline(request.data.getlist('material'))
status_code = status.HTTP_200_OK
except Exception as e:
response = {"message": "Please provide a proper data.",
"error": str(e) }
status_code = status.HTTP_400_BAD_REQUEST
return Response(response, status=status_code)
def _validate_post_data(self, data):
if "pipeline" not in data:
raise InvalidRequest("Pipeline field is missing. Should be array of components used in analysis. Available components at /api/classifiers")
if len(data["pipeline"]) < 1:
raise InvalidRequest("Pipeline array is empty.")
if "material" not in data:
raise InvalidRequest("Material to be analyzed is missing. Please provide an array of strings.")
if len(data["material"]) < 1:
raise InvalidRequest("Material to be analyzed is missing, array is empty. Please provide an array of strings.")
Try to do something like that on the test:
import json
def test_classification(self):
""" Make sure that the API will respond correctly when required url params are supplied.
"""
response = self.client.post(
reverse('analyzer_api:classifytext'),
json.dumps({
"pipeline": ["Bayes"],
"material": [
"Rakastan iloisuutta!",
"Autojen kanssa pitää olla varovainen.",
"Paska kesä taas. Kylmää ja sataa",
]
}),
content_type='application/json'
)
self.assertTrue(status.is_success(response.status_code))
self.assertEqual(response.data[0], 1)
Perhaps if you send the data as json it will work.
I'm trying to implement a simple Django service with a RESTful API using tastypie. My problem is that when I try to create a WineResource with PUT, it works fine, but when I use POST, it returns a HTTP 501 error. Reading the tastypie documentation, it seems like it should just work, but it's not.
Here's my api.py code:
class CustomResource(ModelResource):
"""Provides customizations of ModelResource"""
def determine_format(self, request):
"""Provide logic to provide JSON responses as default"""
if 'format' in request.GET:
if request.GET['format'] in FORMATS:
return FORMATS[request.GET['format']]
else:
return 'text/html' #Hacky way to prevent incorrect formats
else:
return 'application/json'
class WineValidation(Validation):
def is_valid(self, bundle, request=None):
if not bundle.data:
return {'__all__': 'No data was detected'}
missing_fields = []
invalid_fields = []
for field in REQUIRED_WINE_FIELDS:
if not field in bundle.data.keys():
missing_fields.append(field)
for key in bundle.data.keys():
if not key in ALLOWABLE_WINE_FIELDS:
invalid_fields.append(key)
errors = missing_fields + invalid_fields if request.method != 'PATCH' \
else invalid_fields
if errors:
return 'Missing fields: %s; Invalid fields: %s' % \
(', '.join(missing_fields), ', '.join(invalid_fields))
else:
return errors
class WineProducerResource(CustomResource):
wine = fields.ToManyField('wines.api.WineResource', 'wine_set',
related_name='wine_producer')
class Meta:
queryset = WineProducer.objects.all()
resource_name = 'wine_producer'
authentication = Authentication() #allows all access
authorization = Authorization() #allows all access
class WineResource(CustomResource):
wine_producer = fields.ForeignKey(WineProducerResource, 'wine_producer')
class Meta:
queryset = Wine.objects.all()
resource_name = 'wine'
authentication = Authentication() #allows all access
authorization = Authorization() #allows all access
validation = WineValidation()
filtering = {
'percent_new_oak': ('exact', 'lt', 'gt', 'lte', 'gte'),
'percentage_alcohol': ('exact', 'lt', 'gt', 'lte', 'gte'),
'color': ('exact', 'startswith'),
'style': ('exact', 'startswith')
}
def hydrate_wine_producer(self, bundle):
"""Use the provided WineProducer ID to properly link a PUT, POST,
or PATCH to the correct WineProducer instance in the db"""
#Workaround since tastypie has bug and calls hydrate more than once
try:
int(bundle.data['wine_producer'])
except ValueError:
return bundle
bundle.data['wine_producer'] = '/api/v1/wine_producer/%s/' % \
bundle.data['wine_producer']
return bundle
Any help is greatly appreciated! :-)
This usually means that you are sending the POST to a detail uri, e.g. /api/v1/wine/1/. Since POST means treat the enclosed entity as a subordinate, sending the POST to the list uri, e.g. /api/v1/wine/, is probably what you want.
Is there a better pattern for input validation than I'm using in this function?
https://github.com/nathancahill/clearbit-intercom/blob/133e4df0cfd1a146cedb3c749fc1b4fac85a6e1b/server.py#L71
Here's the same function without any validation. It's much more readable, it's short and to the point (9 LoC vs 53 LoC).
def webhook(clearbitkey, appid, intercomkey):
event = request.get_json()
id = event['data']['item']['id']
email = event['data']['item']['email']
person = requests.get(CLEARBIT_USER_ENDPOINT.format(email=email), auth=(clearbitkey, '')).json()
domain = person['employment']['domain']
company = requests.get(CLEARBIT_COMPANY_ENDPOINT.format(domain=domain), auth=(clearbitkey, '')).json()
note = create_note(person, company)
res = requests.post(INTERCOM_ENDPOINT,
json=dict(user=dict(id=id), body=note),
headers=dict(accept='application/json'),
auth=(appid, intercomkey))
return jsonify(note=res.json())
However, it doesn't handle any of these errors:
dict KeyError's (especially nested dicts)
HTTP errors
Invalid JSON
Unexpected responses
Is there a better pattern to follow? I looked into using a data validation library like voluptous but it seems like I'd still have the same problem of verbosity.
Your original code on github seems fine to me. It's a little over complicated, but also handle all cases of error. You can try to improve readability by abstract things.
Just for demonstration, I may write code like this:
class ValidationError(Exception):
"Raises when data validation fails"
pass
class CallExternalApiError(Exception):
"Raises when calling external api fails"
pass
def get_user_from_event(event):
"""Get user profile from event
:param dict event: request.get_json() result
:returns: A dict of user profile
"""
try:
event_type = event['data']['item']['type']
except KeyError:
raise ValidationError('Unexpected JSON format.')
if event_type != 'user':
return ValidationError('Event type is not supported.')
try:
id = event['data']['item']['id']
email = event['data']['item']['email']
except KeyError:
return ValidationError('User object missing fields.')
return {'id': id, 'email': email}
def call_json_api(request_function, api_name, *args, **kwargs):
"""An simple wrapper for sending request
:param request_function: function used for sending request
:param str api_name: name for this api call
"""
try:
res = request_function(*args, **kwargs)
except:
raise CallExternalApiError('API call failed to %s.' % api_name)
try:
return res.json()
except:
raise CallExternalApiError('Invalid response from %s.' % api_name)
#app.route('/<clearbitkey>+<appid>:<intercomkey>', methods=['POST'])
def webhook(clearbitkey, appid, intercomkey):
"""
Webhook endpoint for Intercom.io events. Uses this format for Clearbit and
Intercom.io keys:
/<clearbitkey>+<appid>:<intercomkey>
:clearbitkey: Clearbit API key.
:appid: Intercom.io app id.
:intercomkey: Intercom.io API key.
Supports User events, specifically designed for the User Created event.
Adds a note to the user with their employment and company metrics.
"""
event = request.get_json()
try:
return handle_event(event, clearbitkey, appid, intercomkey)
except (ValidationError, CallExternalApiError) as e:
# TODO: include **res_objs in response
return jsonify(error=str(e))
def handle_event(event):
"""Handle the incoming event
"""
user = get_user_from_event(event)
res_objs = dict(event=event)
person = call_json_api(
requests.get,
'Clearbit',
CLEARBIT_USER_ENDPOINT.format(email=user['email']),
auth=(clearbitkey, '')
)
res_objs['person'] = person
if 'error' in person:
raise CallExternalApiError('Error response from Clearbit.')
domain = person['employment']['domain']
company = None
if domain:
try:
company = call_json_api(
requests.get,
'Clearbit',
CLEARBIT_COMPANY_ENDPOINT.format(domain=domain),
auth=(clearbitkey, ''))
)
if 'error' in company:
company = None
except:
company = None
res_objs['company'] = company
try:
note = create_note(person, company)
except:
return jsonify(error='Failed to generate note for user.', **res_objs)
result = call_json_api(
requests.post,
'Intercom',
(INTERCOM_ENDPOINT, json=dict(user=dict(id=id), body=note),
headers=dict(accept='application/json'),
auth=(appid, intercomkey)
)
return jsonify(note=result, **res_objs)
I hope it helps.