In flask-restplus API , I need to make validation on request JSON data where I already defined request body schema with api.model. Basically I want to pass input JSON data to API function where I have to validate input JSON data before using API function. To do so, I used RequestParser for doing this task, but the API function was expecting proper JSON data as parameters after request JSON is validated and parsed. To do request JSON validation, first I have to parse received input JSON data, parse its JSON body, validate each then reconstructs it as JSON object, and pass to the API function. Is there any easier way to do this?
input JSON data
{
"body": {
"gender": "M",
"PCT": {
"value": 12,
"device": "roche_cobas"
},
"IL6": {
"value": 12,
"device": "roche_cobas"
},
"CRP": {
"value": 12,
"device": "roche_cobas"
}
}
}
my current attempt in flask
from flask_restplus import Api, Namespace, Resource, fields, reqparse, inputs
from flask import Flask, request, make_response, Response, jsonify
app = Flask(__name__)
api = Api(app)
ns = Namespace('')
feature = api.model('feature', {
'value': fields.Integer(required=True),
'time': fields.Date(required=True)
})
features = api.model('featureList', {
'age': fields.String,
'gender': fields.String(required=True),
'time': fields.Date,
'features': fields.List(fields.Nested(feature, required=True))
})
#ns.route('/hello')
class helloResource(Resource):
#ns.expect(features)
def post(self):
json_dict = request.json ## get input JSON data
## need to parse json_dict to validate expected argument in JSON body
root_parser = reqparse.RequestParser()
root_parser.add_argument('body', type=dict)
root_args = root_parser.parse_args()
jsbody_parser = reqparse.RequestParser()
jsbody_parser.add_argument('age', type=dict, location = ('body',))
jsbody_parser.add_argument('gender', type=dict, location=('body',))
## IL6, CRP could be something else, how to use **kwargs here
jsbody_parser.add_argument('IL6', type=dict, location=('body',))
jsbody_parser.add_argument('PCT', type=dict, location=('body',))
jsbody_parser.add_argument('CRP', type=dict, location=('body',))
jsbody_parser = jsbody_parser.parse_args(req=root_args)
## after validate each argument on input JSON request body, all needs to be constructed as JSON data
json_data = json.dumps(jsonify(jsbody_parser)) ## how can I get JSON again from jsbody_parser
func_output = my_funcs(json_data)
rest = make_response(jsonify(str(func_output)), 200)
return rest
if __name__ == '__main__':
api.add_namespace(ns)
app.run(debug=True)
update: dummy api function
Here is dummy function that expecting json data after validation:
import json
def my_funcs(json_data):
a =json.loads(json_data)
for k,v in a.iteritems():
print k,v
return jsonify(a)
current output of above attempt:
I have this on response body:
{
"errors": {
"gender": "'gender' is a required property"
},
"message": "Input payload validation failed"
}
obviously, request JSON input is not handled and not validated in my attempt. I think I have to pass json_dict to RequestParser object, but still can't validate request JSON here. how to make this happen?
I have to validate expected arguments from JSON body, after validation, I want to construct JSON body that gonna be used as a parameter for API function. How can I make this happen? any workaround to achieve this?
parsed JSON must pass to my_funcs
in my post, request JSON data should be parsed, such as age, gender should be validated as expected arguments in the request JSON, then jsonify added arguments as JSON and pass the my_funcs. how to make this happen easily in fl
I want to make sure flask should parse JSON body and add arguments as it expected, otherwise throw up error. for example:
{
"body": {
"car": "M",
"PCT": {
"value": 12,
"device": "roche_cobas"
},
"IL6": {
"device": "roche_cobas"
},
"CRP": {
"value": 12
}
}
}
if I give JSON data like above for making POST request to a server endpoint, it should give the error. How to make this happen? how to validate POST request JSON for flask API call?
As I tried to convey in our conversation it appears you are after a serilzation and deserilization tool. I have found Marshmallow to be an exceptional tool for this (it is not the only one). Here's a working example of using Marshmallow to validate a request body, converting the validated data back to a JSON string and passing it to a function for manipulation, and returning a response with JSON data:
from json import dumps, loads
from flask import Flask, jsonify, request
from marshmallow import Schema, fields, ValidationError
class BaseSchema(Schema):
age = fields.Integer(required=True)
gender = fields.String(required=True)
class ExtendedSchema(BaseSchema):
# have a look at the examples in the Marshmallow docs for more complex data structures, such as nested fields.
IL6 = fields.String()
PCT = fields.String()
CRP = fields.String()
def my_func(json_str:str):
""" Your Function that Requires JSON string"""
a_dict = loads(json_str)
return a_dict
app = Flask(__name__)
#app.route('/base', methods=["POST"])
def base():
# Get Request body from JSON
request_data = request.json
schema = BaseSchema()
try:
# Validate request body against schema data types
result = schema.load(request_data)
except ValidationError as err:
# Return a nice message if validation fails
return jsonify(err.messages), 400
# Convert request body back to JSON str
data_now_json_str = dumps(result)
response_data = my_func(data_now_json_str)
# Send data back as JSON
return jsonify(response_data), 200
#app.route('/extended', methods=["POST"])
def extended():
""" Same as base function but with more fields in Schema """
request_data = request.json
schema = ExtendedSchema()
try:
result = schema.load(request_data)
except ValidationError as err:
return jsonify(err.messages), 400
data_now_json_str = dumps(result)
response_data = my_func(data_now_json_str)
return jsonify(response_data), 200
Here's some quick tests to show validation, as well as extending the fields in your request body:
import requests
# Request fails validation
base_data = {
'age': 42,
}
r1 = requests.post('http://127.0.0.1:5000/base', json=base_data)
print(r1.content)
# Request passes validation
base_data = {
'age': 42,
'gender': 'hobbit'
}
r2 = requests.post('http://127.0.0.1:5000/base', json=base_data)
print(r2.content)
# Request with extended properties
extended_data = {
'age': 42,
'gender': 'hobbit',
'IL6': 'Five',
'PCT': 'Four',
'CRP': 'Three'}
r3 = requests.post('http://127.0.0.1:5000/extended', json=extended_data)
print(r3.content)
Hope this helps gets you where you're going.
Related
I am new to flask and I was trying to make GET request for url containing "?" symbol but it look like my program is just skipping work with it. I am working with flask-sql alchemy, flask and flask-restful. Some simplified look of my program looks like this:
fields_list = ['id']
db = SQLAlchemy(app)
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
class Get(Resource):
#staticmethod
def get():
users = User.query.all()
usr_list = Collection.user_to_json(users)
return {"Users": usr_list}, 200
class GetSorted(Resource):
#staticmethod
def get(field, type):
if field not in fields_list or type not in ['acs', 'desc']:
return {'Error': 'Wrong field or sort type'}, 400
users = db.session.execute(f"SELECT * FROM USER ORDER BY {field} {type}")
usr_list = Collection.user_to_json(users)
return {"Users": usr_list}, 200
api.add_resource(GetSorted, '/api/customers?sort=<field>&sort_type=<type>')
api.add_resource(Get, '/api/customers')
Output with url "http://127.0.0.1:5000/api/customers?sort=id&sort_type=desc" looks like this
{
"Users": [
{
"Id": 1
},
{
"Id": 2
},
{
"Id": 3
},
]
}
But I expect it to look like this
{
"Users": [
{
"Id": 3
},
{
"Id": 2
},
{
"Id": 1
},
]
}
Somehow if I replace "?" with "/" in url everything worked fine, but I want it to work with "?"
In order to get the information after ?, you have to use request.args. This information is Query Parameters, which are part of the Query String: a section of the URL that contains key-value parameters.
If your route is:
api.add_resource(GetSorted, '/api/customers?sort=<field>&sort_type=<type>')
Your key-values would be:
sort=<field>
sort_type=<type>
And you could get the values of the field and type keys like this:
sort = request.args.get('field', 'field_defaul_value')
sort_type = request.args.get('type', 'type_defaul_value')
More info:
1
2
With Flask you can define path variables like you did, but they must be part of the path. For example, defining a path of /api/customers/<id> can be used to get a specific customer by id, defining the function as def get(id):. Query parameters cannot be defined in such a way, and as you mentioned in your comment, you need to somehow "overload" the get function. Here is one way to do it:
from flask import Flask, request
from flask_restful import Resource, Api
app = Flask(__name__)
api = Api(app)
USERS = [
{"id": 1},
{"id": 3},
{"id": 2},
]
class Get(Resource):
#classmethod
def get(cls):
if request.args:
return cls._sorted_get()
return {"Users": USERS, "args":request.args}, 200
#classmethod
def _sorted_get(cls):
field = request.args.get("sort")
type = request.args.get("sort_type")
if field not in ("id",) or type not in ['acs', 'desc']:
return {'Error': 'Wrong field or sort type'}, 400
sorted_users = sorted(USERS, key=lambda x: x[field], reverse=type=="desc")
return {"Users": sorted_users}, 200
api.add_resource(Get, '/api/customers')
if __name__ == '__main__':
app.run(debug=True)
Here is Flask's documentation regarding accessing request data, and Flask-Restful's quickstart guide.
I have a file called main.py as follows:
from typing import Optional
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
fake_db = {
"Foo": {"id": "foo", "title": "Foo", "description": "There goes my hero"},
"Bar": {"id": "bar", "title": "Bar", "description": "The bartenders"},
}
class Item(BaseModel):
id: str
title: str
description: Optional[str] = None
#app.post("/items/", response_model=Item)
async def create_item(item: Item, key: str):
fake_db[key] = item
return item
Now, if I run the code for the test, saved in the file test_main.py
from fastapi.testclient import TestClient
from main import app
client = TestClient(app)
def test_create_item():
response = client.post(
"/items/",
{"id": "baz", "title": "A test title", "description": "A test description"},
"Baz"
)
return response.json()
print(test_create_item())
I don't get the desired result, that is
{"id": "baz", "title": "A test title", "description": "A test description"}
What is the mistake?
Let's start by explaining what you are doing wrong.
FastAPI's TestClient is just a re-export of Starlette's TestClient which is a subclass of requests.Session. The requests library's post method has the following signature:
def post(self, url, data=None, json=None, **kwargs):
r"""Sends a POST request. Returns :class:`Response` object.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
object to send in the body of the :class:`Request`.
:param json: (optional) json to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:rtype: requests.Response
"""
Your code
response = client.post(
"/items/",
{"id": "baz", "title": "A test title", "description": "A test description"},
"Baz",
)
is doing multiple things wrong:
It is passing in arguments to both the data and the json parameter, which is wrong because you can't have 2 different request bodies. You either pass in data or json, but not both. The data is typically used for form-encoded inputs from HTML forms, while json is for raw JSON objects. See the requests docs on "More complicated POST requests".
The requests library will simply drop the json argument because:
Note, the json parameter is ignored if either data or files is passed.
It is passing-in the plain string "Baz" to the json parameter, which is not a valid JSON object.
The data parameter expects form-encoded data.
The full error returned by FastAPI in the response is:
def test_create_item():
response = client.post(
"/items/", {"id": "baz", "title": "A test title", "description": "A test description"}, "Baz"
)
print(response.status_code)
print(response.reason)
return response.json()
# 422 Unprocessable Entity
# {'detail': [{'loc': ['query', 'key'],
# 'msg': 'field required',
# 'type': 'value_error.missing'},
# {'loc': ['body'],
# 'msg': 'value is not a valid dict',
# 'type': 'type_error.dict'}]}
The 1st error says key is missing from the query, meaning the route parameter key value "Baz" was not in the request body and FastAPI tried to look for it from the query parameters (see the FastAPI docs on Request body + path + query parameters).
The 2nd error is from point #4 I listed above about data not being properly form-encoded (that error does go away when you wrap the dict value in json.dumps, but that's not important nor is it part of the solution).
You said in a comment that you were trying to do the same thing as in the FastAPI Testing Tutorial. The difference of that tutorial from your code is that was POSTing all the attributes of the Item object in 1 body, and that it was using the json= parameter of .post.
Now on the solutions!
Solution #1: Have a separate class for POSTing the item attributes with a key
Here, you'll need 2 classes, one with a key attribute that you use for the POST request body (let's call it NewItem), and your current one Item for the internal DB and for the response model. Your route function will then have just 1 parameter (new_item) and you can just get the key from that object.
main.py
class Item(BaseModel):
id: str
title: str
description: Optional[str] = None
class NewItem(Item):
key: str
#app.post("/items/", response_model=Item)
async def create_item(new_item: NewItem):
# See Pydantic https://pydantic-docs.helpmanual.io/usage/exporting_models/#modeldict
# Also, Pydantic by default will ignore the extra attribute `key` when creating `Item`
item = Item(**new_item.dict())
print(item)
fake_db[new_item.key] = item
return item
For the test .post code, use json= to pass all the fields in 1 dictionary.
test_main.py
def test_create_item():
response = client.post(
"/items/",
json={
"key": "Baz",
"id": "baz",
"title": "A test title",
"description": "A test description",
},
)
print(response.status_code, response.reason)
return response.json()
Output
id='baz' title='A test title' description='A test description'
200 OK
{'description': 'A test description', 'id': 'baz', 'title': 'A test title'}
Solution #2: Have 2 body parts, 1 for the item attributes and 1 for the key
You can structure the POSTed body like this instead:
{
"item": {
"id": "baz",
"title": "A test title",
"description": "A test description",
},
"key": "Baz",
},
where you have the Item attributes in a nested dict and then have a simple key-value pair in the same level as item. FastAPI can handle this, see the docs on Singular values in body, which fits your example quite nicely:
For example, extending the previous model, you could decide that you want to have another key importance in the same body, besides the item and user.
If you declare it as is, because it is a singular value, FastAPI will assume that it is a query parameter.
But you can instruct FastAPI to treat it as another body key using Body
Note the parts I emphasized, about telling FastAPI to look for key in the same body. It is important here that the parameter names item and key match the ones in the request body.
main.py
from fastapi import Body, FastAPI
class Item(BaseModel):
id: str
title: str
description: Optional[str] = None
#app.post("/items/", response_model=Item)
async def create_item(item: Item, key: str = Body(...)):
print(item)
fake_db[key] = item
return item
Again, for making the .post request, use json= to pass the entire dictionary.
test_main.py
def test_create_item():
response = client.post(
"/items/",
json={
"item": {
"id": "baz",
"title": "A test title",
"description": "A test description",
},
"key": "Baz",
},
)
print(response.status_code, response.reason)
return response.json()
Output
id='baz' title='A test title' description='A test description'
200 OK
{'description': 'A test description', 'id': 'baz', 'title': 'A test title'}
I try to test an endpoint with the TestClient from FastAPI (which is the Scarlett TestClient basically).
The response code is always 422 Unprocessable Entity.
This is my current Code:
from typing import Dict, Optional
from fastapi import APIRouter
from pydantic import BaseModel
router = APIRouter()
class CreateRequest(BaseModel):
number: int
ttl: Optional[float] = None
#router.post("/create")
async def create_users(body: CreateRequest) -> Dict:
return {
"msg": f"{body.number} Users are created"
}
As you can see I'm also passing the application/json header to the client to avoid a potential error.
And this is my Test:
from fastapi.testclient import TestClient
from metusa import app
def test_create_50_users():
client = TestClient(app)
client.headers["Content-Type"] = "application/json"
body = {
"number": 50,
"ttl": 2.0
}
response = client.post('/v1/users/create', data=body)
assert response.status_code == 200
assert response.json() == {"msg": "50 Users created"}
I also found this error message in the Response Object
b'{"detail":[{"loc":["body",0],"msg":"Expecting value: line 1 column 1 (char 0)","type":"value_error.jsondecode","ctx":{"msg":"Expecting value","doc":"number=50&ttl=2.0","pos":0,"lineno":1,"colno":1}}]}'
Thank you for your support and time!
You don't need to set headers manualy. You can use json argument insteed of data in client.post method.
def test_create_50_users():
client = TestClient(router)
body = {
"number": 50,
"ttl": 2.0
}
response = client.post('/create', json=body)
If you still want to use data attribute, you need to use json.dumps
def test_create_50_users():
client = TestClient(router)
client.headers["Content-Type"] = "application/json"
body = {
"number": 50,
"ttl": 2.0
}
response = client.post('/create', data=json.dumps(body))
I am trying to print the values from an API Request. The JSON file returned is large(4,000 lines) so I am just trying to get specific values from the key value pair and automate a message.
Here is what I have so far:
import requests
import json
import urllib
url = "https://api.github.com/repos/<companyName>/<repoName>/issues" #url
payload = {}
headers = {
'Authorization': 'Bearer <masterToken>' #authorization works fine
}
name = (user.login) #pretty sure nothing is being looked out
url = (url)
print(hello %name, you have a pull request to view. See here %url for more information) # i want to print those keys here
The JSON file (exported from the API get request is as followed:
[
{
**"url": "https://github.com/<ompanyName>/<repo>/issues/1000",**
"repository_url": "https://github.com/<ompanyName>/<repo>",
"labels_url": "https://github.com/<ompanyName>/<repo>/issues/1000labels{/name}",
"comments_url": "https://github.com/<ompanyName>/<repo>/issues/1000",
"events_url": "https://github.com/<ompanyName>/<repo>/issues/1000",
"html_url": "https://github.com/<ompanyName>/<repo>/issues/1000",
"id": <id>,
"node_id": "<nodeID>",
"number": 702,
"title": "<titleName>",
"user": {
**"login": "<userName>",**
"id": <idNumber>,
"node_id": "nodeID",
"avatar_url": "https://avatars3.githubusercontent.com/u/urlName?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/<userName>",
"html_url": "https://github.com/<userName>",
"followers_url": "https://api.github.com/users/<userName>/followers",
"following_url": "https://api.github.com/users/<userName>/following{/other_user}",
"gists_url": "https://api.github.com/users/<userName>/gists{/gist_id}",
"starred_url": "https://api.github.com/users/<userName>/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/<userName>/subscriptions",
"organizations_url": "https://api.github.com/users/<userName>/orgs",
"repos_url": "https://api.github.com/users/<userName>/repos",
"events_url": "https://api.github.com/users/<userName>/events{/privacy}",
"received_events_url": "https://api.github.com/users/<userName>/received_events",
"type": "User",
"site_admin": false
},
]
(note this JSON file repeats a few hundred times)
From the API request, I am trying to get the nested "login" and the url.
What am I missing?
Thanks
Edit:
Solved:
import requests
import json
import urllib
url = "https://api.github.com/repos/<companyName>/<repoName>/issues"
payload = {}
headers = {
'Authorization': 'Bearer <masterToken>'
}
response = requests.get(url).json()
for obj in response:
name = obj['user']['login']
url = obj['url']
print('Hello {0}, you have an outstanding ticket to review. For more information see here:{1}.'.format(name,url))
Since it's a JSON array you have to loop over it. And JSON objects are converted to dictionaries, so you use ['key'] to access the elements.
for obj in response:
name = obj['user']['login']
url = obj['url']
print(f'hello {name}, you have a pull request to view. See here {url} for more information')
you can parse it into a python lists/dictionaries and then access it like any other python object.
response = requests.get(...).json()
login = response[0]['user']
You can convert JSON formatted data to a Python dictionary like this:
https://www.w3schools.com/python/python_json.asp
json_data = ... # response from API
dict_data = json.loads(json_data)
login = response[0]['user']['login']
url = response[0]['url']
I'm trying to post the following data. But I'm getting an error. Can you please take look? Thanks a lot.
I'm posting the same data using Postman. And it works.
def _build_post_data(bike_instance):
"""
data = {
"apikey": "XXX",
"data": {
"created_at": "date_XX",
"Price": "Decimal_XX"
}
}
"""
data = {}
raw_data = serializers.serialize('python', [bike_instance])
actual_data = [d['fields'] for d in raw_data]
data.update(
{
"apikey": XXX,
"data": actual_data[0]
}
)
return data
Posting data
bike = Bike.objects.get(pk=XXX)
data = _build_post_data(bike)
dump_data = json.dumps(data, cls=DjangoJSONEncoder)
requests.post(url, data=dump_data)
error
u'{"error":{"message":"422 Unprocessable Entity","errors":[["The data field is required."],["The apikey field is required."]],"status_code":422}}'
data and apikey already in the dict. then why I'm getting an error? Any idea?
Postman works
With Postman you are sending a multipart/form-data request, with requests you only send JSON (the value of the data field in Postman), and are not including the apikey field.
Use a dictionary with the JSON data as one of the values, and pass that in as the files argument. It probably also works as the data argument (sent as application/x-www-urlencoded):
form_structure = {'apikey': 'XXXX', 'data': dump_data}
requests.post(url, files=form_structure)
# probably works too: requests.post(url, data=form_structure)