I am requesting an api and get the following response and not getting my results.
For IDE I am using pycharm community edition.
import requests
import json
def tes():
url="https://user-service.abc.co.in/api/user/admin/roles/"
header ={'Content-Type': 'application/json' }
payload ={'phone_number': '9999999999'}
resp=requests.get(url,headers=header,data=json.dump(payload,indent=4))
assert resp.status_code==200
resp_body=resp.json()
assert resp_body['url']==url
print(resp.text)
Please help me why this is happening.
In the line resp=requests.get(url,headers=header,data=json.dump(payload,indent=4)) you are trying to convert the payload to json but you called json.dump() which expects a file-like object as second parameter. From the docs:
json.dump(obj, fp)
Serialize obj as a JSON formatted stream to fp (a .write()-supporting file-like object) Python docs for json.dump()
For your purposes you want to use json.dumps(obj) (notice the 's' at the end) which
Serialize obj to a JSON formatted str
Python docs for json.dumps()
Related
This question already has answers here:
What is JSONP, and why was it created?
(10 answers)
Django - Parse JSONP (Json with Padding)
(2 answers)
Closed last month.
I'm using the API of an affiliate network (Sovrn), expecting to retrieve a product's specification using the URL.
As per their documentation, I use:
url = 'URL-goes-here'
headers = {
"accept": "application/json",
"authorization": "VERY-HARD-TO-GUESS"
}
response = requests.get(url, headers=headers)
The code is working, the response I get is 200, the header contains the magical content-type application/json line
when I do
print(response.text)
I get
NULL({"merchantName":"Overstock","canonicalUrl":"URL goes here","title":"product name",...});
I tested for response type of response.text, it's <class 'str'> as expected. But when I try to process the response as json:
product_details = json.load(response.text)
I get an error message:
requests.exceptions.JSONDecodeError: [Errno Expecting value]
I'm new to JSON, but I assume the error is due to the outer NULL that the (seemingly valid) data is wrapped in.
After spending a few hours searching for a solution, it seems that I must be missing something obvious, but not sure what.
Any pointers would be extremely helpful.
That's clearly a bug in the API. Assuming it will be fixed after you complain, you could add a hack to your code
def sovrn_json_load_hack(json_text):
"""sovrn is returning invalid json as of (revision here)."""
if not json_text.startswith ("NULL("):
return json.loads(json_text)
else:
return json.loads(json_text[5:-2])
You can ignore NULL( at the beginning and ); at the end by using string slicing:
product_details = json.loads(response.text[5:-2])
Additionally, you should be using json.loads() as the content is a string.
I have written the same API application with the same function in both FastAPI and Flask. However, when returning the JSON, the format of data differs between the two frameworks. Both use the same json library and even the same exact code:
import json
from google.cloud import bigquery
bigquery_client = bigquery.Client()
#router.get('/report')
async def report(request: Request):
response = get_clicks_impression(bigquery_client, source_id)
return response
def get_user(client, source_id):
try:
query = """ SELECT * FROM ....."""
job_config = bigquery.QueryJobConfig(
query_parameters=[
bigquery.ScalarQueryParameter("source_id", "STRING", source_id),
]
)
query_job = client.query(query, job_config=job_config) # Wait for the job to complete.
result = []
for row in query_job:
result.append(dict(row))
json_obj = json.dumps(result, indent=4, sort_keys=True, default=str)
except Exception as e:
return str(e)
return json_obj
The returned data in Flask was dict:
{
"User": "fasdf",
"date": "2022-09-21",
"count": 205
},
{
"User": "abd",
"date": "2022-09-27",
"count": 100
}
]
While in FastAPI was string:
"[\n {\n \"User\": \"aaa\",\n \"date\": \"2022-09-26\",\n \"count\": 840,\n]"
The reason I use json.dumps() is that date cannot be itterable.
If you serialise the object before returning it—using, for instance, json.dumps(), as in your example—the object will end up being serialised twice, as FastAPI will automatically serialise the return value. Hence, the reason for the output string you ended up with, that is:
"[\n {\n \"User\": \"aaa\",\n \"date\": \"2022-09-26\",\n ...
Have a look at the available solutions below.
Option 1
You could normally return data such as dict, list, etc., and FastAPI would automatically convert that return value into JSON, after first converting the data into JSON-compatible data (e.g., a dict) using the jsonable_encoder. The jsonable_encoder ensures that objects that are not serializable, such as datetime objects, are converted to a str. Then, behind the scenes, FastAPI would put that JSON-compatible data inside of a JSONResponse, which will return an application/json encoded response to the client. The JSONResponse, as can be seen in Starlette's source code here, will use the Python standard json.dumps() to serialise the dict (for alternatvie/faster JSON encoders, see this answer).
Data:
from datetime import date
d = [{'User': 'a', 'date': date.today(), 'count': 1},
{'User': 'b', 'date': date.today(), 'count': 2}]
API Endpoint:
#app.get('/')
def main():
return d
The above is equivalent to:
from fastapi.responses import JSONResponse
from fastapi.encoders import jsonable_encoder
#app.get('/')
def main():
return JSONResponse(content=jsonable_encoder(d))
Output:
[{"User":"a","date":"2022-10-21","count":1},{"User":"b","date":"2022-10-21","count":2}]
Option 2
If, for any reason (e.g., trying to force some custom JSON format), you have to serialise the object before returning it, you can then return a custom Response directly, as described in this answer. As per the documentation:
When you return a Response directly its data is not validated,
converted (serialized), nor documented automatically.
Additionally, as described here:
FastAPI (actually Starlette) will automatically include a
Content-Length header. It will also include a Content-Type header,
based on the media_type and appending a charset for text types.
Hence, you can also set the media_type to whatever type you are expecting the data to be; in this case, that is application/json. Example is given below.
Note 1: The JSON outputs posted in this answer (in both Options 1 & 2) are the result of accessing the API endpoint through the browser directly (i.e., by typing the URL in the address bar of the browser and then hitting the enter key). If you tested the endpoint through Swagger UI at /docs instead, you would see that the indentation differs (in both options). This is due to how Swagger UI formats application/json responses. If you needed to force your custom indentation on Swagger UI as well, you could avoid specifying the media_type for the Response in the example below. This would result in displaying the content as text, as the Content-Type header would be missing from the response, and hence, Swagger UI couldn't recognise the type of the data, in order to format them.
Note 2: Setting the default argument to str in json.dumps() is what makes it possible to serialise the date object, otherwise if it wasn't set, you would get: TypeError: Object of type date is not JSON serializable. The default is a function that gets called for objects that can't otherwise be serialized. It should return a JSON-encodable version of the object. In this case it is str, meaning that every object that is not serializable, it is converted to string. You could also use a custom function or JSONEncoder subclass, as demosntrated here, if you would like to serialise an object in a custom way.
Note 3: FastAPI/Starlette's Response accepts as a content argument either a str or bytes object. As shown in the implementation here, if you don't pass a bytes object, Starlette will try to encode it using content.encode(self.charset). Hence, if, for instance, you passed a dict, you would get: AttributeError: 'dict' object has no attribute 'encode'. In the example below, a JSON str is passed, which will later be encoded into bytes (you could alternatively encode it yourself before passing it to the Response object).
API Endpoint:
from fastapi import Response
import json
#app.get('/')
def main():
json_str = json.dumps(d, indent=4, default=str)
return Response(content=json_str, media_type='application/json')
Output:
[
{
"User": "a",
"date": "2022-10-21",
"count": 1
},
{
"User": "b",
"date": "2022-10-21",
"count": 2
}
]
I am trying to post a kafka schema using python.
From the CLI I would use a syntax like:
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"type\":\"record\",\"name\":\"VisualDetections\",\"namespace\":\"com.namespace.something\",\"fields\":[{\"name\":\"vehicle_id\",\"type\":\"int\"},{\"name\":\"source\",\"type\":\"string\"},{\"name\":\"width\",\"type\":\"int\"},{\"name\":\"height\",\"type\":\"int\"},{\"name\":\"annotated_frame\",\"type\":[\"string\",\"null\"]},{\"name\":\"version\",\"type\":\"string\"},{\"name\":\"fps\",\"type\":\"int\"},{\"name\":\"mission_id\",\"type\":\"int\"},{\"name\":\"sequence\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"sequence_record\",\"fields\":[{\"name\":\"frame_id\",\"type\":\"int\"},{\"name\":\"timestamp\",\"type\":\"long\"},{\"name\":\"localization\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"localization_record\",\"fields\":[{\"name\":\"latitude\",\"type\":\"double\"},{\"name\":\"longitude\",\"type\":\"double\"},{\"name\":\"class\",\"type\":\"string\"},{\"name\":\"object_id\",\"type\":\"int\"},{\"name\":\"confidence\",\"type\":\"double\"},{\"name\":\"bbox\",\"type\":{\"type\":\"record\",\"name\":\"bbox\",\"fields\":[{\"name\":\"x_min\",\"type\":\"int\"},{\"name\":\"y_min\",\"type\":\"int\"},{\"name\":\"x_max\",\"type\":\"int\"},{\"name\":\"y_max\",\"type\":\"int\"}]}}]}}}]}}}]}"}' http://server_ip:8081/subjects/VisualDetections-value/versions/
When I tried to tranfer this function to python I tried something like:
import requests
import json
topic = 'VisualDetections'
headers = {'Content-Type': 'application/vnd.schemaregistry.v1+json'}
with open(avro_path) as fp:
data = {'schema': json.load(fp)}
data_json = json.dumps(data)
cmd = 'http://server_ip:8081/subjects/{}-value/versions/'.format(topic)
response = requests.post(cmd, headers=headers, data=data_json)
The above returns a code {"error_code":500,"message":"Internal Server Error"}. I have tried other options like:
with open(avro_path) as fp:
data = json.load(fp)
with error code:
"error_code":422,"message":"Unrecognized field: name"
In the above the avro_path just contains the avro schema in a json file (can be uploaded if useful also).
I am not sure how I could post this data exactly. Also, I did not take into consideration the -H argument of post in CLI since I couldn't find a equivalent python argument (not sure it plays any role though). Can anyone provide a solution to this issue.
For the second error, the payload needs to be {'schema': "schema string"}
For the first, I think its a matter of the encoding; json.load will read the file to a dict rather than just a string.
Notice
>>> import json
>>> schema = {"type":"record"} # example when using json.load() ... other data excluded
>>> json.dumps({'schema': schema})
'{"schema": {"type": "record"}}' # the schema value is not a string
>>> json.dumps({'schema': json.dumps(schema)})
'{"schema": "{\\"type\\": \\"record\\"}"}' # here it is
Try just reading the file
url = 'http://server_ip:8081/subjects/{}-value/versions/'.format(topic)
with open(avro_path) as fp:
data = {'schema': fp.read().strip()}
response = requests.post(cmd, headers=headers, data=json.dumps(data))
Otherwise, you would json.load then use json.dumps twice as shown above
You may also try json=data rather than data=json.dumps(data)
I need to add some extra key, value to response in requests lib of python3 without changing response structure (because the response will be sent into another service for the process). for example, I received this response:
>>import requests
>>r = requests.post(url=input_kong_address, data=data)
>>print(r.json())
{
"foo":"bar",
"key1":"val1"
}
I need to add "extra_key":"extra_value" to response:
{
"foo":"bar",
"key1":"val1",
"extra_key":"extra_value"
}
and now I want to add some extra key to response and sent it to the next service without changing in structure (class, type and etc):
>>import requests
>>import json
>>r = requests.post(url=input_kong_address, data=data)
>>response_data = r.json()
>>response_data['extra_key']='extra_value' # trying to add extra key and value to response
>>r.json = json.loads(json.dumps(response_data)) # trying to attach new dict to response
>>r.json() # check is worked?
{TypeError}'dict' object is not callable
thank you.
response.json() is a method, so rebinding it to a dict can only lead to this behaviour indeed. Now if you read the source of the response class, you'll find out that this method actually operates on the ._content attribute (accessed via either the .content and/or .text properties). IOW, you just have to assign your serialized json string to response._content:
>>> import requests
>>> import json
>>> r = requests.get("https://www.google.com")
>>> r._content = json.dumps({"foo": "bar"})
>>> r.json()
{u'foo': u'bar'}
>>>
This being said:
the response will be sent into another service for the process
you may want to think twice about your design then. It's of course impossible to tell without knowing much more about your concrete use case, but ask yourself whether this "other service" really needs the whole response object.
My API works fine and I see a 200 status when I test it using Postman. However I'm trying to access it using a Kivy application but I'm seeing a 400 response from the server after some waiting or quitting the app. By the way when testing with Postman I specify header as Content-Type: application/json and in body I see my parameters
{
"search_text": "Hello",
"num_results": 1
}
being sent as raw data.
My code
def search(self, search_text):
header = {'Content-Type':'application/json'}
req = UrlRequest('http://127.0.0.1:5000/search',req_body={"search_text": search_text,"num_results": 1},on_success=Test.got_json,req_headers=header)
print("Search method called")
#staticmethod
def got_json(req,result):
print(result)
Kivy docs say that you don't have to specify a method as this would send a POST request so I've not specified that here
Edit: The code for the server is kind of irrelevant for my issue here so I've removed it
UrlRequest should be passed a str object as request body. You can serialize the request dictionary as a string object by dumping it. Pass this dumped dictionary as the request body to UrlRequest.
import json
req_body=json.dumps({'search_text': search_text, 'num_results': 1})
req = UrlRequest(
'http://127.0.0.1:5000/search',
req_body=req_body,
on_success=Test.got_json,
req_headers=header)
req_body is a string parameter, might be a bit confusing as req_headers is a dict. You can use:
req_body=json.dumps({"search_text": search_text,"num_results": 1})