This question already has answers here:
What is JSONP, and why was it created?
(10 answers)
Django - Parse JSONP (Json with Padding)
(2 answers)
Closed last month.
I'm using the API of an affiliate network (Sovrn), expecting to retrieve a product's specification using the URL.
As per their documentation, I use:
url = 'URL-goes-here'
headers = {
"accept": "application/json",
"authorization": "VERY-HARD-TO-GUESS"
}
response = requests.get(url, headers=headers)
The code is working, the response I get is 200, the header contains the magical content-type application/json line
when I do
print(response.text)
I get
NULL({"merchantName":"Overstock","canonicalUrl":"URL goes here","title":"product name",...});
I tested for response type of response.text, it's <class 'str'> as expected. But when I try to process the response as json:
product_details = json.load(response.text)
I get an error message:
requests.exceptions.JSONDecodeError: [Errno Expecting value]
I'm new to JSON, but I assume the error is due to the outer NULL that the (seemingly valid) data is wrapped in.
After spending a few hours searching for a solution, it seems that I must be missing something obvious, but not sure what.
Any pointers would be extremely helpful.
That's clearly a bug in the API. Assuming it will be fixed after you complain, you could add a hack to your code
def sovrn_json_load_hack(json_text):
"""sovrn is returning invalid json as of (revision here)."""
if not json_text.startswith ("NULL("):
return json.loads(json_text)
else:
return json.loads(json_text[5:-2])
You can ignore NULL( at the beginning and ); at the end by using string slicing:
product_details = json.loads(response.text[5:-2])
Additionally, you should be using json.loads() as the content is a string.
Related
I'm trying to send this json array to an API using request.post():
{
"insert": [
{
"data": "48bPRVkgvHwjG2VUTkPLaGazynZ6RxETuNGsYZNBrtb7ZkAUqY1NE2iGqoLd8EFsvhbDGW8gNb96Jce8fg2aiY8A5mbd8zf",
"tag": "0x1001"
}
]
}
I was told the json post has to be in an array by one of the devs at the site I'm posting to.
I also pored over these two examples in the API docs.1(edit emojiId section),2(adding an emoji id record)
What I tried so far is posting this dictionary:
{'insert': [{'data': '48bPRVkgvHwjG2VUTkPLaGazynZ6RxETuNGsYZNBrtb7ZkAUqY1NE2iGqoLd8EFsvhbDGW8gNb96Jce8fg2aiY8A5mbd8zf', 'tag': '0x1001'}]}
Using both of these post requests:
requests.post(base_url + '/emoji_id/🤘🐺🤘', json=dict_data, headers=headers)
requests.post(base_url + '/emoji_id/🤘🐺🤘', data=dict_data, headers=headers)
I then converted the dictionary to a string using
dict_data = json.dumps(dict_data, skipkeys=True, separators=(',', ':'))
and again sent it using both json and data.
Each returned a 405 error.
The next thing I tried is using the example this stack post to send a nested json. Here's my code for that attempt:
params = [{'data':'48bPRVkgvHwjG2VUTkPLaGazynZ6RxETuNGsYZNBrtb7ZkAUqY1NE2iGqoLd8EFsvhbDGW8gNb96Jce8fg2aiY8A5mbd8zf','tag':'0x1001'}]
# Tried above both with [ ] on outside making it a list and without
payload = {'insert': json.dumps(params, skipkeys=True, separators=(',', ':'))}
requests.post(base_url + '/emoji_id/🤘🐺🤘', json=payload, headers=headers)
Again, I used both json=payload and data=payload and again I tried it both as a dictionary and converting to a string using the same json.dumps method as above. Same as before each returned a 405 response.
My question is, am I using request.post() correctly in at least one of these attempts? If so, I must be missing something about what the API expects and can start down that rabbit hole.
#ewong nailed it. I didn't realize I had to use the patch method instead of the post method.
So my issue was fixed using:
requests.patch(base_url + '/emoji_id/🤘🐺🤘', json=dict_data, headers=headers)
instead of request.post. The difference between the two still seems subtle to me at this point but the example given in the API docs clearly show a patch method being used and that's what worked.
I'm very new to coding, and I'm building my first web application using open REST api with python flask.
I think the api is returning jsonp which looks like this - callbackfunction{ json }; and I get from other posts that all I need to do is getting rid of this padding. However, I can't figure out at which point I should implement the stripping.
This is my code. 5th line is throwing an error "the JSON object must be str, bytes or bytearray, not HTTPResponse"
def lookup(title):
try:
url = "http://www.aladin.co.kr/ttb/api/ItemSearch.aspx?ttbkey=foo&Query=bar"
result = urllib.request.urlopen(url)
data = json.loads(result)
data_json = data.split("{", 1)[1].strip("}")
return data_json
except requests.RequestException:
return None
I'm sure it's working well until 4th line. When I tried the code below, at least it returned result, though cryptic, like this.
b'{ "version" : "20070901", "title" :
"\xec\x95\x8c\xeb\x9d\xbc\xeb\x94\x98 \xea\xb2\x80 ...
"customerReviewRank":9 } ] };'
Judging by the keys, I'm pretty sure this is the information I requested. So what can I do to fix this? Thanks in advance!
def lookup(title):
try:
url = "http://www.aladin.co.kr/ttb/api/ItemSearch.aspx?ttbkey=foo&Query=bar"
result = urllib.request.urlopen(url)
res = result.readline()
return res
I tested using Python to translate a curl to get some data.
import requests
import json
username="abc"
password="123"
headers = {
'Content-Type': 'application/json',
}
params = (
('version', '2017-05-01'),
)
data = '{"text":["This is message one."], "id":"en-es"}'
response = requests.post('https://somegateway.service/api/abc', headers=headers, params=params, data=data, auth=(username, password))
print(response.text)
The above works fine. It returns json data.
It seems ["This is message one."] is a list. I want to use a variable that loads a file to replace this list.
I tried:
with open(f,"r",encoding='utf-8') as fp:
file_in_list=fp.read().splitlines()
toStr=str(file_in_list)
data = '{"text":'+toStr+', "id":"en-es"}'
response = requests.post('https://somegateway.service/api/abc', headers=headers, params=params, data=data, auth=(username, password))
print(response.text)
But it returned error below.
{
"code" : 400,
"error" : "Mapping error, invalid JSON"
}
Can you help? How can I have valid response.text?
Thanks.
update:
The content of f contains only five lines below:
This is message one.
this is 2.
this is three.
this is four.
this is five.
The reason your existing code fails is that str applied to a list of strings will only rarely give you valid JSON. They're not intended to do the same thing. JSON only allows double-quoted strings; Python allows both single- and double-quoted strings. And, unless your strings all happen to include ' characters, Python will render them with single quotes:
>>> print(["abc'def"]) # gives you valid JSON, but only by accident
["abc'def"]
>>> print(["abc"]) # does not give you valid JSON
['abc']
If you want to get the valid JSON encoding of a list of strings, don't try to trick str into giving you valid JSON by accident, just use the json module:
toStr = json.dumps(file_in_list)
But, even more simply, you shouldn't be trying to figure out how to construct JSON strings in the first place. Just create a dict and json.dumps the whole thing:
data = {"text": file_in_list, "id": "en-es"}
data_str = json.dumps(data)
Being able to do this is pretty much the whole point of JSON: it's a simple way to automatically serialize all of the types that are common to all the major scripting languages.
Or, even better, let requests do it for you by passing a json argument instead of a data argument:
data = {"text": file_in_list, "id": "en-es"}
response = requests.post('https://somegateway.service/api/abc', headers=headers, params=params, json=data, auth=(username, password))
This also automatically takes care of setting the Content-Type header to application/json for you. You weren't doing that—and, while many servers will accept your input without it, it's illegal, and some servers will not allow it.
For more details, see the section More complicated POST requests in the requests docs. But there really aren't many more details.
tldr;
toStr = json.dumps(file_in_list)
Explanation
Assuming your file contains something like
String_A
String_B
You need to ensure that toStr is:
Enclosed by [ and ]
Every String in the list is enclosed by quotation marks.
So your raw json (as a String) is equal to '{"text":["String_A", "String_B"], "id":"en-es"}'
There is a POST request which works perfectly when I pass the data as below:
url = 'https://www.nnnow.com/api/product/details'
requests.post(url, data="{\"styleId\":\"BMHSUR2HTS\"}", headers=headers)
But when I use json.dumps() on a dictionary and send the response, I do not get the response (response code 504), using headers={'Content-Type': 'application/json'} . Have also tried json parameter of Post requests.
requests.post(url, data=json.dumps({"styleId":"BMHSUR2HTS"}), headers={'content-type': 'application/json'})
Now, the data returned by json.dumps({"styleId":"BMHSUR2HTS"}) and
"{\"styleId\":\"BMHSUR2HTS\"}" is not the same.
json.dumps({"styleId":"BMHSUR2HTS"}) == "{\"styleId\":\"BMHSUR2HTS\"}" gives False even though a print on both shows a similar string.
How can I get the same format as "{\"styleId\":\"BMHSUR2HTS\"}" from a dictionary {"styleId":"BMHSUR2HTS"} ?
If you print the json.dumps({"styleId":"BMHSUR2HTS"}), you will notice two things:
your output is a string (just try type(json.dumps({"styleId":"BMHSUR2HTS"})));
if you pay attention the output will add a space between the json name and value: {"styleId": "BMHSURT2HTS"}.
Not sure how do you want to handle this, and in your entry code, but there are 2 main options to workaround this issue:
Replace the space on json.dumps output: json.dumps({"styleId":"BMHSUR2HTS"}).replace(': ', ':')
Convert all to json by using eval(): eval(json.dumps({"styleId":"BMHSUR2HTS"})) and eval(YOUR_JSON_STRING)
I hope this helps you.
This question already has answers here:
HTTP requests and JSON parsing in Python [duplicate]
(8 answers)
Closed 7 years ago.
I am creating a Django web app.
There is a function which creates a JSON response like this:
def rest_get(request, token):
details = Links.get_url(Links, token)
result={}
if len(details)>0:
result['status'] = 200
result['status_message'] = "OK"
result['url'] = details[0].url
else:
result['status'] = 404
result['status_message'] = "Not Found"
result['url'] = None
return JsonResponse(result)
And I get the response in the web browser like this:
{"status": 200, "url": "http://www.bing.com", "status_message": "OK"}
Now from another function I want to consume that response and extract the data out of it. How do I do it?
You can use the json library in python to do your job. for example :
json_string = '{"first_name": "tom", "last_name":"harry"}'
import json
parsed_json = json.loads(json_string)
print(parsed_json['first_name'])
"tom"
Since you have created a web app. I am assuming you have exposed a URL from which you can get you JSON response, for example http://jsonplaceholder.typicode.com/posts/1.
import urllib2
import json
data = urllib2.urlopen("http://jsonplaceholder.typicode.com/posts/1").read()
parsed_json = json.loads(data)
The urlopen function sends a HTTP GET request at the given URL. parsed_json is a variable of the type map and you can extract the required data from it.
print parsed_json['userId']
1
The answer I want to suggest is a little different. In your scenario - where one function needs to be accessed from both server and client end, I would suggest provide some extra parameter and change the output based on that. This reduces overheads and unnecessary conversions.
For example, if you pass in an extra parameter and change the result like this, you don't need JSON parsing on python. Of course there are solutions to do that, but why need converting to json and then parsing back when you can avoid that totally?
def rest_get(request, token, return_json=True):
details = Links.get_url(Links, token)
result={}
if len(details)>0:
result['status'] = 200
result['status_message'] = "OK"
result['url'] = details[0].url
else:
result['status'] = 404
result['status_message'] = "Not Found"
result['url'] = None
if return_json: # this is web response, so by default return_json = True
return JsonResponse(result)
return result
Then in your python code call like this -
rest_get(request, token, return_json=False): # we are passing False, so the return type is dictionary and we can use it right away.