Develop Reference

JSON wrapped in NULL? [duplicate]

This question already has answers here:
What is JSONP, and why was it created?
(10 answers)
Django - Parse JSONP (Json with Padding)
(2 answers)
Closed last month.
I'm using the API of an affiliate network (Sovrn), expecting to retrieve a product's specification using the URL.
As per their documentation, I use:
url = 'URL-goes-here'
headers = {
"accept": "application/json",
"authorization": "VERY-HARD-TO-GUESS"
}
response = requests.get(url, headers=headers)
The code is working, the response I get is 200, the header contains the magical content-type application/json line
when I do
print(response.text)
I get
NULL({"merchantName":"Overstock","canonicalUrl":"URL goes here","title":"product name",...});
I tested for response type of response.text, it's <class 'str'> as expected. But when I try to process the response as json:
product_details = json.load(response.text)
I get an error message:
requests.exceptions.JSONDecodeError: [Errno Expecting value]
I'm new to JSON, but I assume the error is due to the outer NULL that the (seemingly valid) data is wrapped in.
After spending a few hours searching for a solution, it seems that I must be missing something obvious, but not sure what.
Any pointers would be extremely helpful.

That's clearly a bug in the API. Assuming it will be fixed after you complain, you could add a hack to your code
def sovrn_json_load_hack(json_text):
"""sovrn is returning invalid json as of (revision here)."""
if not json_text.startswith ("NULL("):
return json.loads(json_text)
else:
return json.loads(json_text[5:-2])

You can ignore NULL( at the beginning and ); at the end by using string slicing:
product_details = json.loads(response.text[5:-2])
Additionally, you should be using json.loads() as the content is a string.

Am I sending a this json data correctly using the python requests library?

I'm trying to send this json array to an API using request.post():
{
"insert": [
{
"data": "48bPRVkgvHwjG2VUTkPLaGazynZ6RxETuNGsYZNBrtb7ZkAUqY1NE2iGqoLd8EFsvhbDGW8gNb96Jce8fg2aiY8A5mbd8zf",
"tag": "0x1001"
}
]
}
I was told the json post has to be in an array by one of the devs at the site I'm posting to.
I also pored over these two examples in the API docs.1(edit emojiId section),2(adding an emoji id record)
What I tried so far is posting this dictionary:
{'insert': [{'data': '48bPRVkgvHwjG2VUTkPLaGazynZ6RxETuNGsYZNBrtb7ZkAUqY1NE2iGqoLd8EFsvhbDGW8gNb96Jce8fg2aiY8A5mbd8zf', 'tag': '0x1001'}]}
Using both of these post requests:
requests.post(base_url + '/emoji_id/🤘🐺🤘', json=dict_data, headers=headers)
requests.post(base_url + '/emoji_id/🤘🐺🤘', data=dict_data, headers=headers)
I then converted the dictionary to a string using
dict_data = json.dumps(dict_data, skipkeys=True, separators=(',', ':'))
and again sent it using both json and data.
Each returned a 405 error.
The next thing I tried is using the example this stack post to send a nested json. Here's my code for that attempt:
params = [{'data':'48bPRVkgvHwjG2VUTkPLaGazynZ6RxETuNGsYZNBrtb7ZkAUqY1NE2iGqoLd8EFsvhbDGW8gNb96Jce8fg2aiY8A5mbd8zf','tag':'0x1001'}]
# Tried above both with [ ] on outside making it a list and without
payload = {'insert': json.dumps(params, skipkeys=True, separators=(',', ':'))}
requests.post(base_url + '/emoji_id/🤘🐺🤘', json=payload, headers=headers)
Again, I used both json=payload and data=payload and again I tried it both as a dictionary and converting to a string using the same json.dumps method as above. Same as before each returned a 405 response.
My question is, am I using request.post() correctly in at least one of these attempts? If so, I must be missing something about what the API expects and can start down that rabbit hole.

#ewong nailed it. I didn't realize I had to use the patch method instead of the post method.
So my issue was fixed using:
requests.patch(base_url + '/emoji_id/🤘🐺🤘', json=dict_data, headers=headers)
instead of request.post. The difference between the two still seems subtle to me at this point but the example given in the API docs clearly show a patch method being used and that's what worked.

Simple way to extract key and values data from nested Json Response

Hello everyone here is a snippet of my nested json Response.
https://pastebin.com/qTk21iwX
What I want to accomplished to get all the "Row" data from this json response but I have yet to figure out the correct method of approaching this.
This is what I have tried is:
response = requests.get(url, headers=headers, json={"script": sql}, verify=verify).json()
print(response)
responseobject = json.dumps(response)
print(responseobject)
for object in responseobject['Result']['Results']:
print (object)
My output after running this code
TypeError: string indices must be integers
Any suggestions ?

responseobject = json.dumps(response)
Here you are turning the response object into a string. So when you do responseobject['Result'], you are indexing a string, not a dictionary.
To fix the problem, just remove responseobject = json.dumps(response) and do response['Result']['Results'] instead of responseobject['Result']['Results'].

How to return this valid json data in Python?

I tested using Python to translate a curl to get some data.
import requests
import json
username="abc"
password="123"
headers = {
'Content-Type': 'application/json',
}
params = (
('version', '2017-05-01'),
)
data = '{"text":["This is message one."], "id":"en-es"}'
response = requests.post('https://somegateway.service/api/abc', headers=headers, params=params, data=data, auth=(username, password))
print(response.text)
The above works fine. It returns json data.
It seems ["This is message one."] is a list. I want to use a variable that loads a file to replace this list.
I tried:
with open(f,"r",encoding='utf-8') as fp:
file_in_list=fp.read().splitlines()
toStr=str(file_in_list)
data = '{"text":'+toStr+', "id":"en-es"}'
response = requests.post('https://somegateway.service/api/abc', headers=headers, params=params, data=data, auth=(username, password))
print(response.text)
But it returned error below.
{
"code" : 400,
"error" : "Mapping error, invalid JSON"
}
Can you help? How can I have valid response.text?
Thanks.
update:
The content of f contains only five lines below:
This is message one.
this is 2.
this is three.
this is four.
this is five.

The reason your existing code fails is that str applied to a list of strings will only rarely give you valid JSON. They're not intended to do the same thing. JSON only allows double-quoted strings; Python allows both single- and double-quoted strings. And, unless your strings all happen to include ' characters, Python will render them with single quotes:
>>> print(["abc'def"]) # gives you valid JSON, but only by accident
["abc'def"]
>>> print(["abc"]) # does not give you valid JSON
['abc']
If you want to get the valid JSON encoding of a list of strings, don't try to trick str into giving you valid JSON by accident, just use the json module:
toStr = json.dumps(file_in_list)
But, even more simply, you shouldn't be trying to figure out how to construct JSON strings in the first place. Just create a dict and json.dumps the whole thing:
data = {"text": file_in_list, "id": "en-es"}
data_str = json.dumps(data)
Being able to do this is pretty much the whole point of JSON: it's a simple way to automatically serialize all of the types that are common to all the major scripting languages.
Or, even better, let requests do it for you by passing a json argument instead of a data argument:
data = {"text": file_in_list, "id": "en-es"}
response = requests.post('https://somegateway.service/api/abc', headers=headers, params=params, json=data, auth=(username, password))
This also automatically takes care of setting the Content-Type header to application/json for you. You weren't doing that—and, while many servers will accept your input without it, it's illegal, and some servers will not allow it.
For more details, see the section More complicated POST requests in the requests docs. But there really aren't many more details.

tldr;
toStr = json.dumps(file_in_list)
Explanation
Assuming your file contains something like
String_A
String_B
You need to ensure that toStr is:
Enclosed by [ and ]
Every String in the list is enclosed by quotation marks.
So your raw json (as a String) is equal to '{"text":["String_A", "String_B"], "id":"en-es"}'

Sending a variable as data parameter in requests.post() in Python

I'm trying to pass a variable to the data field in requests.post() and I continue to get the error,
Error Response: {'error': {'message': 'Exception while reading request',
'detail': 'Cannod decode: java.io.StringReader#1659711'}, 'status': 'failure'}
Here is my code
#Fill array from CSV
temp=[]
for row in csv.iterrows():
index, data = row
temp.append(data.tolist())
#Create new asset for all assets in CSV
for index, row in enumerate(temp):
make = temp[index][0]
serial = str(temp[index][1])
date = str(temp[index][2])
response = requests.post(url, auth=(user, pwd), headers=headers,
data='{"asset_tag":"test1", "assigned_to":"test2",
"company":"test3", "serial_number":serial}')
I originally tried feeding it directly from the CSV using
str(temp[index][1])
This did not work, so I tried assigning str(temp[index][1]) to the variable serial and then passing the variable like that but that also results in the same error.
A point in the right direction would be great, thanks!

Instead of sending the request payload body in string, pass it in json form.
requests.post accepts string in data variable and json in json variable. I faced a same issue while trying to make my first REST call to ServiceNow instance via Python. Hope this helps.
response = requests.post(url, auth=(user, pwd), headers=headers,
json={"asset_tag":"test1", "assigned_to":"test2",
"company":"test3", "serial_number":serial})

Remove the single quotes from the following :
data='{"asset_tag":"test1", "assigned_to":"test2",
"company":"test3", "serial_number":serial}'
Use
data = {"asset_tag":"test1", "assigned_to":"test2",
"company":"test3", "serial_number":serial}

rather than passing data=data, take data as dict and pass it as json=data.