Develop Reference

JSON wrapped in NULL? [duplicate]

This question already has answers here:
What is JSONP, and why was it created?
(10 answers)
Django - Parse JSONP (Json with Padding)
(2 answers)
Closed last month.
I'm using the API of an affiliate network (Sovrn), expecting to retrieve a product's specification using the URL.
As per their documentation, I use:
url = 'URL-goes-here'
headers = {
"accept": "application/json",
"authorization": "VERY-HARD-TO-GUESS"
}
response = requests.get(url, headers=headers)
The code is working, the response I get is 200, the header contains the magical content-type application/json line
when I do
print(response.text)
I get
NULL({"merchantName":"Overstock","canonicalUrl":"URL goes here","title":"product name",...});
I tested for response type of response.text, it's <class 'str'> as expected. But when I try to process the response as json:
product_details = json.load(response.text)
I get an error message:
requests.exceptions.JSONDecodeError: [Errno Expecting value]
I'm new to JSON, but I assume the error is due to the outer NULL that the (seemingly valid) data is wrapped in.
After spending a few hours searching for a solution, it seems that I must be missing something obvious, but not sure what.
Any pointers would be extremely helpful.

That's clearly a bug in the API. Assuming it will be fixed after you complain, you could add a hack to your code
def sovrn_json_load_hack(json_text):
"""sovrn is returning invalid json as of (revision here)."""
if not json_text.startswith ("NULL("):
return json.loads(json_text)
else:
return json.loads(json_text[5:-2])

You can ignore NULL( at the beginning and ); at the end by using string slicing:
product_details = json.loads(response.text[5:-2])
Additionally, you should be using json.loads() as the content is a string.

Mapbox API PUT Datasets Feature return "Provide a single Feature to insert"

I am trying to add a feature to the Dataset via Mapbox API using Python. I'm following this instruction https://docs.mapbox.com/api/maps/#update-a-dataset but keep getting this error:
{'message': 'Provide a single Feature to insert'}
The code looks like this:
rs = []
dictionary = {
"id":1,
"type":"Feature",
"properties":{},
"geometry":{"coordinates":[-83.750246, 42.269375],"type":"Point"}}
url = "https://api.mapbox.com/datasets/v1/voratima/"+dataset+"/features/1?access_token="+access_token
rs.append(grequests.put(url, data=dictionary, hooks = {'response' : do_something}))
grequests.map(rs, exception_handler=exception_handler)
I've tried the following but none of them work:
using requests instead of grequests
wrapping the dictionary with json.dumps()
changing the put parameter from data=dictionary to json=dictionary
Making sure the id for both data and URL are set to 1.
Postman of the exact same request does not have the error. What am I missing?

Given a dataset with dataset ID dataset exists, your request body looks ok.
Please add the header
headers = {'Content-type': 'application/json'}
Also can you check if you meet these specs:
This should be one individual GeoJSON feature, not a GeoJSON
FeatureCollection. If the GeoJSON feature has a top-level id property,
it must match the feature_id you use in the URL endpoint.

It turns out I forgot the header. Thanks to Mortiz for pointing that out. After the update I got
<Response [400]> {'message': 'Unexpected token i'}
That's because I need to wrap the dictionary inside json.dumps(). Then the error became
<Response [422]> {'message': 'Request URI does not match feature id'}
That's because the id in the dictionary has to be a string i.e. "id":"1" not "id":1. Here's the code that works:
rs = []
dictionary = {
"id":"1",
"type":"Feature",
"properties":{},
"geometry":{"coordinates":[-83.750246, 42.269375],"type":"Point"}}
headers = {'Content-type': 'application/json'}
url = "https://api.mapbox.com/datasets/v1/voratima/"+dataset+"/features/1?access_token="+access_token
rs.append(grequests.put(url, data=json.dumps(dictionary), headers=headers, hooks = {'response' : do_something}))
grequests.map(rs, exception_handler=exception_handler)

How to remove u(unicode) from response and replace it in body using robotframework?

I have a response in list format from an API.
${list}=[u'e2e', u'e2etest', u'123e2e', u'Cost', u'App', u'Env']
I need to replace this list in another API post body.
list_filter= set variable ${list}
apibody: dir(filter: {name: {in_: "list_filter"}})
But when I do that I get an error as
Response status code is not equal 200: 200 != 415(Unsupported Media Type)
detail: "Invalid Content-type (application/json), expected JSON data"
status: "415"
title: "Unsupported Media Type"
type: "about:blank"
how to remove u (unicode character from list when I get the response or post response?
How to replace the string list in 2nd body?

GCM get invalid tokens when sending to multiple devices at once

I'm trying to remove invalid GCM Tokens from my database after they failed in a broadcast request.
I send it via:
payload = {
"registration_ids": gcm_keys,
"data": messageData
}
headers = {
'Content-type': 'application/json',
'Authorization': Message_Broker.api_key
}
try:
return requests.post(Message_Broker.host, data=json.dumps(payload), headers=headers)
Lets say I try to send a message to:
gcm_keys = ['daöodaeöoijiö','12345','fwiorjpfwj'] # Lets pretend the second one is a valid token
From the content of the response object I get a dict like this:
response_results = [{u'error': u'InvalidRegistration'}, {u'registration_id': u'1234567', u'message_id': u'0:14339323424213768%540eeb39f9fd7aed'}, {u'error': u'InvalidRegistration'}]
To find out which tokens failed I made a set substraction with a list comprehension:
failed_keys = list(set(gcm_keys) - set([r.get('registration_id') for r in response_results]))
This should give me back only the tokens which produced an error.
My first question is, is there a more common way to do so or any kind of best practice?
Second question is, as you can see in the response_results and the gcm_keys, the valid token is not returned. Instead a kinda similar token is given back.
I did not find any on this. Why do I get a different token back?
-> Just found out that in case I get a different token back, I should replace the old one. That leads to another question. When I send to several tokens, how can I find out to which gcm token in the request this returned token belongs?
Is the order in the results always the same as in the request?

The response body's results parameter should have an array of objects that are listed in the same order as the request. Please refer here.

How to get data from webapp2.Request

I am having trouble sending a string of bytes (an image) to my backend.
In my code I have:
# sends a httplib2.Request
backend_resp, backend_content = self.mirror_service._http.request(
uri=backend_path,
body=urllib.urlencode({"img":content}))
This sends a request where content is a large string of bytes.
in my backend I have:
class Handler(webapp2.RequestHandler):
def get(self):
image_bytes = self.request.get("img")
logging.info(image_bytes) # output is empty string
Which logs an empty string.
I have also tried
image_bytes = self.request.body
and just setting body = content in the request, but these also return nothing
I know the backend is receiving the request because the backend logs have messages I have placed.
What is the correct way to send and retrieve my GET data?
EDIT:
Here's what content logs before trying to send it to my backend:
logging.info(str(type(content)))
# returns <type 'str'>
logging.info(content)
# logs a long string of bytes
On another note, I also get this warning in the logs when sending the request, but I'm not sure how to fix it:
new_request() takes at most 1 positional argument (2 given)
I'm guessing that this warning means that the 1 positional argument it takes is path=, and it's ignoring my body= argument. I think the warning changes to (3 given) if I add method="POST" or method="GET"
I tried using a POST method too, but logging.info won't display in my logs. I tried just writing self.request.body or self.request.get('img') back to the response, and it still just returns an empty string like the GET method.

To send a post from httplib2 :
import urllib
import httplib2
http = httplib2.Http()
url = '<your post url>'
body = {'img': 'all your image bytes...'}
headers = {'Content-type': 'application/x-www-form-urlencoded'}
response, content = http.request(url, 'POST', headers=headers, body=urllib.urlencode(body))
see httplib2 docs
To receive a post in Webapp2:
class Handler(webapp2.RequestHandler):
def post(self):
image_bytes = self.request.POST.get("img")
logging.info(image_bytes) # output is empty string
I haven't tested this code, but it should give you and idea how it should be done.