This question already has answers here:
Let JSON object accept bytes or let urlopen output strings
(12 answers)
Closed 6 years ago.
I'm using flask in a web application that uses service api generates JSON response. The following part of the function works fine and returns JSON text output:
def get_weather(query = 'london'):
api_url = "http://api.openweathermap.org/data/2.5/weather?q={}&units=metric&appid=XXXXX****2a6eaf86760c"
query = urllib.request.quote(query)
url = api_url.format(query)
response = urllib.request.urlopen(url)
data = response.read()
return data
The output returned is:
{"coord":{"lon":-0.13,"lat":51.51},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04d"}],"base":"cmc stations","main":{"temp":12.95,"pressure":1030,"humidity":68,"temp_min":12.95,"temp_max":12.95,"sea_level":1039.93,"grnd_level":1030},"wind":{"speed":5.11,"deg":279.006},"clouds":{"all":76},"dt":1462290955,"sys":{"message":0.0048,"country":"GB","sunrise":1462249610,"sunset":1462303729},"id":2643743,"name":"London","cod":200}
This mean that data is a string, does not it?
However, commenting the return data and then adding the following two lines:
jsonData = json.loads(data)
return jsonData
generates the following error:
TypeError: the JSON object must be str, not 'bytes'
What's wrong? data, the JSON object, previously returned as a string! I need to know where is the mistake?
The data returned by request library is a binary string while json.loads accepts strings so you need to convert the data (decode) to a string using the encoding that your request returns (which is usually ok to assume that it is UTF-8).
You should be able to just change your code to this:
return json.loads(data.decode("utf-8"))
PS: Storing the variable right before returning it is redundant so I simplified things
Related
This question already has answers here:
Loading JSON object in Python using urllib.request and json modules
(3 answers)
Closed 11 months ago.
URL = "API URL"
response = urllib.request.urlopen(URL)
standards = response.read() #returning type <bytes>
standards = standards.decode('utf-8') #converting type <str>
I actually want to read through the data and extract the values of "referenceNumber" & "title" from the data given below, I have 755 records of same structure. I want to extract the above given fields for each record.
[{"larsCode":0,"referenceNumber":"ST0870","title":"Business support assistant","status":"Withdrawn","url":"https://www.instituteforapprenticeships.org/api/apprenticeshipstandards/0","versionNumber":"0.0","change":"Withdrawn","changedDate":"2019-07-31T00:00:00","earliestStartDate":null,"latestStartDate":null,"latestEndDate":null,"overviewOfRole":"","level":2,"typicalDuration":0,"maxFunding":0,"route":"Business and administration","keywords":["Business","support","assistant","admin","office","office administration"],"jobRoles":[],"entryRequirements":"","assessmentPlanUrl":"","ssa1":"","ssa2":"","version":"0.0","standardInformation":"","occupationalSummary":"","knowledges":[],"behaviours":[],"skills":[],"options":[],"optionsUnstructuredTemplate":[]]
The server returns json encoded bytes. Python has a module for json.
You dont need to convert it to str in order to decode the json bytes.
Look at the python documentation for handling json objects.
Specifically json.loads for handling str or json.load for handling file like objects.
https://docs.python.org/3/library/json.html#json.loads
Just an fyi, your response is a list of objects so json.loads will return a list of dict.
I am putting a JSON response into a variable via requests.json() like this:
response = requests.get(some_url, params=some_params).json()
This however converts JSON's original " to Python's ', true to True, null to None.
This poses a problem when trying to save the response as text and the convert it back to JSON - sure, I can use .replace() for all conversions mentioned above, but even once I do that, I get other funny json decoder errors.
Is there any way in Python to get JSON response and keep original JavaScript format?
json() is the JSON decoder method. You are looking at a Python object, that is why it looks like Python.
Other formats are listed on the same page, starting from Response Content
.text: text - it has no separate link/paragraph, it is right under "Response Content"
.content: binary, as bytes
.json(): decoded JSON, as Python object
.raw: streamed bytes (so you can get parts of content as it comes)
You need .text for getting text, including JSON data.
You can get the raw text of your response with requests.get(some_url, params=some_params).text
It is the json method which converts to a Python friendly format.
header = {'Content-type': 'application/json','Authorization': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' }
url = 'https://sandbox-authservice.priaid.ch/login'
response = requests.post(url, headers = header, verify=False).json()
token = json.dumps(response)
print token['ValidThrough']
I want to print the ValidThrough Attribute in my webhook, which is received as JSON data via a POST call. I know this has been asked a number of times here, but print token['ValidThrough']isnt working for me.I receive the error "TypeError: string indices must be integers, not str"
Since the response already seems to be in json, there is no need to use json.dumps.
json.dumps on a dictionary will return a string which cannot be indexed obviously and hence that error.
a requests response .json() method already loads the content of the string to json.
You should use that, but your code later serializes it back to a string, and hence the error (token is a string representation of the dict you are expecting, not the dict). You should just omit the json.dumps(response) line, and use response['ValidThrough']
There's another error here, even if you assume that the .json() returns a string that should be unserialized again you should've used json.loads(response) in order to load it into a dict (not dumps to serialize it again)
I am trying to print out at least one key value from the returned Json, as following this basic tutorial
response=None
booking_source = 'sourceBusinessName'
api_request ='http://api.com'
r = requests.get(api_request)
while response is None:
response = r.content.decode('utf-8')
data = json.loads(response)
print (data[booking_source])
return HttpResponse(data[booking_source])
But it returns TypeError: list indices must be integers or slices, not str
probably because I am giving an string instead of an integer to data when printing, but then what I am doing wrong here ?
With requests you can skip the decoding of the response and parsing it as JSON by using the response's json method:
r = requests.get(api_request)
data = r.json()
print data # so you can see what you're dealing with
At this point I suggest dumping out the value of data so that you can see the structure of the JSON data. Probably it is a JSON array (converted to a Python list) and you simply need to take the first element of that array before accessing the dictionary, but it's difficult to tell without seeing the actual data. You might like to add a sample of the data to your question.
Your JSON is an array at the top level, but you're trying to address it as if it were:
{
"sourceBusinessName": {
...
},
...
}
I asked a similar question earlier
python JSON feed returns string not object
but I am having a little more trouble and don't understand it.
For about half of the dates this works and returns a JSON object
for example November 9 2013 works
url = 'http://data.ncaa.com/jsonp/scoreboard/basketball-men/d1/2013/11/09/scoreboard.html?callback=c'
r = requests.get(url)
jsonObj = json.loads(r.content[2:-2])
but if I try November 11 2013:
url = 'http://data.ncaa.com/jsonp/scoreboard/basketball-men/d1/2013/11/11/scoreboard.html?callback=c'
r = requests.get(url)
jsonObj = json.loads(r.content[2:-2])
I get this error
ValueError: No JSON object could be decoded
I dont understand why. When I put both urls into a browser they look exactly the same.
The JSON in the second feed is, in fact, invalid JSON. Found this by removing the callback function and running it through: http://jsonlint.com/
To see for yourself, search for the following ID: 336252
The lines just above that ID contain two commas in a row, which is disallowed by the JSON spec.
My guess is that the server at data.ncaa.com is trying to generate JSON itself rather than using a JSON library. You should contact the site administrator and make them aware of this error.
Using demjson
demjson.decode(r.content[2:-2])
seems to work