I am trying to prints json in readable form. I already checked previous threads and tried out.
using
JSON.stringify(response)
gives error:
NameError: name 'JSON' is not defined
Using
response = json.loads(urllib.urlopen(url).read())
parsed = json.loads(response)
print json.dumps(parsed, indent=4, sort_keys=True)
gives error:
Traceback (most recent call last):
File "p6.py", line 15, in <module>
parsed = json.loads(response)
File "/usr/lib/python2.7/json/__init__.py", line 328, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer
You get
NameError: name 'JSON' is not defined
because the first snippet is in JavaScript, not in Python.
As to the second snippet, you are calling json.loads() twice:
response = json.loads(urllib.urlopen(url).read()) # calling once
parsed = json.loads(response) # calling twice
Just call it once (and ensure that what you get from the HTTP server is actually JSON).
Related
See below. Given a well-known Google URL, I'm trying to retrieve data from that URL. That data will provide me another Google URL from which I can retrieve a list of JWKs.
>>> import requests, json
>>> open_id_config_url = 'https://ggp.sandbox.google.com/.well-known/openid-configuration'
>>> response = requests.get(open_id_config_url)
>>> r.status_code
200
>>> response.text
u'{\n "issuer": "https://www.stadia.com",\n "jwks_uri": "https://www.googleapis.com/service_accounts/v1/jwk/stadia-jwt#system.gserviceaccount.com",\n "claims_supported": [\n "iss",\n "aud",\n "sub",\n "iat",\n "exp",\n "s_env",\n "s_app_id",\n "s_gamer_tag",\n "s_purchase_country",\n "s_current_country",\n "s_session_id",\n "s_instance_ip",\n "s_restrict_text_chat",\n "s_restrict_voice_chat",\n "s_restrict_multiplayer",\n "s_restrict_stream_connect",\n ],\n "id_token_signing_alg_values_supported": [\n "RS256"\n ],\n}'
Above I have successfully retrieved the data from the first URL. I can see the entry jwks_uri contains the second URL I need. But when I try to convert that blob of text to a python dictionary, it fails.
>>> response.json()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/saqib.ali/saqib-env-99/lib/python2.7/site-packages/requests/models.py", line 889, in json
self.content.decode(encoding), **kwargs
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
>>> json.loads(response.text)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
The only way I can get out the JWKs URL is by doing this ugly regular expression parsing:
>>> re.compile('(?<="jwks_uri": ")[^"]+').findall(response.text)[0]
u'https://www.googleapis.com/service_accounts/v1/jwk/stadia-jwt#system.gserviceaccount.com'
Is there a cleaner, more Pythonic way to extract this string?
I really wish Google would send back a string that could be cleanly JSON-ified.
The returned json string is incorrect because last item of the dictionary ends with ,, which json cannot parse.
": [\n "RS256"\n ],\n}'
^^^
But ast.literal_eval can do that (as python parsing accepts lists/dicts that end with a comma). As long as you don't have booleans or null values, it is possible and pythonic
>>> ast.literal_eval(response.text)["jwks_uri"]
'https://www.googleapis.com/service_accounts/v1/jwk/stadia-jwt#system.gserviceaccount.com'
Your JSON is invalid because it has an extra comma after the last value in the claims_supported array.
I wouldn't necessarily recommend it, but you could use the similarity of JSON and Python syntax to parse this directly, since Python is much less picky:
ast.literal_eval(response.tezt)
As suggested in this answer use yaml to parse json. It will tolerate the trailing comma as well as other deviations from the json standard.
import yaml
d = yaml.load(response.text)
I'm using the requests library to make an api call. The json response is then formatted as a string and sent as part of a result to my server as shown by the code snippet:
def get_and_send(url, method):
resp = requests.request(url=url, method=method, **kwargs)
result = f'{{ "status_code":{resp.status_code}, "content":{resp.json()} }}'
send_to_server(result)
I intend to convert this result back to a dictionary object from the string result on the server.
The problem I have is that when I use json.loads(result) to convert the string to dictionary object, it throws the following error
Exception in thread Thread-2: Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/adipster/PycharmProjects/ScriptBackbone/ts_server/agent_thread.py", line 39, in run
resp_data = self._task_formatter.format_response(response) # Formats the response
File "/home/adipster/PycharmProjects/ScriptBackbone/utils/task_formatter.py", line 26, in format_response
response = self.get_dict_response(response.decode().strip())
File "/home/adipster/PycharmProjects/ScriptBackbone/utils/task_formatter.py", line 36, in get_dict_response
raise exp
File "/home/adipster/PycharmProjects/ScriptBackbone/utils/task_formatter.py", line 34, in get_dict_response
return json.loads(response)
File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 32 (char 31)
I understand that the error is because all my keys have to be in double quotes which is not the case when resp.json() is formatted to a string.
Please can someone help out on how I could ensure that all the keys of my dictionary object are in double quotes?
Or any other alternative to this will be welcomed. Thanks
The issue is, as you point out, that if you just do string interpolation (with an f-string, for example), the quotes for strings in the JSON object will be single quotes, but JSON format requires double quotes.
To fix this you can use json.dumps which takes a JSON object (in Python) and converts it to a properly formatted JSON string. Edit thanks to Charles Duffy: You can avoid the f-string entirely by just creating the whole result_data object as a dictionary, then converting it to JSON all at once using json.dumps.
For example:
import json
def get_and_send(url, method):
resp = requests.request(url=url, method=method, **kwargs)
result_data = {
"status_code": resp.status_code,
"content": resp.json(),
}
result = json.dumps(result_data)
send_to_server(result)
I wrote a simple python script to put the JSON file to Elasticsearch.I want to store it based on the id field I am extracting from the JSON file.
But when I try to insert into elastic search.It raises an error TypeError: expected string or buffer
Here is the code I am working on...
#! /usr/bin/python
import requests
from elasticsearch import Elasticsearch
import json
es = Elasticsearch([{'host':'localhost','port':9200}])
r = requests.get('http://127.0.0.1:9200')
i = 1
if r.status_code == 200:
with open('d1.json') as json_data:
d = json.load(json_data)
for i in d['tc'][0]['i]['f']['h']:
if i['name'] == 'm':
m = i['value']
dope=str(m)
print dope
print type(dope)
#print type(md5)
es.index(index='lab', doc_type='report',id=dope,body=json.loads(json_data))
Error Log:
44d88612fea8a8f36de82e1278abb02f
<type 'str'>
Traceback (most recent call last):
File "elastic_insert.py", line 22, in <module>
es.index(index='labs', doc_type='report',id=dope,body=json.loads(json_data))
File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer
Any suggestions on how to solve this error.I even tried to convert the m to int but it gave another error.
int(m)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '44d88612fea8a8f36de82e1278abb02f'
P.S: ElasticSearch service is up and running.
The problem is not related to the id. the problem is with
"json_data". it is a file stream so you need json.load and not json.loads in your es.index
I'm trying to store an exception error to json. Even though I'm pretty sure I'm storing a string, it's still giving me a typeerror.
Relevant section of code:
except ConnectionError as e:
s = str(e)
print type(s)
data = json.loads({'error message': s})
print "JSON load succeeded"
Traceback:
<type 'str'>
Traceback (most recent call last):
File "[REDACTED]", line 36, in <module>
ping(SERVERS)
File "[REDACTED]", line 29, in ping
data = json.loads({'error message': s})
File "C:\Python27\Lib\json\__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "C:\Python27\Lib\json\decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer
This is quite baffling to me. I'd appreciate any help with this matter.
You are looking for json.dumps(), not json.loads(). Try this:
data = json.dumps({'error message': s})
json.dumps(obj): Serialize obj to a JSON formatted str
json.loads(s): Deserialize s (a str instance containing a JSON document) to a Python object
I'm trying to take the JSON from a twitter get_user query and turn it into a Python object that I can extract data from (twitter handle, location, screen name, etc.)
Here is what I created. I am not sure why it doesn't work.
api = tweepy.API(auth,parser=tweepy.parsers.JSONParser())
user = api.search_users('google.com')
t_dict = json.loads(user)
pprint(t_dict)
Error:
Traceback (most recent call last):
File "Get_User_By_URL.py", line 23, in <module>
t_dict = json.loads(user)
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer
api.search_users is already returning a python object. It isn't a json string that needs to be parsed. According to tweetpy documentation search_users actually returns a list of users. So the following is possible:
for user in api.search_users('google.com'):
print user.screen_name