Develop Reference

What's the most Pythonic way to parse out this value from a JSON-like blob?

See below. Given a well-known Google URL, I'm trying to retrieve data from that URL. That data will provide me another Google URL from which I can retrieve a list of JWKs.
>>> import requests, json
>>> open_id_config_url = 'https://ggp.sandbox.google.com/.well-known/openid-configuration'
>>> response = requests.get(open_id_config_url)
>>> r.status_code
200
>>> response.text
u'{\n "issuer": "https://www.stadia.com",\n "jwks_uri": "https://www.googleapis.com/service_accounts/v1/jwk/stadia-jwt#system.gserviceaccount.com",\n "claims_supported": [\n "iss",\n "aud",\n "sub",\n "iat",\n "exp",\n "s_env",\n "s_app_id",\n "s_gamer_tag",\n "s_purchase_country",\n "s_current_country",\n "s_session_id",\n "s_instance_ip",\n "s_restrict_text_chat",\n "s_restrict_voice_chat",\n "s_restrict_multiplayer",\n "s_restrict_stream_connect",\n ],\n "id_token_signing_alg_values_supported": [\n "RS256"\n ],\n}'
Above I have successfully retrieved the data from the first URL. I can see the entry jwks_uri contains the second URL I need. But when I try to convert that blob of text to a python dictionary, it fails.
>>> response.json()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/saqib.ali/saqib-env-99/lib/python2.7/site-packages/requests/models.py", line 889, in json
self.content.decode(encoding), **kwargs
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
>>> json.loads(response.text)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
The only way I can get out the JWKs URL is by doing this ugly regular expression parsing:
>>> re.compile('(?<="jwks_uri": ")[^"]+').findall(response.text)[0]
u'https://www.googleapis.com/service_accounts/v1/jwk/stadia-jwt#system.gserviceaccount.com'
Is there a cleaner, more Pythonic way to extract this string?
I really wish Google would send back a string that could be cleanly JSON-ified.

The returned json string is incorrect because last item of the dictionary ends with ,, which json cannot parse.
": [\n "RS256"\n ],\n}'
^^^
But ast.literal_eval can do that (as python parsing accepts lists/dicts that end with a comma). As long as you don't have booleans or null values, it is possible and pythonic
>>> ast.literal_eval(response.text)["jwks_uri"]
'https://www.googleapis.com/service_accounts/v1/jwk/stadia-jwt#system.gserviceaccount.com'

Your JSON is invalid because it has an extra comma after the last value in the claims_supported array.
I wouldn't necessarily recommend it, but you could use the similarity of JSON and Python syntax to parse this directly, since Python is much less picky:
ast.literal_eval(response.tezt)

As suggested in this answer use yaml to parse json. It will tolerate the trailing comma as well as other deviations from the json standard.
import yaml
d = yaml.load(response.text)

json.loads(string) throws json.decoder.JSONDecodeError when trying to convert string to dictionary

I'm using the requests library to make an api call. The json response is then formatted as a string and sent as part of a result to my server as shown by the code snippet:
def get_and_send(url, method):
resp = requests.request(url=url, method=method, **kwargs)
result = f'{{ "status_code":{resp.status_code}, "content":{resp.json()} }}'
send_to_server(result)
I intend to convert this result back to a dictionary object from the string result on the server.
The problem I have is that when I use json.loads(result) to convert the string to dictionary object, it throws the following error
Exception in thread Thread-2: Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/adipster/PycharmProjects/ScriptBackbone/ts_server/agent_thread.py", line 39, in run
resp_data = self._task_formatter.format_response(response) # Formats the response
File "/home/adipster/PycharmProjects/ScriptBackbone/utils/task_formatter.py", line 26, in format_response
response = self.get_dict_response(response.decode().strip())
File "/home/adipster/PycharmProjects/ScriptBackbone/utils/task_formatter.py", line 36, in get_dict_response
raise exp
File "/home/adipster/PycharmProjects/ScriptBackbone/utils/task_formatter.py", line 34, in get_dict_response
return json.loads(response)
File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 32 (char 31)
I understand that the error is because all my keys have to be in double quotes which is not the case when resp.json() is formatted to a string.
Please can someone help out on how I could ensure that all the keys of my dictionary object are in double quotes?
Or any other alternative to this will be welcomed. Thanks

The issue is, as you point out, that if you just do string interpolation (with an f-string, for example), the quotes for strings in the JSON object will be single quotes, but JSON format requires double quotes.
To fix this you can use json.dumps which takes a JSON object (in Python) and converts it to a properly formatted JSON string. Edit thanks to Charles Duffy: You can avoid the f-string entirely by just creating the whole result_data object as a dictionary, then converting it to JSON all at once using json.dumps.
For example:
import json
def get_and_send(url, method):
resp = requests.request(url=url, method=method, **kwargs)
result_data = {
"status_code": resp.status_code,
"content": resp.json(),
}
result = json.dumps(result_data)
send_to_server(result)

JSON TypeError: expected string or buffer

I'm trying to store an exception error to json. Even though I'm pretty sure I'm storing a string, it's still giving me a typeerror.
Relevant section of code:
except ConnectionError as e:
s = str(e)
print type(s)
data = json.loads({'error message': s})
print "JSON load succeeded"
Traceback:
<type 'str'>
Traceback (most recent call last):
File "[REDACTED]", line 36, in <module>
ping(SERVERS)
File "[REDACTED]", line 29, in ping
data = json.loads({'error message': s})
File "C:\Python27\Lib\json\__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "C:\Python27\Lib\json\decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer
This is quite baffling to me. I'd appreciate any help with this matter.

You are looking for json.dumps(), not json.loads(). Try this:
data = json.dumps({'error message': s})
json.dumps(obj): Serialize obj to a JSON formatted str
json.loads(s): Deserialize s (a str instance containing a JSON document) to a Python object

Error when passing urllib.urlopen result to json.load

I'm new to python but would like to use urllib to download tweets, I'm following a tutorial instructions but get the same error every time, I print:
import urllib
import json
response = urllib.urlopen("https://twitter.com/search?q=Microsoft&src=tyah")
print json.load(response)
But everytime I get the error:
Traceback (most recent call last):
File "C:\Python27\print.py", line 4, in <module>
print json.load(response)
File "C:\Python27\Lib\json\__init__.py", line 278, in load
**kw)
File "C:\Python27\Lib\json\__init__.py", line 326, in loads
return _default_decoder.decode(s)
File "C:\Python27\Lib\json\decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Python27\Lib\json\decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

As noted in comments, the answer is: nothing is wrong with your code, per se.
The problem is that when json.load looks at response, it does not find JSON in there - it is finding HTML.
You need to pass a file-like object containing JSON into the json.load function, or it will raise the exception you see here.
To get JSON from Twitter, you need to call a URL that gives a JSON response. I can tell you now, that none of the Web interface URLs do this directly. You should use the Twitter API.
However, purely for sake of demonstration, if you deconstruct the page at the URL you are calling now, you will find that to load the tweet data, the page makes the following request:
https://twitter.com/i/search/timeline?q=Microsoft&src=tyah&composed_count=0&include_available_features=1&include_entities=1
And this URL does return JSON in response, which would work just fine with your current code.
Of course, I'm pretty sure doing so violates some sort of Twitter TOS, so if you do this there are all sorts of potential negative repercussions to consider. Plus it's just not good sportsmanship. :)

No JSON object could be decoded. Threading Python

I've been trying to make multiple HTTP request using threading.
First, I have a method that calls to an API, and that method takes a dict as an argument and returns a JSON object. Additionally, it works perfectly fine when I run it without threading. However, here is the code and error I get when trying to use threading.
import sys
sys.path.append(r'C:/dict/path')
import apiModule
import threading
token = 'xxxx'
apiModule = apiModule.Module(token)
urls = [{'url': 'http://www.example.com'}, {'url': 'http://www.example.com/2'}]
data = []
for element in urls:
thread = threading.Thread(target=apiModule.method(),kwargs=element)
thread.start()
data.append(thread)
Traceback (most recent call last):
File "<pyshell#72>", line 2, in <module>
thread = threading.Thread(target=apiModule.method(),kwargs=element)
File "C:/dict/path", line 54, in method
return json.loads(r.content)
File "C:\Python27\lib\json\__init__.py", line 326, in loads
return _default_decoder.decode(s)
File "C:\Python27\lib\json\decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Python27\lib\json\decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Any help would be appreciated!

To expand on #falsetru's reply: According to the threading documentation:
target is the callable object to be invoked by the run() method
In your example you pass target=apiModule.method(), immediately invoking the method "method" on apiModule. This raises the error you're seeing. Currently the code is not even reaching the thread.start() call.