Develop Reference

What's the most Pythonic way to parse out this value from a JSON-like blob?

See below. Given a well-known Google URL, I'm trying to retrieve data from that URL. That data will provide me another Google URL from which I can retrieve a list of JWKs.
>>> import requests, json
>>> open_id_config_url = 'https://ggp.sandbox.google.com/.well-known/openid-configuration'
>>> response = requests.get(open_id_config_url)
>>> r.status_code
200
>>> response.text
u'{\n "issuer": "https://www.stadia.com",\n "jwks_uri": "https://www.googleapis.com/service_accounts/v1/jwk/stadia-jwt#system.gserviceaccount.com",\n "claims_supported": [\n "iss",\n "aud",\n "sub",\n "iat",\n "exp",\n "s_env",\n "s_app_id",\n "s_gamer_tag",\n "s_purchase_country",\n "s_current_country",\n "s_session_id",\n "s_instance_ip",\n "s_restrict_text_chat",\n "s_restrict_voice_chat",\n "s_restrict_multiplayer",\n "s_restrict_stream_connect",\n ],\n "id_token_signing_alg_values_supported": [\n "RS256"\n ],\n}'
Above I have successfully retrieved the data from the first URL. I can see the entry jwks_uri contains the second URL I need. But when I try to convert that blob of text to a python dictionary, it fails.
>>> response.json()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/saqib.ali/saqib-env-99/lib/python2.7/site-packages/requests/models.py", line 889, in json
self.content.decode(encoding), **kwargs
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
>>> json.loads(response.text)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
The only way I can get out the JWKs URL is by doing this ugly regular expression parsing:
>>> re.compile('(?<="jwks_uri": ")[^"]+').findall(response.text)[0]
u'https://www.googleapis.com/service_accounts/v1/jwk/stadia-jwt#system.gserviceaccount.com'
Is there a cleaner, more Pythonic way to extract this string?
I really wish Google would send back a string that could be cleanly JSON-ified.

The returned json string is incorrect because last item of the dictionary ends with ,, which json cannot parse.
": [\n "RS256"\n ],\n}'
^^^
But ast.literal_eval can do that (as python parsing accepts lists/dicts that end with a comma). As long as you don't have booleans or null values, it is possible and pythonic
>>> ast.literal_eval(response.text)["jwks_uri"]
'https://www.googleapis.com/service_accounts/v1/jwk/stadia-jwt#system.gserviceaccount.com'

Your JSON is invalid because it has an extra comma after the last value in the claims_supported array.
I wouldn't necessarily recommend it, but you could use the similarity of JSON and Python syntax to parse this directly, since Python is much less picky:
ast.literal_eval(response.tezt)

As suggested in this answer use yaml to parse json. It will tolerate the trailing comma as well as other deviations from the json standard.
import yaml
d = yaml.load(response.text)

json.loads(string) throws json.decoder.JSONDecodeError when trying to convert string to dictionary

I'm using the requests library to make an api call. The json response is then formatted as a string and sent as part of a result to my server as shown by the code snippet:
def get_and_send(url, method):
resp = requests.request(url=url, method=method, **kwargs)
result = f'{{ "status_code":{resp.status_code}, "content":{resp.json()} }}'
send_to_server(result)
I intend to convert this result back to a dictionary object from the string result on the server.
The problem I have is that when I use json.loads(result) to convert the string to dictionary object, it throws the following error
Exception in thread Thread-2: Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/adipster/PycharmProjects/ScriptBackbone/ts_server/agent_thread.py", line 39, in run
resp_data = self._task_formatter.format_response(response) # Formats the response
File "/home/adipster/PycharmProjects/ScriptBackbone/utils/task_formatter.py", line 26, in format_response
response = self.get_dict_response(response.decode().strip())
File "/home/adipster/PycharmProjects/ScriptBackbone/utils/task_formatter.py", line 36, in get_dict_response
raise exp
File "/home/adipster/PycharmProjects/ScriptBackbone/utils/task_formatter.py", line 34, in get_dict_response
return json.loads(response)
File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 32 (char 31)
I understand that the error is because all my keys have to be in double quotes which is not the case when resp.json() is formatted to a string.
Please can someone help out on how I could ensure that all the keys of my dictionary object are in double quotes?
Or any other alternative to this will be welcomed. Thanks

The issue is, as you point out, that if you just do string interpolation (with an f-string, for example), the quotes for strings in the JSON object will be single quotes, but JSON format requires double quotes.
To fix this you can use json.dumps which takes a JSON object (in Python) and converts it to a properly formatted JSON string. Edit thanks to Charles Duffy: You can avoid the f-string entirely by just creating the whole result_data object as a dictionary, then converting it to JSON all at once using json.dumps.
For example:
import json
def get_and_send(url, method):
resp = requests.request(url=url, method=method, **kwargs)
result_data = {
"status_code": resp.status_code,
"content": resp.json(),
}
result = json.dumps(result_data)
send_to_server(result)

Using Python's json.loads with JSON containing regex value, breaks?

TL,DR; How can JSON containing a regex with escaped backslahes, be loaded using Python's JSON decoder?
Detail; The regular expression \\[0-9]\\ will match (for example):
\2\
The same regular expression could be encoded as a JSON value:
{
"pattern": "\\[0-9]\\"
}
And in turn, the JSON value could be encoded as a string in Python (note the single quotes):
'{"pattern": "\\[0-9]\\"}'
When loading the JSON in Python, a JSONDecodeError is raised:
import json
json.loads('{"pattern": "\\[0-9]\\"}')
The problem is caused by the regular expression escaping the blackslashes:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid \escape: line 1 column 14 (char 13)
>>> json.loads('{"pattern": "\\[0-9]\\"}')
This surprised me since each step seems reasonable (i.e. valid regex, valid JSON, and valid Python).
How can JSON containing a regex with escaped backslahes, be loaded using Python's JSON decoder?

What's happening is that Python is first escaping the input to loads as a string literal, making it '{"pattern": "\[0-9]\"}' (double backslash -> single backslash). Then, loads now attempts to escape \[, which is invalid. To fix, escape the backslashes again. However, it's easier and more practical to specify it as a raw string:
>>> import json
>>> json.loads('{"pattern": "\\[0-9]\\"}')
json.decoder.JSONDecodeError: Invalid \escape: line 1 column 14 (char 13)
>>> json.loads(r'{"pattern": "\\[0-9]\\"}')
{'pattern': '\\[0-9]\\'} # No error
Note that this problem won't apply if loading from a file.
test.json:
{"pattern": "\\[0-9]\\"}
Python:
import json
with open('test.json', 'r') as infile:
json.load(infile) # no problem
Basically, the problem arises with the fact that you're passing in a string literal, but ironically, your string literal isn't being taken literally.

The r means that the string is to be treated as a raw string, which means all escape codes will be ignored:
json.loads(r'{"pattern": "\\[0-9]\\"}')

how to pass directly variable name as dict key with json.loads in python?

I'm making a script that write other script, arguments are passed as JSON while calling the script from terminal.
The script that need to be written contains a dictionnary.
One of the key value in this dict is a variable name (not a string) call strategy.
My problem looks like this.
d = json.loads(sys.argv[2])
# d should looks like that
d = {
"stopLossValue": 5,
"strategy": strategy,
"strategyTitle": "week5"
}
dic = """
parameterDict = {}
""".format(json.dumps(d, sort_keys=True, indent=4))
Running the script return an error that disapear if i set strategy key value as string.
Error:
Traceback (most recent call last):
File "updateCandleStrategy.py", line 11, in <module>
d = json.loads(sys.argv[2])
File "/usr/lib/python3.4/json/__init__.py", line 318, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.4/json/decoder.py", line 343, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.4/json/decoder.py", line 361, in raw_decode
raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 506 (char 505)
Is there a simple way to achieve my goal?
Thanks

when passing a json string from command line, there's a great chance that one of the quotes / escape char is interpreted by the underlying shell.
So that's not a viable/reliable method to pass json strings. Pass a file containing json data instead and read it:
with open(sys.argv[2]) as f:
d = json.load(f)
Example from windows console, just printing second argument:
S:\python>foo.py ff "d = {"s":12,"d":15}"
d = {s:12,d:15}
the quotes have been removed. Would need to double them.
On a Linux terminal, wrapping your argument into single quotes could solve most situations, though, until you stumble on a value containing a single quote...
Instead of passing a dictionary, why not using getopt or argparse and build/parse a proper command line?

Read JSON formatted string in Python

I have a simple Websockets server in python, it receives messages from Android app clients, I tried to make the message payload from the client in JSON but I felt. It is only working when it is in String.
One solution I found is to keep the message string but with JSON format:
try {
json.put("name", "Jack");
json.put("age", "24");
message = json.toString(2);
} catch (JSONException e) {
e.printStackTrace();
}
webSocket.send(message);
Inspired by the Javascript JSON.stringify(message)
I printed the message on the server and it seems to be formatted
My question is how can I reverse back it into JSON on the server when it received?
I tried this way in Python:
def on_message(self,message):
data = json.loads(message)
self.write_message(data['name'])
but I got this error:
ERROR:tornado.application:Uncaught exception in /
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/tornado/websocket.py", line 494, in _run_callback
result = callback(*args, **kwargs)
File "index.py", line 24, in on_message
data = json.loads(message)
File "/usr/lib/python3.4/json/__init__.py", line 318, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.4/json/decoder.py", line 343, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.4/json/decoder.py", line 361, in raw_decode
raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)

You should use the json Python package. To have a JSON, you could simply do import json and json.dumps(message).

Use a Json package in python
import json
data = json.loads(your_var)
In data variable you get a json format data
hope this will help you

Will something like this work for you?
import json
# assume this is the JSON you receive
text = json.dumps(dict(name='Jack', age='24'))
# show the text to be converted
print(text)
# outputs: {"name": "Jack", "age": "24"}
# load a string and convert to Python object
# see `json` module more for details
obj = json.loads(text)

I tried this way and it worked for me, I converted the message into a dictionary using ast and then used the new message as I wanted:
formattedMessage = ast.literal_eval(format(message))
self.write_message(formattedMessage["name"])