See below. Given a well-known Google URL, I'm trying to retrieve data from that URL. That data will provide me another Google URL from which I can retrieve a list of JWKs.
>>> import requests, json
>>> open_id_config_url = 'https://ggp.sandbox.google.com/.well-known/openid-configuration'
>>> response = requests.get(open_id_config_url)
>>> r.status_code
200
>>> response.text
u'{\n "issuer": "https://www.stadia.com",\n "jwks_uri": "https://www.googleapis.com/service_accounts/v1/jwk/stadia-jwt#system.gserviceaccount.com",\n "claims_supported": [\n "iss",\n "aud",\n "sub",\n "iat",\n "exp",\n "s_env",\n "s_app_id",\n "s_gamer_tag",\n "s_purchase_country",\n "s_current_country",\n "s_session_id",\n "s_instance_ip",\n "s_restrict_text_chat",\n "s_restrict_voice_chat",\n "s_restrict_multiplayer",\n "s_restrict_stream_connect",\n ],\n "id_token_signing_alg_values_supported": [\n "RS256"\n ],\n}'
Above I have successfully retrieved the data from the first URL. I can see the entry jwks_uri contains the second URL I need. But when I try to convert that blob of text to a python dictionary, it fails.
>>> response.json()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/saqib.ali/saqib-env-99/lib/python2.7/site-packages/requests/models.py", line 889, in json
self.content.decode(encoding), **kwargs
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
>>> json.loads(response.text)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/Cellar/python#2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
The only way I can get out the JWKs URL is by doing this ugly regular expression parsing:
>>> re.compile('(?<="jwks_uri": ")[^"]+').findall(response.text)[0]
u'https://www.googleapis.com/service_accounts/v1/jwk/stadia-jwt#system.gserviceaccount.com'
Is there a cleaner, more Pythonic way to extract this string?
I really wish Google would send back a string that could be cleanly JSON-ified.
The returned json string is incorrect because last item of the dictionary ends with ,, which json cannot parse.
": [\n "RS256"\n ],\n}'
^^^
But ast.literal_eval can do that (as python parsing accepts lists/dicts that end with a comma). As long as you don't have booleans or null values, it is possible and pythonic
>>> ast.literal_eval(response.text)["jwks_uri"]
'https://www.googleapis.com/service_accounts/v1/jwk/stadia-jwt#system.gserviceaccount.com'
Your JSON is invalid because it has an extra comma after the last value in the claims_supported array.
I wouldn't necessarily recommend it, but you could use the similarity of JSON and Python syntax to parse this directly, since Python is much less picky:
ast.literal_eval(response.tezt)
As suggested in this answer use yaml to parse json. It will tolerate the trailing comma as well as other deviations from the json standard.
import yaml
d = yaml.load(response.text)
I want to load this JSON data in python :
JSON data:
{"status":"success","result":{"users":{"5543295":{"status":{"payment_verified":false,"identity_verified":false,"email_verified":true,"deposit_made":false,"phone_verified":false,"facebook_connected":false,"profile_complete":true},"avatar_large":"/img/unknown.png","avatar_cdn":"//cdn6.f-cdn.com/img/unknown.png","spam_profile":null,"search_languages":null,"corporate_users":null,"support_status":null,"last_name":null,"suspended":null,"primary_language":"en","timezone":{"country":"US","offset":-7.0,"id":105,"timezone":"America/Los_Angeles"},"membership_package":null,"qualifications":[{"description":"Foundation vWorker Member","level":null,"icon_url":"/img/insignia/vworker.png","icon_name":"vworker","score_percentage":null,"user_percentile":null,"type":null,"id":null,"name":null}],"id":5543295,"badges":null,"hourly_rate":null,"responsiveness":null,"first_name":null,"display_name":"vw693015vw","tagline":null,"account_balances":null,"public_name":"vw693015vw","role":"employer","location":{"administrative_area":null,"city":null,"country":{"highres_flag_url":"/img/flags/highres_png/unknown.png","code":null,"name":"","flag_url_cdn":"//cdn5.f-cdn.com/img/flags/png/unknown.png","highres_flag_url_cdn":"//cdn5.f-cdn.com/img/flags/highres_png/unknown.png","flag_url":"/img/flags/png/unknown.png"},"vicinity":null,"longitude":null,"full_address":null,"latitude":null},"closed":false,"email":null,"username":"vw693015vw","is_local":null,"endorsements":null,"jobs":[],"employer_reputation":{"user_id":5543295,"last3months":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"entire_history":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"earnings_score":0.0,"role":"employer","job_history":{"job_counts":[],"count_other":0},"last12months":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"project_stats":{"draft":0,"open":0,"complete":0,"pending":0,"work_in_progress":0}},"company":"","registration_date":1041968572,"is_active":null,"avatar_large_cdn":"//cdn6.f-cdn.com/img/unknown.png","profile_description":"","address":null,"limited_account":false,"portfolio_count":0,"preferred_freelancer":false,"true_location":null,"primary_currency":{"code":"USD","name":"US Dollar","country":"US","sign":"$","exchange_rate":1.0,"id":1},"mobile_tracking":null,"test_user":false,"chosen_role":"both","reputation":{"user_id":5543295,"last3months":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_again":0.0,"quality":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":0},"entire_history":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_again":0.0,"quality":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":0},"earnings_score":0.0,"role":"freelancer","job_history":{"job_counts":[],"count_other":0},"last12months":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_{"status":"success","result":{"users":{"5543296":{"status":{"payment_verified":false,"identity_verified":false,"email_verified":true,"deposit_made":false,"phone_verified":false,"facebook_connected":false,"profile_complete":true},"avatar_large":"/img/unknown.png","avatar_cdn":"//cdn6.f-cdn.com/img/unknown.png","spam_profile":null,"search_languages":null,"corporate_users":null,"support_status":null,"last_name":null,"suspended":null,"primary_language":"en","timezone":{"country":"IN","offset":5.5,"id":164,"timezone":"Asia/Calcutta"},"membership_package":null,"qualifications":[{"description":"Foundation vWorker Member","level":null,"icon_url":"/img/insignia/vworker.png","icon_name":"vworker","score_percentage":null,"user_percentile":null,"type":null,"id":null,"name":null}],"id":5543296,"badges":null,"hourly_rate":null,"responsiveness":null,"first_name":null,"display_name":"vw1619364vw","tagline":null,"account_balances":null,"public_name":"vw1619364vw","role":"employer","location":{"administrative_area":null,"city":null,"country":{"highres_flag_url":"/img/flags/highres_png/unknown.png","code":null,"name":"","flag_url_cdn":"//cdn5.f-cdn.com/img/flags/png/unknown.png","highres_flag_url_cdn":"//cdn5.f-cdn.com/img/flags/highres_png/unknown.png","flag_url":"/img/flags/png/unknown.png"},"vicinity":null,"longitude":null,"full_address":null,"latitude":null},"closed":false,"email":null,"username":"vw1619364vw","is_local":null,"endorsements":null,"jobs":[],"employer_reputation":{"user_id":5543296,"last3months":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"entire_history":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"earnings_score":0.0,"role":"employer","job_history":{"job_counts":[],"count_other":0},"last12months":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"project_stats":{"draft":0,"open":0,"complete":0,"pending":0,"work_in_progress":0}},"company":"","registration_date":1124680669,"is_active":null,"avatar_large_cdn":"//cdn6.f-cdn.com/img/unknown.png","profile_description":"","address":null,"limited_account":false,"portfolio_count":0,"preferred_freelancer":false,"true_location":null,"primary_currency":{"code":"USD","name":"US Dollar","country":"US","sign":"$","exchange_rate":1.0,"id":1},"mobile_tracking":null,"test_user":false,"chosen_role":"both","reputation":{"user_id":5543296,"last3months":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_again":0.0,"quality":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":0},"entire_history":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_again":0.0,"quality":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":0},"earnings_score":0.0,"role":"freelancer","job_history":{"job_counts":[],"count_other":0},"last12months":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_agaiagain":0.0,"quality":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":0},"project_stats":null},"avatar":"/img/unknown.png","cover_image":{"current_image":{"url":"//cdn2.f-cdn.com/static/img/profiles/cover-product.jpg","width":1920,"height":550,"id":null,"description":""},"past_images":null},"corporate":null,"force_verify":null}}},"request_id":"2520481ca4d6b8130bdc20f6114d8037"}
My Python code :
import json
json_data = open('test.txt', 'r').read()
data = json.loads(json_data)
I'm sure JSON data is valid, because is result of one website, I need to load this in python and get username value.
Actual error :
MacBook-Pro-di-admin:freelancer11 admin$ python load.py
Traceback (most recent call last):
File "load.py", line 6, in <module>
data = json.loads(json_data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting : delimiter: line 1 column 4099 (char 4098)
So if you head over to your favorite JSON validator, like JSONlint, you will see the JSON is not valid, and that is precisely what the error in python means.