Develop Reference

Referencing Values in a List (syntax issue?) [duplicate]

I wrote some code to get data from a web API. I was able to parse the JSON data from the API, but the result I gets looks quite complex. Here is one example:
>>> my_json
{'name': 'ns1:timeSeriesResponseType', 'declaredType': 'org.cuahsi.waterml.TimeSeriesResponseType', 'scope': 'javax.xml.bind.JAXBElement$GlobalScope', 'value': {'queryInfo': {'creationTime': 1349724919000, 'queryURL': 'http://waterservices.usgs.gov/nwis/iv/', 'criteria': {'locationParam': '[ALL:103232434]', 'variableParam': '[00060, 00065]'}, 'note': [{'value': '[ALL:103232434]', 'title': 'filter:sites'}, {'value': '[mode=LATEST, modifiedSince=null]', 'title': 'filter:timeRange'}, {'value': 'sdas01', 'title': 'server'}]}}, 'nil': False, 'globalScope': True, 'typeSubstituted': False}
Looking through this data, I can see the specific data I want: the 1349724919000 value that is labelled as 'creationTime'.
How can I write code that directly gets this value?
I don't need any searching logic to find this value. I can see what I need when I look at the response; I just need to know how to translate that into specific code to extract the specific value, in a hard-coded way. I read some tutorials, so I understand that I need to use [] to access elements of the nested lists and dictionaries; but I can't figure out exactly how it works for a complex case.
More generally, how can I figure out what the "path" is to the data, and write the code for it?

For reference, let's see what the original JSON would look like, with pretty formatting:
>>> print(json.dumps(my_json, indent=4))
{
"name": "ns1:timeSeriesResponseType",
"declaredType": "org.cuahsi.waterml.TimeSeriesResponseType",
"scope": "javax.xml.bind.JAXBElement$GlobalScope",
"value": {
"queryInfo": {
"creationTime": 1349724919000,
"queryURL": "http://waterservices.usgs.gov/nwis/iv/",
"criteria": {
"locationParam": "[ALL:103232434]",
"variableParam": "[00060, 00065]"
},
"note": [
{
"value": "[ALL:103232434]",
"title": "filter:sites"
},
{
"value": "[mode=LATEST, modifiedSince=null]",
"title": "filter:timeRange"
},
{
"value": "sdas01",
"title": "server"
}
]
}
},
"nil": false,
"globalScope": true,
"typeSubstituted": false
}
That lets us see the structure of the data more clearly.
In the specific case, first we want to look at the corresponding value under the 'value' key in our parsed data. That is another dict; we can access the value of its 'queryInfo' key in the same way, and similarly the 'creationTime' from there.
To get the desired value, we simply put those accesses one after another:
my_json['value']['queryInfo']['creationTime'] # 1349724919000

I just need to know how to translate that into specific code to extract the specific value, in a hard-coded way.
If you access the API again, the new data might not match the code's expectation. You may find it useful to add some error handling. For example, use .get() to access dictionaries in the data, rather than indexing:
name = my_json.get('name') # will return None if 'name' doesn't exist
Another way is to test for a key explicitly:
if 'name' in resp_dict:
name = resp_dict['name']
else:
pass
However, these approaches may fail if further accesses are required. A placeholder result of None isn't a dictionary or a list, so attempts to access it that way will fail again (with TypeError). Since "Simple is better than complex" and "it's easier to ask for forgiveness than permission", the straightforward solution is to use exception handling:
try:
creation_time = my_json['value']['queryInfo']['creationTime']
except (TypeError, KeyError):
print("could not read the creation time!")
# or substitute a placeholder, or raise a new exception, etc.

Here is an example of loading a single value from simple JSON data, and converting back and forth to JSON:
import json
# load the data into an element
data={"test1": "1", "test2": "2", "test3": "3"}
# dumps the json object into an element
json_str = json.dumps(data)
# load the json to a string
resp = json.loads(json_str)
# print the resp
print(resp)
# extract an element in the response
print(resp['test1'])

Try this.
Here, I fetch only statecode from the COVID API (a JSON array).
import requests
r = requests.get('https://api.covid19india.org/data.json')
x = r.json()['statewise']
for i in x:
print(i['statecode'])

Try this:
from functools import reduce
import re
def deep_get_imps(data, key: str):
split_keys = re.split("[\\[\\]]", key)
out_data = data
for split_key in split_keys:
if split_key == "":
return out_data
elif isinstance(out_data, dict):
out_data = out_data.get(split_key)
elif isinstance(out_data, list):
try:
sub = int(split_key)
except ValueError:
return None
else:
length = len(out_data)
out_data = out_data[sub] if -length <= sub < length else None
else:
return None
return out_data
def deep_get(dictionary, keys):
return reduce(deep_get_imps, keys.split("."), dictionary)
Then you can use it like below:
res = {
"status": 200,
"info": {
"name": "Test",
"date": "2021-06-12"
},
"result": [{
"name": "test1",
"value": 2.5
}, {
"name": "test2",
"value": 1.9
},{
"name": "test1",
"value": 3.1
}]
}
>>> deep_get(res, "info")
{'name': 'Test', 'date': '2021-06-12'}
>>> deep_get(res, "info.date")
'2021-06-12'
>>> deep_get(res, "result")
[{'name': 'test1', 'value': 2.5}, {'name': 'test2', 'value': 1.9}, {'name': 'test1', 'value': 3.1}]
>>> deep_get(res, "result[2]")
{'name': 'test1', 'value': 3.1}
>>> deep_get(res, "result[-1]")
{'name': 'test1', 'value': 3.1}
>>> deep_get(res, "result[2].name")
'test1'

How do I get the value of a dict item within a list, within a dict?

How do I get the value of a dict item within a list, within a dict in Python? Please see the following code for an example of what I mean.
I use the following lines of code in Python to get data from an API.
res = requests.get('https://api.data.amsterdam.nl/bag/v1.1/nummeraanduiding/', params)
data = res.json()
data then returns the following Python dictionary:
{
'_links': {
'next': {
'href': null
},
'previous': {
"href": null
},
'self': {
'href': 'https://api.data.amsterdam.nl/bag/v1.1/nummeraanduiding/'
}
},
'count': 1,
'results': [
{
'_display': 'Maple Street 99',
'_links': {
'self': {
'href': 'https://api.data.amsterdam.nl/bag/v1.1/nummeraanduiding/XXXXXXXXXXXXXXXX/'
}
},
'dataset': 'bag',
'landelijk_id': 'XXXXXXXXXXXXXXXX',
'type_adres': 'Hoofdadres',
'vbo_status': 'Verblijfsobject in gebruik'
}
]
}
Using Python, how do I get the value for 'landelijk_id', represented by the twelve Xs?

This should work:
>>> data['results'][0]['landelijk_id']
"XXXXXXXXXXXXXXXX"
You can just chain those [] for each child you need to access.

I'd recommend using the jmespath package to make handling nested Dictionaries easier. https://pypi.org/project/jmespath/
import jmespath
import requests
res = requests.get('https://api.data.amsterdam.nl/bag/v1.1/nummeraanduiding/', params)
data = res.json()
print(jmespath.search('results[].landelijk_id', data)

Generating specific fields from JSON objects

I have a rather massive JSON objects and I'm trying to generate a specific JSON using only certain elements from it.
My current code looks like this:
data = get.req("/v2/users")
data = json.loads(data)
print (type(data)) # This returns <class 'list'>
print (data) # Below for what this returns
all_data = []
for d in data:
login_value = d['login']
if login_value.startswith('fe'):
continue
s = get.req("/v2/users/" + str(login_value)) # Sending another request with each
# login from the first request
all_data.append(s)
print (all_data) # Below for what this looks like this
print (data) before json.loads is str for the information, it returns data like this:
[
{
"id": 68663,
"login": "test1",
"url": "https://x.com/test"
},
{
"id": 67344,
"login": "test2",
"url": "https://x.com/test"
},
{
"id": 66095,
"login": "hi",
"url": "https://x.com/test"
}
]
print (all_data) returns a similar result to this for every user a request was sent for the first time
[b'{"id":68663,"email":"x#gmail.com","login":"xg","phone":"hidden","fullname":"xg gx","image_url":"https://imgur.com/random.png","mod":false,"points":5,"activity":0,"groups":['skill': 'archery']}
And this repeats for every user.
What I'm attempting to do is filtering by a few fields from all those results I received, so the final JSON I have will look something like this
[
{
"email": "x#gmail.com",
"fullname": "xg gf",
"points": 5,
"image_url", "https://imgur.com/random.pmg"
},
{
... similar json for the next user and so on
}
]
I feel as if the way I'm iterating over the data might be inefficient, so if you could guide me to a better way it would be wonderful.

In order for you to fetch login value, you have to iterate over data atleast once and for fetching details for every user, you have to make one call and that is exactly what you have done.
After you receive the user details instead of appending the whole object to the all_data list just take the field you need and construct a dict of it and then append it to all_data.
So your code has time complexity of O(n) which is best I understand.
Edit :
For each user you are receiving a byte response like below.
byte_response = [ b'{"id":68663,"email":"x#gmail.com","login":"xg","phone":"hidden","fullname":"xg gx","image_url":"https://imgur.com/random.png","mod":false,"points":5,"activity":0,"groups":[]}']
I'm not sure why would you get a response in a list [], but if it like that then take byte_response[0] so that we have the actual byte data like below.
byte_response = b'{"id":68663,"email":"x#gmail.com","login":"xg","phone":"hidden","fullname":"xg gx","image_url":"https://imgur.com/random.png","mod":false,"points":5,"activity":0,"groups":[]}'
response_decoded = byte_response.decode("utf-8") #decode it
import json
json_object_in_dict_form = json.loads(response_decoded) #convert it into dictionary
and then...
json_object_in_dict_form['take the field u want']

you can write:
data = get.req("/v2/users")
data = json.loads(data)
all_data = []
for d in data:
...
s = get.req("/v2/users/" + str(login_value))
new_data = {
'email': s['email'],
'fullname': s['fullname'],
'points': s['points'],
'image_url': s['image_url']
}
all_data.append(new_data)
print (all_data)
or you can make it fancy using an array with the fields you need:
data = get.req("/v2/users")
data = json.loads(data)
all_data = []
fields = ['email', 'fullname', 'point', 'image_url']
for d in data:
...
s = get.req("/v2/users/" + str(login_value))
new_data = dict()
for field in fields:
new_data[field] = s[field]
all_data.append(new_data)
print (all_data)

error in accessing a nested json within a list in a json in python

I have this sample json data that is coming as a response from a POST request- response is the name of below sample json data variable:
{
"statusCode": "OK",
"data": [
{
"id": -199,
"result": {
"title": "test1",
"group": "test_grp2"
}
},
{
"id": -201,
"result": {
"title": "test2",
"group": "test_grp2"
}
}
]
}
Now what I want to do is form a list of list where each list will have id,title,and group values from above json data. Here is what my code looks like:
def get_list():
# code to get the response from POST request
json_data = json.loads(response.text)
group_list = []
if json_data['statusCode'] == 'OK':
for data in json_data['data']:
print(data)
for result_data in data['result']:
title = result_data['title']
group = result_data['group']
group_list.append([data['id'],title,group])
return(group_list)
When I execute this I get error as list indices must be int not str at line title = result_data['title]. How can I form the list of list above?

data['result'] in your case is a dictionary. When you iterate over it using for result_data in data['result'], you get the keys - in other words, result_data is a string and it does not support string indexing.
Instead, you meant:
for data in json_data['data']:
result_data = data['result']
title = result_data['title']
group = result_data['group']
Or, alternatively, you can make use of a list comprehension:
group_list = [(data['id'], data['result']['title'], data['result']['group'])
for data in json_data['data']]
print(group_list)

You can extract the list of lists directly using a list comprehension:
>>> [[d['id'], d['result']['title'], d['result']['group']] for d in json_data['data']]
[[-199, 'test1', 'test_grp2'], [-201, 'test2', 'test_grp2']]

Replace value in JSON file for key which can be nested by n levels

I have JSON that looks like this:
{
"ROLE_NAME": {
"FOO": {
"download_url": "http: //something.staging/12345/buzz.zip"
},
"BAR": {
"download_url": "http: //something.staging/12345/fizz.zip"
},
"download_url": "http: //something.staging/12345/fizzbuzz.zip",
"db_name": "somedb",
"db_server": "dbserver.staging.dmz",
"plugin": {
"server_url": "http: //lab.staging.corp/server/"
}
}
}
I wrote a bit of python that replaces the "download_url" k:v with a new value (i.e. new download_url). Unfortunately it only replaces one of the three download_urls in that json snippet. I understand why, but am having a little difficulty getting the solution, and so I am here asking for help.
The entire json object is "data"
So I do something like this:
data["ROLE_NAME"]["download_url"] = download_url
Where download_url is a new value I have assigned to that variable
What I need to do is for any key called ["download_url"] then update it, rather than the one I have specified at the layer I am going to.
Some of my code to help:
I take some values obtained earlier in my code and build a url which returns a response. I extract a value from the response which will be used to build the value of download_url
buildinfo_url = "http://something.staging/guestAuth/app/rest/builds/?locator=buildType:%s,tags:%s,branch:branched:any" % (
bt_number,
list_json_load[role_name][0]['tag']
)
Send HTTP request
client = httplib2.Http()
response, xml = client.request(buildinfo_url)
Extract some value from the response xml and set download_url variable
doc = ElementTree.fromstring(xml)
for id in doc.findall('build'):
build_id = "%s" % (id.attrib['id'])
try:
download_url = "http://something.staging/guestAuth/repository/download/%s/%s:id/%s" % (
bt_number,
build_id,
build_artifact_zip
)
data[role_name]["download_url"] = download_url
except NameError:
print "something"
I think I should be recursively searching and updating

Using recursion
import json
json_txt = """
{
"ROLE_NAME": {
"FOO": {
"download_url": "http: //something.staging/12345/buzz.zip"
},
"BAR": {
"download_url": "http: //something.staging/12345/fizz.zip"
},
"download_url": "http: //something.staging/12345/fizzbuzz.zip",
"db_name": "somedb",
"db_server": "dbserver.staging.dmz",
"plugin": {
"server_url": "http: //lab.staging.corp/server/"
}
}
}
"""
data = json.loads(json_txt)
def fixup(adict, k, v):
for key in adict.keys():
if key == k:
adict[key] = v
elif type(adict[key]) is dict:
fixup(adict[key], k, v)
import pprint
pprint.pprint( data )
fixup(data, 'download_url', 'XXX')
pprint.pprint( data )
Output:
{u'ROLE_NAME': {u'BAR': {u'download_url': u'http: //something.staging/12345/fizz.zip'},
u'FOO': {u'download_url': u'http: //something.staging/12345/buzz.zip'},
u'db_name': u'somedb',
u'db_server': u'dbserver.staging.dmz',
u'download_url': u'http: //something.staging/12345/fizzbuzz.zip',
u'plugin': {u'server_url': u'http: //lab.staging.corp/server/'}}}
{u'ROLE_NAME': {u'BAR': {u'download_url': 'XXX'},
u'FOO': {u'download_url': 'XXX'},
u'db_name': u'somedb',
u'db_server': u'dbserver.staging.dmz',
u'download_url': 'XXX',
u'plugin': {u'server_url': u'http: //lab.staging.corp/server/'}}}