I've a huge json file which has a lot of nested key value pairs. So I thought I should save the keys as dictionary values and use that dictionary values as keys to access the values from the json file. Say for example:
json_obj = {
"calendar" :{
"day": {
"activities" : {
"morning:"walk"
}
}
}
}
so I thought to access the key morning value, instead of writing
json_obj['calendar']['day']['activities']['morning']
I should keep a dictionary which will contain the query parameters like
query_parameters = {
0 :[['calendar'],['day'],['activities'],['morning']]
}
And use this dictionary to query from the json object.
But
Here is my question?
Can I use this dictionary to write query or to access values from my json_obj without using any loops ?
say something like json_obj[query_parameters[0]] # this is not right syntax I know
Or do you have any suggestions when accessing these long key value pair from an object?
You can write a function like this
This function will return value if exist otherwise returns None
def fun(query, data):
if not query:
return data
if query[0][0] in data:
return fun(query[1:], data[query[0][0]])
print(fun(query_parameters[0], json_obj)) # walk
Related
I have a json data stored in a variable.
Json Data:
{ "XYZ": { "abc":[{....}] } }
From above JSON Data, I should not have XYZ node. I need JSON Data as
{ "abc":[{....}] }
How can I remove using Python?
Thanks a lot!
You want to look up the dictionary via it's key ("XYZ") and assign it to a new variable:
data = request_data.get("XYZ")
this will return None if there is no "XYZ" key, but if you know the "XYZ" key is always going to be there you can lookup like so:
data = requests_data["XYZ"]
I need to pull data in from elasticsearch, do some cleaning/munging and export as table/rds.
To do this I have a long list of variable names required to pull from elasticsearch. This list of variables is required for the pull, but the issue is that not all fields may be represented within a given pull, meaning that I need to add the fields after the fact. I can do this using a schema (in nested json format) of the same list of variable names.
To try and [slightly] future proof this work I would ideally like to only maintain the list/schema in one place, and convert from list to schema (or vice-versa).
Is there a way to do this in python? Please see example below of input and desired output.
Small part of schema:
{
"_source": {
"filters": {"group": {"filter_value": 0}},
"user": {
"email": "",
"uid": ""
},
"status": {
"date": "",
"active": True
}
}
}
Desired string list output:
[
"_source.filters.group.filter_value",
"_source.user.email",
"_source.user.uid",
"_source.status.date",
"_source.status.active"
]
I believe that schema -> list might be an easier transformation than list -> schema, though am happy for it to be the other way round if that is simpler (though need to ensure the schema variables have the correct type, i.e. str, bool, float).
I have explored the following answers which come close, but I am struggling to understand since none appear to be in python:
Convert dot notation to JSON
Convert string with dot notation to JSON
Where d is your json as a dictionary,
def full_search(d):
arr = []
def dfs(d, curr):
if not type(d) == dict or curr[-1] not in d or type(d[curr[-1]]) != dict:
arr.append(curr)
return
for key in d[curr[-1]].keys():
dfs(d[curr[-1]], curr + [key])
for key in d.keys():
dfs(d, [key])
return ['.'.join(x) for x in arr]
If d is in json form, use
import json
res = full_search(json.loads(d))
The issue I'm having is that when i try to access the values within a nested dictionary, i cannot because it's returning a list instead of a dictionary.
I have a .json file with this format;
{
"users": [
{
"1": {
"1": "value",
"2": "value"
}
}
]
}
I load the .json file, and access the value i want by using this function
def load_json(fn):
with open(fn) as pf:
data = json.load(pf)
return data['users']['1']['2']
If i simply do return data it is a dictionary, but if try to access further by adding ['users'], it turns into a list and will give an index error if i try to access key #1 or #2 inside of that..
My objective is to obtain the value of the nested key #2 for example, ideally without having loop through it.
Your JSON contains an array (Python list) wrapping the inner dicts (that's what the [ and ] in your JSON indicate). All you need to do is change:
return data['users']['1']['2']
to:
return data['users'][0]['1']['2']
# ^^^ Added
to index the list to get into the inner dicts.
given your data structure, and following it down :
data is a dictionary - with one key 'users' and a value of a list
data['users'] is a list - with one entry
data['users'][0] is a dictionary - with one key '1' and a value of a dictionary
data['users'][0][1] is a dictionary - with two keys '1' and '2'
So you need to do do :
def load_json(fn):
with open(fn) as pf:
data = json.load(pf)
return data['users'][0]['1']['2']
I'm using this as a reference: Elegant way to remove fields from nested dictionaries
I have a large number of JSON-formatted data here and we've determined a list of unnecessary keys (and all their underlying values) that we can remove.
I'm a bit new to working with JSON and Python specifically (mostly did sysadmin work) and initially thought it was just a plain dictionary of dictionaries. While some of the data looks like that, several more pieces of data consists of dictionaries of lists, which can furthermore contain more lists or dictionaries with no specific pattern.
The idea is to keep the data identical EXCEPT for the specified keys and associated values.
Test Data:
to_be_removed = ['leecher_here']
easy_modo =
{
'hello_wold':'konnichiwa sekai',
'leeching_forbidden':'wanpan kinshi',
'leecher_here':'nushiyowa'
}
lunatic_modo =
{
'hello_wold':
{'
leecher_here':'nushiyowa','goodbye_world':'aokigahara'
},
'leeching_forbidden':'wanpan kinshi',
'leecher_here':'nushiyowa',
'something_inside':
{
'hello_wold':'konnichiwa sekai',
'leeching_forbidden':'wanpan kinshi',
'leecher_here':'nushiyowa'
},
'list_o_dicts':
[
{
'hello_wold':'konnichiwa sekai',
'leeching_forbidden':'wanpan kinshi',
'leecher_here':'nushiyowa'
}
]
}
Obviously, the original question posted there isn't accounting for lists.
My code, modified appropriately to work with my requirements.
from copy import deepcopy
def remove_key(json,trash):
"""
<snip>
"""
keys_set = set(trash)
modified_dict = {}
if isinstance(json,dict):
for key, value in json.items():
if key not in keys_set:
if isinstance(value, dict):
modified_dict[key] = remove_key(value, keys_set)
elif isinstance(value,list):
for ele in value:
modified_dict[key] = remove_key(ele,trash)
else:
modified_dict[key] = deepcopy(value)
return modified_dict
I'm sure something's messing with the structure since it doesn't pass the test I wrote since the expected data is exactly the same, minus the removed keys. The test shows that, yes it's properly removing the data but for the parts where it's supposed to be a list of dictionaries, it's only getting returned as a dictionary instead which will have unfortunate implications down the line.
I'm sure it's because the function returns a dictionary but I don't know to proceed from here in order to maintain the structure.
At this point, I'm needing help on what I could have overlooked.
When you go through your json file, you only need to determine whether it is a list, a dict or neither. Here is a recursive way to modify your input dict in place:
def remove_key(d, trash=None):
if not trash: trash = []
if isinstance(d,dict):
keys = [k for k in d]
for key in keys:
if any(key==s for s in trash):
del d[key]
for value in d.values():
remove_key(value, trash)
elif isinstance(d,list):
for value in d:
remove_key(value, trash)
remove_key(lunatic_modo,to_be_removed)
remove_key(easy_modo,to_be_removed)
Result:
{
"hello_wold": {
"goodbye_world": "aokigahara"
},
"leeching_forbidden": "wanpan kinshi",
"something_inside": {
"hello_wold": "konnichiwa sekai",
"leeching_forbidden": "wanpan kinshi"
},
"list_o_dicts": [
{
"hello_wold": "konnichiwa sekai",
"leeching_forbidden": "wanpan kinshi"
}
]
}
{
"hello_wold": "konnichiwa sekai",
"leeching_forbidden": "wanpan kinshi"
}
import json
data ='''
{
"names": {"first_boy" : "khaled"},
"names": {"second_boy" : "waseem"}
}
'''
info = json.loads(data)
for line in info:
print(info["names"])
I expected it to print the first_boy and the second_boy dictionary ,but it's printing
{'second_boy': 'waseem'}
Dicts in python can only support one of the same key. Similarly, most implementations of JSON do not allow duplicate keys. The way python handles this, when using json.loads() (or anything else that constructs a dict) is to simply use the most recent definition of any given key.
In this case, {"second_boy":"waseem"} overwrites {"first_boy":"khaled"}.
The problem here is that the key "names" exists 2 times.
Maybe you can do this:
import json
data ='''
{
"names": {"first_boy" : "khaled",
"second_boy" : "waseem"}
}
'''
info = json.loads(data)
for key, value in info['names'].items():
print(key, value)