Develop Reference

Python how to parse a list[dict] in python and convert values from nested dictionaries as keys

Need help in writing clean code , I have a yaml parsed output which looks like this :
yaml_output = [{'name' : 'alex', 'subjects' : {'maths' : ['grade_1', 'grade_2']}},
{'name' : 'rio', 'subjects' : {'maths' : ['grade_3', 'grade_2'], 'science : ['grade_4', 'grade_6']}}]
I want it create a list of dictionaries with subjects as key and name of the students as value in a dictionary where grades are the keys.
desired_output = [{'maths' : {'grade_1' : ['alex'], 'grade_2' : ['alex', 'rio'], 'grade_3' : ['rio']}}, {'science' : {'grade_4' : ['rio'], 'grade_6' : ['rio']}
needs subjects as key and name of the students as value in a dictionary where grades are the keys.
new_dict = []
for dict in yaml_output:
for k,v in dict:
for i,j in dict['subjects']:
if any(i in dict_list for dict_list in new_dict):
dict_list[i].append(v)

You were close. Your for k,v loop is looking at the wrong data. You don't want to look at ALL the keys, you want to unravel the subjects key and reference the "name" specifically.
yaml_output = [{'name' : 'alex', 'subjects' : {'maths' : ['grade_1', 'grade_2']}},
{'name' : 'rio', 'subjects' : {'maths' : ['grade_3', 'grade_2'], 'science': ['grade_4', 'grade_6']}}]
out = dict()
for data in yaml_output:
for k,v in data['subjects'].items():
if k not in out:
out[k] = {}
for g in v:
if g not in out[k]:
out[k][g] = []
out[k][g].append( data['name'] )
print(out)
Output:
{'maths': {'grade_1': ['alex'], 'grade_2': ['alex', 'rio'], 'grade_3': ['rio']}, 'science': {'grade_4': ['rio'], 'grade_6': ['rio']}}

Dictionary key name from variable

I am trying to create a nested dictionary, whereby the key to each nested dictionary is named from the value from a variable. My end result should look something like this:
data_dict = {
'jane': {'name': 'jane', 'email': 'jane#example.com'},
'jim': {'name': 'jim', 'email': 'jim#example.com'}
}
Here is what I am trying:
data_dict = {}
s = "jane"
data_dict[s][name] = 'jane'
To my surprise, this does not work. Is this possible?

You want something like:
data_dict = {}
s = "jane"
data_dict[s] = {}
data_dict[s]['name'] = s
That should work, though I would recommend instead of a nested dictionary that you use a dictionary of names to either namedtuples or instances of a class.

Try this:
data_dict = {}
s = ["jane", "jim"]
for name in s:
data_dict[name] = {}
data_dict[name]['name'] = name
data_dict[name]['email'] = name + '#example.com'

as #Milad in the comment mentioned, you first need to initialize s as empty dictionary first
data={}
data['Tom']={}
data['Tom']['name'] = 'Tom Marvolo Riddle'
data['Tom']['email'] = 'iamlordvoldermort.com'

For existing dictionaries you can do dict[key] = value although if there is no dict that would raise an error. I think this is the code you want to have:
data_dict = {}
s = "jane"
data_dict[s] = {"name": s, "email": f"{s}#example.com"}
print(data_dict)
I just realized when I got a notification about this question:
data_dict = defaultdict(dict)
data_dict["jane"]["name"] = "jane"
Would be a better answer I think.

Update keys in nested dictionary with some exception in key strings

So I've a list of students which looks something like this :
students = [ {'name': 'Jack' , 'status' : 'Average' , 'subjects' : { 'subject1' : 'English' , 'subject2' : 'Math' } , 'height' : '20cm' },
{'name': 'Tom' , 'status' : 'Good' , 'subjects' : { 'subject1' : 'English' , 'subject2' : 'Science' } , 'height' : '30cm' }
]
So the above list is of size 2. Assume that the size is pretty big, lets say 50 or 60 or more.
I want to return a list students_output & for each student I want to return a dictionary which contains the following values for each student which are fetched from the above list but have slightly modified 'keys'. The end output should be something like this :
students_output = [ {'student_name': 'Jack' , 'student_status' : 'Average' , 'student_subjects' : { 'student_subject1' : 'English' , 'student_subject2' : 'Math' } , 'child_height' : '20cm' },
{'student_name': 'Tom' , 'student_status' : 'Good' , 'student_subjects' : { 'student_subject1' : 'English' , 'student_subject2' : 'Science' } , 'child_height' : '30cm' }
]
I am not able to understand how I can create an effective loop so that the keys in my resultant data structure are maintained as provided in the output and i can fetch the data from the first list.
for example, in students_output, I know
students_output[0]['student_name']=students[0]['name']
But can anyone help me do it iteratively ?

In order to achieve this, you have to concatenate "student_" at the start of each key with some exception as "height" key. You may do it via combination of list comprehension and dict comprehension expression as:
students = [
{'name': 'Jack' , 'status' : 'Average' , 'subjects' : { 'subject1' : 'English' , 'subject2' : 'Math' } , 'height' : '20cm' },
{'name': 'Tom' , 'status' : 'Good' , 'subjects' : { 'subject1' : 'English' , 'subject2' : 'Science' } , 'height' : '30cm' }
]
def get_key(key):
return {
'height': 'child_height', # All exception you need in `key`
# apart from concatenating `"student_"`
}.get(key, 'student_' + key)
new_list = [{
get_key(k): ({
get_key(kk):v for kk, vv in v.items()} if isinstance(v, dict) else v) \
for k, v in s.items()
} for s in students]
Value hold by new_list will be:
[{'student_name': 'Jack', 'child_height': '20cm', 'student_status': 'Average', 'student_subjects': {'student_subject1': {'subject1': 'English', 'subject2': 'Math'}, 'student_subject2': {'subject1': 'English', 'subject2': 'Math'}}},
{'student_name': 'Tom', 'child_height': '30cm', 'student_status': 'Good', 'student_subjects': {'student_subject1': {'subject1': 'English', 'subject2': 'Science'}, 'student_subject2': {'subject1': 'English', 'subject2': 'Science'}}}]

Here's a quick-and-dirty function that will do what you need:
In [10]: def rename_keys(students):
...: d = {}
...: for k,v in students.items():
...: if isinstance(v,dict):
...: k = "student_" + k
...: v = rename_keys(v)
...: d[k] = v
...: elif k == 'height':
...: k = "child_height"
...: d[k] = v
...: else:
...: k = "student_" + k
...: d[k] = v
...: return d
...:
...:
In [11]: [rename_keys(d) for d in students]
Out[11]:
[{'child_height': '20cm',
'student_name': 'Jack',
'student_status': 'Average',
'student_subjects': {'student_subject1': 'English',
'student_subject2': 'Math'}},
{'child_height': '30cm',
'student_name': 'Tom',
'student_status': 'Good',
'student_subjects': {'student_subject1': 'English',
'student_subject2': 'Science'}}]
And really, this doesn't have to be recursive, you could substitute the recursive call with a dictionary comprehension:
v = {'student_'+key:value for key,value in v.items()}

You can use the following function inside a list comprehension like this:
def new_dict(d):
res = {}
for key, value in d.iteritems():
student_or_child = 'student' if key != 'height' else 'child'
if type(value) == dict:
res['{}_{}'.format(student_or_child, key)] = new_dict(value)
else:
res['{}_{}'.format(student_or_child, key)] = value
return res
The above function takes a dict as argument, for each key, value in the passed dict, if value is of type dict then the same function is called on value, and the result is added to res dict, else the same value is added.
Now, with a list comprehension, we can do:
[new_dict(d) for d in students]
Output:
>>> [new_dict(d) for d in students]
[{'child_height': '20cm', 'student_name': 'Jack', 'student_status': 'Average', 'student_subjects': {'student_subject1': 'English', 'student_subject2': 'Math'}}, {'child_height': '30cm', 'student_name': 'Tom', 'student_status': 'Good', 'student_subjects': {'student_subject1': 'English', 'student_subject2': 'Science'}}]

How to replace a Value in a Dict from another Dict but based on a key value in the first Dict

For say I have two dictionaries like so:
dict1 = {'project' : 'Administration', 'cost_center' : 'N/A'}
{'project' : 'Non-Work Time', 'cost_center' : 'N/A'}
dict2 = {'project':'Administration', 'cost_center': 'F99'}
{'project':'Non-Work Time', 'cost_center': 'F88'}
I need help finding a way to replace 'N/A' with the value from dict2 cost_center. I think an IF statement would be best like: if it finds "N/A" then replace with the value that is in dict2 if it doesn't find that value keep it "N/A"
Example of desired output:
dict1 = {'project' : 'Administration', 'cost_center' : 'F99'}
{'project' : 'Non-Work Time', 'cost_center' : 'F88'}
Pseudo:
if dict1['cost_center'] == "N/A':
Replace 'cost' if 'project' == exists in dict2
else:
continue

You can use a dict comprehension :
>>> dict1 = {'project' : 'Non-Work Time', 'cost' : 'N/A'}
>>> dict2 = {'Administration': 'F99' , 'Non-Work Time': 'Q99'}
>>> dict1 = {i:dict2[dict1['project']] if i=='cost' and j=='N/A' else j for i,j in dict1.items()}
>>> dict1
{'project': 'Non-Work Time', 'cost': 'Q99'}
But note that your dictionaries must contain the expected keys in otherwise it will raise a KeyError.

Xpath like query for nested python dictionaries

Is there a way to define a XPath type query for nested python dictionaries.
Something like this:
foo = {
'spam':'eggs',
'morefoo': {
'bar':'soap',
'morebar': {'bacon' : 'foobar'}
}
}
print( foo.select("/morefoo/morebar") )
>> {'bacon' : 'foobar'}
I also needed to select nested lists ;)
This can be done easily with #jellybean's solution:
def xpath_get(mydict, path):
elem = mydict
try:
for x in path.strip("/").split("/"):
try:
x = int(x)
elem = elem[x]
except ValueError:
elem = elem.get(x)
except:
pass
return elem
foo = {
'spam':'eggs',
'morefoo': [{
'bar':'soap',
'morebar': {
'bacon' : {
'bla':'balbla'
}
}
},
'bla'
]
}
print xpath_get(foo, "/morefoo/0/morebar/bacon")
[EDIT 2016] This question and the accepted answer are ancient. The newer answers may do the job better than the original answer. However I did not test them so I won't change the accepted answer.

One of the best libraries I've been able to identify, which, in addition, is very actively developed, is an extracted project from boto: JMESPath. It has a very powerful syntax of doing things that would normally take pages of code to express.
Here are some examples:
search('foo | bar', {"foo": {"bar": "baz"}}) -> "baz"
search('foo[*].bar | [0]', {
"foo": [{"bar": ["first1", "second1"]},
{"bar": ["first2", "second2"]}]}) -> ["first1", "second1"]
search('foo | [0]', {"foo": [0, 1, 2]}) -> [0]

There is an easier way to do this now.
http://github.com/akesterson/dpath-python
$ easy_install dpath
>>> dpath.util.search(YOUR_DICTIONARY, "morefoo/morebar")
... done. Or if you don't like getting your results back in a view (merged dictionary that retains the paths), yield them instead:
$ easy_install dpath
>>> for (path, value) in dpath.util.search(YOUR_DICTIONARY, "morefoo/morebar", yielded=True)
... and done. 'value' will hold {'bacon': 'foobar'} in that case.

Not exactly beautiful, but you might use sth like
def xpath_get(mydict, path):
elem = mydict
try:
for x in path.strip("/").split("/"):
elem = elem.get(x)
except:
pass
return elem
This doesn't support xpath stuff like indices, of course ... not to mention the / key trap unutbu indicated.

There is the newer jsonpath-rw library supporting a JSONPATH syntax but for python dictionaries and arrays, as you wished.
So your 1st example becomes:
from jsonpath_rw import parse
print( parse('$.morefoo.morebar').find(foo) )
And the 2nd:
print( parse("$.morefoo[0].morebar.bacon").find(foo) )
PS: An alternative simpler library also supporting dictionaries is python-json-pointer with a more XPath-like syntax.

dict > jmespath
You can use JMESPath which is a query language for JSON, and which has a python implementation.
import jmespath # pip install jmespath
data = {'root': {'section': {'item1': 'value1', 'item2': 'value2'}}}
jmespath.search('root.section.item2', data)
Out[42]: 'value2'
The jmespath query syntax and live examples: http://jmespath.org/tutorial.html
dict > xml > xpath
Another option would be converting your dictionaries to XML using something like dicttoxml and then use regular XPath expressions e.g. via lxml or whatever other library you prefer.
from dicttoxml import dicttoxml # pip install dicttoxml
from lxml import etree # pip install lxml
data = {'root': {'section': {'item1': 'value1', 'item2': 'value2'}}}
xml_data = dicttoxml(data, attr_type=False)
Out[43]: b'<?xml version="1.0" encoding="UTF-8" ?><root><root><section><item1>value1</item1><item2>value2</item2></section></root></root>'
tree = etree.fromstring(xml_data)
tree.xpath('//item2/text()')
Out[44]: ['value2']
Json Pointer
Yet another option is Json Pointer which is an IETF spec that has a python implementation:
https://github.com/stefankoegl/python-json-pointer
From the jsonpointer-python tutorial:
from jsonpointer import resolve_pointer
obj = {"foo": {"anArray": [ {"prop": 44}], "another prop": {"baz": "A string" }}}
resolve_pointer(obj, '') == obj
# True
resolve_pointer(obj, '/foo/another%20prop/baz') == obj['foo']['another prop']['baz']
# True
>>> resolve_pointer(obj, '/foo/anArray/0') == obj['foo']['anArray'][0]
# True

If terseness is your fancy:
def xpath(root, path, sch='/'):
return reduce(lambda acc, nxt: acc[nxt],
[int(x) if x.isdigit() else x for x in path.split(sch)],
root)
Of course, if you only have dicts, then it's simpler:
def xpath(root, path, sch='/'):
return reduce(lambda acc, nxt: acc[nxt],
path.split(sch),
root)
Good luck finding any errors in your path spec tho ;-)

Another alternative (besides that suggested by jellybean) is this:
def querydict(d, q):
keys = q.split('/')
nd = d
for k in keys:
if k == '':
continue
if k in nd:
nd = nd[k]
else:
return None
return nd
foo = {
'spam':'eggs',
'morefoo': {
'bar':'soap',
'morebar': {'bacon' : 'foobar'}
}
}
print querydict(foo, "/morefoo/morebar")

More work would have to be put into how the XPath-like selector would work.
'/' is a valid dictionary key, so how would
foo={'/':{'/':'eggs'},'//':'ham'}
be handled?
foo.select("///")
would be ambiguous.

Is there any reason for you to the query it the way like the XPath pattern? As the commenter to your question suggested, it just a dictionary, so you can access the elements in a nest manner. Also, considering that data is in the form of JSON, you can use simplejson module to load it and access the elements too.
There is this project JSONPATH, which is trying to help people do opposite of what you intend to do (given an XPATH, how to make it easily accessible via python objects), which seems more useful.

def Dict(var, *arg, **kwarg):
""" Return the value of an (imbricated) dictionnary, if all fields exist else return "" unless "default=new_value" specified as end argument
Avoid TypeError: argument of type 'NoneType' is not iterable
Ex: Dict(variable_dict, 'field1', 'field2', default = 0)
"""
for key in arg:
if isinstance(var, dict) and key and key in var: var = var[key]
else: return kwarg['default'] if kwarg and 'default' in kwarg else "" # Allow Dict(var, tvdbid).isdigit() for example
return kwarg['default'] if var in (None, '', 'N/A', 'null') and kwarg and 'default' in kwarg else "" if var in (None, '', 'N/A', 'null') else var
foo = {
'spam':'eggs',
'morefoo': {
'bar':'soap',
'morebar': {'bacon' : 'foobar'}
}
}
print Dict(foo, 'morefoo', 'morebar')
print Dict(foo, 'morefoo', 'morebar', default=None)
Have a SaveDict(value, var, *arg) function that can even append to lists in dict...

I reference form this link..
Following code is for json xpath base parse implemented in python :
import json
import xmltodict
# Parse the json string
class jsonprase(object):
def __init__(self, json_value):
try:
self.json_value = json.loads(json_value)
except Exception :
raise ValueError('must be a json str value')
def find_json_node_by_xpath(self, xpath):
elem = self.json_value
nodes = xpath.strip("/").split("/")
for x in range(len(nodes)):
try:
elem = elem.get(nodes[x])
except AttributeError:
elem = [y.get(nodes[x]) for y in elem]
return elem
def datalength(self, xpath="/"):
return len(self.find_json_node_by_xpath(xpath))
#property
def json_to_xml(self):
try:
root = {"root": self.json_value}
xml = xmltodict.unparse(root, pretty=True)
except ArithmeticError :
pyapilog().error(e)
return xml
Test Json :
{
"responseHeader": {
"zkConnected": true,
"status": 0,
"QTime": 2675,
"params": {
"q": "TxnInitTime:[2021-11-01T00:00:00Z TO 2021-11-30T23:59:59Z] AND Status:6",
"stats": "on",
"stats.facet": "CountryCode",
"rows": "0",
"wt": "json",
"stats.field": "ItemPrice"
}
},
"response": {
"numFound": 15162439,
"start": 0,
"maxScore": 1.8660598,
"docs": []
}
}
Test Code to read the values from above input json.
numFound = jsonprase(ABOVE_INPUT_JSON).find_json_node_by_xpath('/response/numFound')
print(numFound)