Develop Reference

Convert an uneven nested dictioary into tabular form

res = {'Head': {'Ide': 'GLE', 'ID': '7b', 'Source': 'CARS', 'Target': 'TULUM', 'Country': 'GL'},
'Load': {'Stat': {'Code': '21', 'Reason': 'invalid'}, 'SrcFilePath': '/path.xls'}}
res is the nested dictionary that needs to be converted into a tabular form.
With the following columns and respective values:
Ide ID Source Target Country Code Reason SrcFilePath
Code:
for col,data in res.items():
final_data = dict(data.items())
df = pd.DataFrame(final_data)
print(df)
Error:
ValueError: If using all scalar values, you must pass an index

You can try:
pd.DataFrame.from_dict(res, orient='index')

You could try using:
pd.json_normalize(res)
Although the output can be a bit "ugly", but it actually works.

I assume that res isn't the only record and there's data like:
data = [
{'Head': {'Ide': 'GLE', 'ID': '7b', 'Source': 'CARS', 'Target': 'TULUM', 'Country': 'GL'}, 'Load': {'Stat': {'Code': '21', 'Reason': 'invalid'}, 'SrcFilePath': '/path.xls'}}
, {'Head': {'Ide': 'ABC', 'ID': '8b', 'Source': 'CARS', 'Target': 'TULUM', 'Country': 'AB'}, 'Load': {'Stat': {'Code': '21', 'Reason': 'invalid'}, 'SrcFilePath': '/path.xls'}}
, {'Head': {'Ide': 'EFG', 'ID': '9b', 'Source': 'CARS', 'Target': 'TULUM', 'Country': 'EF'}, 'Load': {'Stat': {'Code': '21', 'Reason': 'invalid'}, 'SrcFilePath': '/path.xls'}}
]
So we have to write a procedure to flatten records and apply it by map to the data before transforming records into a frame:
def flatten_dict(d:dict) -> dict:
res = {}
for k, v in d.items():
if type(v) is dict:
res.update(flatten_dict(v))
else:
res[k] = v
return res
output = pd.DataFrame(map(flatten_dict, data))
The output:
Ide ID Source Target Country Code Reason SrcFilePath
0 GLE 7b CARS TULUM GL 21 invalid /path.xls
1 ABC 8b CARS TULUM AB 21 invalid /path.xls
2 EFG 9b CARS TULUM EF 21 invalid /path.xls

replacing keys from a dictionary inside a list python

I have a list inside a nested dictionary
body = {'Ready Date': '2020-01-31T12:00:00','Shipment Line List': [{'Description': 'Test', 'Weigth': '5',
'Height': '4.0','Length': '2.0', 'Width': '3.0'}, {'Description': 'Test', 'Weigth': '20', 'Height': '5',
'Length': '30', 'Width': '10']}
I want to iterate over the keys in the nested dictionary and replace "Weigth" with the correct spelling "Weight"
I tried this approach, but I am not getting the expected output
key = {"Weigth":"Weight"}
def find_replace(dict_body, dictionary):
# is the item in the dict?
for item in dict_body:
# iterate by keys
if item in dictionary.keys():
# look up and replace
dict_body = dict_body.replace(item, dictionary[item])
# return updated dict
return dict_body
a = find_replace(body,key)
print(a)

I think a better idea in this particular case is to treat everything as a string, replace and back as a dictionary. Because if you have multiple nested keys, it might be just be easier this way in two lines of code:
from ast import literal_eval
body = literal_eval(str(body).replace("Weigth","Weight"))
This outputs:
{'Ready Date': '2020-01-31T12:00:00',
'Shipment Line List': [{'Description': 'Test',
'Height': '4.0',
'Length': '2.0',
'Weight': '5',
'Width': '3.0'},
{'Description': 'Test',
'Height': '5',
'Length': '30',
'Weight': '20',
'Width': '10'}]}

I want to iterate over the keys in the nested dictionary and replace "Weigth" with the correct spelling "Weight"
something like the below
body = {'Ready Date': '2020-01-31T12:00:00', 'Shipment Line List': [{'Description': 'Test', 'Weigth': '5',
'Height': '4.0', 'Length': '2.0', 'Width': '3.0'},
{'Description': 'Test', 'Weigth': '20',
'Height': '5',
'Length': '30', 'Width': '10'}]}
for entry in body['Shipment Line List']:
entry['Weight'] = entry['Weigth']
del entry['Weigth']
print(body)
output
{'Ready Date': '2020-01-31T12:00:00', 'Shipment Line List': [{'Description': 'Test', 'Height': '4.0', 'Length': '2.0', 'Width': '3.0', 'Weight': '5'}, {'Description': 'Test', 'Height': '5', 'Length': '30', 'Width': '10', 'Weight': '20'}]}

inserting the new json data in the already existing json list

am trying to insert the json data into another json data list data using update function
datasets:
data = {'name':{'first' : 'xx', 'last': 'yy'}, 'class': {'standard': '5', 'section': '6'}}
maindata = [{'school': {'govt':{ 'value':'public1', 'item':'zphs1'}, 'convent':{'value':'private1','item':'HPS1'}}},
{'school': {'govt':{ 'value':'public2', 'item':'zphs2'}, 'convent':{'value':'private2','item':'HPS2'}}}]
Trying to fit in the data into main data after the tag, 'school'
expected output:
maindata = [{'school': {'govt':{ 'value':'public1', 'item':'zphs1'}, 'convent':{'value':'private1','item':'HPS1'}, 'name':{'first' : 'xx', 'last': 'yy'}, 'class': {'standard': '5', 'section': '6'}}},
{'school': {'govt':{ 'value':'public2', 'item':'zphs2'}, 'convent':{'value':'private2','item':'HPS2'},'name':{'first' : 'xx', 'last': 'yy'}, 'class': {'standard': '5', 'section': '6'}}}]
code used:
for i in maindata:
i['school'].update(data)
which returns None

The dictionary update method is not expected to return anything. It performs an in-place update that mutates the target dictionary. To see what this returns, try looking at the content of maindata after the update has run.

Your code is correct: the maindata dictionary is being updated. After making the following update:
for i in maindata:
i['school'].update(data)
check the value of maindata, it should be exactly what you're expecting it to be. You can also verify this by doing the following:
desired_output = [
{'school': {'govt':{ 'value':'public1', 'item':'zphs1'}, 'convent':{'value':'private1','item':'HPS1'}, 'name':{'first' : 'xx', 'last': 'yy'}, 'class': {'standard': '5', 'section': '6'}}},
{'school': {'govt':{ 'value':'public2', 'item':'zphs2'}, 'convent' {'value':'private2','item':'HPS2'},'name':{'first' : 'xx', 'last': 'yy'}, 'class': {'standard': '5', 'section': '6'}}}
]
for i in maindata:
i['school'].update(data)
maindata == desired_output
That works for me.

Accessing YAML data in Python

I have a YAML file that parses into an object, e.g.:
{'name': [{'proj_directory': '/directory/'},
{'categories': [{'quick': [{'directory': 'quick'},
{'description': None},
{'table_name': 'quick'}]},
{'intermediate': [{'directory': 'intermediate'},
{'description': None},
{'table_name': 'intermediate'}]},
{'research': [{'directory': 'research'},
{'description': None},
{'table_name': 'research'}]}]},
{'nomenclature': [{'extension': 'nc'}
{'handler': 'script'},
{'filename': [{'id': [{'type': 'VARCHAR'}]},
{'date': [{'type': 'DATE'}]},
{'v': [{'type': 'INT'}]}]},
{'data': [{'time': [{'variable_name': 'time'},
{'units': 'minutes since 1-1-1980 00:00 UTC'},
{'latitude': [{'variable_n...
I'm having trouble accessing the data in python and regularly see the error TypeError: list indices must be integers, not str
I want to be able to access all elements corresponding to 'name' so to retrieve each data field I imagine it would look something like:
import yaml
settings_stream = open('file.yaml', 'r')
settingsMap = yaml.safe_load(settings_stream)
yaml_stream = True
print 'loaded settings for: ',
for project in settingsMap:
print project + ', ' + settingsMap[project]['project_directory']
and I would expect each element would be accessible via something like ['name']['categories']['quick']['directory']
and something a little deeper would just be:
['name']['nomenclature']['data']['latitude']['variable_name']
or am I completely wrong here?

The brackets, [], indicate that you have lists of dicts, not just a dict.
For example, settingsMap['name'] is a list of dicts.
Therefore, you need to select the correct dict in the list using an integer index, before you can select the key in the dict.
So, giving your current data structure, you'd need to use:
settingsMap['name'][1]['categories'][0]['quick'][0]['directory']
Or, revise the underlying YAML data structure.
For example, if the data structure looked like this:
settingsMap = {
'name':
{'proj_directory': '/directory/',
'categories': {'quick': {'directory': 'quick',
'description': None,
'table_name': 'quick'}},
'intermediate': {'directory': 'intermediate',
'description': None,
'table_name': 'intermediate'},
'research': {'directory': 'research',
'description': None,
'table_name': 'research'},
'nomenclature': {'extension': 'nc',
'handler': 'script',
'filename': {'id': {'type': 'VARCHAR'},
'date': {'type': 'DATE'},
'v': {'type': 'INT'}},
'data': {'time': {'variable_name': 'time',
'units': 'minutes since 1-1-1980 00:00 UTC'}}}}}
then you could access the same value as above with
settingsMap['name']['categories']['quick']['directory']
# quick

Accessing python variables in a list

In the following code below, how to retrieve the value of id,Id has multiple values in it.How to access the values of id and update it to result1
def parse_results ():
try:
xml = minidom.parseString(new_results)
for xmlchild in xmldoc.childNodes[0].childNodes :
result1 = {}
result1.update ({'firstname': xmlchild.getElementsByTagName("firstname")[0].childNodes[0].nodeValue})
result1.update ({'lastname': xmlchild.getElementsByTagName("lastname")[0].childNodes[0].nodeValue})
result1.update ({'address': address})
if xmlchild.getElementsByTagName("id")[0].childNodes[0].nodeValue:
logging.debug(xmlchild.getElementsByTagName("id")[0].childNodes[0].nodeValue.lower())
result1.update ({'id': id})
Edit:
xmlchild.getElementsByTagName("id")[0].childNodes[0].nodeValue -this statement gives an exception
Adding XML:
<info><firstname>firstname</firstname><lastname>lastname</lastname><id>2</id></info>
<info><firstname>firstname</firstname><lastname>lastname</lastname><id>3</id></info>
<info><firstname>firstname</firstname><lastname>lastname</lastname><id>4</id></info>

Why are you using minidom? It is really boring to use.
I suggest you move to element tree:
import xml.etree.ElementTree as et
d = et.fromstring('''
<doc>
<info><firstname>firstname</firstname><lastname>lastname</lastname><id>2</id></info>
<info><firstname>firstname</firstname><lastname>lastname</lastname><id>3</id></info>
<info><firstname>firstname</firstname><lastname>lastname</lastname><id>4</id></info>
</doc>
''')
result = [dict((el.tag, el.text) for el in info) for info in d.findall('info')]
print result
That prints:
[{'firstname': 'firstname', 'id': '2', 'lastname': 'lastname'},
{'firstname': 'firstname', 'id': '3', 'lastname': 'lastname'},
{'firstname': 'firstname', 'id': '4', 'lastname': 'lastname'}]