Develop Reference

JSON viewers don't accept my pattern even after dict going through json.dumps() + json.loads()

The result when printing after a = json.dumps(dicter) and print(json.loads(a)) is this:
{
'10432981': {
'tournament': {
'name': 'Club Friendly Games',
'slug': 'club-friendly-games',
'category': {
'name': 'World',
'slug': 'world',
'sport': {
'name': 'Football',
'slug': 'football',
'id': 1
},
'id': 1468,
'flag': 'international'
},
'uniqueTournament': {
'name': 'Club Friendly Games',
'slug': 'club-friendly-games',
'category': {
'name': 'World',
'slug': 'world',
'sport': {
'name': 'Football',
'slug': 'football',
'id': 1
},
'id': 1468,
'flag': 'international'
},
'userCount': 0,
'hasPositionGraph': False,
'id': 853,
'hasEventPlayerStatistics': False,
'displayInverseHomeAwayTeams': False
},
'priority': 0,
'id': 86
}
}
}
But when trying to read in any json viewer, they warn that the format is incorrect but don't specify where the problem is.
If it doesn't generate any error when converting the dict to JSON and not even when reading it, why do views warn of failure?

You must enclose the strings using double quotes ("). The json.loads returns a python dictionary, so it is not a valid JSON object. If you want to get valid JSON you can get the string that json.dumps returns.

How to get key values in a json object(Python)

This is my json :
{'1': {'name': 'poulami', 'password': 'paul123', 'profession': 'user', 'uid': 'poulamipaul'}, '2': {'name': 'test', 'password': 'testing', 'profession': 'tester', 'uid': 'jarvistester'}}
I want to get a list of all the values of name.
What should be my code in python

d.values gives all the values, then you can get the attribute name of each value.
d = {'1': {'name': 'poulami', 'password': 'paul123', 'profession': 'user', 'uid': 'poulamipaul'}, '2': {'name': 'test', 'password': 'testing', 'profession': 'tester', 'uid': 'jarvistester'}}
[i['name'] for i in d.values()]
['poulami', 'test']
Also note that d.values returns a generator and not a list so to convert to list use list(d.values())

That is not JSON format. It is a Python Dictionary.
Iterate over the values of the dictionary(d.values()) and get the name from each item.
d = {'1': {'name': 'poulami', 'password': 'paul123', 'profession': 'user', 'uid': 'poulamipaul'}, '2': {'name': 'test', 'password': 'testing', 'profession': 'tester', 'uid': 'jarvistester'}}
names_list = []
for i in d.values():
names_list.append(i['name'])
names_list = ['poulami', 'test']

Python String to List of Dicts

I have reviewed a number of similar questions on stackoverflow, but been unable to locate an answer which applies to my data/string.
I have a string which is effectively a list of dictionaries. In the fields, numbers are not surrounded by double quotes. If I try to use ast to evaluate the string, part of the string is cut off and I am unsure why. Could someone help me determine an appropriate way to read in this string and create a list of dicts.
Thanks,
>>> print(ascii_data)
[{"measurement": "cpu_load_short","tags": {"host": "server999","region": "us-west-1"},"fields": {"value": 0.99}},{"measurement": "cpu_load_short","tags": {"host": "server888","region": "us-east-1"},"fields": {"value": 0.88}}]
>>> x = ast.literal_eval(ascii_data)
>>> print(x)
[{'fields': {'value': 0.99}, 'tags': {'host': 'server999', 'region': 'us-west-1'}, 'measurement': 'cpu_load_short'}, {'fields': {'value': 0.88}, 'tags': {'host': 'server888', 'region': 'us-east-1'}, 'measurement': 'cpu_load_short'}]

Given:
>>> s
'[{"measurement": "cpu_load_short","tags": {"host": "server999","region": "us-west-1"},"fields": {"value": 0.99}},{"measurement": "cpu_load_short","tags": {"host": "server888","region": "us-east-1"},"fields": {"value": 0.88}}]'
You can use json
>>> import json
>>> json.loads(s)
[{u'fields': {u'value': 0.99}, u'tags': {u'host': u'server999', u'region': u'us-west-1'}, u'measurement': u'cpu_load_short'}, {u'fields': {u'value': 0.88}, u'tags': {u'host': u'server888', u'region': u'us-east-1'}, u'measurement': u'cpu_load_short'}]
Or ast:
>>> import ast
>>> ast.literal_eval(s)
[{'fields': {'value': 0.99}, 'tags': {'host': 'server999', 'region': 'us-west-1'}, 'measurement': 'cpu_load_short'}, {'fields': {'value': 0.88}, 'tags': {'host': 'server888', 'region': 'us-east-1'}, 'measurement': 'cpu_load_short'}]
And they produce the same Python data structure (at least with ascii input...):
>>> json.loads(s)==ast.literal_eval(s)
True
Since in each case the result is a Python dict know that the order may be different than the string's order. Python dicts are unordered and will usually be different than the creation order (at least prior to Python 3.6).
Under Python 3.6, they resulting dict is in the same order:
>>> json.loads(s)
[{'measurement': 'cpu_load_short', 'tags': {'host': 'server999', 'region': 'us-west-1'}, 'fields': {'value': 0.99}}, {'measurement': 'cpu_load_short', 'tags': {'host': 'server888', 'region': 'us-east-1'}, 'fields': {'value': 0.88}}]
>>> ast.literal_eval(s)
[{'measurement': 'cpu_load_short', 'tags': {'host': 'server999', 'region': 'us-west-1'}, 'fields': {'value': 0.99}}, {'measurement': 'cpu_load_short', 'tags': {'host': 'server888', 'region': 'us-east-1'}, 'fields': {'value': 0.88}}]
Python 3.6 is great...

Use json.
In [1]: s = '''[{"measurement": "cpu_load_short","tags": {"host": "server999","region": "us-west-1"},"fields": {"value": 0.99}},{"measuremen
...: t": "cpu_load_short","tags": {"host": "server888","region": "us-east-1"},"fields": {"value": 0.88}}]'''
In [2]: import json
In [3]: import pprint
In [4]: pprint.pprint(json.loads(s))
[{'fields': {'value': 0.99},
'measurement': 'cpu_load_short',
'tags': {'host': 'server999', 'region': 'us-west-1'}},
{'fields': {'value': 0.88},
'measurement': 'cpu_load_short',
'tags': {'host': 'server888', 'region': 'us-east-1'}}]
In [11]: json.loads(s)[0]['tags']['host']
Out[11]: 'server999'

How about json.loads?
j = json.loads(ascii_data)
ast.literal_eval may not be the best option. If your data source comes from some API, it would definitely be json format.
And if the order of dict keys matters to you, try specifying the object_pairs_hook argument to JSONDecoder. (ref: Can I get JSON to load into an OrderedDict in Python?)

Accessing YAML data in Python

I have a YAML file that parses into an object, e.g.:
{'name': [{'proj_directory': '/directory/'},
{'categories': [{'quick': [{'directory': 'quick'},
{'description': None},
{'table_name': 'quick'}]},
{'intermediate': [{'directory': 'intermediate'},
{'description': None},
{'table_name': 'intermediate'}]},
{'research': [{'directory': 'research'},
{'description': None},
{'table_name': 'research'}]}]},
{'nomenclature': [{'extension': 'nc'}
{'handler': 'script'},
{'filename': [{'id': [{'type': 'VARCHAR'}]},
{'date': [{'type': 'DATE'}]},
{'v': [{'type': 'INT'}]}]},
{'data': [{'time': [{'variable_name': 'time'},
{'units': 'minutes since 1-1-1980 00:00 UTC'},
{'latitude': [{'variable_n...
I'm having trouble accessing the data in python and regularly see the error TypeError: list indices must be integers, not str
I want to be able to access all elements corresponding to 'name' so to retrieve each data field I imagine it would look something like:
import yaml
settings_stream = open('file.yaml', 'r')
settingsMap = yaml.safe_load(settings_stream)
yaml_stream = True
print 'loaded settings for: ',
for project in settingsMap:
print project + ', ' + settingsMap[project]['project_directory']
and I would expect each element would be accessible via something like ['name']['categories']['quick']['directory']
and something a little deeper would just be:
['name']['nomenclature']['data']['latitude']['variable_name']
or am I completely wrong here?

The brackets, [], indicate that you have lists of dicts, not just a dict.
For example, settingsMap['name'] is a list of dicts.
Therefore, you need to select the correct dict in the list using an integer index, before you can select the key in the dict.
So, giving your current data structure, you'd need to use:
settingsMap['name'][1]['categories'][0]['quick'][0]['directory']
Or, revise the underlying YAML data structure.
For example, if the data structure looked like this:
settingsMap = {
'name':
{'proj_directory': '/directory/',
'categories': {'quick': {'directory': 'quick',
'description': None,
'table_name': 'quick'}},
'intermediate': {'directory': 'intermediate',
'description': None,
'table_name': 'intermediate'},
'research': {'directory': 'research',
'description': None,
'table_name': 'research'},
'nomenclature': {'extension': 'nc',
'handler': 'script',
'filename': {'id': {'type': 'VARCHAR'},
'date': {'type': 'DATE'},
'v': {'type': 'INT'}},
'data': {'time': {'variable_name': 'time',
'units': 'minutes since 1-1-1980 00:00 UTC'}}}}}
then you could access the same value as above with
settingsMap['name']['categories']['quick']['directory']
# quick

Getting a python dict.keys trail for each item

I have a python dict that looks like this
{'data': [{'data': [{'data': 'gen1', 'name': 'objectID'},
{'data': 'familyX', 'name': 'family'}],
'name': 'An-instance-of-A'},
{'data': [{'data': 'gen2', 'name': 'objectID'},
{'data': 'familyY', 'name': 'family'},
{'data': [{'data': [{'data': '21',
'name': 'objectID'},
{'data': 'name-for-21',
'name': 'name'},
{'data': 'no-name', 'name': None}],
'name': 'An-instance-of-X:'},
{'data': [{'data': '22',
'name': 'objectID'}],
'name': 'An-instance-of-X:'}],
'name': 'List-of-2-X-elements:'}],
'name': 'An-instance-of-A'}],
'name': 'main'}
The structure is repeating and its rule is like:
A dict contains 'name' and 'data'
'data' can contain a list of dicts
If 'data' is not a list, it is a value I need.
'name' is a just a name
The problem is that for each value, I need to know every info for each parent.
So at the end, I need to print a list with items that looks something like:
objectID=gen2 family=familyY An-instance-of-X_objectID=21 An-instance-of-X_name=name-for-21
Edit: This is only one of several lines I want as the output. I need one line like this for each item that doesn’t have a dict as 'data'.
So, for each data that is not a dict, traverse up, find info and print it..
I don't know every function in modules like itertools and collections. But is there something in there I can use? What is this called (when I am trying to do research on my own)?
I can find many "flatten dict" methods, but not like this, not when I have 'data', 'name' like this..

This is a wonderful example what recursion is good for:
input_ = {'data': [{'data': [{'data': 'gen1', 'name': 'objectID'},
{'data': 'familyX', 'name': 'family'}],
'name': 'An-instance-of-A'},
{'data': [{'data': 'gen2', 'name': 'objectID'},
{'data': 'familyY', 'name': 'family'},
{'data': [{'data': [{'data': '21',
'name': 'objectID'},
{'data': 'name-for-21',
'name': 'name'},
{'data': 'no-name', 'name': None}],
'name': 'An-instance-of-X:'},
{'data': [{'data': '22',
'name': 'objectID'}],
'name': 'An-instance-of-X:'}],
'name': 'List-of-2-X-elements:'}],
'name': 'An-instance-of-A'}],
'name': 'main'}
def parse_dict(d, predecessors, output):
"""Recurse into dict and fill list of path-value-pairs"""
data = d["data"]
name = d["name"]
name = name.strip(":") if type(name) is str else name
if type(data) is list:
for d_ in data:
parse_dict(d_, predecessors + [name], output)
else:
output.append(("_".join(map(str,predecessors+[name])), data))
result = []
parse_dict(input_, [], result)
print "\n".join(map(lambda x: "%s=%s"%(x[0],x[1]),result))
Output:
main_An-instance-of-A_objectID=gen1
main_An-instance-of-A_family=familyX
main_An-instance-of-A_objectID=gen2
main_An-instance-of-A_family=familyY
main_An-instance-of-A_List-of-2-X-elements_An-instance-of-X_objectID=21
main_An-instance-of-A_List-of-2-X-elements_An-instance-of-X_name=name-for-21
main_An-instance-of-A_List-of-2-X-elements_An-instance-of-X_None=no-name
main_An-instance-of-A_List-of-2-X-elements_An-instance-of-X_objectID=22
I hope I understood your requirements correctly. If you don't want to join the paths into strings, you can keep the list of predecessors instead.
Greetings,
Thorsten