Develop Reference

Import JSON file with dicts of dicts of nested dicts into Pandas

I am downloading a json file from the RKI in Germany (CDC equivalent). It seems to have dictionaries inside dictionaries inside dictionaries. I'm really only interested in the dictionary of data nested in the "features" dictionary. My problem is that each entry in this dictionary is nested with the same key - "attributes". This is what it looks like as text (I have to work with the text because I can't download it directly into python due to proxy issues - grrr.).
{"objectIdFieldName":"ObjectId","uniqueIdField":
{"name":"ObjectId","isSystemMaintained":true},
"globalIdFieldName":"","fields":
[{"name":"AdmUnitId","type":"esriFieldTypeInteger","alias":"AdmUnitId","sqlType":"sqlTypeInteger","domain":null,"defaultValue":null},
...etc... {"name":"ObjectId","type":"esriFieldTypeOID","alias":"ObjectId","sqlType":"sqlTypeInteger","domain":null,"defaultValue":null}],
"features":
[{"attributes":{"AdmUnitId":0,"BundeslandId":0,"AnzFall":3741781,"AnzTodesfall":91337,"AnzFallNeu":1456,"AnzTodesfallNeu":18,"AnzFall7T":7178,"AnzGenesen":3638200,"AnzGenesenNeu":700,"AnzAktiv":12300,"AnzAktivNeu":700,"Inz7T":8.6,"ObjectId":1}},
{"attributes":{"AdmUnitId":1,"BundeslandId":1,"AnzFall":64221,"AnzTodesfall":1628,"AnzFallNeu":35,"AnzTodesfallNeu":1,"AnzFall7T":181,"AnzGenesen":62300,"AnzGenesenNeu":0,"AnzAktiv":300,"AnzAktivNeu":0,"Inz7T":6.2,"ObjectId":2}},
{"attributes":{"AdmUnitId":2,"BundeslandId":2,"AnzFall":77823,"AnzTodesfall":1603,"AnzFallNeu":50,"AnzTodesfallNeu":0,"AnzFall7T":217,"AnzGenesen":75700,"AnzGenesenNeu":0,"AnzAktiv":500,"AnzAktivNeu":0,"Inz7T":11.7,"ObjectId":3}},
...etc
When I try to pd.read_json(the_file), I get the Value Error: arrays must all be same length.
If I open as json and load, create a dictionary, I get my dictionary of dictionaries with the dictionary I want. I can almost get there, as below, but I end up with a list of nested dictionaries, where the key is always - "attributes" - which throws the error.
with open(r"Q:\AbisF\Covid-19\Lageberichte\Misc\RKI_7Tages.json") as json_data:
data = json.load(json_data)
# dig down to the data
features = data["features"]
attributes = features["attributes"] # TypeError: list indices must be integers or slices, not str
I am wondering if I am coming at this the wrong way, of if there is a way to clean up my list (get rid of the attributes level).

I think that your features = data["features"] is now a list of dicts.
You can iterate over those:
features = data["features"]
for feature in features:
attributes = feature["attributes"]
print(attributes['AdmUnitId']) # example item in attributes

Conv json obj w/ varying keys & key:value pairs to python list for specific keys:values

I have received a json file from an api and want to convert it from to a dictionary to a list in python. I already have it loaded as a python dictionary, so I'm really just looking to iterate over it and convert from a dictionary to a list. However, the dictionary has nested lists and not every object within the json file's many objects is always structured the same (i.e. the api will not always return the same json object for every event). I want to convert the dictionary to 3 separate lists, each list for a specific key:value pair I am parsing for.
I've tried using KeyError in a try/except statement to account for cases where there is no value but haven't had much luck. Also tried importing defaultdict from collections to no success as well. I gather I should be able to make the rests of my lists once I get some help with this first one. Hopefully I didn't overcomplicate the question
data = load_json()# function responsible for loading the json from a stored file
new_list = []
for some_key1 in data
if new_list['some_key1'] > 0:
try:
new_list.append(['some_key1'], some_key1)
except:
new_list.get(['some_key1'], '0')
for x in range(len(new_list)):
print(new_list[x])
I am looking to make one list for storing each object's (i.e. each python 'breaking' dictionary) key:value pair where some_key1 exists (i.e. this means in this case I will have a list for aDifferentCherry and yetAnotherDifferentCherry.

Extract a value from within a nested key python

I need to extract out a specific value from within a nested key in python
for example from the below I want to extract out the Start from the key Params
{'Key': 'Params', 'Value': `'{"Shut":false,"Remove":false,"SnapshotRequired":false,"Start":"Never","End":"Never"}'}
This is as far as I have got
for tag in i["Tags"]:
if 'Params' in tag['Key']:
then I can get the value but this is the whole string.

You can use dict.get(key) method to get a value of the key you specify in get(), where dict is variable which stores your dictionary.
Also you can use dict[key]. It gives the same result.
In your case, for example, dict['Key'] will return 'Params',
dict['Value'] will return nested dictionary.

Ok after fixing an issue in the string, an extra '`' you can simply make dict(string) to make it a valid python dict, which then accepts nested dicts as values.
also have in mine that "false" is not a valid python type, so you would have to make it a string or convert to python format which is False with the capital F
and finally do:
variable['Value']['Start']
test = dict({'Key': 'Params', 'Value': {"Shut":False,"Remove":False,"SnapshotRequired":False,"Start":"Never","End":"Never"}})
test['Value']['Start']

Clean way to manage parse-dictionaries that contain function names

Well to directly explain it, I wish to "parse" user input. - Simple input string -> output string.
However there isn't any "logical" way to parse this apart from simply checking the input string against a dictionary (regex check). Now this isn't difficult at all - I'd just create a dictionary with regex search string keys & regex/function pointer values.
However the problem is: there will be around ~100-200 "keys" probably. And I can easily see myself wishing to add/remove keys (maybe merge) in the future.
So is there a way that creating such a dictionary looks "structured"? Keeping the "data" away from the "code". (Data would be the regex-functionname pairs)?

Store the dictionary in the JSON format in file, with the function names as ordinary strings. Demo how to load a JSON file:
Content of sample file:
{"somestring":"myfunction"}
Code:
import json
d = json.load(open('very_small_dic.txt', 'r'))
print(d) # {'somestring': 'myfunction'}
How to get the string:function mapping:
First you load the dictionary from a file as illustrated in the code above. After that, you build a new dictionary where the strings of the function names are replaced by the actual functions. Demo:
def myfunction(x):
return 2*x
d = {'somestring': 'myfunction'} # in the real code this came from json.load
d = {k:globals()[v] for k,v in d.items()}
print(d) # {'somestring': <function myfunction at 0x7f36e69d8c20>}
print(d['somestring'](42)) # 84
You could also store your functions in a separate file myfunctions.py and use getattr. This is probably a cleaner way than using globals.
import myfunctions # for this demo, this module only contains the function myfunction
d = {'somestring': 'myfunction'} # in the real code this came from json.load
d = {k:getattr(myfunctions,v) for k,v in d.items()}
print(d) # {'somestring': <function myfunction at 0x7f36e69d8c20>}
print(d['somestring'](42)) # 84

You could also use JsonSchema (http://json-schema.org/example1.html).
I think in order to transform values belonging to certain keys you'd certainly have to write a function to perform the conversions.
If you'd just like to sanitize the input based on the presence of certain keys - It would be ideal to convert it into a dictionary, define a schema and then check for the (required/optional fields)/the type or validate a field against a list of enums.

Convert One Key Value from List of Dictionaries to List

I have a list of dictionaries that is encoded:
[u"{'name':'Tom', 'uid':'asdlfkj223'}", u"{'name':'Jerry', 'uid':'alksd32'}", ...]
Is there anyway I can create a list of just the values of the key name?
Even better if someone knows Django ORM well enough to pull down a list of a data/column with properties from a PostgreSQL database.
Thanks!

To get only that value for the name column from the DB table, use:
names = Person.objects.values_list('name', flat=True)
(as per https://docs.djangoproject.com/en/dev/ref/models/querysets/#django.db.models.query.QuerySet.values_list)
otherwise, given
people = [{'name':'Tom', 'uid':'asdlfkj223'}, {'name':'Jerry', 'uid':'alksd32'},]
this should do the job:
names = [person['name'] for person in people]
And you should find out why your data items are strings (containing a string representation of a dict) to start with—it doesn't look like the way it's supposed to be.
Or, if you're actually storing dict's in your database as strings, either prefer JSON over the Python string representation, or if you must use the current format, the AST parsing solution provided in another question here should do the job.

You can use ast.literal_eval:
>>> data = [u"{'name':'Tom', 'uid':'asdlfkj223'}",u"{'name':'Jerry', 'uid':'alksd32'}"]
>>> import ast
>>> [ast.literal_eval(d)['name'] for d in data]
['Tom', 'Jerry']