Related
I am using the data from the League of Legends API to learn Python, JSON, and Data Classes. Using dacite, I have created parent and child classes that allow access to the data using this syntax: champs.data['Ahri']['key']. However, I wonder if there is a way to create a class that returns the keys as fields so one could access the data using this syntax: champs.data.Ahri.key.
Here is the working code:
from dataclasses import dataclass
from dacite import from_dict
j1 = {'type': 'champion',
'data': {'Aatrox': {'id': 'Aatrox', 'key': '266', 'name': 'Aatrox'},
'Ahri': {'id': 'Ahri', 'key': '103', 'name': 'Ahri'}}}
#dataclass
class C:
type: str
data: dict
#dataclass
class P:
type: str
data: dict
champs = from_dict(data_class=P, data=j1)
champs.data['Ahri']['key']
If it were me, I would probably leave/make champions a dictionary. Then access it like champions['Ahri'].key
Something like:
import dataclasses
#dataclasses.dataclass
class Champion:
id: str
key: str
name: str
j1 = {
'type': 'champion',
'data': {
'Aatrox': {'id': 'Aatrox', 'key': '266', 'name': 'Aatrox'},
'Ahri': {'id': 'Ahri', 'key': '103', 'name': 'Ahri'}
}
}
champions = {
champion["id"]: Champion(**champion)
for champion in j1["data"].values()
}
print(champions['Ahri'].key)
resulting in 103
However if you were really keen on champions.Ahri.key then you can implement Champions as an empty class and use setattr()
import dataclasses
#dataclasses.dataclass
class Champion:
id: str
key: str
name: str
#dataclasses.dataclass
class Champions:
pass
j1 = {
'type': 'champion',
'data': {
'Aatrox': {'id': 'Aatrox', 'key': '266', 'name': 'Aatrox'},
'Ahri': {'id': 'Ahri', 'key': '103', 'name': 'Ahri'}
}
}
champions = Champions()
for champion in j1["data"].values():
setattr(champions, champion["id"], Champion(**champion))
print(champions.Ahri.key)
again giving you 103
Note: The #dataclass decorator can likely be omitted from Champion().
The closest you can probably get - at least in a safe enough manner - is as #JonSG suggests, using champs.data['Ahri'].key.
Here's a straightforward example using the dataclass-wizard. It doesn't do a strict type checking as I know dacite does.
Instead, it opts to do implicit type coercision where possible, which is useful in some cases; you can see an example of this below - str to annotated int in this case.
Note: This example should work for Python 3.7+ with the included __future__ import.
from __future__ import annotations
from dataclasses import dataclass
from dataclass_wizard import fromdict
data = {
'type': 'champion',
'data': {
'Aatrox': {'id': 'Aatrox', 'key': '266', 'name': 'Aatrox'},
'Ahri': {'id': 'Ahri', 'key': '103', 'name': 'Ahri'},
}
}
#dataclass
class P:
type: str
data: dict[str, Character]
#dataclass
class Character:
id: str
key: int
name: str
champs = fromdict(P, data)
print(champs)
print(champs.data['Ahri'].key)
Output:
P(type='champion', data={'Aatrox': Character(id='Aatrox', key=266, name='Aatrox'), 'Ahri': Character(id='Ahri', key=103, name='Ahri')})
103
How to do this
d = {
"type": "champion",
"data": {
"Aatrox": {"id": "Aatrox", "key": "266", "name": "Aatrox"},
"Ahri": {"id": "Ahri", "key": "103", "name": "Ahri"},
},
}
def dict_to_class(d) -> object:
if isinstance(d, dict):
class C:
pass
for k, v in d.items():
setattr(C, k, dict_to_class(v))
return C
else:
return d
champ = dict_to_class(d)
print(champ.data.Ahri.key)
# 103
The key here is the setatter builtin method, which takes an object, a string, and some value, and creates an attribute (field) on that object, named according to the string and containing the value.
Don't do this!
I must stress that there is almost never a good reason to do this. When dealing with JSON data of an unknown shape, the correct way to represent it is a dict.
If you do know the shape of the data, you should create a specialized dataclass, like so:
from dataclasses import dataclass
d = {
"type": "champion",
"data": {
"Aatrox": {"id": "Aatrox", "key": "266", "name": "Aatrox"},
"Ahri": {"id": "Ahri", "key": "103", "name": "Ahri"},
},
}
#dataclass
class Champion:
id: str
key: str
name: str
champions = {name: Champion(**attributes) for name, attributes in d["data"].items()}
print(champions)
# {'Aatrox': Champion(id='Aatrox', key='266', name='Aatrox'), 'Ahri': Champion(id='Ahri', key='103', name='Ahri')}
print(champions["Aatrox"].key)
# 266
The dacite docs have a section about nested structures that is very close to what you want. The example they use, verbatim, is as follows:
#dataclass
class A:
x: str
y: int
#dataclass
class B:
a: A
data = {
'a': {
'x': 'test',
'y': 1,
}
}
result = from_dict(data_class=B, data=data)
assert result == B(a=A(x='test', y=1))
We can access fields at arbitrary depth as e.g. result.a.x == 'test'.
The critical difference between this and your data is that the dictionary under the data key has keys with arbitrary values (Aatrox, Ahri, etc.). dacite isn't set up to create new field names on the fly, so the best you're going to get is something like the latter part of #JonSG's answer, which uses setattr to dynamically build new fields.
Let's imagine how you would use this data for a moment, though. Probably you'd want a some point to be able to iterate over your champions in order to perform a filter/transform/etc. operation. It's possible to iterate over fields in python, but you have to really dig into python internals, which means your code will be less readable/generally comprehensible.
Much better would be one of the following:
Preprocess j1 into a shape that fits the structure you want to use, and then use dacite with a dataclass that fits the new structure. For example, maybe it makes sense to pull the values of the data dict out into a list.
Process in steps using dacite. For example, something like the following:
from dataclasses import dataclass
from dacite import from_dict
#dataclass
class TopLevel:
type: str
data: dict
j1 = {
"type": "champion",
"data": {
"Aatrox": {"id": "Aatrox", "key": "266", "name": "Aatrox"},
"Ahri": {"id": "Ahri", "key": "103", "name": "Ahri"},
},
}
champions = from_dict(data_class=TopLevel, data=j1)
# champions.data is a dict of dicts
#dataclass
class Champion:
id: str
key: str
name: str
# transform champions.data into a dict of Champions
for k, v in champions.data.items():
champions.data[k] = from_dict(data_class=Champion, data=v)
# now, you can do interesting things like the following filter operation
start_with_a = [
champ for champ in champions.data.values() if champ.name.lower().startswith("a")
]
print(start_with_a)
# [Champion(id='Aatrox', key='266', name='Aatrox'), Champion(id='Ahri', key='103', name='Ahri')]
I am trying to validate JSON, the schema for which specifies a list of dicts with arbitrary string keys, the corresponding values of which are dicts with a strict schema (i.e, the keys of the inner dict are strictly some string, here 'a'). From the Cerberus docs, I think that what I want is the 'keysrules' rule. The example in the docs seems to only show how to use 'keysrules' to validate arbitrary keys, but not their values. I wrote the below code as an example; the best I could do was assume that 'keysrules' would support a 'schema' argument for defining a schema for these values.
keysrules = {
'myDict': {
'type': 'dict',
'keysrules': {
'type': 'string',
'schema': {
'type': 'dict',
'schema': {
'a': {'type': 'string'}
}
}
}
}
}
keysRulesTest = {
'myDict': {
'arbitraryStringKey': {
'a': 'arbitraryStringValue'
},
'anotherArbitraryStringKey': {
'shouldNotValidate': 'arbitraryStringValue'
}
}
}
def test_rules():
v = Validator(keysrules)
if not v.validate(keysRulesTest):
print(v.errors)
assert(0)
This example does validate, and I would like it to not validate on 'shouldNotValidate', because that key should be 'a'. Does the flexibility implied by 'keysrules' (i.e, keys governed by 'keysrules' have no constraint other than {'type': 'string'}) propagate down recursively to all schemas underneath it? Or have I made some different error? How can I achieve my desired outcome?
I didn't want keysrules, I wanted valuesrules:
keysrules = {
'myDict': {
'type': 'dict',
'valuesrules': {
'type': 'dict',
'schema': {
'a': {'type': 'string'}
}
}
}
}
keysRulesTest = {
'myDict': {
'arbitraryStringKey': {
'a': 'arbitraryStringValue'
},
'anotherArbitraryStringKey': {
'shouldNotValidate': 'arbitraryStringValue'
}
}
}
def test_rules():
v = Validator(keysrules)
if not v.validate(keysRulesTest):
print(v.errors)
assert(0)
This produces my desired outcome.
I'm trying to de-serialize a JSON string into an object in Python, while indicating the specific class type for each object.
Here is an example JSON:
{
"Vehicle":
[
{
"$type": "Car",
"Make": 1982,
"Settings":
{
"$type": "CarSettings",
"ESP": true
}
},
{
"$type": "Motorcycle",
"Make": 2010,
"Settings":
{
"$type": "MotorcycleSettings",
"ABS": true
}
}
]
}
Note that my classes are built this way:
Vehicle
Car : Vehicle
Motorcycle : Vehicle
and:
Settings
CarSettings : Settings
MotorcycleSettings : Settings
I figured it out for a single type representing the entire JSON using "json.loads" with "object_hook" but I can't seem to make it work for sub-classes.
Thanks!
The object_hook you define will get the dictionaries from the JSON string. Take this example:
def hook(dic):
print(dic)
return dic
Using it as a hook, with your example will produce:
{'$type': 'CarSettings', 'ESP': True}
{'$type': 'Car', 'Make': 1982, 'Settings': {'$type': 'CarSettings', 'ESP': True}}
{'$type': 'MotorcycleSettings', 'ABS': True}
{'$type': 'Motorcycle', 'Make': 2010, 'Settings': {'$type': 'MotorcycleSettings', 'ABS': True}}
Therefore in the hook() function you take the $type element and act according to that.
def hook(dic):
if dic['$type'] == "Car":
return Car(**dic) # assuming Car constructor can take kwargs
if dic['$type'] == "Motorcycle":
return Motorcycle(**dic) # assuming Motorcycle constructor can take kwargs
The hook() will be called on the most nested dictionary first. So if you parse the CarSettings and return that object from the hook() than the dictionary for the Car ('$type': 'Car') will already contain the CarSettings type object.
{'$type': 'Car', 'Make': 1982, 'Settings': <__main__.CarSettings object at 0x000001FF4FD6A550>}
If you have many classes, make a dictionary ordering a type for a typename, and use that for object creation.
types = {
'Car' : Car,
'Motorcycle' : Motorcycle,
}
def hook(dic):
try:
return types[dic['$type']](**dic)
except KeyError:
pass
I have a dictionary with config info:
my_conf = {
'version': 1,
'info': {
'conf_one': 2.5,
'conf_two': 'foo',
'conf_three': False,
'optional_conf': 'bar'
}
}
I want to check if the dictionary follows the structure I need.
I'm looking for something like this:
conf_structure = {
'version': int,
'info': {
'conf_one': float,
'conf_two': str,
'conf_three': bool
}
}
is_ok = check_structure(conf_structure, my_conf)
Is there any solution done to this problem or any library that could make implementing check_structure more easy?
You may use schema (PyPi Link)
schema is a library for validating Python data structures, such as those obtained from config-files, forms, external services or command-line parsing, converted from JSON/YAML (or something else) to Python data-types.
from schema import Schema, And, Use, Optional, SchemaError
def check(conf_schema, conf):
try:
conf_schema.validate(conf)
return True
except SchemaError:
return False
conf_schema = Schema({
'version': And(Use(int)),
'info': {
'conf_one': And(Use(float)),
'conf_two': And(Use(str)),
'conf_three': And(Use(bool)),
Optional('optional_conf'): And(Use(str))
}
})
conf = {
'version': 1,
'info': {
'conf_one': 2.5,
'conf_two': 'foo',
'conf_three': False,
'optional_conf': 'bar'
}
}
print(check(conf_schema, conf))
Without using libraries, you could also define a simple recursive function like this:
def check_structure(struct, conf):
if isinstance(struct, dict) and isinstance(conf, dict):
# struct is a dict of types or other dicts
return all(k in conf and check_structure(struct[k], conf[k]) for k in struct)
if isinstance(struct, list) and isinstance(conf, list):
# struct is list in the form [type or dict]
return all(check_structure(struct[0], c) for c in conf)
elif isinstance(conf, type):
# struct is the type of conf
return isinstance(struct, conf)
else:
# struct is neither a dict, nor list, not type
return False
This assumes that the config can have keys that are not in your structure, as in your example.
Update: New version also supports lists, e.g. like 'foo': [{'bar': int}]
Advice for the future: use Pydantic!
Pydantic enforces type hints at runtime, and provides user friendly errors when data is invalid. Define how data should be in pure, canonical python; validate it with pydantic, as simple as that:
from pydantic import BaseModel
class Info(BaseModel):
conf_one: float
conf_two: str
conf_three: bool
class Config:
extra = 'forbid'
class ConfStructure(BaseModel):
version: int
info: Info
If validation fails pydantic will raise an error with a breakdown of what was wrong:
my_conf_wrong = {
'version': 1,
'info': {
'conf_one': 2.5,
'conf_two': 'foo',
'conf_three': False,
'optional_conf': 'bar'
}
}
my_conf_right = {
'version': 10,
'info': {
'conf_one': 14.5,
'conf_two': 'something',
'conf_three': False
}
}
model = ConfStructure(**my_conf_right)
print(model.dict())
# {'version': 10, 'info': {'conf_one': 14.5, 'conf_two': 'something', 'conf_three': False}}
res = ConfStructure(**my_conf_wrong)
# pydantic.error_wrappers.ValidationError: 1 validation error for ConfStructure
# info -> optional_conf
# extra fields not permitted (type=value_error.extra)
You can build structure using recursion:
def get_type(value):
if isinstance(value, dict):
return {key: get_type(value[key]) for key in value}
else:
return str(type(value))
And then compare required structure with your dictionary:
get_type(current_conf) == get_type(required_conf)
Example:
required_conf = {
'version': 1,
'info': {
'conf_one': 2.5,
'conf_two': 'foo',
'conf_three': False,
'optional_conf': 'bar'
}
}
get_type(required_conf)
{'info': {'conf_two': "<type 'str'>", 'conf_one': "<type 'float'>", 'optional_conf': "<type 'str'>", 'conf_three': "<type 'bool'>"}, 'version': "<type 'int'>"}
Looks like the dict-schema-validator package does exactly what you need:
Here is a simple schema representing a Customer:
{
"_id": "ObjectId",
"created": "date",
"is_active": "bool",
"fullname": "string",
"age": ["int", "null"],
"contact": {
"phone": "string",
"email": "string"
},
"cards": [{
"type": "string",
"expires": "date"
}]
}
Validation:
from datetime import datetime
import json
from dict_schema_validator import validator
with open('models/customer.json', 'r') as j:
schema = json.loads(j.read())
customer = {
"_id": 123,
"created": datetime.now(),
"is_active": True,
"fullname": "Jorge York",
"age": 32,
"contact": {
"phone": "559-940-1435",
"email": "york#example.com",
"skype": "j.york123"
},
"cards": [
{"type": "visa", "expires": "12/2029"},
{"type": "visa"},
]
}
errors = validator.validate(schema, customer)
for err in errors:
print(err['msg'])
Output:
[*] "_id" has wrong type. Expected: "ObjectId", found: "int"
[+] Extra field: "contact.skype" having type: "str"
[*] "cards[0].expires" has wrong type. Expected: "date", found: "str"
[-] Missing field: "cards[1].expires"
You can also use dataclasses_json library. Here is how I would normally do it
from dataclasses import dataclass
from dataclasses_json import dataclass_json, Undefined
from dataclasses_json.undefined import UndefinedParameterError
from typing import Optional
#### define schema #######
#dataclass_json(undefined=Undefined.RAISE)
#dataclass
class Info:
conf_one: float
# conf_two: str
conf_three: bool
optional_conf: Optional[str]
#dataclass_json
#dataclass
class ConfStructure:
version: int
info: Info
####### test for compliance####
try:
ConfStructure.from_dict(my_conf).to_dict()
except KeyError as e:
print('theres a missing parameter')
except UndefinedParameterError as e:
print('extra parameters')
You can use dictify from https://pypi.org/project/dictify/.
Read docs here https://dictify.readthedocs.io/en/latest/index.html
This is how it can be done.
from dictify import Field, Model
class Info(Model):
conf_one = Field(required=True).instance(float)
conf_two = Field(required=True).instance(str)
conf_three = Field(required=True).instance(bool)
optional_conf = Field().instance(str)
class MyConf(Model):
version = Field(required=True).instance(int)
info = Field().model(Info)
my_conf = MyConf() # Invalid without required fields
# Valid
my_conf = MyConf({
'version': 1,
'info': {
'conf_one': 2.5,
'conf_two': 'foo',
'conf_three': False,
'optional_conf': 'bar'
}
})
my_conf['info']['conf_one'] = 'hi' # Invalid, won't be assinged
There is a standard for validating JSON files called JSON Schema.
Validators have been implemented in many languages, including the Python. Read also the documentation for more details. In the following example I will use a Python package jsonschema (docs) that I am familiar with.
Given the config data
my_conf = {
'version': 1,
'info': {
'conf_one': 2.5,
'conf_two': 'foo',
'conf_three': False,
'optional_conf': 'bar',
},
}
and the corresponding config schema
conf_structure = {
'type': 'object',
'properties': {
'version': {'type': 'integer'},
'info': {
'type': 'object',
'properties': {
'conf_one': {'type': 'number'},
'conf_two': {'type': 'string'},
'conf_three': {'type': 'boolean'},
'optional_conf': {'type': 'string'},
},
'required': ['conf_one', 'conf_two', 'conf_three'],
},
},
}
the actual code to validate this data is then as simple as this:
import jsonschema
jsonschema.validate(my_conf, schema=conf_structure)
A big advantage of this approach is that you can store both data and schema as JSON-formatted files.
#tobias_k beat me to it (both in time and quality probably) but here is another recursive function for the task that might be a bit easier for you (and me) to follow:
def check_dict(my_dict, check_against):
for k, v in check_against.items():
if isinstance(v, dict):
return check_dict(my_dict[k], v)
else:
if not isinstance(my_dict[k], v):
return False
return True
The nature of dictionaries, if they are being used in python and not exported as some JSON, is that the order of the dictionary need not be set. Instead, looking up keys returns values (hence a dictionary).
In either case, these functions should provide you with what your looking for for the level of nesting present in the samples you provided.
#assuming identical order of keys is required
def check_structure(conf_structure,my_conf):
if my_conf.keys() != conf_structure.keys():
return False
for key in my_conf.keys():
if type(my_conf[key]) == dict:
if my_conf[key].keys() != conf_structure[key].keys():
return False
return True
#assuming identical order of keys is not required
def check_structure(conf_structure,my_conf):
if sorted(my_conf.keys()) != sorted(conf_structure.keys()):
return False
for key in my_conf.keys():
if type(my_conf[key]) != dict:
return False
else:
if sorted(my_conf[key].keys()) != sorted(conf_structure[key].keys()):
return False
return True
This solution would obviously need to be changed if the level of nesting was greater (i.e. it is configured to assess the similarity in structure of dictionaries that have some values as dictionaries, but not dictionaries where some values these latter dictionaries are also dictionaries).
So say i have the following document:
test_obj = {
'my_things':{
'id17': {
'blah': 3,
'weird': 'yay',
'thechallenge': ObjectId('5712d06fdb4d0856551300d2')
},
'id32': {
'blah': 62,
'weird': 'hoorah',
'thechallenge': ObjectId('5712d06fdb4d0856551300d4')
}
},
'_id': 12,
'an_extra_field': 'asdf'
}
for this document i have the following schema:
API.config['DOMAIN']['test_obj']['schema'] = {
'id': {'type': 'int'},
'an_extra_field': {'type': 'string'},
'my_things': {
'type': 'dict',
'valueschema': {
'type': 'dict',
'schema': {
'blah': {'type': 'dict'},
'weird': {'type': 'string'},
'thechallenge': {'type': 'objectid'}
}
}
}
}
Now say i make a patch with the following pseudocode:
data = {
'mythings': {
'id17': {
'thechallenge': '5712d06fdb4d0856551300d8'
}
}
}
PATCH(url='/v1/test_objs/12', data=data)
When I make this patch Cerberus raises an error during validation, saying "value '5712d06fdb4d0856551300d8' cannot be converted to a ObjectId". Now this is a valid object id, and i find that if I make a patch to other non-valueschema fields it does not raise this error. It seems like valueschema was not meant to have a value of dict, and adding an extra 'schema' attribute was the only way i could get around cerberus raising a schemaerror/having cerberus actually validate my fields. But eve does not appear to actually be serializing my fields in my dictionary correctly. It should be of type ObjectId when it gets passed to Cerberus.
The way i'm temporarily getting around this is by manipulating my the code in Eve. In common.py (module) in serialize (function) in line 398 i added, where it checks if the field schema is a 'valueschema':
elif field_type == 'dict' and 'schema' in field_schema['valueschema']:
for subdocument in document[field].values():
serialize(subdocument, schema=field_schema['valueschema']['schema'])
Should i not be using type dict for the valueschema? If not how else should i handle this scenario? I would like to not have to maintain my own fork of Eve, so if others do want the ability to have valueschema be of type dict should i submit a pull-request for this change?
This has been fixed with Eve v0.6.4, which has just been released.