Develop Reference

How to get my class (which inherets dict) to allow double lookups? (typeerror)

I am trying (for fun) to implement a kind of defaultdict that always has the same structure. It is a nested defaultdict if n > 1 and a defaultdict (Lm) with defaultvalue 0 otherwise.
class Lm(dict):
def __init__(self, n=1):
if n == 1:
self.default = 0
else:
self.default = Lm(1)
dict.__init__(self)
def __missing__(self, key):
return dict.__setitem__(self, key, self.default)
def __getitem__(self, key):
try:
return dict.__getitem__(self, key)
except KeyError:
return self.__missing__(key)
Since it is sometimes nested, I would like to be able to access entries like so (desired):
test = Lm(2)
test['i']['am']
print(test)
{'i': {'am': 0}}
However this does not work, I get the following error:
test = Lm(2)
test['i']['am']
TypeError: 'NoneType' object is not subscriptable
I have found that if I lookup from Lm once before with the ['i'] it works as intended:
test = Lm(2)
test['i']
print(test)
{'i': {}}
test['i']['am']
print(test)
{'i': {'am': 0}}
Why does doing a lookup twice in the same line not work here?
Edit:
I figured it out, here is my updated code with the suggestions from below (including the bit about the defaultfactory).
class Lm(dict):
def __init__(self, n=1):
if n == 1:
self.default = lambda val: 0
else:
self.default = lambda val: Lm(1)
def __missing__(self, key):
self.__setitem__(key, self.default(key))
return self.default(key)
def __getitem__(self, key):
try:
val = super().__getitem__(key)
self.__setitem__(key, val)
return val
except KeyError:
self.__missing__(self, key, self.default(key))

Python property setting does not assign the right instance attributes (leading underscores) when assigning via __setattr__

I can't set the right properties of an instance when setting their attributes via setattr in a factory method.
Given the following code where data is a simple dict containing e.g. { "age": "64", ...}
def factory(data):
obj = MyClass()
for k, v in data.items():
setattr(obj, k, v)
return obj
class MyClass(object):
def __init__(self):
self._age = None
# more...
#property
def age(self):
return self._age
#age.setter
def age(self, value):
some_validation(value)
self._age = value
def __setattr__(self, name, value):
object.__setattr__(self, name, value)
def __getitem__(self, item):
return self.__dict__.get(item, None)
def __getattr__(self, item):
self.__dict__[item] = None
return None
def __str__(self):
return json.dumps(self, default=lambda o: o.__dict__)
c = factory(data)
print(c)
I always get the following output when printing the created object:
{"_age": "64", ...}
But I need to have
{"age": "64", ...}
Why does the setattr method assign the leading underscore?

Some of the things you are trying to achieve get mixed up, like wanting to print __dict__ for a readable representation, but using private attributes for properties. Let's start from scratch and see how we can implement your class correctly.
You are trying to implement a class which attributes can be accessed both as keys and attributes. That is fine and can be accomplished in a more concise way.
class MyClass:
...
def __getitem__(self, item):
return self.__getattribute__(item)
def __setitem__(self, key, value):
return self.__setattr__(key, value)
You also want None to be returned when an attribute does not exist. This is covered by __getattr__ which is called exactly when an attribute does not exist.
def __getattr__(self, _):
return None
Then you want to add some validation to some attributes with property. It is indeed the correct way to proceed.
#property
def age(self):
return self._age
#age.setter
def age(self, value):
# some validation here
self._age = value
And finally you want to be able to have a nice string representation of your instance. We have to be careful for that since we had to add some private attributes that we do not want to print.
What we are going to do is implement a method keys to allow casting to dict. This method will only return keys for attributes which are not private nor methods.
def keys(self):
return [k for k in dir(self) if not k.startswith('_') and not callable(self[k])]
def __str__(self):
return json.dumps(dict(self))
This does the right thing.
obj = MyClass()
obj.age = 3
print(obj)
# prints: {"age": 3}

Python: from a dict, how retrieve object as key

If I have a dictionary of several Object:value,, How can I retrieve certain Object using it as [key]?
For example
class Obj():
def __init__(self, value):
self.value = value
dct = {Obj(foo):foo_value, Obj(bar):bar_value}
#How to do something like
#>>> dct[foo]
#foo_value
Suppose that foo_value can't be aasigned as property of Obj.
So far, this is what I get (abstracted)
class Obj():
def __init__(self, name):
self.name = name
def __hash__(self):
return hash(tuple(sorted(self.__dict__.items())))
def __eq__(self, other):
if isinstance(other, self.__class__):
return self.__dict__ == other.__dict__
else:
return False
def __repr__(self):
return str(self.name)
dct = {Obj('item1'):1, Obj('item2'):2}
print(dct.keys())
dct['item1']
And the output
dict_keys([item1, item2])
Traceback (most recent call last):
File "C:\Users\ivan\Desktop\multi_e.py", line 197, in <module>
dct['item1']
KeyError: 'item1'

What about using a custom implementation of dict?
class FieldDict(dict):
def __getitem__(self, item):
return dict.__getitem__(self, Obj(item))
dct = FieldDict({Obj('item1'):1, Obj('item2'):2})
print(dct.keys())
print(dct['item1']) # prints 1

That will not work as the keys are not the strings, but they are objects of type Obj. Even though all the objects hold that string
You could do this. Store variables as references to objects as keys
x = Obj("item1")
y = Obj("item2")
dct= {x:1, y:2}
And to retrieve you need to do:
>>> dct[x]
1

You could roll-out your own dict subclass with custom __getitem__ and __setitem__ and you really don't need any complicated __hash__ method on your Obj class after this, just __init__ will suffice.
class Obj:
def __init__(self, value):
self.name = value
class MyDict(dict):
def __setitem__(self, key, value):
if isinstance(key, Obj):
dict.__setitem__(self, key.name, value)
else:
dict.__setitem__(self, key, value)
def __getitem__(self, key):
if isinstance(key, Obj):
return dict.__getitem__(self, key.name)
return dict.__getitem__(self, key)
Demo:
>>> dct = MyDict()
>>> dct[Obj('item1')] = 1
>>> dct[Obj('item2')] = 2
>>> dct
{'item1': 1, 'item2': 2}
>>> dct['item1']
1
>>> dct[Obj('item1')]
1

I was a trying for a little bit time, but I think I got what you want, look:
edit thanks to #user2357112
class Obj():
def __init__(self, value):
self.value = value
def __eq__(self, other):
"""Override the default Equals behavior"""
if isinstance(other, self.__class__):
return self.value == other.value
return False
def __ne__(self, other):
"""Define a non-equality test"""
return not self.__eq__(other)
def __hash__(self):
return id(self.value)
class Custom_dict(dict):
def __getitem__(self, item):
return dict.__getitem__(self, Obj(item))
x = Custom_dict()
x[Obj('asdf')] = 5
print(x['asdf'])
dct = Custom_dict({Obj('item1'):1, Obj('item2'):2})
print([key.value for key in dct.keys()])
print(dct['item1'])
5
['item1', 'item2']
1

Is there a better way to create a class which is a subclass of `dict` but case-insensitive?

I want to create a class inheriting from dict type but could use case-insensitive key to visit the data. I implement a simple one but I don't think using instance.__dict__ variable is a proper approach.
Is there a better way to do this?
Here is my code:
class MyDict(dict):
def __init__(self, *args, **kwargs):
if args:
for k, v in args[0].iteritems():
self.__dict__.update({k.lower(): v})
def __getitem__(self, k):
return self.__dict__.get(k.lower())
def __setitem__(self, k, v):
self.__dict__.update({k.lower(): v})
def __delitem__(self, k):
self.__dict__.pop(k, None)
if __name__ == '__main__':
test_0 = MyDict({'naME': 'python', 'Age': 24})
print(test_0['name']) # return 'python'
print(test_0['AGE']) # return 24
test_1 = MyDict()
test_1['StaCk'] = 23
print(test_1['stack']) # return 23
print(test_1['STACK']) # return 23

Edit: See Janne Karila's link, that contains a better solution.
Instead of using self.__dict__, which has a special meaning unrelated to this being a dict, you should use super() to call the corresponding function on the superclass.
E.g.,
def __setitem__(self, k, v):
if hasattr(k, 'lower'):
k = k.lower()
return super(MyDict, self).__setitem__(k, v)

SQLAlchemy JSON as blob/text

I'm storing JSON down as blob/text in a column using MySQL. Is there a simple way to convert this into a dict using python/SQLAlchemy?

You can very easily create your own type with SQLAlchemy
For SQLAlchemy versions >= 0.7, check out Yogesh's answer below
import jsonpickle
import sqlalchemy.types as types
class JsonType(types.MutableType, types.TypeDecorator):
impl = types.Unicode
def process_bind_param(self, value, engine):
return unicode(jsonpickle.encode(value))
def process_result_value(self, value, engine):
if value:
return jsonpickle.decode(value)
else:
# default can also be a list
return {}
This can be used when you are defining your tables (example uses elixir):
from elixir import *
class MyTable(Entity):
using_options(tablename='my_table')
foo = Field(String, primary_key=True)
content = Field(JsonType())
active = Field(Boolean, default=True)
You can also use a different json serialiser to jsonpickle.

sqlalchemy.types.MutableType has been deprecated (v0.7 onward), the documentation recommends using sqlalchemy.ext.mutable instead.
I found a Git gist by dbarnett that I have tested for my usage. It has worked well so far, for both dictionary and lists.
Pasting below for posterity:
import simplejson
import sqlalchemy
from sqlalchemy import String
from sqlalchemy.ext.mutable import Mutable
class JSONEncodedObj(sqlalchemy.types.TypeDecorator):
"""Represents an immutable structure as a json-encoded string."""
impl = String
def process_bind_param(self, value, dialect):
if value is not None:
value = simplejson.dumps(value)
return value
def process_result_value(self, value, dialect):
if value is not None:
value = simplejson.loads(value)
return value
class MutationObj(Mutable):
#classmethod
def coerce(cls, key, value):
if isinstance(value, dict) and not isinstance(value, MutationDict):
return MutationDict.coerce(key, value)
if isinstance(value, list) and not isinstance(value, MutationList):
return MutationList.coerce(key, value)
return value
#classmethod
def _listen_on_attribute(cls, attribute, coerce, parent_cls):
key = attribute.key
if parent_cls is not attribute.class_:
return
# rely on "propagate" here
parent_cls = attribute.class_
def load(state, *args):
val = state.dict.get(key, None)
if coerce:
val = cls.coerce(key, val)
state.dict[key] = val
if isinstance(val, cls):
val._parents[state.obj()] = key
def set(target, value, oldvalue, initiator):
if not isinstance(value, cls):
value = cls.coerce(key, value)
if isinstance(value, cls):
value._parents[target.obj()] = key
if isinstance(oldvalue, cls):
oldvalue._parents.pop(target.obj(), None)
return value
def pickle(state, state_dict):
val = state.dict.get(key, None)
if isinstance(val, cls):
if 'ext.mutable.values' not in state_dict:
state_dict['ext.mutable.values'] = []
state_dict['ext.mutable.values'].append(val)
def unpickle(state, state_dict):
if 'ext.mutable.values' in state_dict:
for val in state_dict['ext.mutable.values']:
val._parents[state.obj()] = key
sqlalchemy.event.listen(parent_cls, 'load', load, raw=True, propagate=True)
sqlalchemy.event.listen(parent_cls, 'refresh', load, raw=True, propagate=True)
sqlalchemy.event.listen(attribute, 'set', set, raw=True, retval=True, propagate=True)
sqlalchemy.event.listen(parent_cls, 'pickle', pickle, raw=True, propagate=True)
sqlalchemy.event.listen(parent_cls, 'unpickle', unpickle, raw=True, propagate=True)
class MutationDict(MutationObj, dict):
#classmethod
def coerce(cls, key, value):
"""Convert plain dictionary to MutationDict"""
self = MutationDict((k,MutationObj.coerce(key,v)) for (k,v) in value.items())
self._key = key
return self
def __setitem__(self, key, value):
dict.__setitem__(self, key, MutationObj.coerce(self._key, value))
self.changed()
def __delitem__(self, key):
dict.__delitem__(self, key)
self.changed()
class MutationList(MutationObj, list):
#classmethod
def coerce(cls, key, value):
"""Convert plain list to MutationList"""
self = MutationList((MutationObj.coerce(key, v) for v in value))
self._key = key
return self
def __setitem__(self, idx, value):
list.__setitem__(self, idx, MutationObj.coerce(self._key, value))
self.changed()
def __setslice__(self, start, stop, values):
list.__setslice__(self, start, stop, (MutationObj.coerce(self._key, v) for v in values))
self.changed()
def __delitem__(self, idx):
list.__delitem__(self, idx)
self.changed()
def __delslice__(self, start, stop):
list.__delslice__(self, start, stop)
self.changed()
def append(self, value):
list.append(self, MutationObj.coerce(self._key, value))
self.changed()
def insert(self, idx, value):
list.insert(self, idx, MutationObj.coerce(self._key, value))
self.changed()
def extend(self, values):
list.extend(self, (MutationObj.coerce(self._key, v) for v in values))
self.changed()
def pop(self, *args, **kw):
value = list.pop(self, *args, **kw)
self.changed()
return value
def remove(self, value):
list.remove(self, value)
self.changed()
def JSONAlchemy(sqltype):
"""A type to encode/decode JSON on the fly
sqltype is the string type for the underlying DB column.
You can use it like:
Column(JSONAlchemy(Text(600)))
"""
class _JSONEncodedObj(JSONEncodedObj):
impl = sqltype
return MutationObj.as_mutable(_JSONEncodedObj)

I think the JSON example from the SQLAlchemy docs is also worth mentioning:
https://docs.sqlalchemy.org/en/13/core/custom_types.html#marshal-json-strings
However, I think it can be improved to be less strict regarding NULL and empty strings:
class JSONEncodedDict(TypeDecorator):
impl = VARCHAR
def process_bind_param(self, value, dialect):
if value is None:
return None
return json.dumps(value, use_decimal=True)
def process_result_value(self, value, dialect):
if not value:
return None
return json.loads(value, use_decimal=True)

There is a recipe for this in the official documentation:
from sqlalchemy.types import TypeDecorator, VARCHAR
import json
class JSONEncodedDict(TypeDecorator):
"""Represents an immutable structure as a json-encoded string.
Usage::
JSONEncodedDict(255)
"""
impl = VARCHAR
def process_bind_param(self, value, dialect):
if value is not None:
value = json.dumps(value)
return value
def process_result_value(self, value, dialect):
if value is not None:
value = json.loads(value)
return value

How about json.loads()?
>>> d= {"foo":1, "bar":[2,3]}
>>> s='{"foo":1, "bar":[2,3]}'
>>> import json
>>> json.loads(s) == d
True

Based on #snapshoe answer and to answer #Timmy's comment:
You can do it by using properties. Here is an example of a table:
class Providers(Base):
__tablename__ = "providers"
id = Column(
Integer,
Sequence('providers_id', optional=True),
primary_key=True
)
name = Column(Unicode(40), index=True)
_config = Column("config", Unicode(2048))
#property
def config(self):
if not self._config:
return {}
return json.loads(self._config)
#config.setter
def config(self, value):
self._config = json.dumps(value)
def set_config(self, field, value):
config = self.config
config[field] = value
self.config = config
def get_config(self):
if not self._config:
return {}
return json.loads(self._config)
def unset_config(self, field):
config = self.get_config()
if field in config:
del config[field]
self.config = config
Now you can use it on a Providers() object:
>>> p = Providers()
>>> p.set_config("foo", "bar")
>>> p.get_config()
{"foo": "bar"}
>>> a.config
{u'foo': u'bar'}
I know this is an old Question maybe even dead, but I hope this could help someone.

This is what I came up with based on the two answers above.
import json
class JsonType(types.TypeDecorator):
impl = types.Unicode
def process_bind_param(self, value, dialect):
if value :
return unicode(json.dumps(value))
else:
return {}
def process_result_value(self, value, dialect):
if value:
return json.loads(value)
else:
return {}

As an update to the previous responses, which we've used with success so far. As of MySQL 5.7 and SQLAlchemy 1.1 you can use the native MySQL JSON data type, which gives you better performance and a whole range of operators for free.
It lets you to create virtual secondary indexes on JSON elements too.
But of course you will lock yourself into running your app on MySQL only when moving the logic into the database itself.