Develop Reference

pickle and dill can't load objects with overridden __hash__ function (AttributeError)

In the next few lines of code I'll replicate on a smaller scale what's happening with my program.
Class A must store a dictionary with keys that have type A (values can be any type to replicate the error).
class A:
def __init__(self, name):
self.name = name
self.dic = dict() # it'll be a mapping from A objects to <?> objects
def __repr__(self): return self.name
def __hash__(self): return hash(self.name)
The same is needed with class B. Besides, class B is a more complex object that takes a while to build, and thus I need to store it locally and load it when I need it.
class B:
def __init__(self, dic):
self.dic = dic # it'll be a mapping from A objects to <?> objects
def __repr__(self): return str(self.dic)
# saving the model with pickle
def save(self, filename):
with open("objects/" + filename + ".fan", "wb+") as filehandler:
pickle.dump(self, filehandler)
# loading the model with pickle
#staticmethod
def load(filename):
with open("objects/" + filename + ".fan", "rb") as filehandler:
return pickle.load(filehandler)
Let's instantiate some objects:
# instantiate two A objects
obj1 = A("name")
obj2 = A("name2")
# fill their dic field
obj1.dic[obj2] = 0
obj2.dic[obj1] = 1
# create a dictionary object with type(key) = A
# and instantiate a B object with that
dic = {obj1: (0, 0), obj2: (1, 4)}
obj3 = B(dic)
Now if I try to dump and load B with pickle/dill:
obj3.save("try") # all goes well
B.load("try") # nothing goes well
I get the following error:
Traceback (most recent call last):
File "C:\Users\SimoneZannini\Documents\Fantacalcio\try.py", line 40, in <module>
B.load("try")
File "C:\Users\SimoneZannini\Documents\Fantacalcio\try.py", line 29, in load
return pickle.load(filehandler)
File "C:\Users\SimoneZannini\Documents\Fantacalcio\try.py", line 11, in __hash__
def __hash__(self): return hash(self.name)
AttributeError: 'A' object has no attribute 'name'
Process finished with exit code 1
I know there's a similar problem that was solved, but this isn't exactly my case and the __getstate__ and __setstate__ workaround doesn't seem to work. I think this is due to A class having a dict object inside of it, but it's just an assumption.
Thanks in advance for your time.

Two things:
1
I'm not sure exactly why the error occurs but you can avoid it by declaring name as a class member variable like so
class A:
name = ""
def __init__(self, name):
self.name = name
self.dic = dict() # it'll be a mapping from A objects to <?> objects
def __repr__(self): return self.name
def __hash__(self): return hash(self.name)
2
Objects that keep a reference to other objects of the same class are often (though not always) an indicator of sub-optimal design. Why keep a dict inside each A when you could simply keep a dict (or dicts) outside the class?
To address the comments
Now I get KeyError with the dic field of A, something that doesn't happen without dumping and loading the object B
Consider the following
class C:
def __hash__(self):
return 1
c1 = C()
c2 = C()
mydict = {c1:1}
print(mydict[c1]) # 1
print(mydict[c2]) # key error
When you un-pickle a B, its self.dic now contains new As (not the original ones) so when you try to use the old As as keys in the new Bs dic, it doesn't work. Again, you could work around this but I think re-designing your app will be easier in the long run. You will need to override __eq__() in A for it to work:
class D:
def __hash__(self):
return 1
def __eq__(self, other):
return True
d1 = D()
d2 = D()
mydict = {d1:1}
print(mydict[d1]) # 1
print(mydict[d2]) # 1

Python make dictionary items accessible as object property

This is mostly syntactic sugar but I'd like to access the items of a dictionary as object properties.
Example:
class CoolThing():
def __init__(self):
self.CoolDict = {'a': 1, 'b': 2}
and I'd like to have
my_cool_thing.a # => 1
my_cool_thing.b # => 2
Edit: some code of a potential solution with a nested structure with dot notation: device.property.field
class Parameters():
def __init__(self, ids, devices):
self._ids = ids
self._devices = devices
for p in self._devices:
p = p[0]
if self.__dict__.get(p.device) is None:
self.__dict__[p.device] = SmartDict()
else:
if self.__dict__[p.device].get(p.property) is None:
self.__dict__[p.device][p.property] = SmartDict()
else:
if self.__dict__[p.device][p.property].get(p.field) is None:
self.__dict__[p.device][p.property][p.field] = ParameterData(p)
class SmartDict():
def __init__(self):
self.__dict__ = {}
def __getitem__(self, k):
return self.__dict__[k]
def __setitem__(self, k, v):
self.__dict__[k] = v
def get(self, k):
return self.__dict__.get(k)
def __len__(self):
return len(self.__dict__)

You want __getattr__ and __setattr__, though you'll have to roll your own class (I'm not aware of any builtins, though namedtuple might work if you don't need to change values much)
class AttrDict(dict):
def __getattr__(self, attr):
return self[attr]
def __setattr__(self, attr, value):
self[attr] = value
If you just want to access a sub-dictionary that way, you just change self to self.cool_dict
class CoolThing:
def __init__(self):
self.cool_dict = {'a': 1, 'b': 2}
def __getattr__(self, attr):
return self.cool_dict[attr]
def __setattr__(self, attr, value):
# Note, you'll have to do this for anything that you want to set
# in __init__.
if attr == 'cool_dict':
super().__setattr__(attr, value)
else:
self.cool_dict[attr] = value
Note that __getattr__ is used after any other lookups fail, but if you want to ensure that your function is called first, you can use __getattribute__
Also note that self.cool_dict does not exist on CoolThing until after __init__ is called. My initial version of this would throw a maximum recursion depth exceeded, because as you created the class it would go to set self.cool_dict in init, call __setattr__, which would try to get self.cool_dict so it could set [attr] = value on it. Naturally it can't find cool_dict yet, and so it will try to call __getattr__ again... which can't find cool_dict and round and round it goes.
Another option would be to use a class-level variable instead, but that's probably not at all what you want :)

CoolDict already exists, it's named __dict__:
>>> class CoolThing(object):
... def __init__(self):
... self.__dict__['a'] = 1
... self.__dict__['b'] = 2
...
>>> thing = CoolThing()
>>> thing.a
1
>>> thing.b
2
>>> thing.c = 3
>>> thing.__dict__
{'a': 1, 'b': 2, 'c': 3}

Recursive DotDict

I have a utility class that makes Python dictionaries behave somewhat like JavaScript objects as far as getting and setting attributes.
class DotDict(dict):
"""
a dictionary that supports dot notation
as well as dictionary access notation
usage: d = DotDict() or d = DotDict({'val1':'first'})
set attributes: d.val2 = 'second' or d['val2'] = 'second'
get attributes: d.val2 or d['val2']
"""
__getattr__ = dict.__getitem__
__setattr__ = dict.__setitem__
__delattr__ = dict.__delitem__
I would like to make it so it also converts nested dictionaries into DotDict() instances. I was hoping to be able to do something like this with __init__ or __new__, but I haven't come up with anything that works:
def __init__(self, dct):
for key in dct.keys():
if hasattr(dct[key], 'keys'):
dct[key] = DotDict(dct[key])
How can I recursively convert the nested dictionaries into DotDict() instances?
>>> dct = {'scalar_value':1, 'nested_dict':{'value':2}}
>>> dct = DotDict(dct)
>>> print dct
{'scalar_value': 1, 'nested_dict': {'value': 2}}
>>> print type(dct)
<class '__main__.DotDict'>
>>> print type(dct['nested_dict'])
<type 'dict'>

I don't see where you are copying the values in the constructor. Here DotDict is always empty because of that. When I added the key assignment, it worked:
class DotDict(dict):
"""
a dictionary that supports dot notation
as well as dictionary access notation
usage: d = DotDict() or d = DotDict({'val1':'first'})
set attributes: d.val2 = 'second' or d['val2'] = 'second'
get attributes: d.val2 or d['val2']
"""
__getattr__ = dict.__getitem__
__setattr__ = dict.__setitem__
__delattr__ = dict.__delitem__
def __init__(self, dct):
for key, value in dct.items():
if hasattr(value, 'keys'):
value = DotDict(value)
self[key] = value
dct = {'scalar_value':1, 'nested_dict':{'value':2, 'nested_nested': {'x': 21}}}
dct = DotDict(dct)
print dct.nested_dict.nested_nested.x
It looks a bit dangerous and error prone, not to mention source of countless surprises to other developers, but seems to be working.

Shamelessly plugging my own package
There is a package doing exactly what you want and also something more and it is called Prodict.
from prodict import Prodict
life_dict = {'bigBang':
{'stars':
{'planets': []}
}
}
life = Prodict.from_dict(life_dict)
print(life.bigBang.stars.planets)
# prints []
# you can even add new properties dynamically
life.bigBang.galaxies = []
PS 1: I'm the author of the Prodict.
PS 2: This is a direct copy paste of an answer of another question.

I've been slightly unhappy with all the different answers I have found to this problem. My goals in my implementation were:
1) Don't create more new object attributes than necessary.
2) Don't allow overwriting access to built-in attributes.
3) The class converts added items to maintain consistency.
class attrdict(dict):
"""
Attribute Dictionary.
Enables getting/setting/deleting dictionary keys via attributes.
Getting/deleting a non-existent key via attribute raises `AttributeError`.
Objects are passed to `__convert` before `dict.__setitem__` is called.
This class rebinds `__setattr__` to call `dict.__setitem__`. Attributes
will not be set on the object, but will be added as keys to the dictionary.
This prevents overwriting access to built-in attributes. Since we defined
`__getattr__` but left `__getattribute__` alone, built-in attributes will
be returned before `__getattr__` is called. Be careful::
>>> a = attrdict()
>>> a['key'] = 'value'
>>> a.key
'value'
>>> a['keys'] = 'oops'
>>> a.keys
<built-in method keys of attrdict object at 0xabcdef123456>
Use `'key' in a`, not `hasattr(a, 'key')`, as a consequence of the above.
"""
def __init__(self, *args, **kwargs):
# We trust the dict to init itself better than we can.
dict.__init__(self, *args, **kwargs)
# Because of that, we do duplicate work, but it's worth it.
for k, v in self.iteritems():
self.__setitem__(k, v)
def __getattr__(self, k):
try:
return dict.__getitem__(self, k)
except KeyError:
# Maintain consistent syntactical behaviour.
raise AttributeError(
"'attrdict' object has no attribute '" + str(k) + "'"
)
def __setitem__(self, k, v):
dict.__setitem__(self, k, attrdict.__convert(v))
__setattr__ = __setitem__
def __delattr__(self, k):
try:
dict.__delitem__(self, k)
except KeyError:
raise AttributeError(
"'attrdict' object has no attribute '" + str(k) + "'"
)
#staticmethod
def __convert(o):
"""
Recursively convert `dict` objects in `dict`, `list`, `set`, and
`tuple` objects to `attrdict` objects.
"""
if isinstance(o, dict):
o = attrdict(o)
elif isinstance(o, list):
o = list(attrdict.__convert(v) for v in o)
elif isinstance(o, set):
o = set(attrdict.__convert(v) for v in o)
elif isinstance(o, tuple):
o = tuple(attrdict.__convert(v) for v in o)
return o

The accepted answer has some gotchas, such as failing on hasattr(). Using the keys to simulate properties means you need to do a tad more than assign __getattr__ = dict.__getitem__. Here's a more robust implementation with tests:
from collections import OrderedDict, Mapping
class DotDict(OrderedDict):
'''
Quick and dirty implementation of a dot-able dict, which allows access and
assignment via object properties rather than dict indexing.
'''
def __init__(self, *args, **kwargs):
# we could just call super(DotDict, self).__init__(*args, **kwargs)
# but that won't get us nested dotdict objects
od = OrderedDict(*args, **kwargs)
for key, val in od.items():
if isinstance(val, Mapping):
value = DotDict(val)
else:
value = val
self[key] = value
def __delattr__(self, name):
try:
del self[name]
except KeyError as ex:
raise AttributeError(f"No attribute called: {name}") from ex
def __getattr__(self, k):
try:
return self[k]
except KeyError as ex:
raise AttributeError(f"No attribute called: {k}") from ex
__setattr__ = OrderedDict.__setitem__
And the tests:
class DotDictTest(unittest.TestCase):
def test_add(self):
exp = DotDict()
# test that it's not there
self.assertFalse(hasattr(exp, 'abc'))
with self.assertRaises(AttributeError):
_ = exp.abc
with self.assertRaises(KeyError):
_ = exp['abc']
# assign and test that it is there
exp.abc = 123
self.assertTrue(hasattr(exp, 'abc'))
self.assertTrue('abc' in exp)
self.assertEqual(exp.abc, 123)
def test_delete_attribute(self):
exp = DotDict()
# not there
self.assertFalse(hasattr(exp, 'abc'))
with self.assertRaises(AttributeError):
_ = exp.abc
# set value
exp.abc = 123
self.assertTrue(hasattr(exp, 'abc'))
self.assertTrue('abc' in exp)
self.assertEqual(exp.abc, 123)
# delete attribute
delattr(exp, 'abc')
# not there
self.assertFalse(hasattr(exp, 'abc'))
with self.assertRaises(AttributeError):
delattr(exp, 'abc')
def test_delete_key(self):
exp = DotDict()
# not there
self.assertFalse('abc' in exp)
with self.assertRaises(KeyError):
_ = exp['abc']
# set value
exp['abc'] = 123
self.assertTrue(hasattr(exp, 'abc'))
self.assertTrue('abc' in exp)
self.assertEqual(exp.abc, 123)
# delete key
del exp['abc']
# not there
with self.assertRaises(KeyError):
del exp['abc']
def test_change_value(self):
exp = DotDict()
exp.abc = 123
self.assertEqual(exp.abc, 123)
self.assertEqual(exp.abc, exp['abc'])
# change attribute
exp.abc = 456
self.assertEqual(exp.abc, 456)
self.assertEqual(exp.abc, exp['abc'])
# change key
exp['abc'] = 789
self.assertEqual(exp.abc, 789)
self.assertEqual(exp.abc, exp['abc'])
def test_DotDict_dict_init(self):
exp = DotDict({'abc': 123, 'xyz': 456})
self.assertEqual(exp.abc, 123)
self.assertEqual(exp.xyz, 456)
def test_DotDict_named_arg_init(self):
exp = DotDict(abc=123, xyz=456)
self.assertEqual(exp.abc, 123)
self.assertEqual(exp.xyz, 456)
def test_DotDict_datatypes(self):
exp = DotDict({'intval': 1, 'listval': [1, 2, 3], 'dictval': {'a': 1}})
self.assertEqual(exp.intval, 1)
self.assertEqual(exp.listval, [1, 2, 3])
self.assertEqual(exp.listval[0], 1)
self.assertEqual(exp.dictval, {'a': 1})
self.assertEqual(exp.dictval['a'], 1)
self.assertEqual(exp.dictval.a, 1) # nested dotdict works
And just for fun, you can turn an object into a DotDict with this:
def to_dotdict(obj):
''' Converts an object to a DotDict '''
if isinstance(obj, DotDict):
return obj
elif isinstance(obj, Mapping):
return DotDict(obj)
else:
result = DotDict()
for name in dir(obj):
value = getattr(obj, name)
if not name.startswith('__') and not inspect.ismethod(value):
result[name] = value
return result

Is it possible to overload a call to vars(object) in Python?

I have a class that overloads object attribute access by returning the attributes of its "row" attribute, along the following lines:
from collections import namedtuple
class MyObj(object):
def __init__(self, y, z):
r = namedtuple('row', 'a b')
self.row = r(y, z)
self.arbitrary = True
def __getattr__(self, attr):
return getattr(self.row, attr)
def __dir__(self):
return list(self.row._fields)
In [2]: m = MyObj(1, 2)
In [3]: dir(m)
Out[3]: ['a', 'b']
In [4]: m.a
Out[4]: 1
In [5]: vars(m)
Out[5]: {'arbitrary': True, 'row': row(a=1, b=2)}
In [6]: output = '{a} -> {b}'
In [7]: output.format(**vars(m.row))
Out[7]: '1 -> 2'
In [8]: output.format(**vars(m))
KeyError: 'a'
As I quite often do string formatting using vars() I'd like to be able to access row's attributes directly from the call to vars(). Is this possible?
Edit following aaronsterling's answer
The key to solving this, thanks to Aaron's pointer, is to check for __dict__ in __getattribute__
from collections import namedtuple
class MyObj(object):
def __init__(self, y, z):
r = namedtuple('row', 'a b')
self.row = r(y, z)
self.arbitrary = True
def __getattr__(self, attr):
return getattr(self.row, attr)
def __getattribute__(self, attribute):
if attribute == '__dict__':
return self.row._as_dict()
else:
return object.__getattribute__(self, attribute)
def __dir__(self):
return list(self.row._fields)
In [75]: m = MyObj(3, 4)
In [76]: m.a
Out[76]: 3
In [77]: vars(m)
Out[77]: OrderedDict([('a', 3), ('b', 4)])

You can also define a #property for __dict__ instead of overriding __getattribute__.
class Example:
#property
def __dict__(self):
return "From vars()"
e = Example()
print(vars(e)) # From vars()

The docs are so kind as to specify that vars works by returning the __dict__ attribute of the object it's called on. Hence overriding __getattribute__ does the trick. Y
I subclass dict (but see later) to override the __str__ function. The subclass accepts a function str_func which gets called to return a string representation of how you want your __dict__ object to appear. It does this by constructing a regular dictionary with the entries that you want and then calling str on that.
This is very hacky. In particular, it will break any code that depends on doing anything like
myobj.__dict__[foo] = bar
This code will now update a phantom dictionary and not the real one.
A much more robust solution would depend on completely replacing all methods that set values on the SpoofedDict with methods that actually update myobj.__dict__. This would require SpoofedDict instances to hold a reference to myobj.__dict__. Then of course, the methods that read values would have to fetch them out of myobj.__dict__ as well.
At that point, you're better off using collections.Mapping to construct a custom class rather than subclassing from dict.
Here's the proof of concept code, hackish as it may be:
from collections import namedtuple
class SpoofDict(dict):
def __init__(self, *args, **kwargs):
self.str_func = kwargs['str_func']
del kwargs['str_func']
dict.__init__(self, *args, **kwargs)
def __str__(self):
return self.str_func()
class MyObj(object):
def __init__(self, y, z):
r = namedtuple('row', 'a b')
self.row = r(y, z)
self.arbitrary = True
def __getattr__(self, attr):
return getattr(self.row, attr)
def __dir__(self):
return list(self.row._fields)
def str_func(self):
attrs = list(self.row._fields)
str_dict = {}
row = object.__getattribute__(self, 'row')
for attr in attrs:
str_dict[attr] = getattr(row, attr)
return str(str_dict)
def __getattribute__(self, attribute):
if attribute == '__dict__':
spoof_dict = SpoofDict(str_func=self.str_func)
spoof_dict.update(object.__getattribute__(self, '__dict__'))
return spoof_dict
else:
return object.__getattribute__(self, attribute)
if __name__=='__main__':
m = MyObj(1, 2)
print "dir(m) = {0}".format(dir(m))
print "vars(m) = {0}".format(vars(m))
print "m.row = {0}".format(m.row)
print "m.arbitrary = {0}".format(m.arbitrary)

Python - Access a class from a list using a key

Is there any way to make a list of classes behave like a set in python?
Basically, I'm working on a piece of software that does some involved string comparison, and I have a custom class for handling the strings. Therefore, there is an instance of the class for each string.
As a result, I have a large list containing all these classes. I would like to be able to access them like list[key], where in this case, the key is a string the class is based off of (note: the string will never change once the class is instantiated, so it should be hashable).
It seems to me that I should be able to do this somewhat easily, by adding something like __cmp__ to the class, but either I'm being obtuse (likely), or I'm missing something in the docs.
Basically, I want to be able to do something like this (Python prompt example):
>>class a:
... def __init__(self, x):
... self.var = x
...
>>> from test import a
>>> cl = set([a("Hello"), a("World"), a("Pie")])
>>> print cl
set([<test.a instance at 0x00C866C0>, <test.a instance at 0x00C866E8>, <test.a instance at 0x00C86710>])
>>> cl["World"]
<test.a instance at 0x00C866E8>
Thanks!
Edit Some additional Tweaks:
class a:
... def __init__(self, x):
... self.var = x
... def __hash__(self):
... return hash(self.var)
...
>>> v = a("Hello")
>>> x = {}
>>> x[v]=v
>>> x["Hello"]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'Hello'
>>> x["Hello"]

Just write a class that behaves a bit like a mapping:
class ClassDict(object):
def __init__(self):
self.classes = {}
def add(self, cls):
self.classes[cls.__name__] = cls
def remove(self, cls):
if self.classes[cls.__name__] == cls:
del self.classes[cls.__name__]
else:
raise KeyError('%r' % cls)
def __getitem__(self, key):
return self.classes[key]
def __repr__(self):
return 'ClassDict(%s)' % (', '.join(self.classes),)
class C(object):
pass
class D(object):
pass
cd = ClassDict()
cd.add(C)
cd.add(D)
print cd
print cd['C']

Why don't you just do:
>>> v = MyStr("Hello")
>>> x = {}
>>> x[v.val]=v
>>> x["Hello"]
MyStr("Hello")
Why go through all the trouble of trying to create a hand-rolled dict that uses different keys than the ones you pass in? (i.e. "Hello" instead of MyStr("Hello")).
ex.
class MyStr(object):
def __init__(self, val):
self.val = str(val)
def __hash__(self):
return hash(self.val)
def __str__(self):
return self.val
def __repr__(self):
return 'MyStr("%s")' % self.val
>>> v = MyStr("Hello")
>>> x = {}
>>> x[str(v)]=v
>>> x["Hello"]
MyStr("Hello")

Set and dict use the value returned by an object's __hash__ method to look up the object, so this will do what you want:
>>class a:
... def __init__(self, x):
... self.var = x
...
... def __hash__(self):
... return hash(self.var)

As I remember "set" and "dict" uses also __hash__
From Python 2.x doc:
A dictionary’s keys are almost arbitrary values. Values that are not hashable, that is, values containing lists, dictionaries or other mutable types (that are compared by value rather than by object identity) may not be used as keys.

Do you want something like this
class A(object):
ALL_INSTANCES = {}
def __init__(self, text):
self.text = text
self.ALL_INSTANCES[self.text] = self
a1 = A("hello")
a2 = A("world")
print A.ALL_INSTANCES["hello"]
output:
<__main__.A object at 0x00B7EA50>