I want to make a data object:
class GameData:
def __init__(self, data={}):
self.data = data
def __getitem__(self, item):
return self.data[item]
def __setitem__(self, key, value):
self.data[key] = value
def __getattr__(self, item):
return self.data[item]
def __setattr__(self, key, value):
self.data[kay] = value
def __repr__(self):
return str(self.data)
When I create a GameData object, I get RecursionError. How can I avoid setitem recall itself?
In the assignment self.data = data, __setattr__ is called because self has no attribute called data at the moment. __setattr__ then calls __getattr__ to obtain the non-existing attribute data. __getattr__ itself calls __getattr__ again. This is a recursion.
Use object.__setattr__(self, 'data', data) to do the assignment when implementing __setattr__.
class GameData:
def __init__(self, data=None):
object.__setattr__(self, 'data', {} if data is None else data)
def __getitem__(self, item):
return self.data[item]
def __setitem__(self, key, value):
self.data[key] = value
def __getattr__(self, item):
return self.data[item]
def __setattr__(self, key, value):
self.data[key] = value
def __repr__(self):
return str(self.data)
For details, see the __getattr__ manual
Additionally, do not use mutable objects as default parameter because the same object {} in the default argument is shared between GameData instances.
SQLAlchemy offers the PickleType and offers mutation tracking for any type that is mutable (like a dict).
The SQLAlchemy documentation mentions that this is the way to implement a mutable PickleType but it does not state exactly how to proceed with it.
Note: I want to store a dict in the PickleType.
How do you implement this?
While the documentation mentions some examples, it is not sufficient in my eyes, so I will add my implementation here that can be used to implement a mutable dict that is pickled and stored in the database.
Use the MutableDict example from the docs:
class MutableDict(Mutable, dict):
#classmethod
def coerce(cls, key, value):
if not isinstance(value, MutableDict):
if isinstance(value, dict):
return MutableDict(value)
return Mutable.coerce(key, value)
else:
return value
def __delitem(self, key):
dict.__delitem__(self, key)
self.changed()
def __setitem__(self, key, value):
dict.__setitem__(self, key, value)
self.changed()
def __getstate__(self):
return dict(self)
def __setstate__(self, state):
self.update(self)
Now create a column to be tracked:
class MyModel(Base):
data = Column(MutableDict.as_mutable(PickleType))
I would like to see some other examples that are maybe more advanced or possibly use different data structures. What would a generic approach for pickle look like? Is there one (I suppose not, or SQLAlchemy would have one).
Here's a solution I came up with. It wraps any type and detects any attribute sets and calls Mutable.changed(). It also wraps function calls and detects changes by taking a snapshot of the object before and after and comparing. Should work for Pickleable types...
from sqlalchemy.ext.mutable import Mutable
class MutableTypeWrapper(Mutable):
top_attributes = ['_underlying_object',
'_underlying_type',
'_last_state',
'_snapshot_update',
'_snapshot_changed',
'_notify_if_changed',
'changed',
'__getstate__',
'__setstate__',
'coerce']
#classmethod
def coerce(cls, key, value):
if not isinstance(value, MutableTypeWrapper):
try:
return MutableTypeWrapper(value)
except:
return Mutable.coerce(key, value)
else:
return value
def __getstate__(self):
return self._underlying_object
def __setstate__(self, state):
self._underlying_type = type(state)
self._underlying_object = state
def __init__(self, underlying_object, underlying_type=None):
if (underlying_object is None and underlying_type is None):
print('Both underlying object and type are none.')
raise RuntimeError('Unable to create MutableTypeWrapper with no underlying object or type.')
if (underlying_object is not None):
self._underlying_object = underlying_object
else:
self._underlying_object = underlying_type()
if (underlying_type is not None):
self._underlying_type = underlying_type
else:
self._underlying_type = type(underlying_object)
def __getattr__(self, attr):
if (attr in MutableTypeWrapper.top_attributes):
return object.__getattribute__(self, attr)
orig_attr = self._underlying_object.__getattribute__(attr)
if callable(orig_attr):
def hooked(*args, **kwargs):
self._snapshot_update()
result = orig_attr(*args, **kwargs)
self._notify_if_changed()
# prevent underlying from becoming unwrapped
if result == self._underlying_object:
return self
return result
return hooked
else:
return orig_attr
def __setattr__(self, attr, value):
if (attr in MutableTypeWrapper.top_attributes):
object.__setattr__(self, attr, value)
return
self._underlying_object.__setattr__(attr, value)
self.changed()
def _snapshot_update(self):
self._last_state = pickle.dumps(self._underlying_object,
pickle.HIGHEST_PROTOCOL)
def _snapshot_changed(self):
return self._last_state != pickle.dumps(self._underlying_object,
pickle.HIGHEST_PROTOCOL)
def _notify_if_changed(self):
if (self._snapshot_changed()):
self.changed()
And then use it with PickleType as follows:
class TestModel(Base):
__tablename__ = 'testtable'
id = Column(Integer, primary_key=True)
obj = Column(MutableTypeWrapper.as_mutable(PickleType))
The disadvantage here is the underlying class is snapshotted before every function call, and then changes are compared after in order to verify if the underlying object has changed. This will have a significant performance impact.
The other way to ensure that your PickleType objects are updated when you modify them is to copy and assign them before committing changes.
Ok, so I am trying to write code that let's one extend classes to have "properties" or metadata key value pairs.
For example in my app I want to use it for
obj.color = 'test"
But it gives me a recursion error on
File "/home/ekarlso/bookie-frontend/bookie/models/base.py", line 96, in __getattr__
if is_instrumented(self, key):
File "/home/ekarlso/venv/lib/python2.6/site-packages/sqlalchemy/orm/instrumentation.py", line 460, in is_instrumented
return manager_of_class(instance.__class__).\
RuntimeError: maximum recursion depth exceeded
I tried overriding __getattr__ as per
https://bitbucket.org/zzzeek/sqlalchemy-2208/src/f9fac8a6cc88/custom_management.py
But it doesn't work, anyone got some clues on fixing up the implementation?
from sqlalchemy.orm import class_mapper, object_mapper, scoped_session, sessionmaker
from sqlalchemy.orm.attributes import set_attribute, get_attribute, del_attribute
from sqlalchemy.orm.instrumentation import is_instrumented
class MetadataMixin(object):
"""
Mixin to extend a class with attributes / info in a related table
"""
__meta_attr__ = "metadata"
def meta_set(self, meta, cls=None):
"""
Create or update metadata for an Entity
:param meta: Metadata to be set
"""
meta_cls = object_mapper(self).get_property(self.meta_attr(cls)).\
mapper.class_
meta_current = self.meta_dict(cls=cls)
for key, value in meta.items():
values = {"entity_id": self.id, "key": key, "value": value}
if key in meta_current:
meta_ref = meta_current[key]
else:
meta_ref = meta_cls()
meta_ref.from_dict(values)
meta_ref.save()
return self
def meta_get(self, key, cls=None):
"""
Get metadata based on key
:param key: The key to get
"""
meta_ref = self.meta_dict(cls=cls).get(key, None)
return meta_ref.value if meta_ref else meta_ref
def meta_delete(self, key):
meta = self.meta_get(key).delete()
def meta_dict(self, cls=None):
"""
Return metadata as a dict
"""
meta_attr = getattr(self, self.meta_attr(cls=cls))
return dict([(m.key, m) for m in meta_attr])
def meta_attr(self, cls=None):
"""
This returns the attribute which other methods act upon
"""
return cls or self.__meta_attr__
def __setattr__(self, key, value):
if is_instrumented(self, key):
set_attribute(self, key, value)
else:
self.set_meta({name: value})
def __getattr__(self, key):
if is_instrumented(self, key):
return get_attribute(self, key)
else:
return self.meta_by_key(key)
def __delattr__(self, key):
if is_instrumented(self, key):
del_attribute(self, key)
else:
self.meta_delete(self, name)
In the example enumeration code given in this question, reproduced below, why does TOKEN contain the implementations of __contains__ and __repr__ from the metaclass EnumerationType?
from ctypes import *
class EnumerationType(type(c_uint)):
def __new__(metacls, name, bases, dict):
if not "_members_" in dict:
_members_ = {}
for key,value in dict.items():
if not key.startswith("_"):
_members_[key] = value
dict["_members_"] = _members_
cls = type(c_uint).__new__(metacls, name, bases, dict)
for key,value in cls._members_.items():
globals()[key] = value
return cls
def __contains__(self, value):
return value in self._members_.values()
def __repr__(self):
return "<Enumeration %s>" % self.__name__
class Enumeration(c_uint):
__metaclass__ = EnumerationType
_members_ = {}
def __init__(self, value):
for k,v in self._members_.items():
if v == value:
self.name = k
break
else:
raise ValueError("No enumeration member with value %r" % value)
c_uint.__init__(self, value)
#classmethod
def from_param(cls, param):
if isinstance(param, Enumeration):
if param.__class__ != cls:
raise ValueError("Cannot mix enumeration members")
else:
return param
else:
return cls(param)
def __repr__(self):
return "<member %s=%d of %r>" % (self.name, self.value, self.__class__)
class TOKEN(Enumeration):
_members_ = {'T_UNDEF':0, 'T_NAME':1, 'T_NUMBER':2, 'T_STRING':3, 'T_OPERATOR':4, 'T_VARIABLE':5, 'T_FUNCTION':6}
I would expect to have exceptions thrown by the following code to the effect that __contains__ is not implemented, instead however, I receive True False.
print 2 in TOKEN, 7 in TOKEN
Both Enumeration and TOKEN are instances of EnumerationType:
>>> isinstance(Enumeration, EnumerationType)
True
>>> isinstance(TOKEN, EnumerationType)
True
And special methods for instances of new style classes are looked up in class, e.g. repr(TOKEN) is equivalent to type(TOKEN).__repr__(TOKEN), which is EnumerationType.__repr__(TOKEN).
I'd like my dictionary to be case insensitive.
I have this example code:
text = "practice changing the color"
words = {'color': 'colour',
'practice': 'practise'}
def replace(words,text):
keys = words.keys()
for i in keys:
text= text.replace(i ,words[i])
return text
text = replace(words,text)
print text
Output = practise changing the colour
I'd like another string, "practice changing the Color", (where Color starts with a capital) to also give the same output.
I believe there is a general way to convert to lowercase using
mydictionary[key.lower()] but I'm not sure how to best integrate this into my existing code. (If this would be a reasonable, simple approach anyway).
The currently accepted answer wouldn't work for lots of cases, so it cannot be used as a drop-in dict replacement. Some tricky points in getting a proper dict replacement:
overloading all of the methods that involve keys
properly handling non-string keys
properly handling the constructor of the class
The following should work much better:
class CaseInsensitiveDict(dict):
#classmethod
def _k(cls, key):
return key.lower() if isinstance(key, basestring) else key
def __init__(self, *args, **kwargs):
super(CaseInsensitiveDict, self).__init__(*args, **kwargs)
self._convert_keys()
def __getitem__(self, key):
return super(CaseInsensitiveDict, self).__getitem__(self.__class__._k(key))
def __setitem__(self, key, value):
super(CaseInsensitiveDict, self).__setitem__(self.__class__._k(key), value)
def __delitem__(self, key):
return super(CaseInsensitiveDict, self).__delitem__(self.__class__._k(key))
def __contains__(self, key):
return super(CaseInsensitiveDict, self).__contains__(self.__class__._k(key))
def has_key(self, key):
return super(CaseInsensitiveDict, self).has_key(self.__class__._k(key))
def pop(self, key, *args, **kwargs):
return super(CaseInsensitiveDict, self).pop(self.__class__._k(key), *args, **kwargs)
def get(self, key, *args, **kwargs):
return super(CaseInsensitiveDict, self).get(self.__class__._k(key), *args, **kwargs)
def setdefault(self, key, *args, **kwargs):
return super(CaseInsensitiveDict, self).setdefault(self.__class__._k(key), *args, **kwargs)
def update(self, E={}, **F):
super(CaseInsensitiveDict, self).update(self.__class__(E))
super(CaseInsensitiveDict, self).update(self.__class__(**F))
def _convert_keys(self):
for k in list(self.keys()):
v = super(CaseInsensitiveDict, self).pop(k)
self.__setitem__(k, v)
Just for the record. I found an awesome impementation on Requests:
https://github.com/kennethreitz/requests/blob/v1.2.3/requests/structures.py#L37
If I understand you correctly and you want a way to key dictionaries in a non case-sensitive fashion, one way would be to subclass dict and overload the setter / getter:
class CaseInsensitiveDict(dict):
def __setitem__(self, key, value):
super(CaseInsensitiveDict, self).__setitem__(key.lower(), value)
def __getitem__(self, key):
return super(CaseInsensitiveDict, self).__getitem__(key.lower())
In my particular instance, I needed a case insensitive lookup, however, I did not want to modify the original case of the key. For example:
>>> d = {}
>>> d['MyConfig'] = 'value'
>>> d['myconfig'] = 'new_value'
>>> d
{'MyConfig': 'new_value'}
You can see that the dictionary still has the original key, however it is accessible case-insensitively. Here's a simple solution:
class CaseInsensitiveKey(object):
def __init__(self, key):
self.key = key
def __hash__(self):
return hash(self.key.lower())
def __eq__(self, other):
return self.key.lower() == other.key.lower()
def __str__(self):
return self.key
The __hash__ and __eq__ overrides are required for both getting and setting entries in the dictionary. This is creating keys that hash to the same position in the dictionary if they are case-insensitively equal.
Now either create a custom dictionary that initializes a CaseInsensitiveKey using the provided key:
class CaseInsensitiveDict(dict):
def __setitem__(self, key, value):
key = CaseInsensitiveKey(key)
super(CaseInsensitiveDict, self).__setitem__(key, value)
def __getitem__(self, key):
key = CaseInsensitiveKey(key)
return super(CaseInsensitiveDict, self).__getitem__(key)
or simply make sure to always pass an instance of CaseInsensitiveKey as the key when using the dictionary.
Would you consider using string.lower() on your inputs and using a fully lowercase dictionary? It's a bit of a hacky solution, but it works
I've modified the simple yet good solution by pleasemorebacon (thanks!) making it slightly more compact, self-contained and with minor updates to allow construction from {'a':1, 'B':2} and support __contains__ protocol.
Finally, since the CaseInsensitiveDict.Key is expected to be string (what else can be case-sensitive or not), it is a good idea to derive Key class from the str, then it is possible, for instance, to dump CaseInsensitiveDict with json.dumps out of the box.
# caseinsensitivedict.py
class CaseInsensitiveDict(dict):
class Key(str):
def __init__(self, key):
str.__init__(key)
def __hash__(self):
return hash(self.lower())
def __eq__(self, other):
return self.lower() == other.lower()
def __init__(self, data=None):
super(CaseInsensitiveDict, self).__init__()
if data is None:
data = {}
for key, val in data.items():
self[key] = val
def __contains__(self, key):
key = self.Key(key)
return super(CaseInsensitiveDict, self).__contains__(key)
def __setitem__(self, key, value):
key = self.Key(key)
super(CaseInsensitiveDict, self).__setitem__(key, value)
def __getitem__(self, key):
key = self.Key(key)
return super(CaseInsensitiveDict, self).__getitem__(key)
Here is a basic test script for those who like to check things in action:
# test_CaseInsensitiveDict.py
import json
import unittest
from caseinsensitivedict import *
class Key(unittest.TestCase):
def setUp(self):
self.Key = CaseInsensitiveDict.Key
self.lower = self.Key('a')
self.upper = self.Key('A')
def test_eq(self):
self.assertEqual(self.lower, self.upper)
def test_hash(self):
self.assertEqual(hash(self.lower), hash(self.upper))
def test_str(self):
self.assertEqual(str(self.lower), 'a')
self.assertEqual(str(self.upper), 'A')
class Dict(unittest.TestCase):
def setUp(self):
self.Dict = CaseInsensitiveDict
self.d1 = self.Dict()
self.d2 = self.Dict()
self.d1['a'] = 1
self.d1['B'] = 2
self.d2['A'] = 1
self.d2['b'] = 2
def test_contains(self):
self.assertIn('B', self.d1)
d = self.Dict({'a':1, 'B':2})
self.assertIn('b', d)
def test_init(self):
d = self.Dict()
self.assertFalse(d)
d = self.Dict({'a':1, 'B':2})
self.assertTrue(d)
def test_items(self):
self.assertDictEqual(self.d1, self.d2)
self.assertEqual(
[v for v in self.d1.items()],
[v for v in self.d2.items()])
def test_json_dumps(self):
s = json.dumps(self.d1)
self.assertIn('a', s)
self.assertIn('B', s)
def test_keys(self):
self.assertEqual(self.d1.keys(), self.d2.keys())
def test_values(self):
self.assertEqual(
[v for v in self.d1.values()],
[v for v in self.d2.values()])
While a case insensitive dictionary is a solution, and there are answers to how to achieve that, there is a possibly easier way in this case. A case insensitive search is sufficient:
import re
text = "Practice changing the Color"
words = {'color': 'colour', 'practice': 'practise'}
def replace(words,text):
keys = words.keys()
for i in keys:
exp = re.compile(i, re.I)
text = re.sub(exp, words[i], text)
return text
text = replace(words,text)
print text
You can do a dict key case insensitive search with a one liner:
>>> input_dict = {'aBc':1, 'xyZ':2}
>>> search_string = 'ABC'
>>> next((value for key, value in input_dict.items() if key.lower()==search_string.lower()), None)
1
>>> search_string = 'EFG'
>>> next((value for key, value in input_dict.items() if key.lower()==search_string.lower()), None)
>>>
You can place that into a function:
def get_case_insensitive_key_value(input_dict, key):
return next((value for dict_key, value in input_dict.items() if dict_key.lower() == key.lower()), None)
Note that only the first match is returned.
If you only need to do this once in your code (hence, no point to a function), the most straightforward way to deal with the problem is this:
lowercase_dict = {key.lower(): value for (key, value) in original_dict}
I'm assuming here that the dict in question isn't all that large--it might be inelegant to duplicate it, but if it's not large, it isn't going to hurt anything.
The advantage of this over #Fred's answer (though that also works) is that it produces the same result as a dict when the key isn't present: a KeyError.
There are multiple approaches to this problem, each has its set of pros and cons. Just to add to the list (looks like this option wasn't mentioned), it's possible to extend str class and use it as a key:
class CaseInsensitiveStr(str):
def __hash__(self) -> 'int':
return hash(self.lower())
def __eq__(self, other:'str') -> 'bool':
return self.lower() == other.lower()
It can work well if dictionary in question is private and some kind of interface is used to access it.
class MyThing:
def __init__(self):
self._d: 'dict[CaseInsensitiveStr, int]' = dict()
def set(self, key:'str', value:'int'):
self._d[CaseInsensitiveStr(key)] = value
def get(self, key:'str') -> 'int':
return self._d[CaseInsensitiveStr(key)]
I just set up a function to handle this:
def setLCdict(d, k, v):
k = k.lower()
d[k] = v
return d
myDict = {}
So instead of
myDict['A'] = 1
myDict['B'] = 2
You can:
myDict = setLCdict(myDict, 'A', 1)
myDict = setLCdict(myDict, 'B', 2)
You can then either lower case the value before looking it up or write a function to do so.
def lookupLCdict(d, k):
k = k.lower()
return d[k]
myVal = lookupLCdict(myDict, 'a')
Probably not ideal if you want to do this globally but works well if its just a subset you wish to use it for.