Develop Reference

Mutable Wrapper for Immutable Dictionary

Here is the problem; I have an immutable dictionary with huge set of items. The key type and the value types contained within this dict are themselves immutable. I would like to be able to mutate this dict (adding/removing/replacing key-value pairs) without having to do a full copy of the dict.
I am imagining some wrapper class for the immutable dict which adheres to the dict contract, and defaults to the immutable dict for values that have not been updated. I see the post How to “perfectly” override a dict? which I plan to leverage to make this wrapper.
Before I embark on implementing this design I just wanted to ask- is this construct already provided by the language? Or how else can I achieve the desired effect? I am on the latest version of Python (3.7) so I can use all language features available. Thanks!

Take a look at collections.ChainMap. It's a wrapper around multiple dictionaries: all writes go to the first dictionary, and lookups are searched in order of the maps. So I think you could just do something like:
modified_map = {}
mutable_map = collections.ChainMap(modified_map, huge_immutable_map)

Let's say you used a frozendict implementation like this one:
class frozendict(collections.Mapping):
"""
An immutable wrapper around dictionaries that implements the complete :py:class:`collections.Mapping`
interface. It can be used as a drop-in replacement for dictionaries where immutability is desired.
"""
dict_cls = dict
def __init__(self, *args, **kwargs):
self._dict = self.dict_cls(*args, **kwargs)
self._hash = None
def __getitem__(self, key):
return self._dict[key]
def __contains__(self, key):
return key in self._dict
def copy(self, **add_or_replace):
return self.__class__(self, **add_or_replace)
def __iter__(self):
return iter(self._dict)
def __len__(self):
return len(self._dict)
def __repr__(self):
return '<%s %r>' % (self.__class__.__name__, self._dict)
def __hash__(self):
if self._hash is None:
h = 0
for key, value in self._dict.items():
h ^= hash((key, value))
self._hash = h
return self._hash
If you wanted to mutate it, you could just reach in and mutate self._dict:
d = frozendict({'a': 1, 'b': 2})
d['a'] = 3 # This fails
mutable_dict = d._dict
mutable_dict['a'] = 3 # This works
print(d['a'])
It's a little yucky reaching into a class's protected members, but I'd say that's okay because what you're trying to do is a little yucky. If you want a mutable dictionary (just a dict) then use one. If you never want to mutate it, use a frozendict implementation like the one above. A hyrid of mutable and immutable makes no sense. All frozendict does is it doesn't implement the mutation dunder methods (__setitem__, __delitem__, etc.). Under the hood a frozendict is represented by a regular, mutable dict.
The above approach is superior in my view to the one you linked. Composability (having frozendict have a _dict property) is much easier to reason about than inheritance (subclassing dict) in many cases.

Delay creating python dict until set/update

I have a Python dict-of-dict structure with a large number of outer-dict keys (millions to billions). The inner dicts are mostly empty, but can store key-value pairs. Currently I create a separate dict as each of the inner dicts. But it uses a lot of memory that I don't end up using. Each empty dict is small, but I have a lot of them. I'd like to delay creating the inner dict until needed.
Ideally, I'd like to even delay creating the inner dict until a key-value pair is set in the inner dict. I envision using a single DelayDict object for ALL outer-dict values. This object would act like an empty dict for get and getitem calls, but as soon as a setitem or update call comes in it would create an empty dict to take its place. I run into trouble having the delaydict object know how to connect the new empty dict with the dict-of-dict structure.
class DelayDict(object): % can do much more - only showing get/set
def __init__(self, dod):
self.dictofdict = dod % the outer dict
def __getitem__(self, key):
raise KeyError(key)
def __setitem__(self, key, value):
replacement = {key: value}
% replace myself in the outer dict!!
self.dict-of-dict[?????] = replacement
I can't think of how to store the new replacement dict in the dict-of-dict structure so that it replaces the DelayDict class as the inner dict. I know properties can do similar things, but I believe the same fundamental trouble arises when I try to replace myself inside the outer dict.

Old question, but I came across a similar problem. I'm not sure that it's a
good idea to try to spare some memory, but if you really need to do that, you should try to build your own data structure.
If you are stuck with the dict of dict, here's a solution.
First, you need a way to create keys in the OuterDict without value (value is {}) by default). if OuterDict is a wrapper around a dict __d:
def create(self, key):
self.__d[key] = None
How much memory will you spare?
>>> import sys
>>> a = {}
>>> sys.getsizeof(a)
136
As you pointed out, None is created only once, but you have to keep a reference on it. In Cpython (64 bits), it's 8 bytes. For 1 billion elements, you spare (136-8)* 10**9 bytes = 128 Gb (and not Mb, thanks!). You need to give a
placeholder when someone ask for the value. The placeholder keeps track of the outer dict and the key in the outer dict. It wraps a dict and assigns this dict to outer[key] when you assign a value.
No more talking, code:
class OuterDict():
def __init__(self):
self.__d = {}
def __getitem__(self, key):
v = self.__d[key]
if v is None: # an orphan key
v = PlaceHolder(self.__d, key)
return v
def create(self, key):
self.__d[key] = None
class PlaceHolder():
def __init__(self, parent, key):
self.__parent = parent
self.__key = key
self.__d = {}
def __getitem__(self, key):
return self.__d[key]
def __setitem__(self, key, value):
if not self.__d:
self.__parent[self.__key] = self.__d # copy me in the outer dict
self.__d[key] = value
def __repr__(self):
return repr("PlaceHolder for "+str(self.__d))
# __len__, ...
A test:
o = OuterDict()
o.create("a") # a is empty
print (o["a"])
try:
o["a"]["b"] # Key Error
except KeyError as e:
print ("KeyError", e)
o["a"]["b"] = 2
print (o["a"])
# output:
# 'PlaceHolder for {}'
# KeyError 'b'
# {'b': 2}
Why it doesn't use much memory? Because you are not building billions of placeholders. You release them when you don't need them anymore. Maybe you will need just one at a time.
Possible improvements: you can create a pool of PlaceHolders. A stack may be a good data structure: recently created placeholder are likely to be released soon. When you need a new PlaceHolder, you
look into the stack, and if a placeholder has only one ref (sys.getrefcount(ph) == 1), you can use it. To fasten the process, when you are looking for
a free placeholder, you can remember the placeholder with the maximum refcount. You switch the free placeholder with this "max refcount" placeholder. Hence, the placeholders with the maximum
refcount are sent to the bottom of the stack.

Choosing variables to create when initializing classes

I have a class which would be a container for a number of variables of different types. The collection is finite and not very large so I didn't use a dictionary. Is there a way to automate, or shorten the creation of variables based on whether or not they are requested (specified as True/False) in the constructor?
Here is what I have for example:
class test:
def __init__(self,a=False,b=False,c=False):
if a: self.a = {}
if b: self.b = 34
if c: self.c = "generic string"
For any of a,b,c that are true in the constructor they will be created in the object.
I have a collection of standard variables (a,b,c,d..) that some objects will have and some objects won't. The number of combinations is too large to create separate classes, but the number of variables isn't enough to have a dictionary for them in each class.
Is there any way in python to do something like this:
class test:
def __init__(self,*args):
default_values = {a:{},b:34,c:"generic string"}
for item in args:
if item: self.arg = default_values[arg]
Maybe there is a whole other way to do this?
EDIT:
To clarify this a class which represents different type of bounding boxes on a 2D surface. Depending on the function of the box it can have any of frame coordinates, internal cross coordinates, id, population statistics (attached to that box), and some other cached values for easy calculation.
I don't want to have each object as a dictionary because there are methods attached to it which allow it to export and modify its internal data and interact with other objects of the same type (similar to how strings interact with + - .join, etc.). I also don't want to have a dictionary inside each object because the call to that variable is inelegant:
print foo.info["a"]
versus
print foo.a
Thanks to ballsdotball I've come up with a solution:
class test:
def __init__(self, a=False, b=False,c =False):
default_values = {"a":{},"b":34,"c":"generic string"}
for k, v in default_values.iteritems():
if eval(k): setattr(self,k,v)

Maybe something like:
def __init__(self,*args,**kwargs):
default_values = {a:{},b:34,c:"generic string"}
for k,v in kwargs.iteritems():
try:
if not v is False:
setattr(self,k,default_values[k])
except Exception, e:
print "Argument has no default value.",e
But to be honest I would just put the default values in with the init arguments instead of having to test for them like that.
*Edited a couple times for syntax.

You can subclass dict (if you aren't using positional arguments):
class Test(dict):
def your_method(self):
return self['foo'] * 4
You can also override __getattr__ and __setattr__ if the self['foo'] syntax bothers you:
class Test(dict):
def __getattr__(self, key):
return dict.__getattr__(self, key)
def __setattr__(self, key, value):
return dict.__getattr__(self, key, value)
def your_method(self):
return self.foo * 4

Is there a "standard" basic class in python?

I am looking for a way to create a basic python "object" which I can externally assign attributes to.
Currently I am doing it the following way:
I define an empty class with
class C(object):
pass
and then I instantiate an object and assign attributes like this:
c = C()
c.attr = 2
Coming to my question
Is there a way to instantiate an empty class object, which I can then assign attributes like shown above without defining a class C?
Is there maybe an other better way to accomplish what I am after?

It looks like you are looking for a flexible container that has no methods and can take attributes with arbitrary names. That's a dict.
d = dict()
d['myattr'] = 42
If you prefer the attribute syntax that you get with a class (c.myattr = 42), then use a class just as per the code in your question.

Is there a way to instantiate an empty class object, which I can then assign attributes like shown above without defining a class C?
Yes:
>>> C = type("C", (object,), {})
>>> c = C()
>>> c.attr = 2
But as you can see, it's not much of an improvement, and the end result is the same -- it's just another way of creating the same class C.
Addendum:
You can make it prettier by "hiding" it in a function:
def attr_holder(cls=type("C", (object,), {})):
return cls()
c = attr_holder()
c.attr = 2
Though this is just reinventing the wheel -- replace the two line function with
class attr_holder(object):
pass
and it'll work exactly the same, and we've come full circle. So: go with what David or Reorx suggests.

I had come to the same question long ago, and then create this class to use in many of my projects:
class DotDict(dict):
"""
retrieve value of dict in dot style
"""
def __getattr__(self, key):
try:
return self[key]
except KeyError:
raise AttributeError('has no attribute %s' % key)
def __setattr__(self, key, value):
self[key] = value
def __delattr__(self, key):
try:
del self[key]
except KeyError:
raise AttributeError(key)
def __str__(self):
return '<DotDict %s >' % self.__to_dict()
def __to_dict(self):
return dict(self)
When I want a object to store data or want to retrieve value easily from a dict, I always use this class.
Additionally, it can help me serialize the attributes that I set in the object, and reversely get the original dict when needed.
So I think this may be a good solution in many situations, though other tricks look simple,
they are not very helpful further.

What do I do when I need a self referential dictionary?

I'm new to Python, and am sort of surprised I cannot do this.
dictionary = {
'a' : '123',
'b' : dictionary['a'] + '456'
}
I'm wondering what the Pythonic way to correctly do this in my script, because I feel like I'm not the only one that has tried to do this.
EDIT: Enough people were wondering what I'm doing with this, so here are more details for my use cases. Lets say I want to keep dictionary objects to hold file system paths. The paths are relative to other values in the dictionary. For example, this is what one of my dictionaries may look like.
dictionary = {
'user': 'sholsapp',
'home': '/home/' + dictionary['user']
}
It is important that at any point in time I may change dictionary['user'] and have all of the dictionaries values reflect the change. Again, this is an example of what I'm using it for, so I hope that it conveys my goal.
From my own research I think I will need to implement a class to do this.

No fear of creating new classes -
You can take advantage of Python's string formating capabilities
and simply do:
class MyDict(dict):
def __getitem__(self, item):
return dict.__getitem__(self, item) % self
dictionary = MyDict({
'user' : 'gnucom',
'home' : '/home/%(user)s',
'bin' : '%(home)s/bin'
})
print dictionary["home"]
print dictionary["bin"]

Nearest I came up without doing object:
dictionary = {
'user' : 'gnucom',
'home' : lambda:'/home/'+dictionary['user']
}
print dictionary['home']()
dictionary['user']='tony'
print dictionary['home']()

>>> dictionary = {
... 'a':'123'
... }
>>> dictionary['b'] = dictionary['a'] + '456'
>>> dictionary
{'a': '123', 'b': '123456'}
It works fine but when you're trying to use dictionary it hasn't been defined yet (because it has to evaluate that literal dictionary first).
But be careful because this assigns to the key of 'b' the value referenced by the key of 'a' at the time of assignment and is not going to do the lookup every time. If that is what you are looking for, it's possible but with more work.

What you're describing in your edit is how an INI config file works. Python does have a built in library called ConfigParser which should work for what you're describing.

This is an interesting problem. It seems like Greg has a good solution. But that's no fun ;)
jsbueno as a very elegant solution but that only applies to strings (as you requested).
The trick to a 'general' self referential dictionary is to use a surrogate object. It takes a few (understatement) lines of code to pull off, but the usage is about what you want:
S = SurrogateDict(AdditionSurrogateDictEntry)
d = S.resolve({'user': 'gnucom',
'home': '/home/' + S['user'],
'config': [S['home'] + '/.emacs', S['home'] + '/.bashrc']})
The code to make that happen is not nearly so short. It lives in three classes:
import abc
class SurrogateDictEntry(object):
__metaclass__ = abc.ABCMeta
def __init__(self, key):
"""record the key on the real dictionary that this will resolve to a
value for
"""
self.key = key
def resolve(self, d):
""" return the actual value"""
if hasattr(self, 'op'):
# any operation done on self will store it's name in self.op.
# if this is set, resolve it by calling the appropriate method
# now that we can get self.value out of d
self.value = d[self.key]
return getattr(self, self.op + 'resolve__')()
else:
return d[self.key]
#staticmethod
def make_op(opname):
"""A convience class. This will be the form of all op hooks for subclasses
The actual logic for the op is in __op__resolve__ (e.g. __add__resolve__)
"""
def op(self, other):
self.stored_value = other
self.op = opname
return self
op.__name__ = opname
return op
Next, comes the concrete class. simple enough.
class AdditionSurrogateDictEntry(SurrogateDictEntry):
__add__ = SurrogateDictEntry.make_op('__add__')
__radd__ = SurrogateDictEntry.make_op('__radd__')
def __add__resolve__(self):
return self.value + self.stored_value
def __radd__resolve__(self):
return self.stored_value + self.value
Here's the final class
class SurrogateDict(object):
def __init__(self, EntryClass):
self.EntryClass = EntryClass
def __getitem__(self, key):
"""record the key and return"""
return self.EntryClass(key)
#staticmethod
def resolve(d):
"""I eat generators resolve self references"""
stack = [d]
while stack:
cur = stack.pop()
# This just tries to set it to an appropriate iterable
it = xrange(len(cur)) if not hasattr(cur, 'keys') else cur.keys()
for key in it:
# sorry for being a duche. Just register your class with
# SurrogateDictEntry and you can pass whatever.
while isinstance(cur[key], SurrogateDictEntry):
cur[key] = cur[key].resolve(d)
# I'm just going to check for iter but you can add other
# checks here for items that we should loop over.
if hasattr(cur[key], '__iter__'):
stack.append(cur[key])
return d
In response to gnucoms's question about why I named the classes the way that I did.
The word surrogate is generally associated with standing in for something else so it seemed appropriate because that's what the SurrogateDict class does: an instance replaces the 'self' references in a dictionary literal. That being said, (other than just being straight up stupid sometimes) naming is probably one of the hardest things for me about coding. If you (or anyone else) can suggest a better name, I'm all ears.
I'll provide a brief explanation. Throughout S refers to an instance of SurrogateDict and d is the real dictionary.
A reference S[key] triggers S.__getitem__ and SurrogateDictEntry(key) to be placed in the d.
When S[key] = SurrogateDictEntry(key) is constructed, it stores key. This will be the key into d for the value that this entry of SurrogateDictEntry is acting as a surrogate for.
After S[key] is returned, it is either entered into the d, or has some operation(s) performed on it. If an operation is performed on it, it triggers the relative __op__ method which simple stores the value that the operation is performed on and the name of the operation and then returns itself. We can't actually resolve the operation because d hasn't been constructed yet.
After d is constructed, it is passed to S.resolve. This method loops through d finding any instances of SurrogateDictEntry and replacing them with the result of calling the resolve method on the instance.
The SurrogateDictEntry.resolve method receives the now constructed d as an argument and can use the value of key that it stored at construction time to get the value that it is acting as a surrogate for. If an operation was performed on it after creation, the op attribute will have been set with the name of the operation that was performed. If the class has a __op__ method, then it has a __op__resolve__ method with the actual logic that would normally be in the __op__ method. So now we have the logic (self.op__resolve) and all necessary values (self.value, self.stored_value) to finally get the real value of d[key]. So we return that which step 4 places in the dictionary.
finally the SurrogateDict.resolve method returns d with all references resolved.
That'a a rough sketch. If you have any more questions, feel free to ask.

If you, just like me wandering how to make #jsbueno snippet work with {} style substitutions, below is the example code (which is probably not much efficient though):
import string
class MyDict(dict):
def __init__(self, *args, **kw):
super(MyDict,self).__init__(*args, **kw)
self.itemlist = super(MyDict,self).keys()
self.fmt = string.Formatter()
def __getitem__(self, item):
return self.fmt.vformat(dict.__getitem__(self, item), {}, self)
xs = MyDict({
'user' : 'gnucom',
'home' : '/home/{user}',
'bin' : '{home}/bin'
})
>>> xs["home"]
'/home/gnucom'
>>> xs["bin"]
'/home/gnucom/bin'
I tried to make it work with the simple replacement of % self with .format(**self) but it turns out it wouldn't work for nested expressions (like 'bin' in above listing, which references 'home', which has it's own reference to 'user') because of the evaluation order (** expansion is done before actual format call and it's not delayed like in original % version).

Write a class, maybe something with properties:
class PathInfo(object):
def __init__(self, user):
self.user = user
#property
def home(self):
return '/home/' + self.user
p = PathInfo('thc')
print p.home # /home/thc

As sort of an extended version of #Tony's answer, you could build a dictionary subclass that calls its values if they are callables:
class CallingDict(dict):
"""Returns the result rather than the value of referenced callables.
>>> cd = CallingDict({1: "One", 2: "Two", 'fsh': "Fish",
... "rhyme": lambda d: ' '.join((d[1], d['fsh'],
... d[2], d['fsh']))})
>>> cd["rhyme"]
'One Fish Two Fish'
>>> cd[1] = 'Red'
>>> cd[2] = 'Blue'
>>> cd["rhyme"]
'Red Fish Blue Fish'
"""
def __getitem__(self, item):
it = super(CallingDict, self).__getitem__(item)
if callable(it):
return it(self)
else:
return it
Of course this would only be usable if you're not actually going to store callables as values. If you need to be able to do that, you could wrap the lambda declaration in a function that adds some attribute to the resulting lambda, and check for it in CallingDict.__getitem__, but at that point it's getting complex, and long-winded, enough that it might just be easier to use a class for your data in the first place.

This is very easy in a lazily evaluated language (haskell).
Since Python is strictly evaluated, we can do a little trick to turn things lazy:
Y = lambda f: (lambda x: x(x))(lambda y: f(lambda *args: y(y)(*args)))
d1 = lambda self: lambda: {
'a': lambda: 3,
'b': lambda: self()['a']()
}
# fix the d1, and evaluate it
d2 = Y(d1)()
# to get a
d2['a']() # 3
# to get b
d2['b']() # 3
Syntax wise this is not very nice. That's because of us needing to explicitly construct lazy expressions with lambda: ... and explicitly evaluate lazy expression with ...(). It's the opposite problem in lazy languages needing strictness annotations, here in Python we end up needing lazy annotations.
I think with some more meta-programmming and some more tricks, the above could be made more easy to use.
Note that this is basically how let-rec works in some functional languages.

The jsbueno answer in Python 3 :
class MyDict(dict):
def __getitem__(self, item):
return dict.__getitem__(self, item).format(self)
dictionary = MyDict({
'user' : 'gnucom',
'home' : '/home/{0[user]}',
'bin' : '{0[home]}/bin'
})
print(dictionary["home"])
print(dictionary["bin"])
Her ewe use the python 3 string formatting with curly braces {} and the .format() method.
Documentation : https://docs.python.org/3/library/string.html