Is there something like an "infinite dictionary" in Python?
More precisely, is there something where
- i can put in values like in a dictionary,
- but maybe also a function which tells me how to map a key to a value,
- and maybe also something that maps a key to a (finite) set of keys and then gives the corresponding value?
Formulated in another way, what I want to have is the following "thing":
I initialize it in a way (give values, functions, whatever) and then it just gives me for each key a value (on request).
You will want to create a class with the special method __getitem__(self,key) that returns the appropriate value for that key.
What you need is called a "function".
Now, on a less sarcastic note: I don't know exactly what you are trying to achieve, but here's an example:
You want a piece of code that returns the nth element in an arithmetic progression. You can do it this way with functions:
def progression(first_element, ratio):
def nth_element(n):
return n*ratio + first_element
return nth_element
my_progression = progression(2, 32)
print my_progression(17) # prints 546
This can be extended if, for example, you need a function that retains state.
Hope this helps
If you want normal behaviour for existing keys, and special behavior for non-existing keys, there's the __missing__ method that's called for missing keys.
class funny_dict(dict):
def __missing__(self, key):
return "funny" * key
d = funny_dict()
d[1] = "asdf"
d[3] = 3.14
for i in range(5):
print(i, d[i])
print(d)
Output:
0
1 asdf
2 funnyfunny
3 3.14
4 funnyfunnyfunnyfunny
{1: 'asdf', 3: 3.14}
An easy way to do this would be to use a function object for both use cases. If you want to use a key-value function, you just just use it directly as a reference. To adapt an ordinary dictionary to this interface, you can wrap it in a lambda block. Like so:
# Use function as dictionary
def dict_func(key):
return key * key
dictionary = dict_func
print dictionary(2) # prints 4
# Use normal dictionary with the same interface
normal_dict = {1: 1, 2: 4, 3: 9}
dictionary = lambda(key): normal_dict[key]
print dictionary(2) # also prints 4
# Lambda functions store references to the variables they use,
# so this works too:
def fn_dict(normal_dict):
return lambda(key): normal_dict[key]
dictionary = fn_dict({1: 1, 2: 4, 3: 9})
print dictionary(3) # prints 9
I think you want something like this, where you dict act like a normal dictionary but for special keys you want to change the behavior e.g.
class InfiniteDict(dict):
def __init__(self, *args, **kwargs):
self.key_funcs = kwargs.pop('key_funcs', [])
super(InfiniteDict, self).__init__(*args, **kwargs)
def __getitem__(self, key):
try:
return super(InfiniteDict, self).__getitem__(key)
except KeyError:
return self._get_value_from_functions(key)
def _get_value_from_functions(self, key):
"""
go thru list of user defined functions and return first match
"""
for key_func in self.key_funcs:
try:
return key_func(key)
except KeyError:
pass
raise KeyError(key)
def double_even_int(key):
try:
if int(key)%2 == 0:
return int(key)*2
else:
raise KeyError(key)
except ValueError:
raise KeyError(key)
def tripple_odd_int(key):
try:
if int(key)%2 == 1:
return int(key)*3
else:
raise KeyError(key)
except ValueError:
raise KeyError(key)
inf = InfiniteDict(key_funcs=[double_even_int, tripple_odd_int])
inf['a'] = 'A'
print inf['a'], inf[1], inf['2']
output:
A 3 4
Related
I am trying to provide a function as the default argument for the dictionary's get function, like this
def run():
print "RUNNING"
test = {'store':1}
test.get('store', run())
However, when this is run, it displays the following output:
RUNNING
1
so my question is, as the title says, is there a way to provide a callable as the default value for the get method without it being called if the key exists?
Another option, assuming you don't intend to store falsy values in your dictionary:
test.get('store') or run()
In python, the or operator does not evaluate arguments that are not needed (it short-circuits)
If you do need to support falsy values, then you can use get_or_run(test, 'store', run) where:
def get_or_run(d, k, f):
sentinel = object() # guaranteed not to be in d
v = d.get(k, sentinel)
return f() if v is sentinel else v
See the discussion in the answers and comments of dict.get() method returns a pointer. You have to break it into two steps.
Your options are:
Use a defaultdict with the callable if you always want that value as the default, and want to store it in the dict.
Use a conditional expression:
item = test['store'] if 'store' in test else run()
Use try / except:
try:
item = test['store']
except KeyError:
item = run()
Use get:
item = test.get('store')
if item is None:
item = run()
And variations on those themes.
glglgl shows a way to subclass defaultdict, you can also just subclass dict for some situations:
def run():
print "RUNNING"
return 1
class dict_nokeyerror(dict):
def __missing__(self, key):
return run()
test = dict_nokeyerror()
print test['a']
# RUNNING
# 1
Subclassing only really makes sense if you always want the dict to have some nonstandard behavior; if you generally want it to behave like a normal dict and just want a lazy get in one place, use one of my methods 2-4.
I suppose you want to have the callable applied only if the key does not exist.
There are several approaches to do so.
One would be to use a defaultdict, which calls run() if key is missing.
from collections import defaultdict
def run():
print "RUNNING"
test = {'store':1}
test.get('store', run())
test = defaultdict(run, store=1) # provides a value for store
test['store'] # gets 1
test['runthatstuff'] # gets None
Another, rather ugly one, one would be to only save callables in the dict which return the apropriate value.
test = {'store': lambda:1}
test.get('store', run)() # -> 1
test.get('runrun', run)() # -> None, prints "RUNNING".
If you want to have the return value depend on the missing key, you have to subclass defaultdict:
class mydefaultdict(defaultdict):
def __missing__(self, key):
val = self[key] = self.default_factory(key)
return val
d = mydefaultdict(lambda k: k*k)
d[10] # yields 100
#mydefaultdict # decorators are fine
def d2(key):
return -key
d2[5] # yields -5
And if you want not to add this value to the dict for the next call, you have a
def __missing__(self, key): return self.default_factory(key)
instead which calls the default factory every time a key: value pair was not explicitly added.
If you only know what the callable is likely to be at he get call site you could subclass dict something like this
class MyDict(dict):
def get_callable(self,key,func,*args,**kwargs):
'''Like ordinary get but uses a callable to
generate the default value'''
if key not in self:
val = func(*args,**kwargs)
else:
val = self[key]
return val
This can then be used like so:-
>>> d = MyDict()
>>> d.get_callable(1,complex,2,3)
(2+3j)
>>> d[1] = 2
>>> d.get_callable(1,complex,2,3)
2
>>> def run(): print "run"
>>> repr(d.get_callable(1,run))
'2'
>>> repr(d.get_callable(2,run))
run
'None'
This is probably most useful when the callable is expensive to compute.
I have a util directory in my project with qt.py, general.py, geom.py, etc. In general.py I have a bunch of python tools like the one you need:
# Use whenever you need a lambda default
def dictGet(dict_, key, default):
if key not in dict_:
return default()
return dict_[key]
Add *args, **kwargs if you want to support calling default more than once with differing args:
def dictGet(dict_, key, default, *args, **kwargs):
if key not in dict_:
return default(*args, **kwargs)
return dict_[key]
Here's what I use:
def lazy_get(d, k, f):
return d[k] if k in d else f(k)
The fallback function f takes the key as an argument, e.g.
lazy_get({'a': 13}, 'a', lambda k: k) # --> 13
lazy_get({'a': 13}, 'b', lambda k: k) # --> 'b'
You would obviously use a more meaningful fallback function, but this illustrates the flexibility of lazy_get.
Here's what the function looks like with type annotation:
from typing import Callable, Mapping, TypeVar
K = TypeVar('K')
V = TypeVar('V')
def lazy_get(d: Mapping[K, V], k: K, f: Callable[[K], V]) -> V:
return d[k] if k in d else f(k)
I have been investigating and found out that using if in is the fastest compare to ->
benchmark
and I have been trying to create a function where I can pass arguments on what path I want the if in will follow e.g.
def main():
d = {"foo": "spam"}
if "bar" in d:
if "eggs" in d["bar"]:
d["bar"]["eggs"]
else:
{}
else:
{}
But instead of having a long code, I was trying to do a function where I can pass argument e.g. get_path(json_data, 'foo', 'eggs') which would try to do something similar to the code above and return if value found else return empty.
My question is how can I create a function where we can pass argument to do the if in checks and return the value if it's found?
You could pass your keys as tuple/list:
def main(data, keys):
for k in keys:
if k not in data:
return {}
data = data[k]
return data
d = {"foo": "spam", "bar": {"eggs": "HAM!"}}
print(main(d, ('bar', 'eggs')))
Out:
HAM!
This is a nice little problem that has a fairly easy solution as long as everything is dicts:
def get_path(data, *path):
node = data
for step in path:
if step in node:
node = node[step]
else:
return {} # None might be more appropriate here
return node
note that it won't work quite right if you encounter a list along the way: although lists support [] and they support in, in means something different to them ("is this value found", rather than "is this key found"), so the test generally won't succeed.
One minor annoyance with dict.setdefault is that it always evaluates its second argument (when given, of course), even when the first the first argument is already a key in the dictionary.
For example:
import random
def noisy_default():
ret = random.randint(0, 10000000)
print 'noisy_default: returning %d' % ret
return ret
d = dict()
print d.setdefault(1, noisy_default())
print d.setdefault(1, noisy_default())
This produces ouptut like the following:
noisy_default: returning 4063267
4063267
noisy_default: returning 628989
4063267
As the last line confirms, the second execution of noisy_default is unnecessary, since by this point the key 1 is already present in d (with value 4063267).
Is it possible to implement a subclass of dict whose setdefault method evaluates its second argument lazily?
EDIT:
Below is an implementation inspired by BrenBarn's comment and Pavel Anossov's answer. While at it, I went ahead and implemented a lazy version of get as well, since the underlying idea is essentially the same.
class LazyDict(dict):
def get(self, key, thunk=None):
return (self[key] if key in self else
thunk() if callable(thunk) else
thunk)
def setdefault(self, key, thunk=None):
return (self[key] if key in self else
dict.setdefault(self, key,
thunk() if callable(thunk) else
thunk))
Now, the snippet
d = LazyDict()
print d.setdefault(1, noisy_default)
print d.setdefault(1, noisy_default)
produces output like this:
noisy_default: returning 5025427
5025427
5025427
Notice that the second argument to d.setdefault above is now a callable, not a function call.
When the second argument to LazyDict.get or LazyDict.setdefault is not a callable, they behave the same way as the corresponding dict methods.
If one wants to pass a callable as the default value itself (i.e., not meant to be called), or if the callable to be called requires arguments, prepend lambda: to the appropriate argument. E.g.:
d1.setdefault('div', lambda: div_callback)
d2.setdefault('foo', lambda: bar('frobozz'))
Those who don't like the idea of overriding get and setdefault, and/or the resulting need to test for callability, etc., can use this version instead:
class LazyButHonestDict(dict):
def lazyget(self, key, thunk=lambda: None):
return self[key] if key in self else thunk()
def lazysetdefault(self, key, thunk=lambda: None):
return (self[key] if key in self else
self.setdefault(key, thunk()))
This can be accomplished with defaultdict, too. It is instantiated with a callable which is then called when a nonexisting element is accessed.
from collections import defaultdict
d = defaultdict(noisy_default)
d[1] # noise
d[1] # no noise
The caveat with defaultdict is that the callable gets no arguments, so you can not derive the default value from the key as you could with dict.setdefault. This can be mitigated by overriding __missing__ in a subclass:
from collections import defaultdict
class defaultdict2(defaultdict):
def __missing__(self, key):
value = self.default_factory(key)
self[key] = value
return value
def noisy_default_with_key(key):
print key
return key + 1
d = defaultdict2(noisy_default_with_key)
d[1] # prints 1, sets 2, returns 2
d[1] # does not print anything, does not set anything, returns 2
For more information, see the collections module.
You can do that in a one-liner using a ternary operator:
value = cache[key] if key in cache else cache.setdefault(key, func(key))
If you are sure that the cache will never store falsy values, you can simplify it a little bit:
value = cache.get(key) or cache.setdefault(key, func(key))
No, evaluation of arguments happens before the call. You can implement a setdefault-like function that takes a callable as its second argument and calls it only if it is needed.
There seems to be no one-liner that doesn't require an extra class or extra lookups. For the record, here is a easy (even not concise) way of achieving that without either of them.
try:
value = dct[key]
except KeyError:
value = noisy_default()
dct[key] = value
return value
I want to do something like this:
f[frozenset((1,3,4))] = 5
f[frozenset((1,))] = 3
but it's just painful to type these all the time, is there anyway to have alias for this? I know in C++ it's possible to have a helper function which return a reference so you can just type:
F(1,3,4) = 5
F(1) = 3
with F as a helper function. Thanks very much!
I think that this can really only be achieved via a subclass:
class FrozenSetDict(dict):
def __setitem__(self,idx,value):
try:
dict.__setitem__(self,frozenset(idx),value)
except TypeError:
dict.__setitem__(self,frozenset((idx,)),value)
d = FrozenSetDict()
d[1,2,3] = 4
d[1] = 5
print d
yields:
{frozenset([1, 2, 3]): 4, frozenset([1]): 5}
This introduces an asymmetry between __getitem__ and __setitem__ which could easily be fixed by re-defining __getitem__ in the same way.
This might seem a little messy -- Indeed it is. Why require a subclass? That just makes it harder to put non-frozenset objects into your dictionary as keys. You could easily use this recipe though to create a proxy object which will do this with your dict:
#I don't like the name of this class -- I'm open to suggestions :)
class FrozenSetProxy(object):
def __init__(self,obj):
self.obj = obj
def __setitem__(self,idx,value):
try:
self.obj[frozenset(idx)] = value
except TypeError:
self.obj[frozenset((idx,))] = value
def __getitem__(self,idx):
try:
return self.obj[frozenset(idx)]
except TypeError:
return self.obj[frozenset((idx,))]
d = dict()
F = FrozenSetProxy(d)
F[1,2,3] = 4
F[1] = 5
print d
print F[1]
There's nothing like a C++ reference in Python, and the syntax you use is illegal to boot (in the words of the parser: can't assign to function call). You could emulate it with an object or subclass dict to customize its __getitem__. But there's a simpler and less intrusive way: Pass the value to the helper too, and let it handle the assignment:
def blah(f):
def F(*args, value):
f[frozenset(args)] = value
F(1, 3, 4, value=5)
F(1, value=3)
Note that this uses a Python 3 feature, keyword-only parameters. If you need it to work with Python 2, you can emulate the call syntax by accepting **kwdargs:
def F(*args, **kwds):
# optional: check that no other keyword arguments were passed
f[frozenset(args)] = kwds['value']
The default behavior for python dictionary is to create a new key in the dictionary if that key does not already exist. For example:
d = {}
d['did not exist before'] = 'now it does'
this is all well and good for most purposes, but what if I'd like python to do nothing if the key isn't already in the dictionary. In my situation:
for x in exceptions:
if masterlist.has_key(x):
masterlist[x] = False
in other words, i don't want some incorrect elements in exceptions to corrupt my masterlist. Is this as simple as it gets? it FEELS like I should be able to do this in one line inside the for loop (i.e., without explicitly checking that x is a key of masterlist)
UPDATE:
To me, my question is asking about the lack of a parallel between a list and a dict. For example:
l = []
l[0] = 2 #fails
l.append(2) #works
with the subclassing answer, you could modify the dictionary (maybe "safe_dict" or "explicit_dict" to do something similar:
d = {}
d['a'] = '1' #would fail in my world
d.insert('a','1') #what my world is missing
You could use .update:
masterlist.update((x, False) for x in exceptions if masterlist.has_key(x))
You can inherit a dict class, override it's __setitem__ to check for existance of key (or do the same with monkey-patching only one instance).
Sample class:
class a(dict):
def __init__(self, *args, **kwargs):
dict.__init__(self, *args, **kwargs)
dict.__setitem__(self, 'a', 'b')
def __setitem__(self, key, value):
if self.has_key(key):
dict.__setitem__(self, key, value)
a = a()
print a['a'] # prints 'b'
a['c'] = 'd'
# print a['c'] - would fail
a['a'] = 'e'
print a['a'] # prints 'e'
You could also use some function to make setting values without checking for existence simpler.
However, I though it would be shorter... Don't use it unless you need it in many places.
You can also use in instead of has_key, which is a little nicer.
for x in exceptions:
if x in masterlist:
masterlist[x] = False
But I don't see the issue with having an if statement for this purpose.
For long lists try to use the & operator with set() function embraced with ():
for x in (set(exceptions) & set(masterlist)):
masterlist[x] = False
#or masterlist[x] = exceptions[x]
It'll improve the reading and the iterations at the same time by reading the masterlist's keys only once.