python dict: get vs setdefault - python
The following two expressions seem equivalent to me. Which one is preferable?
data = [('a', 1), ('b', 1), ('b', 2)]
d1 = {}
d2 = {}
for key, val in data:
# variant 1)
d1[key] = d1.get(key, []) + [val]
# variant 2)
d2.setdefault(key, []).append(val)
The results are the same but which version is better or rather more pythonic?
Personally I find version 2 harder to understand, as to me setdefault is very tricky to grasp. If I understand correctly, it looks for the value of "key" in the dictionary, if not available, enters "[]" into the dict, returns a reference to either the value or "[]" and appends "val" to that reference. While certainly smooth it is not intuitive in the least (at least to me).
To my mind, version 1 is easier to understand (if available, get the value for "key", if not, get "[]", then join with a list made up from [val] and place the result in "key"). But while more intuitive to understand, I fear this version is less performant, with all this list creating. Another disadvantage is that "d1" occurs twice in the expression which is rather error-prone. Probably there is a better implementation using get, but presently it eludes me.
My guess is that version 2, although more difficult to grasp for the inexperienced, is faster and therefore preferable. Opinions?
Your two examples do the same thing, but that doesn't mean get and setdefault do.
The difference between the two is basically manually setting d[key] to point to the list every time, versus setdefault automatically setting d[key] to the list only when it's unset.
Making the two methods as similar as possible, I ran
from timeit import timeit
print timeit("c = d.get(0, []); c.extend([1]); d[0] = c", "d = {1: []}", number = 1000000)
print timeit("c = d.get(1, []); c.extend([1]); d[0] = c", "d = {1: []}", number = 1000000)
print timeit("d.setdefault(0, []).extend([1])", "d = {1: []}", number = 1000000)
print timeit("d.setdefault(1, []).extend([1])", "d = {1: []}", number = 1000000)
and got
0.794723378711
0.811882272256
0.724429205999
0.722129751973
So setdefault is around 10% faster than get for this purpose.
The get method allows you to do less than you can with setdefault. You can use it to avoid getting a KeyError when the key doesn't exist (if that's something that's going to happen frequently) even if you don't want to set the key.
See Use cases for the 'setdefault' dict method and dict.get() method returns a pointer for some more info about the two methods.
The thread about setdefault concludes that most of the time, you want to use a defaultdict. The thread about get concludes that it is slow, and often you're better off (speed wise) doing a double lookup, using a defaultdict, or handling the error (depending on the size of the dictionary and your use case).
The accepted answer from agf isn't comparing like with like. After:
print timeit("d[0] = d.get(0, []) + [1]", "d = {1: []}", number = 10000)
d[0] contains a list with 10,000 items whereas after:
print timeit("d.setdefault(0, []) + [1]", "d = {1: []}", number = 10000)
d[0] is simply []. i.e. the d.setdefault version never modifies the list stored in d. The code should actually be:
print timeit("d.setdefault(0, []).append(1)", "d = {1: []}", number = 10000)
and in fact is faster than the faulty setdefault example.
The difference here really is because of when you append using concatenation the whole list is copied every time (and once you have 10,000 elements that is beginning to become measurable. Using append the list updates are amortised O(1), i.e. effectively constant time.
Finally, there are two other options not considered in the original question: defaultdict or simply testing the dictionary to see whether it already contains the key.
So, assuming d3, d4 = defaultdict(list), {}
# variant 1 (0.39)
d1[key] = d1.get(key, []) + [val]
# variant 2 (0.003)
d2.setdefault(key, []).append(val)
# variant 3 (0.0017)
d3[key].append(val)
# variant 4 (0.002)
if key in d4:
d4[key].append(val)
else:
d4[key] = [val]
variant 1 is by far the slowest because it copies the list every time, variant 2 is the second slowest, variant 3 is the fastest but won't work if you need Python older than 2.5, and variant 4 is just slightly slower than variant 3.
I would say use variant 3 if you can, with variant 4 as an option for those occasional places where defaultdict isn't an exact fit. Avoid both of your original variants.
For those who are still struggling in understanding these two term, let me tell you basic difference between get() and setdefault() method -
Scenario-1
root = {}
root.setdefault('A', [])
print(root)
Scenario-2
root = {}
root.get('A', [])
print(root)
In Scenario-1 output will be {'A': []} while in Scenario-2 {}
So setdefault() sets absent keys in the dict while get() only provides you default value but it does not modify the dictionary.
Now let come where this will be useful-
Suppose you are searching an element in a dict whose value is a list and you want to modify that list if found otherwise create a new key with that list.
using setdefault()
def fn1(dic, key, lst):
dic.setdefault(key, []).extend(lst)
using get()
def fn2(dic, key, lst):
dic[key] = dic.get(key, []) + (lst) #Explicit assigning happening here
Now lets examine timings -
dic = {}
%%timeit -n 10000 -r 4
fn1(dic, 'A', [1,2,3])
Took 288 ns
dic = {}
%%timeit -n 10000 -r 4
fn2(dic, 'A', [1,2,3])
Took 128 s
So there is a very large timing difference between these two approaches.
You might want to look at defaultdict in the collections module. The following is equivalent to your examples.
from collections import defaultdict
data = [('a', 1), ('b', 1), ('b', 2)]
d = defaultdict(list)
for k, v in data:
d[k].append(v)
There's more here.
1. Explained with a good example here:
http://code.activestate.com/recipes/66516-add-an-entry-to-a-dictionary-unless-the-entry-is-a/
dict.setdefault typical usage
somedict.setdefault(somekey,[]).append(somevalue)
dict.get typical usage
theIndex[word] = 1 + theIndex.get(word,0)
2. More explanation : http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html
dict.setdefault() is equivalent to get or set & get. Or set if necessary then get. It's especially efficient if your dictionary key is expensive to compute or long to type.
The only problem with dict.setdefault() is that the default value is always evaluated, whether needed or not. That only matters if the default value is expensive to compute. In that case, use defaultdict.
3. Finally the official docs with difference highlighted http://docs.python.org/2/library/stdtypes.html
get(key[, default])
Return the value for key if key is in the dictionary, else default. If
default is not given, it defaults to None, so that this method never
raises a KeyError.
setdefault(key[, default])
If key is in the dictionary, return its value. If not, insert key with a value of default and return default. default defaults to None.
The logic of dict.get is:
if key in a_dict:
value = a_dict[key]
else:
value = default_value
Take an example:
In [72]: a_dict = {'mapping':['dict', 'OrderedDict'], 'array':['list', 'tuple']}
In [73]: a_dict.get('string', ['str', 'bytes'])
Out[73]: ['str', 'bytes']
In [74]: a_dict.get('array', ['str', 'byets'])
Out[74]: ['list', 'tuple']
The mechamism of setdefault is:
levels = ['master', 'manager', 'salesman', 'accountant', 'assistant']
#group them by the leading letter
group_by_leading_letter = {}
# the logic expressed by obvious if condition
for level in levels:
leading_letter = level[0]
if leading_letter not in group_by_leading_letter:
group_by_leading_letter[leading_letter] = [level]
else:
group_by_leading_letter[leading_letter].append(word)
In [80]: group_by_leading_letter
Out[80]: {'a': ['accountant', 'assistant'], 'm': ['master', 'manager'], 's': ['salesman']}
The setdefault dict method is for precisely this purpose. The preceding for loop can be rewritten as:
In [87]: for level in levels:
...: leading = level[0]
...: group_by_leading_letter.setdefault(leading,[]).append(level)
Out[80]: {'a': ['accountant', 'assistant'], 'm': ['master', 'manager'], 's': ['salesman']}
It's very simple, means that either a non-null list append an element or a null list append an element.
The defaultdict, which makes this even easier. To create one, you pass a type or function for generating the default value for each slot in the dict:
from collections import defualtdict
group_by_leading_letter = defaultdict(list)
for level in levels:
group_by_leading_letter[level[0]].append(level)
There is no strict answer to this question. They both accomplish the same purpose. They can both be used to deal with missing values on keys. The only difference that I have found is that with setdefault(), the key that you invoke (if not previously in the dictionary) gets automatically inserted while it does not happen with get(). Here is an example:
Setdefault()
>>> myDict = {'A': 'GOD', 'B':'Is', 'C':'GOOD'} #(1)
>>> myDict.setdefault('C') #(2)
'GOOD'
>>> myDict.setdefault('C','GREAT') #(3)
'GOOD'
>>> myDict.setdefault('D','AWESOME') #(4)
'AWESOME'
>>> myDict #(5)
{'A': 'GOD', 'B': 'Is', 'C': 'GOOD', 'D': 'AWSOME'}
>>> myDict.setdefault('E')
>>>
Get()
>>> myDict = {'a': 1, 'b': 2, 'c': 3} #(1)
>>> myDict.get('a',0) #(2)
1
>>> myDict.get('d',0) #(3)
0
>>> myDict #(4)
{'a': 1, 'b': 2, 'c': 3}
Here is my conclusion: there is no specific answer to which one is best specifically when it comes to default values imputation. The only difference is that setdefault() automatically adds any new key with a default value in the dictionary while get() does not. For more information, please go here !
In [1]: person_dict = {}
In [2]: person_dict['liqi'] = 'LiQi'
In [3]: person_dict.setdefault('liqi', 'Liqi')
Out[3]: 'LiQi'
In [4]: person_dict.setdefault('Kim', 'kim')
Out[4]: 'kim'
In [5]: person_dict
Out[5]: {'Kim': 'kim', 'liqi': 'LiQi'}
In [8]: person_dict.get('Dim', '')
Out[8]: ''
In [5]: person_dict
Out[5]: {'Kim': 'kim', 'liqi': 'LiQi'}
Related
Update multiple key/value pairs in python at once
Say I have d = {"a":0,"b":0,"c":0} is there a way to update the keys a and b at the same time, instead of looping over them, such like update_keys = ["a","b"] d.some_function(update_keys) +=[10,5] print(d) {"a":10,"b":5,"c":0}
Yes, you can use update like this: d.update({'a':10, 'b':5}) Thus, your code would look this way: d = {"a":0,"b":0,"c":0} d.update({'a':10, 'b':5}) print(d) and shows: {"a":10,"b":5,"c":0}
If you mean a function that can add a new value to the existing value without an explict loop, you can definitely do it like this. add_value = lambda d,k,v: d.update(zip(k,list(map(lambda _k,_v:d[_k]+_v,k,v)))) or d and you can use it like this >>> d = {"a":2,"b":3} >>> add_value(d,["a","b"],[2,-3]) {'a': 4, 'b': 0} There is nothing tricky here, I just replace the loop with a map and a lambda to do the update job and use list to wrap them up so Python will immediately evaluate the result of map. Then I use zip to create an updated key-value pair and use dict's update method the update the dictionary. However I really doubt if this has any practical usage since this is definitely more complex than a for loop and introduces extra complexity to the code.
Update values of multiple keys in dictionary d = {"a":0,"b":0,"c":0} d.update({'a': 40, 'b': 41, 'c': 89}) print(d) {'a': 40, 'b': 41, 'c': 89}
If you are just storing integer values, then you can use the Counter class provided by the python module "collections": from collections import Counter d = Counter({"a":0,"b":0,"c":0}) result = d + Counter({"a":10, "b":5}) 'result' will have the value of Counter({'a': 10, 'b': 5}) And since Counter is subclassed from Dict, you have probably do not have to change anything else in your code. >>> isinstance(result, dict) True You do not see the 'c' key in the result because 0-values are not stored in a Counter instance, which saves space. You can check out more about the Counter instance here. Storing other numeric types is supported, with some conditions: "For in-place operations such as c[key] += 1, the value type need only support addition and subtraction. So fractions, floats, and decimals would work and negative values are supported. The same is also true for update() and subtract() which allow negative and zero values for both inputs and outputs." Performing the inverse operation of "+" requires using the method "subtract", which is a note-worthy "gotcha". >>> d = Counter({"a":10, "b":15}) >>> result.subtract(d) >>> c Counter({'a': 0, 'b': 0})
Python reverse dictionary items order
Assume I have a dictionary: d = {3: 'three', 2: 'two', 1: 'one'} I want to rearrange the order of this dictionary so that the dictionary is: d = {1: 'one', 2: 'two', 3: 'three'} I was thinking something like the reverse() function for lists, but that did not work. Thanks in advance for your answers!
Since Python 3.8 and above, the items view is iterable in reverse, so you can just do: d = dict(reversed(d.items())) On 3.7 and 3.6, they hadn't gotten around to implementing __reversed__ on dict and dict views (issue33462: reversible dict), so use an intermediate list or tuple, which do support reversed iteration: d = {3: 'three', 2: 'two', 1: 'one'} d = dict(reversed(list(d.items()))) Pre-3.6, you'd need collections.OrderedDict (both for the input and the output) to achieve the desired result. Plain dicts did not preserve any order until CPython 3.6 (as an implementation detail) and Python 3.7 (as a language guarantee).
Standard Python dictionaries (Before Python 3.6) don't have an order and don't guarantee order. This is exactly what the creation of OrderedDict is for. If your Dictionary was an OrderedDict you could reverse it via: import collections mydict = collections.OrderedDict() mydict['1'] = 'one' mydict['2'] = 'two' mydict['3'] = 'three' collections.OrderedDict(reversed(list(mydict.items())))
Another straightforward solution, which is guaranteed to work for Python v3.7 and over: d = {'A':'a', 'B':'b', 'C':'c', 'D':'d'} dr = {k: d[k] for k in reversed(d)} print(dr) Output: {'D': 'd', 'C': 'c', 'B': 'b', 'A': 'a'} Note that reversed dictionaries are still considered equal to their unreversed originals, i.e.: (d == dr) == True In response to someone upvoting this comment, I was curious to see which solution is actually faster. As usual, it depends. Reversing a 10,000 item dictionary 10,000 times is faster with the solution using list and reversed on the items. But reversing a 1,000,000 item dictionary 100 times (i.e. the same number of items in total reversed dictionaries, just a bigger starting dictionary) is faster with the comprehension - it's left up to the reader to find the exact point where it flips. If you deal with large dictionaries, you may want to benchmark either if performance matters: from random import randint from timeit import timeit def f1(d): return dict(reversed(list(d.items()))) def f2(d): return {k: d[k] for k in reversed(d)} def compare(n): d = {i: randint(1, 100) for i in range(n)} print(timeit(lambda: f1(d), number=100000000 // n)) print(timeit(lambda: f2(d), number=100000000 // n)) compare(10000) compare(1000000) Results (one run, typical results): 4.1554735 4.7047593 8.750093200000002 6.7306311
Python update a key in dict if it doesn't exist
I want to insert a key-value pair into dict if key not in dict.keys(). Basically I could do it with: if key not in d.keys(): d[key] = value But is there a better way? Or what's the pythonic solution to this problem?
You do not need to call d.keys(), so if key not in d: d[key] = value is enough. There is no clearer, more readable method. You could update again with dict.get(), which would return an existing value if the key is already present: d[key] = d.get(key, value) but I strongly recommend against this; this is code golfing, hindering maintenance and readability.
Use dict.setdefault(): >>> d = {'key1': 'one'} >>> d.setdefault('key1', 'some-unused-value') 'one' >>> d # d has not changed because the key already existed {'key1': 'one'} >>> d.setdefault('key2', 'two') 'two' >>> d {'key1': 'one', 'key2': 'two'}
Since Python 3.9 you can use the merge operator | to merge two dictionaries. The dict on the right takes precedence: new_dict = old_dict | { key: val } For example: new_dict = { 'a': 1, 'b': 2 } | { 'b': 42 } print(new_dict) # {'a': 1, 'b': 42} Note: this creates a new dictionary with the updated values.
With the following you can insert multiple values and also have default values but you're creating a new dictionary. d = {**{ key: value }, **default_values} I've tested it with the most voted answer and on average this is faster as it can be seen in the following example, . Speed test comparing a for loop based method with a dict comprehension with unpack operator method. if no copy (d = default_vals.copy()) is made on the first case then the most voted answer would be faster once we reach orders of magnitude of 10**5 and greater. Memory footprint of both methods are the same.
You can also use this solution in only one line of code: dict[dict_key] = dict.get(dict_key,value) The second argument of dict.get is the value you want to assign to the key in case the key does not exist. Since this evaluates before the assignment to dict[dict_key] = , we can be sure that they key will exist when we try to access it.
Python get remaining runoff voting
I am a little stuck on writing a function for a project. This function takes a dictionary of candidates who's values are the number of votes they received. I then have to return a set containing the remaining_candidates. In other words the candidate with the least amount of votes should not be in the set being returned and if for example all of the candidates have the same votes, the set should be empty. I am having trouble getting started here. For example I know I can sort the dictionary like so: x = min(canadites, key=canadites.__getitem__) but that will not work if the candidates have the same value, as it just pops up the last one in the dict. Any ideas? Update: To make things clear. Lets say I have the following dictionary: canadites = {'X':22,'Y':1, 'Z':0} Ideally the function should return a set containing only X and Y. But if Y and Z where both 1 x = min(canadites, key=canadites.__getitem__) seems to only return Z
It's cleaner to create a new dict instead of popping items from the old one: >>> d = {'a':1, 'b':2, 'c':1, 'd':3} >>> min_val = min(d.values()) >>> {k:v for k,v in d.items() if v > min_val} {'b': 2, 'd': 3} In python2, itervalues and iteritems would be more efficient, although this is a micro-optimization in most cases.
Destructuring-bind dictionary contents
I am trying to 'destructure' a dictionary and associate values with variables names after its keys. Something like params = {'a':1,'b':2} a,b = params.values() But since dictionaries are not ordered, there is no guarantee that params.values() will return values in the order of (a, b). Is there a nice way to do this?
from operator import itemgetter params = {'a': 1, 'b': 2} a, b = itemgetter('a', 'b')(params) Instead of elaborate lambda functions or dictionary comprehension, may as well use a built in library.
One way to do this with less repetition than Jochen's suggestion is with a helper function. This gives the flexibility to list your variable names in any order and only destructure a subset of what is in the dict: pluck = lambda dict, *args: (dict[arg] for arg in args) things = {'blah': 'bleh', 'foo': 'bar'} foo, blah = pluck(things, 'foo', 'blah') Also, instead of joaquin's OrderedDict you could sort the keys and get the values. The only catches are you need to specify your variable names in alphabetical order and destructure everything in the dict: sorted_vals = lambda dict: (t[1] for t in sorted(dict.items())) things = {'foo': 'bar', 'blah': 'bleh'} blah, foo = sorted_vals(things)
How come nobody posted the simplest approach? params = {'a':1,'b':2} a, b = params['a'], params['b']
Python is only able to "destructure" sequences, not dictionaries. So, to write what you want, you will have to map the needed entries to a proper sequence. As of myself, the closest match I could find is the (not very sexy): a,b = [d[k] for k in ('a','b')] This works with generators too: a,b = (d[k] for k in ('a','b')) Here is a full example: >>> d = dict(a=1,b=2,c=3) >>> d {'a': 1, 'c': 3, 'b': 2} >>> a, b = [d[k] for k in ('a','b')] >>> a 1 >>> b 2 >>> a, b = (d[k] for k in ('a','b')) >>> a 1 >>> b 2
Here's another way to do it similarly to how a destructuring assignment works in JS: params = {'b': 2, 'a': 1} a, b, rest = (lambda a, b, **rest: (a, b, rest))(**params) What we did was to unpack the params dictionary into key values (using **) (like in Jochen's answer), then we've taken those values in the lambda signature and assigned them according to the key name - and here's a bonus - we also get a dictionary of whatever is not in the lambda's signature so if you had: params = {'b': 2, 'a': 1, 'c': 3} a, b, rest = (lambda a, b, **rest: (a, b, rest))(**params) After the lambda has been applied, the rest variable will now contain: {'c': 3} Useful for omitting unneeded keys from a dictionary. Hope this helps.
Maybe you really want to do something like this? def some_func(a, b): print a,b params = {'a':1,'b':2} some_func(**params) # equiv to some_func(a=1, b=2)
If you are afraid of the issues involved in the use of the locals dictionary and you prefer to follow your original strategy, Ordered Dictionaries from python 2.7 and 3.1 collections.OrderedDicts allows you to recover you dictionary items in the order in which they were first inserted
(Ab)using the import system The from ... import statement lets us desctructure and bind attribute names of an object. Of course, it only works for objects in the sys.modules dictionary, so one could use a hack like this: import sys, types mydict = {'a':1,'b':2} sys.modules["mydict"] = types.SimpleNamespace(**mydict) from mydict import a, b A somewhat more serious hack would be to write a context manager to load and unload the module: with obj_as_module(mydict, "mydict_module"): from mydict_module import a, b By pointing the __getattr__ method of the module directly to the __getitem__ method of the dict, the context manager can also avoid using SimpleNamespace(**mydict). See this answer for an implementation and some extensions of the idea. One can also temporarily replace the entire sys.modules dict with the dict of interest, and do import a, b without from.
Warning 1: as stated in the docs, this is not guaranteed to work on all Python implementations: CPython implementation detail: This function relies on Python stack frame support in the interpreter, which isn’t guaranteed to exist in all implementations of Python. If running in an implementation without Python stack frame support this function returns None. Warning 2: this function does make the code shorter, but it probably contradicts the Python philosophy of being as explicit as you can. Moreover, it doesn't address the issues pointed out by John Christopher Jones in the comments, although you could make a similar function that works with attributes instead of keys. This is just a demonstration that you can do that if you really want to! def destructure(dict_): if not isinstance(dict_, dict): raise TypeError(f"{dict_} is not a dict") # the parent frame will contain the information about # the current line parent_frame = inspect.currentframe().f_back # so we extract that line (by default the code context # only contains the current line) (line,) = inspect.getframeinfo(parent_frame).code_context # "hello, key = destructure(my_dict)" # -> ("hello, key ", "=", " destructure(my_dict)") lvalues, _equals, _rvalue = line.strip().partition("=") # -> ["hello", "key"] keys = [s.strip() for s in lvalues.split(",") if s.strip()] if missing := [key for key in keys if key not in dict_]: raise KeyError(*missing) for key in keys: yield dict_[key] In [5]: my_dict = {"hello": "world", "123": "456", "key": "value"} In [6]: hello, key = destructure(my_dict) In [7]: hello Out[7]: 'world' In [8]: key Out[8]: 'value' This solution allows you to pick some of the keys, not all, like in JavaScript. It's also safe for user-provided dictionaries
With Python 3.10, you can do: d = {"a": 1, "b": 2} match d: case {"a": a, "b": b}: print(f"A is {a} and b is {b}") but it adds two extra levels of indentation, and you still have to repeat the key names.
Look for other answers as this won't cater to the unexpected order in the dictionary. will update this with a correct version sometime soon. try this data = {'a':'Apple', 'b':'Banana','c':'Carrot'} keys = data.keys() a,b,c = [data[k] for k in keys] result: a == 'Apple' b == 'Banana' c == 'Carrot'
Well, if you want these in a class you can always do this: class AttributeDict(dict): def __init__(self, *args, **kwargs): super(AttributeDict, self).__init__(*args, **kwargs) self.__dict__.update(self) d = AttributeDict(a=1, b=2)
Based on #ShawnFumo answer I came up with this: def destruct(dict): return (t[1] for t in sorted(dict.items())) d = {'b': 'Banana', 'c': 'Carrot', 'a': 'Apple' } a, b, c = destruct(d) (Notice the order of items in dict)
An old topic, but I found this to be a useful method: data = {'a':'Apple', 'b':'Banana','c':'Carrot'} for key in data.keys(): locals()[key] = data[key] This method loops over every key in your dictionary and sets a variable to that name and then assigns the value from the associated key to this new variable. Testing: print(a) print(b) print(c) Output Apple Banana Carrot
An easy and simple way to destruct dict in python: params = {"a": 1, "b": 2} a, b = [params[key] for key in ("a", "b")] print(a, b) # Output: # 1 2
I don't know whether it's good style, but locals().update(params) will do the trick. You then have a, b and whatever was in your params dict available as corresponding local variables.
Since dictionaries are guaranteed to keep their insertion order in Python >= 3.7, that means that it's complete safe and idiomatic to just do this nowadays: params = {'a': 1, 'b': 2} a, b = params.values() print(a) print(b) Output: 1 2