This question already has answers here:
How to merge dicts, collecting values from matching keys?
(17 answers)
Closed 5 months ago.
I'm trying to merge three dictionaries, which all have the same keys, and either lists of values, or single values.
one={'a': [1, 2], 'c': [5, 6], 'b': [3, 4]}
two={'a': [2.4, 3.4], 'c': [5.6, 7.6], 'b': [3.5, 4.5]}
three={'a': 1.2, 'c': 3.4, 'b': 2.3}
What I need is for all the items in the values to be added to one list.
result={'a': [1, 2, 2.4, 3.4, 1.2], 'c': [5, 6, 5.6, 7.6, 2.3], 'b': [3, 4, 3.5, 4.5, 3.4]}
I have tried several things, but most put the values into nested lists.
E.g.
out=dict((k, [one[k], two.get(k), three.get(k)]) for k in one)
{'a': [[1, 2], [2.4, 3.4], 1.2], 'c': [[5, 6], [5.6, 7.6], 3.4], 'b': [[3, 4], [3.5, 4.5], 2.3]}
I tried updating it by looping through the values:
out.update((k, [x for x in v]) for k,v in out.iteritems())
but the results was exactly the same.
I have tried to simply add the lists, but because the third dictionary has only a float, I couldn't do it.
check=dict((k, [one[k]+two[k]+three[k]]) for k in one)
So I tried to first add the lists in values of one and two, and then append the value of three. Adding the lists worked well, but then when I tried to append the float from the third dictionary, suddenly the whole value went to 'None'
check=dict((k, [one[k]+two[k]]) for k in one)
{'a': [[1, 2, 2.4, 3.4]], 'c': [[5, 6, 5.6, 7.6]], 'b': [[3, 4, 3.5, 4.5]]}
new=dict((k, v.append(three[k])) for k,v in check.items())
{'a': None, 'c': None, 'b': None}
As a one-liner, with a dictionary comprehension:
new = {key: value + two[key] + [three[key]] for key, value in one.iteritems()}
This creates new lists, concatenating the list from one with the corresponding list from two, putting the single value in three into a temporary list to make concatenating easier.
Or with a for loop updating one in-place:
for key, value in one.iteritems():
value.extend(two[key])
value.append(three[key])
This uses list.extend() to update original list in-place with the list from two, and list.append() to add the single value from three.
Where you went wrong:
your first attempt creates a new list with the values from one, two and three nested within rather than concatenating the existing lists. Your attempt to clean that up just copied those nested lists across.
Your second attempt didn't work because the value in three is not a list so could not be concatenated. I created a new list just for that one value.
Your last attempt should not have used list.append() in a generator expression, because you store the return value of that method, which is always None (its change is stored in v directly and the list doesn't need returning again).
Demo of the first approach:
>>> one={'a': [1, 2], 'c': [5, 6], 'b': [3, 4]}
>>> two={'a': [2.4, 3.4], 'c': [5.6, 7.6], 'b': [3.5, 4.5]}
>>> three={'a': 1.2, 'c': 3.4, 'b': 2.3}
>>> {key: value + two[key] + [three[key]] for key, value in one.iteritems()}
{'a': [1, 2, 2.4, 3.4, 1.2], 'c': [5, 6, 5.6, 7.6, 3.4], 'b': [3, 4, 3.5, 4.5, 2.3]}
Arbitrary dictionary number and keys
The issues with your attempt are covered by #MartijnPieters' solution.
For a generalised solution, consider using itertools.chain to chain multiple dictionaries. You can also use a defaultdict for the more general case where you do not find the same keys in each dictionary.
from collections import defaultdict
from itertools import chain
from operator import methodcaller
# dictionaries with non-equal keys, values all lists for simplicity
one = {'a': [1, 2], 'c': [5, 6], 'b': [3, 4], 'e': [6.2]}
two = {'a': [2.4, 3.4], 'c': [5.6, 7.6], 'b': [3.5, 4.5], 'f': [1.3]}
three = {'a': [1.2], 'c': [3.4], 'b': [2.3], 'e': [3.1]}
# initialise defaultdict of lists
dd = defaultdict(list)
# iterate dictionary items
dict_items = map(methodcaller('items'), (one, two, three))
for k, v in chain.from_iterable(dict_items):
dd[k].extend(v)
print(dd)
# defaultdict(list,
# {'a': [1, 2, 2.4, 3.4, 1.2],
# 'b': [3, 4, 3.5, 4.5, 2.3],
# 'c': [5, 6, 5.6, 7.6, 3.4],
# 'e': [6.2, 3.1],
# 'f': [1.3]})
Note defaultdict is a subclass of dict so there's generally no need to convert the result to a regular dict.
A robust solution. =)
def FullMergeDict(D1, D2):
for key, value in D1.items():
if key in D2:
if type(value) is dict:
FullMergeDict(D1[key], D2[key])
else:
if type(value) in (int, float, str):
D1[key] = [value]
if type(D2[key]) is list:
D1[key].extend(D2[key])
else:
D1[key].append(D2[key])
for key, value in D2.items():
if key not in D1:
D1[key] = value
if __name__ == '__main__':
X = {
'a': 'aaa',
'c': [1,3,5,7],
'd': 100,
'e': {'k': 1, 'p': 'aa','t': [-1,-2]},
'f': {'j':1}
}
Y = {
'b': 'bbb',
'd': 200,
'e': {'k': 2, 'p': 'bb','o': [-4]},
'c': [2,4,6],
'g': {'v':2}
}
FullMergeDict(X, Y)
exit(0)
Result:
X = {
'a': 'aaa',
'b': 'bbb',
'c': [1, 3, 5, 7, 2, 4, 6],
'd': [100, 200],
'e': {'k': [1, 2], 'o': [-4], 'p': ['aa', 'bb'], 't': [-1, -2]},
'f': {'j': 1},
'g': {'v': 2}}
If you have different keys and different types of values in dictionaries you can use the following approach:
from collections import defaultdict, Iterable
dct1 = {'a': [1, 2]}
dct2 = {'a': [3], 'b': [5, 6]}
dct3 = {'a': 4, 'c': 7}
result = defaultdict(list)
for dct in [dct1, dct2, dct3]:
for k, v in dct.items():
if isinstance(v, Iterable):
result[k].extend(v)
else:
result[k].append(v)
print(result)
# defaultdict(<class 'list'>, {'a': [1, 2, 3, 4], 'b': [5, 6], 'c': [7]})
One line solution (also handles for keys present in only one dict):
{ key:one.get(key,[])+two.get(key,[]) for key in set(list(one.keys())+list(two.keys())) }
Example 1:
one = {'a': [1, 2], 'c': [5, 6], 'b': [3, 4]}
two = {'a': [2.4, 3.4], 'c': [5.6, 7.6], 'd': [3.5, 4.5]}
{ key:one.get(key,[])+two.get(key,[]) for key in set(list(one.keys())+list(two.keys())) }
Output:
{'a': [1, 2, 2.4, 3.4], 'b': [3, 4], 'c': [5, 6, 5.6, 7.6], 'd': [3.5,
4.5]}
Example 2:
x={1:['a','b','c']}
y={1:['d','e','f'],2:['g']}
{ key:x.get(key,[])+y.get(key,[]) for key in set(list(x.keys())+list(y.keys())) }
Output:
{1: ['a', 'b', 'c', 'd', 'e', 'f'], 2: ['g']}
See this help or not:
>>> dic={}
>>> k=[]
>>> for i in 'abc':
k=one[i]+two[i]
k.append(three[i])
dic[i]=k
>>> dic
{'c': [5, 6, 5.6, 7.6, 3.4], 'a': [1, 2, 2.4, 3.4, 1.2], 'b': [3, 4, 3.5, 4.5, 2.3]}
Related
I want to merge list of dictionaries in python. The number of dictionaries contained inside the list is not fixed and the nested dictionaries are being merged on both same and different keys. The dictionaries within the list do not contain nested dictionary. The values from same keys can be stored in a list.
My code is:
list_of_dict = [{'a': 1, 'b': 2, 'c': 3}, {'a': 3, 'b': 5}, {'k': 5, 'j': 5}, {'a': 3, 'k': 5, 'd': 4}, {'a': 3} ...... ]
output = {}
for i in list_of_dict:
for k,v in i.items():
if k in output:
output[k].append(v)
else:
output[k] = [v]
Is there a shorter and faster way of implementing this?
I am actually trying to implement the most fast way of doing this because the list of dictionary is very large and then there are lots of rows with such data.
One way using collections.defaultdict:
from collections import defaultdict
res = defaultdict(list)
for d in list_of_dict:
for k, v in d.items():
res[k].append(v)
Output:
defaultdict(list,
{'a': [1, 3, 3, 3],
'b': [2, 5],
'c': [3],
'k': [5, 5],
'j': [5],
'd': [4]})
items() is a dictionary method, but list_of_dict is a list. You need a nested loop so you can loop over the dictionaries and then loop over the items of each dictionary.
ou = {}
for d in list_of_dict:
for key, value in d.items():
output.setdefault(key, []).append(value)
another shorten version can be,
list_of_dict = [{'a': 1, 'b': 2, 'c': 3}, {'a': 3, 'b': 5}, {'k': 5, 'j': 5}, {'a': 3, 'k': 5, 'd': 4}, {'a': 3}]
output = {
k: [d[k] for d in list_of_dict if k in d]
for k in set().union(*list_of_dict)
}
print(output)
{'d': [4], 'k': [5, 5], 'a': [1, 3, 3, 3], 'j': [5], 'c': [3], 'b': [2, 5]}
Python 3.9+ you can use the merge operator for this.
def merge_dicts(dicts):
result = dict()
for _dict in dicts:
result |= _dict
return result
One of the shortest way would be to
prepare a list/set of all the keys from all the dictionaries
and call that key on all the dictionary in the list.
list_of_dict = [{'a': 1, 'b': 2, 'c': 3}, {'a': 3, 'b': 5}, {'k': 5, 'j': 5}, {'a': 3, 'k': 5, 'd': 4}, {'a': 3}]
# prepare a list/set of all the keys from all the dictionaries
# method 1: use sum
all_keys = sum([[a for a in x.keys()] for x in list_of_dict], [])
# method 2: use itertools
import itertools
all_keys = list(itertools.chain.from_iterable(list_of_dict))
# method 3: use union of the set
all_keys = set().union(*list_of_dict)
print(all_keys)
# ['a', 'b', 'c', 'a', 'b', 'k', 'j', 'a', 'k', 'd', 'a']
# convert the list to set to remove duplicates
all_keys = set(all_keys)
print(all_keys)
# {'a', 'k', 'c', 'd', 'b', 'j'}
# now merge the dictionary
merged = {k: [d.get(k) for d in list_of_dict if k in d] for k in all_keys}
print(merged)
# {'a': [1, 3, 3, 3], 'k': [5, 5], 'c': [3], 'd': [4], 'b': [2, 5], 'j': [5]}
In short:
all_keys = set().union(*list_of_dict)
merged = {k: [d.get(k) for d in list_of_dict if k in d] for k in all_keys}
print(merged)
# {'a': [1, 3, 3, 3], 'k': [5, 5], 'c': [3], 'd': [4], 'b': [2, 5], 'j': [5]}
I want to change the value of a key inside a 2D dictionary while looping, but the program is behaving in a way I did not expect.
So first I initiated my 2D dictionary:
dict1 = dict()
dict2 = dict()
list = ['a', 'b', 'c']
list2 = ['A', 'B', 'C']
for i in list2:
dict1[i] = []
for i in list:
dict2[i] = dict1
Now I created a nested loop to change the key by appending a value to a list:
count = 0
for i in range(3):
for j in range(3):
dict2[list[i]][list2[j]].append(count)
count += 1
print(dict2)
The outcome is as follows:
{'a': {'A': [0, 3, 6], 'B': [1, 4, 7], 'C': [2, 5, 8]}, 'b': {'A': [0, 3, 6], 'B': [1, 4, 7], 'C': [2, 5, 8]}, 'c': {'A': [0, 3, 6], 'B': [1, 4, 7], 'C': [2, 5, 8]}}
whereas I expected to see this:
{'a': {'A': [0], 'B': [1], 'C': [2]}, 'b': {'A': [3], 'B': [4], 'C': [5]}, 'c': {'A': [6], 'B': [7], 'C': [8]}}
Why is the code behaving in this way and what can I change to get the outcome that I'm looking for?
Thanks!
What #darrylg says in the comment is correct. You can also combine a number of the for-loops you are using, e.g.:
keys1 = ['a', 'b', 'c']
keys2 = ['A', 'B', 'C']
dict2 = {k1: {k2: [] for k2 in keys2} for k1 in keys1}
for count, (i, j) in enumerate((i, j) for i in range(3) for j in range(3)):
dict2[keys1[i]][keys2[j]].append(count)
or, maybe better:
for count, (k1, k2) in enumerate((k1, k2) for k1 in keys1 for k2 in keys2):
dict2[k1][k2].append(count)
or perhaps, using itertools.product:
import itertools
for count, (k1, k2) in enumerate(itertools.product(keys1, keys2)):
dict2[k1][k2].append(count)
you could write it as a one-liner, although I'd strongly suggest not doing this :-) :
dict2 = (count := -1) and {k1: {k2: [count := count+1] for k2 in 'ABC'} for k1 in 'abc'}
which given:
import pprint
pprint.pprint(dict2)
will print:
{'a': {'A': [0], 'B': [1], 'C': [2]},
'b': {'A': [3], 'B': [4], 'C': [5]},
'c': {'A': [6], 'B': [7], 'C': [8]}}
I want to merge list of dictionaries in python. The number of dictionaries contained inside the list is not fixed and the nested dictionaries are being merged on both same and different keys. The dictionaries within the list do not contain nested dictionary. The values from same keys can be stored in a list.
My code is:
list_of_dict = [{'a': 1, 'b': 2, 'c': 3}, {'a': 3, 'b': 5}, {'k': 5, 'j': 5}, {'a': 3, 'k': 5, 'd': 4}, {'a': 3} ...... ]
output = {}
for i in list_of_dict:
for k,v in i.items():
if k in output:
output[k].append(v)
else:
output[k] = [v]
Is there a shorter and faster way of implementing this?
I am actually trying to implement the most fast way of doing this because the list of dictionary is very large and then there are lots of rows with such data.
One way using collections.defaultdict:
from collections import defaultdict
res = defaultdict(list)
for d in list_of_dict:
for k, v in d.items():
res[k].append(v)
Output:
defaultdict(list,
{'a': [1, 3, 3, 3],
'b': [2, 5],
'c': [3],
'k': [5, 5],
'j': [5],
'd': [4]})
items() is a dictionary method, but list_of_dict is a list. You need a nested loop so you can loop over the dictionaries and then loop over the items of each dictionary.
ou = {}
for d in list_of_dict:
for key, value in d.items():
output.setdefault(key, []).append(value)
another shorten version can be,
list_of_dict = [{'a': 1, 'b': 2, 'c': 3}, {'a': 3, 'b': 5}, {'k': 5, 'j': 5}, {'a': 3, 'k': 5, 'd': 4}, {'a': 3}]
output = {
k: [d[k] for d in list_of_dict if k in d]
for k in set().union(*list_of_dict)
}
print(output)
{'d': [4], 'k': [5, 5], 'a': [1, 3, 3, 3], 'j': [5], 'c': [3], 'b': [2, 5]}
Python 3.9+ you can use the merge operator for this.
def merge_dicts(dicts):
result = dict()
for _dict in dicts:
result |= _dict
return result
One of the shortest way would be to
prepare a list/set of all the keys from all the dictionaries
and call that key on all the dictionary in the list.
list_of_dict = [{'a': 1, 'b': 2, 'c': 3}, {'a': 3, 'b': 5}, {'k': 5, 'j': 5}, {'a': 3, 'k': 5, 'd': 4}, {'a': 3}]
# prepare a list/set of all the keys from all the dictionaries
# method 1: use sum
all_keys = sum([[a for a in x.keys()] for x in list_of_dict], [])
# method 2: use itertools
import itertools
all_keys = list(itertools.chain.from_iterable(list_of_dict))
# method 3: use union of the set
all_keys = set().union(*list_of_dict)
print(all_keys)
# ['a', 'b', 'c', 'a', 'b', 'k', 'j', 'a', 'k', 'd', 'a']
# convert the list to set to remove duplicates
all_keys = set(all_keys)
print(all_keys)
# {'a', 'k', 'c', 'd', 'b', 'j'}
# now merge the dictionary
merged = {k: [d.get(k) for d in list_of_dict if k in d] for k in all_keys}
print(merged)
# {'a': [1, 3, 3, 3], 'k': [5, 5], 'c': [3], 'd': [4], 'b': [2, 5], 'j': [5]}
In short:
all_keys = set().union(*list_of_dict)
merged = {k: [d.get(k) for d in list_of_dict if k in d] for k in all_keys}
print(merged)
# {'a': [1, 3, 3, 3], 'k': [5, 5], 'c': [3], 'd': [4], 'b': [2, 5], 'j': [5]}
This question already has answers here:
Return a default value if a dictionary key is not available
(15 answers)
Closed 1 year ago.
I want to take the elements of a list one by one and search them in 4 different dictionaries in python. then I want to create a new dictionary and put the elements of that list as keys and the values I found from those 4 dictionaries as values?
For example:
list = ['a', 'b', 'c', 'd', 'e']
dict1 = {'a': 10, 'b': 2, 'c': 45}
dict2 = {'a': 15, 'b': 55}
dict3 = {'a': 79, 'b': 6, 'c': 3}
dict4 = {'d': 600, 'e': 30}
The result I want:
newlist = {'a': [10, 15, 79, 0],
'b': [2, 55, 6, 0],
'c': [45, 0, 3, 0],
'd': [0, 0, 0, 600],
'e': [0, 0, 0, 30]}
This dict comprehension will result in what you're looking for:
dicts = dict1, dict2, dict3, dict4
{k: [d.get(k, 0) for d in dicts] for k in list1}
Something like this?
from collections import defaultdict
list1 = ['a', 'b', 'c', 'd', 'e']
dict1 = {'a': 10, 'b': 2, 'c': 45}
dict2 = {'a':11, 'b':20, 'z':100}
def collect_values(list1, dictionaries):
result = defaultdict(list)
for key in list1:
for d in dictionaries:
result[key].append(d.get(key, 0))
return result
print(collect_values(list1, [dict1, dict2]))
Which would return
defaultdict(<class 'list'>, {'a': [10, 11], 'b': [2, 20], 'c': [45, 0], 'd': [0, 0], 'e': [0, 0]})
EDIT
If appending zeros each time the key is not found in the dictionary is undesired behavior, the function below can be used:
def collect_values(list1, dictionaries):
result = defaultdict(list)
for key in list1:
for d in dictionaries:
if key in d.keys():
result[key].append(d[key])
elif not result[key]:
result[key].append(0)
return result
Which returns
defaultdict(<class 'list'>, {'a': [10, 11], 'b': [2, 20], 'c': [45], 'd': [0], 'e': [0]})
This question already has answers here:
How do I initialize a dictionary of empty lists in Python?
(7 answers)
Closed last year.
I'm sure i'm doing something silly, but I'm trying to take a list of dictionaries and make a dictionary where one of the key:value pairs in the dictionary becomes a key and the remaining key:value pairs become part of the value where the value is a list of dictionaries. An example is here:
In [115]: aa = ['a', 'b', 'c', 'd']
In [116]: my_dict = dict.fromkeys(key_list, [])
In [143]: my_dict
Out[143]: {'a': [], 'b': [], 'c': [], 'd': []}
In [117]: list_of_dicts = [{'letter':'a', 'val1':2, 'val2':3}, {'letter':'b', 'val1':4, 'val2':5}, {'letter':'c', 'val1':6, 'val2':7}]
In [118]: %paste
for entry in list_of_dicts:
remainder = {k:v for k, v in entry.iteritems() if k!='letter'}
my_dict[entry['letter']].append(remainder)
## -- End pasted text --
In [119]: my_dict
Out[119]:
{'a': [{'val1': 2, 'val2': 3}, {'val1': 4, 'val2': 5}, {'val1': 6, 'val2': 7}],
'b': [{'val1': 2, 'val2': 3}, {'val1': 4, 'val2': 5}, {'val1': 6, 'val2': 7}],
'c': [{'val1': 2, 'val2': 3}, {'val1': 4, 'val2': 5}, {'val1': 6, 'val2': 7}],
'd': [{'val1': 2, 'val2': 3}, {'val1': 4, 'val2': 5}, {'val1': 6, 'val2': 7}]}
Why are my values being applied to every entry in the dictionary?
Why isnt' the final dictionary:
In [121]: final_dict
Out[121]:
{'a': [{'val1': 2, 'val2': 3}],
'b': [{'val1': 4, 'val2': 5}],
'c': [{'val1': 6, 'val2': 7}]}
my_dict = dict.fromkeys(key_list, []) uses the same list for all the keys
You can use
my_dict = {k: [] for k in key_list}
to get a new list for each key
alternatively, you can use collections.defaultdict
my_dict = defaultdict(list)
which calls list() to make a new list every time you insert a new key