Related
I want to merge list of dictionaries in python. The number of dictionaries contained inside the list is not fixed and the nested dictionaries are being merged on both same and different keys. The dictionaries within the list do not contain nested dictionary. The values from same keys can be stored in a list.
My code is:
list_of_dict = [{'a': 1, 'b': 2, 'c': 3}, {'a': 3, 'b': 5}, {'k': 5, 'j': 5}, {'a': 3, 'k': 5, 'd': 4}, {'a': 3} ...... ]
output = {}
for i in list_of_dict:
for k,v in i.items():
if k in output:
output[k].append(v)
else:
output[k] = [v]
Is there a shorter and faster way of implementing this?
I am actually trying to implement the most fast way of doing this because the list of dictionary is very large and then there are lots of rows with such data.
One way using collections.defaultdict:
from collections import defaultdict
res = defaultdict(list)
for d in list_of_dict:
for k, v in d.items():
res[k].append(v)
Output:
defaultdict(list,
{'a': [1, 3, 3, 3],
'b': [2, 5],
'c': [3],
'k': [5, 5],
'j': [5],
'd': [4]})
items() is a dictionary method, but list_of_dict is a list. You need a nested loop so you can loop over the dictionaries and then loop over the items of each dictionary.
ou = {}
for d in list_of_dict:
for key, value in d.items():
output.setdefault(key, []).append(value)
another shorten version can be,
list_of_dict = [{'a': 1, 'b': 2, 'c': 3}, {'a': 3, 'b': 5}, {'k': 5, 'j': 5}, {'a': 3, 'k': 5, 'd': 4}, {'a': 3}]
output = {
k: [d[k] for d in list_of_dict if k in d]
for k in set().union(*list_of_dict)
}
print(output)
{'d': [4], 'k': [5, 5], 'a': [1, 3, 3, 3], 'j': [5], 'c': [3], 'b': [2, 5]}
Python 3.9+ you can use the merge operator for this.
def merge_dicts(dicts):
result = dict()
for _dict in dicts:
result |= _dict
return result
One of the shortest way would be to
prepare a list/set of all the keys from all the dictionaries
and call that key on all the dictionary in the list.
list_of_dict = [{'a': 1, 'b': 2, 'c': 3}, {'a': 3, 'b': 5}, {'k': 5, 'j': 5}, {'a': 3, 'k': 5, 'd': 4}, {'a': 3}]
# prepare a list/set of all the keys from all the dictionaries
# method 1: use sum
all_keys = sum([[a for a in x.keys()] for x in list_of_dict], [])
# method 2: use itertools
import itertools
all_keys = list(itertools.chain.from_iterable(list_of_dict))
# method 3: use union of the set
all_keys = set().union(*list_of_dict)
print(all_keys)
# ['a', 'b', 'c', 'a', 'b', 'k', 'j', 'a', 'k', 'd', 'a']
# convert the list to set to remove duplicates
all_keys = set(all_keys)
print(all_keys)
# {'a', 'k', 'c', 'd', 'b', 'j'}
# now merge the dictionary
merged = {k: [d.get(k) for d in list_of_dict if k in d] for k in all_keys}
print(merged)
# {'a': [1, 3, 3, 3], 'k': [5, 5], 'c': [3], 'd': [4], 'b': [2, 5], 'j': [5]}
In short:
all_keys = set().union(*list_of_dict)
merged = {k: [d.get(k) for d in list_of_dict if k in d] for k in all_keys}
print(merged)
# {'a': [1, 3, 3, 3], 'k': [5, 5], 'c': [3], 'd': [4], 'b': [2, 5], 'j': [5]}
I want to merge list of dictionaries in python. The number of dictionaries contained inside the list is not fixed and the nested dictionaries are being merged on both same and different keys. The dictionaries within the list do not contain nested dictionary. The values from same keys can be stored in a list.
My code is:
list_of_dict = [{'a': 1, 'b': 2, 'c': 3}, {'a': 3, 'b': 5}, {'k': 5, 'j': 5}, {'a': 3, 'k': 5, 'd': 4}, {'a': 3} ...... ]
output = {}
for i in list_of_dict:
for k,v in i.items():
if k in output:
output[k].append(v)
else:
output[k] = [v]
Is there a shorter and faster way of implementing this?
I am actually trying to implement the most fast way of doing this because the list of dictionary is very large and then there are lots of rows with such data.
One way using collections.defaultdict:
from collections import defaultdict
res = defaultdict(list)
for d in list_of_dict:
for k, v in d.items():
res[k].append(v)
Output:
defaultdict(list,
{'a': [1, 3, 3, 3],
'b': [2, 5],
'c': [3],
'k': [5, 5],
'j': [5],
'd': [4]})
items() is a dictionary method, but list_of_dict is a list. You need a nested loop so you can loop over the dictionaries and then loop over the items of each dictionary.
ou = {}
for d in list_of_dict:
for key, value in d.items():
output.setdefault(key, []).append(value)
another shorten version can be,
list_of_dict = [{'a': 1, 'b': 2, 'c': 3}, {'a': 3, 'b': 5}, {'k': 5, 'j': 5}, {'a': 3, 'k': 5, 'd': 4}, {'a': 3}]
output = {
k: [d[k] for d in list_of_dict if k in d]
for k in set().union(*list_of_dict)
}
print(output)
{'d': [4], 'k': [5, 5], 'a': [1, 3, 3, 3], 'j': [5], 'c': [3], 'b': [2, 5]}
Python 3.9+ you can use the merge operator for this.
def merge_dicts(dicts):
result = dict()
for _dict in dicts:
result |= _dict
return result
One of the shortest way would be to
prepare a list/set of all the keys from all the dictionaries
and call that key on all the dictionary in the list.
list_of_dict = [{'a': 1, 'b': 2, 'c': 3}, {'a': 3, 'b': 5}, {'k': 5, 'j': 5}, {'a': 3, 'k': 5, 'd': 4}, {'a': 3}]
# prepare a list/set of all the keys from all the dictionaries
# method 1: use sum
all_keys = sum([[a for a in x.keys()] for x in list_of_dict], [])
# method 2: use itertools
import itertools
all_keys = list(itertools.chain.from_iterable(list_of_dict))
# method 3: use union of the set
all_keys = set().union(*list_of_dict)
print(all_keys)
# ['a', 'b', 'c', 'a', 'b', 'k', 'j', 'a', 'k', 'd', 'a']
# convert the list to set to remove duplicates
all_keys = set(all_keys)
print(all_keys)
# {'a', 'k', 'c', 'd', 'b', 'j'}
# now merge the dictionary
merged = {k: [d.get(k) for d in list_of_dict if k in d] for k in all_keys}
print(merged)
# {'a': [1, 3, 3, 3], 'k': [5, 5], 'c': [3], 'd': [4], 'b': [2, 5], 'j': [5]}
In short:
all_keys = set().union(*list_of_dict)
merged = {k: [d.get(k) for d in list_of_dict if k in d] for k in all_keys}
print(merged)
# {'a': [1, 3, 3, 3], 'k': [5, 5], 'c': [3], 'd': [4], 'b': [2, 5], 'j': [5]}
This question already has answers here:
Return a default value if a dictionary key is not available
(15 answers)
Closed 1 year ago.
I want to take the elements of a list one by one and search them in 4 different dictionaries in python. then I want to create a new dictionary and put the elements of that list as keys and the values I found from those 4 dictionaries as values?
For example:
list = ['a', 'b', 'c', 'd', 'e']
dict1 = {'a': 10, 'b': 2, 'c': 45}
dict2 = {'a': 15, 'b': 55}
dict3 = {'a': 79, 'b': 6, 'c': 3}
dict4 = {'d': 600, 'e': 30}
The result I want:
newlist = {'a': [10, 15, 79, 0],
'b': [2, 55, 6, 0],
'c': [45, 0, 3, 0],
'd': [0, 0, 0, 600],
'e': [0, 0, 0, 30]}
This dict comprehension will result in what you're looking for:
dicts = dict1, dict2, dict3, dict4
{k: [d.get(k, 0) for d in dicts] for k in list1}
Something like this?
from collections import defaultdict
list1 = ['a', 'b', 'c', 'd', 'e']
dict1 = {'a': 10, 'b': 2, 'c': 45}
dict2 = {'a':11, 'b':20, 'z':100}
def collect_values(list1, dictionaries):
result = defaultdict(list)
for key in list1:
for d in dictionaries:
result[key].append(d.get(key, 0))
return result
print(collect_values(list1, [dict1, dict2]))
Which would return
defaultdict(<class 'list'>, {'a': [10, 11], 'b': [2, 20], 'c': [45, 0], 'd': [0, 0], 'e': [0, 0]})
EDIT
If appending zeros each time the key is not found in the dictionary is undesired behavior, the function below can be used:
def collect_values(list1, dictionaries):
result = defaultdict(list)
for key in list1:
for d in dictionaries:
if key in d.keys():
result[key].append(d[key])
elif not result[key]:
result[key].append(0)
return result
Which returns
defaultdict(<class 'list'>, {'a': [10, 11], 'b': [2, 20], 'c': [45], 'd': [0], 'e': [0]})
I have 2 lists like this:
l1 = [{'a': 1, 'b': 2, 'c': 3, 'd': 4}, {'a': 5, 'b': 6, 'c': 7, 'd': 8}]
l2 = [{'a': 5, 'b': 6, 'e': 100}, {'a': 1, 'b': 2, 'e': 101}]
and I want to obtain a list l3, which is a join of l1 and l2 where values of 'a' and 'b' are equal in both l1 and l2
i.e.
l3 = [{'a': 1, 'b: 2, 'c': 3, 'd': 4, 'e': 101}, {'a': 5, 'b: 6, 'c': 7, 'd': 8, 'e': 100}]
How can I do this?
You should accumulate the results in a dictionary. You should use the values of 'a' and 'b' to form a key of this dictionary
Here, I have used a defaultdict to accumulate the entries
l1 = [{'a': 1, 'b': 2, 'c': 3, 'd': 4}, {'a': 5, 'b': 6, 'c': 7, 'd': 8}]
l2 = [{'a': 5, 'b': 6, 'e': 100}, {'a': 1, 'b': 2, 'e': 101}]
from collections import defaultdict
D = defaultdict(dict)
for lst in l1, l2:
for item in lst:
key = item['a'], item['b']
D[key].update(item)
l3 = D.values()
print l3
output:
[{'a': 1, 'c': 3, 'b': 2, 'e': 101, 'd': 4}, {'a': 5, 'c': 7, 'b': 6, 'e': 100, 'd': 8}]
Simple list operations would do the thing for you as well:
l1 = [{'a': 1, 'b': 2, 'c': 3, 'd': 4}, {'a': 5, 'b': 6, 'c': 7, 'd': 8}]
l2 = [{'a': 5, 'b': 6, 'e': 100}, {'a': 1, 'b': 2, 'e': 101}]
l3 = []
for i in range(len(l1)):
for j in range(len(l2)):
if l1[i]['a'] == l2[j]['a'] and l1[i]['b'] == l2[j]['b']:
l3.append(dict(l1[i]))
l3[i].update(l2[j])
My approach is to sort the the combined list by the key, which is keys a + b. After that, for each group of dictionaries with similar key, combine them:
from itertools import groupby
def ab_key(dic):
return dic['a'], dic['b']
def combine_lists_of_dicts(list_of_dic1, list_of_dic2, keyfunc):
for key, dic_of_same_key in groupby(sorted(list_of_dic1 + list_of_dic2, key=keyfunc), keyfunc):
combined_dic = {}
for dic in dic_of_same_key:
combined_dic.update(dic)
yield combined_dic
l1 = [{'a': 1, 'b': 2, 'c': 3, 'd': 4}, {'a': 5, 'b': 6, 'c': 7, 'd': 8}]
l2 = [{'a': 5, 'b': 6, 'e': 100}, {'a': 1, 'b': 2, 'e': 101}]
for dic in combine_lists_of_dicts(l1, l2, ab_key):
print dic
Discussion
The function ab_key returns a tuple of value for key a and b, used for sorting a groupping
The groupby function groups all the dictionaries with similar keys together
This solution is less efficient than that of John La Rooy, but should work fine for small lists
One can achieve a nice and quick solution using pandas.
l1 = [{'a': 1, 'b': 2, 'c': 3, 'd': 4}, {'a': 5, 'b': 6, 'c': 7, 'd': 8}]
l2 = [{'a': 5, 'b': 6, 'e': 100}, {'a': 1, 'b': 2, 'e': 101}]
import pandas as pd
pd.DataFrame(l1).merge(pd.DataFrame(l2), on=['a','b']).to_dict('records')
I have the following dict:
my_dict = {'A': [1, 2], 'B': [1, 4]}
And I want to end up with a list of dicts like this:
[
{'A': 1, 'B': 1},
{'A': 1, 'B': 4},
{'A': 2, 'B': 1},
{'A': 2, 'B': 4}
]
So, I'm after the product of dict's lists, expressed as a list of dicts using the same keys as the incoming dict.
The closest I got was:
my_dict = {'A': [1, 2], 'B': [1, 4]}
it = []
for k in my_dict.keys():
current = my_dict.pop(k)
for i in current:
it.append({k2: i2 for k2, i2 in my_dict.iteritems()})
it[-1].update({k: i})
Which, apart from looking hideous, doesn't give me what I want:
[
{'A': 1, 'B': [1, 4]},
{'A': 2, 'B': [1, 4]},
{'B': 1},
{'B': 4}
]
If anyone feels like solving a riddle, I'd love to see how you'd approach it.
You can use itertools.product for this, i.e calculate cartesian product of the value and then simply zip each of the them with the keys from the dictionary. Note that ordering of a dict's keys() and corresponding values() remains same if it is not modified in-between hence ordering won't be an issue here:
>>> from itertools import product
>>> my_dict = {'A': [1, 2], 'B': [1, 4]}
>>> keys = list(my_dict)
>>> [dict(zip(keys, p)) for p in product(*my_dict.values())]
[{'A': 1, 'B': 1}, {'A': 1, 'B': 4}, {'A': 2, 'B': 1}, {'A': 2, 'B': 4}]
you can use itertools.product function within a list comprehension :
>>> from itertools import product
>>> [dict(i) for i in product(*[[(i,k) for k in j] for i,j in my_dict.items()])]
[{'A': 1, 'B': 1}, {'A': 1, 'B': 4}, {'A': 2, 'B': 1}, {'A': 2, 'B': 4}]
You can get the pairs contain your key and values with the following list comprehension :
[(i,k) for k in j] for i,j in my_dict.items()]
[[('A', 1), ('A', 2)], [('B', 1), ('B', 4)]]
Then you can use product to calculate the product of the preceding lists and then convert them to dictionary with dict function.
With itertools:
>>> from itertools import product
>>> my_dict = {'A': [1, 2], 'B': [1, 4]}
>>> keys, items = zip(*my_dict.items())
>>> [dict(zip(keys, x)) for x in product(*items)]
[{'A': 1, 'B': 1}, {'A': 1, 'B': 4}, {'A': 2, 'B': 1}, {'A': 2, 'B': 4}]
Try this:
from itertools import product
def dict_product(values, first, second):
return [
{first: first_value, second: second_value}
for first_value, second_value in product(values[first], values[second])
]
This is the result:
>>> dict_product({'A': [1, 2], 'B': [1, 4]}, 'A', 'B')
[{'A': 1, 'B': 1}, {'A': 1, 'B': 4}, {'A': 2, 'B': 1}, {'A': 2, 'B': 4}]