I'm trying to split a list of dictionaries into two lists based on a conditional being the type of the value in the dictionaries. Is there anyway to do this in a one pass list comprehension?
Currently, I am doing this:
nonvals = [{k: v for k, v in act.items() if type(v) != int} for act in actual]
vals = [{k: v for k, v in act.items() if type(v) == int} for act in actual]
Is there anyway I can return two lists based on the conditional in one list comprehension? If not what's a more pythonic way to split this list of dictionaries?
I am attempting to do the following as a final solution, but I feel like there is a lot of code repetition and extraneous looping that can be avoided - for reference both actual and expected are lists of dictionaries. I would just like to get to the final solution in the fewest lines as possible.
I want to check if the non-int key, value pairs in expected are in the non-int key, value pairs in actual. And in actual I just want to check if all the int key, value pairs are in the [-11, 11] range.
expected = [{'time': '12:34:22', 'place': 'LA', 'person': 'Mike', 'val1': 2, 'val2': 3, 'val3': 4},
{'time': '11:45:15', 'place': 'SF', 'person': 'Emily', 'val1': 2, 'val2': 3, 'val3': 4}]
actual = [{'time': '12:34:22', 'place': 'LA', 'person': 'Mike', 'val1': 2, 'val2': 3, 'val3': 4},
{'time': '11:45:15', 'place': 'SF', 'person': 'Emily', 'val1': 2, 'val2': 3, 'val3': 4},
{'time': '21:19:57', 'place': 'LA', 'person': 'Leo', 'val1': 2, 'val2': 3, 'val3': 4},
{'time': '15:43:11', 'place': 'LA', 'person': 'Marge', 'val1': 2, 'val2': 3, 'val3': 4}]
def check(expected, actual):
nonvals = [{k: v for k, v in act.items() if type(v) != int} for act in actual]
vals = [{k: v for k, v in act.items() if type(v) == int} for act in actual]
for act in actual:
for k, v in act.items():
if v in vals and v not in range(-11, 11):
return False
for exp in expected:
if {k: v for k, v in exp.items() if type(v) != int} not in nonvals:
return False
return True
There is no general pythonic solution that will split a list based on some condition, let alone do the even more complicated job of splitting a dictionary based on some condition (like the type of the value matching a specific type).
Doing what you're doing is fairly readable and not a bad solution, but if you run into this problem more often, you could just write a function you can then apply to the dictionaries in the list:
def pdict(d, condition):
"""
Partition a dictionary based on some condition function
:param d: a dict
:param condition: a function with parameters k, v returning a bool for k: v in d
:return: two dictionaries, with the contents of d, split according to condition
"""
return {
k: v for k, v in d.items() if condition(k, v)
}, {
k: v for k, v in d.items() if not condition(k, v)
}
original = [{'a': 1, 'b': 'two', 'c': 3}, {'a': 'one', 'b': 2}, {'a': 1, 'b': 2}]
int_result, nonint_result = [
list(t) for t in zip(*[
pdict(d, lambda k, v: isinstance(v, int)) for d in original
])
]
print(int_result)
print(nonint_result)
This is clean and allows you to simply reuse partition_dict for similar cases.
The output produced by the example:
[{'a': 1, 'c': 3}, {'b': 2}, {'a': 1, 'b': 2}]
[{'b': 'two'}, {'a': 'one'}, {}]
A more simple example of (re)using pdict():
d = {1: 42, 2: 33, 3: 5, 4: 10}
odd, even = pdict(d, lambda k, v: v % 2 == 1)
print(odd, even)
Output:
{2: 33, 3: 5} {1: 42, 4: 10}
A rewrite of pdict() that only loops once (as suggested by #blues), but is a bit wordier:
def pdict(d, condition):
"""
Partition a dictionary based on some condition function
:param d: a dict
:param condition: a function with parameters k, v returning a bool for k: v in d
:return: two dictionaries, with the contents of d, split according to condition
"""
meets_condition = {}
does_not_meet_condition = {}
for k, v in d.items():
if condition(k, v):
meets_condition[k] = v
else:
does_not_meet_condition[k] = v
return meets_condition, does_not_meet_condition
A disadvantage of this may be that for each element being added to the dictionaries, there may be some overhead in look-ups that the original example code can avoid in the dictionary comprehension. You could run performance tests to decide what is preferable, but the latter solution does avoid looping through the entire original dictionary twice. I think I'd prefer the first solution for anything that's not crazy large.
I have:
myDict = {'a': [1,2,3], 'b':[4,5,6], 'c':[7,8,9]}
I want:
myDict = {'a': set([1,2,3]), 'b':set([4,5,6]), 'c':set([7,8,9])}
Is there a one-liner I can use to do this rather than looping through it and converting the type of the values?
You'll have to loop anyway:
{key: set(value) for key, value in yourData.items()}
If you're using Python 3.6+, you can also do this:
dict(zip(myDict.keys(), map(set, myDict.values())))
This can be done with map by mapping the values to type set
myDict = dict(map(lambda x: (x[0], set(x[1])), myDict.items()))
Or with either version of dictionary comprehension as well
myDict = {k: set(v) for k, v in myDict.items()}
myDict = {k: set(myDict[k]) for k in myDict}
You can use comprehension for it:
Basically, loop through the key-value pairs and create set out of each value for the corresponding key.
>>> myDict = {'a': [1,2,3], 'b':[4,5,6], 'c':[7,8,9]}
>>> myDict = {k: set(v) for k, v in myDict.items()}
>>> myDict
{'a': {1, 2, 3}, 'b': {4, 5, 6}, 'c': {8, 9, 7}}
You can't do it without looping anyway, but you can have the looping done in one line, with the following code:
myDict = {k:set(v) for k, v in myDict.items()}
This is basically traversing each item in your dictionary and converting the lists to sets and combining the key(str):value(set) pairs to a new dictionary and assigning it back to myDict variable.
I am new at python and here. I need to create N new dictionaries based on N keys from an original dictionary.
Lets say I have an OriginalDict {a:1, b:2, c:3, d:4, e:5, f:6} I need to create 6 new dictionaries having as keys (new keys are same) the value of the original. Something like that:
Dict1 {Name:a,...}, Dict2 {Name:b,...}, Dict3 {Name:c,...}.....
Dict6 {Name:f...}
This is my code:
d = {}
for key in OriginalDict:
d['Name'] = key
I got a new dictionary but only for the last key.
print d
{Name:f}
I guess cos' last value in a dictionary overrides the previous one if the keys are the same
please advise... :)
If, for example, we put all those dicts in a list, we can use this comprehensions
dicts = [{'Name': k} for k in OriginalDict]
Let's try it out
>>> OriginalDict = {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6}
>>> dicts = [{'Name': k} for k in OriginalDict]
>>> dicts
[{'Name': 'd'}, {'Name': 'c'}, {'Name': 'a'}, {'Name': 'b'}, {'Name': 'e'}, {'Name': 'f'}]
The statement 6 new dictionaries having as keys (new keys are same) the value of the original seems to contradict your example, at least to me.
In such case we can do
dicts = [{v: k} for k, v in OriginalDict.items()]
Let's try it out:
>>> dicts = [{v: k} for k, v in OriginalDict.items()]
>>> dicts
[{4: 'd'}, {3: 'c'}, {1: 'a'}, {2: 'b'}, {5: 'e'}, {6: 'f'}]
In python 3.x:
for key in OriginalDict.keys():
d = {}
d ['Name'] = key
This will give you a new Dictionary for every key of the Original one. Now, you could save them inside a list or inside another dictionary like:
New_Dicts = []
for key in OriginalDict.keys():
d = {}
d ['Name'] = key
New_Dicts.append(d)
Or,
New_Dicts = {}
for i,key in enumerate(OriginalDict.keys()):
d = {}
d ['Name'] = key
New_Dicts[i] = d
I think you want to create a function which is a generator, then call that function passing in the dictionary and then yielding your new ones:
orig = {'a':1, 'b':2, 'c':3, 'd':4, 'e':5, 'f':6}
def make_sub_dictionaries(big_dictionary):
for key, val in big_dictionary.iteritems():
yield {'name': key, 'val': val }
# if you want them one by one, call next
first = make_sub_dictionaries(orig).next()
print first
# if you want all of them
for d in make_sub_dictionaries(orig):
print str(d)
# do whatever stuff you need to do
One method is as follows:
OriginalDict = {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6}
newDicts = [{'Name':v} for k,v in enumerate(OriginalDict.values())]
which will give you
>>> newDicts
[{'Name': 1}, {'Name': 3}, {'Name': 2}, {'Name': 5}, {'Name': 4}, {'Name': 6}]
First of all, this won't keep the order. You need to use OriginalDict to preserve the order.
my_dict={'a':1, 'b':2, 'c':3, 'd':4, 'e':5, 'f':6}
j=0
for key, value in my_dict.iteritems():
j +=1
exec("Dict%s = %s" % (j,{key:value}))
Now when you type.
>>> print Dict1
{'a':1}
>>> print Dict2
{'c':3}
This question already has answers here:
How to merge dicts, collecting values from matching keys?
(17 answers)
Closed 3 months ago.
I have to merge list of python dictionary. For eg:
dicts[0] = {'a':1, 'b':2, 'c':3}
dicts[1] = {'a':1, 'd':2, 'c':'foo'}
dicts[2] = {'e':57,'c':3}
super_dict = {'a':[1], 'b':[2], 'c':[3,'foo'], 'd':[2], 'e':[57]}
I wrote the following code:
super_dict = {}
for d in dicts:
for k, v in d.items():
if super_dict.get(k) is None:
super_dict[k] = []
if v not in super_dict.get(k):
super_dict[k].append(v)
Can it be presented more elegantly / optimized?
Note
I found another question on SO but its about merging exactly 2 dictionaries.
You can iterate over the dictionaries directly -- no need to use range. The setdefault method of dict looks up a key, and returns the value if found. If not found, it returns a default, and also assigns that default to the key.
super_dict = {}
for d in dicts:
for k, v in d.iteritems(): # d.items() in Python 3+
super_dict.setdefault(k, []).append(v)
Also, you might consider using a defaultdict. This just automates setdefault by calling a function to return a default value when a key isn't found.
import collections
super_dict = collections.defaultdict(list)
for d in dicts:
for k, v in d.iteritems(): # d.items() in Python 3+
super_dict[k].append(v)
Also, as Sven Marnach astutely observed, you seem to want no duplication of values in your lists. In that case, set gets you what you want:
import collections
super_dict = collections.defaultdict(set)
for d in dicts:
for k, v in d.iteritems(): # d.items() in Python 3+
super_dict[k].add(v)
from collections import defaultdict
dicts = [{'a':1, 'b':2, 'c':3},
{'a':1, 'd':2, 'c':'foo'},
{'e':57, 'c':3} ]
super_dict = defaultdict(set) # uses set to avoid duplicates
for d in dicts:
for k, v in d.items(): # use d.iteritems() in python 2
super_dict[k].add(v)
you can use this behaviour of dict. (a bit elegant)
a = {'a':1, 'b':2, 'c':3}
b = {'d':1, 'e':2, 'f':3}
c = {1:1, 2:2, 3:3}
merge = {**a, **b, **c}
print(merge) # {'a': 1, 'b': 2, 'c': 3, 'd': 1, 'e': 2, 'f': 3, 1: 1, 2: 2, 3: 3}
and you are good to go :)
Merge the keys of all dicts, and for each key assemble the list of values:
super_dict = {}
for k in set(k for d in dicts for k in d):
super_dict[k] = [d[k] for d in dicts if k in d]
The expression set(k for d in dicts for k in d) builds a set of all unique keys of all dictionaries. For each of these unique keys, we use the list comprehension [d[k] for d in dicts if k in d] to build the list of values from all dicts for this key.
Since you only seem to one the unique value of each key, you might want to use sets instead:
super_dict = {}
for k in set(k for d in dicts for k in d):
super_dict[k] = set(d[k] for d in dicts if k in d)
It seems like most of the answers using comprehensions are not all that readable. In case any gets lost in the mess of answers above this might be helpful (although extremely late...). Just loop over the items of each dict and place them in a separate one.
super_dict = {key:val for d in dicts for key,val in d.items()}
When the value of the keys are in list:
from collections import defaultdict
dicts = [{'a':[1], 'b':[2], 'c':[3]},
{'a':[11], 'd':[2], 'c':['foo']},
{'e':[57], 'c':[3], "a": [1]} ]
super_dict = defaultdict(list) # uses set to avoid duplicates
for d in dicts:
for k, v in d.items(): # use d.iteritems() in python 2
super_dict[k] = list(set(super_dict[k] + v))
combined_dict = {}
for elem in super_dict.keys():
combined_dict[elem] = super_dict[elem]
combined_dict
## output: {'a': [1, 11], 'b': [2], 'c': [3, 'foo'], 'd': [2], 'e': [57]}
I have a very easy to go solution without any imports.
I use the dict.update() method.
But sadly it will overwrite, if same key appears in more than one dictionary, then the most recently merged dict's value will appear in the output.
dict1 = {'Name': 'Zara', 'Age': 7}
dict2 = {'Sex': 'female' }
dict3 = {'Status': 'single', 'Age': 27}
dict4 = {'Occupation':'nurse', 'Wage': 3000}
def mergedict(*args):
output = {}
for arg in args:
output.update(arg)
return output
print(mergedict(dict1, dict2, dict3, dict4))
The output is this:
{'Name': 'Zara', 'Age': 27, 'Sex': 'female', 'Status': 'single', 'Occupation': 'nurse', 'Wage': 3000}
Perhaps a more modern and concise approach for those who use python 3.3 or later versions is the use of ChainMap from the collections module.
from collections import ChainMap
d1 = {'a': 1, 'b': 3}
d2 = {'c': 2}
d3 = {'d': 7, 'a': 9}
d4 = {}
combo = dict(ChainMap(d1, d2, d3, d4))
# {'d': 7, 'a': 1, 'c': 2, 'b': 3}
For a larger collection of dict objects then star operator works
dict(ChainMap(*dict_collection))
Note that the resulting dictionary seems to only keep the value of the first key it encounters in the ordered collection and ignores any further duplicates.
This may be a bit more elegant:
super_dict = {}
for d in dicts:
for k, v in d.iteritems():
l=super_dict.setdefault(k,[])
if v not in l:
l.append(v)
UPDATE: made change suggested by Sven
UPDATE: changed to avoid duplicates (thanks Marcin and Steven)
Never forget that the standard libraries have a wealth of tools for dealing with dicts and iteration:
from itertools import chain
from collections import defaultdict
super_dict = defaultdict(list)
for k,v in chain.from_iterable(d.iteritems() for d in dicts):
if v not in super_dict[k]: super_dict[k].append(v)
Note that the if v not in super_dict[k] can be avoided by using defaultdict(set) as per Steven Rumbalski's answer.
If you assume that the keys in which you are interested are at the same nested level, you can recursively traverse each dictionary and create a new dictionary using that key, effectively merging them.
merged = {}
for d in dicts:
def walk(d,merge):
for key, item in d.items():
if isinstance(item, dict):
merge.setdefault(key, {})
walk(item, merge[key])
else:
merge.setdefault(key, [])
merge[key].append(item)
walk(d,merged)
For example, say you have the following dictionaries you want to merge.
dicts = [{'A': {'A1': {'FOO': [1,2,3]}}},
{'A': {'A1': {'A2': {'BOO': [4,5,6]}}}},
{'A': {'A1': {'FOO': [7,8]}}},
{'B': {'B1': {'COO': [9]}}},
{'B': {'B2': {'DOO': [10,11,12]}}},
{'C': {'C1': {'C2': {'POO':[13,14,15]}}}},
{'C': {'C1': {'ROO': [16,17]}}}]
Using the key at each level, you should get something like this:
{'A': {'A1': {'FOO': [[1, 2, 3], [7, 8]],
'A2': {'BOO': [[4, 5, 6]]}}},
'B': {'B1': {'COO': [[9]]},
'B2': {'DOO': [[10, 11, 12]]}},
'C': {'C1': {'C2': {'POO': [[13, 14, 15]]},
'ROO': [[16, 17]]}}}
Note: I assume the leaf at each branch is a list of some kind, but you can obviously change the logic to do whatever is necessary for your situation.
This is a more recent enhancement over the prior answer by ElbowPipe, using newer syntax introduced in Python 3.9 for merging dictionaries. Note that this answer does not merge conflicting values into a list!
> import functools
> import operator
> functools.reduce(operator.or_, [{0:1}, {2:3, 4:5}, {2:6}])
{0: 1, 2: 6, 4: 5}
For a oneliner, the following could be used:
{key: {d[key] for d in dicts if key in d} for key in {key for d in dicts for key in d}}
although readibility would benefit from naming the combined key set:
combined_key_set = {key for d in dicts for key in d}
super_dict = {key: {d[key] for d in dicts if key in d} for key in combined_key_set}
Elegance can be debated but personally I prefer comprehensions over for loops. :)
(The dictionary and set comprehensions are available in Python 2.7/3.1 and newer.)
python 3.x (reduce is builtin for python 2.x, so no need to import if in 2.x)
import operator
from functools import operator.add
a = [{'a': 1}, {'b': 2}, {'c': 3, 'd': 4}]
dict(reduce(operator.add, map(list,(map(dict.items, a))))
map(dict.items, a) # converts to list of key, value iterators
map(list, ... # converts to iterator equivalent of [[[a, 1]], [[b, 2]], [[c, 3],[d,4]]]
reduce(operator.add, ... # reduces the multiple list down to a single list
My solution is similar to #senderle proposed, but instead of for loop I used map
super_dict = defaultdict(set)
map(lambda y: map(lambda x: super_dict[x].add(y[x]), y), dicts)
The use of defaultdict is good, this also can be done with the use of itertools.groupby.
import itertools
# output all dict items, and sort them by key
dicts_ele = sorted( ( item for d in dicts for item in d.items() ), key = lambda x: x[0] )
# groups items by key
ele_groups = itertools.groupby( dicts_ele, key = lambda x: x[0] )
# iterates over groups and get item value
merged = { k: set( v[1] for v in grouped ) for k, grouped in ele_groups }
and obviously, you can merge this block of code into one-line style
merged = {
k: set( v[1] for v in grouped )
for k, grouped in (
itertools.groupby(
sorted(
( item for d in dicts for item in d.items() ),
key = lambda x: x[0]
),
key = lambda x: x[0]
)
)
}
I'm a bit late to the game but I did it in 2 lines with no dependencies beyond python itself:
flatten = lambda *c: (b for a in c for b in (flatten(*a) if isinstance(a, (tuple, list)) else (a,)))
o = reduce(lambda d1,d2: dict((k, list(flatten([d1.get(k), d2.get(k)]))) for k in set(d1.keys() + d2.keys())), dicts)
# output:
# {'a': [1, 1, None], 'c': [3, 'foo', 3], 'b': [2, None, None], 'e': [None, 57], 'd': [None, 2, None]}
Though if you don't care about nested lists, then:
o2 = reduce(lambda d1,d2: dict((k, [d1.get(k), d2.get(k)]) for k in set(d1.keys() + d2.keys())), dicts)
# output:
# {'a': [[1, 1], None], 'c': [[3, 'foo'], 3], 'b': [[2, None], None], 'e': [None, 57], 'd': [[None, 2], None]}