I would like to convert a Python dictionary of the following form:
D = {'a':[1,2,3], 'b':[0.1,0.5], 'c':[10,20]}
into a list of dictionaries the following form:
E = [{a:1,b:0.1,c:10}, {a:1,b:0.1,c:20},{a:1,b:0.5,c:10}, ...., {a:3,b:0.5,c:20}]
I have tried using itertools but I do not understand how to use it to make combinations of dictionaries.
Here is a solution:
import itertools
D = {'a':[1,2,3], 'b':[0.1,0.5], 'c':[10,20]}
E = [dict(zip(D.keys(), a)) for a in itertools.product(*D.values())]
This yields:
E= [{'a': 1, 'b': 0.1, 'c': 10},
{'a': 1, 'b': 0.1, 'c': 20},
{'a': 1, 'b': 0.5, 'c': 10},
{'a': 1, 'b': 0.5, 'c': 20},
{'a': 2, 'b': 0.1, 'c': 10},
{'a': 2, 'b': 0.1, 'c': 20},
{'a': 2, 'b': 0.5, 'c': 10},
{'a': 2, 'b': 0.5, 'c': 20},
{'a': 3, 'b': 0.1, 'c': 10},
{'a': 3, 'b': 0.1, 'c': 20},
{'a': 3, 'b': 0.5, 'c': 10},
{'a': 3, 'b': 0.5, 'c': 20}]
Edit: Removed ordered dict as Aran points out, here is documentation to support it:Python dictionary: are keys() and values() always the same order?
Why not just:
from itertools import product
D = {'a':[1,2,3], 'b':[0.1,0.5], 'c':[10,20]}
print([dict(zip(D.keys(),v)) for v in product(*D.values())])
Output:
[{'a': 1, 'b': 0.1, 'c': 10}, {'a': 1, 'b': 0.1, 'c': 20}, {'a': 1, 'b': 0.5, 'c': 10}, {'a': 1, 'b': 0.5, 'c': 20}, {'a': 2, 'b': 0.1, 'c': 10}, {'a': 2, 'b': 0.1, 'c': 20}, {'a': 2, 'b': 0.5, 'c': 10}, {'a': 2, 'b': 0.5, 'c': 20}, {'a': 3, 'b': 0.1, 'c': 10}, {'a': 3, 'b': 0.1, 'c': 20}, {'a': 3, 'b': 0.5, 'c': 10}, {'a': 3, 'b': 0.5, 'c': 20}]
One option is to simply use nested for loops; if the original data is not too big, then this will work fine and you won't need itertools to get it working.
>>> origin = {
'a': [1, 2, 3],
'b': [0.1, 0.5],
'c': [10, 20],
}
>>> result = [
{
'a': a,
'b': b,
'c': c,
}
for a in origin['a']
for b in origin['b']
for c in origin['c']]
>>> result
[{'a': 1, 'b': 0.1, 'c': 10}, {'a': 1, 'b': 0.1, 'c': 20},
{'a': 1, 'b': 0.5, 'c': 10}, {'a': 1, 'b': 0.5, 'c': 20},
{'a': 2, 'b': 0.1, 'c': 10}, {'a': 2, 'b': 0.1, 'c': 20},
{'a': 2, 'b': 0.5, 'c': 10}, {'a': 2, 'b': 0.5, 'c': 20},
{'a': 3, 'b': 0.1, 'c': 10}, {'a': 3, 'b': 0.1, 'c': 20},
{'a': 3, 'b': 0.5, 'c': 10}, {'a': 3, 'b': 0.5, 'c': 20}]
I developed a round about answer
parameter_values_each = {'a':[1,2,3], 'b':[0.1,0.5], 'c':[10,20]}
param_possibilities = []
for name in parameter_values_each:
temp = []
for val in parameter_values_each[name]:
temp.append((name,val))
param_possibilities.append(temp)
result = list(itertools.product(*param_possibilities))
print result
Long way with no itertools (or this is what going on inside product):
D = {'a': [1, 2, 3], 'b': [0.1, 0.5], 'c': [10, 20]}
E = []
list_of_keys = list(D.keys())
list_of_lengths = [len(D[key]) for key in list_of_keys]
product_number = 1
for length in list_of_lengths:
product_number *= length
for n in range(product_number):
index = n
index_list = []
for length in reversed(list_of_lengths):
index_list.insert(0, index % length)
index = index // length
keys_with_values = {}
for j, key in enumerate(list_of_keys):
keys_with_values[key] = D[key][index_list[j]]
E.append(keys_with_values)
for e in E:
print(e)
Result:
{'a': 1, 'b': 0.1, 'c': 10}
{'a': 1, 'b': 0.1, 'c': 20}
{'a': 1, 'b': 0.5, 'c': 10}
{'a': 1, 'b': 0.5, 'c': 20}
{'a': 2, 'b': 0.1, 'c': 10}
{'a': 2, 'b': 0.1, 'c': 20}
{'a': 2, 'b': 0.5, 'c': 10}
{'a': 2, 'b': 0.5, 'c': 20}
{'a': 3, 'b': 0.1, 'c': 10}
{'a': 3, 'b': 0.1, 'c': 20}
{'a': 3, 'b': 0.5, 'c': 10}
{'a': 3, 'b': 0.5, 'c': 20}
Related
I have a list of dictionaries and a separate dictionary having the same keys and only the values are different. For example the list of dictionaries look like this:
[{'A': 0.102, 'B': 0.568, 'C': 0.33}, {'A': 0.026, 'B': 0.590, 'C': 0.382}, {'A': 0.005, 'B': 0.857, 'C': 0.137}, {'A': 0.0, 'B': 0.962, 'C': 0.036}, {'A': 0.0, 'B': 0.991, 'C': 0.008}]
and the separate dictionary looks like this:
{'A': 0.005, 'B': 0.956, 'C': 0.038}
I want to compare the separate dictionary with the list of dictionaries and return the index from the list which has higher value than the separate dictionary. In this example, the indices would be 3, 4 as the dictionary in indices 3 and 4 has a higher value for key 'B' since 'B' has the highest value in the separate dictionary.
Any ideas on how I should I proceed the problem?
You can use enumerate for finding index of max value:
org = [
{'A': 0.102, 'B': 0.568, 'C': 0.33},
{'A': 0.026, 'B': 0.590, 'C': 0.382},
{'A': 0.005, 'B': 0.857, 'C': 0.137},
{'A': 0.0, 'B': 0.962, 'C': 0.036},
{'A': 0.0, 'B': 0.991, 'C': 0.008}
]
com = {'A': 0.005, 'B': 0.956, 'C': 0.038}
def fnd_index(org, com):
key_max, val_max = max(com.items(), key=lambda x: x[1])
print('key_max:', key_max)
print('val_max:', val_max)
res = []
for idx, dct in enumerate(org):
if dct[key_max] > val_max:
res.append(idx)
return res
res = fnd_index(org, com)
print('result:', res)
Output:
key_max: B
val_max: 0.956
result: [3, 4]
are you sure that it should be only index 4?
dict_list = [{'A': 0.102, 'B': 0.568, 'C': 0.33},
{'A': 0.026, 'B': 0.590, 'C': 0.382},
{'A': 0.005, 'B': 0.857, 'C': 0.137},
{'A': 0.0, 'B': 0.962, 'C': 0.036},
{'A': 0.0, 'B': 0.991, 'C': 0.008}]
d = {'A': 0.005, 'B': 0.956, 'C': 0.038}
max_val = max(d.values())
idxmax = [i for i,j in enumerate(dict_list) if max(j.values()) > max_val]
print(idxmax) # [3, 4]
I have a dict I'm trying to loop through which stores integer values:
myDict = {"a":1,"b":2,"c":3}
And I'm trying to loop through every combination of these integers from 1 to their value. For example:
{"a":1,"b":1,"c":1}
{"a":1,"b":1,"c":2}
{"a":1,"b":1,"c":3}
{"a":1,"b":2,"c":1}
{"a":1,"b":2,"c":2}
And so on...
Ending at their max values at the start while going through all the combinations.
Is there an elegant way to this that applies to any size dictionary? :
myDict = {"a":1,"b":2,"c":3,"d":5}
I'm currently doing this with nested if statements that edit a clone dictionary, so any solution setting dict values would work :) :
copyDict["c"] = 2
Yes, this will do it, for any size input (that fits in memory, of course):
import itertools
myDict = {"a":1,"b":2,"c":3,"d":5}
keys = list(myDict.keys())
ranges = tuple(range(1,k+1) for k in myDict.values())
outs = [dict(zip(keys,s)) for s in itertools.product(*ranges)]
print(outs)
You can actually replace *ranges with the *(range(1,k+1) for k in myDict.values()) and reduce it by one line, but it's not easier to read.
Another way with itertools.product:
>>> [dict(zip(myDict.keys(), p)) for p in itertools.product(myDict.values(), repeat=len(myDict))]
[{'a': 1, 'b': 1, 'c': 1},
{'a': 1, 'b': 1, 'c': 2},
{'a': 1, 'b': 1, 'c': 3},
{'a': 1, 'b': 2, 'c': 1},
{'a': 1, 'b': 2, 'c': 2},
{'a': 1, 'b': 2, 'c': 3},
{'a': 1, 'b': 3, 'c': 1},
{'a': 1, 'b': 3, 'c': 2},
{'a': 1, 'b': 3, 'c': 3},
{'a': 2, 'b': 1, 'c': 1},
{'a': 2, 'b': 1, 'c': 2},
{'a': 2, 'b': 1, 'c': 3},
{'a': 2, 'b': 2, 'c': 1},
{'a': 2, 'b': 2, 'c': 2},
{'a': 2, 'b': 2, 'c': 3},
{'a': 2, 'b': 3, 'c': 1},
{'a': 2, 'b': 3, 'c': 2},
{'a': 2, 'b': 3, 'c': 3},
{'a': 3, 'b': 1, 'c': 1},
{'a': 3, 'b': 1, 'c': 2},
{'a': 3, 'b': 1, 'c': 3},
{'a': 3, 'b': 2, 'c': 1},
{'a': 3, 'b': 2, 'c': 2},
{'a': 3, 'b': 2, 'c': 3},
{'a': 3, 'b': 3, 'c': 1},
{'a': 3, 'b': 3, 'c': 2},
{'a': 3, 'b': 3, 'c': 3}]
My question is somewhat similar to this question: https://codereview.stackexchange.com/questions/175079/removing-key-value-pairs-in-list-of-dicts. Essentially, I have a list of dictionaries, and I want to remove duplicates from the list based on the unique combination of two (or more) keys within each dictionary.
Suppose I have the following list of dictionaries:
some_list_of_dicts = [
{'a': 1, 'b': 1, 'c': 1, 'd': 2, 'e': 4},
{'a': 1, 'b': 1, 'c': 1, 'd': 5, 'e': 1},
{'a': 1, 'b': 1, 'c': 1, 'd': 7, 'e': 8},
{'a': 1, 'b': 1, 'c': 1, 'd': 9, 'e': 6},
{'a': 1, 'b': 1, 'c': 2, 'd': 2, 'e': 3},
{'a': 1, 'b': 1, 'c': 3, 'd': 2, 'e': 3},
{'a': 1, 'b': 1, 'c': 4, 'd': 2, 'e': 3}
]
And let's suppose the combination of a, b, and c have to be unique; any other values can be whatever they want, but the combination of these three must be unique to this list. I would want to take whichever unique combo of a, b, and c came first, keep that, and discard everything else where that combination is the same.
The new list, after running it through some remove_duplicates function would look like this:
new_list = [
{'a': 1, 'b': 1, 'c': 1, 'd': 2, 'e': 4},
{'a': 1, 'b': 1, 'c': 2, 'd': 2, 'e': 3},
{'a': 1, 'b': 1, 'c': 3, 'd': 2, 'e': 3},
{'a': 1, 'b': 1, 'c': 4, 'd': 2, 'e': 3}
]
I've only managed to come up with this:
def remove_duplicates(old_list):
uniqueness_check_list = []
new_list = []
for item in old_list:
# The unique combination is 'a', 'b', and 'c'
uniqueness_check = "{}{}{}".format(
item["a"], item["b"], item["c"]
)
if uniqueness_check not in uniqueness_check_list:
new_list.append(item)
uniqueness_check_list.append(uniqueness_check)
return new_list
But this doesn't feel very Pythonic. It also has the problem that I've hardcoded in the function which keys have to be unique; it would be better if I could specify that as an argument to the function itself, but again, not sure what's the most elegant way to do this.
You can use a dict comprehension to construct a dict from the list of dicts in the reversed order so that the values of the first of any unique combinations would take precedence. Use operator.itemgetter to get the unique keys as a tuple. Reverse again in the end for the original order:
from operator import itemgetter
list({itemgetter('a', 'b', 'c')(d): d for d in reversed(some_list_of_dicts)}.values())[::-1]
This returns:
[{'a': 1, 'b': 1, 'c': 1, 'd': 2, 'e': 4},
{'a': 1, 'b': 1, 'c': 2, 'd': 2, 'e': 3},
{'a': 1, 'b': 1, 'c': 3, 'd': 2, 'e': 3},
{'a': 1, 'b': 1, 'c': 4, 'd': 2, 'e': 3}]
With the help of a function to keep track of duplicates, you can use some list comprehension:
def remove_duplicates(old_list, cols=('a', 'b', 'c')):
duplicates = set()
def is_duplicate(item):
duplicate = item in duplicates
duplicates.add(item)
return duplicate
return [x for x in old_list if not is_duplicate(tuple([x[col] for col in cols]))]
To use:
>>> remove_duplicates(some_list_of_dicts)
[
{'a': 1, 'c': 1, 'b': 1, 'e': 4, 'd': 2},
{'a': 1, 'c': 2, 'b': 1, 'e': 3, 'd': 2},
{'a': 1, 'c': 3, 'b': 1, 'e': 3, 'd': 2},
{'a': 1, 'c': 4, 'b': 1, 'e': 3, 'd': 2}
]
You can also provide different columns to key on:
>>> remove_duplicates(some_list_of_dicts, cols=('a', 'd'))
[
{'a': 1, 'c': 1, 'b': 1, 'e': 4, 'd': 2},
{'a': 1, 'c': 1, 'b': 1, 'e': 1, 'd': 5},
{'a': 1, 'c': 1, 'b': 1, 'e': 8, 'd': 7},
{'a': 1, 'c': 1, 'b': 1, 'e': 6, 'd': 9}
]
I have some code that generates all the combinations for a dictionary of lists
import itertools
import collections
def gen_combinations(d):
keys, values = d.keys(), d.values()
combinations = itertools.product(*values)
for c in combinations:
yield dict(zip(keys, c))
Let's say I have a dictionary A like
A = {'a': [0, 1],
'b': [2, 3, 4]}
It yields:
{'a': 0, 'b': 2}
{'a': 0, 'b': 3}
{'a': 0, 'b': 4}
{'a': 1, 'b': 2}
{'a': 1, 'b': 3}
{'a': 1, 'b': 4}
Now I'd like it to work with nested dictionary of lists like
B = {'s1': {'a': [0, 1],
'b': [0, 1, 2] },
's2': {'c': [0, 1],
'd': [0, 1] }}
which should yield something like:
{'s1': {'a': 0, 'b': 0},
's2': {'c': 0, 'd': 0}}
{'s1': {'a': 0, 'b': 0},
's2': {'c': 0, 'd': 1}}
{'s1': {'a': 0, 'b': 0},
's2': {'c': 1, 'd': 0}}
{'s1': {'a': 0, 'b': 0},
's2': {'c': 1, 'd': 1}}
and so on for all the combinations...
I'm looking for an elegant way to do this. Don't have any particular performance restrictions, they are small dictionaries.
Just create the product of the output of gen_combinations() for each key in the outer dictionary:
def gen_dict_combinations(d):
keys, values = d.keys(), d.values()
for c in itertools.product(*(gen_combinations(v) for v in values)):
yield dict(zip(keys, c))
This is basically the same pattern, only now instead of using values directly, we use the output of gen_combinations():
>>> for c in gen_dict_combinations(B):
... print(c)
...
{'s1': {'a': 0, 'b': 0}, 's2': {'c': 0, 'd': 0}}
{'s1': {'a': 0, 'b': 0}, 's2': {'c': 0, 'd': 1}}
{'s1': {'a': 0, 'b': 0}, 's2': {'c': 1, 'd': 0}}
{'s1': {'a': 0, 'b': 0}, 's2': {'c': 1, 'd': 1}}
{'s1': {'a': 0, 'b': 1}, 's2': {'c': 0, 'd': 0}}
{'s1': {'a': 0, 'b': 1}, 's2': {'c': 0, 'd': 1}}
{'s1': {'a': 0, 'b': 1}, 's2': {'c': 1, 'd': 0}}
{'s1': {'a': 0, 'b': 1}, 's2': {'c': 1, 'd': 1}}
{'s1': {'a': 0, 'b': 2}, 's2': {'c': 0, 'd': 0}}
{'s1': {'a': 0, 'b': 2}, 's2': {'c': 0, 'd': 1}}
{'s1': {'a': 0, 'b': 2}, 's2': {'c': 1, 'd': 0}}
{'s1': {'a': 0, 'b': 2}, 's2': {'c': 1, 'd': 1}}
{'s1': {'a': 1, 'b': 0}, 's2': {'c': 0, 'd': 0}}
{'s1': {'a': 1, 'b': 0}, 's2': {'c': 0, 'd': 1}}
{'s1': {'a': 1, 'b': 0}, 's2': {'c': 1, 'd': 0}}
{'s1': {'a': 1, 'b': 0}, 's2': {'c': 1, 'd': 1}}
{'s1': {'a': 1, 'b': 1}, 's2': {'c': 0, 'd': 0}}
{'s1': {'a': 1, 'b': 1}, 's2': {'c': 0, 'd': 1}}
{'s1': {'a': 1, 'b': 1}, 's2': {'c': 1, 'd': 0}}
{'s1': {'a': 1, 'b': 1}, 's2': {'c': 1, 'd': 1}}
{'s1': {'a': 1, 'b': 2}, 's2': {'c': 0, 'd': 0}}
{'s1': {'a': 1, 'b': 2}, 's2': {'c': 0, 'd': 1}}
{'s1': {'a': 1, 'b': 2}, 's2': {'c': 1, 'd': 0}}
{'s1': {'a': 1, 'b': 2}, 's2': {'c': 1, 'd': 1}}
Ended up with a recursive version for arbitrarily nested dictionaries:
import itertools
def gen_combinations(d):
keys, values = d.keys(), d.values()
values_choices = (gen_combinations(v) if isinstance(v, dict) else v for v in values)
for comb in itertools.product(*values_choices):
yield dict(zip(keys, comb))
B = {'s1': {'a': [0, 1],
'b': [0, 1, 2] },
's2': {'c': [0, 1],
'd': [0, 1] }}
for c in gen_combinations(B):
print(c)
Output:
{'s1': {'a': 0, 'b': 0}, 's2': {'c': 0, 'd': 0}}
{'s1': {'a': 0, 'b': 0}, 's2': {'c': 0, 'd': 1}}
{'s1': {'a': 0, 'b': 0}, 's2': {'c': 1, 'd': 0}}
{'s1': {'a': 0, 'b': 0}, 's2': {'c': 1, 'd': 1}}
{'s1': {'a': 0, 'b': 1}, 's2': {'c': 0, 'd': 0}}
{'s1': {'a': 0, 'b': 1}, 's2': {'c': 0, 'd': 1}}
{'s1': {'a': 0, 'b': 1}, 's2': {'c': 1, 'd': 0}}
{'s1': {'a': 0, 'b': 1}, 's2': {'c': 1, 'd': 1}}
{'s1': {'a': 0, 'b': 2}, 's2': {'c': 0, 'd': 0}}
{'s1': {'a': 0, 'b': 2}, 's2': {'c': 0, 'd': 1}}
{'s1': {'a': 0, 'b': 2}, 's2': {'c': 1, 'd': 0}}
{'s1': {'a': 0, 'b': 2}, 's2': {'c': 1, 'd': 1}}
{'s1': {'a': 1, 'b': 0}, 's2': {'c': 0, 'd': 0}}
{'s1': {'a': 1, 'b': 0}, 's2': {'c': 0, 'd': 1}}
{'s1': {'a': 1, 'b': 0}, 's2': {'c': 1, 'd': 0}}
{'s1': {'a': 1, 'b': 0}, 's2': {'c': 1, 'd': 1}}
{'s1': {'a': 1, 'b': 1}, 's2': {'c': 0, 'd': 0}}
{'s1': {'a': 1, 'b': 1}, 's2': {'c': 0, 'd': 1}}
{'s1': {'a': 1, 'b': 1}, 's2': {'c': 1, 'd': 0}}
{'s1': {'a': 1, 'b': 1}, 's2': {'c': 1, 'd': 1}}
{'s1': {'a': 1, 'b': 2}, 's2': {'c': 0, 'd': 0}}
{'s1': {'a': 1, 'b': 2}, 's2': {'c': 0, 'd': 1}}
{'s1': {'a': 1, 'b': 2}, 's2': {'c': 1, 'd': 0}}
{'s1': {'a': 1, 'b': 2}, 's2': {'c': 1, 'd': 1}}
I extended Martijn Pieters answer so that the function can also handle non-iterable values in the nested dictionary. So the code below works on nested dicts (just one level of nesting though) that look like this:
dictionary = {
'a': 31415,
'b': 'strings too',
'c': [1, 2, 3],
'd': {'e': [1, 2, 3], 'f': 42, 'g': 'strings here too'}
}
from itertools import product
def _gen_combinations(d):
keys, values = d.keys(), d.values()
list_keys = [k for k in keys if isinstance(d[k], list)]
nonlist_keys = [k for k in keys if k not in list_keys]
list_values = [v for v in values if isinstance(v, list)]
nonlist_values = [v for v in values if v not in list_values]
combinations = product(*list_values)
for c in combinations:
result = dict(zip(list_keys, c))
result.update({k: v for k, v in zip(nonlist_keys, nonlist_values)})
yield result
def generate_dict_combinations(d):
keys, values = d.keys(), d.values()
dict_values = [v for v in values if isinstance(v, dict)]
dict_keys = [k for k in keys if isinstance(d[k], dict)]
nondict_values = [v for v in values if v not in dict_values]
nondict_keys = [k for k in keys if k not in dict_keys]
for c in product(*(_gen_combinations(v) for v in dict_values)):
result = dict(zip(dict_keys, c))
result.update({k: v for k, v in zip(nondict_keys, nondict_values)})
yield result
This question already has answers here:
How do I merge two dictionaries in a single expression in Python?
(43 answers)
Closed 5 years ago.
I have two lists of dictionaries which I am trying to get the product of:
from itertools import product
list1 = [{'A': 1, 'B': 1}, {'A': 2, 'B': 2}, {'A': 2, 'B': 1}, {'A': 1, 'B': 2}]
list2 = [{'C': 1, 'D': 1}, {'C': 1, 'D': 2}]
for p in product(list1, list2):
print p
and this gives me the output:
({'A': 1, 'B': 1}, {'C': 1, 'D': 1})
({'A': 1, 'B': 1}, {'C': 1, 'D': 2})
({'A': 2, 'B': 2}, {'C': 1, 'D': 1})
({'A': 2, 'B': 2}, {'C': 1, 'D': 2})
({'A': 2, 'B': 1}, {'C': 1, 'D': 1})
({'A': 2, 'B': 1}, {'C': 1, 'D': 2})
({'A': 1, 'B': 2}, {'C': 1, 'D': 1})
({'A': 1, 'B': 2}, {'C': 1, 'D': 2})
How would I flatten these so the output is a single dict rather than a tuple of dicts?:
{'A': 1, 'B': 1, 'C': 1, 'D': 1}
{'A': 1, 'B': 1, 'C': 1, 'D': 2}
{'A': 2, 'B': 2, 'C': 1, 'D': 1}
{'A': 2, 'B': 2, 'C': 1, 'D': 2}
{'A': 2, 'B': 1, 'C': 1, 'D': 1}
{'A': 2, 'B': 1, 'C': 1, 'D': 2}
{'A': 1, 'B': 2, 'C': 1, 'D': 1}
{'A': 1, 'B': 2, 'C': 1, 'D': 2}
Looks like you want to merge the dictionaries
for p1, p2 in product(list1, list2):
merged = {**p1, **p2}
print(merged)
In earlier versions of Python, you can't merge with this expression. Use p1.update(p2) instead.