This question already has answers here:
Python dict.fromkeys return same id element
(2 answers)
Closed 3 years ago.
I've a dict code snippets which is not behaving as expected
a = {"d1":{"a":1,"b":2,"c":4},"d2":{"a":1,"b":2,"c":4},"d3":{"a":1,"b":2,"c":4}}
b = {"d1":{"a":1,"b":0},"d2":{"a":0,"c":4},"d3":{"a":1,"b":2,"c":4}}
c = dict.fromkeys(a.keys(),{})
print(c)
for doc in b.keys():
for word in b[doc].keys():
c[doc][word] = a[doc][word]*b[doc][word]
print(c)
output is:
{'d1': {}, 'd2': {}, 'd3': {}}
{'d1': {'a': 1, 'b': 4, 'c': 16}, 'd2': {'a': 1, 'b': 4, 'c': 16}, 'd3': {'a': 1, 'b': 4, 'c': 16}}
instead of:
{'d1': {}, 'd2': {}, 'd3': {}}
{'d1': {'a': 1, 'b': 0}, 'd2': {'a': 0, 'c': 16}, 'd3': {'a': 1, 'b': 4, 'c': 16}}
I very confused now any insights would be helpful.
The problem is because you are using a mutable object as the second argument for fromkeys.
This is much clearer here:
d = dict.fromkeys(['a', 'b'], [])
d['a'].append(1)
print(d)
Outputs
{'a': [1], 'b': [1]}
Made a modification to your for loop :
for doc in b.keys():
for word in b[doc].keys():
if doc not in c:
c[doc]={}
c[doc][word] = a[doc][word]*b[doc][word]
print(c)
#{'d1': {'a': 1, 'b': 0}, 'd2': {'a': 0, 'c': 16}, 'd3': {'a': 1, 'b': 4, 'c': 16}}
Use a dictionary comprehension to create c instead:
c = {k: {} for k in a.keys()}
for doc in b.keys():
for word in b[doc].keys():
c[doc][word] = a[doc][word]*b[doc][word]
print(c)
# {'d1': {'a': 1, 'b': 0}, 'd2': {'a': 0, 'c': 16}, 'd3': {'a': 1, 'b': 4, 'c': 16}}
Notice the difference when you use fromkeys vs dictionary comprehension:
c = dict.fromkeys(a.keys(),{})
print([id(o) for o in c.values()])
# [53649152, 53649152, 53649152]
# same object reference id!
c = {k: {} for k in a.keys()}
print([id(o) for o in c.values()])
# [53710208, 53649104, 14445232]
# each object has different reference id
I have multiples list of dictionaries like this:
list_of_dictionaries_1 = [{
'a':1,
'b':2
}, {
'a':3,
'b':4
}]
list_of_dictionaries_2 = [{
'c':1,
'd':2
}, {
'c':3,
'd':4
}]
And I want to add each element into a new dictionary.
new_dictionary = {
data: [{
'a':1,
'b':2
}, {
'a':3,
'b':4
}, {
'c':1,
'd':2
}, {
'c':3,
'd':4
}]
}
So I made this for each list of dictionaries:
for dictionary_ in list_of_dictionaries_1:
new_dictionary['data'] = dictionary_
But this just return the last element in the list of dictionaries.
new_dictionary = {
data:[{
'c':3,
'd':4
}]
}
How can I add all de dictionaries in the new dictionary?
If I understood correctly, you could do it like this:
new_dictionary = {'data': []}
for elem in list_of_dictionaries_1 + list_of_dictionaries_2:
new_dictionary['data'].append(elem)
print(new_dictionary)
Output:
{'data': [{'a': 1, 'b': 2}, {'a': 3, 'b': 4}, {'c': 1, 'd': 2}, {'c': 3, 'd': 4}]}
You can use itertools.chain to merge the two lists:
from itertools import chain
new_dictionary = {'data': list(chain(list_of_dictionaries_1, list_of_dictionaries_2))}
new_dictionary becomes:
{'data': [{'a': 1, 'b': 2}, {'a': 3, 'b': 4}, {'c': 1, 'd': 2}, {'c': 3, 'd': 4}]}
Your dictionary structure looks inconsistent. But however, you can do the below to achieve what you are trying for.
list_of_dictionaries_1 = [{'a':1, 'b':2 }, {'a':3, 'b':4}]
list_of_dictionaries_2 = [{'c':1, 'd':2 }, {'c':3, 'd':4 }]
list_of_dictionaries_1.extend(list_of_dictionaries_2)
print(list_of_dictionaries_1)
Output:
[{'a': 1, 'b': 2}, {'a': 3, 'b': 4}, {'c': 1, 'd': 2}, {'c': 3, 'd': 4}]
I am trying to update values for a dict() key dynamically with a for loop.
def update_dict():
f = []
for i, j in enumerate(list_range):
test_dict.update({'a': i})
j['c'] = test_dict
print(j)
f.append(j)
print(f)
test_dict = dict({'a': 1})
list_range = [{'b': i} for i in range(0, 5)]
update_dict()
Even print(j) gives iterating value (0,1,2,3,4), somehow the last dict is getting overwritten all over the list and giving wrong output (4,4,4,4,4).
Expected Output,
[{'b': 0, 'c': {'a': 0}}, {'b': 1, 'c': {'a': 1}}, {'b': 2, 'c': {'a': 2}}, {'b': 3, 'c': {'a': 3}}, {'b': 4, 'c': {'a': 4}}]
Output obtained,
[{'b': 0, 'c': {'a': 4}}, {'b': 1, 'c': {'a': 4}}, {'b': 2, 'c': {'a': 4}}, {'b': 3, 'c': {'a': 4}}, {'b': 4, 'c': {'a': 4}}]
I need to understand how the dictionaries are getting overwritten and what could be the best solution to avoid this?
Thanks in advance!
P.S. : please avoid suggesting list or dict comprehension method as bare answer as i am aware of them and the only purpose of this question is to understand the wrong behavior of dict().
The reason of such behaviour is that all references in list points to the same dict. Line j['c'] = test_dict doesn't create copy of dictionary, but just make j['c'] refer to test_dict. To get expected result you need change this line to:
j['c'] = test_dict.copy(). It will make deep copy of test_dict and assign it to j['c'].
You try to add values to same dictionary every time in the loop and as loop progresses, you keep replacing the values.
You need to define dictionary in every iteration to create separate references of the dictionary:
def update_dict():
f = []
for i, j in enumerate(list_range):
test_dict = {'a': i}
j['c'] = test_dict
f.append(j)
print(f)
list_range = [{'b': i} for i in range(0, 5)]
update_dict()
# [{'b': 0, 'c': {'a': 0}},
# {'b': 1, 'c': {'a': 1}},
# {'b': 2, 'c': {'a': 2}},
# {'b': 3, 'c': {'a': 3}},
# {'b': 4, 'c': {'a': 4}}]
A simpler solution could be to iterate through list_range and create c using the values from b
lista = [{'b': i } for i in range(0, 5)]
for i in lista:
i['c'] = {'a': i['b']}
# [{'b': 0, 'c': {'a': 0}}, {'b': 1, 'c': {'a': 1}}, {'b': 2, 'c': {'a': 2}}, {'b': 3, 'c': {'a': 3}}, {'b': 4, 'c': {'a': 4}}]
def update_dict():
f = []
for i, j in enumerate(list_range):
j['c'] = {'a': i}
print(j)
f.append(j)
return f
list_range = [{'b': i} for i in range(0, 5)]
print(update_dict())
#output
{'b': 0, 'c': {'a': 0}}
{'b': 1, 'c': {'a': 1}}
{'b': 2, 'c': {'a': 2}}
{'b': 3, 'c': {'a': 3}}
{'b': 4, 'c': {'a': 4}}
If I have a dictionary containing lists in one or more of its values:
data = {
'a':0,
'b':1,
'c':[0, 1, 2],
'pair':['one','two']
}
How can I get a list of dict tuples paired by pair and iterating over c, with all else remaining constant? E.g.
output = [
({
'a':0,
'b':1,
'c':0,
'pair':'one'
},
{
'a':0,
'b':1,
'c':0,
'pair':'two'
}),
({
'a':0,
'b':1,
'c':1,
'pair':'one'
},
...
]
Well, this doesn't feel especially elegant, but you might use a nested for loop or list comprehension:
output = []
for i in data['c']:
output.append(tuple({'a': 0, 'b': 1, 'c': i, 'pair': p} for p in data))
or
output = [tuple({'a': 0, 'b': 1, 'c': i, 'pair': p} for p in data['pair']) for i in data['c']]
A cleaner solution might separate out the generation of the component dict into a function, like this:
def gen_output_dict(c, pair):
return {'a': 0, 'b': 1, 'c': c, 'pair': pair}
output = []
for i in data['c']:
output.append(tuple(gen_output_dict(i, p) for p in data['pair']))
You can use itertools.product on list values and keep track of the key from which each element originated. Since the key 'pair' has a special meaning, you should treat it separately.
Code
from itertools import product
def unzip_dict(d):
keys = [k for k, v in d.items() if isinstance(v, list) and k != 'pair']
values = [d[k] for k in keys]
for values in product(*values):
yield tuple({**d, **dict(zip(keys, values)), 'pair': pair} for pair in d['pair'])
Example
data = {
'a': 0,
'c': [1, 2],
'pair': ['one', 'two']
}
print(*unzip_dict(data))
Output
({'a': 0, 'c': 1, 'pair': 'one'}, {'a': 0, 'c': 1, 'pair': 'two'})
({'a': 0, 'c': 2, 'pair': 'one'}, {'a': 0, 'c': 2, 'pair': 'two'})
The following is quite an extended solution:
data = {
'a':0,
'b':1,
'c':[0, 1, 2],
'pair':['one','two']
}
# Get the length of the longest sequence
length = max(map(lambda x: len(x) if isinstance(x, list) else 1, data.values()))
# Loop through the data and change scalars to sequences
# while also making sure that smaller sequences are stretched to match
# or exceed the length of the longest sequence
for k, v in data.items():
if isinstance(v, list):
data[k] = v * int(round(length/len(v), 0))
else:
data[k] = [v] * length
# Create a dictionary to keep track of which outputs
# need to end up in which tuple
seen = dict.fromkeys(data.get('pair'), 0)
output = [tuple()] * len(seen)
# Loop through the data and place dictionaries in their
# corresponding tuples.
for v in zip(*data.values()):
d = dict(zip(data, v))
output[seen[d.get('pair')]] += (d,)
seen[d.get('pair')] += 1
print(output)
The idea is to convert the scalars in your data to sequences whose lengths match that of the longest sequence in the original data. Therefore, the first thing I did was assign to the variable length the size of the longest sequence. Armed with that knowledge, we loop through the original data and extend the already existing sequences to match the size of the longest sequence while converting scalars to sequences.
Once that's done, we move to generating the output variable. But first, we create a dictionary called seen to help us both create a list of tuples and keep track of which group of dictionaries ends up in which tuple.
This, then, allows us to run one final loop to place the groups of dictionaries to their corresponding tuples.
The current output looks like the following:
[({'a': 0, 'b': 1, 'c': 0, 'pair': 'one'},
{'a': 0, 'b': 1, 'c': 1, 'pair': 'two'}),
({'a': 0, 'b': 1, 'c': 2, 'pair': 'one'},)]
Please let me know if you need any more clarifying details. Otherwise, I do hope this serves some purpose.
#r3robertson, You can also try the below code. The code is based on the concept of list comprehension, & deepcopy() operation in Python.
Check Shallow copy vs deepcopy in Python.
import pprint;
import copy;
data = {
'a': 0,
'b': 1,
'c': [0, 1, 2],
'pair': ['one','two'],
};
def get_updated_dict(data, index, pair_name):
d = copy.deepcopy(data);
d.update({'c': index, 'pair': pair_name});
return d;
output = [tuple(get_updated_dict(data, index, pair_name) for pair_name in data['pair']) for index in data['c']];
# Pretty printing the output list.
pprint.pprint(output, indent=4);
Output »
[ ( { 'a': 0, 'b': 1, 'c': 0, 'pair': 'one'},
{ 'a': 0, 'b': 1, 'c': 0, 'pair': 'two'}),
( { 'a': 0, 'b': 1, 'c': 1, 'pair': 'one'},
{ 'a': 0, 'b': 1, 'c': 1, 'pair': 'two'}),
( { 'a': 0, 'b': 1, 'c': 2, 'pair': 'one'},
{ 'a': 0, 'b': 1, 'c': 2, 'pair': 'two'})]
Pretty printing using json module »
Note: Tuple will convert into list here as tuples are not supported inside JSON.
import json;
print(json.dumps(output, indent=4));
Output »
[
[
{
"a": 0,
"c": 0,
"b": 1,
"pair": "one"
},
{
"a": 0,
"c": 0,
"b": 1,
"pair": "two"
}
],
[
{
"a": 0,
"c": 1,
"b": 1,
"pair": "one"
},
{
"a": 0,
"c": 1,
"b": 1,
"pair": "two"
}
],
[
{
"a": 0,
"c": 2,
"b": 1,
"pair": "one"
},
{
"a": 0,
"c": 2,
"b": 1,
"pair": "two"
}
]
]
Not too perfect but here's my solution.
data = { 'a':0, 'b':1, 'c':[0, 1, 2], 'pair':['one','two'] }
a,b = data['pair'], data['c']
for t in range(0, len(b)):
for u in range(0, len(a)):
for h in a:
data['c']=b[t]
data['pair']=a[u]
print(tuple([data]))
If you are sure about defined keys then you can use like below
record_dict = {'id':'0123abc', 'roles': ['abc', 'cda', 'xyz']}
output = []
for index, key in enumerate(record_dict['roles']):
output.append({'id': record_dict.get('id'), 'roles': key})
print(output)
You can use itertools:
import itertools
data = {
'a':0,
'b':1,
'c':[0, 1, 2],
'pair':['one','two']
}
def expand_dict(data):
grouped = [a for a, b in data.items() if isinstance(b, list)]
p = [[a, list(b)] for a, b in itertools.groupby(itertools.product(*[data[i] for i in grouped]), key=lambda x:x[0])]
return [tuple({**data, **dict(zip(grouped, i))} for i in c) for _, c in p]
print(expand_dict(data))
Output:
[({'a': 0, 'b': 1, 'c': 0, 'pair': 'one'}, {'a': 0, 'b': 1, 'c': 0, 'pair': 'two'}),
({'a': 0, 'b': 1, 'c': 1, 'pair': 'one'}, {'a': 0, 'b': 1, 'c': 1, 'pair': 'two'}),
({'a': 0, 'b': 1, 'c': 2, 'pair': 'one'}, {'a': 0, 'b': 1, 'c': 2, 'pair': 'two'})]
This solution will also work on input with many possible lists of values:
data = {'a':[5, 6, 1, 3], 'b':1, 'c':[0, 1, 2], 'pair':['one', 'two']}
print(expand_dict(data))
Output:
[({'a': 5, 'b': 1, 'c': 0, 'pair': 'one'}, {'a': 5, 'b': 1, 'c': 0, 'pair': 'two'}, {'a': 5, 'b': 1, 'c': 1, 'pair': 'one'}, {'a': 5, 'b': 1, 'c': 1, 'pair': 'two'}, {'a': 5, 'b': 1, 'c': 2, 'pair': 'one'}, {'a': 5, 'b': 1, 'c': 2, 'pair': 'two'}), ({'a': 6, 'b': 1, 'c': 0, 'pair': 'one'}, {'a': 6, 'b': 1, 'c': 0, 'pair': 'two'}, {'a': 6, 'b': 1, 'c': 1, 'pair': 'one'}, {'a': 6, 'b': 1, 'c': 1, 'pair': 'two'}, {'a': 6, 'b': 1, 'c': 2, 'pair': 'one'}, {'a': 6, 'b': 1, 'c': 2, 'pair': 'two'}), ({'a': 1, 'b': 1, 'c': 0, 'pair': 'one'}, {'a': 1, 'b': 1, 'c': 0, 'pair': 'two'}, {'a': 1, 'b': 1, 'c': 1, 'pair': 'one'}, {'a': 1, 'b': 1, 'c': 1, 'pair': 'two'}, {'a': 1, 'b': 1, 'c': 2, 'pair': 'one'}, {'a': 1, 'b': 1, 'c': 2, 'pair': 'two'}), ({'a': 3, 'b': 1, 'c': 0, 'pair': 'one'}, {'a': 3, 'b': 1, 'c': 0, 'pair': 'two'}, {'a': 3, 'b': 1, 'c': 1, 'pair': 'one'}, {'a': 3, 'b': 1, 'c': 1, 'pair': 'two'}, {'a': 3, 'b': 1, 'c': 2, 'pair': 'one'}, {'a': 3, 'b': 1, 'c': 2, 'pair': 'two'})]