Related
Maybe this is impossible but without a for loop through each key in a given dictionary merge the two based on the key in the dictionary
Given :
dict1 = { 'APPL' : { 'cp': 1, 'sed': 'bull'}, 'BAC' : { 'cp': 1, 'sed': 'bull'}}
dict2 = { 'APPL' : { 'tp': 100}}
dict3 = dict1 | dict2 ## python ≥3.9 only
print(dict3)
{'APPL': {'tp': 100}, 'BAC': {'cp': 1, 'sed': 'bull'}}
dict1.update(dict2)
print(dict1)
{'APPL': {'tp': 100}, 'BAC': {'cp': 1, 'sed': 'bull'}}
Desired output
{'APPL': {'tp': 100,'cp': 1, 'sed': 'bull'}, 'BAC': {'cp': 1, 'sed': 'bull'}}
I can do it now with a for loop , just wondering if there is a more elegant solution
Do:
dict1 = {'APPL': {'cp': 1, 'sed': 'bull'}, 'BAC': {'cp': 1, 'sed': 'bull'}}
dict2 = {'APPL': {'tp': 100}}
res = {key: {**value, **dict2.get(key, {})} for key, value in dict1.items()}
print(res)
Output
{'APPL': {'cp': 1, 'sed': 'bull', 'tp': 100}, 'BAC': {'cp': 1, 'sed': 'bull'}}
No, this isn't possible without iteration. You could use a custom joiner function and then reduce a list of dictionaries however:
data = [{'BAC': {'a': 40}, 'XYZ': {'c': 81, 'b': 16}, 'ABC': {'b': 85}},
{'APPL': {'b': 55},
'BAC': {'b': 16, 'f': 59},
'ABC': {'d': 9, 'c': 43},
'XYZ': {'b': 82}},
{'ABC': {'b': 43, 'c': 35},
'APPL': {'f': 17, 'a': 1, 'd': 16},
'BAC': {'f': 35, 'a': 1},
'XYZ': {'a': 2, 'c': 55}},
{'BAC': {'f': 4, 'd': 87},
'XYZ': {'d': 31, 'f': 92},
'APPL': {'b': 18, 'a': 74, 'c': 69}},
{'XYZ': {'d': 84, 'f': 49},
'ABC': {'d': 88, 'a': 82, 'f': 96},
'APPL': {'a': 23},
'BAC': {'b': 40}},
{'BAC': {'c': 88, 'd': 38},
'APPL': {'c': 48, 'b': 30},
'ABC': {'d': 95, 'b': 38},
'XYZ': {'d': 90, 'a': 5}}]
def join_dict(d1, d2):
result = {k: {**d1[k], **d2[k]} for k in d1}
result.update({k: {**d1[k], **d2[k]} for k in d2})
return result
>>> import functools
>>> functools.reduce(join_dict, data)
{'XYZ': {'a': 39, 'f': 78, 'c': 42, 'd': 30, 'b': 24},
'ABC': {'c': 22, 'f': 69, 'a': 8, 'b': 51, 'd': 70},
'APPL': {'d': 19, 'b': 35, 'a': 6, 'f': 33, 'c': 64},
'BAC': {'f': 97, 'c': 38, 'd': 1, 'b': 63, 'a': 91}}
Of course, this will overwrite any common values in the sub-dictionaries. Assuming that isn't an issue for you, this should work fine as a "more elegant solution".
The next problem you have a list of dictionaries of the format
[{'a': 10, 'b': 11, 'c': 12, 'd': 13, 'e': 14},
{'a': 20, 'b': 21, 'c': 22, 'd': 23, 'e': 24},
{'a': 30, 'b': 31, 'c': 32, 'd': 33, 'e': 34},
{'a': 40, 'b': 41, 'c': 42, 'd': 43, 'e': 44}]
which you want to move to CSV-file, looking like
"a","b","c","d","e"
10,11,12,13,14
20,21,22,23,24
30,31,32,33,34
40,41,42,43,44
Problem is that when you start code:
def write_csv_from_list_dict(filename, table, fieldnames, separator, quote):
table = []
for dit in table:
a_row = []
for fieldname in fieldnames:
a_row.append(dit[fieldname])
table.append(a_row)
file_handle = open(filename, 'wt', newline='')
csv_write = csv.writer(file_handle,
delimiter=separator,
quotechar=quote,
quoting=csv.QUOTE_NONNUMERIC)
csv_write.writerow(fieldnames)
for row in table:
csv_write.writerow(row)
file_handler.close()
raising error
(Exception: AttributeError) "'list' object has no attribute 'keys'"
at line 148, in _dict_to_list wrong_fields = rowdict.keys() - self.fieldnames
Why to be so hard to say, explicitly to close a file, not a string.
The below code should work
data = [{'a': 10, 'b': 11, 'c': 12, 'd': 13, 'e': 14},
{'a': 20, 'b': 21, 'c': 22, 'd': 23, 'e': 24},
{'a': 30, 'b': 31, 'c': 32, 'd': 33, 'e': 34},
{'a': 40, 'b': 41, 'c': 42, 'd': 43, 'e': 44}]
keys = data[0].keys()
with open('data.csv', 'w') as f:
f.write(','.join(keys) + '\n')
for entry in data:
f.write(','.join([str(v) for v in entry.values()]) + '\n')
data.csv
a,b,c,d,e
10,11,12,13,14
20,21,22,23,24
30,31,32,33,34
40,41,42,43,44
Example:
[{"a":{"x":13, "y":32, "z":33}, "b":5, "c":7, "d":8, "e":9}, {"a":{"x":18, "y":28, "z":38}, "b":57, "c":77, "d":87, "e":97}, {"a":{"x":17, "y":72, "z":73}, "b":58, "c":70, "d":80, "e":90}, ...]
This is just a small sample set, but what I would like is a list with a filtered list of items in each dictionary such as below:
Sample Output:
[{"x":13, "b":5, "e"9}, {"x":18, "b":57, "e"97}, {"x":17, "b":58, "e"90}, ...]
I can filter it down to the following:
[{"a":{"x":13, "y":32, "z":33}, "b":5, "e":9}, {"a":{"x":18, "y":28, "z":38}, "b":57, "e":97}, {"a":{"x":17, "y":72, "z":73}, "b":58, "e":90}, ...]
using the following code
for i in range(len(results)):
desired_keys = ['a', 'b', 'e']
bigdict = all_results[i]
filtered = {x: bigdict[x] for x in desired_keys if x in bigdict}
but have yet to be able to figure out how to get the one element of the nested dictionary out.
You cannot just use your approach since it only works for top-level keys. You will need to specify each key and how to access it from the nested dictionary:
>>> [{'x': e['a']['x'], 'b': e['b'], 'e': e['e']} for e in results]
[{'x': 13, 'b': 5, 'e': 9}, {'x': 18, 'b': 57, 'e': 97}, {'x': 17, 'b': 58, 'e': 90}, ...]
As mentioned, all the items in the nested dictionaries must be visited.
This recursive approach
vals = [{"a":{"x":13, "y":32, "z":33}, "b":5, "c":7, "d":8, "e":9}, {"a":{"x":18, "y":28, "z":38}, "b":57, "c":77, "d":87, "e":97}, {"a":{"x":17, "y":72, "z":73}, "b":58, "c":70, "d":80, "e":90}]
def get_items(d, keys):
res = dict()
for k, v in d.items():
if isinstance(v, dict):
res.update(get_items(v, keys))
elif k in keys:
res[k] = v
return res
r = [get_items(d, {'x','b', 'e'}) for d in vals]
print(r)
produces
[{'x': 13, 'b': 5, 'e': 9}, {'x': 18, 'b': 57, 'e': 97}, {'x': 17, 'b': 58, 'e': 90}]
Note: make sure the keys do not appear more than once in any given path along the nested dictionaries.
Another possible recursive approach with a generator function:
def get_vals(d, to_find):
for a, b in d.items():
if a in to_find:
yield (a, b)
yield from [] if not isinstance(b, dict) else get_vals(b, to_find)
data = [{"a":{"x":13, "y":32, "z":33}, "b":5, "c":7, "d":8, "e":9}, {"a":{"x":18, "y":28, "z":38}, "b":57, "c":77, "d":87, "e":97}, {"a":{"x":17, "y":72, "z":73}, "b":58, "c":70, "d":80, "e":90}]
result = [dict(get_vals(i, ['x', 'b', 'e'])) for i in data]
Output:
[{'x': 13, 'b': 5, 'e': 9}, {'x': 18, 'b': 57, 'e': 97}, {'x': 17, 'b': 58, 'e': 90}]
I need to create a list of dictionaries from a dict. The original dict will have to will have keys have do not repeat but values can. For example:
{ 'b': 68, 'c': 68, 'x': 68, 'z': 401, 'aa':401, 'a': 2}
I need to create a list of dicts from the above that would look like this:
[{68: ['b', 'c', 'x']}, {401: ['z', 'aa']}, {2: ['a']}]
What have I tried? I have tried this and it works but I am certain there is probably are better way to accomplish the same result.
lofd=[]
origdict = { 'b': 68, 'c': 68, 'x': 68, 'z': 401, 'aa':401, 'a': 2}
for i in origdict.items():
k = i[0]
v = i[1]
try:
d[v].append(k)
except:
d = {v:[]}
d[v].append(k)
if d not in lofd:
lofd.append(d)
lofd
[{68: ['b', 'c', 'x']}, {401: ['z', 'aa']}, {2: ['a']}]
I have seen other questions that one might consider duplicate to this question, but the answers to those questions do not cover how to convert a dictionary to a list of dictionaries. There are plenty of answer that should how to invert a dictionary which is kinda like what I am trying to do, but I am not inverting the dictionary. There are answers that show create a tuple of tuples from a dict, but I do need a list of dictionaries, not immutable tuples.
You can do this using a list comprehension as well:
[dict((_, )) for _ in {v: [_k for _k, _v in d.items() if _v == v] for v in d.values()}.items()]
To show it works
>>> d = { 'b': 68, 'c': 68, 'x': 68, 'z': 401, 'aa':401, 'a': 2}
>>> [dict((_, )) for _ in {v: [_k for _k, _v in d.items() if _v == v] for v in d.values()}.items()]
[{68: ['b', 'c', 'x']}, {401: ['z', 'aa']}, {2: ['a']}]
Note I don't particularly know what usage this is for but If you could I'd suggest using a dict instead of a list of dicts as this comprehension is slower than just the dict comprehension alone. If possible I'd suggest using just the dict comprehension which would be much better in my opinion although returns a dict instead of a list of dicts which is more pythonic or at least makes more sense especially since you can iterate over it the same with dict.items().
{v: [_k for _k, _v in d.items() if _v == v] for v in d.values()}
This returns a dict as follows:
{68: ['b', 'c', 'x'], 401: ['z', 'aa'], 2: ['a']}
I would suggest the following. Untested.
from collections import defaultdict
a = { 'b': 66, 'b': 68, 'c': 68, 'x': 68, 'z': 401, 'aa':401, 'a': 2}
t = defaultdict(list)
for k, v in a.items():
t[v].append(k)
o = [dict((p,)) for p in t.items()]
Figure out if the list contains the wanted key, and if it does, append the value to the list. Otherwise, append a new dictionary:
a = { 'b': 66, 'b': 68, 'c': 68, 'x': 68, 'z': 401, 'aa':401, 'a': 2 }
b = []
for key, val in a.items():
if (any(val in d for d in b)):
b[next(i for i, d in enumerate(b) if val in d)][val].append(key)
else:
d = {}
d[val] = [key]
b.append(d)
print(b)
Output:
[{68: ['b', 'c', 'x']}, {401: ['z', 'aa']}, {2: ['a']}]
Maybe something like that.
a = { 'b': 68, 'c': 68, 'x': 68, 'z': 401, 'aa':401, 'a': 2}
b = {}
for k, v in a.items():
b.setdefault(v, []).append(k)
c = [{k: b[k]} for k in b]
print(c)
Output:
[{68: ['b', 'c', 'x']}, {401: ['z', 'aa']}, {2: ['a']}]
Somewhat same but a bit optimized:
a = { 'b': 66, 'b': 68, 'c': 68, 'x': 68, 'z': 401, 'aa':401, 'a': 2}
b = {}
for k,v in a.items():
val = b.get(v,[])
val.append(k)
b[v] = val
lst = []
for k,v in b.items():
d = dict()
d[k] = v
lst.append(d)
print(lst)
Output:
[{68: ['b', 'c', 'x']}, {401: ['z', 'aa']}, {2: ['a']}]
use defuhaltdict in this case
from collections import defaultdict
dic = defaultdict(list)
k= { 'b': 66, 'b': 68, 'c': 68, 'x': 68, 'z': 401, 'aa':401, 'a': 2}
for ke, v in k.items():
dic[v]+=[ke]
print(dict(dic))
output
{68: ['b', 'c', 'x'], 401: ['z', 'aa'], 2: ['a']}
Enumerate() method adds a counter to an iterable and returns it in a form of enumerate object.
origdict = { 'b': 66, 'b': 68, 'c': 68, 'x': 68, 'z': 401, 'aa':401, 'a': 2}
my_list = []
my_dict = {}
for index, (key, value) in enumerate(origdict.items()):
if value in my_dict:
my_dict[value].append(key)
else:
#validate is not first index of list
if index > 0:
my_list.append(my_dict)
my_dict = {}
my_dict[value] = [key]
#validate last index of list
if index == len(origdict)-1:
my_list.append(my_dict)
print(my_list)
O/P:
[{68: ['b', 'c', 'x']}, {401: ['z', 'aa']}, {2: ['a']}]
I know I can write a list of dictionaries directly to a CSV file. Similarly, is there a direct way to write a list of list of dictionaries to a csv file in python without iterating through each line manually?
Sample data
[[{'e': 46, 'p': 100, 'n': 0, 'a': 100, ...},
{'e': 29, 'p': 40, 'n': 1, 'a': 40, ...}, ...],
[{...}, ...]
Expected format
e,p,n,a,....
46,100,0,100,....
29,40,1,40,...
.......
Note this is not a list of dictionaries, but a list of list of dictionaries
Without Pandas, you can use itertools.chain to get a flattened list of all dictionaries and then write that to your CSV file with csv.DictWriter:
import csv
from itertools import chain
data = [
[{'e': 46, 'p': 100, 'n': 0, 'a': 100},
{'e': 29, 'p': 40, 'n': 1, 'a': 40}],
[{'e': 56, 'p': 200, 'n': 23, 'a': 10},
{'e': 22, 'p': 41, 'n': 11, 'a': 420}]]
fieldnames = ['e', 'p', 'n', 'a']
with open('mydata.csv', 'w') as f:
writer = csv.DictWriter(f, fieldnames=fieldnames)
writer.writeheader()
writer.writerows(chain.from_iterable(data))
Output (mydata.csv)
e,p,n,a
46,100,0,100
29,40,1,40
56,200,23,10
22,41,11,420
There must be a way to accomplish your task using just core Python, but I would go for Pandas:
import pandas as pd
d = yourListOfListsOfDictionaries
df = pd.concat(map(pd.DataFrame, d), sort=True)
# a e n p
#0 100 46 0 100
#0 40 29 1 40
df.to_csv(index=False)
#'a,e,n,p\n100,46,0,100\n40,29,1,40\n'
If you want the set of keys to be the union of all of the dictionaries in the list of list of dicts, then you can do something like this:
import csv
x = \
[[{'e': 46, 'p': 100, 'n': 0, 'a': 100},
{'e': 29, 'p': 40, 'n': 1, 'a': 40}],
[{'e': 19, 'p': 10, 'n': 1, 'a': 10, 'b':8}]]
key_dict = {}
for l in x:
for d in l:
for k in d:
key_dict[k] = None
with open('file.csv', 'w') as csvfile:
writer = csv.DictWriter(csvfile, key_dict.keys())
writer.writeheader()
for l in x:
for d in l:
writer.writerow(d)
Result:
a,p,b,e,n
100,100,,46,0
40,40,,29,1
10,10,8,19,1
If the dictionaries have the same format, just flatten the list like so (assuming it's indeed a list of lists, two-dimensional):
data = [
[{'a': 10, 'b': 20, 'c': 30}],
[{'a': 20, 'b': 30, 'c': 40},
{'a': 30, 'b': 40, 'c': 50}]
]
rows = [item for sublist in data for item in sublist]
Then just write your rows to the CSV:
with open('my_data.csv', 'wb') as output_file:
dict_writer = csv.DictWriter(output_file, rows[0].keys())
dict_writer.writeheader()
dict_writer.writerows(rows)
A combination of the two following posts:
How to make a flat list out of list of lists?
How do I convert this list of dictionaries to a csv file?