I have an array containing an even number of integers. The array represents a pairing of an identifier and a count. The tuples have already been sorted by the identifier. I would like to merge a few of these arrays together. I have thought of a few ways to do it but they are fairly complicated and I feel there might be an easy way to do this with python.
IE:
[<id>, <count>, <id>, <count>]
Input:
[14, 1, 16, 4, 153, 21]
[14, 2, 16, 3, 18, 9]
Output:
[14, 3, 16, 7, 18, 9, 153, 21]
It would be better to store these as dictionaries than as lists (not just for this purpose, but for other use cases, such as extracting the value of a single ID):
x1 = [14, 1, 16, 4, 153, 21]
x2 = [14, 2, 16, 3, 18, 9]
# turn into dictionaries (could write a function to convert)
d1 = dict([(x1[i], x1[i + 1]) for i in range(0, len(x1), 2)])
d2 = dict([(x2[i], x2[i + 1]) for i in range(0, len(x2), 2)])
print d1
# {16: 4, 153: 21, 14: 1}
After that, you could use any of the solutions in this question to add them together. For example (taken from the first answer):
import collections
def d_sum(a, b):
d = collections.defaultdict(int, a)
for k, v in b.items():
d[k] += v
return dict(d)
print d_sum(d1, d2)
# {16: 7, 153: 21, 18: 9, 14: 3}
collections.Counter() is what you need here:
In [21]: lis1=[14, 1, 16, 4, 153, 21]
In [22]: lis2=[14, 2, 16, 3, 18, 9]
In [23]: from collections import Counter
In [24]: dic1=Counter(dict(zip(lis1[0::2],lis1[1::2])))
In [25]: dic2=Counter(dict(zip(lis2[0::2],lis2[1::2])))
In [26]: dic1+dic2
Out[26]: Counter({153: 21, 18: 9, 16: 7, 14: 3})
or :
In [51]: it1=iter(lis1)
In [52]: it2=iter(lis2)
In [53]: dic1=Counter(dict((next(it1),next(it1)) for _ in xrange(len(lis1)/2)))
In [54]: dic2=Counter(dict((next(it2),next(it2)) for _ in xrange(len(lis2)/2)))
In [55]: dic1+dic2
Out[55]: Counter({153: 21, 18: 9, 16: 7, 14: 3})
Use collections.Counter:
import itertools
import collections
def grouper(n, iterable, fillvalue=None):
args = [iter(iterable)] * n
return itertools.izip_longest(fillvalue=fillvalue, *args)
count1 = collections.Counter(dict(grouper(2, lst1)))
count2 = collections.Counter(dict(grouper(2, lst2)))
result = count1 + count2
I've used the itertools library grouper recipe here to convert your data to dictionaries, but as other answers have shown you there are more ways to skin that particular cat.
result is a Counter with each id pointing to a total count:
Counter({153: 21, 18: 9, 16: 7, 14: 3})
Counters are multi-sets and will keep track of the count of each key with ease. It feels like a much better data structure for your data. They support summing, as used above, for example.
All of the previous answers look good, but I think that the JSON blob should be properly formed to begin with or else (from my experience) it can cause some serious problems down the road during debugging etc. In this case with id and count as the fields, the JSON should look like
[{"id":1, "count":10}, {"id":2, "count":10}, {"id":1, "count":5}, ...]
Properly formed JSON like that is much easier to deal with, and probably similar to what you have coming in anyway.
This class is a bit general, but certainly extensible
from itertools import groupby
class ListOfDicts():
def init_(self, listofD=None):
self.list = []
if listofD is not None:
self.list = listofD
def key_total(self, group_by_key, aggregate_key):
""" Aggregate a list of dicts by a specific key, and aggregation key"""
out_dict = {}
for k, g in groupby(self.list, key=lambda r: r[group_by_key]):
print k
total=0
for record in g:
print " ", record
total += record[aggregate_key]
out_dict[k] = total
return out_dict
if __name__ == "__main__":
z = ListOfDicts([ {'id':1, 'count':2, 'junk':2},
{'id':1, 'count':4, 'junk':2},
{'id':1, 'count':6, 'junk':2},
{'id':2, 'count':2, 'junk':2},
{'id':2, 'count':3, 'junk':2},
{'id':2, 'count':3, 'junk':2},
{'id':3, 'count':10, 'junk':2},
])
totals = z.key_total("id", "count")
print totals
Which gives
1
{'count': 2, 'junk': 2, 'id': 1}
{'count': 4, 'junk': 2, 'id': 1}
{'count': 6, 'junk': 2, 'id': 1}
2
{'count': 2, 'junk': 2, 'id': 2}
{'count': 3, 'junk': 2, 'id': 2}
{'count': 3, 'junk': 2, 'id': 2}
3
{'count': 10, 'junk': 2, 'id': 3}
{1: 12, 2: 8, 3: 10}
Related
I've searched quite a lot but I haven't found any similar question to that one.
I have two lists of dictionaries in following format:
data1 = [
{'id': 4, 'date_time': datetime.datetime(2020, 4, 3, 12, 34, 40)},
{'id': 4, 'date_time': datetime.datetime(2020, 4, 3, 12, 34, 40)},
{'id': 6, 'date_time': datetime.datetime(2020, 4, 3, 12, 34, 40)},
{'id': 7, 'date_time': datetime.datetime(2020, 4, 3, 16, 14, 21)},
]
data2 = [
{'id': 4, 'date_time': datetime.datetime(2020, 4, 3, 12, 34, 40)},
{'id': 6, 'date_time': datetime.datetime(2020, 4, 3, 12, 34, 40)},
]
desired output:
final_data = [
{'id': 4, 'date_time': datetime.datetime(2020, 4, 3, 12, 34, 40)},
{'id': 7, 'date_time': datetime.datetime(2020, 4, 3, 16, 14, 21)},
]
I want only dictionaries which are in data1 and not in data2.
Until now when I found a match in two for loops I popped the dictionary out of the list but that does not seem like a good approach to me. How can I achieve desired output?
It doesn't have to be time efficient since there will be max tens of dictionaries in each list
Current implementation:
counter_i = 0
for i in range(len(data1)):
counter_j = 0
for j in range(len(data2)):
if data1[i-counter_i]['id'] == data2[j-counter_j]['id'] and data1[i-counter_i]['date_time'] == data2[j-counter_j]['date_time']
data1.pop(i-counter_i)
data2.pop(j-counter_j)
counter_i += 1
counter_j += 1
break
If performance is not an issue, why not:
for d in data2:
try:
data1.remove(d)
except ValueError:
pass
list.remove checks for object equality, not identity, so will work for dicts with equal keys and values. Also, list.remove only removes one occurrence at a time.
schwobaseggl's answer is probably the cleanest solution (just make a copy before removing if you need to keep data1 intact).
But if you want to use a set difference... well dicts are not hashable, because their underlying data could change and lead to issues (same reason why lists or sets are not hashable either).
However, you can get all the dict pairs in a frozenset to represent a dictionary (assuming the dictionary values are hashable -schwobaseggl). And frozensets are hashable, so you can add those to a set a do normal set difference. And reconstruct the dictionaries at the end :D.
I don't actually recommend doing it, but here we go:
final_data = [
dict(s)
for s in set(
frozenset(d.items()) for d in data1
).difference(
frozenset(d.items()) for d in data2
)
]
you can go in either way:
Method 1:
#using filter and lambda function
final_data = filter(lambda i: i not in data2, data1)
final_data = list(final_data)
Method 2:
# using list comprehension to perform task
final_data = [i for i in data1 if i not in data2]
I need to make two dictionaries from list1:
list1 = [['8/16/2016 9:55', 6], ['11/22/2015 13:43', 29], ['5/2/2016 10:14', 1],
['8/2/2016 14:20', 3], ['10/15/2015 16:38', 17], ['9/26/2015 23:23', 1],
['4/22/2016 12:24', 4], ['11/16/2015 9:22', 1], ['2/24/2016 17:57', 1],
['6/4/2016 17:17', 2]]
count_by_hour = {} # this is created by extracting the hour from index[0] of list1
I was able to get this with help from replies to my previously posted question.
for each in list1:
if each[0].split(':')[0][-2] == " ": #split by : to get second last char and check if >9
hours.append(each[0].split(':')[0][-1:]) # if hour is <9 take last char which is hour
else:
hours.append(each[0].split(':')[0][-2:]) #else take last 2 chars
print('Hour extracted:')
print(hours)
Output:
Counts by hour:
{'9': 2, '13': 1, '10': 1, '14': 1, '16': 1, '23': 1, '12': 1, '17': 2}
Now, how do I do the following:
comments_by_hour = {}
Expected Outcome:
{9:7, 13:29, 10:1, 14:3, 16:17, 23:1, 12:4, 17:2} #value is a total for every hour that exists as a key in list1
As always, any help is appreciated.
Notice that we need to accumulate the sum separately for each of many categories (hours). A simple solution (in pure Python) combines the accumulator pattern while using a dictionary to store all the counts.
First, let's use time.strptime to extract the hours using a list comprehension.
In [1]: list1 = [['8/16/2016 9:55', 6], ['11/22/2015 13:43', 29], ['5/2/2016 10:14', 1],
: ['8/2/2016 14:20', 3], ['10/15/2015 16:38', 17], ['9/26/2015 23:23', 1],
: ['4/22/2016 12:24', 4], ['11/16/2015 9:22', 1], ['2/24/2016 17:57', 1],
: ['6/4/2016 17:17', 2]]
In [2]: from time import strptime
In [3]: hour_list = [(strptime(time, "%m/%d/%Y %H:%M").tm_hour, val) for time, val in list1]
The solution is to use a dictionary to accumulate statistics for each category. Do this by (a) starting with an empty dictionary and (b) updating the sums for each new value. This can be accomplished as follows.
In [4]: comments_by_hour = {}
In [5]: for hour, val in hour_list:
: comments_by_hour[hour] = val + comments_by_hour.get(hour, 0)
:
In [6]: comments_by_hour
Out[6]: {9: 7, 13: 29, 10: 1, 14: 3, 16: 17, 23: 1, 12: 4, 17: 3}
Note that comments_by_hour.get(hour, 0) used to get the current value for that hour, if it exists, or using the default value of 0 otherwise.
I have a list of dictionary as:
[{
'doc1': {'hyundai': 12,
'mercedez': 34,
'bugatti': 2,
'honda': 19},
'doc2': {'tennis': 20,
'wimbledon': 11,
'nadal': 57},
'doc3':{'world': ,
'politics': 8,
'obama': 7,
'america': 4,
'summit': 14,
'budget': 17,
'germany': 22
}}]
and list of words l= ['trump','mercedez','wimbledon','nadal','hyundai','tennis']
All I want to match the list elements with keys of dictionary and get the sum of its values with sorted along with highest sum and if none elements get matched then sum will be zero for corresponding key.
Expected as,
new_dict={'doc2':88,'doc1':46,'doc3'=0}
you can iterate using .items(),
then iterate the inner dicts the same way.
like this:
d_list = [{'doc1': {'hyundai': 12,
'mercedez': 34,
'bugatti': 2,
'honda': 19},
'doc2': {'tennis': 20,
'wimbledon': 11,
'nadal': 57},
'doc3': {'world': 0,
'politics': 8,
'obama': 7,
'america': 4,
'summit': 14,
'budget': 17,
'germany': 22}}]
l = ['trump', 'mercedez', 'wimbledon', 'nadal', 'hyundai', 'tennis']
result_d = {}
for doc_name, inner_dict in d_list[0].items():
result_d[doc_name] = sum(v for k,v in inner_dict.items() if k in l)
new_list = [result_d]
print(new_list)
Output:
[{'doc1': 46, 'doc2': 88, 'doc3': 0}]
Try this,
In [131]: A = [{'doc1': {'hyundai': 12, 'mercedez': 34, 'bugatti': 2, 'honda': 19},
...: 'doc2': {'tennis': 20, 'wimbledon': 11, 'nadal': 57},
...: 'doc3': {'world': 0, 'politics': 8, 'obama': 7, 'america': 4, 'summit': 14, 'budget': 17, 'germany': 22}}]
In [132]: B = ['trump', 'mercedez', 'wimbledon', 'nadal', 'hyundai', 'tennis']
In [133]: dict_comprehension = {key: sum([value.get(b, 0) for b in B]) for key, value in A[0].items()}
In [134]: result = dict(sorted(dict_comprehension.items(), key=lambda x: x[1], reverse=True))
In [135]: result
Out[135]: {'doc2': 88, 'doc1': 46, 'doc3': 0}
I got your expected output using dict comprehensions.
I have 2 Lists named 'speciality' and 'count', which are part of a Dictionary 'P' . I Zip sorted both the 'Lists' on Descending order of 'count' List.
speciality = ['Cardiology' , 'Nephrology', 'ENT', 'Opthalmology' 'Oncology']
count = [2, 7, 9, 9, 1]
count, speciality = zip(*[[x, y] for x, y in sorted(zip(count, speciality), reverse=True)])
P = {'Speciliaty': speciality, 'Count': count}
print(P)
# {'Speciliaty': ('Opthalmology', 'ENT', 'Nephrology', 'Cardiology', 'Oncology'), 'Count': (9, 9, 7, 2, 1)}
Please notice, the elements 'Opthalmology' and 'ENT' has the same count 9.
But, after doing the Zip Sort.
'Opthalmology' appeared before 'ENT' in the Output Tuple. In the Input the order is 'ENT' first then 'Opthalmology'.
Can we make the output like below:
P = {'Speciliaty': ('ENT', 'Opthalmology', 'Nephrology', 'Cardiology', 'Oncology'), 'Count': (9, 9, 7, 2, 1)}
You need to set the key in sorted to sort by count.
Ex:
speciality = ['Cardiology' , 'Nephrology', 'ENT', 'Opthalmology', 'Oncology']
count = [2, 7, 9, 9, 1]
count, speciality = zip(*[[x, y] for x, y in sorted(zip(count, speciality), key=lambda x: x[0], reverse=True)])
P = {'Speciliaty': speciality, 'Count': count}
print(P)
Output:
{'Count': (9, 9, 7, 2, 1), 'Speciliaty': ('ENT', 'Opthalmology', 'Nephrology', 'Cardiology', 'Oncology')}
Is there a method of logically merging multiple dictionaries if they have common strings between them? Even if these common strings match between values of one dict() to a key of another?
I see a lot of similar questions on SO but none that seem to address my specific issue of relating multiple keys in "lower level files" to those in higher keys/values(level1dict)
Say we have:
level1dict = { '1':[1,3], '2':2 }
level2dict = { '1':4, '3':[5,9], '2':10 }
level3dict = { '1':[6,8,11], '4':12, '2':13, '3':[14,15], '5':16, '9':17, '10':[18,19,20]}
finaldict = level1dict
When I say logically I mean, in level1dict 1=1,3 and in level2dict 1=4 and 3=5,9 so overall (so far) 1 = 1,3,4,5,9 (sorting not important)
The result I would like to get to is
#.update or .append or .default?
finaldict = {'1':[1,3,4,5,9,6,8,11,12,14,15,16,17] '2':[2,10,18,19,20]}
Answered: Thank you Ashwini Chaudhary and Abhijit for the networkx module.
This is a problem of connected component subgraphs and can be best determined if you want to use networkx. Here is a solution to your problem
>>> import networkx as nx
>>> level1dict = { '1':[1,3], '2':2 }
>>> level2dict = { '1':4, '3':[5,9], '2':10 }
>>> level3dict = { '1':[6,8,11], '4':12, '2':13, '3':[14,15], '5':16, '9':17, '10':[18,19,20]}
>>> G=nx.Graph()
>>> for lvl in level:
for key, value in lvl.items():
key = int(key)
try:
for node in value:
G.add_edge(key, node)
except TypeError:
G.add_edge(key, value)
>>> for sg in nx.connected_component_subgraphs(G):
print sg.nodes()
[1, 3, 4, 5, 6, 8, 9, 11, 12, 14, 15, 16, 17]
[2, 10, 13, 18, 19, 20]
>>>
Here is how you visualize it
>>> import matplotlib.pyplot as plt
>>> nx.draw(G)
>>> plt.show()
A couple of notes:
It's not convenient that some values are numbers and some are lists. Try converting numbers to 1-item lists first.
If the order is not important, you'll be better off using sets instead of lists. They have methods for all sorts of "logical" operations.
Then you can do:
In [1]: dict1 = {'1': {1, 3}, '2': {2}}
In [2]: dict2 = {'1': {4}, '2': {10}, '3': {5, 9}}
In [3]: dict3 = {'1': {6, 8, 11}, '2': {13}, '4': {12}}
In [4]: {k: set.union(*(d[k] for d in (dict1, dict2, dict3)))
for k in set.intersection(*(set(d.keys()) for d in (dict1, dict2, dict3)))}
Out[4]: {'1': set([1, 3, 4, 6, 8, 11]), '2': set([2, 10, 13])}
In [106]: level1dict = { '1':[1,3], '2':2 }
In [107]: level2dict = { '1':4, '3':[5,9], '2':10 }
In [108]: level3dict = { '1':[6,8,11], '4':12, '2':13, '3':[14,15], '5':16, '9':17, '10':[18,19,20]}
In [109]: keys=set(level2dict) & set(level1dict) & set(level3dict) #returns ['1','2']
In [110]: dic={}
In [111]: for key in keys:
dic[key]=[]
for x in (level1dict,level2dict,level3dict):
if isinstance(x[key],int):
dic[key].append(x[key])
elif isinstance(x[key],list):
dic[key].extend(x[key])
.....:
In [112]: dic
Out[112]: {'1': [1, 3, 4, 6, 8, 11], '2': [2, 10, 13]}
# now iterate over `dic` again to get the values related to the items present
# in the keys `'1'` and `'2'`.
In [122]: for x in dic:
for y in dic[x]:
for z in (level1dict,level2dict,level3dict):
if str(y) in z and str(y) not in dic:
if isinstance(z[str(y)],(int,str)):
dic[x].append(z[str(y)])
elif isinstance(z[str(y)],list):
dic[x].extend(z[str(y)])
.....:
In [123]: dic
Out[123]:
{'1': [1, 3, 4, 6, 8, 11, 5, 9, 14, 15, 12, 16, 17],
'2': [2, 10, 13, 18, 19, 20]}