Related
I have a dataset where i groupby the monthly data with the same id:
temp1 = listvar[2].groupby(["id", "month"])["value"].mean()
This results in this:
id month
SN10380 1 -9.670370
2 -8.303571
3 -4.932143
4 0.475862
5 5.732000
...
SN99950 8 6.326786
9 4.623529
10 1.290566
11 -0.867273
12 -2.485455
I then want to have each month and the corresponding value as a own column on the same ID, like this:
id month_1 month_2 month_3 month_4 .... month_12
SN10380 -9.670370 -8.303571 .....
SN99950
I have tried different solutions using apply(), transform() and agg(), but aren't able to produce the wanted output.
You could use unstack. Here's the sample code:
import pandas as pd
df = pd.DataFrame({
"id": [1, 1, 1, 1, 1, 2, 2, 2, 2, 2],
"month": [1, 2, 3, 4, 5, 1, 2, 3, 4, 5],
"value": [11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
})
temp1 = df.groupby(["id", "month"])["value"].mean()
temp1.unstack()
I hope it helps!
I want to declare a set of variables with PuLP which contains all the possible combinations of the following lists:
month = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
manufacturer = ['China', 'Mexico', 'Taiwan']
demand = ['London', 'Paris', 'Milan']
Then, I will have a dictionary (for example) as follow:
'1.China.London', '1.China.Paris',...
I tried with the following code, but I don't know how to store all the combinations.
vlbs = {}
for key in month:
for kay in manufacturer:
for eyk in demand:
vlbs = (str(key)+'.'+str(kay)+'.'+str(eyk))
First, I'm not getting properly the dictionary vlbs. And later on:
variables = {var: pl.LpVariable(var, lowBound = 0) for var in vlbs}
How can I solve it properly??
You can use a tuple as a dictionary key. I think this makes filtering/searching easier than doing a bunch of string concatenating and splitting.
months = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
manufacturers = ['China', 'Mexico', 'Taiwan']
demands = ['London', 'Paris', 'Milan']
var_dict = {}
for month in months:
for manufacturer in manufacturers:
for demand in demands:
combo = (month, manufacturer, demand)
var_name = '.'.join([str(c) for c in combo])
var_dict[combo] = LpVariable(var_name, lowBound=0)
# add constraint for month 1
model += lpSum([k for k in var_dict if k[0] == 1]) <= 50
The dict got the keys years and for each year it's a list of all the temperatures in all 12 months of that year. My goal is to print out a table starting with what year it is and then a new line for each month and the temp that month.
The main thing is to mark the highest temp of all years with (ATH) and mark the highest temp in each year with (YearHighest).
My current code:
temp_dict= {
"2010": [2, 3, 4, 5, 7, 3, 20, 29, 34, 2, 10, 1],
"2011": [2, 7, 4, 5, 9, 3, 20, 9, 34, 2, 10, 10]
}
for key in temp_dict:
print("Year",key,":")
x=0
for temps in temp_dict[key]:
x=x+1
print("Month "+str(x)+":%3d"%temps)
print()
I'm not sure how to make the max function, I was thinking something like this but I can't get it to work:
for key in temp_dict:
ATH = temp_dict[key]
YearHigh = temp_dict[key][0]
for temps in temp_dict[key]:
if temps >= temp_dict[key][0]:
YearHigh = temps
if YearHigh >= ATH:
ATH = YearHigh
How I want my output to look like:
Year 2011 :
Month1: 2
Month2: 7
Month3: 4
Month4: 5
Month5: 9
Month6: 3
Month7: 20
Month8: 9
Month9: 34 (YearHighest)(ATH)
Month10: 2
Month11: 10
Month12: 10
Year 2010 :
Month1: 2
Month2: 3
Month3: 4
Month4: 5
Month5: 7
Month6: 3
Month7: 20
Month8: 29
Month9: 34 (YearHighest)(ATH)
Month10: 2
Month11: 10
Month12: 1
Python has built-in function max, it's considered a good practice to use it.
Max in year:
max(temp_dict["2010"])
Max all time:
max(sum(temp_dict.values(), []))
sum(lists, []) does list flattening, equivalent to
[] + lists[0] + lists[1]...
Python has a builtin function max you can utilize:
for key in temp_dict:
print("Year", key,":")
temps = temp_dict[key]
max_temp = max(temps)
max_index = temps.index(max_temp)
for index, temps in enumerate(temps):
r = "Month "+str(index+1)+":%3d"%temps
if index == max_index:
r += "(YearHighest)(ATH)"
print(r)
You can try something like this:
temp_dict= {
"2010": [2, 3, 4, 5, 7, 3, 20, 29, 34, 2, 10, 1],
"2011": [2, 7, 4, 5, 9, 3, 20, 9, 34, 2, 10, 10]
}
# defines the max of all years with a list comprehension
global_max_temp = max([ max(year_temps) for year_temps in temp_dict.values() ])
# iterates through each year
for year, temps in temp_dict.items():
print("Year {}".format(year))
for i, temp in enumerate(temps):
# prepares the output
temp_string = ["Month{}: {}".format(i+1, temp)]
# builds a list of flags to be displayed
flags = []
if temp == max(temps):
# max in year flag
flags.append("YearHighest")
if temp == global_max_temp:
# absolute max flag
flags.append("ATH")
# joins temp_string and flags in a single line and prints it
print(" ".join(temp_string + [ "({})".format(flag) for flag in flags ]))
Useful links from Python's documentation: enumerate, list comprehensions, max
This is my code.
temp_dict= {
"2010": [2, 3, 4, 5, 7, 3, 20, 29, 34, 2, 10, 1],
"2011": [2, 7, 4, 5, 9, 3, 20, 9, 34, 2, 10, 10]
}
# Find the highest temp of all years
ath = max([ max(v) for v in temp_dict.values()])
for key in temp_dict:
# Output Year
print("Year{k}:".format(k=key))
x=0
# Find max
max_value = max(temp_dict[key])
for temps in temp_dict[key]:
# Output Month
x=x+1
s = "Month {x}:{v:3d}".format(x=str(x), v=temps)
# Tag the max value
if max_value == temps:
s += "(YearHighest)"
if ath == temps:
s += "(ATH)"
print(s)
print()
And this is my output.
Year2010:
Month 1: 2
Month 2: 3
Month 3: 4
Month 4: 5
Month 5: 7
Month 6: 3
Month 7: 20
Month 8: 29
Month 9: 34(YearHighest)(ATH)
Month 10: 2
Month 11: 10
Month 12: 1
Year2011:
Month 1: 2
Month 2: 7
Month 3: 4
Month 4: 5
Month 5: 9
Month 6: 3
Month 7: 20
Month 8: 9
Month 9: 34(YearHighest)(ATH)
Month 10: 2
Month 11: 10
Month 12: 10
Here needs to use max function. It can max value from numbers of a list fast.
I have an array containing an even number of integers. The array represents a pairing of an identifier and a count. The tuples have already been sorted by the identifier. I would like to merge a few of these arrays together. I have thought of a few ways to do it but they are fairly complicated and I feel there might be an easy way to do this with python.
IE:
[<id>, <count>, <id>, <count>]
Input:
[14, 1, 16, 4, 153, 21]
[14, 2, 16, 3, 18, 9]
Output:
[14, 3, 16, 7, 18, 9, 153, 21]
It would be better to store these as dictionaries than as lists (not just for this purpose, but for other use cases, such as extracting the value of a single ID):
x1 = [14, 1, 16, 4, 153, 21]
x2 = [14, 2, 16, 3, 18, 9]
# turn into dictionaries (could write a function to convert)
d1 = dict([(x1[i], x1[i + 1]) for i in range(0, len(x1), 2)])
d2 = dict([(x2[i], x2[i + 1]) for i in range(0, len(x2), 2)])
print d1
# {16: 4, 153: 21, 14: 1}
After that, you could use any of the solutions in this question to add them together. For example (taken from the first answer):
import collections
def d_sum(a, b):
d = collections.defaultdict(int, a)
for k, v in b.items():
d[k] += v
return dict(d)
print d_sum(d1, d2)
# {16: 7, 153: 21, 18: 9, 14: 3}
collections.Counter() is what you need here:
In [21]: lis1=[14, 1, 16, 4, 153, 21]
In [22]: lis2=[14, 2, 16, 3, 18, 9]
In [23]: from collections import Counter
In [24]: dic1=Counter(dict(zip(lis1[0::2],lis1[1::2])))
In [25]: dic2=Counter(dict(zip(lis2[0::2],lis2[1::2])))
In [26]: dic1+dic2
Out[26]: Counter({153: 21, 18: 9, 16: 7, 14: 3})
or :
In [51]: it1=iter(lis1)
In [52]: it2=iter(lis2)
In [53]: dic1=Counter(dict((next(it1),next(it1)) for _ in xrange(len(lis1)/2)))
In [54]: dic2=Counter(dict((next(it2),next(it2)) for _ in xrange(len(lis2)/2)))
In [55]: dic1+dic2
Out[55]: Counter({153: 21, 18: 9, 16: 7, 14: 3})
Use collections.Counter:
import itertools
import collections
def grouper(n, iterable, fillvalue=None):
args = [iter(iterable)] * n
return itertools.izip_longest(fillvalue=fillvalue, *args)
count1 = collections.Counter(dict(grouper(2, lst1)))
count2 = collections.Counter(dict(grouper(2, lst2)))
result = count1 + count2
I've used the itertools library grouper recipe here to convert your data to dictionaries, but as other answers have shown you there are more ways to skin that particular cat.
result is a Counter with each id pointing to a total count:
Counter({153: 21, 18: 9, 16: 7, 14: 3})
Counters are multi-sets and will keep track of the count of each key with ease. It feels like a much better data structure for your data. They support summing, as used above, for example.
All of the previous answers look good, but I think that the JSON blob should be properly formed to begin with or else (from my experience) it can cause some serious problems down the road during debugging etc. In this case with id and count as the fields, the JSON should look like
[{"id":1, "count":10}, {"id":2, "count":10}, {"id":1, "count":5}, ...]
Properly formed JSON like that is much easier to deal with, and probably similar to what you have coming in anyway.
This class is a bit general, but certainly extensible
from itertools import groupby
class ListOfDicts():
def init_(self, listofD=None):
self.list = []
if listofD is not None:
self.list = listofD
def key_total(self, group_by_key, aggregate_key):
""" Aggregate a list of dicts by a specific key, and aggregation key"""
out_dict = {}
for k, g in groupby(self.list, key=lambda r: r[group_by_key]):
print k
total=0
for record in g:
print " ", record
total += record[aggregate_key]
out_dict[k] = total
return out_dict
if __name__ == "__main__":
z = ListOfDicts([ {'id':1, 'count':2, 'junk':2},
{'id':1, 'count':4, 'junk':2},
{'id':1, 'count':6, 'junk':2},
{'id':2, 'count':2, 'junk':2},
{'id':2, 'count':3, 'junk':2},
{'id':2, 'count':3, 'junk':2},
{'id':3, 'count':10, 'junk':2},
])
totals = z.key_total("id", "count")
print totals
Which gives
1
{'count': 2, 'junk': 2, 'id': 1}
{'count': 4, 'junk': 2, 'id': 1}
{'count': 6, 'junk': 2, 'id': 1}
2
{'count': 2, 'junk': 2, 'id': 2}
{'count': 3, 'junk': 2, 'id': 2}
{'count': 3, 'junk': 2, 'id': 2}
3
{'count': 10, 'junk': 2, 'id': 3}
{1: 12, 2: 8, 3: 10}
Is there a method of logically merging multiple dictionaries if they have common strings between them? Even if these common strings match between values of one dict() to a key of another?
I see a lot of similar questions on SO but none that seem to address my specific issue of relating multiple keys in "lower level files" to those in higher keys/values(level1dict)
Say we have:
level1dict = { '1':[1,3], '2':2 }
level2dict = { '1':4, '3':[5,9], '2':10 }
level3dict = { '1':[6,8,11], '4':12, '2':13, '3':[14,15], '5':16, '9':17, '10':[18,19,20]}
finaldict = level1dict
When I say logically I mean, in level1dict 1=1,3 and in level2dict 1=4 and 3=5,9 so overall (so far) 1 = 1,3,4,5,9 (sorting not important)
The result I would like to get to is
#.update or .append or .default?
finaldict = {'1':[1,3,4,5,9,6,8,11,12,14,15,16,17] '2':[2,10,18,19,20]}
Answered: Thank you Ashwini Chaudhary and Abhijit for the networkx module.
This is a problem of connected component subgraphs and can be best determined if you want to use networkx. Here is a solution to your problem
>>> import networkx as nx
>>> level1dict = { '1':[1,3], '2':2 }
>>> level2dict = { '1':4, '3':[5,9], '2':10 }
>>> level3dict = { '1':[6,8,11], '4':12, '2':13, '3':[14,15], '5':16, '9':17, '10':[18,19,20]}
>>> G=nx.Graph()
>>> for lvl in level:
for key, value in lvl.items():
key = int(key)
try:
for node in value:
G.add_edge(key, node)
except TypeError:
G.add_edge(key, value)
>>> for sg in nx.connected_component_subgraphs(G):
print sg.nodes()
[1, 3, 4, 5, 6, 8, 9, 11, 12, 14, 15, 16, 17]
[2, 10, 13, 18, 19, 20]
>>>
Here is how you visualize it
>>> import matplotlib.pyplot as plt
>>> nx.draw(G)
>>> plt.show()
A couple of notes:
It's not convenient that some values are numbers and some are lists. Try converting numbers to 1-item lists first.
If the order is not important, you'll be better off using sets instead of lists. They have methods for all sorts of "logical" operations.
Then you can do:
In [1]: dict1 = {'1': {1, 3}, '2': {2}}
In [2]: dict2 = {'1': {4}, '2': {10}, '3': {5, 9}}
In [3]: dict3 = {'1': {6, 8, 11}, '2': {13}, '4': {12}}
In [4]: {k: set.union(*(d[k] for d in (dict1, dict2, dict3)))
for k in set.intersection(*(set(d.keys()) for d in (dict1, dict2, dict3)))}
Out[4]: {'1': set([1, 3, 4, 6, 8, 11]), '2': set([2, 10, 13])}
In [106]: level1dict = { '1':[1,3], '2':2 }
In [107]: level2dict = { '1':4, '3':[5,9], '2':10 }
In [108]: level3dict = { '1':[6,8,11], '4':12, '2':13, '3':[14,15], '5':16, '9':17, '10':[18,19,20]}
In [109]: keys=set(level2dict) & set(level1dict) & set(level3dict) #returns ['1','2']
In [110]: dic={}
In [111]: for key in keys:
dic[key]=[]
for x in (level1dict,level2dict,level3dict):
if isinstance(x[key],int):
dic[key].append(x[key])
elif isinstance(x[key],list):
dic[key].extend(x[key])
.....:
In [112]: dic
Out[112]: {'1': [1, 3, 4, 6, 8, 11], '2': [2, 10, 13]}
# now iterate over `dic` again to get the values related to the items present
# in the keys `'1'` and `'2'`.
In [122]: for x in dic:
for y in dic[x]:
for z in (level1dict,level2dict,level3dict):
if str(y) in z and str(y) not in dic:
if isinstance(z[str(y)],(int,str)):
dic[x].append(z[str(y)])
elif isinstance(z[str(y)],list):
dic[x].extend(z[str(y)])
.....:
In [123]: dic
Out[123]:
{'1': [1, 3, 4, 6, 8, 11, 5, 9, 14, 15, 12, 16, 17],
'2': [2, 10, 13, 18, 19, 20]}