Develop Reference

Python Groupby and Rolling Apply a self-defined function to multiple columns

I would like to groupby id, rolling 7 dates, and apply a simple OLS regression to each id and each period.
Here's my function:
endo_var = 'monthly_gross_ret'
exogenous_vars = ['bcmk_ret']
def get_r2(data_it):
data_it.exog = sm.add_constant(data_it[exogenous_vars], prepend=False)
data_it.endog = data_it[endo_var]
mod = sm.OLS(data_it.endog, data_it.exog)
res = mod.fit()
return res.rsquared
The fabricated sample data is:
ids = ['A','A','A','A','A','A','A','A','B','B','B','B','B','B','B','B']
dates = [1,2,3,4,5,6,7,8,1,2,3,4,5,6,7,8]
bcmk_rets = [4,5,6,3,7,8,3,5,1,5,8,4,7,3,7,3]
monthly_gross_rets = [8,4,7,2,5,7,2,6,8,3,6,6,2,7,3,2]
df = pd.DataFrame({'id':ids,'date':dates,'bcmk_ret':bcmk_rets,'monthly_gross_ret':monthly_gross_rets})
I would like to groupby id and rolling 7 dates then apply the function get_r2, something like this:
data.groupby('id').rolling(7).apply(get_r2)
However, the error says:KeyError: "None of [Index(['bcmk_ret'], dtype='object')] are in the [index]"
I know how to do this in for loop:
record_r2 = pd.DataFrame()
count = 0
for i in data['id'].unique():
data_i = data[(data['id']==i)]
for t in data_i['date'].unique():
data_it = data_i[(data_i['date']>=t) & (data_i['date']<=t + 6)]
if len(data_it) ==7: # should have put 24, but there's literally ZERO result, because the data quality is rather poor. (we have lots of missing months. For e.g., id == 6679 has May to Nov, then missing Dec and Jan, then have Feb to Dec...)
data_it.exog = sm.add_constant(data_it[exogenous_vars], prepend=False)
data_it.endog = data_it[endo_var]
mod = sm.OLS(data_it.endog, data_it.exog)
res = mod.fit()
record_r2.loc[count,'id'] = i
record_r2.loc[count,'date'] = t
record_r2.loc[count,'R2'] = res.rsquared
count = count + 1
The for loop returns the desired dataframe that records the R^2 of each group's regression results:
{'id': {0: 'A', 1: 'A', 2: 'B', 3: 'B'},
'date': {0: 1.0, 1: 2.0, 2: 1.0, 3: 2.0},
'R2': {0: 0.3512152777777777,
1: 0.7200000000000001,
2: 0.4736842105263158,
3: 0.026894573272770783},
'adj_R2': {0: 0.2214583333333333,
1: 0.6640000000000001,
2: 0.368421052631579,
3: -0.16772651207267497}}
But the for loop is computationaly inefficient and takes ages to run my full data sample. That's why I would love to use the groupby method instead of for loop. Could someone help out?
Thank you very much!
Best,
Darcy

How to set a column value by fuzzy string matching with another dataframe?

I have referred to this post but cannot get it to run for my particular case. I have two dataframes:
import pandas as pd
df1 = pd.DataFrame(
{
"ein": {0: 1001, 1: 1500, 2: 3000},
"ein_name": {0: "H for Humanity", 1: "Labor Union", 2: "Something something"},
"lname": {0: "Cooper", 1: "Cruise", 2: "Pitt"},
"fname": {0: "Bradley", 1: "Thomas", 2: "Brad"},
}
)
df2 = pd.DataFrame(
{
"lname": {0: "Couper", 1: "Cruise", 2: "Pit"},
"fname": {0: "Brad", 1: "Tom", 2: "Brad"},
"score": {0: 3, 1: 3.5, 2: 4},
}
)
Then I do:
from fuzzywuzzy import fuzz
from fuzzywuzzy import process
from itertools import product
N = 60
names = {
tup: fuzz.ratio(*tup)
for tup in product(df1["lname"].tolist(), df2["lname"].tolist())
}
s1 = pd.Series(names)
s1 = s1[s1 > N]
s1 = s1[s1.groupby(level=0).idxmax()]
degrees = {
tup: fuzz.ratio(*tup)
for tup in product(df1["fname"].tolist(), df2["fname"].tolist())
}
s2 = pd.Series(degrees)
s2 = s2[s2 > N]
s2 = s2[s2.groupby(level=0).idxmax()]
df2["lname"] = df2["lname"].map(s1).fillna(df2["lname"])
df2["fname"] = df2["fname"].map(s2).fillna(df2["fname"])
df = df1.merge(df2, on=["lname", "fname"], how="outer")
The result is not what I expect. Can you help me with editing this code please? Note that I have millions of lines in df1 and millions in df2, so I need some efficiency as well.
Basically, I need to match people from df1 to people in df2. In the above example, I am matching them on last name (lname) and first name (fname). I also have a third one, which I leave out here for parsimony.
The expected result should look like:
ein ein_name lname fname score
0 1001 H for Humanity Cooper Bradley 3
1 1500 Labor Union Cruise Thomas 3.5
2 3000 Something something Pitt Brad 4

You could try this:
from functools import cache
import pandas as pd
from fuzzywuzzy import fuzz
# First, define indices and values to check for matches
indices_and_values = [(i, value) for i, value in enumerate(df2["lname"] + df2["fname"])]
# Define helper functions to find matching rows and get corresponding score
#cache
def find_match(x):
return [i for i, value in indices_and_values if fuzz.ratio(x, value) > 75]
def get_score(x):
try:
return df2.loc[x[0], "score"]
except (KeyError, IndexError):
return pd.NA
# Add scores to df1:
df1["score"] = (
(df1["lname"] + df1["fname"])
.apply(find_match)
.apply(get_score)
)
And then:
print(df1)
ein ein_name lname fname score
0 1001 H for Humanity Cooper Bradley 3.0
1 1500 Labor Union Cruise Thomas 3.5
2 3000 Something something Pitt Brad 4.0
Given the size of your dataframes, I suppose you have namesakes (identical first and last names), hence the use of #cache decorator from Python standard library in order to try speeding things up (but you can do without it).

How to transpose values from top few rows in python dataframe into new columns

I am trying to select the values from the top 3 records of each group in a python sorted dataframe and put them into new columns. I have a function that is processing each group but I am having difficulties finding the right method to extract, rename the series, then combine the result as a single series to return.
Below is a simplified example of an input dataframe (df_in) and the expected output (df_out):
import pandas as pd
data_in = { 'Product': ['A', 'A', 'A', 'A', 'B', 'C', 'C'],
'Price': [25.0, 30.5, 50.0, 61.5, 120.0, 650.0, 680.0],
'Qty': [15 , 13, 14, 10, 5, 2, 1]}
df_in = pd.DataFrame (data_in, columns = ['Product', 'Price', 'Qty'])
I am reproducing below 2 examples of the functions I've tested and trying to get a more efficient option that works, especially if I have to process many more columns and records.
Function best3_prices_v1 works but have to explicitly specify each column or variable, and is especially an issue as I have to add more columns.
def best3_prices_v1(x):
d = {}
# get best 3 records if records available, else set volumes as zeroes
best_price_lv1 = x.iloc[0].copy()
rec_with_zeroes = best_price_lv1.copy()
rec_with_zeroes['Price'] = 0
rec_with_zeroes['Qty'] = 0
recs = len(x) # number of records
if (recs == 1):
# 2nd and 3rd records not available
best_price_lv2 = rec_with_zeroes.copy()
best_price_lv3 = rec_with_zeroes.copy()
elif (recs == 2):
best_price_lv2 = x.iloc[1]
# 3rd record not available
best_price_lv3 = rec_with_zeroes.copy()
else:
best_price_lv2 = x.iloc[1]
best_price_lv3 = x.iloc[2]
# 1st best
d['Price_1'] = best_price_lv1['Price']
d['Qty_1'] = best_price_lv1['Qty']
# 2nd best
d['Price_2'] = best_price_lv2['Price']
d['Qty_2'] = best_price_lv2['Qty']
# 3rd best
d['Price_3'] = best_price_lv3['Price']
d['Qty_3'] = best_price_lv3['Qty']
# return combined results as a series
return pd.Series(d, index=['Price_1', 'Qty_1', 'Price_2', 'Qty_2', 'Price_3', 'Qty_3'])
Codes to call function:
# sort dataframe by Product and Price
df_in.sort_values(by=['Product', 'Price'], ascending=True, inplace=True)
# get best 3 prices and qty as new columns
df_out = df_in.groupby(['Product']).apply(best3_prices_v1).reset_index()
Second attempt to improve/reduce codes and explicit names for each variable ... not complete and not working.
def best3_prices_v2(x):
d = {}
# get best 3 records if records available, else set volumes as zeroes
best_price_lv1 = x.iloc[0].copy()
rec_with_zeroes = best_price_lv1.copy()
rec_with_zeroes['Price'] = 0
rec_with_zeroes['Qty'] = 0
recs = len(x) # number of records
if (recs == 1):
# 2nd and 3rd records not available
best_price_lv2 = rec_with_zeroes.copy()
best_price_lv3 = rec_with_zeroes.copy()
elif (recs == 2):
best_price_lv2 = x.iloc[1]
# 3rd record not available
best_price_lv3 = rec_with_zeroes.copy()
else:
best_price_lv2 = x.iloc[1]
best_price_lv3 = x.iloc[2]
stats_columns = ['Price', 'Qty']
# get records values for best 3 prices
d_lv1 = best_price_lv1[stats_columns]
d_lv2 = best_price_lv2[stats_columns]
d_lv3 = best_price_lv3[stats_columns]
# How to rename (keys?) or combine values to return?
lv1_stats_columns = [c + '_1' for c in stats_columns]
lv2_stats_columns = [c + '_2' for c in stats_columns]
lv3_stats_columns = [c + '_3' for c in stats_columns]
# return combined results as a series
return pd.Series(d, index=lv1_stats_columns + lv2_stats_columns + lv3_stats_columns)

Let's unstack():
df_in=(df_in.set_index([df_in.groupby('Product').cumcount().add(1),'Product'])
.unstack(0,fill_value=0))
df_in.columns=[f"{x}_{y}" for x,y in df_in]
df_in=df_in.reset_index()
OR via pivot()
df_in=(df_in.assign(key=df_in.groupby('Product').cumcount().add(1))
.pivot('Product','key',['Price','Qty'])
.fillna(0,downcast='infer'))
df_in.columns=[f"{x}_{y}" for x,y in df_in]
df_in=df_in.reset_index()

Based on #AnuragDabas's pivot solution and #ceruler's feedback above, I can now expand it to a more general problem.
New dataframe with more groups and columns:
data_in = { 'Product': ['A', 'A', 'A', 'A', 'B', 'C', 'C'],
'Model': ['A1', 'A1', 'A1', 'A2', 'B1', 'C1', 'C1'],
'Price': [25.0, 30.5, 50.0, 61.5, 120.0, 650.0, 680.0],
'Qty': [15 , 13, 14, 10, 5, 2, 1],
'Ratings': [9, 7, 8, 10, 6, 7, 8 ]}
df_in = pd.DataFrame (data_in, columns = ['Product', 'Model' ,'Price', 'Qty', 'Ratings'])
group_list = ['Product', 'Model']
stats_list = ['Price','Qty', 'Ratings']
df_out = df_in.groupby(group_list).head(3)
df_out=(df_out.assign(key=df_out.groupby(group_list).cumcount().add(1))
.pivot(group_list,'key', stats_list)
.fillna(0,downcast='infer'))
df_out.columns=[f"{x}_{y}" for x,y in df_out]
df_out = df_out.reset_index()

pandas - insert value into a new column based on a list of dictionary

I have a list of dictionary
items = [{'name':'Fruit','title':'Apple','id':'1'},
{'name':'Fruit','title':'Banana','id':'1'},
{'name':'Vegetable','title':'Tomato','id':'2'},
{'name':'Vegetable','title':'Onion','id':'2'}]
and I have a dataframe
df = pd.Dataframe({'Name': {0: 'Banana', 1: 'Apple', 2: 'Orange', 3: 'Tomato', 4: 'Onion'},'Price': {0: 25, 1: 20, 2: 10, 3: 26, 4: 45},'Kg': {0: 1, 1: 25, 2: 3, 3: 55, 4: 10}})
Now I need to map the title in next column, if the value not exist in dict, it can be Empty
Expected output:
What I have tried is
df["Title"] = df["Name"].map(lambda x : for i in items[name] == x)
Adding a new pandas column with mapped value from a dictionary
But its not working as items is a list of dictonary.

You can flatten your list into a single dict and use that to map.
df['Title'] = df['Name'].map({i['title']:i['name'] for i in items})

long-format pandas dataframe to dictionary

While I find help and documentation on how to convert a pandas DataFrame to dictionary so that columns are keys and values are rows, I find myself stuck when I would like to have one of the column's values as keys and the associated values from another column as values, so that a df like this
a b
1 car
1 train
2 boot
2 computer
2 lipstick
converts to the following dictionary {'1': ['car','train'], '2': ['boot','computer','lipstick]}
I have a feeling it's something pretty simple but I'm out of ideas. I tried df.groupby('a').to_dict() but was unsuccessful
Any suggestions?

You could view this as a groupby-aggregation (i.e., an operation which turns each group into one value -- in this case a list):
In [85]: df.groupby(['a'])['b'].agg(lambda grp: list(grp))
Out[85]:
a
1 [car, train]
2 [boot, computer, lipstick]
dtype: object
In [68]: df.groupby(['a'])['b'].agg(lambda grp: list(grp)).to_dict()
Out[68]: {1: ['car', 'train'], 2: ['boot', 'computer', 'lipstick']}

You can't perform a to_dict() on a the result of groupby, but you can use it to perform your own dictionary construction. The following code will work with the example you provided.
import pandas as pd
df = pd.DataFrame(dict(a=[1,1,2,2,2],
b=['car', 'train', 'boot', 'computer', 'lipstick']))
# Using a loop
dt = {}
for g, d in df.groupby('a'):
dt[g] = d['b'].values
# Using dictionary comprehension
dt2 = {g: d['b'].values for g, d in df.groupby('a')}
Now both dt and dt2 will be dictionaries like this:
{1: array(['car', 'train'], dtype=object),
2: array(['boot', 'computer', 'lipstick'], dtype=object)}
Of course you can put the numpy arrays back into lists, if you so desire.

Yes, because DataFrameGroupBy has no attribute of to_dict, only DataFrame has to_dict attribute.
DataFrame.to_dict(outtype='dict')
Convert DataFrame to dictionary.
You can read more about DataFrame.to_dict here
Take a look of this:
import pandas as pd
df = pd.DataFrame([np.random.sample(9), np.random.sample(9)])
df.columns = [c for c in 'abcdefghi']
# it will convert the DataFrame to dict, with {column -> {index -> value}}
df.to_dict()
{'a': {0: 0.53252618404947039, 1: 0.78237275521385163},
'b': {0: 0.43681232450879315, 1: 0.31356312459390356},
'c': {0: 0.84648298651737541, 1: 0.81417040486070058},
'd': {0: 0.48419015448536995, 1: 0.37578177386187273},
'e': {0: 0.39840348154035421, 1: 0.35367537180764919},
'f': {0: 0.050381560155985827, 1: 0.57080653289506755},
'g': {0: 0.96491634442628171, 1: 0.32844653606404517},
'h': {0: 0.68201236712813085, 1: 0.0097104037581828839},
'i': {0: 0.66836630467152902, 1: 0.69104505886376366}}
type(df)
pandas.core.frame.DataFrame
# DataFrame.groupby is another type
type(df.groupby('a'))
pandas.core.groupby.DataFrameGroupBy
df.groupby('a').to_dict()
AttributeError: Cannot access callable attribute 'to_dict' of 'DataFrameGroupBy' objects, try using the 'apply' method