Here, I am trying to assign groups based on multiple criteria and the closest date diff prior to the zero. The groupby should look only within each ID, then find the closest negative datediff value prior to each zero (not positive, I am trying to look back in time), and based on the Location integer, assign a group. I will have hundreds of groups, and the groups should be assigned based on the Location integer. So, multiple IDs can have the same groups if the Location is the same
Please let me know if I should elaborate or reword - thank you for your help!
Input:
ID Location Date Diff (Days)
111 87 -5
111 88 0
123 97 -123
123 98 -21
123 55 0
123 56 -59
123 30 -29
123 46 0
123 46 25
123 31 87
234 87 -32
234 55 0
234 30 -26
234 54 0
Expected Output:
ID Location Date Diff (Days) Group
111 87 -5 1
111 88 0
123 97 -123
123 98 -21 2
123 55 0
123 56 -59
123 30 -29 3
123 46 0
123 46 25
123 31 87
234 87 -32 1
234 55 0
234 30 -26 3
234 54 0
IIUC, you can find the index to add a group value by using where and mask all values in Diff (I renamed the column Date Diff (Days) by Diff for simplicity) greater or equal to 0. Then groupby ID and groups made of where the column Diff, once shift is equal to 0 and cumsum. For each group get the idxmax. Clean the nan and get the list of all indexes. Second step is to use this list of index and the column Location to create unique ID for each Location with pd.factorize
idx = (df['Diff'].where(lambda x: x.lt(0))
.groupby([df['ID'],
df['Diff'].shift().eq(0).cumsum()])
.idxmax().dropna().tolist()
)
df['Group'] = ''
df.loc[idx, 'Group'] = (pd.factorize(df.loc[idx, 'Location'])[0]+1)
print (df)
ID Location Diff Group
0 111 87 -5 1
1 111 88 0
2 123 97 -123
3 123 98 -21 2
4 123 55 0
5 123 56 -59
6 123 30 -29 3
7 123 46 0
8 123 46 25
9 123 31 87
10 234 87 -32 1
11 234 55 0
12 234 30 -26 3
13 234 54 0
Because the order of rows matter, the most straightforward answer that that I can think of (that will have a somewhat readable code) can use a loop... So I sure hope that performance is not an issue.
The code is less cumbersome than it seems. I hope that the code comments are clear enough.
# Your data
df = pd.DataFrame(
data=[[111,87,-5],
[111,88,0],
[123,97,-123],
[123,98,-21],
[123,55,0],
[123,56,-59],
[123,30,-29],
[123,46,0],
[123,46,25],
[123,31,87],
[234,87,-32],
[234,55,0],
[234,30,-26],
[234,54,0]], columns=['ID','Location','Date Diff (Days)'])
N_ID, N_Location, N_Date, N_Group = 'ID', 'Location', 'Date Diff (Days)', 'Group'
# Some preparations
col_group = pd.Series(index=df.index) # The final column we'll add to our `df`
groups_found = 0
location_to_group = dict() # To maintain our mapping of Location to "group" values
# LOOP
prev_id, prev_DD, best_idx = None, None, None
for idx, row in df.iterrows():
#print(idx, row.values)
if prev_id is None:
if row[N_Date] < 0:
best_idx = idx
#best_date_diff_in_this_run = row[N_Date]
else:
if row[N_ID] != prev_id or row[N_Date] < prev_DD:
# Associate a 'group' value to row with index `best_idx`
if best_idx is not None:
best_location = df.loc[best_idx, N_Location]
if best_location in location_to_group:
col_group.loc[best_idx] = location_to_group[best_location]
else:
groups_found += 1
location_to_group[best_location] = groups_found
col_group.loc[best_idx] = groups_found
# New run
best_idx = None
# Regardless, update best_idx
if row[N_Date] < 0:
best_idx = idx
#best_date_diff_in_this_run = row[N_Date]
# Done
prev_id, prev_DD = row[N_ID], row[N_Date]
# Deal with the last "run" (same code as the one inside the loop)
# Associate a 'group' value to row with index `best_idx`
if best_idx is not None:
best_location = df.loc[best_idx, N_Location]
if best_location in location_to_group:
col_group.loc[best_idx] = location_to_group[best_location]
else:
groups_found += 1
location_to_group[best_location] = groups_found
col_group.loc[best_idx] = groups_found
# DONE
df['Group'] = col_group
Related
I'd like to create a lookup (similar to excel for example) with match mode -> an exact match or the next larger item.
Let's say I have these 2 dataframes:
seed(1)
np.random.seed(1)
Wins_Range = np.arange(1,101,1)
Wins = pd.DataFrame({"Wins Needed": Wins_Range})
Wins
Wins Needed
0 1
1 2
2 3
3 4
4 5
... ...
95 96
96 97
97 98
98 99
99 100
And the second one:
Levels_Range = np.arange(1,101,1)
Levels = pd.DataFrame({"Level": Levels_Range})
Levels["Wins"]=np.random.choice([1,2,3,4,5],size=len(Levels), p=[0.2,0.2,0.2,0.2,0.2]).cumsum()
Levels
Level Wins
0 1 3
1 2 7
2 3 8
3 4 10
4 5 11
... ... ...
95 96 281
96 97 286
97 98 289
98 99 290
99 100 294
Now, I'd like to pull the level from Levels df to the Wins df when the condition is Wins Needed=Wins but as I said - the match mode will be an exact match or the next larger item.
BTW - the type of Levels["Wins"] is float and the type of Wins["Win"] is int if that matters.
I've tried to use the merge function but it doesn't work (I'm new at python) -
Wins.merge(Levels, on='Wins Needed', how='left')
Thanks in advance!
You need a merge_asof:
out = pd.merge_asof(Wins, Levels, left_on='Wins Needed', right_on='Wins',
direction='forward')[['Wins Needed', 'Level']]
Or
Wins['Level'] = pd.merge_asof(Wins, Levels, left_on='Wins Needed', right_on='Wins',
direction='forward')['Level']
NB. the keys must be sorted for a merge_asof.
Output:
Wins Needed Level
0 1 1
1 2 1
2 3 1
3 4 2
4 5 2
.. ... ...
95 96 35
96 97 35
97 98 36
98 99 36
99 100 37
[100 rows x 2 columns]
If the values are not initially sorted:
Wins['Level'] = pd.merge_asof(Wins[['Wins Needed']].reset_index().sort_values(by='Wins Needed'),
Levels.sort_values(by='Wins'),
left_on='Wins Needed', right_on='Wins',
direction='forward').set_index('index')['Level']
I was counting the no of occurrence of angle and dist by the code below:
g = new_df.value_counts(subset=['Current_Angle','Current_dist'] ,sort = False)
the output:
current_angle current_dist 0
-50 30 1
-50 40 2
-50 41 6
-50 45 4
try1:
g.columns = ['angle','Distance','count','Percentage Missed'] - result was no change in the name of column
try2:
When I print the columns using print(g.columns) ended with error AttributeError: 'Series' object has no attribute 'columns'
I want to rename the column 0 as count and add a new column to the dataframe g as percent missed which is calculated by 100 - value in column 0
Expected output
current_angle current_dist count percent missed
-50 30 1 99
-50 40 2 98
-50 41 6 94
-50 45 4 96
1:How to modify the code? I mean instead of value_counts, is there any other function that can give the expected output?
2. How to get the expected output with the current method?
EDIT 1(exceptional case)
data:
angle
distance
velocity
0
124
-3
50
24
-25
50
34
25
expected output:
count is calculated based on distance
angle
distance
velocity
count
percent missed
0
124
-3
1
99
50
24
-25
1
99
50
34
25
1
99
First add Series.reset_index, because DataFrame.value_counts return Series, so possible use parameter name for change column 0 to count column and then subtract 100 to new column by Series.rsub for subtract from right side like 100 - df['count']:
df = (new_df.value_counts(subset=['Current_Angle','Current_dist'] ,sort = False)
.reset_index(name='count')
.assign(**{'percent missed': lambda x: x['count'].rsub(100)}))
Or if need also set new columns names use DataFrame.set_axis:
df = (new_df.value_counts(subset=['Current_Angle','Current_dist'] ,sort = False)
.reset_index(name='count')
.set_axis(['angle','Distance','count'], axis=1)
.assign(**{'percent missed': lambda x: x['count'].rsub(100)}))
If need assign new columns names here is alternative solution:
df = (new_df.value_counts(subset=['Current_Angle','Current_dist'] ,sort = False)
.reset_index())
df.columns = ['angle','Distance','count']
df['percent missed'] = df['count'].rsub(100)
Assuming a DataFrame as input (if not reset_index first), simply use rename and a subtraction:
df = df.rename(columns={'0': 'count'}) # assuming string '0' here, else use 0
df['percent missed'] = 100 - df['count']
output:
current_angle current_dist count percent missed
0 -50 30 1 99
1 -50 40 2 98
2 -50 41 6 94
3 -50 45 4 96
alternative: using groupby.size:
(new_df
.groupby(['current_angle','current_dist']).size()
.reset_index(name='count')
.assign(**{'percent missed': lambda d: 100-d['count']})
)
output:
current_angle current_dist count percent missed
0 -50 30 1 99
1 -50 40 2 98
2 -50 41 6 94
3 -50 45 4 96
I have a df with numbers:
numbers = pd.DataFrame(columns=['number'], data=[
50,
65,
75,
85,
90
])
and a df with ranges (look up table):
ranges = pd.DataFrame(
columns=['range','range_min','range_max'],
data=[
['A',90,100],
['B',85,95],
['C',70,80]
]
)
I want to determine what range (in second table) a value (in the first table) falls in. Please note ranges overlap, and limits are inclusive.
Also please note the vanilla dataframe above has 3 ranges, however this dataframe gets generated dynamically. It could have from 2 to 7 ranges.
Desired result:
numbers = pd.DataFrame(columns=['number','detected_range'], data=[
[50,'out_of_range'],
[65, 'out_of_range'],
[75,'C'],
[85,'B'],
[90,'overlap'] * could be A or B *
])
I solved this with a for loop but this doesn't scale well to a big dataset I am using. Also code is too extensive and inelegant. See below:
numbers['detected_range'] = nan
for i, row1 in number.iterrows():
for j, row2 in ranges.iterrows():
if row1.number<row2.range_min and row1.number>row2.range_max:
numbers.loc[i,'detected_range'] = row1.loc[j,'range']
else if (other cases...):
...and so on...
How could I do this?
You can use a bit of numpy vectorial operations to generate masks, and use them to select your labels:
import numpy as np
a = numbers['number'].values # numpy array of numbers
r = ranges.set_index('range') # dataframe of min/max with labels as index
m1 = (a>=r['range_min'].values[:,None]).T # is number above each min
m2 = (a<r['range_max'].values[:,None]).T # is number below each max
m3 = (m1&m2) # combine both conditions above
# NB. the two operations could be done without the intermediate variables m1/m2
m4 = m3.sum(1) # how many matches?
# 0 -> out_of_range
# 2 -> overlap
# 1 -> get column name
# now we select the label according to the conditions
numbers['detected_range'] = np.select([m4==0, m4==2], # out_of_range and overlap
['out_of_range', 'overlap'],
# otherwise get column name
default=np.take(r.index, m3.argmax(1))
)
output:
number detected_range
0 50 out_of_range
1 65 out_of_range
2 75 C
3 85 B
4 90 overlap
edit:
It works with any number of intervals in ranges
example output with extra['D',50,51]:
number detected_range
0 50 D
1 65 out_of_range
2 75 C
3 85 B
4 90 overlap
Pandas IntervalIndex fits in here; however, since your data has overlapping points, a for loop is the approach I'll use here (for unique, non-overlapping indices, pd.get_indexer is a fast approach):
intervals = pd.IntervalIndex.from_arrays(ranges.range_min,
ranges.range_max,
closed='both')
box = []
for num in numbers.number:
bools = intervals.contains(num)
if bools.sum()==1:
box.append(ranges.range[bools].item())
elif bools.sum() > 1:
box.append('overlap')
else:
box.append('out_of_range')
numbers.assign(detected_range = box)
number detected_range
0 50 out_of_range
1 65 out_of_range
2 75 C
3 85 B
4 90 overlap
firstly，explode the ranges:
df1=ranges.assign(col1=ranges.apply(lambda ss:range(ss.range_min,ss.range_max),axis=1)).explode('col1')
df1
range range_min range_max col1
0 A 90 100 90
0 A 90 100 91
0 A 90 100 92
0 A 90 100 93
0 A 90 100 94
0 A 90 100 95
0 A 90 100 96
0 A 90 100 97
0 A 90 100 98
0 A 90 100 99
1 B 85 95 85
1 B 85 95 86
1 B 85 95 87
1 B 85 95 88
1 B 85 95 89
1 B 85 95 90
secondly，judge wether each of numbers in first df
def function1(x):
df11=df1.loc[df1.col1==x]
if len(df11)==0:
return 'out_of_range'
if len(df11)>1:
return 'overlap'
return df11.iloc[0,0]
numbers.assign(col2=numbers.number.map(function1))
number col2
0 50 out_of_range
1 65 out_of_range
2 75 C
3 85 B
4 90 overlap
the logic is simple and clear
Three columns exist in this data set: ID (unique employee identification), WorkComplete (indicating when all work has been completed), and DateDiff (number of days from their start date). I am looking to group the DaysDiff column based on certain time periods with an added layer of tolerance or leniency. For my mock data, I am spacing the time periods by 30 days.
Group 0: 0-30 DateDiff (with a 30 day extra window if 'Y' is not found)
Group 1: 31-60 DateDiff (with a 30 day extra window if 'Y' is not found)
Group 2: 61-90 DateDiff (with a 30 day extra window if 'Y' is not found)
I was able to create very basic code and assign the groupings, but I am having trouble with the extra 30 day window. For example, if an employee completed their work (Y) during the time periods above, then they receive the attributed grouping. For ID 111 below, you can see that the person did not complete their work within the first 30 days, so I am giving them an addition 30 days to complete their work. If they complete their work, then the first instance we see a 'Y', it is grouped in the previous grouping.
df = pd.DataFrame({'ID':[111, 111, 111, 111, 111, 111, 112, 112, 112],
'WorkComplete':['N', 'N', 'Y', 'N', 'N', 'N', 'N', 'Y', 'Y'],
'DaysDiff': [0, 29, 45, 46, 47, 88, 1, 12, 89]})
Input
ID WorkComplete DaysDiff
111 N 0
111 N 29
111 Y 45
111 N 46
111 N 47
111 N 88
123 N 1
123 Y 12
123 Y 89
Output
ID WorkComplete DaysDiff Group
111 N 0 0
111 N 29 0
111 Y 45 0 <---- note here the grouping is 0 to provide extra time
111 N 46 1 <---- back to normal
111 N 47 1
111 N 88 2
123 N 1 0
123 Y 12 0
123 Y 89 2
minQ1 = 0
highQ1 = 30
minQ2 = 31
highQ2 = 60
minQ2 = 61
highQ2 = 90
def Group_df(df):
if (minQ1 <= df['DateDiff'] <= highQ1): return '0'
elif (minQ1 <= df['DateDiff'] <= highQ1): return '1'
elif (minQ2 <= df['DateDiff'] <= highQ2): return '2'
df['Group'] = df.apply(Group_df, axis = 1)
The trouble I am having is allowing for the additional 30 days if the person did not complete the work. My above attempt is partial at trying to resolve the issue.
You can use np.select for the primary conditions.
Then, use mask for the specific condition you mention. s is the first index location for all Y values per group. I then temporarily assign s as a new column, so that I can check for rows against df.index (the index) to return rows that meet the condition. The second condition is if the group number is 1 from the previos line of code:
df['Group'] = np.select([df['DaysDiff'].between(0,30),
df['DaysDiff'].between(31,60),
df['DaysDiff'].between(61,90)],
[0,1,2])
s = df[df['WorkComplete'] == 'Y'].groupby('ID')['DaysDiff'].transform('idxmin')
df['Group'] = df['Group'].mask((df.assign(s=s)['s'].eq(df.index)) & (df['Group'].eq(1)), 0)
df
Out[1]:
ID WorkComplete DaysDiff Group
0 111 N 0 0
1 111 N 29 0
2 111 Y 45 0
3 111 N 46 1
4 111 N 47 1
5 111 N 88 2
6 123 N 1 0
7 123 Y 12 0
8 123 Y 89 2
I have two dataframes, both have same shapes.
dfA
2008LG 2007LG 2006LG 2005LG
0 44 65 30 20
1 10 16 56 70
2 65 30 20 122
3 0.0 0.00 679 158
4 0.0 0.00 30 20
dfB
2008Net 2007Net 2006Net 2005Net
0 0 0 0 452
1 0 0 0 365
2 0 0 0 778
3 0 0 0 78
4 0 0 0 60
The calculation logic is: for each row in dfB , start from the very end 2005Net column and use 2005LG - 2005net and get a value which gets assigned to the first right columns of 2005Net.
For example: for the first iteration 2005LG - 2005Net = 20-452 = -432 and assign -432 to 2006Net. and the second iteration will start from 2006LG - 2006Net= 30 - -432 = 462 and assign to 2007Net.
below is my code, but it is not cutting it, what exactly is wrong here ?
import pandas as pd
import numpy as np
from tqdm import tqdm
for index in tqdm(range(dfA.shape[0])):
for col_index in reversed(range(4)):
the_value = 0
the_value = dfA[dfA.columns[col_index]].iloc[index] - dfB[dfB.columns[col_index]].iloc[index]
dfB[dfB.columns[col_index-1]].iloc[index] = the_value
Try something like this.
for index in reverse(range(4)):
dfB[index - 1] = dfA.iloc[:, index] - dfB.iloc[:,index]
This assume that each column you want to subtract have the same lenght.