Develop Reference

Second largest value is showing wrong value on pandas dataframe, when groupby

I have a table, and I'm trying to get the second largest "percent" value by column "Day".
I can get the second largest value, but the column 'Hour', is not the right one
Table:df
name
Day
Hour
percent
000_RJ_S1
26
10
0.908494
000_RJ_S1
26
11
0.831482
000_RJ_S1
26
12
0.843846
000_RJ_S1
26
13
0.877238
000_RJ_S1
26
17
0.163908
000_RJ_S1
26
18
0.230296
000_RJ_S1
26
19
0.359440
000_RJ_S1
26
20
0.379988
Script Used:
df = df.groupby(['name','Day'])[['Hour','percent']].apply(lambda x: x.nlargest(2, columns='percent').min())
Output:
As you can see, the "Hour" value is wrong. It should be "13" and not "10". The second largest value is right.
name
Day
Hour
percent
000_RJ_S1
26
10
0.877238
It should be:
name
Day
Hour
percent
000_RJ_S1
26
13
0.877238
I can't figure out what's is wrong. Could you guys help me with this issue.
Thanks a lot

Sort the percent columns before grouping, and use the nth function instead:
(df.sort_values('percent', ascending=False)
.groupby(['name', 'Day'],sort=False, as_index = False)
.nth(1)
)
name Day Hour percent
3 000_RJ_S1 26 13 0.877238

The reason you have got 10 is because of the min() function.
The nlargest() in the lambda would return the two rows with largest percent values and when you apply min() what it does is it selects the minimum values from each column separately which gave you that output.
You can use iloc[1] instead of min() to get the desired result
Here's the code using iloc:
df.groupby(['name','Day'])[['Hour','percent']].apply(lambda x: x.nlargest(2, columns='percent')).iloc[1]

One solution is to use a double groupby:
cols = ['name','Day']
# get the top 2 per group
s = df.groupby(cols)['percent'].nlargest(2)
# get the index of min per group
idx = s.droplevel(cols).groupby(s.droplevel(-1).index).idxmin()
# slice original data with those indexes
df2 = df.loc[idx.values]
Output:
name Day Hour percent
3 000_RJ_S1 26 13 0.877238

Pandas Time Series: Remove Rows Per ID

I have a Pandas dataframe of the form:
Date ID Temp
2019/03/27 1 23
2019/04/27 2 32
2019/04/27 1 42
2019/04/28 1 41
2019/01/27 2 33
2019/08/27 2 23
What I need to do?
Select Rows which are at least 30 days old from their latest measurement for each id.
i.e. the latest date for Id = 2 is 2019/08/27, so for ID =2 I need to select rows which are at least 30 days older. So, the row with 2019/08/27 for ID=2 will itself be dropped.
Similarly, the latest date for ID = 1 is 2019/04/28. This means I can select rows for ID =1 only if the date is less than 2019/03/28 (30 days older). So, the row 2019/04/27 with ID=1 will be dropped.
How to do this in Pandas. Any help is greatly appreciated.
Thank you.
Final dataframe will be:
Date ID Temp
2019/03/27 1 23
2019/04/27 2 32
2019/01/27 2 33

In your case using groupby + transform('last') and filter the original df
Yourdf=df[df.Date<df.groupby('ID').Date.transform('last')-pd.Timedelta('30 days')].copy()
Date ID Temp
0 2019-03-27 1 23
1 2019-04-27 2 32
4 2019-01-27 2 33
Notice I am adding the .copy at the end to prevent the setting copy error.

Grouping the data with pandas TimeGrouper with interval between 5 to 25 min, 25 to 45 min, 45 to 05 minutes

I am new to python pandas and I am trying to group my data on 20 Minute interval. If I use Data.groupby([pd.TimeGrouper('20Min')) it is working but it is giving the grouped data from 0 to 20 min, 20-40 etc. But I want to group my data between 5 to 25 min, 25 to 45 min etc.
Can you help how to achieve this pandas TimeGrouper.
Thanks in advance.

Data.groupby([pd.TimeGrouper(freq='20Min',base=5, label='right')
will give you data grouped by 20 Min, starting with forward 5 min (since we are using base as 5 and label as right. i.e, it will be grouped as 05 to 25, 25 to 45 etc. and if you use this:
Data.groupby([pd.TimeGrouper(freq='20Min',base=55, label='left')
will give you data grouped by 20 Min with backward 5 min (i.e, your data will group as :55 to :15, :15 to :35 etc)

Subsetting a Pandas.DataFrame object only where there is a difference between two rows in python

I was wondering if it there were an easy way in python to return a subset of my DataFrame rows only where there is a change between two consecutive rows. For example, my dataframe object might look like this:
Date A B
20160713070000 20 21
20160713070100 20 23
20160713070128 20 23
20160713070128 21 24
20160713070134 23 24
In this case, I would want to return the following dataframe object:
Date A B
20160713070000 20 21
20160713070100 20 23
20160713070128 21 24
20160713070134 23 24
Thanks for the help!

I'd use drop_duplicates() function:
In [262]: df.drop_duplicates(subset=['A','B'])
Out[262]:
Date A B
0 20160713070000 20 21
1 20160713070100 20 23
3 20160713070128 21 24
4 20160713070134 23 24

Assuming your dataframe is df, try the following:
sub_df = df[df.groupby('Date')['A'].transform(lambda x: x.index[-1])==df.index]

Group by to create unique values appearing by date, as well as non-unique values by date

I have a data frame that looks like:
app_id subproduct date
0 23 3 2015-05-29
1 23 4 2015-05-29
2 25 5 2015-05-29
3 23 3 2015-05-29
4 24 7 2015-05-29
....
I run:
groupings =insightevents.groupby([insightevents['created_at_date'].dt.year,\
insightevents['created_at_date'].dt.month,\
insightevents['created_at_date'].dt.week,insightevents['created_at_date'].dt.day,
insightevents['created_at_date'].dt.dayofweek]);
inboxinsights=pd.DataFrame([groupings['app_id'].unique(),groupings['subproduct'].unique()]).transpose()
This gives me:
app_id subproduct
2015 5 22 29 4 [23,24,25] [3,4,5,7]
However, what I want is actually not to get just the unique values, but overall just the app_ids and sub_product loads on the day as additional columns, so:
unique_ app_id unique_subproduct subproduct app_id
2015 5 22 29 4 [23,24,25] [3,4,5,7] [3,3,4,5,7] [23,23,23,24,25]
I find that just doing:
inboxinsights=pd.DataFrame([groupings['app_id'].unique(), groupings['subproduct'].unique(),groupings['app_id'],groupings['subproduct']]).transpose()
Doesn't work and just gives me:
AttributeError: 'Series' object has no attribute 'type'

If you wanted just the number of unique values, that's easy:
inboxinsights.groupby('date').agg({'app_id': 'nunique', 'subproduct': 'nunique'})
returns:
But it looks like you want the list of what those actually are. I found this other SO question helpful:
not_unique_inboxinsights = groupby('date').agg(lambda x: tuple(x))
And then you say want both the unique and not-unique. For that, I would make two groupby dataframes and concatenate them, like this:
unique_inboxinsights = groupby('date').agg(lambda x: set(tuple(x)))
Hope that helps.