Develop Reference

Efficently slicing non-integer multilevel indexes with integers in Pandas

The following code generates a sample DataFrame with a multilevel index. The first level is a string, the second level is a datetime.
Script
import pandas as pd
from datetime import datetime
import random
df = pd.DataFrame(columns=['network','time','active_clients','throughput','speed'])
networks = ['ALPHA','BETA','GAMMA']
times = pd.date_range(datetime.strptime('2021-01-01 00:00:00','%Y-%m-%d %H:%M:%S'),datetime.strptime('2021-01-01 12:00:00','%Y-%m-%d %H:%M:%S'),7).tolist()
for n in networks:
for t in times:
df = df.append({'network':n,'time':t,'active_clients':random.randint(10,30),'throughput':random.randint(1500,5000),'speed':random.randint(10000,12000)},ignore_index=True)
df.set_index(['network','time'],inplace=True)
print(df.to_string())
Output
active_clients throughput speed
network time
ALPHA 2021-01-01 00:00:00 16 4044 11023
2021-01-01 02:00:00 17 2966 10933
2021-01-01 04:00:00 10 4649 11981
2021-01-01 06:00:00 23 3629 10113
2021-01-01 08:00:00 30 2520 11159
2021-01-01 10:00:00 10 4200 11309
2021-01-01 12:00:00 16 3878 11366
BETA 2021-01-01 00:00:00 17 3073 11798
2021-01-01 02:00:00 20 1941 10640
2021-01-01 04:00:00 17 1980 11869
2021-01-01 06:00:00 23 3346 10002
2021-01-01 08:00:00 10 1952 10063
2021-01-01 10:00:00 28 3788 11047
2021-01-01 12:00:00 24 4993 10487
GAMMA 2021-01-01 00:00:00 21 4366 11587
2021-01-01 02:00:00 22 3404 11669
2021-01-01 04:00:00 20 1608 10344
2021-01-01 06:00:00 28 1849 10278
2021-01-01 08:00:00 14 3229 11925
2021-01-01 10:00:00 21 3408 10411
2021-01-01 12:00:00 12 1799 10492
For each item in the first level, I want to select the last three records in the second level. The catch is that I don't know the datetime values, so I need to select by integer-based index location instead. What's the most efficient way of slicing the DataFrame to achieve the following.
Desired output
active_clients throughput speed
network time
ALPHA 2021-01-01 08:00:00 30 2520 11159
2021-01-01 10:00:00 10 4200 11309
2021-01-01 12:00:00 16 3878 11366
BETA 2021-01-01 08:00:00 10 1952 10063
2021-01-01 10:00:00 28 3788 11047
2021-01-01 12:00:00 24 4993 10487
GAMMA 2021-01-01 08:00:00 14 3229 11925
2021-01-01 10:00:00 21 3408 10411
2021-01-01 12:00:00 12 1799 10492
My attempts
Returns the full dataframe:
df_sel = df.iloc[:,-3:]
Raises an error because loc doesn't support using integer values on datetime objects:
df_sel = df.loc[:,-3:]
Returns the last three entries in the second level, but only for the last entry in the first level:
df_sel = df.loc[:].iloc[-3:]

I have 2 methods to solve this problem:
Method 1:
As it mentions from the first comment from Quang Hoang, you can use groupby to do this, which I believe has the shortest code:
df.groupby(level=0).tail(3)
Method 2:
You can also slice each one in networks then concat them:
pd.concat([df.loc[[i]][-3:] for i in networks])
Both of these 2 methods will output the result you want:

Another method is to do some reshaping:
df.unstack(0).iloc[-3:].stack().swaplevel(0,1).sort_index()
Output:
active_clients throughput speed
network time
ALPHA 2021-01-01 08:00:00 26 4081 11325
2021-01-01 10:00:00 13 3370 10716
2021-01-01 12:00:00 13 3691 10737
BETA 2021-01-01 08:00:00 28 2105 10465
2021-01-01 10:00:00 21 2444 10158
2021-01-01 12:00:00 24 1947 11226
GAMMA 2021-01-01 08:00:00 13 1850 10288
2021-01-01 10:00:00 23 2241 11521
2021-01-01 12:00:00 30 3515 11138
Details:
unstack the outer most index level, level=0
Use, iloc to select the last three records in the dataframe
stack that level back to the index swaplevel and sort_index

Pandas, insert datetime values that increase one hour for each row

I made predictions with an Arima model that predict the next 168 hours (one week) of cars on the road. I also want to add a column called "datetime" that starts with 00:00 01-01-2021 and increases with one hour for each row.
Is there an intelligent way of doing this?

You can do:
x=pd.to_datetime('2021-01-01 00:00')
y=pd.to_datetime('2021-01-07 23:59')
pd.Series(pd.date_range(x,y,freq='H'))
output:
pd.Series(pd.date_range(x,y,freq='H'))
Out[153]:
0 2021-01-01 00:00:00
1 2021-01-01 01:00:00
2 2021-01-01 02:00:00
3 2021-01-01 03:00:00
4 2021-01-01 04:00:00
163 2021-01-07 19:00:00
164 2021-01-07 20:00:00
165 2021-01-07 21:00:00
166 2021-01-07 22:00:00
167 2021-01-07 23:00:00
Length: 168, dtype: datetime64[ns]

How to extract hourly data from a df in python?

I have the following df
dates Final
2020-01-01 00:15:00 94.7
2020-01-01 00:30:00 94.1
2020-01-01 00:45:00 94.1
2020-01-01 01:00:00 95.0
2020-01-01 01:15:00 96.6
2020-01-01 01:30:00 98.4
2020-01-01 01:45:00 99.8
2020-01-01 02:00:00 99.8
2020-01-01 02:15:00 98.0
2020-01-01 02:30:00 95.1
2020-01-01 02:45:00 91.9
2020-01-01 03:00:00 89.5
The entire dataset is till 2021-01-01 00:00:00 95.6 with a gap of 15mins.
Since the freq is 15mins, I would like to change it to 1 hour and maybe drop the middle values
Expected output
dates Final
2020-01-01 01:00:00 95.0
2020-01-01 02:00:00 99.8
2020-01-01 03:00:00 89.5
With the last row being 2021-01-01 00:00:00 95.6
How can this be done?
Thanks

Use Series.dt.minute to performance a boolean indexing:
df_filtered = df.loc[df['dates'].dt.minute.eq(0)]
#if necessary
#df_filtered = df.loc[pd.to_datetime(df['dates']).dt.minute.eq(0)]
print(df_filtered)
dates Final
3 2020-01-01 01:00:00 95.0
7 2020-01-01 02:00:00 99.8
11 2020-01-01 03:00:00 89.5

If you're doing data analysis or data science I don't think dropping the middle values is a good approach at all! You should sum them I guess (I don't know about your use case but I know some stuff about Time Series data).

Select groups using slicing based on the group index in pandas DataFrame

I have a Dataframe with a users indicated by the column: 'user_id'. Each of these users have several entries in the dataframe based on the date on which they did something, which is also a column. The dataframe looks somthing like
df:
user_id date
0 2019-04-13 02:00:00
0 2019-04-13 03:00:00
3 2019-02-18 22:00:00
3 2019-02-18 23:00:00
3 2019-02-19 00:00:00
3 2019-02-19 02:00:00
3 2019-02-19 03:00:00
3 2019-02-19 04:00:00
8 2019-04-05 04:00:00
8 2019-04-05 05:00:00
8 2019-04-05 06:00:00
8 2019-04-05 15:00:00
15 2019-04-28 19:00:00
15 2019-04-28 20:00:00
15 2019-04-29 01:00:00
23 2019-06-24 02:00:00
23 2019-06-24 05:00:00
23 2019-06-24 06:00:00
24 2019-03-27 12:00:00
24 2019-03-27 13:00:00
What I want to do is, for example, select the first 3 users. I wanted to do this with a code like this:
df.groupby('user_id').iloc[:3]
I know that groupby doesn't have an iloc so how could I achieve the same thing like an iloc in the groups, so I am able to slice them?

I found a way based on crayxt's answer:
df[df['user_id'].isin(df['user_id'].unique()[:3])]

scipy UnivariateSpline always return linear-ish spline when plotting

I have following set of data (pandas.DataFrame) which I would like to use scipy.interpolate.UnivariateSpline to fit. Let's call the data data.
Date
2018-04-02 09:00:00 16249
2018-04-02 10:00:00 45473
2018-04-02 11:00:00 32050
2018-04-02 12:00:00 35898
2018-04-02 13:00:00 21577
2018-04-02 14:00:00 30545
2018-04-02 15:00:00 60925
2018-04-02 16:00:00 47124
2018-04-03 09:00:00 18534
2018-04-03 10:00:00 36064
2018-04-03 11:00:00 32387
2018-04-03 12:00:00 15903
2018-04-03 13:00:00 22291
2018-04-03 14:00:00 26367
2018-04-03 15:00:00 66269
2018-04-03 16:00:00 38478
2018-04-04 09:00:00 15803
2018-04-04 10:00:00 22511
2018-04-04 11:00:00 33123
2018-04-04 12:00:00 21000
2018-04-04 13:00:00 23132
2018-04-04 14:00:00 39270
2018-04-04 15:00:00 102544
2018-04-04 16:00:00 143421
2018-04-04 17:00:00 200
2018-04-05 09:00:00 23377
2018-04-05 10:00:00 52089
2018-04-05 11:00:00 99298
2018-04-05 12:00:00 24627
2018-04-05 13:00:00 33467
2018-04-05 14:00:00 26498
2018-04-05 15:00:00 114794
2018-04-05 16:00:00 44904
2018-04-06 09:00:00 12180
2018-04-06 10:00:00 41658
2018-04-06 11:00:00 64066
2018-04-06 12:00:00 12517
2018-04-06 13:00:00 12610
2018-04-06 14:00:00 43544
2018-04-06 15:00:00 65533
2018-04-06 16:00:00 123885
2018-04-09 09:00:00 13425
2018-04-09 10:00:00 38354
2018-04-09 11:00:00 59491
2018-04-09 12:00:00 21402
2018-04-09 13:00:00 24550
2018-04-09 14:00:00 25189
2018-04-09 15:00:00 67751
2018-04-09 16:00:00 16071
2018-04-10 09:00:00 35587
2018-04-10 10:00:00 58667
2018-04-10 11:00:00 41831
2018-04-10 12:00:00 35196
2018-04-10 13:00:00 22611
2018-04-10 14:00:00 23070
2018-04-10 15:00:00 40819
2018-04-10 16:00:00 20337
2018-04-11 09:00:00 7962
2018-04-11 10:00:00 23982
2018-04-11 11:00:00 21794
2018-04-11 12:00:00 16835
2018-04-11 13:00:00 16821
2018-04-11 14:00:00 13270
2018-04-11 15:00:00 34954
2018-04-11 16:00:00 15772
2018-04-12 09:00:00 8587
2018-04-12 10:00:00 47950
2018-04-12 11:00:00 24742
2018-04-12 12:00:00 16743
2018-04-12 13:00:00 21917
2018-04-12 14:00:00 43272
2018-04-12 15:00:00 50630
2018-04-12 16:00:00 104656
2018-04-13 09:00:00 15282
2018-04-13 10:00:00 30304
2018-04-13 11:00:00 65737
2018-04-13 12:00:00 17467
2018-04-13 13:00:00 10439
2018-04-13 14:00:00 19836
2018-04-13 15:00:00 52051
2018-04-13 16:00:00 99462
what I have done so far is:
import matplotlib.pyplot as plt
import numpy as np
import scipy.interpolate as interp
x = [i for i in range(1, data.size+1)] # this gives x as an array from 1 to 82.
spl = interp.UnivariateSpline(x, data.values, s=0.5)
xx = np.linspace(min(x), max(x), 1000) # 1000 is an arbitrary number here.
plt.plot(x, data.values, 'bo')
plt.plot(xx, spl(xx), 'r')
plt.show()
# the plot is below and it seems to be very linear and does not look like a cubic spline at all. Cubic Spline is the default.
when I run spl against x, others remain unchanged, which is:
plt.plot(x, spl(x), 'r')
I get following:
the only different is the y axis is topped at 14,000, which seems to mean the previous plot showed some degree of curvature. (or not?)
I am not sure what I am missing here but I apparently missed something. I am still very new to spline fitting in python generally.
can you tell me how I can correctly spline fit my time series above?
EDIT
upon comment from you, I wanted to add another plot to hopefully explain myself a bit better. I didn't really mean it is linear but I couldn't find a better word. To illustrate,
xxx = [10,20,40,60,80]
plt(x, data.values, 'bo')
plt(xx, sp(xx), 'r')
plt.show()
I think below plot looks reasonably linear-ish in my sense. I am guessing, probably my question should be, how scipy.UnivariateSpline really works?
does it only show the plot for the values evaluated at the points we supplied (e.g. for this plot it is xxx) ?
I was expecting a much smoother plot with decent curvature demonstrated. this question's answer is showing a plot that I would expect; it looks more like a plot that piece-wise cubic functions would generate, whereas mine looks, to me, and compared to that plot, linear-ish (or first order if it is more appropriate.)

The data set you have looks more like Rexthor, the dog-bearer than something that a smooth curve can follow. You don't have an issue with SciPy; you have an issue with data.
By increasing the parameter s you can get progressively smoother plots that deviate further and further from the data, eventually approaching the cubic polynomial that is the "best" least-squares fit for the data. But here "best" means "very bad, probably worthless". A smooth curve can be useful to display a pattern that the data already follows. If the data does not follow a smooth pattern, one should not draw a curve for the sake of drawing. The data points on the first plot should just be presented as is, without any connecting or approximating curves.
The data comes from hourly reading taken from 9:00 to 16:00 (with one stray 17:00 value mixed it - throw it out.) This structure matters. Do not pretend that Tuesday 9:00 is what happens one hour after Monday 16:00.
The data can be meaningfully summarized by daily totals
Day Total
2018-04-02 289841
2018-04-03 256293
2018-04-04 401004
2018-04-05 419054
2018-04-06 375993
2018-04-09 266233
2018-04-10 278118
2018-04-11 151390
2018-04-12 318497
2018-04-13 310578
and by hourly averages (average number of events at 9:00, across all days, etc).
Hour Average
9:00:00 16698.6
10:00:00 39705.2
11:00:00 47451.9
12:00:00 21758.8
13:00:00 20941.5
14:00:00 29086.1
15:00:00 65627
16:00:00 65411
In these things we can maybe observe some pattern. Here is the hourly one:
hourly_averages = np.array([16698.6, 39705.2, 47451.9, 21758.8, 20941.5, 29086.1, 65627, 65411])
hours = np.arange(9, 17)
hourly_s = 0.1*np.diff(hourly_averages).max()**2
hourly_spline = interp.UnivariateSpline(hours, hourly_averages, s=hourly_s)
xx = np.linspace(min(hours), max(hours), 1000) # 1000 is an arbitrary number here.
plt.plot(hours, hourly_averages, 'bo')
plt.plot(xx, hourly_spline(xx), 'r')
plt.show()
The curve shows the lunch break and the end-of-day rush. My choice of s as 0.1*np.diff(hourly_averages).max()**2 is not canonical, but it recognizes the fact that s scales as the square of the residuals. (Documentation). I'll use the same choice for daily averages:
daily_totals = np.array([289841, 256293, 401004, 419054, 375993, 266233, 278118, 151390, 318497, 310578])
days = np.arange(len(daily_totals))
daily_s = 0.1*np.diff(daily_totals).max()**2
daily_spline = interp.UnivariateSpline(days, daily_totals, s=daily_s)
xx = np.linspace(min(days), max(days), 1000) # 1000 is an arbitrary number here.
plt.plot(days, daily_totals, 'bo')
plt.plot(xx, daily_spline(xx), 'r')
plt.show()
This is less useful. Maybe we need a longer period of observations. Maybe we should not pretend that Monday comes after Friday. Maybe averages should be taken for each day of week to uncover a weekly pattern, but with only two weeks there is not enough to play with.
Technical details: the method UnivariateSpline chooses as few knots as possible so that a certain weighed sum of squared deviations from the data is at most s. With large s this will mean very few knots, until none remain, and we get a single cubic polynomial. How large s needs to be depends on the amount of oscillation in the vertical direction, which is extremely high in this example.