Given the following code with rolling windows of 3:
import pandas as pd
df=pd.DataFrame({"date":['1/1/2021','1/2/2021','1/3/2021','1/4/2021','1/5/2021','1/1/2021','1/2/2021','1/3/2021','1/4/2021','1/5/2021'],"item_name":["bracelet","bracelet","bracelet","bracelet","bracelet","earring","earring","earring","earring","earring"],"quantity_sold":[1,2,3,4,5,100,200,300,400,500]})
df['date']=pd.to_datetime(df['date'])
display(df)
#sort on the right fields before the calculation
df=df.sort_values(['date','item_name'])
#sum of quantity for last 3 days (curr_day-2,curr_day-1,curr_day)
display(df.set_index("date").groupby("item_name").rolling(3).agg('sum'))
the result is:
Is it possible to have the first two value calculated without NaN - e.g., on bracelet, 2021-01-01, since we have 1 element, we use rolling windows of 1 and get value=1; on bracelet, 2021-01-02, since we have 2 element, we use rolling windows of 2 and get value=3?
(So similarly we have bracelet, 2021-01-01 with value =100 and bracelet, 2021-01-02 with value =300)
You can use the min_periods keyword from the documentation of rolling():
df.set_index("date").groupby("item_name").rolling(3,min_periods=1).agg('sum')
min_periods: int, default None
Minimum number of observations in window
required to have a value (otherwise result is NA). For a window that
is specified by an offset, min_periods will default to 1. Otherwise,
min_periods will default to the size of the window.
This will give you:
quantity_sold
item_name date
bracelet 2021-01-01 1.0
2021-01-02 3.0
2021-01-03 6.0
2021-01-04 9.0
2021-01-05 12.0
earring 2021-01-01 100.0
2021-01-02 300.0
2021-01-03 600.0
2021-01-04 900.0
2021-01-05 1200.0
Related
I have a pd.Series object with a pd.DatetimeIndex containing dates. I would like to calculate the difference from a past value, for example the one month before. The values are not exactly aligned to the months, so I cannot simply add a monthly date offset. There might also be missing data.
So I would like to match the previous value using an offset and a tolerance. One way to do this is using the .reindex() method with method='nearest' which matches the previous data point almost like I want to:
shifted = data.copy()
shifted.index = shifted.index + pd.DateOffset(months=1)
shifted = shifted.reindex(
data.index,
method="nearest",
tolerance=timedelta(days=100),
)
return data - shifted
Here we calculate the difference from the value one month before, but we tolerate finding a value 100 days around that timestamp.
This is almost what I want, but I want to avoid subtracting the value from itself. I always want to subtract a value in the past, or no value at all.
For example: if this is the data
2020-01-02 1.0
2020-02-03 2.0
2020-04-05 3.0
And I use the code above, the last data point, 3.0 will be subtracted from itself, since its date is closer to 2020-05-05 than to 2020-03-03. And the result will be
2020-01-02 0.0
2020-02-03 1.0
2020-04-05 0.0
While the goal is to get
2020-01-02 NaN
2020-02-03 1.0
2020-04-05 1.0
Additional edit after Baron Legendre's answer (thanks for pointing out the flaw in my question):
The tolerance variable is also important to me. So let's say there is a gap of a year in the data, that falls outside the tolerance of 100 days, and the result should be NaN:
2015-12-04 10.0
2020-01-02 1.0
2020-02-03 2.0
2020-04-05 3.0
Should result in:
2015-12-05 NaN (because there is no past value to subtract)
2020-01-02 NaN (because the past value is too far back)
2020-02-03 1.0
2020-04-05 1.0
Hope that explains the problem well enough. Any ideas on how to do this efficiently, without looping over every single data point?
ser
###
2015-12-04 10
2020-01-02 1
2020-02-03 2
2020-04-05 3
dtype: int64
df = ser.reset_index()
tdiff = df['index'].diff().dt.days
ser[:] = np.where(tdiff > 100, np.nan, ser - ser.shift())
ser
###
2015-12-04 NaN
2020-01-02 NaN
2020-02-03 1.0
2020-04-05 1.0
dtype: float64
I have a DataFrame that tracks the 'Adj Closing' price for several global markets causing there to be repeating dates. To clean this up I use .set_index(['Index Ticker', 'Date']).
DataFrame sample
My issue is that the Closing Prices run as far back as 1997-07-02 but I only need 2020-01-01 and forward. I tried using idx = pd.IndexSlice followed by df.loc[idx[ :, '2020-01-01':], :] as well as df.loc[(slice(None), '2020-01-01':), :], but both methods return a syntax error on the : that I'm using to slice across a range of dates. Any tips on getting the data I need past a specific date? Thank you in advance!
Try:
# create dataframe to approximate your data
df = pd.DataFrame({'ticker' : ['A']*5 + ['M']*5,
'Date' : pd.date_range(start='2021-01-01', periods=5).tolist() + pd.date_range(start='2021-01-01', periods=5).tolist(),
'high' : range(10)}
).groupby(['ticker', 'Date']).sum()
high
ticker Date
A 2021-01-01 0
2021-01-02 1
2021-01-03 2
2021-01-04 3
2021-01-05 4
M 2021-01-01 5
2021-01-02 6
2021-01-03 7
2021-01-04 8
2021-01-05 9
# evaluate conditions against level 1 (Date) of your multiIndex; level 0 is ticker
df[df.index.get_level_values(1) > '2021-01-03']
high
ticker Date
A 2021-01-04 3
2021-01-05 4
M 2021-01-04 8
2021-01-05 9
Alternatively, if possible, remove the unwanted dates prior to setting your multiIndex.
Thankfully, the contents of the last question were solved well, so I was making a dataset without any problems.(Thank you Ecker!!)
But, there was a new problem.
In the process of calculating the cumulative weighted average through the same process,
When the dataset and type were changed, there were cases where it was not possible to join.
For example, in the dataset below
firm
date
reviewer
compound
A
2021-01-01
a
0.6531
A
2021-01-01
b
-0.7213
A
2021-01-01
c
-0.3168
A
2021-01-02
d
0.3548
A
2021-01-02
e
0.5783
A
2021-01-03
f
0.4298
A
2021-01-04
g
0.8769
B
2021-01-01
h
0.7895
B
2021-01-01
i
-0.4924
B
2021-01-02
j
0.0245
B
2021-01-02
k
0.4982
B
2021-01-03
a
0.1597
B
2021-01-04
b
0.6254
‘The compound value’ is a real number (float64) including a decimal point between -1 and 1.
‘The count number’ is the number of reviewers on a specific date.(int64)
I would like to add a column that calculates the cumulative weighted average as shown in the table below.
firm
date
reviewer
rate
cum_avg_compound
A
2021-01-01
a
0.6531
-0.12833
A
2021-01-01
b
-0.7213
-0.12833
A
2021-01-01
c
-0.3168
-0.12833
A
2021-01-02
d
0.3548
0.10962
A
2021-01-02
e
0.5783
0.10962
A
2021-01-03
f
0.4298
0.162983
A
2021-01-04
g
0.8769
0.264971
B
2021-01-01
h
0.7895
0.14855
B
2021-01-01
i
-0.4924
0.14855
B
2021-01-02
j
0.0245
0.20495
B
2021-01-02
k
0.4982
0.20495
B
2021-01-03
a
0.1597
0.1959
B
2021-01-04
b
0.6254
0.26748
Even if this is converted to the same float64 format, it cannot be combined using join.
The code I tried is as follows.
g = (
df.groupby(['firm', 'date'])['compound']
.agg(['sum', 'count'])
.groupby(level='sid').cumsum()
)
df = df.join(
g ['sum'].div(g_text_emo['count']).rename('cum_avg_compound'),
on=['firm', 'date']
)
Is there any way to solve this problem?
Thank you in advance.
I am having some trouble managing and combining columns in order to get one datetime column out of three columns containing the date, the hours and the minutes.
Assume the following df (copy and type df= = pd.read_clipboard() to reproduce) with the types as noted below:
>>>df
date hour minute
0 2021-01-01 7.0 15.0
1 2021-01-02 3.0 30.0
2 2021-01-02 NaN NaN
3 2021-01-03 9.0 0.0
4 2021-01-04 4.0 45.0
>>>df.dtypes
date object
hour float64
minute float64
dtype: object
I want to replace the three columns with one called 'datetime' and I have tried a few things but I face the following problems:
I first create a 'time' column df['time']= (pd.to_datetime(df['hour'], unit='h') + pd.to_timedelta(df['minute'], unit='m')).dt.time and then I try to concatenate it with the 'date' df['datetime']= df['date'] + ' ' + df['time'] (with the purpose of converting the 'datetime' column pd.to_datetime(df['datetime']). However, I get
TypeError: can only concatenate str (not "datetime.time") to str
If I convert 'hour' and 'minute' to str to concatenate the three columns to 'datetime', then I face the problem with the NaN values, which prevents me from converting the 'datetime' to the corresponding type.
I have also tried to first convert the 'date' column df['date']= df['date'].astype('datetime64[ns]') and again create the 'time' column df['time']= (pd.to_datetime(df['hour'], unit='h') + pd.to_timedelta(df['minute'], unit='m')).dt.time to combine the two: df['datetime']= pd.datetime.combine(df['date'],df['time']) and it returns
TypeError: combine() argument 1 must be datetime.date, not Series
along with the warning
FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime module instead.
Is there a generic solution to combine the three columns and ignore the NaN values (assume it could return 00:00:00).
What if I have a row with all NaN values? Would it possible to ignore all NaNs and 'datetime' be NaN for this row?
Thank you in advance, ^_^
First convert date to datetimes and then add hour and minutes timedeltas with replace missing values to 0 timedelta:
td = pd.Timedelta(0)
df['datetime'] = (pd.to_datetime(df['date']) +
pd.to_timedelta(df['hour'], unit='h').fillna(td) +
pd.to_timedelta(df['minute'], unit='m').fillna(td))
print (df)
date hour minute datetime
0 2021-01-01 7.0 15.0 2021-01-01 07:15:00
1 2021-01-02 3.0 30.0 2021-01-02 03:30:00
2 2021-01-02 NaN NaN 2021-01-02 00:00:00
3 2021-01-03 9.0 0.0 2021-01-03 09:00:00
4 2021-01-04 4.0 45.0 2021-01-04 04:45:00
Or you can use Series.add with fill_value=0:
df['datetime'] = (pd.to_datetime(df['date'])
.add(pd.to_timedelta(df['hour'], unit='h'), fill_value=0)
.add(pd.to_timedelta(df['minute'], unit='m'), fill_value=0))
I would recommend converting hour and minute columns to string and constructing the datetime string from the provided components.
Logically, you need to perform the following steps:
Step 1. Fill missing values for hour and minute with zeros.
df['hour'] = df['hour'].fillna(0)
df['minute'] = df['minute'].fillna(0)
Step 2. Convert float values for hour and minute into integer ones, because your final output should look like 2021-01-01 7:15, not 2021-01-01 7.0:15.0.
df['hour'] = df['hour'].astype(int)
df['minute'] = df['minute'].astype(int)
Step 3. Convert integer values for hour and minute to the string representation.
df['hour'] = df['hour'].astype(str)
df['minute'] = df['minute'].astype(str)
Step 4. Concatenate date, hour and minute into one column of the correct format.
df['result'] = df['date'].str.cat(df['hour'].str.cat(df['minute'], sep=':'), sep=' ')
Step 5. Convert your result column to datetime object.
pd.to_datetime(df['result'])
It is also possible to fullfill all of this steps in one command, though it will read a bit messy:
df['result'] = pd.to_datetime(df['date'].str.cat(df['hour'].fillna(0).astype(int).astype(str).str.cat(df['minute'].fillna(0).astype(int).astype(str), sep=':'), sep=' '))
Result:
date hour minute result
0 2020-01-01 7.0 15.0 2020-01-01 07:15:00
1 2020-01-02 3.0 30.0 2020-01-02 03:30:00
2 2020-01-02 NaN NaN 2020-01-02 00:00:00
3 2020-01-03 9.0 0.0 2020-01-03 09:00:00
4 2020-01-04 4.0 45.0 2020-01-04 04:45:00
I have a Python Dataframe that looks like this:
Facility PUE PUEraw Servers
2016-11-14 00:00:00 6.0 NaN 1.2 5.0
2016-11-14 00:30:00 6.0 NaN 1.2 5.0
2016-11-14 01:00:00 6.0 NaN 1.2 5.0
etc.
As you can see, the index is date/time. The dataframe is updated with a new value every half hour.
I'm trying to write a script that removes all rows except those that correspond to TODAY's date, for which I am utilising date = dt.datetime.today(). However, I am struggling, partly perhaps because the index also contains the time.
Does anyone have any suggestions? Alternatively, a script that removes all but the last 48 rows would also work for me (the last 48 x half hourly values = the latest day's data).
Here are two options you can use to extract data on a specific day:
df['2016-11-16']
# Facility PUE PUEraw Servers
# 2016-11-16 01:00:00 6.0 NaN 1.2 5.0
import datetime
df[df.index.date == datetime.datetime.today().date()]
# Facility PUE PUEraw Servers
# 2016-11-16 01:00:00 6.0 NaN 1.2 5.0
You can always access the last rows in a DataFrame with df.tail()
df = df.tail(48)
For further information:
Pandas Documentation