I have a dataframe similar to the one below.
dry_bulb_tmp_mean Time Timedelta
Date
2011-01-14 -11.245833 2011-01-14 NaN
2011-01-15 -12.608333 2011-01-15 1.0
2011-01-16 -15.700000 2011-01-16 1.0
2011-01-17 -19.954167 2011-01-17 1.0
2011-01-20 -13.654167 2011-01-20 3.0
2011-01-21 -11.887500 2011-01-21 1.0
2011-01-22 -17.866667 2011-01-22 1.0
2011-01-23 -23.250000 2011-01-23 1.0
2011-01-24 -23.654167 2011-01-24 1.0
2011-01-25 -16.254167 2011-01-25 1.0
2011-01-30 -12.233333 2011-01-30 5.0
2011-01-31 -19.041667 2011-01-31 1.0
I am tasked with creating a new dataframe that gives me the lengths for different runs. Basically, a run is however many consecutive days occur in the dataframe. For example, from the 14th, to the 17th I get a run of 4, but then at the 20th I get a run of 1. I have attempted to do this as follows.
if temp_persis33['Timedelta'].iloc[row] == 1:
length += 1
Every time a value greater than 1 is found in the Timedelta column, it will append the counter to a list, and then reset the counter. However, I am not sure how to compare the values in the dataframe. I have tried a few different things and nothing has worked. Any help would be appreciated, thank you.
IIUC try grouping on a boolean array where Timedelta does not equal 1
df.groupby(df['Timedelta'].ne(1).cumsum())['Time'].count().to_numpy()
# array([4, 6, 2])
You don't even need to create the Timedelta column. You can apply diff directly to it. pd.DataFrame.diff() works directly with datetime:
df.groupby(df.Time.diff().ne(1).cumsum()).Time.count()
Related
I have a pd.Series object with a pd.DatetimeIndex containing dates. I would like to calculate the difference from a past value, for example the one month before. The values are not exactly aligned to the months, so I cannot simply add a monthly date offset. There might also be missing data.
So I would like to match the previous value using an offset and a tolerance. One way to do this is using the .reindex() method with method='nearest' which matches the previous data point almost like I want to:
shifted = data.copy()
shifted.index = shifted.index + pd.DateOffset(months=1)
shifted = shifted.reindex(
data.index,
method="nearest",
tolerance=timedelta(days=100),
)
return data - shifted
Here we calculate the difference from the value one month before, but we tolerate finding a value 100 days around that timestamp.
This is almost what I want, but I want to avoid subtracting the value from itself. I always want to subtract a value in the past, or no value at all.
For example: if this is the data
2020-01-02 1.0
2020-02-03 2.0
2020-04-05 3.0
And I use the code above, the last data point, 3.0 will be subtracted from itself, since its date is closer to 2020-05-05 than to 2020-03-03. And the result will be
2020-01-02 0.0
2020-02-03 1.0
2020-04-05 0.0
While the goal is to get
2020-01-02 NaN
2020-02-03 1.0
2020-04-05 1.0
Additional edit after Baron Legendre's answer (thanks for pointing out the flaw in my question):
The tolerance variable is also important to me. So let's say there is a gap of a year in the data, that falls outside the tolerance of 100 days, and the result should be NaN:
2015-12-04 10.0
2020-01-02 1.0
2020-02-03 2.0
2020-04-05 3.0
Should result in:
2015-12-05 NaN (because there is no past value to subtract)
2020-01-02 NaN (because the past value is too far back)
2020-02-03 1.0
2020-04-05 1.0
Hope that explains the problem well enough. Any ideas on how to do this efficiently, without looping over every single data point?
ser
###
2015-12-04 10
2020-01-02 1
2020-02-03 2
2020-04-05 3
dtype: int64
df = ser.reset_index()
tdiff = df['index'].diff().dt.days
ser[:] = np.where(tdiff > 100, np.nan, ser - ser.shift())
ser
###
2015-12-04 NaN
2020-01-02 NaN
2020-02-03 1.0
2020-04-05 1.0
dtype: float64
I am working with UPC (product#), date_expected, and quantity_picked columns and need my data organized to show the total quantity_picked per day (for every day) for each UPC. Example data below:
UPC quantity_picked date_expected
0 0001111041660 1.0 2019-05-14 15:00:00
1 0001111045045 1.0 2019-05-14 15:00:00
2 0001111050268 1.0 2019-05-14 15:00:00
3 0001111086132 1.0 2019-05-14 15:00:00
4 0001111086983 1.0 2019-05-14 15:00:00
5 0001111086984 1.0 2019-05-14 15:00:00
... ... ...
39694 0004470036000 6.0 2019-06-24 20:00:00
39695 0007225001116 1.0 2019-06-24 20:00:00
I was able to successfully organize my data in this manner using the code below, but the output leaves out dates with quantity_picked=0
orders = pd.read_sql_query(SQL, con=sql_conn)
order_daily = orders.copy()
order_daily['date_expected'] = order_daily['date_expected'].dt.normalize()
order_daily['date_expected'] = pd.to_datetime(order_daily.date_expected, format='%Y-%m-%d')
# Groups by date and UPC getting the sum of quanitity picked for each
# then resets index to fill in dates for all rows
tipd = order_daily.groupby(['UPC', 'date_expected']).sum().reset_index()
# Rearranging of columns to put UPC column first
tipd = tipd[['UPC','date_expected','quantity_picked']]
gives the following output:
UPC date_expected quantity_picked
0 0000000002554 2019-05-21 4.0
1 0000000002554 2019-05-24 2.0
2 0000000002554 2019-06-02 2.0
3 0000000002554 2019-06-17 2.0
4 0000000003082 2019-05-15 2.0
5 0000000003082 2019-05-16 2.0
6 0000000003082 2019-05-17 8.0
... ... ...
31588 0360600051715 2019-06-17 1.0
31589 0501072452748 2019-06-15 1.0
31590 0880100551750 2019-06-07 2.0
When I try to follow the solution given in:
Pandas filling missing dates and values within group
I adjust my code to
tipd = order_daily.groupby(['UPC', 'date_expected']).sum().reindex(idx, fill_value=0).reset_index()
# Rearranging of columns to put UPC column first
tipd = tipd[['UPC','date_expected','quantity_picked']]
# Viewing first 10 rows to check format of dataframe
print('Preview of Total per Item per Day')
print(tipd.iloc[0:10])
And receive the following error:
TypeError: Argument 'tuples' has incorrect type (expected numpy.ndarray, got DatetimeArray)
I need each date to be listed for each product, even when quantity picked is zero. I plan on creating two new columns using .shift and .diff for calculations, and those columns will not be accurate if my data is skipping dates.
Any guidance is very much appreciated.
I'm having trouble using a custom transformation with an expanding window. I have a function:
def minutes_since_previous(x):
'''given datetime-indexed series x,
returns time difference in minutes between the last datetime
and the datetime of the previous nonnull value in each column.'''
x = pd.Series(x) # assumes aready sorted, else add .sort_index()
tfinal=x.index[-1]
allprev = x[:-1].dropna() #all prev nonnulls.
if (len(allprev) > 0):
tprev = allprev.index[-1]
return (tfinal - tprev)
else:
return np.nan
and a series:
x
Out[116]:
datetime
2017-06-05 22:01:20.000 81.070099
2017-06-05 22:25:24.235 NaN
2017-06-05 22:33:13.000 NaN
2017-06-05 23:10:06.208 NaN
2017-06-05 23:11:40.437 NaN
2017-06-05 23:30:32.911 NaN
2017-06-06 06:19:41.934 NaN
2017-06-06 06:37:11.432 NaN
2017-06-06 06:41:37.000 93.681000
dtype: float32
The function works as expected on the series as a whole:
In [117]:
minutes_since_previous(x)
Out[117]:
Timedelta('0 days 08:40:17')
But when applied to the expanding window, I lose the datetime nature of the index, and instead of giving timedeltas since the first timestamp, I just get integer differences:
In [118]:
x.expanding().apply(minutes_since_previous)
Out[118]:
datetime
2017-06-05 22:01:20.000 NaN
2017-06-05 22:25:24.235 1.0
2017-06-05 22:33:13.000 2.0
2017-06-05 23:10:06.208 3.0
2017-06-05 23:11:40.437 4.0
2017-06-05 23:30:32.911 5.0
2017-06-06 06:19:41.934 6.0
2017-06-06 06:37:11.432 7.0
2017-06-06 06:41:37.000 8.0
dtype: float64
Perhaps .expanding() resets the index internally, but if so, how can I get what I'm looking for?
I searched "pandas expanding index" and found a question about multiple columns,
and many, many that were really about various kinds of resampling, but none dealing with losing index information.
Thanks ...
I have a Python Dataframe that looks like this:
Facility PUE PUEraw Servers
2016-11-14 00:00:00 6.0 NaN 1.2 5.0
2016-11-14 00:30:00 6.0 NaN 1.2 5.0
2016-11-14 01:00:00 6.0 NaN 1.2 5.0
etc.
As you can see, the index is date/time. The dataframe is updated with a new value every half hour.
I'm trying to write a script that removes all rows except those that correspond to TODAY's date, for which I am utilising date = dt.datetime.today(). However, I am struggling, partly perhaps because the index also contains the time.
Does anyone have any suggestions? Alternatively, a script that removes all but the last 48 rows would also work for me (the last 48 x half hourly values = the latest day's data).
Here are two options you can use to extract data on a specific day:
df['2016-11-16']
# Facility PUE PUEraw Servers
# 2016-11-16 01:00:00 6.0 NaN 1.2 5.0
import datetime
df[df.index.date == datetime.datetime.today().date()]
# Facility PUE PUEraw Servers
# 2016-11-16 01:00:00 6.0 NaN 1.2 5.0
You can always access the last rows in a DataFrame with df.tail()
df = df.tail(48)
For further information:
Pandas Documentation
I have a dataframe (df) where column A is drug units that is dosed at time point given by Timestamp. I want to fill the missing values (NaN) with the drug concentration given the half-life of the drug (180mins). I am struggling with the code in pandas . Would really appreciate help and insight. Thanks in advance
df
A
Timestamp
1991-04-21 09:09:00 9.0
1991-04-21 3:00:00 NaN
1991-04-21 9:00:00 NaN
1991-04-22 07:35:00 10.0
1991-04-22 13:40:00 NaN
1991-04-22 16:56:00 NaN
Given the half -life of the drug is 180 mins. I wanted to fillna(values) as a function of time elapsed and the half life of the drug
something like
Timestamp A
1991-04-21 09:00:00 9.0
1991-04-21 3:00:00 ~2.25
1991-04-21 9:00:00 ~0.55
1991-04-22 07:35:00 10.0
1991-04-22 13:40:00 ~2.5
1991-04-22 16:56:00 ~0.75
Your timestamps are not sorted and I'm assuming this was a typo. I fixed it below.
import pandas as pd
import numpy as np
from StringIO import StringIO
text = """TimeStamp A
1991-04-21 09:09:00 9.0
1991-04-21 13:00:00 NaN
1991-04-21 19:00:00 NaN
1991-04-22 07:35:00 10.0
1991-04-22 13:40:00 NaN
1991-04-22 16:56:00 NaN """
df = pd.read_csv(StringIO(text), sep='\s{2,}', engine='python', parse_dates=[0])
This is the magic code.
# half-life of 180 minutes is 10,800 seconds
# we need to calculate lamda (intentionally mis-spelled)
lamda = 10800 / np.log(2)
# returns time difference for each element
# relative to first element
def time_diff(x):
return x - x.iloc[0]
# create partition of non-nulls with subsequent nulls
partition = df.A.notnull().cumsum()
# calculate time differences in seconds for each
# element relative to most recent non-null observation
# use .dt accessor and method .total_seconds()
tdiffs = df.TimeStamp.groupby(partition).apply(time_diff).dt.total_seconds()
# apply exponential decay
decay = np.exp(-tdiffs / lamda)
# finally, forward fill the observations and multiply by decay
decay * df.A.ffill()
0 9.000000
1 3.697606
2 0.924402
3 10.000000
4 2.452325
5 1.152895
dtype: float64