df = pd.DataFrame({'date': ['3/10/2000', '3/11/2000', '3/12/2000','3/13/2000','3/14/2000','3/15/2000','3/16/2000','3/17/2000','3/18/2000'],
'value': [2,NaN,NaN,NaN,NaN,NaN,NaN,NaN,25]})
In this dataframe, I want to replace the Nan values as with the following logic:
In this case the difference between two dates in terms of days when the value column is not Nan is 8 days i.e. 3/18/2000 - 3/10/2000 = 8 days. And lets say the delta = 23 which we get from subtracting 25-2.
I want to replace the Nan values for all the other t day as 2+(delta)*(t/8) where t is any day with a nan value between the given two non nan value
My desired outcome of value column is :
[2,4.875,7.75,10.625,13.5,16.375,19.25,22.125,25]
You can set the date to timedelta, then as index and interpolate with the 'index' method:
df['value'] = (df
.assign(date=pd.to_datetime(df['date']))
.set_index('date')['value']
.interpolate('index')
.values
)
output:
date value
0 3/10/2000 2.000
1 3/11/2000 4.875
2 3/12/2000 7.750
3 3/13/2000 10.625
4 3/14/2000 13.500
5 3/15/2000 16.375
6 3/16/2000 19.250
7 3/17/2000 22.125
8 3/18/2000 25.000
Raw table:
Column A
5
nan
nan
15
New table:
Column A
5
10
10
15
One option might be the following (using fillna twice (with options ffill and bfill) and then averaging them):
import pandas as pd
import numpy as np
df = pd.DataFrame({'x': [np.nan, 5, np.nan, np.nan, 15]})
filled_series = [df['x'].fillna(method=m) for m in ('ffill', 'bfill')]
print(pd.concat(filled_series, axis=1).mean(axis=1))
# 0 5.0
# 1 5.0
# 2 10.0
# 3 10.0
# 4 15.0
As you can see, this works even if nan happens at the beginning or at the end.
I am to perform seasonal decomposition on around 1000 time series which can contain outliers. I want to replace the outliers with a mean value, since the outlier can corrupt the seasonality extraction.
df
TimeStamp | value
2021-01-01 1
2021-01-02 5
2021-01-03 23
2021-01-04 18
2021-01-05 7
2021-01-06 3
...
Outliers are defined as any sample with an absolute z-score larger than 3.
df['zscore'] = scipy.stats.zscore(df['value'])
I can identify the timestamp of all outliers
(df['zscore'].abs() >= 3]).index
which in above example would return
[2021-01-03,2021-01-04]
Given this list of indexes, how do I replace the value with the mean of the closest previous and next neighbors such that I get the following output?
df_mod
TimeStamp | value
2021-01-01 1
2021-01-02 5
2021-01-03 6
2021-01-04 6
2021-01-05 7
2021-01-06 3
...
Would appreciate any help on how to realize this type of function / logic.
EDIT
There can exists NaN values in the time series from the beginning, which I do not want to replace with mean.
You can replace value to NaN by condition in Series.mask and then for replace by mean sum forward and back filling values with divide by 2:
df = df.reset_index(drop=True)
orig = df.index
df = df.dropna(subset=['value'])
df['value'] = df['value'].mask(df['zscore'].abs() >= 3)
df['value'] = df['value'].ffill().add(df['value'].bfill()).div(2)
df = df.reindex(orig)
Solution without helper column:
zscore = scipy.stats.zscore(df['value'])
df['value'] = df['value'].mask(zscore.abs() >= 3)
df['value'] = df['value'].ffill().add(df['value'].bfill()).div(2)
I have a DataFrame that has dates, assets, and then price/volume data. I'm trying to pull in data from 7 days ago, but the issue is that I can't use shift() because my table has missing dates in it.
date cusip price price_7daysago
1/1/2017 a 1
1/1/2017 b 2
1/2/2017 a 1.2
1/2/2017 b 2.3
1/8/2017 a 1.1 1
1/8/2017 b 2.2 2
I've tried creating a lambda function to try to use loc and timedelta to create this shifting, but I was only able to output empty numpy arrays:
def row_delta(x, df, days, colname):
if datetime.strptime(x['recorddate'], '%Y%m%d') - timedelta(days) in [datetime.strptime(x,'%Y%m%d') for x in df['recorddate'].unique().tolist()]:
return df.loc[(df['recorddate_date'] == df['recorddate_date'] - timedelta(days)) & (df['cusip'] == x['cusip']) ,colname]
else:
return 'nothing'
I also thought of doing something similar to this in order to fill in missing dates, but my issue is that I have multiple indexes, the dates and the cusips so I can't just reindex on this.
merge the DataFrame with itself while adding 7 days to the date column for the right Frame. Use the suffixes argument to name the columns appropriately.
import pandas as pd
df['date'] = pd.to_datetime(df.date)
df.merge(df.assign(date = df.date+pd.Timedelta(days=7)),
on=['date', 'cusip'],
how='left', suffixes=['', '_7daysago'])
Output: df
date cusip price price_7daysago
0 2017-01-01 a 1.0 NaN
1 2017-01-01 b 2.0 NaN
2 2017-01-02 a 1.2 NaN
3 2017-01-02 b 2.3 NaN
4 2017-01-08 a 1.1 1.0
5 2017-01-08 b 2.2 2.0
you can set date and cusip as index and use unstack and shift together
shifted = df.set_index(["date", "cusip"]).unstack().shift(7).stack()
then simply merge shifted with your original df
Having this DataFrame:
import pandas
dates = pandas.date_range('2016-01-01', periods=5, freq='H')
s = pandas.Series([0, 1, 2, 3, 4], index=dates)
df = pandas.DataFrame([(1, 2, s, 8)], columns=['a', 'b', 'foo', 'bar'])
df.set_index(['a', 'b'], inplace=True)
df
I would like to replace the Series in there with a new one that is simply the old one, but resampled to a day period (i.e. x.resample('D').sum().dropna()).
When I try:
df['foo'][0] = df['foo'][0].resample('D').sum().dropna()
That seems to work well:
However, I get a warning:
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
The question is, how should I do this instead?
Notes
Things I have tried but do not work (resampling or not, the assignment raises an exception):
df.iloc[0].loc['foo'] = df.iloc[0].loc['foo']
df.loc[(1, 2), 'foo'] = df.loc[(1, 2), 'foo']
df.loc[df.index[0], 'foo'] = df.loc[df.index[0], 'foo']
A bit more information about the data (in case it is relevant):
The real DataFrame has more columns in the multi-index. Not all of them necessarily integers, but more generally numerical and categorical. The index is unique (i.e.: there is only one row with a given index value).
The real DataFrame has, of course, many more rows in it (thousands).
There are not necessarily only two columns in the DataFrame and there may be more than 1 columns containing a Series type. Columns usually contain series, categorical data and numerical data as well. Any single column is always single-typed (either numerical, or categorical, or series).
The series contained in each cell usually have a variable length (i.e.: two series/cells in the DataFrame do not, unless pure coincidence, have the same length, and will probably never have the same index anyway, as dates vary as well between series).
Using Python 3.5.1 and Pandas 0.18.1.
This should work:
df.iat[0, df.columns.get_loc('foo')] = df['foo'][0].resample('D').sum().dropna()
Pandas is complaining about chained indexing but when you don't do it that way it's facing problems assigning whole series to a cell. With iat you can force something like that. I don't think it would be a preferable thing to do, but seems like a working solution.
Simply set df.is_copy = False before asignment of new value.
Hierarchical data in pandas
It really seems like you should consider restructure your data to take advantage of pandas features such as MultiIndexing and DateTimeIndex. This will allow you to still operate on a index in the typical way while being able to select on multiple columns across the hierarchical data (a,b, andbar).
Restructured Data
import pandas as pd
# Define Index
dates = pd.date_range('2016-01-01', periods=5, freq='H')
# Define Series
s = pd.Series([0, 1, 2, 3, 4], index=dates)
# Place Series in Hierarchical DataFrame
heirIndex = pd.MultiIndex.from_arrays([1,2,8], names=['a','b', 'bar'])
df = pd.DataFrame(s, columns=heirIndex)
print df
a 1
b 2
bar 8
2016-01-01 00:00:00 0
2016-01-01 01:00:00 1
2016-01-01 02:00:00 2
2016-01-01 03:00:00 3
2016-01-01 04:00:00 4
Resampling
With the data in this format, resampling becomes very simple.
# Simple Direct Resampling
df_resampled = df.resample('D').sum().dropna()
print df_resampled
a 1
b 2
bar 8
2016-01-01 10
Update (from data description)
If the data has variable length Series each with a different index and non-numeric categories that is ok. Let's make an example:
# Define Series
dates = pandas.date_range('2016-01-01', periods=5, freq='H')
s = pandas.Series([0, 1, 2, 3, 4], index=dates)
# Define Series
dates2 = pandas.date_range('2016-01-14', periods=6, freq='H')
s2 = pandas.Series([-200, 10, 24, 30, 40,100], index=dates2)
# Define DataFrames
df1 = pd.DataFrame(s, columns=pd.MultiIndex.from_arrays([1,2,8,'cat1'], names=['a','b', 'bar','c']))
df2 = pd.DataFrame(s2, columns=pd.MultiIndex.from_arrays([2,5,5,'cat3'], names=['a','b', 'bar','c']))
df = pd.concat([df1, df2])
print df
a 1 2
b 2 5
bar 8 5
c cat1 cat3
2016-01-01 00:00:00 0.0 NaN
2016-01-01 01:00:00 1.0 NaN
2016-01-01 02:00:00 2.0 NaN
2016-01-01 03:00:00 3.0 NaN
2016-01-01 04:00:00 4.0 NaN
2016-01-14 00:00:00 NaN -200.0
2016-01-14 01:00:00 NaN 10.0
2016-01-14 02:00:00 NaN 24.0
2016-01-14 03:00:00 NaN 30.0
2016-01-14 04:00:00 NaN 40.0
2016-01-14 05:00:00 NaN 100.0
The only issues is that after resampling. You will want to use how='all' while dropping na rows like this:
# Simple Direct Resampling
df_resampled = df.resample('D').sum().dropna(how='all')
print df_resampled
a 1 2
b 2 5
bar 8 5
c cat1 cat3
2016-01-01 10.0 NaN
2016-01-14 NaN 4.0