Develop Reference

Forecasting product return rates based on past returns

I have a waterfall dataset which shows the returns of laptops over different weeks due to defects. For sales of each month it shows returns over different months
I transformed the data to a weekly level: For example the returns column will be the returns of the first month for each sale month.(it will be the sum of the first diagonal
I fit the pdf into a 2 parameter weibull distribution which models failure rates. (to approximate a bathtub curve)
I then used the fit curve to estimate reliability rates form the predicted cdf. Using that the unreliability rate is predicted.
However the Weibull distribution does not accurately model a bathtub curve which is frequent with failure rates and hence the accuracy is quite low.
Is there a better approach to this problem of predicted return rates?

A variance of how much is acceptable for a column before we can use the column variable for Modelling?

I am trying to create a classification model. While pre-processing the data. I look at the variance in each column. This is the amount of variance in each column. I am confused on which all columns should I log transform before modelling. How much variance is acceptable? Could somebody please shed some light on this please.
Temparature 2.318567e-01
HR 4.747868e+02
SpO2 1.179291e+01
SBP 6.263887e+02
MAP 2.905884e+02
RR 2.794205e+01
FiO2 9.061920e+00
PaO2 1.327011e+03
PaCO2 7.466527e+01
pH 4.851681e-03
A.a.gradient 0.000000e+00
HCO3 1.358290e+01
Hb 5.337076e+00
TLC 6.326940e+07
Platelets 1.062145e+10
K 3.332203e-01
Na 4.429681e+01
Serum.Cr 1.897277e+00
Blood.Urea 7.321509e+02
Bili 3.352918e+00
Urine.output 5.157271e+05
Lactate 3.795719e+00
INR 5.362644e-01
dtype: float64

I would say that looking only at variance of columns is mostly useful to delete columns with 0 variance.
If your column has at least minimal variance, you cannot conclude that the column is useless without further investigation.

I would say it depends on the priors you have on the data. There isn't an "acceptable range of variance" unless it comes with context.
For classification purposes it is best to train on as much samples as you can, but you do want to leave some for validation as #desertnaut has suggested.
buttom line: I would take the upper (say) 80% most variable columns and log transform them. The other 20% will remain for validation.

fbprophet yearly seasonality values too high

I have recently started using fbprophet package in python. I have monthly data for last 2 years and forecasting for next 9 months.
Since, I have monthly data I have only included yearly seasonality (Prophet(yearly_seasonality = True)).
When I plot components, trend values seem to be fine however, Yearly seasonality values are too high, I don't understand why?
Seasonality is showing 300 increase or -200 decrease. However, in actual graph it is not happening in any months in the past - what I can do to correct?
Code Used is as follows:
m = Prophet(yearly_seasonality = True)
m.fit(df_bu_country1)
future = m.make_future_dataframe(periods=9, freq='M')
forecast = m.predict(future)
m.plot(forecast)
m.plot_components(forecast)

There is no seasonality at all in your data. For there to be yearly seasonality, you should have a pattern that repeats year after year, but the shape of your time series from 10/2015 to 10/2016 is completely different from the shape between 10/2016 to 10/2017. So forcing a yearly seasonality is going to give you strange results, you should switch it off (i.e. just use Prophet's default settings).

There is an inconsistency in the seasonality factor of your data, there seems a little yearly seasonality between 2017-04 to 2018-10 . The first answer is absolutely true but incase if you feel there some seasonality you can reduce its impact by altering fourier order it has.
https://facebook.github.io/prophet/docs/seasonality,_holiday_effects,_and_regressors.html#fourier-order-for-seasonalities
This page has how to do so, the default fourier order is 10, reducing the values cahnges it effects.
Try this hope it helps you

Python - How do I check time series stationarity?

I have a car speed dataset on a highway. The observations are collected at 15 min steps, which means I have 96 observations per day and 672 per week.
I have a whole month dataset (2976 observations)
My goal is to predict future values using an Autoregressive AR(p) model.
Here's my data repartition over the month.
In addition, here's the autocorrelation plot (ACF)
The visualization of the 2 plots above lead to think of a seasonal component and hence, a non-stationnary time series, which for me makes no doubt.
However, to make sure of the non-stationarity, I applied on it a Dickey-Fuller test. Here are the results.
Results of Dickey-Fuller Test:
Test Statistic -1.666334e+01
p-value 1.567300e-29
#Lags Used 3.000000e+00
Number of Observations Used 2.972000e+03
Critical Value (5%) -2.862513e+00
Critical Value (1%) -3.432552e+00
Critical Value (10%) -2.567288e+00
dtype: float64
The results clearly show that the absolute value of Test statistic is greater than the critical values, therefore, we reject the null hypothesis which means we have a stationary series !
So I'm very confused of the seasonality and stationarity of my time series.
Any help about that would be appreciated.
Thanks a lot

Actually, stationarity and seasonality are not controversial qualities. Stationarity represent a constancy (no variation) on the series moments (such as mean, variance for weak stationarity), and seasonality is a periodic component of the series that can be extracted with filters.
Seasonality and cyclical patterns are not exactly the same thing, but are very close. You can think as if this series in the images that you show can have a sum of sines and cosines that repeats itself for weekly (or monthly, yearly, ...) periods. It does not have any correlation with the fact that the mean value of the series seems to be constant over the period, or even variance.

Holt-Winters for multi-seasonal forecasting in Python

My data: I have two seasonal patterns in my hourly data... daily and weekly. For example... each day in my dataset has roughly the same shape based on hour of the day. However, certain days like Saturday and Sunday exhibit increases in my data, and also slightly different hourly shapes.
(Using holt-winters, as I found discovered here: https://gist.github.com/andrequeiroz/5888967)
I ran the algorithm, using 24 as my periods per season, and forecasting out 7 seasons (1 week), I noticed that it would over-forecast the weekdays and under-forecast the weekend since its estimating the saturday curve based on fridays curve and not a combination of friday's curve and saturday(t-1)'s curve. What would be a good way to include a secondary period in my data, as in, both 24 and 7? Is their a different algorithm that I should be using?

One obvious way to account for different shapes would be to use just one sort of period, but make it have a periodicity of 7*24, so you would be forecasting the entire week as a single shape.
Have you tried linear regression, in which the predicted value is a linear trend plus a contribution from dummy variables? The simplest example to explain would be trend plus only a daily contribution. Then you would have
Y = X*t + c + A*D1 + B*D2 + ... F * D6 (+ noise)
Here you use linear regression to find the best fitting values of X, c, and A...F. t is the time, counting up 0, 1, 2, 3,... indefinitely, so the fitted value of X gives you a trend. c is a constant value, so it moves all the predicted Ys up or down. D1 is set to 1 on Tuesdays and 0 otherwise, D2 is set to 1 on Wednesdays and 0 otherwise... D6 is set to 1 on Sundays and 0 otherwise, so the A..F terms give contributions for days other than Mondays. We don't fit a term for Mondays because if we did then we could not distinguish the c term - if you added 1 to c and subtracted one from each of A..F the predictions would be unchanged.
Hopefully you can now see that we could add 23 terms to account for an shape for the 24 hours of each day, and a total of 46 terms to account for a shape for the 24 hours of each weekday and the different 24 hours of each weekend day.
You would be best to look for a statistical package to handle this for you, such as the free R package (http://www.r-project.org/). It does have a bit of a learning curve, but you can probably find books or articles that take you through using it for just this sort of prediction.
Whatever you do, I would keep on checking forecasting methods against your historical data - people have found that the most accurate forecasting methods in practice are often surprisingly simple.