I have a time series in which i am trying to detect anomalies. The thing is that with those anomalies i want to have a range for which the data points should lie to avoid being the anomaly point. I am using the ML .Net algorithm to detect anomalies and I have done that part but how to get range?
If by some way I can get the range for the points in time series I can plot them and show that the points outside this range are anomalies.
I have tried to calculate the range using prediction interval calculation but that doesn't work for all the data points in the time series.
Like, assume I have 100 points, I take 100/4, i.e 25 as the sliding window to calculate the prediction interval for the next point, i.e 26th point but the problem then arises is that how to calculate the prediction interval for the first 25 points?
A method operating on a fixed-length sliding window generally needs that entire window to be filled, in order to make an output. In that case you must pad the input sequence in the beginning if you want to get predictions (and thus anomaly scores) for the first datapoints. It can be hard to make that padded data realistic, however, which can lead to poor predictions.
A nifty technique is to compute anomaly scores with two different models, one going in the forward direction, the other in the reverse direction, to get scores everywhere. However now you must decide how to handle the ares where you have two sets of predictions - to use min/max/average anomaly score.
There are some models that can operate well on variable-length inputs, like sequence to sequence models made with Recurrent Neural Networks.
Related
I set up a sensor which measures temperature data every 3 seconds. I collected the data for 3 days and have 60.000 rows in my csv export. Now I would like to forecast the next few days. When looking at the data you can already see a "seasonality" which displays the fridges heating and cooling cycle so I guess it shouldn't be too difficult to predict. I am not really sure if my data is too granular and if I should do some kind of undersampling. I thought about using a seasonal ARIMA model but I am having difficulties with picking parameters. As the seasonality in the data is pretty obious is there maybe a model that fits better? Please bear with me I'm pretty new to machine learning.
When the goal is to forecast rising temperatures, you can forecast the lower and upper peaks, i.e., their hight and distances. Assuming (simplified model) that the temperature change in between is linear we can, model each complete peak starting from a first lower peak of the temperature curve to the next upper peak down to next lower peak. So a complete peak can be seen as triangle which we easily integrate (calculate its area + the area of the rectangle below of it). The estimation can now be done by a integrating a number of complete peaks we have already measured. By repeating this procedure, we can do now a linear regression on the average temperatures and alert when the slope is above a defined threshold.
As this only tackles a certain kind of errors, one can do the same for the average distances between the upper peaks and the also for the lower peaks. I.e., take the times between them for a certain periode, fit a curve (linear regression can possibly be sufficient) and alert when the slope of the curve is indicating too long distances.
It's mission impossible. If fridge work without interference, then graph always looks the same. The change can be caused, for example, by opening a door, a breakdown, a major change in external conditions. But you cannot predict such events. Instead, you can try to warn about the possibility of problems in the near future, for example, based on a constant increase in average temperature. This situation may indicate a leak in the cooling system.
By the way, have you considered logging the temperature every 3 seconds? This is usually unjustified, because it is physically impossible for the temperature to change to a measurable degree in such an interval. Our team usually sets the login interval to 30 or 60 seconds in such cases. Sometimes even more. Depending on the size of the chamber, the way the air is circulated, the ratio of volume to power of the refrigeration unit, etc.
I have one FB Prophet demand forecast time-series for each of three consumer classes. I want to add these three models and get a confidence interval for the total demand---the stakeholders may look at everything and choose different assumptions for each consumer class. How should I approach that?
I have tried:
using the usual square root of the summation of variances plus two times the covariances for each point in time. That returned a much wider uncertainty interval than it makes sense (the summation of the historical series falling entirely well within two standard deviations), maybe because of trend uncertainty. Also, the distribution around each point in the series time is not normal, so just getting the standard deviation won't do.
adding sample forecasts from each model and using them to estimate the confidence intervals. But then I remembered that the samples wouldn't be correlated.
Any other ideas? Is there any way for me to correlate samples from the three models?
With python I want to compare a simulated light curve with the real light curve. It should be mentioned that the measured data contain gaps and outliers and the time steps are not constant. The model, however, contains constant time steps.
In a first step I would like to compare with a statistical method how similar the two light curves are. Which method is best suited for this?
In a second step I would like to fit the model to my measurement data. However, the model data is not calculated in Python but in an independent software. Basically, the model data depends on four parameters, all of which are limited to a certain range, which I am currently feeding mannualy to the software (planned is automatic).
What is the best method to create a suitable fit?
A "Brute-Force-Fit" is currently an option that comes to my mind.
This link "https://imgur.com/a/zZ5xoqB" provides three different plots. The simulated lightcurve, the actual measurement and lastly both together. The simulation is not good, but by playing with the parameters one can get an acceptable result. Which means the phase and period are the same, magnitude is in the same order and even the specular flashes should occur at the same period.
If I understand this correctly, you're asking a more foundational question that could be better answered in https://datascience.stackexchange.com/, rather than something specific to Python.
That said, as a data science layperson, this may be a problem suited for gradient descent with a mean-square-error cost function. You initialize the parameters of the curve (possibly randomly), then calculate the square error at your known points.
Then you make tiny changes to each parameter in turn, and calculate how the cost function is affected. Then you change all the parameters (by a tiny amount) in the direction that decreases the cost function. Repeat this until the parameters stop changing.
(Note that this might trap you in a local minimum and not work.)
More information: https://towardsdatascience.com/implement-gradient-descent-in-python-9b93ed7108d1
Edit: I overlooked this part
The simulation is not good, but by playing with the parameters one can get an acceptable result. Which means the phase and period are the same, magnitude is in the same order and even the specular flashes should occur at the same period.
Is the simulated curve just a sum of sine waves, and are the parameters just phase/period/amplitude of each? In this case what you're looking for is the Fourier transform of your signal, which is very easy to calculate with numpy: https://docs.scipy.org/doc/scipy/reference/tutorial/fftpack.html
I am wondering what is difference between filtering and interpolating data.
I am now comparing
savgol_filter(itp(xx), window_size, poly_order)
and
itp = interp1d(x,y, kind='nearest')
I understand that filter filters noise in data, so that they are more smooth.
But the same does the interpolating.
My purpose is to smooth data, so that data will be ever rising.
If they are not ever rising then adjust only those values that break it.
And if there are rising = NO ADJUSTMENT.
What would you recommend to use?
Thanks!
Suppose you have a series of discrete data points, measured for example at specific time.
Interpolation is a way to guess the value of the series at a time in-between two measurements. For example, the temperature is measured every hours, but you want to have the temperature value every half-hour. If the data is noisy, the interpolation will be noisy too.
Filtering is a way to reduce the noise in the data. The value given by the measurement is the real value plus a random noise. For multiple measurements, the real value is assumed to remain identical, while the noise value will change positively and negatively. Therefore, by taking the average over enough multiple measurements the contribution of the noise averaged to zero.
Fitting a model over the data is another similar way to remove the noise from the data. Actually, taking the average is similar to fit the data using a straight horizontal line as a model. Doing a linear regression is fitting with any line i.e. finding the straight line which best describes the data.
The Savitzky–Golay filter performs successive fits over successive parts of the data series (window) in order to reduce locally the noise.
I'm trying to compare the data (black) and the model (color). [Fig. 1]
There is another example [Fig. 2]. The data set and the model are different for Fig. 1 and Fig. 2.
In both the cases, it appears that there are overlaps between the model and data, however, the overlap/matching is better for Fig. 2. I would like to quantify the correlation of the data and the model for both the cases in order to distinguish between the "goodness of fit" of both the figures. I was wondering which (statistical) method I should use to estimate the correlation quantitatively.
You could start by calculating the center of gravity for each dataset using numpy.mean and compare how close their are to one another.
Next step is to check the if each center is inside the uncertainty ellipse (http://www.visiondummy.com/2014/04/draw-error-ellipse-representing-covariance-matrix/) of the other dataset.
Finally, I would recommend to using hypotheses testing like student's test or f-test. There are some methods in scipy for these kind of test, just look at the documentation