I have two signals that I'm trying to see their correlation lag:
It looks like they are synced, so I expect the correlate function to give minimum at zero (because they have anti-correlation every ~100 timesteps).
However, using this code:
yhat1 = np.load('cor1.npy')
yhat2 = np.load('cor2.npy')
corr = np.correlate(yhat1 - np.mean(yhat1),
yhat2 - np.mean(yhat2),
mode='same')
plt.plot(corr)
plt.show()
I'm getting the following (I tried to use 'full ' and 'same' in the mode and got the same result):
Why the minimum is not at 0 as expected but at 250?
Why it seems like there are other significant peaks on both sides of the minimum?
data is here
Numpy's correlation function returns you the auto/cross correlation function depending on the inputs you give. Correlation is same as convolution except you dont apply time reversal to one of the signals. In other words, apply a sliding dot-product between signals.
At t=0, it's normal to get zero correlation as one signal has zero at t=0. However,as you perform this further, signals are fluctuating both in magnitude and sign. Due to (relatively) extreme peaks of signals to each other at different times, correlatino is fluctuiating. THe huge peak is at t=500 because at theat time full overlap occurs between two signals. This happens as your signals extreme peaks are aligned at that moment. After t=500, your overlapped regions decrease and obsreve that the behavior is similar to the case before we've reached to t<500.
Related
I have a noisy square signal which looks like this:
The amplitude is known. To match the complete square, I can create a pattern of the square and apply np.correlate to find where the signal and the pattern maximally overlap. I wanted to apply a similar approach to find the edge, try to correlate with the 2 patterns below:
As the correlation is nothing more than a convolution, this doesn't work. Half the pattern is equal to 0, and the convolution of this half will return 0 no matter the position on the signal; while the other half is equal to -X with X the amplitude. This second half convoluted with the signal will be maximal when the signal amplitude is maximal. On the signal plot, you can observe that the square is not perfect and that the beginning has a slightly larger amplitude. Basically, both correlation leads to a match on the beginning of the square, where the convolution is maximal. The ramp up (end of the square) is not detected.
To avoid this problem, I would like to use a different operation. As I do know the amplitude of the square signal, I can generate a pattern with the correct amplitude, in this case about -0.3. Thus, I would like to take the pattern and slide it across the signal. At each step, I would compute the mean square error and my pattern would match with the signal at the position where the mean square error is minimized. Moreover, I would like to use the same type of setting as for a convolution, 'valid', where the operation is performed only when the 2 arrays fully overlap.
Do you know of an other method; and/or which function, methods I should use? I couldn't find a all-in-one function line np.convolve or np.correlate.
EDIT: Since I couldn't find a pre-coded function in a library, I've coded mine with a while loop. It's pretty inefficient... It's up here on codereview for upgrades.
I think that convolving/correlating your signal with a step function is still pretty close to the optimal solution, since this is similar to matched filtering, which can be proven to be optimal (under certain conditions, noise likely needs to be Gaussian).
The only issue you have is that your template (the step function) contains a DC part. Removing this, will give you the result you want:
import numpy as np
import matplotlib.pyplot as plt
# simulate the signal
signal = np.zeros(4000)
signal[200:-400] = -0.3
signal += 0.005 * np.random.randn(*signal.shape)
plt.plot(signal)
plt.title('Simulated signal')
plt.show()
# convolve with template with non-zero DC
templ = np.zeros(200)
templ[100:] = 1 # step from 0 to 1
plt.plot(np.convolve(signal, templ))
plt.title('Convolution with template with DC component')
plt.show()
# convolve with template without DC
templ_ac = templ - templ.mean() # step from -0.5 to +0.5
plt.plot(np.convolve(signal, templ_ac))
plt.title('Convolution with template without DC component')
plt.show()
Results:
The way to understand this is that convolve(signal, template) = convolve(signal, template_DC) + convolve(signal, template_AC), where template_DC = mean(template) and template_AC = template - template_DC. The first part is the convoltion of the signal with a flat template, which is just a smoothed version of your signal. The second part is the 'edge detection' signal you want. If you do not subtract the AC part of the template, the uninteresting first part dominates the interesting part.
Note that the scaling of the template is not important, the step in the template doesn't have to be 0.3. This will just cause a scale factor in the end result. Also note that this method does not depend on the exact value of the step, so a larger step in your signal will cause a large effect in the edge detection.
If you know that the step is always exactly 0.3, and you want to be insensitive to steps of different amplitude, you could do some sort of least square fitting of the signal with the template, for every possible shift of the templatate, and only trigger a detection of an edge if the residual is small enough. This will be slow, but might give better rejection of steps with the wrong amplitude.
Since you have very little noise you can calculte where the signal drastically changes with a loop, for example:
for i in range(begin+10,end):
if(abs(data[i-10]-data[i])>0.1):
foundChange()
I want to apply Fourier transformation using fft function to my time series data to find "patterns" by extracting the dominant frequency components in the observed data, ie. the lowest 5 dominant frequencies to predict the y value (bacteria count) at the end of each time series.
I would like to preserve the smallest 5 coefficients as features, and eliminate the rest.
My code is as below:
df = pd.read_csv('/content/drive/My Drive/df.csv', sep=',')
X = df.iloc[0:2,0:10000]
dft_X = np.fft.fft(X)
print(dft_X)
print(len(dft_X))
plt.plot(dft_X)
plt.grid(True)
plt.show()
# What is the graph about(freq/amplitude)? How much data did it use?
for i in dft_X:
m = i[np.argpartition(i,5)[:5]]
n = i[np.argpartition(i,range(5))[:5]]
print(m,'\n',n)
Here is the output:
But I am not sure how to interpret this graph. To be precise,
1) Does the graph show the transformed values of the input data? I only used 2 rows of data(each row is a time series), thus data is 2x10000, why are there so many lines in the graph?
2) To obtain frequency value, should I use np.fft.fftfreq(n, d=timestep)?
Parameters:
n : int
Window length.
d : scalar, optional
Sample spacing (inverse of the sampling rate). Defaults to 1.
Returns:
f : ndarray
Array of length n containing the sample frequencies.
How to determine n(window length) and sample spacing?
3) Why are transformed values all complex numbers?
Thanks
I'm gonna answer in reverse order of your questions
3) Why are transformed values all complex numbers?
The output of a Fourier Transform is always complex numbers. To get around this fact, you can either apply the absolute value on the output of the transform, or only plot the real part using:
plt.plot(dft_X.real)
2) To obtain frequency value, should I use np.fft.fftfreq(n, d=timestep)?
No, the "frequency values" will be visible on the output of the FFT.
1) Does the graph show the transformed values of the input data? I only used 2 rows of data(each row is a time series), thus data is 2x10000, why are there so many lines in the graph?
Your graph has so many lines because it's making a line for each column of your data set. Apply the FFT on each row separately (or possibly just transpose your dataframe) and then you'll get more actual frequency domain plots.
Follow up
Would using absolute value or real part of the output as features for a later model have different effect than using the original output?
Absolute values are easier to work with usually.
Using real part
Using absolute value
Here's the Octave code that generated this:
Fs = 4000; % Sampling rate of signal
T = 1/Fs; % Period
L = 4000; % Length of signal
t = (0:L-1)*T; % Time axis
freq = 1000; % Frequency of our sinousoid
sig = sin(freq*2*pi*t); % Fill Time-Domain with 1000 Hz sinusoid
f_sig = fft(sig); % Apply FFT
f = Fs*(0:(L/2))/L; % Frequency axis
figure
plot(f,abs(f_sig/L)(1:end/2+1)); % peak at 1kHz)
figure
plot(f,real(f_sig/L)(1:end/2+1)); % main peak at 1kHz)
In my example, you can see the absolute value returned no noise at frequencies other than the sinusoid of frequency 1kHz I generated while the real part had a bigger peak at 1kHz but also had much more noise.
As for effects, I don't know what you mean by that.
is it expected that "frequency values" always be complex numbers
Always? No. The Fourier series represents the frequency coefficients at which the sum of sines and cosines completely equate any continuous periodic function. Sines and cosines can be written in complex forms through Euler's formula. This is the most convenient way to store Fourier coefficients. In truth, the imaginary part of your frequency-domain signal represents the phase of the signal. (i.e if I have 2 sine functions of the same frequency, they can have different complex forms depending on the time shifting). However, most libraries that provide an FFT function will, by default, store FFT coefficients as complex numbers, to facilitate phase and magnitude calculations.
Is it convention that FFT use each column of dataset when plotting a line
I think it is an issue with mathplotlib.plot, not np.fft.
Could you please show me how to apply FFT on each row separately
There are many ways to go around this and I don't want to force you down one path, so I will propose the general solution to iterate over each row of your dataframe and apply the FFT on each specific row. Otherwise, in your case, I believe transposing your output could also work.
my original problem was the following:
I have a pulse-envelope in an array a (0-element = time 0, last element = time T).
I want to fourier spectrum of the pulse. So what I did was np.fft.fftshift(np.fft.fft(a)). All good.
But then I was told to do a shift beforehand, too: np.fft.fftshift(np.fft.fft(np.fft.fftshift(a))). Then oscillations arised.
Now I wonder why one would do 2 shifts as shown above and why oscillations arise...
Here the example:
I have the following code
x = np.arange(100)
a =np.sin(np.pi*x**2/1000)
a:
a_fft = np.fft.fft(a)
a_fft:
a_fft_shift = np.fft.fftshift(a_fft)
a_fft_shift:
a_shift = np.fft.fftshift(a)
a_shift_fft = np.fft.fft(a_shift)
a_shift_fft:
a_shift_fft_shift = np.fft.fftshift(a_shift_fft)
a_shift_fft_shift:
Your line
a_shift = np.fft.fftshift(a)
reorders your original time-domain signal. That means in terms of FFT that you are altering the phases.
Note also that there is a discontinuity in your signal. By the line above, this discontinuity is shifted to the center of the signal, which causes the FFT to produce an infinite amount of high frequency cosine components. If you shift it to another place, the energy will be distributed accordingly.
The other problem is that you are only considering the real part of the spectrum, i. e., the cosine components. Always look at the imaginary part, too!
Take also a look at the magnitude spectrum to see that the position of the discontinuity only affects the phase. The total energy remains always the same.
I am trying to use a fast fourier transform to extract the phase shift of a single sinusoidal function. I know that on paper, If we denote the transform of our function as T, then we have the following relations:
However, I am finding that while I am able to accurately capture the frequency of my cosine wave, the phase is inaccurate unless I sample at an extremely high rate. For example:
import numpy as np
import pylab as pl
num_t = 100000
t = np.linspace(0,1,num_t)
dt = 1.0/num_t
w = 2.0*np.pi*30.0
phase = np.pi/2.0
amp = np.fft.rfft(np.cos(w*t+phase))
freqs = np.fft.rfftfreq(t.shape[-1],dt)
print (np.arctan2(amp.imag,amp.real))[30]
pl.subplot(211)
pl.plot(freqs[:60],np.sqrt(amp.real**2+amp.imag**2)[:60])
pl.subplot(212)
pl.plot(freqs[:60],(np.arctan2(amp.imag,amp.real))[:60])
pl.show()
Using num=100000 points I get a phase of 1.57173880459.
Using num=10000 points I get a phase of 1.58022110476.
Using num=1000 points I get a phase of 1.6650441064.
What's going wrong? Even with 1000 points I have 33 points per cycle, which should be enough to resolve it. Is there maybe a way to increase the number of computed frequency points? Is there any way to do this with a "low" number of points?
EDIT: from further experimentation it seems that I need ~1000 points per cycle in order to accurately extract a phase. Why?!
EDIT 2: further experiments indicate that accuracy is related to number of points per cycle, rather than absolute numbers. Increasing the number of sampled points per cycle makes phase more accurate, but if both signal frequency and number of sampled points are increased by the same factor, the accuracy stays the same.
Your points are not distributed equally over the interval, you have the point at the end doubled: 0 is the same point as 1. This gets less important the more points you take, obviusly, but still gives some error. You can avoid it totally, the linspace has a flag for this. Also it has a flag to return you the dt directly along with the array.
Do
t, dt = np.linspace(0, 1, num_t, endpoint=False, retstep=True)
instead of
t = np.linspace(0,1,num_t)
dt = 1.0/num_t
then it works :)
The phase value in the result bin of an unrotated FFT is only correct if the input signal is exactly integer periodic within the FFT length. Your test signal is not, thus the FFT measures something partially related to the phase difference of the signal discontinuity between end-points of the test sinusoid. A higher sample rate will create a slightly different last end-point from the sinusoid, and thus a possibly smaller discontinuity.
If you want to decrease this FFT phase measurement error, create your test signal so the your test phase is referenced to the exact center (sample N/2) of the test vector (not the 1st sample), and then do an fftshift operation (rotate by N/2) so that there will be no signal discontinuity between the 1st and last point in your resulting FFT input vector of length N.
This snippet of code might help:
def reconstruct_ifft(data):
"""
In this function, we take in a signal, find its fft, retain the dominant modes and reconstruct the signal from that
Parameters
----------
data : Signal to do the fft, ifft
Returns
-------
reconstructed_signal : the reconstructed signal
"""
N = data.size
yf = rfft(data)
amp_yf = np.abs(yf) #amplitude
yf = yf*(amp_yf>(THRESHOLD*np.amax(amp_yf)))
reconstructed_signal = irfft(yf)
return reconstructed_signal
The 0.01 is the threshold of amplitudes of the fft that you would want to retain. Making the THRESHOLD greater(more than 1 does not make any sense), will give
fewer modes and cause higher rms error but ensures higher frequency selectivity.
(Please adjust the TABS for the python code)
I have a WAV file which I would like to visualize in the frequency domain. Next, I would like to write a simple script that takes in a WAV file and outputs whether the energy at a certain frequency "F" exceeds a threshold "Z" (whether a certain tone has a strong presence in the WAV file). There are a bunch of code snippets online that show how to plot an FFT spectrum in Python, but I don't understand a lot of the steps.
I know that wavfile.read(myfile) returns the sampling rate (fs) and the data array (data), but when I run an FFT on it (y = numpy.fft.fft(data)), what units is y in?
To get the array of frequencies for the x-axis, some posters do this where n = len(data):
X = numpy.linspace(0.0, 1.0/(2.0*T), n/2)
and others do this:
X = numpy.fft.fftfreq(n) * fs)[range(n/2)]
Is there a difference between these two methods and is there a good online explanation for what these operations do conceptually?
Some of the online tutorials about FFTs mention windowing, but not a lot of posters use windowing in their code snippets. I see that numpy has a numpy.hamming(N), but what should I use as the input to that method and how do I "apply" the output window to my FFT arrays?
For my threshold computation, is it correct to find the frequency in X that's closest to my desired tone/frequency and check if the corresponding element (same index) in Y has an amplitude greater than the threshold?
FFT data is in units of normalized frequency where the first point is 0 Hz and one past the last point is fs Hz. You can create the frequency axis yourself with linspace(0.0, (1.0 - 1.0/n)*fs, n). You can also use fftfreq but the components will be negative.
These are the same if n is even. You can also use rfftfreq I think. Note that this is only the "positive half" of your frequencies, which is probably what you want for audio (which is real-valued). Note that you can use rfft to just produce the positive half of the spectrum, and then get the frequencies with rfftfreq(n,1.0/fs).
Windowing will decrease sidelobe levels, at the cost of widening the mainlobe of any frequencies that are there. N is the length of your signal and you multiply your signal by the window. However, if you are looking in a long signal you might want to "chop" it up into pieces, window them, and then add the absolute values of their spectra.
"is it correct" is hard to answer. The simple approach is as you said, find the bin closest to your frequency and check its amplitude.