Develop Reference

How to find period of signal (autocorrelation vs fast fourier transform vs power spectral density)?

Suppose one wanted to find the period of a given sinusoidal wave signal. From what I have read online, it appears that the two main approaches employ either fourier analysis or autocorrelation. I am trying to automate the process using python and my usage case is to apply this concept to similar signals that come from the time-series of positions (or speeds or accelerations) of simulated bodies orbiting a star.
For simple-examples-sake, consider x = sin(t) for 0 ≤ t ≤ 10 pi.
import numpy as np
from scipy import signal
import matplotlib.pyplot as plt
## sample data
t = np.linspace(0, 10 * np.pi, 100)
x = np.sin(t)
fig, ax = plt.subplots()
ax.plot(t, x, color='b', marker='o')
ax.grid(color='k', alpha=0.3, linestyle=':')
plt.show()
plt.close(fig)
Given a sine-wave of the form x = a sin(b(t+c)) + d, the period of the sine-wave is obtained as 2 * pi / b. Since b=1 (or by visual inspection), the period of our sine wave is 2 * pi. I can check the results obtained from other methods against this baseline.
Attempt 1: Autocorrelation
As I understand it (please correct me if I'm wrong), correlation can be used to see if one signal is a time-lagged copy of another signal (similar to how cosine and sine differ by a phase difference). So autocorrelation is testing a signal against itself to measure the times at which the time-lag repeats said signal. Using the example posted here:
result = np.correlate(x, x, mode='full')
Since x and t each consist of 100 elements and result consists of 199 elements, I am not sure why I should arbitrarily select the last 100 elements.
print("\n autocorrelation (shape={}):\n{}\n".format(result.shape, result))
autocorrelation (shape=(199,)):
[ 0.00000000e+00 -3.82130761e-16 -9.73648712e-02 -3.70014208e-01
-8.59889695e-01 -1.56185995e+00 -2.41986054e+00 -3.33109112e+00
-4.15799070e+00 -4.74662427e+00 -4.94918053e+00 -4.64762251e+00
-3.77524157e+00 -2.33298717e+00 -3.97976240e-01 1.87752669e+00
4.27722402e+00 6.54129270e+00 8.39434617e+00 9.57785701e+00
9.88331103e+00 9.18204933e+00 7.44791758e+00 4.76948221e+00
1.34963425e+00 -2.50822289e+00 -6.42666652e+00 -9.99116299e+00
-1.27937834e+01 -1.44791297e+01 -1.47873668e+01 -1.35893098e+01
-1.09091510e+01 -6.93157447e+00 -1.99159756e+00 3.45267493e+00
8.86228186e+00 1.36707567e+01 1.73433176e+01 1.94357232e+01
1.96463736e+01 1.78556800e+01 1.41478477e+01 8.81191526e+00
2.32100171e+00 -4.70897483e+00 -1.15775811e+01 -1.75696560e+01
-2.20296487e+01 -2.44327920e+01 -2.44454330e+01 -2.19677060e+01
-1.71533510e+01 -1.04037163e+01 -2.33560966e+00 6.27458308e+00
1.45655029e+01 2.16769872e+01 2.68391837e+01 2.94553896e+01
2.91697473e+01 2.59122266e+01 1.99154591e+01 1.17007613e+01
2.03381596e+00 -8.14633251e+00 -1.78184255e+01 -2.59814393e+01
-3.17580589e+01 -3.44884934e+01 -3.38046447e+01 -2.96763956e+01
-2.24244433e+01 -1.26974172e+01 -1.41464998e+00 1.03204331e+01
2.13281784e+01 3.04712823e+01 3.67721634e+01 3.95170295e+01
3.83356037e+01 3.32477037e+01 2.46710643e+01 1.33886439e+01
4.77778141e-01 -1.27924775e+01 -2.50860560e+01 -3.51343866e+01
-4.18671622e+01 -4.45258983e+01 -4.27482779e+01 -3.66140001e+01
-2.66465884e+01 -1.37700036e+01 7.76494745e-01 1.55574483e+01
2.90828312e+01 3.99582426e+01 4.70285203e+01 4.95000000e+01
4.70285203e+01 3.99582426e+01 2.90828312e+01 1.55574483e+01
7.76494745e-01 -1.37700036e+01 -2.66465884e+01 -3.66140001e+01
-4.27482779e+01 -4.45258983e+01 -4.18671622e+01 -3.51343866e+01
-2.50860560e+01 -1.27924775e+01 4.77778141e-01 1.33886439e+01
2.46710643e+01 3.32477037e+01 3.83356037e+01 3.95170295e+01
3.67721634e+01 3.04712823e+01 2.13281784e+01 1.03204331e+01
-1.41464998e+00 -1.26974172e+01 -2.24244433e+01 -2.96763956e+01
-3.38046447e+01 -3.44884934e+01 -3.17580589e+01 -2.59814393e+01
-1.78184255e+01 -8.14633251e+00 2.03381596e+00 1.17007613e+01
1.99154591e+01 2.59122266e+01 2.91697473e+01 2.94553896e+01
2.68391837e+01 2.16769872e+01 1.45655029e+01 6.27458308e+00
-2.33560966e+00 -1.04037163e+01 -1.71533510e+01 -2.19677060e+01
-2.44454330e+01 -2.44327920e+01 -2.20296487e+01 -1.75696560e+01
-1.15775811e+01 -4.70897483e+00 2.32100171e+00 8.81191526e+00
1.41478477e+01 1.78556800e+01 1.96463736e+01 1.94357232e+01
1.73433176e+01 1.36707567e+01 8.86228186e+00 3.45267493e+00
-1.99159756e+00 -6.93157447e+00 -1.09091510e+01 -1.35893098e+01
-1.47873668e+01 -1.44791297e+01 -1.27937834e+01 -9.99116299e+00
-6.42666652e+00 -2.50822289e+00 1.34963425e+00 4.76948221e+00
7.44791758e+00 9.18204933e+00 9.88331103e+00 9.57785701e+00
8.39434617e+00 6.54129270e+00 4.27722402e+00 1.87752669e+00
-3.97976240e-01 -2.33298717e+00 -3.77524157e+00 -4.64762251e+00
-4.94918053e+00 -4.74662427e+00 -4.15799070e+00 -3.33109112e+00
-2.41986054e+00 -1.56185995e+00 -8.59889695e-01 -3.70014208e-01
-9.73648712e-02 -3.82130761e-16 0.00000000e+00]
Attempt 2: Fourier
Since I am not sure where to go from the last attempt, I sought a new attempt. To my understanding, Fourier analysis basically shifts a signal from/to the time-domain (x(t) vs t) to/from the frequency domain (x(t) vs f=1/t); the signal in frequency-space should appear as a sinusoidal wave that dampens over time. The period is obtained from the most observed frequency since this is the location of the peak of the distribution of frequencies.
Since my values are all real-valued, applying the Fourier transform should mean my output values are all complex-valued. I wouldn't think this is a problem, except for the fact that scipy has methods for real-values. I do not fully understand the differences between all of the different scipy methods. That makes following the algorithm proposed in this posted solution hard for me to follow (ie, how/why is the threshold value picked?).
omega = np.fft.fft(x)
freq = np.fft.fftfreq(x.size, 1)
threshold = 0
idx = np.where(abs(omega)>threshold)[0][-1]
max_f = abs(freq[idx])
print(max_f)
This outputs 0.01, meaning the period is 1/0.01 = 100. This doesn't make sense either.
Attempt 3: Power Spectral Density
According to the scipy docs, I should be able to estimate the power spectral density (psd) of the signal using a periodogram (which, according to wikipedia, is the fourier transform of the autocorrelation function). By selecting the dominant frequency fmax at which the signal peaks, the period of the signal can be obtained as 1 / fmax.
freq, pdensity = signal.periodogram(x)
fig, ax = plt.subplots()
ax.plot(freq, pdensity, color='r')
ax.grid(color='k', alpha=0.3, linestyle=':')
plt.show()
plt.close(fig)
The periodogram shown below peaks at 49.076... at a frequency of fmax = 0.05. So, period = 1/fmax = 20. This doesn't make sense to me. I have a feeling it has something to do with the sampling rate, but don't know enough to confirm or progress further.
I realize I am missing some fundamental gaps in understanding how these things work. There are a lot of resources online, but it's hard to find this needle in the haystack. Can someone help me learn more about this?

Let's first look at your signal (I've added endpoint=False to make the division even):
t = np.linspace(0, 10*np.pi, 100, endpoint=False)
x = np.sin(t)
Let's divide out the radians (essentially by taking t /= 2*np.pi) and create the same signal by relating to frequencies:
fs = 20 # Sampling rate of 100/5 = 20 (e.g. Hz)
f = 1 # Signal frequency of 1 (e.g. Hz)
t = np.linspace(0, 5, 5*fs, endpoint=False)
x = np.sin(2*np.pi*f*t)
This makes it more salient that f/fs == 1/20 == 0.05 (i.e. the periodicity of the signal is exactly 20 samples). Frequencies in a digital signal always relate to its sampling rate, as you have already guessed. Note that the actual signal is exactly the same no matter what the values of f and fs are, as long as their ratio is the same:
fs = 1 # Natural units
f = 0.05
t = np.linspace(0, 100, 100*fs, endpoint=False)
x = np.sin(2*np.pi*f*t)
In the following I'll use these natural units (fs = 1). The only difference will be in t and hence the generated frequency axes.
Autocorrelation
Your understanding of what the autocorrelation function does is correct. It detects the correlation of a signal with a time-lagged version of itself. It does this by sliding the signal over itself as seen in the right column here (from Wikipedia):
Note that as both inputs to the correlation function are the same, the resulting signal is necessarily symmetric. That is why the output of np.correlate is usually sliced from the middle:
acf = np.correlate(x, x, 'full')[-len(x):]
Now index 0 corresponds to 0 delay between the two copies of the signal.
Next you'll want to find the index or delay that presents the largest correlation. Due to the shrinking overlap this will by default also be index 0, so the following won't work:
acf.argmax() # Always returns 0
Instead I recommend to find the largest peak instead, where a peak is defined to be any index with a larger value than both its direct neighbours:
inflection = np.diff(np.sign(np.diff(acf))) # Find the second-order differences
peaks = (inflection < 0).nonzero()[0] + 1 # Find where they are negative
delay = peaks[acf[peaks].argmax()] # Of those, find the index with the maximum value
Now delay == 20, which tells you that the signal has a frequency of 1/20 of its sampling rate:
signal_freq = fs/delay # Gives 0.05
Fourier transform
You used the following to calculate the FFT:
omega = np.fft.fft(x)
freq = np.fft.fftfreq(x.size, 1)
Thhese functions re designed for complex-valued signals. They will work for real-valued signals, but you'll get a symmetric output as the negative frequency components will be identical to the positive frequency components. NumPy provides separate functions for real-valued signals:
ft = np.fft.rfft(x)
freqs = np.fft.rfftfreq(len(x), t[1]-t[0]) # Get frequency axis from the time axis
mags = abs(ft) # We don't care about the phase information here
Let's have a look:
plt.plot(freqs, mags)
plt.show()
Note two things: the peak is at frequency 0.05, and the maximum frequency on the axis is 0.5 (the Nyquist frequency, which is exactly half the sampling rate). If we had picked fs = 20, this would be 10.
Now let's find the maximum. The thresholding method you have tried can work, but the target frequency bin is selected blindly and so this method would suffer in the presence of other signals. We could just select the maximum value:
signal_freq = freqs[mags.argmax()] # Gives 0.05
However, this would fail if, e.g., we have a large DC offset (and hence a large component in index 0). In that case we could just select the highest peak again, to make it more robust:
inflection = np.diff(np.sign(np.diff(mags)))
peaks = (inflection < 0).nonzero()[0] + 1
peak = peaks[mags[peaks].argmax()]
signal_freq = freqs[peak] # Gives 0.05
If we had picked fs = 20, this would have given signal_freq == 1.0 due to the different time axis from which the frequency axis was generated.
Periodogram
The method here is essentially the same. The autocorrelation function of x has the same time axis and period as x, so we can use the FFT as above to find the signal frequency:
pdg = np.fft.rfft(acf)
freqs = np.fft.rfftfreq(len(x), t[1]-t[0])
plt.plot(freqs, abs(pdg))
plt.show()
This curve obviously has slightly different characteristics from the direct FFT on x, but the main takeaways are the same: the frequency axis ranges from 0 to 0.5*fs, and we find a peak at the same signal frequency as before: freqs[abs(pdg).argmax()] == 0.05.
Edit:
To measure the actual periodicity of np.sin, we can just use the "angle axis" that we passed to np.sin instead of the time axis when generating the frequency axis:
freqs = np.fft.rfftfreq(len(x), 2*np.pi*f*(t[1]-t[0]))
rad_period = 1/freqs[mags.argmax()] # 6.283185307179586
Though that seems pointless, right? We pass in 2*np.pi and we get 2*np.pi. However, we can do the same with any regular time axis, without presupposing pi at any point:
fs = 10
t = np.arange(1000)/fs
x = np.sin(t)
rad_period = 1/np.fft.rfftfreq(len(x), 1/fs)[abs(np.fft.rfft(x)).argmax()] # 6.25
Naturally, the true value now lies in between two bins. That's where interpolation comes in and the associated need to choose a suitable window function.

Detecting Alpha waves in EEG data through SNR? python

I am writing a function that detect Alpha waves in real time. My function receives 256 sample values of single channel as an argument.
After that its fft has to be found and then classified in alpha , beta and gamma ranges. Then I have to find SNR to check whether at alpha waves are present or not i.e is there any peak that exists at 10 hz frequency. So i need to find square of amplitude of values at 10hz divided by sum of square of all values in b/w range of 8-12hz divided by N values.
SNR = Square of Amp Value at 10hz/ (Square of rest values in 8-12hz / No. of these values)
Then 20 log SNR and check threshold.
So basically how to get square of Amp of values that are at 10 hz and then exclude this value and divided by rest of values.
I have written kick starter code below can someone guide or help to complete the code to get the desired job done.
Many thanks.
def classification(flag, data=[]):
fs = 200 # Sampling rate (512 Hz)
# Get real amplitudes of FFT (only in postive frequencies)
fft_vals = np.absolute(np.fft.rfft(data)) #these are my fft values rfft returns only the part of the result that corresponds to nonpositive frequences. (Avoids complex conjugaes) faster and for plotting
# Get frequencies for amplitudes in Hz
fft_freq = np.fft.rfftfreq(len(data), 1.0 / fs) # that might be fixed (window length n , and sample spacing) inverse of the sampling rate returns sample freq of length n .
# Define EEG bands
eeg_bands = {'Delta': (0, 4),
'Theta': (4, 8),
'Alpha': (8, 12),
'Beta': (12, 30),
'Gamma': (30, 45)}
# Take the mean of the fft amplitude for each EEG band
eeg_band_fft = dict()
for band in eeg_bands:
freq_ix = np.where((fft_freq >= eeg_bands[band][0]) & #np.where is like asking "tell me where in this array, entries satisfy a given condition".
(fft_freq <= eeg_bands[band][1]))[0] #for fft_frreq at all point where it satisfies it returns the index (in array)
#if fftfreq[np.where bla bla] will give values array
eeg_band_fft[band] = np.mean(fft_vals[freq_ix])

This code is already extracting the alpha frequencies using band-pass filter. Now to find SNR at specific frequency you simply sum square the values at that frequency divided by rest of frequncies and take 20 log of the division.

How to get frequency axis from an fft function?

So, I am probably missing something obvious, but I have searched through lots of tutorials and documentation and can't seem to find a straight answer. How do you find the frequency axis of a function that you performed an fft on in Python(specifically the fft in the scipy library)?
I am trying to get a raw EMG signal, perform a bandpass filter on it, and then perform an fft to see the remaining frequency components. However, I am not sure how to find an accurate x component list. The specific signal I am working on currently was sampled at 1000 Hz and has 5378 samples.
Is it just creating a linear x starting from 0 and going to the length of the fft'd data? I see a lot of people creating a linspace from 0 to sample points times the sample spacing. But what would be my sample spacing in this case? Would it just be samples/sampling rate? Or is it something else completely?

Here is an example.
First create a sine wave with sampling interval pre-determined. we will combine two sine waves with frequencies 20 and 40. Remember high frequencies might be aliased if the time interval is large.
#Import the necessary packages
from scipy import fftpack
import matplotlib.pyplot as plt
import numpy as np
# sampling freq in herts 20Hz, and 40Hz
freq_sampling1 = 10
freq_sampling2 = 20
amplitude1 = 2 # amplitude of first sine wave
amplitude2 = 4 # amplitude of second sine wave
time = np.linspace(0, 6, 500, endpoint=True) # time range with total samples of 500 from 0 to 6 with time interval equals 6/500
y = amplitude1*np.sin(2*np.pi*freq_sampling1*time) + amplitude2*np.sin(2*np.pi*freq_sampling2*time)
plt.figure(figsize=(10, 4))
plt.plot(time,y, 'k', lw=0.8)
plt.xlim(0,6)
plt.show()
Notice in the figure that two sine waves are superimposed. One with freq. 10 and amplitude 2 and the other with freq. 20 and amplitude 4.
# apply fft function
yf = fftpack.fft(y, time.size)
amp = np.abs(yf) # get amplitude spectrum
freq = np.linspace(0.0, 1.0/(2.0*(6/500)), time.size//2) # get freq axis
# plot the amp spectrum
plt.figure(figsize=(10,6))
plt.plot(freq, (2/amp.size)*amp[0:amp.size//2])
plt.show()
Notice in the amplitude spectrum the two frequencies are recovered while amplitude is zero at other frequencies. the Amplitude values are also 2 and 4 respectively.
you can use instead fftpack.fftfreq to obtain frequency axis as suggested by tom10
Therefore, the code changes to
yf = fftpack.fft(y, time.size)
amp = np.abs(yf) # get amplitude spectrum
freq = fftpack.fftfreq(time.size, 6/500)
plt.figure(figsize=(10,6))
plt.plot(freq[0:freq.size//2], (2/amp.size)*amp[0:amp.size//2])
plt.show()
We are only plotting the positive part of the amplitude spectrum [0:amp.size//2]

Once you feed your window of samples into the FFT call it will return an array of imaginary points ... the freqency separation between each element of returned array is determined by
freq_resolution = sampling_freq / number_of_samples
the 0th element is your DC offset which will be zero if your input curve is balanced straddling the zero crossing point ... so in your case
freq_resolution = 1000 / 5378
In general, for efficiency, you will want to feed an even power of 2 number of samples into your FFT call, important if you are say sliding your window of samples forward in time and repeatedly calling FFT on each window
To calculate the magnitude of a frequency in a given freq_bin (an element of the returned imaginary array)
X = A + jB
A on real axis
B on imag axis
for above formula its
mag = 2.0 * math.Sqrt(A*A+B*B) / number_of_samples
phase == arctan( B / A )
you iterate across each element up to the Nyquist limit which is why you double above magnitude
So yes its a linear increment with same frequency spacing between each freq_bin

Getting the width and area of peaks from Scipy Signal object

How do I get the peak objects with the properties such as position, peak aarea, peak width etc from Scipy Signal function using cwt get peaks method:
def CWT(trace):
x = []
y = []
for i in range(len(trace)):
x.append(trace[i].Position)
y.append(trace[i].Intensity)
x = np.asarray(x)
y = np.asarray(y)
return signal.find_peaks_cwt(x,y)
This just returns an array?

First, it looks like you are using find_peaks_cwt incorrectly. Its two positional parameters are not x and y coordinates of data points. The first parameter is y-values. The x-values are not taken at all, they are assumed to be 0,1,2,.... The second parameter is a list of peak widths that you are interested in;
1-D array of widths to use for calculating the CWT matrix. In general, this range should cover the expected width of peaks of interest.
There is no reason for width parameter to be of the same size as the data array. In my example below, the data has 500 values, but the widths I use are 30...99.
Second, this method only finds the position of peaks (the array you get has the indexes of peaks). There is no analysis of their widths and areas. You will either have to look elsewhere (blog post Peak Detection in the Python World lists some alternatives, though none of them return the data you want), or come up with your own method of estimating those things.
My attempt is below. It does the following:
Cuts the signal by midpoints between peaks
For each piece, uses the median of values in it as the baseline
Declares the peak to consist of all values that are greater than 0.5*(peak value + baseline), i.e., midway between median and maximum.
Finds where the peak begins and where it ends. (The width is just the difference of these)
Declares the area of the peak to be the sum of (y - baseline) over the interval found in step 4.
Complete example:
t = np.linspace(0, 4.2, 500)
y = np.sin(t**2) + np.random.normal(0, 0.03, size=t.shape) # simulated noisy signal
peaks = find_peaks_cwt(y, np.arange(30, 100, 10))
cuts = (peaks[1:] + peaks[:-1])//2 # where to cut the signal
cuts = np.insert(cuts, [0, cuts.size], [0, t.size])
peak_begins = np.zeros_like(peaks)
peak_ends = np.zeros_like(peaks)
areas = np.zeros(peaks.shape)
for i in range(peaks.size):
peak_value = y[peaks[i]]
y_cut = y[cuts[i]:cuts[i+1]] # piece of signal with 1 peak
baseline = np.median(y_cut)
large = np.where(y_cut > 0.5*(peak_value + baseline))[0]
peak_begins[i] = large.min() + cuts[i]
peak_ends[i] = large.max() + cuts[i]
areas[i] = np.sum(y[peak_begins[i]:peak_ends[i]] - baseline)
The arrays areas, peak_begins and peak_ends are of interest here. The widths are [84 47 36], indicating the peaks get thinner (recall these are in index units, the width is the number of data points in the peak). I use this data to color the peaks in red:
widths = peak_ends - peak_begins
print(widths, areas)
plt.plot(t, y)
for i in range(peaks.size):
plt.plot(t[peak_begins[i]:peak_ends[i]], y[peak_begins[i]:peak_ends[i]], 'r')
plt.show()

Drawing FFT discrete signal using continuous sine-waves ?!

I do np.fft.fft() to decompose discrete-signal to frequencies.
Then I pick top N freqs and now I want to draw a signal of the sum of those frequencies using the following formula :
amp * np.sin( 2 * np.pi * freq * time + phase )
I extract the freqs based on their position in the array (fftfreqs), amplitudes with np.abs(complex_num) and phases with np.angle(complex_num, deg=True), from the data I got from from fft() call.
It does not seem to work very well.
What I'm doing wrong ?
Do I understand correctly that the complex-num contains the amp&phase of sinusoid ? Do I have to use sin&cos for the drawing ?
The documentation is very sparse on exact interpretation of the result of fft().
PS> I know I can use np.fft.ifft() to draw close approximation, but this way of doing it is limited to the time-frame of the original signal.
I want to draw/fit the original signal with the top N freqs in the original timeframe, but also to draw the "continuation" of the signal after this timeframe using those freqs i.e. the extended signal.

I found my mistakes.. first the correct way to calculate the waves is (sum of those for topN freqs) :
cos_amp * np.cos( 2 * np.pi * freq * time) + sin_amp * np.sin( 2 * np.pi * freq * time)
not just sin()..second it is incorrect to use abs-amplitude and angle-phase extraction from the complex array.
The real part contains the cos-amp and the negative-imaginary part contains the sin-amplitude i.e. once I get the complex array out of fft(), I do something like this :
fft_res = np.fft.fft(signal)
.... topN_freqs_idx = find top freqs from PSD
all_freqs = np.fft.fftfreq(N)
freqs = all_freqs[topN_freq_idx]
norm = fft_res[topN_freq_idx] / (N/2)
cos_amp = np.real(norm)
sin_amp = -np.imag(norm)
I figured out this from here :
http://www.dspguide.com/ch8/5.htm