Problem
I am trying to remove a frequency from a set of data obtained from an audio file.
To simplify down my problem, I have created the code below which creates a set of waves and merges them into a complex wave. Then it finds the fourier transform of this complex wave and inverses it.
I am expecting to see the original wave as a result since there should be no data loss, however I receive a very different wave instead.
Code:
import numpy as np
import matplotlib.pyplot as plt
import random
#Get plots
fig, c1 = plt.subplots()
c2 = c1.twinx()
fs = 100 # sample rate
f_list = [5,10,15,20,100] # the frequency of the signal
x = np.arange(fs) # the points on the x axis for plotting
# compute the value (amplitude) of the sin wave for each sample
wave = []
for f in f_list:
wave.append(list(np.sin(2*np.pi*f * (x/fs))))
#Adds the sine waves together into a single complex wave
wave4 = []
for i in range(len(wave[0])):
data = 0
for ii in range(len(wave)):
data += wave[ii][i]
wave4.append(data)
#Get frequencies from complex wave
fft = np.fft.rfft(wave4)
fft = np.abs(fft)
#Note: Here I will add some code to remove specific frequencies
#Get complex wave from frequencies
waveV2 = np.fft.irfft(fft)
#Plot the complex waves, should be the same
c1.plot(wave4, color="orange")
c1.plot(waveV2)
plt.show()
Results: (Orange is created wave, blue is original wave)
Expected results:
The blue and orange lines (the original and new wave created) should have the exact same values
You took the absolute value of the FFT before you do the inverse FFT. That changes things, and is probably the cause of your problem.
Related
This is my first post so apologies for any formatting related issues.
So I have a dataset which was obtained from an atomic microscope. The data looks like a 1024x1024 matrix which is composed of different measurements taken from the sample in units of meters, eg.
data = [[1e-07 ... 4e-08][ ... ... ... ][3e-09 ... 12e-06]]
np.size(data) == (1024,1024)
From this data, I was hoping to 1) derive some statistics about the real data; and 2) using the power spectrum density (PSD) distribution, hopefully create a new dataset which is different, but statistically similar to the characteristics of the original data. My plan to do this was 2a) take a 2d fft of data, calculate the power spectrum density 2b) some method?, 2c) take the 2d ifft of the modified signal to turn it back into a new sample with the same power spectrum density as the original.
Moreover, regarding part 2b) this was the closest link I could find regarding a time series based solution; however, I am not understanding exactly how to implement this so far, since I am not exactly sure what the phase, frequency, and amplitudes of the fft data represent in this 2d case, and also since we are now talking about a 2d ifft I'm not exactly sure how to construct this complex matrix while incorporating the random number generation, and amplitude/phase shifts in a way that will translate back to something meaningful.
So basically, I have been having some trouble with my intuition. For this problem, we are working with a 2d Fourier transform of spatial data with no temporal component, so I believe that methods which are applied to time series data could be applied here as well. Since the fft of the original data is the 'frequency in the spatial domain', the x-axis of the PSD should be either pixels or meters, but then what is the 'power' in the y-axis describing? I was hoping that someone could help me figure this problem out.
My code is below, hopefully someone could help me solve my problem. Bonus if someone could help me understand what this shifted frequency vs amplitude plot is saying:
here is the image with the fft, shifted fft, and freq. vs aplitude plots.
Fortunately the power spectrum density function is a bit easier to understand
Thank you all for your time.
data = np.genfromtxt('asample3.0_00001-filter.txt')
x = np.arange(0,int(np.size(data,0)),1)
y = np.arange(0,int(np.size(data,1)),1)
z = data
npix = data.shape[0]
#taking the fourier transform
fourier_image = np.fft.fft2(data)
#Get power spectral density
fourier_amplitudes = np.abs(fourier_image)**2
#calculate sampling frequency fs (physical distance between pixels)
fs = 92e-07/npix
freq_shifted = fs/2 * np.linspace(-1,1,npix)
freq = fs/2 * np.linspace(0,1,int(npix/2))
print("Plotting 2d Fourier Transform ...")
fig, axs = plt.subplots(2,2,figsize=(15, 15))
axs[0,0].imshow(10*np.log10(np.abs(fourier_image)))
axs[0,0].set_title('fft')
axs[0,1].imshow(10*np.log10(np.abs(np.fft.fftshift(fourier_image))))
axs[0,1].set_title('shifted fft')
axs[1,0].plot(freq,10*np.log10(np.abs(fourier_amplitudes[:npix//2])))
axs[1,0].set_title('freq vs amplitude')
for ii in list(range(npix//2)):
axs[1,1].plot(freq_shifted,10*np.log10(np.fft.fftshift(np.abs(fourier_amplitudes[ii]))))
axs[1,1].set_title('shifted freq vs amplitude')
#constructing a wave vector array
## Get frequencies corresponding to signal PSD
kfreq = np.fft.fftfreq(npix) * npix
kfreq2D = np.meshgrid(kfreq, kfreq)
knrm = np.sqrt(kfreq2D[0]**2 + kfreq2D[1]**2)
knrm = knrm.flatten()
fourier_amplitudes = fourier_amplitudes.flatten()
#creating the power spectrum
kbins = np.arange(0.5, npix//2+1, 1.)
kvals = 0.5 * (kbins[1:] + kbins[:-1])
Abins, _, _ = stats.binned_statistic(knrm, fourier_amplitudes,
statistic = "mean",
bins = kbins)
Abins *= np.pi * (kbins[1:]**2 - kbins[:-1]**2)
print("Plotting power spectrum of surface ...")
fig = plt.figure(figsize=(10, 10))
plt.loglog(fs/kvals, Abins)
plt.xlabel("Spatial Frequency $k$ [meters]")
plt.ylabel("Power per Spatial Frequency $P(k)$")
plt.tight_layout()
I am implementing the method from this paper:
https://dspace.mit.edu/bitstream/handle/1721.1/66243/Picard_Noncontact%20Automated.pdf?sequence=1&isAllowed=y
The main idea is cardiac pulse measurement using a set of frames (N=300) from a 10s video so the frame rate equals 30 fps.
red = [item[:,:,0] for item in imgs]
green = [item[:,:,1] for item in imgs]
blue = [item[:,:,2] for item in imgs]
red_avg = [item.mean() for item in red]
green_avg = [item.mean() for item in green]
blue_avg = [item.mean() for item in blue]
red_mean, red_std = np.array(red_avg).mean(), np.array(red_avg).std()
green_mean, green_std = np.array(green_avg).mean(), np.array(green_avg).std()
blue_mean, blue_std = np.array(blue_avg).mean(), np.array(blue_avg).std()
red_avg = [(item - red_mean)/red_std for item in red_avg]
green_avg = [(item - green_mean)/green_std for item in green_avg]
blue_avg = [(item - blue_mean)/blue_std for item in blue_avg]
data = np.vstack([signal.detrend(red_avg), signal.detrend(green_avg), signal.detrend(blue_avg)]).reshape(300,3)
from sklearn.decomposition import FastICA
transformer = FastICA(n_components=3)
X_transformed = transformer.fit_transform(data)
from scipy.fftpack import fft
first = X_transformed.T[0]
second = X_transformed.T[1]
third = X_transformed.T[2]
ff = np.fft.fft(first)
fs = np.fft.fft(second)
ft = np.fft.fft(third)
imgs - is the initial list of arrays with 300 image pixel values.
As you can see, I split all frames into RGB channels and thus have traces x_i(t), where i = 1,2,3
After standardization, we detrend all the traces and stack them to further apply ICA and then FFT all the three components.
The method then claims that we need to plot power vs frequency (Hz) and select the component that is most likely to be heart pulse.
Finally, we applied the fast Fourier transform (FFT) on the selected source signal to
obtain the power spectrum. The pulse frequency was designated as the frequency that
corresponded to the highest power of the spectrum within an operational frequency band. For
our experiments, we set the operational range to [0.75, 4] Hz (corresponding to [45, 240]
bpm) to provide a wide range of heart rate measurements.
Here's how I try to visualize the frequencies:
import numpy as np
import matplotlib.pyplot as plt
from scipy import signal
data = ft
print(fs.size)
ps = np.abs(np.fft.fft(data))**2
sampling_rate = 30
freqs = np.fft.fftfreq(data.size, 1/sampling_rate)
idx = np.argsort(freqs)
#print(idx)
plt.plot(freqs[idx], ps[idx])
What I get is totally different since the range of frequencies is from $-15$ to $15$ and I have no idea whether this is in Hz or not.
The three images above are what I get when I execute the code to visualize frequencies and signal power.
I would appreciate any help or suggestions.
You should really learn how to work with images/videos as nD-tensors. Doing that you can replace all your data wrangling with much more concise code:
import numpy as np
import matplotlib.pyplot as plt
from scipy.fftpack import fft
from sklearn.decomposition import FastICA
images = [np.random.rand(640, 480, 3) for _ in range(30)]
# Create tensor with all images
images = np.array(images)
images.shape
# Take average of all pixels, for each image and each channel individually
avgs = np.mean(images, axis=(1, 2))
mean, std = np.mean(avgs), np.std(avgs)
# Normalize all average channels
avgs = (avgs - mean) / std
# Detrend across images
avgs = scipy.signal.detrend(avgs, axis=0)
transformer = FastICA(n_components=3)
X_transformed = transformer.fit_transform(avgs)
X_ff = np.fft.fft(X_transformed, axis=0)
plt.plot(np.abs(X_ff) ** 2)
To answer your question a little bit: I think you are mistakenly taking the fourier spectrum of the fourier spectrum of the third PCA component.
FFT(FFT(PCA[:, 2]))
while you meant to take one FFT only:
FFT(PCA[:, 2]))
Regarding the -15...15 axis: You have set your sampling frequency as 30Hz (or 30fps in video-terms). That means you can detect anything up to 15Hz in your video.
In fourier theory, there exists a thing as called "negative frequency". Now since we're mostly analyzing real signals (as opposed to complex signals), that negative frequency is always the same as the positive frequency. This means your spectrum is always symmetric and you can disregard the left half.
However since you've taken the FFT twice, you're looking at the FFT of a complex signal, which does have negative frequencies. That's the reason why your spectra are asymmetric and confusing.
Additionally, I believe you're confusing reshaping and transposing. Before PCA, you're assembling your data like
np.vstack([red, green, blue]) # shape = (3, 300)
which you want to transpose to get (300, 3). If you reshape instead, you're not swapping rows and columns but are interpreting the same data in a different shape.
So, I am probably missing something obvious, but I have searched through lots of tutorials and documentation and can't seem to find a straight answer. How do you find the frequency axis of a function that you performed an fft on in Python(specifically the fft in the scipy library)?
I am trying to get a raw EMG signal, perform a bandpass filter on it, and then perform an fft to see the remaining frequency components. However, I am not sure how to find an accurate x component list. The specific signal I am working on currently was sampled at 1000 Hz and has 5378 samples.
Is it just creating a linear x starting from 0 and going to the length of the fft'd data? I see a lot of people creating a linspace from 0 to sample points times the sample spacing. But what would be my sample spacing in this case? Would it just be samples/sampling rate? Or is it something else completely?
Here is an example.
First create a sine wave with sampling interval pre-determined. we will combine two sine waves with frequencies 20 and 40. Remember high frequencies might be aliased if the time interval is large.
#Import the necessary packages
from scipy import fftpack
import matplotlib.pyplot as plt
import numpy as np
# sampling freq in herts 20Hz, and 40Hz
freq_sampling1 = 10
freq_sampling2 = 20
amplitude1 = 2 # amplitude of first sine wave
amplitude2 = 4 # amplitude of second sine wave
time = np.linspace(0, 6, 500, endpoint=True) # time range with total samples of 500 from 0 to 6 with time interval equals 6/500
y = amplitude1*np.sin(2*np.pi*freq_sampling1*time) + amplitude2*np.sin(2*np.pi*freq_sampling2*time)
plt.figure(figsize=(10, 4))
plt.plot(time,y, 'k', lw=0.8)
plt.xlim(0,6)
plt.show()
Notice in the figure that two sine waves are superimposed. One with freq. 10 and amplitude 2 and the other with freq. 20 and amplitude 4.
# apply fft function
yf = fftpack.fft(y, time.size)
amp = np.abs(yf) # get amplitude spectrum
freq = np.linspace(0.0, 1.0/(2.0*(6/500)), time.size//2) # get freq axis
# plot the amp spectrum
plt.figure(figsize=(10,6))
plt.plot(freq, (2/amp.size)*amp[0:amp.size//2])
plt.show()
Notice in the amplitude spectrum the two frequencies are recovered while amplitude is zero at other frequencies. the Amplitude values are also 2 and 4 respectively.
you can use instead fftpack.fftfreq to obtain frequency axis as suggested by tom10
Therefore, the code changes to
yf = fftpack.fft(y, time.size)
amp = np.abs(yf) # get amplitude spectrum
freq = fftpack.fftfreq(time.size, 6/500)
plt.figure(figsize=(10,6))
plt.plot(freq[0:freq.size//2], (2/amp.size)*amp[0:amp.size//2])
plt.show()
We are only plotting the positive part of the amplitude spectrum [0:amp.size//2]
Once you feed your window of samples into the FFT call it will return an array of imaginary points ... the freqency separation between each element of returned array is determined by
freq_resolution = sampling_freq / number_of_samples
the 0th element is your DC offset which will be zero if your input curve is balanced straddling the zero crossing point ... so in your case
freq_resolution = 1000 / 5378
In general, for efficiency, you will want to feed an even power of 2 number of samples into your FFT call, important if you are say sliding your window of samples forward in time and repeatedly calling FFT on each window
To calculate the magnitude of a frequency in a given freq_bin (an element of the returned imaginary array)
X = A + jB
A on real axis
B on imag axis
for above formula its
mag = 2.0 * math.Sqrt(A*A+B*B) / number_of_samples
phase == arctan( B / A )
you iterate across each element up to the Nyquist limit which is why you double above magnitude
So yes its a linear increment with same frequency spacing between each freq_bin
The figure I plot via the code below is just a peak around ZERO, no matter how I change the data. My data is just one column which records every timing points of some kind of signal. Is the time_step a value I should define according to the interval of two neighbouring points in my data?
data=np.loadtxt("timesequence",delimiter=",",usecols=(0,),unpack=True)
ps = np.abs(np.fft.fft(data))**2
time_step = 1
freqs = np.fft.fftfreq(data.size, time_step)
idx = np.argsort(freqs)
pl.plot(freqs[idx], ps[idx])
pl.show()
As others have hinted at your signals must have a large nonzero component. A peak at 0 (DC) indicates the average value of your signal. This is derived from the Fourier transform itself. This cosine function cos(0)*ps(0) indicates a measure of the average value of the signal. Other Fourier transform components are cosine waves of varying amplitude which show frequency content at those values.
Note that stationary signals will not have a large DC component as they are already zero mean signals. If you do not want a large DC component then you should compute the mean of your signal and subtract values from that. Regardless of whether your data is 0,...,999 or 1,...,1000, or even 1000, ..., 2000 you will get a peak at 0Hz. The only difference will be the magnitude of the peak since it measures the average value.
data1 = arange(1000)
data2 = arange(1000)+1000
dataTransformed3 = data - mean(data)
data4 = numpy.zeros(1000)
data4[::10] = 1 #simulate a photon counter where a 1 indicates a photon came in at time indexed by array.
# we could assume that the sample rate was 10 Hz for example
ps1 = np.abs(np.fft.fft(data))**2
ps2 = np.abs(np.fft.fft(data))**2
ps3 = np.abs(np.fft.fft(dataTransformed))**2
figure()
plot(ps1) #shows the peak at 0 Hz
figure()
plot(ps2) #shows the peak at 0 Hz
figure()
plot(ps3) #shows the peak at 1 Hz this is because we removed the mean value but since
#the function is a step function the next largest component is the 1 Hz cosine wave.
#notice the order of magnitude difference in the two plots.
Here is a bare-bones example that shows input and output with a peak as you'd expect it:
import numpy as np
from scipy.fftpack import rfft, irfft, fftfreq
time = np.linspace(0,10,2000)
signal = np.cos(5*np.pi*time)
W = fftfreq(signal.size, d=time[1]-time[0])
f_signal = rfft(signal)
import pylab as plt
plt.subplot(121)
plt.plot(time,signal)
plt.subplot(122)
plt.plot(W,f_signal)
plt.xlim(0,10)
plt.show()
I use rfft since, more than likely, your input signal is from a physical data source and as such is real.
If you make your data all positive:
ps = np.abs(np.fft.fft(data))**2
time_step = 1
then most probably you will create a large 'DC', or 0 Hz component. So if your actual data has little amplitude, compared to that component, it will disappear from the plot, by the autoscaling feature.
I'm trying to smooth and interpolate some periodic data in python using scipy.fftp. I have managed to take the fft of the data, remove the higher order frequencies above wn (by doing myfft[wn:-wn] = 0) and then reconstruct a "smoothed" version of the data with ifft(myfft). The array created by the ifft has the same number of points as the original data. How can I use that fft to create an array with more points.
x = [i*2*np.pi/360 for i in range(0,360,30)]
data = np.sin(x)
#get fft
myfft = fftp.fft(data)
#kill feqs above wn
myfft[wn:-wn] = 0
#make new series
newdata = fftp.ifft(myfft)
I've also been able to manually recreate the series at the same resolution as demonstrated here
Recreating time series data using FFT results without using ifft
but when I tried upping the resolution of the x-values array it didn't give me the right answer either.
Thanks in advance
Niall
What np.fft.fft returns has the DC component at position 0, followed by all positive frequencies, then the Nyquist frequency (only if the number of elements is even), then the negative frequencies in reverse order. So to add more resolution you could add zeros at both sides of the Nyquist frequency:
import numpy as np
import matplotlib.pyplot as plt
y = np.sin(np.linspace(0, 2*np.pi, 32, endpoint=False))
f = np.fft.fft(y)
n = len(f)
f_ = np.concatenate((f[0:(n+1)//2],
np.zeros(n//2),
[] if n%2 != 0 else f[(n+1)//2:(n+3)//2],
np.zeros(n//2),
f[(n+3)//2:]))
y_ = np.fft.ifft(f_)
plt.plot(y, 'ro')
plt.plot(y_, 'bo')
plt.show()