I implemented an high pass filter in python using this code:
from scipy.signal import butter, filtfilt
import numpy as np
def butter_highpass(cutoff, fs, order=5):
nyq = 0.5 * fs
normal_cutoff = cutoff / nyq
b, a = butter(order, normal_cutoff, btype='high', analog=False)
return b, a
def butter_highpass_filter(data, cutoff, fs, order=5):
b, a = butter_highpass(cutoff, fs, order=order)
y = filtfilt(b, a, data)
return y
rawdata = np.loadtxt('sampleSignal.txt', skiprows=0)
signal = rawdata
fs = 100000.0
cutoff = 100
order = 6
conditioned_signal = butter_highpass_filter(signal, cutoff, fs, order)
I am applying this filter on a 100 kHz voltage signal and it works fine for cutoff frequencies >= 60 Hz. But it doesn't work below. I would like to cut off all the frequencies below 10 Hz. Any hints where my mistake is? What I observed is the lower the order of the filter the lower the cutoff frequency could be.
The sample Signal can be found here.
I hope this can help you:
import numpy as np
import pandas as pd
from scipy import signal
import matplotlib.pyplot as plt
def sine_generator(fs, sinefreq, duration):
T = duration
nsamples = fs * T
w = 2. * np.pi * sinefreq
t_sine = np.linspace(0, T, nsamples, endpoint=False)
y_sine = np.sin(w * t_sine)
result = pd.DataFrame({
'data' : y_sine} ,index=t_sine)
return result
def butter_highpass(cutoff, fs, order=5):
nyq = 0.5 * fs
normal_cutoff = cutoff / nyq
b, a = signal.butter(order, normal_cutoff, btype='high', analog=False)
return b, a
def butter_highpass_filter(data, cutoff, fs, order=5):
b, a = butter_highpass(cutoff, fs, order=order)
y = signal.filtfilt(b, a, data)
return y
fps = 30
sine_fq = 10 #Hz
duration = 10 #seconds
sine_5Hz = sine_generator(fps,sine_fq,duration)
sine_fq = 1 #Hz
duration = 10 #seconds
sine_1Hz = sine_generator(fps,sine_fq,duration)
sine = sine_5Hz + sine_1Hz
filtered_sine = butter_highpass_filter(sine.data,10,fps)
plt.figure(figsize=(20,10))
plt.subplot(211)
plt.plot(range(len(sine)),sine)
plt.title('generated signal')
plt.subplot(212)
plt.plot(range(len(filtered_sine)),filtered_sine)
plt.title('filtered signal')
plt.show()
Since my reputation is low I am unable to comment on your question - "What is the relationship between cutoff and filter order?" This is not an answer to your original question.
For an FIR filter, for a given cutoff frequency, the slope of the impulse response plot (|H(f)| vs f) is steeper for a higher order filter. So, to achieve higher attenuation for the undesired frequency range, you increase the filter order. But what happens when the filter order is so high that the impulse response is an ideal box function? You will see an inter-symbol interference (ISI in digital communications) like effect. The intensity of this effect increases when the ratio of cutoff frequency to the sampling frequency gets smaller (think of the relation between the width of box function in frequency domain and the width of the main lobe of the sinc function).
I first observed this when I tried to implement a very narrow band low-pass IIR filter on a TI DSP microcontroller. The TI library, implemented the filter as a cascaded bi-quad structure to handle the well known truncation effects. This still did not solve the problem because the problem is not due to truncation alone. The way I solved this problem was to use an anti-aliasing filter followed by down-sampling the input signal, followed my desired low-pass IIR filter.
I understand that you are implementing a HPF, which is an LPF translated in frequency domain. Hope this answers some of your questions. Let me know if down-sampling works for you.
I add function to verify above Konstantin Purtov's code, so you could see relationship between order and cut off frequency. The code is mostly from Warren Weckesser.
import scipy
import sys
from scipy import signal
from scipy import pi
from scipy.io.wavfile import write
import matplotlib.pyplot as plt
import numpy as np
from scipy.signal import butter, lfilter, freqz
# ----- ----- ----- -----
def butter_highpass(cutoff, fs, order=5):
nyq = 0.5 * fs
normal_cutoff = cutoff / nyq
b, a = signal.butter(order, normal_cutoff, btype='high', analog=False)
return b, a
def butter_highpass_filter(data, cutoff, fs, order=5):
b, a = butter_highpass(cutoff, fs, order=order)
y = signal.filtfilt(b, a, data)
return y
# ----- -----
# (1)
def foo(sel):
if (sel == 1):
# Filter requirements.
order = 6
fs = 300.0 # sample rate, Hz
cutoff = 10 # desired cutoff frequency of the filter, Hz
# Get the filter coefficients so we can check its frequency response.
b, a = butter_highpass(cutoff, fs, order)
# Plot the frequency response.
w, h = freqz(b, a, worN=8000)
plt.subplot(2, 1, 1)
plt.plot(0.5 * fs * w / np.pi, np.abs(h), 'b')
plt.plot(cutoff, 0.5 * np.sqrt(2), 'ko')
plt.axvline(cutoff, color='k')
plt.xlim(0, 0.5 * fs)
plt.title("High Filter Frequency Response")
plt.xlabel('Frequency [Hz]')
plt.grid()
# Demonstrate the use of the filter.
# First make some data to be filtered.
T = 0.5 # seconds
n = int(T * fs) # total number of samples
t = np.linspace(0, T, n, endpoint=False)
# "Noisy" data. We want to recover the 20 Hz signal from this.
data = np.sin(1.2 * 2 * np.pi * t) + 1.5 * np.cos(5 * 2 * np.pi * t) + 0.5 * np.sin(20.0 * 2 * np.pi * t)
# Filter the data, and plot both the original and filtered signals.
y = butter_highpass_filter(data, cutoff, fs, order)
plt.subplot(2, 1, 2)
plt.plot(t, data, 'b-', label='data')
plt.plot(t, y, 'g-', linewidth=2, label='filtered data')
plt.xlabel('Time [sec]')
plt.grid()
plt.legend()
plt.subplots_adjust(hspace=0.35)
plt.show()
else:
print ('Please, choose among choices, thanks.')
# ----- -----
def main():
sel = int (sys.argv[1])
foo(sel)
# ----- ----- ----- ----- ----- -----
if __name__ == '__main__':
main()
Related
I'm working on a nerd project for fun.
The project is analog video recorded onto an audio cassette.
The challenge lies in the very limited bandwidth.
I have a method to record color video at 24fps along with mono audio.
I got the video stuff working but need some help with the audio.
Here is the signal I have to work with:
Note: using YUV color space
Left channel:
Sync Pulses &
Y (luma) data
Right channel:
U & V (chroma) data
mixed with
Mono audio (Amplitude Modulated at 14kHz)
I'm not sure how to separate the color data from the audio.
I've looked into FFT with numpy a bit but am not fully understanding it.
Basically what I need is a band filter to separate 13990Hz - 14010Hz (to account for wow and flutter)
Ok here is a little test code that shows how this works.
import matplotlib.pyplot as plt
import numpy as np
import math as mt
from scipy.signal import butter, sosfilt, lfilter
def butter_bandpass(lowcut, highcut, fs, order=5):
nyq = 0.5 * fs
low = lowcut / nyq
high = highcut / nyq
sos = butter(order, [low, high], analog=False, btype='band', output='sos')
return sos
def bandpass(data, lowcut, highcut, fs, order=5):
sos = butter_bandpass(lowcut, highcut, fs, order=order)
y = sosfilt(sos, data)
return y
def bandstop(data, lowcut, highcut, fs, order):
nyq = 0.5 * fs
low = lowcut / nyq
high = highcut / nyq
i, u = butter(order, [low, high], btype='bandstop')
y = lfilter(i, u, data)
return y
# Modulation & Bandpass Test
# note that the data is just test data and isn't actual audio or color data
Fs = 192000 # rate
f = 14000 # carrier frequency (Hz)
sample = 192000 # total length
x = np.arange(sample)
signal = (np.sin(2 * np.pi * 1000 * x / Fs)) # audio
noise = 0.5*(np.sin(2 * np.pi * 10000 * x / Fs)) # color data
y = [] # combined AM audio and color data
for t in range(0, sample, 1):
amp = (signal[t] + 1) / 2
sine = amp * (mt.sin(2*mt.pi * f * t / Fs))
y.append((sine+noise[t])/2)
# y is raw signal
w = bandpass(y, 1600, 1800, 24000, order=1) # only AM audio signal
v = bandstop(y, 1450, 1950, 24000, order=6) # Rest of the signal
# Note: lowered the sample rate input for the filters (and frequencies accordingly)
# since it didn't like 192khz
# The color data does get impacted by the bandstop filter but this is
# mostly not noticable as the YUV color space is used meaning
# the color resolution can be reduced drastically without noticable effect
plt.plot(y)
plt.plot(w)
plt.plot(v)
plt.xlabel('sample')
plt.ylabel('amplitude')
plt.show()
If you want to check out the full code along with a wav of the signal and an example video of the output here's a link:
https://drive.google.com/drive/folders/18ogpK4n43d_Q0tjdmlm2uIRZBIrRu01y?usp=sharing
I want to detrend a signal using a bandpass filter. I used a Butterworth filter with FL=0.1 Hz and FH=20Hz in python but after applying this bandpass filter I observed a large spike at the beginning of the detrended signal.
what is this spike for? and how do I remove this spike in python?
you can download "data1.csv" using this link.
from scipy.signal import butter, lfilter
from numpy import genfromtxt
import numpy as np
import matplotlib.pyplot as plt
def butter_bandpass(lowcut, highcut, fs, order=5):
nyq = 0.5 * fs
low = lowcut / nyq
high = highcut / nyq
b, a = butter(order, [low, high], btype='band')
return b, a
def butter_bandpass_filter(data, lowcut, highcut, fs, order=5):
b, a = butter_bandpass(lowcut, highcut, fs, order=order)
y = lfilter(b, a, data)
return y
BP_without_NaN = genfromtxt('data1.csv', delimiter=',')
framerate=1024
# detrending [0.1Hz 20Hz]
OMW = butter_bandpass_filter(data = BP_without_NaN, lowcut = 0.1, highcut = 20 , fs = framerate, order = 3)
# plot OMW
time = np.linspace(0, len(OMW)/framerate ,len(OMW))
plt.plot(time, OMW)
plt.show()
When applying a Butterworth (or any IIR) filter, each output sample is computed based on previous output samples,
y[n] = b0 * x[n] + b1 * x[n-1] + ... + bk * x[n-k]
- a1 * y[n-1] + ... + ak * y[n-k]
This makes a problem for how start the filter at the beginning of the signal, since no output samples have been computed yet. A typical way handling this is to assume that all outputs at negative times n < 0 are zero, or equivalently, to conceptually extrapolate the input as zero. This zero initialization approach is what scipy.signal.lfilter does by default.
However, this approach is not always a good fit. If the input signal does not begin close to zero, then extrapolating zeros introduces an artificial step or jump discontinuity in the signal, and the output signal will begin with an unwanted transient, the filter's response to this artificial step.
Looking at your linked data, the input begins with a few hundred samples equal to 154. This explains the large spike at the beginning of your plot. You can mostly remove the spike by changing the filter initialization. Use scipy.signal.lfilter_zi to compute initial conditions for lfilter for step response steady-state of value 154:
zi = lfilter_zi(b, a) * data[0]
Then use it with lfilter as y, _ = lfilter(b, a, data, zi=zi). Here is what I get with this change:
I want to read data from the file and plot the fft of it.
I have used the scipy's fft function to plot the fft of the synthetically generated signal data however while I am trying to save the generated signal data and plot the fft I am getting the peaks as I was getting but the value of the amplitude and frequency is off as compared to the Original plotting.I found problem with the second fft function. Can anyone help me with the same.
import numpy as np
import pandas as pd
import scipy
from scipy.fftpack import fft,fftfreq
import matplotlib.pyplot as plt
import csv
#Function to plot the raw data
def raw_plot(signals,time):
'''Plot the raw values in the time domain '''
plt.figure(0)
plt.plot(time,signals)
plt.xlabel("Time(s)")
plt.ylabel("Amplitude(g)")
plt.show()
#Function to plot the fft of the data
def fft_signal(signal, sampling_frequency, timestart, timeend):
T = 1 / sampling_frequency
N = (timeend - timestart) * sampling_frequency
x = np.linspace(0.0, 1.0 / (2.0 * T), int(N / 2))
# print("Values of X are :",x)
# print(len(x))
yr2 = fft(signal)
y2 = 2 / N * np.abs(yr2[0:int(N / 2)])
# amp_spec = abs(fft(signal))/N
# freq = np.linspace(0,2,num=N)
plt.figure(2)
plt.plot(x, y2)
plt.xlabel("Frequency")
plt.ylabel("Amplitude")
plt.show()
def fftsignal2(data, sampling_freq = 100, timestep = 1 ):
N = len(data)
FS = 1000
T= 1/FS
timestep = timestep
yr2 = fft(data)
y2 = 2 / N * np.abs(yr2[0:int(N / 2)])
# f= np.arange(0,N)*FS/N
# f= np.linspace(0,1.0/T,N)
frq = np.fft.fftfreq(N,d=timestep)
# f= np.linspace(0.0,int(N/2)*sampling_freq)
frq= frq[0:len(frq)//2]
print("Frequencies Length :",len(frq))
print("DATA LENGTH :",len(y2))
plt.figure(3)
plt.plot(frq,y2)
plt.xlabel("Frequency")
plt.ylabel("Amplitude")
plt.show()
Mag1_Hz = 70
Mag1_AMP = 10
Mag4_AMP = 60
Mag4_Hz= 50
Mag2_Hz = 20
Mag2_AMP = 8
time_start = 0
time_end = 4
Sampling_frequency = 100
No_of_samples = (time_end-time_start) * Sampling_frequency
time = np.linspace(time_start,time_end,No_of_samples)
#Generate the signals values
data_signal = Mag1_AMP * np.sin(2* np.pi * Mag1_Hz * time ) + Mag2_AMP * np.cos(2* np.pi * Mag2_Hz * time)
raw_plot(signals=data_signal[:800],time=time[:800])
fft_signal(data_signal,sampling_frequency=Sampling_frequency,timestart=time_start,timeend=time_end)
# for (time, datavalues) in zip(time,data_signal):
# with open("data_signals_27.csv","a",newline="") as file:
# writefile = csv.writer(file)
# print("Time%f,Data %f"%(time,datavalues))
# print(time,datavalues)
# writefile.writerow([time,datavalues])
fftsignal2(data= data_signal, sampling_freq= Sampling_frequency,timestep=2)
The peaks are coming right in the fft_signal function no 1 while I am trying to construct the fftsignal2 to plot the fft of the data so the peaks which I am getting is correct but the magnitude and frequency is off.I don't understand why ? Can anyone help me with that.
I tried to filter some signal with fft.
The signal I am working on is quite complicated and im not really experienced in this topic.
That's why I created a simple sin wave 3Hz and tried to cut off the 3 Hz.
and so far, so good
import numpy as np
import matplotlib.pyplot as plt
from scipy.fftpack import fftfreq, irfft, rfft
t = np.linspace(0, 2*np.pi, 1000, endpoint=True)
f = 3.0 # Frequency in Hz
A = 100.0 # Amplitude in Unit
s = A * np.sin(2*np.pi*f*t) # Signal
dt = t[1] - t[0] # Sample Time
W = fftfreq(s.size, d=dt)
f_signal = rfft(s)
cut_f_signal = f_signal.copy()
cut_f_signal[(np.abs(W)>3)] = 0 # cut signal above 3Hz
cs = irfft(cut_f_signal)
fig = plt.figure(figsize=(10,5))
plt.plot(s)
plt.plot(cs)
What i expected
What i got
I don't really know where the noise is coming from.
I think it is some basic stuff, but i dont get it.
Can someone explain to to me?
Edit
Just further information
Frequency
yf = fft(s)
N = s.size
xf = np.linspace(0, fa/2, N/2, endpoint=True)
fig, ax = plt.subplots()
ax.plot(xf,(2.0/N * np.abs(yf[:N//2])))
plt.xlabel('Frequency ($Hz$)')
plt.ylabel('Amplitude ($Unit$)')
plt.show()
You could change the way you create your signal and use a sample frequency:
fs = 1000
t = np.linspace(0, 1000 / fs, 1000, endpoint=False) # 1000 samples
f = 3.0 # Frequency in Hz
A = 100.0 # Amplitude in Unit
s = A * np.sin(2*np.pi*f*t) # Signal
dt = 1/fs
And here the whole code:
import numpy as np
import matplotlib.pyplot as plt
from scipy.fftpack import fftfreq, irfft, rfft
fs = 1000
t = np.linspace(0, 1000 / fs, 1000, endpoint=False)
f = 3.0 # Frequency in Hz
A = 100.0 # Amplitude in Unit
s = A * np.sin(2*np.pi*f*t) # Signal
dt = 1/fs
W = fftfreq(s.size, d=dt)
f_signal = rfft(s)
cut_f_signal = f_signal.copy()
cut_f_signal[(np.abs(W)>3)] = 0 # cut signal above 3Hz
cs = irfft(cut_f_signal)
fig = plt.figure(figsize=(10,5))
plt.plot(s)
plt.plot(cs)
And with f = 3.0 Hz and (np.abs(W) >= 3):
And with f = 1.0 Hz:
Just some additional information about why A. As solution works better than yours:
A. A's model doesn't include any non-integer frequencies in its Solution and after filtering out the higher frequencies the result looks like:
1.8691714842589136e-12 * exp(2*pi*n*t*0.0)
1.033507502555532e-12 * exp(2*pi*n*t*1.0)
2.439774536202658e-12 * exp(2*pi*n*t*2.0)
-8.346741339115191e-13 * exp(2*pi*n*t*3.0)
-5.817427588021649e-15 * exp(2*pi*n*t*-3.0)
4.476938066992472e-14 * exp(2*pi*n*t*-2.0)
-3.8680170177940454e-13 * exp(2*pi*n*t*-1.0)
while your solution includes components like:
...
177.05936105690256 * exp(2*pi*n*t*1.5899578814880346)
339.28717376420747 * exp(2*pi*n*t*1.7489536696368382)
219.76658524130005 * exp(2*pi*n*t*1.9079494577856417)
352.1094590251063 * exp(2*pi*n*t*2.0669452459344453)
267.23939871205346 * exp(2*pi*n*t*2.2259410340832484)
368.3230130593005 * exp(2*pi*n*t*2.384936822232052)
321.0888818355804 * exp(2*pi*n*t*2.5439326103808555)
...
Please refer to this question regarding possible side effects of zeroing FFT bins out.
I am newbie in signal processing, in this question, i want to ask how to obtain energy for each frequency band around interested frequency F. I have found a formula, but I dont know how to implement it in Python. This is the formula and my Fourier transform plot:
x = np.linspace(0,5,100)
y = np.sin(2*np.pi*x)
## fourier transform
f = np.fft.fft(y)
## sample frequencies
freq = np.fft.fftfreq(len(y), d=x[1]-x[0])
plt.plot(freq, abs(f)**2) ## will show a peak at a frequency of 1 as it should.
You are almost there as Mike pointed but here is a different approach which is simpler to understand.you can set a variable that holds the filtered signal and return a 1d array of Af, then apply the above formula which is quite simple (squared sum of these amplitudes)
Filter out the signals like this
from scipy.signal import butter, lfilter
def butter_bandpass(lowcut, highcut, fs, order=5):
nyq = 0.5 * fs
low = lowcut / nyq
high = highcut / nyq
b, a = butter(order, [low, high], btype='band')
return b, a
def butter_bandpass_filter(data, lowcut, highcut, fs, order=5):
b, a = butter_bandpass(lowcut, highcut, fs, order=order)
y = lfilter(b, a, data)
return y
now assuming y is your original signal and you need energy of 5Hz component in signal ,
#let fs = 250
#let order = 5
oneD_array_of_amps_of_fiveHz_component = butter_bandpass_filter(y, 4, 6, 250, 5)
#calculate energy like this
energy_of_fiveHz_comp = sum([x*2 for x in oneD_array_of_amps_of_fiveHz_component])
Using the signal processing module from Finding local maxima/minima with Numpy in a 1D numpy array you would do the following:
from scipy.signal import argrelextrema
import numpy as np
delta = 0.1
F = 1
X_delta = abs(freq - F) < delta
A_f_delta = abs(f[X_delta])**2
maximum_ix = argrelextrema(A_f, np.greater)
E_F_delta = np.sum(A_f[maximum_ix])/2
E_F_delta
2453.8235776144866