I want to amplify audio input at specific frequencies, and I use numpy.fft.
So my question is: When changing the amplitudes of the signal, what happens with phase?
For example, if I multiply amplitudes in some frequency range, by some factor, let's say 2, do I need to change the phases, and if so, what should I do with them?
I've done the amplification without changing phases, and the result was not what I wanted. It's pretty much the same signal, with some unwanted noise.
You shouldn't need the change the phase for something like this. More likely the problem is that you need to be a bit more gentle about applying the boost. It sounds like you are taking some frequency window and multiplying by a constant while leaving everything else unchanged. This will cause ringing in the time domain with a very long tail. You need to smooth the transition from the gain=1 region to the gain=2 region, for instance by using a gaussian waveform with code that looks something like this:
x, t = get_waveform()
f0, df = get_parameters() # Center frequency and bandwidth of gain region
f = np.fft.rfft(x)
freqs = np.fft.fftfreq(len(x), t[1]-t[0])
freqs = freqs[0:len(f)] # rfft has only non-negative frequency components
gain_window = 1 + np.exp(-(freqs-f0)**2/(df)**2)
f = f * gain_window
x = np.fft.irfft(f)
return x
If that works, you can experiment with more aggressive functions that have sharper turn-on and a flatter top.
The FFT may not actually be what you want. FFTs are not normally used for real-time / streaming applications. This is because in the naive approach you have to collect the whole sample buffer before you start processing. For simple filtering applications it is often easier to do filtering directly in the time domain. This is what FIR and IIR filters do.
In order to filter with the fourier transform in real time what you have to do is break your data stream into overlapping blocks of a fixed length, FFT, filter, reverse FFT, and stich them back together without introducing glitches. This is possible, but it is tricky to get right. For a full-blown multi-channel EQ it might still be the best option, but you should at least consider time domain filtering.
If this is not a real-time application, then FFT is the way to go. For medium sized data sets (up to a few hundred megabytes) you can just FFT the whole data set. For much larger data sets you still have to break the data up into blocks, but they can be much larger blocks and you don't have to worry about the latency introduced.
Also, remember the FFT treats the signal as periodic, so if your signal doesn't go to zero at the beginning and end you will need to do some sort of windowing.
Related
I am in the process of making my own system monitoring tool. I'm looking to run a filter (like a Gaussian filter or similar) on a continuous stream of raw data that i'm receiving from a device (My cpu % in this case).
The collection of data values is n elements long. Every time this piece of code runs it appends the new cpu value and removes the oldest keeping the collection at a length of n essentially a deque([float('nan')] * n, maxlen=n) where n is the length of the graph i'm plotting to.
then it filters the whole collection through a Gaussian filter creating the smoothed data points and then plots them, creating an animated graph similar to most system monitors cpu % graphs found on your computer.
This works just fine... However there has to be a more efficient way to filter the incoming data instead of running a filter on the whole data set every time a new data val is added (in my case the graph updates every .2 sec)
I can think of ways to do it without filtering the whole list but im not sure they are very efficient. Is there anything out there in the signal processing world that will work for me? Apologies if my explanation is a bit confusing, I'm very new to this.
from scipy.ndimage.filters import gaussian_filter1d
# Not my actual code but hopefully describes what im doing
def animate(): # function that is called every couple of milliseconds to animate the graph
# ... other stuff
values.append(get_new_val) # values = collection of data vals from cpu
line.set_ydata(gaussian_filter1d(values, sigma=4)) # line = the line object used for graphing
# ... other stuff
graph_line(line) # function that graphs the line
tl;dr: looking for an optimized way to smooth raw streaming data instead of filtering the whole data set every pass.
I've never used one, but what you need like sounds what a Savitzky–Golay filter is for. It is a local smoothing filter that can be used to make data more differentiable (and to differentiate it, while we're at it).
The good news is that scipy supports this filter as of version 0.14. The relevant part of the documentation:
scipy.signal.savgol_filter(x, window_length, polyorder, deriv=0, delta=1.0, axis=-1, mode='interp', cval=0.0)
Apply a Savitzky-Golay filter to an array.
This is a 1-d filter. If x has dimension greater than 1, axis determines the axis along which the filter is applied.
Parameters:
x : array_like
The data to be filtered. If x is not a single or double precision floating point array, it will be converted to type numpy.float64 before ftering.
window_length : int
The length of the filter window (i.e. the number of coefficients). window_length must be a positive odd integer.
polyorder : int
The order of the polynomial used to fit the samples. polyorder must be less than window_length.
[...]
I would first determine a small pair of polynomial order and window size. Instead of working with the full n data points, you only need to smooth a much smaller deque of a length of roughly window_length. As each new data point comes in, you have to append it to your smaller deque, apply the Savitzky–Golay filter, take the new filtered point, and append it to your graph.
Note, however, that it seems to me that the method is mostly well-defined when not on the edge of the data set. This might mean that for precision's sake you might have to introduce a few measurements' worth of delay, so that you can always use points which are inside a given window (what I mean is that for a given time point you likely need "future" data points to get a reliable filtered value). Considering that your data is measured five times every second, this might be a reasonable compromise if necessary.
I want to know how much energy is at a specific frequency in a signal. I am using FFT to get the spectrum, and the frequency step is determined by the length of my signal.
My spectrum looks, for example, like this :
I want to get the spectrum peak at a specific frequency, -0.08. However, the discretization of the spectrum only give me a peaks at -0.0729 and -0.0833.
Is there a way to shift the spectrum to make sure there is a data point at the frequency I want? Or a way to get the value without necessarily using fft?
Thank you very much!
What you're actually doing when you take a DFT (or any Fourier Transform) is measuring how much of your signal "intersects" with sines of certain frequencies. This is done by summing product of your signal with the complex conjugate of the wave of whatever frequency. Technically, this is called an inner product, which is a generalization of the dot product, and measures how "close" a signal is to another. So if you're only interested in one frequency, don't take the whole DFT, just look at one you want.
I'm not sure what your units are, so I'll assume you want the peak at f0 = -0.08 Hz (If your units are something else, like normalized to the sampling frequency, then you'll need to account for that). This corresponds to the complex exponential exp(2*pi*j*f0*t). Because you're sampling, your t is discrete, so t = n/fs, where fs is the sampling frequency (in Hz).
# assuming you're using numpy arrays
w = exp(-2*pi*1j*f0*arange(len(signal))/fs)
peak = abs(sum(signal*w))
There are different definitions of the DFT; I'm pretty sure numpy's corresponds to what I have above. The extra minus in the exponential is because it's the complex conjugate.
Notice that it's unlikely that w is actually periodic. If the number of samples is large enough this doesn't really matter. A good heuristic is at least 10 periods.
If you have discrete data but need an output for a continuous variable you'll necessarily need some kind of interpolation function. For a value by request style I would advise Scipy interp1d (example of the use of a interp1d function). I believe it's the fastest way to achieve your intended results.
I am attempting to filter a list of 16-bit two's-complement integer using a Butterworth filter generated using scipy.signal.butter and scipy.signal.lfilter (the list was extracted from a PCM-encoded .wav file). My primary language is C, however the majority of this project already exists in Python, and so I need to code this feature within the existing Python framework.
The butterworth filters produced by scipy.signal (as far as I'm aware) are unity gain, and based upon the plot of the frequency response of the filter, that should be the case. I designed the filter using butter() and filtered my dataset using:
data.l = lfilter(b, a, data.l)
where data.l is a class containing an list 'l' containing PCM data, and 'b' and 'a' are the coefficients produced by butter().
However, regardless of the PCM data that I send in, the filter appears to be applying non-unity gain.
For example, filtering full-bandwidth pink noise with max(data.l) = 32,767, and min(data.l) = -32,768 before filtering (as expected for a 16-bit PCM signal) returns a signal with approximately 5% increased gain in the passband. i.e max(data.l) = 34,319.0057 and min(data.l) = -37,593.
The filter appears to be correctly filtering the signal apart from the gain; if I save this PCM data back into a .wav file, and compare a spectrogram of the data to the original signal, the frequency response is exactly as would be expected from my test filters. It seems to be functioning perfectly except for the odd increase in gain?
Obviously I can just rescale this output down to fit into my 16-bit PCM dataset, however I am writing this as part of a wider set of signal processing modules that are designed to be flexible and eventually include non-unity gain filters. For this reason, I want to attempt to figure out why this gain is being applied so as to potentially fix the issue, and not be arbitrarily rescaling the output of my butter() filter.
Does anyone with experience with scipy.signal have an idea as to why this may be the case?
It is normal for a discrete time Butterworth filter to have ringing transients, and these can overshoot the bounds of the input. Take a look at the step response of your filter (as opposed to the frequency response, which is a steady state calculation).
For example, the following code
In [638]: from scipy.signal import butter, lfilter, freqz
In [639]: b, a = butter(3, 0.2)
In [640]: step_response = lfilter(b, a, np.ones(50))
In [641]: plot(step_response)
Out[641]: [<matplotlib.lines.Line2D at 0x10ecb2850>]
generates the plot
Note the overshoot.
The frequency response shows the expected gains for a (discrete time) Butterworth filter.
In [642]: w, h = freqz(b, a, worN=1000)
In [643]: plot(w/np.pi, np.abs(h))
Out[643]: [<matplotlib.lines.Line2D at 0x10f7a90d0>]
See also: http://www.dspguide.com/ch20/3.htm
Concerning your second question: If you remove the overshoots, you'll either cause distortions (clipping) or you'll end up with a different frequency response as impulse / step and frequency response are chained together by the Laplace transform.
By filtering, you change your samples, so the concept of "preserving signal level" is questionable in my opinion - the level of a stationary (e.g. sinusoidal) signal in the passband should remain more or less the same as a Butterworth filter has no ripple in the passband, but components in the stop band (transients -> high frequency) are changed of course.
You could try using a filter with Bessel characteristics that has no overshoot (at the cost of a more gentle slope between pass and stop band) if you want to avoid the rescaling.
I'm using a slightly modified version of this python code to do frequency analysis:
FFT wrong value?
Lets say I have a pack of sine waves in the time domain that are very close together in frequency, while sharing the same amplitude. This is how they look like in the frequency domain, using FFT on 1024 samples from which I strip out the second half, giving 512 bins of resolution:
This is when I apply a FFT over the same group of waves but this time with 128 samples (64 bins):
I expected a plateau-ish frequency response but it looks like the waves in the center are being cancelled. What are those "horns" I see? Is this normal?
I believe your result is correct. The peaks are at ±f1 and ±f2), corresponding to the respective frequency components of the two signals shown in your first plot.
I assume that you are shifting the DC component back to the center? What "waves in the center" are you referring to?
There are a couple of other potential issues that you should be aware of:
Aliasing: by inspection it appears that you have enough samples across your signal but keep in mind that artificial (or aliased) frequencies can be created by the FFT, if there are not enough sample points to capture the underlying frequency. Specifically, if your frequency is f, then you need your data sample spacing to be at least, Δx = 1/(2*f), or smaller.
Windowing: your signal is windowed (has a finite extent) so there will also be some broadening, ringing, or redistribution of power about each spatial frequency due to edge affects.
Since I don't know the details of your data, I went ahead and created a sinusoid and then sampled the data close to what appears to be your sampling rate. For example, below is a sinusoid with 64 points and with a signal frequency at 10 cycles (count the peaks):
The FFT result is then:
which shows the same quantitative features as yours, but without having your data, its difficult for me to match your exact situation (spacing and taper).
Next I applied a super-Gauss window function (shown below) to simulate the finite extent of your data:
After applying the window to the input signal we have:
The corresponding FFT result shows some additional power redistribution, due to the finite extent of the data:
Although I can't match your exact situation, I believe your results appear as expected and some qualitative features of your data have been identified. Hope this helps.
Sine waves closely spaced in the frequency domain will occasionally nearly cancel out in the time domain. Since your second FFT is 8 times shorter than your first FFT, you may have windowed just such an short area of cancellation. Try a different time location of shorter time window to see something different (or different phases of sinusoids).
I'm working on a project to find the instantaneous frequency of a multicomponent audio signal in Python. I am currently using a Butterworth bandpass filter combined with scipy.signal.lfilter to extract around my desired frequency region. I then use the analytic signal (from scipy.signal.hilbert) to get the instantaneous phase, which can be unwrapped to give frequency.
As a relative novice to signal processing, I have two main questions:
I have read that in many applications, it is preferable to use scipy.signal.filtfilt over scipy.signal.lfilter. Certainly when I apply filtfilt to my data, I get a significantly smoother looking instantaneous frequency signal. I would like to know the major differences between the two, bearing in mind that I would like to get an output which is as close to the "true" instantaneous frequency as possible.
The instantaneous frequency data is nonstationary, which means that in some instances I must use a wider bandpass filter to capture all my desired data. This appears to introduce additional noise, and occassional instabilities, into my signal. Are there ways to deal with these kinds of problems, for example with a better designed filter?
EDIT
In response to flebool, below are some images of the data I am looking at. First, a comparison of filt and filtfilt:
Both the above signals have had the same Butterworth filter applied (although the filter function is different), followed by extraction of instantaneous frequency (which is what is plotted, as a function of time). filtfilt seems to shift and smooth the data. Is one of these signals a better approximation of the "true" signal?
Note that this graph shows just a subset of a particular signal.
Second, the effect of increasing the Butterworth filter size:
This is for the same subset of data as figure 1. The legend shows the lower and upper bound for the filter, respectively (the red trace is the filt version of the data in figure 1).
Although it may not be clear here why I would use a larger pass band, in some instances the data may be located at various points between, say, 600 and 800Hz. It is here that I would require a broader filter design. You can see that additional noise enters the trace as the filter gets wider; I'd like to know if there's a way to optimise/improve my filter design.
Some sparse comments:
1) On the top picture: I can't comment on what is best between filt and filtfilt, though the shift in frequency of filtfilt is worrying. You can obtain a similar result by applying a low-pass filter to the filt signal.
2) There isn't a "true" instantaneous frequency, unless the signal was specifically generated with a certain tone. In my experience unwrapping the phase of the Hilbert transform does a good job in many cases, though. It becomes less and less reliable as the ratio of noise to signal intensity grows.
3) Regarding the bottom picture, you say that sometimes you need a large bandpass filter. Is this because the signal is very long, and the instantaneous frequency moves around between 500 and 800 Hz? If so, you may want to proceed windowing the signal to a length at which the filtered signal has a distinct peak in the Fourier spectrum, extract that peak, tailor your bandbass filter to that peak, apply Hilbert to the windowed signal, extract the phase, filter the phase.
This is worth doing if you are sure the signal has other harmonics except noise and the one you are interested in, and it takes a while. Before doing so I would want to be sure the data I obtain is wrong.
If it is simply 1 harmonic + noise, I would lowpass+hilbert+extract instantaneous phase + lowpass again on the instantaneous phase
To your first problem I can't speak intelligently on but scipy is generally well documented so I'd start reading through some of their stuff.
To your second problem a better designed filter would certainly help. You say the data is "non-stationary," do you know where it will be? Or what kind frequencies it might occupy? For example if the signal is centered around 1 of 3 frequencies that you know a-priori you could have three different filters and run the signal through all 3 (only one giving you the output you want of course).
If you don't know have that kind of knowledge about the signal I would first do a wider BPF, then do some peak detection, and apply a more stringent BPF when you know where the data you would like is located