I have a series of signals, sample data looks like this:
We can see that there are 5 peaks there. I can assume that there won't be more than 1 pick every 10 samples, usually there is one pick every 20 to 40 samples.
I was trying to fit a polynomial and then use scipy.signal.find_peaks and it kind of works but I have to choose different numbers of spline knots to approximate each series correctly and the number of knots correlates to the number of peaks so I sort of ended up where I begun - but now I'd need only a rough idea about the number of peaks.
Then I tried it by dividing the signal into parts:
window = 10 # the smallest range potentially containing whole peak
parts = np.array_split(data, len(data)//window) # divide data set into parts
lengths = []
d = np.nan
for i in parts:
d = abs(i.max() - i.min())
lengths.append(d) # differences between max and min values in each part
av = sum(lengths)/len(lengths)
for i in lengths:
if i < some_tolerance_fraction*av:
window = window+1 # make part for the next check bigger
break
The idea was that the difference between min and max values in these parts should be smaller than the height of an actual pick I'm looking for unless the parts are large enough to contain whole peak - then the differences should be similar in each part and the average should also be similar to the actual height of the pick.
But this doesn't work at all and possibly doesn't even make sense - depending on the tolerance it divides window all the time or doesn't divide it at all.
this is the array from the image:
array([254256., 254390., 251546., 250561., 250603., 250128., 251000.,
252612., 253552., 253776., 252843., 251800., 250808., 250569.,
249804., 247755., 247685., 247111., 242320., 242580., 243462.,
240383., 239689., 240730., 239508., 239604., 238544., 240174.,
240806., 240218., 239956., 241325., 241343., 241532., 240696.,
242064., 241830., 237569., 237392., 236353., 234819., 234430.,
233890., 233215., 233745., 232159., 231778., 230307., 228754.,
225823., 225139., 223737., 222078., 221188., 220669., 221944.,
223928., 224996., 223405., 223018., 224966., 226590., 226166.,
226012., 226192., 224900., 224439., 223179., 222375., 221509.,
220734., 219686., 218656., 217792., 215934., 214829., 213673.,
212837., 211604., 210748., 210216., 209974., 209659., 209707.,
210131., 210663., 212113., 213078., 214476., 215087., 216220.,
216831., 217286., 217373., 217030., 216491., 215642., 214249.,
213273., 212148., 210846., 209570., 208202., 207165., 206677.,
205703., 203837., 202620., 201530., 198812., 197654., 196506.,
194163., 193736., 193945., 193785., 193417., 193044., 193768.,
194690., 195739., 198592., 199237., 199932., 200142., 199859.,
199593., 199337., 198403., 197500., 195988., 195114., 194278.,
193837., 193861.])
I would use find_peaks of scipy but filtering the signal with a moving average mean:
import numpy as np
import matplotlib.pyplot as plt
arr = np.array([254256., 254390., 251546., 250561., 250603., 250128., 251000.,
252612., 253552., 253776., 252843., 251800., 250808., 250569.,
249804., 247755., 247685., 247111., 242320., 242580., 243462.,
240383., 239689., 240730., 239508., 239604., 238544., 240174.,
240806., 240218., 239956., 241325., 241343., 241532., 240696.,
242064., 241830., 237569., 237392., 236353., 234819., 234430.,
233890., 233215., 233745., 232159., 231778., 230307., 228754.,
225823., 225139., 223737., 222078., 221188., 220669., 221944.,
223928., 224996., 223405., 223018., 224966., 226590., 226166.,
226012., 226192., 224900., 224439., 223179., 222375., 221509.,
220734., 219686., 218656., 217792., 215934., 214829., 213673.,
212837., 211604., 210748., 210216., 209974., 209659., 209707.,
210131., 210663., 212113., 213078., 214476., 215087., 216220.,
216831., 217286., 217373., 217030., 216491., 215642., 214249.,
213273., 212148., 210846., 209570., 208202., 207165., 206677.,
205703., 203837., 202620., 201530., 198812., 197654., 196506.,
194163., 193736., 193945., 193785., 193417., 193044., 193768.,
194690., 195739., 198592., 199237., 199932., 200142., 199859.,
199593., 199337., 198403., 197500., 195988., 195114., 194278.,
193837., 193861.])
def moving_average(x, w):
"""calculate moving average with window size w"""
return np.convolve(x, np.ones(w), 'valid') / w
#moving average with size 5
n=5
arr_f = moving_average(arr, 5)
#to show in same plot
arr_f_ext= np.hstack([np.ones(n//2)*arr_f[0],arr_f])
plt.figure()
plt.plot(arr,'o')
plt.plot(arr_f_ext)
This will show:
Then find peaks:
from scipy.signal import find_peaks
#n//2 is the offset of the averaged signal (2 in this example)
peaks =find_peaks(arr_f)[0] + n//2
plt.plot(peaks,arr[peaks],'xr',ms=10)
wich will show:
Note that,
the filtered signal will have a delay of n/2 samples (rounding down) so add n//2 to the peaks finded in filtered signal.
2)the filtered signal does not have the same values that the original, but same behaviour, Then to extract peak value use the original signal.
My informal definition of a peak is a point surrounded by two vectors, one ascending and one descending. It's pretty easy to implement it by iterating the array and comparing two neighbouring segments.
If they are both in the same direction, we merge the 2 segments by deleting the middle point.
To determine if they are in the same direction, I used multiplication. The product is positive if the 2 segments are in same direction.
At the end, every point will be a peak (we cannot determine for the first and last two).
i = 0 # position cursor at beginning
while i <= (len(t)-3):
if (t[i] - t[i+1]) * (t[i+1] - t[i+2]) >= 0:
# Same direction: join 2 segments by removing the middlepoint.
# This test also include the case of an horizontal segment \
# formed by the first 2 points. We remove the second.
del( t[i+1])
else:
# different directions. Delete nothing. Move cursor by 1
i += 1
see plot. You can see the reduction from 135 to 34 points.
Each blue mark is a peak.
Some of these peaks are non-significant and some more filtering is required. But the best method depend on your application. You may filter on vertical distance between 2 adjacent peaks or the horizontal distance between 2 adjacent peaks. For this last case, we need the x value of each peak so I rewrote the program using x-y data points.
t0 = [254256, 254390, 251546, 250561, 250603, 250128, 251000,
252612, 253552, 253776, 252843, 251800, 250808, 250569,
249804, 247755, 247685, 247111, 242320, 242580, 243462,
240383, 239689, 240730, 239508, 239604, 238544, 240174,
240806, 240218, 239956, 241325, 241343, 241532, 240696,
242064, 241830, 237569, 237392, 236353, 234819, 234430,
233890, 233215, 233745, 232159, 231778, 230307, 228754,
225823, 225139, 223737, 222078, 221188, 220669, 221944,
223928, 224996, 223405, 223018, 224966, 226590, 226166,
226012, 226192, 224900, 224439, 223179, 222375, 221509,
220734, 219686, 218656, 217792, 215934, 214829, 213673,
212837, 211604, 210748, 210216, 209974, 209659, 209707,
210131, 210663, 212113, 213078, 214476, 215087, 216220,
216831, 217286, 217373, 217030, 216491, 215642, 214249,
213273, 212148, 210846, 209570, 208202, 207165, 206677,
205703, 203837, 202620, 201530, 198812, 197654, 196506,
194163, 193736, 193945, 193785, 193417, 193044, 193768,
194690, 195739, 198592, 199237, 199932, 200142, 199859,
199593, 199337, 198403, 197500, 195988, 195114, 194278,
193837, 193861]
def graph( t1, t2):
import matplotlib.pyplot as plt
fig=plt.figure()
plt.plot( [p[0] for p in t1], [p[1] for p in t1], color='r', label="raw data")
plt.plot( [p[0] for p in t2], [p[1] for p in t2], marker='.', color='b', label="reduced data")
plt.title('Peak identification')
plt.legend()
plt.show()
def reduce( t):
i = 0 # position cursor at beginning
while i < (len(t)-2):
if (t[i][1] - t[i+1][1]) * (t[i+1][1] - t[i+2][1]) >= 0:
# Same direction: join 2 segments by removing the middlepoint.
# This test also include the case of an horizontal segment \
# formed by the first 2 points. We remove the second.
del( t[i+1])
else:
# different directions. Delete nothing. Move cursor by 1
i += 1
t1 = [(i,t) for i,t in enumerate(t0)] # add x to every data point
t = t1.copy()
reduce( t)
graph( t1, t)
Have fun!
I have 16 guassian curves which I have to fit with one guassian curve. I was unable to imply the sum of guassian(multiple regression) in python.
Here is the code I am using:
import matplotlib.pyplot as plt
import numpy as np
a=np.array([3750.0, -250.0, 6750.0, 2750.0, -2050.0, 6350.0, 1550.0, -4050.0, 5750.0, 150.0, -6250.0, 4950.0, -1450.0, -8650.0, 3950.0, -3250.0])
v1=np.array( [2.5470357695283954, 0.1937004980283323, 0.43831655553839766, 6.07645636407398, 0.6331239135554633, 0.969937308645575, 13.38133838752005, 1.3226417845166933, 1.5531178254607325, 27.599625693090765, 2.031000233294804, 1.635762971986014, 53.83073800155456, 2.0719664311822843, 0.0, 100.0])
x=[]
s=[]
v5=9.9e2
for j in range(0,len(a)):
for i in range(-1500,1500):
v11=a[j]+i
x.append(v11)
z=np.exp((-4*np.log(2)*((v11-a[j])/(v5))**2))*((4.5*np.log(2)/(np.pi))**0.5)
s.append(z*v1[j])
plt.plot(x,s,'--r',)
plt.stem(a,v1)
Which generates the following plot (with the problem circled):
Instead of the desired output:
The output of your code shows this overlapping because you are not summing the 16 gaussians but instead creating an array containing [x1_g1,x1_g1,...,x3000_g1,x1_g2,...,x3000_g16] and the same for s. It is a 1d array containing the 3000 x values of the first gaussian, then the 3000 x values of the second gaussian and so on. But they are not added. Thus, the plot shows the 16 independent gaussians instead of the sum which is the desired output.
In the actual code, the x values of each gaussian are different (going -1500 and +1500 around its center) which makes adding the 16 gaussians more complicated.
If we consider only the first 2 gaussians for instance, centered at 3750 and -250, the values appended in x from the first gaussian go from 2250 to 5250 in steps of 1, as well as their images in s which are s(2250)... Afterwards, the values of the second gaussian (x between -1750 and 1250) are appended (not added), which will result in an x list like that:
x = [2250,2251,<in steps of 1>,5249,5250,-1750,-1749,<in steps of 1>,1250]
And s is a list where each position contains the image of the same position in x. Strating from this format, getting the final output which is the sum of the gaussians id difficult, because we wolud have to check for equivalent values of x, and sum their contributions...
However, if instead we always evaluated the gaussians at the same positions (in the exemple between -1750 and 5250 in steps of 1), we will have much more values stored, and most of them will be zero, but adding them will be straightforward.
Half-way vectorization
One option similar to the code in the question is the following:
a = np.array([3750.0, -250.0, 6750.0, 2750.0, -2050.0, 6350.0, 1550.0, -4050.0, 5750.0, 150.0, -6250.0, 4950.0, -1450.0, -8650.0, 3950.0, -3250.0])
v1 = np.array( [2.5470357695283954, 0.1937004980283323, 0.43831655553839766, 6.07645636407398, 0.6331239135554633, 0.969937308645575, 13.38133838752005, 1.3226417845166933, 1.5531178254607325, 27.599625693090765, 2.031000233294804, 1.635762971986014, 53.83073800155456, 2.0719664311822843, 0.0, 100.0])
v5 = 9.9e2
xrange = np.arange(a.min()-1500,a.max()+1500)
# This generates an array between the minimum of a minus 1500 and the maximum of a
# plus 1500. This way, all the values in the old x list are contained in ths array
# Therefore, it becomes really easy to sum the contribution of each gaussian,
# because only an element-wise sum is needed.
s = np.zeros(len(xrange))
for j,aj in enumerate(a):
z = np.exp((-4*np.log(2)*((xrange-aj)/(v5))**2))*((4.5*np.log(2)/(np.pi))**0.5)
s += z*v1[j]
plt.plot(xrange,s,'--r')
plt.stem(a,v1)
The output plot is the same as for the completely vectorized solution.
Completely vectorized solution
One simple solution is to define a unique xrange for all 16 gaussians, then calculate s for each of them (on the same x values) and finally sum over the 16 gaussians:
a = np.array([3750.0, -250.0, 6750.0, 2750.0, -2050.0, 6350.0, 1550.0, -4050.0, 5750.0, 150.0, -6250.0, 4950.0, -1450.0, -8650.0, 3950.0, -3250.0])
v1 = np.array( [2.5470357695283954, 0.1937004980283323, 0.43831655553839766, 6.07645636407398, 0.6331239135554633, 0.969937308645575, 13.38133838752005, 1.3226417845166933, 1.5531178254607325, 27.599625693090765, 2.031000233294804, 1.635762971986014, 53.83073800155456, 2.0719664311822843, 0.0, 100.0])
v5 = 9.9e2
xrange = np.arange(a.min()-1500,a.max()+1500)
z = np.exp((-4*np.log(2)*((xrange-a.reshape((len(a),1)))/(v5))**2))*((4.5*np.log(2)/(np.pi))**0.5)
s = z*v1.reshape((len(a),1))
plt.plot(xrange,s.sum(axis=0),'--r')
plt.stem(a,v1)
Note that I have removed the 2 nested loops using numpy.
The loop over range(-1500,1500) can be avoided defining i=np.arange(-1500,1500) instead of the for i in ... and leaving the rest of the code untouched (only indentation has to be updated). Thet is because numpy operated element-wise over the arrays.
The second loop is a bit trickier than that. The a and v1 arrays are reshaped to a 2d array, in order to generate a z with the shape (16,len(xrange)). Thas is why combining an array xrange of length muxh larger than 16 with a does not raise any error of dimensions not matching, because one is the 1st dimension and the other the second.
The code above generates the following plot:
Groupby solution
There is also the option of working with the same code to generate x and s and afterwards, plot every unique value of x (the same value of x can be found in x[i1],x[i2],x[i3]) versus s[i1]+s[i2]+s[i3].
This can be done adding the following code after the loops:
x,s = np.array(x),np.array(s)
ind = np.argsort(x)
x,s = x[ind],s[ind]
unique_x = np.unique(x)
catsums=[]
for k in unique_x:
catsums.append(np.sum(s[np.where(x==k)]))
plt.plot(u,catsums,'--r')
plt.stem(a,v1)
This groupby can also be vectorized using numpy or pandas as it is explained in this other SO answer
This link provides code for an autocorrelation-based pitch detection algorithm. I am using it to detect pitches in simple guitar melodies.
In general, it produces very good results. For example, for the melody C4, C#4, D4, D#4, E4 it outputs:
262.743653536
272.144441273
290.826273006
310.431336809
327.094621169
Which correlates to the correct notes.
However, in some cases like this audio file (E4, F4, F#4, G4, G#4, A4, A#4, B4) it produces errors:
325.861452246
13381.6439242
367.518651703
391.479384923
414.604661221
218.345286173
466.503751322
244.994090035
More specifically, there are three errors here: 13381Hz is wrongly detected instead of F4 (~350Hz) (weird error), and also 218Hz instead of A4 (440Hz) and 244Hz instead of B4 (~493Hz), which are octave errors.
I assume the two errors are caused by something different? Here is the code:
slices = segment_signal(y, sr)
for segment in slices:
pitch = freq_from_autocorr(segment, sr)
print pitch
def segment_signal(y, sr, onset_frames=None, offset=0.1):
if (onset_frames == None):
onset_frames = remove_dense_onsets(librosa.onset.onset_detect(y=y, sr=sr))
offset_samples = int(librosa.time_to_samples(offset, sr))
print onset_frames
slices = np.array([y[i : i + offset_samples] for i
in librosa.frames_to_samples(onset_frames)])
return slices
You can see the freq_from_autocorr function in the first link above.
The only think that I have changed is this line:
corr = corr[len(corr)/2:]
Which I have replaced with:
corr = corr[int(len(corr)/2):]
UPDATE:
I noticed the smallest the offset I use (the smallest the signal segment I use to detect each pitch), the more high-frequency (10000+ Hz) errors I get.
Specifically, I noticed that the part that goes differently in those cases (10000+ Hz) is the calculation of the i_peak value. When in cases with no error it is in the range of 50-150, in the case of the error it is 3-5.
The autocorrelation function in the code snippet that you linked is not particularly robust. In order to get the correct result, it needs to locate the first peak on the left hand side of the autocorrelation curve. The method that the other developer used (calling the numpy.argmax() function) does not always find the correct value.
I've implemented a slightly more robust version, using the peakutils package. I don't promise that it's perfectly robust either, but in any case it achieves a better result than the version of the freq_from_autocorr() function that you were previously using.
My example solution is listed below:
import librosa
import numpy as np
import matplotlib.pyplot as plt
from scipy.signal import fftconvolve
from pprint import pprint
import peakutils
def freq_from_autocorr(signal, fs):
# Calculate autocorrelation (same thing as convolution, but with one input
# reversed in time), and throw away the negative lags
signal -= np.mean(signal) # Remove DC offset
corr = fftconvolve(signal, signal[::-1], mode='full')
corr = corr[len(corr)//2:]
# Find the first peak on the left
i_peak = peakutils.indexes(corr, thres=0.8, min_dist=5)[0]
i_interp = parabolic(corr, i_peak)[0]
return fs / i_interp, corr, i_interp
def parabolic(f, x):
"""
Quadratic interpolation for estimating the true position of an
inter-sample maximum when nearby samples are known.
f is a vector and x is an index for that vector.
Returns (vx, vy), the coordinates of the vertex of a parabola that goes
through point x and its two neighbors.
Example:
Defining a vector f with a local maximum at index 3 (= 6), find local
maximum if points 2, 3, and 4 actually defined a parabola.
In [3]: f = [2, 3, 1, 6, 4, 2, 3, 1]
In [4]: parabolic(f, argmax(f))
Out[4]: (3.2142857142857144, 6.1607142857142856)
"""
xv = 1/2. * (f[x-1] - f[x+1]) / (f[x-1] - 2 * f[x] + f[x+1]) + x
yv = f[x] - 1/4. * (f[x-1] - f[x+1]) * (xv - x)
return (xv, yv)
# Time window after initial onset (in units of seconds)
window = 0.1
# Open the file and obtain the sampling rate
y, sr = librosa.core.load("./Vocaroo_s1A26VqpKgT0.mp3")
idx = np.arange(len(y))
# Set the window size in terms of number of samples
winsamp = int(window * sr)
# Calcualte the onset frames in the usual way
onset_frames = librosa.onset.onset_detect(y=y, sr=sr)
onstm = librosa.frames_to_time(onset_frames, sr=sr)
fqlist = [] # List of estimated frequencies, one per note
crlist = [] # List of autocorrelation arrays, one array per note
iplist = [] # List of peak interpolated peak indices, one per note
for tm in onstm:
startidx = int(tm * sr)
freq, corr, ip = freq_from_autocorr(y[startidx:startidx+winsamp], sr)
fqlist.append(freq)
crlist.append(corr)
iplist.append(ip)
pprint(fqlist)
# Choose which notes to plot (it's set to show all 8 notes in this case)
plidx = [0, 1, 2, 3, 4, 5, 6, 7]
# Plot amplitude curves of all notes in the plidx list
fgwin = plt.figure(figsize=[8, 10])
fgwin.subplots_adjust(bottom=0.0, top=0.98, hspace=0.3)
axwin = []
ii = 1
for tm in onstm[plidx]:
axwin.append(fgwin.add_subplot(len(plidx)+1, 1, ii))
startidx = int(tm * sr)
axwin[-1].plot(np.arange(startidx, startidx+winsamp), y[startidx:startidx+winsamp])
ii += 1
axwin[-1].set_xlabel('Sample ID Number', fontsize=18)
fgwin.show()
# Plot autocorrelation function of all notes in the plidx list
fgcorr = plt.figure(figsize=[8,10])
fgcorr.subplots_adjust(bottom=0.0, top=0.98, hspace=0.3)
axcorr = []
ii = 1
for cr, ip in zip([crlist[ii] for ii in plidx], [iplist[ij] for ij in plidx]):
if ii == 1:
shax = None
else:
shax = axcorr[0]
axcorr.append(fgcorr.add_subplot(len(plidx)+1, 1, ii, sharex=shax))
axcorr[-1].plot(np.arange(500), cr[0:500])
# Plot the location of the leftmost peak
axcorr[-1].axvline(ip, color='r')
ii += 1
axcorr[-1].set_xlabel('Time Lag Index (Zoomed)', fontsize=18)
fgcorr.show()
The printed output looks like:
In [1]: %run autocorr.py
[325.81996740236065,
346.43374761017725,
367.12435233192753,
390.17291696559079,
412.9358117076161,
436.04054933498134,
465.38986619237039,
490.34120132405866]
The first figure produced by my code sample depicts the amplitude curves for the next 0.1 seconds following each detected onset time:
The second figure produced by the code shows the autocorrelation curves, as computed inside of the freq_from_autocorr() function. The vertical red lines depict the location of the first peak on the left for each curve, as estimated by the peakutils package. The method used by the other developer was getting incorrect results for some of these red lines; that's why his version of that function was occasionally returning the wrong frequencies.
My suggestion would be to test the revised version of the freq_from_autocorr() function on other recordings, see if you can find more challenging examples where even the improved version still gives incorrect results, and then get creative and try to develop an even more robust peak finding algorithm that never, ever mis-fires.
The autocorrelation method is not always right. You may want to implement a more sophisticated method like YIN:
http://audition.ens.fr/adc/pdf/2002_JASA_YIN.pdf
or MPM:
http://www.cs.otago.ac.nz/tartini/papers/A_Smarter_Way_to_Find_Pitch.pdf
Both of the above papers are good reads.
i`m having a class called numerical methods, where we learn how to write programs for certain problems in physics. We had to write 4 programs which could solve ODEs (implicit/explicit euler, velocity-verlet, implicit midpoint rule), now we have to calculate the error by using |y_N - y(T)|. We already have a template which we need to fill out.
This is the code which we have to complete.
def ex2_d():
T = 0.2
y0 = np.array([0.3, 0.0])
all_methods = [explicit_euler, implicit_euler, implicit_mid_point, velocity_verlet]
all_rhs = 3*[pendulum_rhs] + [pendulum_verlet_rhs]
resolutions = 2**np.arange(4, 11)
_, y_exact = ode45(pendulum_rhs, (0.0, T), y0, reltol=1e-12)
for method, rhs in zip(all_methods, all_rhs):
error = np.empty(resolutions.size)
for k, N in enumerate(resolutions):
# TODO: Berechen Sie die Lösung und den Fehler
error[k] = np.absolute(methode())
rate = convergence_rate(error, resolutions)
print(method)
print("rate: " + str(rate) + "\n")
The only thing I need to fill out is the TODO part. But I don`t understand, the for loop, which is looping over k and N in enumerate(resolution), and why is the resolution array declared as it is anyways?
Thank you in advance for your help!
In numerically solving an ODE, you want to have doubling resolutions (halving step sizes), to find the convergence rate, using the standard method:
(u_h - u_(h/2))/(u_(h/2) - u_(h/4)) = 2^p + O(h)
with u_h the numerical solution at a step h, u_(h/2) the solution with a step h/2 (e.g. double resolution) and u_(h/4) the solution with a step h/4 (e.g. again double resolution). The order of the error is p, which gives a convergence rate of h^p
This is why the resolutions are declared as 2**np.arange(4,11), which gives[ 16, 32, 64, 128, 256, 512, 1024]`. (You can use other grid sizes, which will change the formula accordingly. For more information, see this.
To store the errors in a list, you need the corresponding indices of the resolutions, which is why enumerate is used:
enumerate(resolutions) -> [(0,16), (1,32), (2,64), (3,128), (4,256), (5,512), (6,1024)]
which is unpacked by the for loop:
iteration k N
1 0 16
2 1 32
etc.
The aim of this excercise is to compare different methods for solving the differential equation given by pendulum_rhs.
The quantity by which the comparison takes place is the convergence rate. In order to determine this rate you need to solve the DE with variing resolution (of the underlying grid) and compute the error for every resolution.
The resolutions to use are given: resolutions =[16, 32, 64, ...].
So for a given method method, you iterate over the resolutions:
for k in range(len(resolutions)):
N = resolutions[k]
# calculate the result using N
result = method(..., N, ...)
#store it in an array called
error[k] = np.abs(y_exact - result)