Develop Reference

Implementation of NLMS for a one dimensional array with padasip

I am trying to implement an NLMS filter per the Padasipexample:
https://matousc89.github.io/padasip/sources/filters/nlms.html
I need to filter a one dimensional array. I adjusted the code but I'm getting an error
File "C:\Python310\lib\site-packages\padasip\filters\base_filter.py", line 194, in run
self.n = len(x[0])
TypeError: object of type 'numpy.float64' has no len()
I understand that a int doesn't have a length, however I'm not clear how do implement NLMS with a single array.
Full Code:
import numpy as np
import matplotlib.pylab as plt
import padasip as pa
# creation of data
N = 10000
x = np.random.normal(0, 1, N) # input matrix
v = np.random.normal(0, 0.1, N) # noise
d = x + v
# identification
TAP=5000
f = pa.filters.FilterNLMS(n=TAP, mu=0.1, w="random")
y, e, w = f.run(d, x)
# show results
plt.figure(figsize=(15,9))
plt.subplot(211);plt.title("Adaptation");plt.xlabel("samples - k")
plt.plot(d,"b", label="d - target")
plt.plot(y,"g", label="y - output");plt.legend()
plt.subplot(212);plt.title("Filter error");plt.xlabel("samples - k")
plt.plot(10*np.log10(e**2),"r", label="e - error [dB]");plt.legend()
plt.tight_layout()
plt.show()

Plotting PMF jupyter notebook : ValueError: shape mismatch: objects cannot be broadcast to a single shape

Currently working on a task which requires me to plot a probability mass function to a graph. The mass function i have is to do with a biased coin being tossed three times :
P(H) = 0.75
P(T) = 0.25
X = 0,1,2,3
F(0) = P(X=0) = P(t,t,t) = 0.015625
F(1) = P(X=1) = P(h,t,t) + P(t,h,t) + P(t,t,h) = ‬0.140625
F(2) = P(X=2) = P(h,h,t) + P(h,t,h) + P(t,h,h) = 0.421875
F(3) = P(X=3) = P(h,h,h) = 0.421875
When i work to plot these points using the following code
import matplotlib.pyplot as plt
prob = np.array([0,0.015625,0.140625,0.421875,0.421875])
x = np.arange(0,3)
plt.bar(x,prob, width = 0.5)
plt.xlim(0.5,3.5)
plt.show()
I am met with this error:
ValueError: shape mismatch: objects cannot be broadcast to a single shape

The shape of the x array must match the shape of the prob array. I can suggest the following:
import matplotlib.pyplot as plt
prob = np.array([0.015625,0.140625,0.421875,0.421875])
x = np.arange(4)
plt.bar(x, prob, width = 0.5)
plt.xticks(x)
plt.xlim(-0.5,3.5)
plt.show()

You have 5 elements in prob and 3 elements in x. Python cannot plot charts if the number of elements in both arrays is different. Since you have 5 elements in prob, you need to have 5 ticks on the x-axis to draw the bar chart.
Change x = np.arange(0, 3) to x = np.arange(0, 5) and plt.xlim(0.5,3.5) to plt.xlim(0.5,4.5) and you should get the plot.

How can I map a vectorized function to a numpy array without using a for loop?

So here's what I already have:
import numpy as np
import matplotlib.pyplot as plt
def monteCarloPi(n):
np.random.seed() #seed the random number generator
y = np.random.rand(n)*2 - 1 #n random samples on (-1,1)
x = np.linspace(-1,1,n) #x axis to plot against
square = np.array([x,y]) #collecting axes as a single object
mask1 = ((x**2 + y**2) < 1) #filters
hits = np.sum(mask1) #calculating approximation
ratio = hits/n
pi_approx = ratio * 4
return pi_approx
Here is what I would like to do:
x = np.arange(100,1000)
y = monteCarloPi(x)
plt.scatter(x,y)
However, when I run the above code block, I get the following error:
---------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-52-bf4dcedaa309> in <module>()
1 x = np.arange(100,1000)
----> 2 y = monteCarloPi(x)
3 plt.scatter(x,y)
<ipython-input-51-8d5b36e22d4b> in monteCarloPi(n)
1 def monteCarloPi(n):
2 np.random.seed() #seed the random number generator
----> 3 y = np.random.rand(n)*2 - 1 #n random samples on (-1,1)
4 x = np.linspace(-1,1,n) #x axis to plot against
5
mtrand.pyx in mtrand.RandomState.rand()
mtrand.pyx in mtrand.RandomState.random_sample()
mtrand.pyx in mtrand.cont0_array()
TypeError: only integer scalar arrays can be converted to a scalar index
Based on my understanding of how broadcasting works in numpy, this should work. I could just use a for loop but that gets really slow really quickly as the number of samples goes up.
halp

Here is one option, where the maximum sample size is based, then subsampling occurs if start>0 (error handling not included).
import numpy as np
import matplotlib.pyplot as plt
def monteCarloPi(n,start=0,stride=1):
np.random.seed() # seed the random number generator
y = np.random.rand(n)*2 - 1 # n random samples on (-1,1)
x = np.linspace(-1,1,n) # x axis to plot against
mask = ( x**2 + y**2 ) < 1 # masking
samples = {}
inds = arange(n)
for k in range(n-start,n+1,stride):
sub_inds = np.random.choice(inds,k,replace=False)
sub_mask = mask[sub_inds]
sub_hits = np.sum(sub_mask)
ratio = sub_hits/n
pi_approx = ratio * 4
samples[k]=pi_approx
return pi_approx
This still requires a for loop, but it's handled inside the method quickly, since you're subsampling from one large random sample. To recreate your original call (running from n=100 to n=1000 [note that I am going up to n=1000 here]):
estimates = monteCarloPi(1000,start=900)
plt.plot(estimates.keys(),estimates.values())
You could of course pass the original x=arange(100,1001), but then there would need to be error checking in the method (to make sure an array or list was passed), and then n would be equal to the last element of x (n=x[-1]), and finally, the looping would be done over the elements of x (for k in x:).

Filtering 1D numpy arrays in Python

Explanation:
I have two numpy arrays: dataX and dataY, and I am trying to filter each array to reduce the noise. The image shown below shows the actual input data (blue dots) and an example of what I want it to be like(red dots). I do not need the filtered data to be as perfect as in the example but I do want it to be as straight as possible. I have provided sample data in the code.
What I have tried:
Firstly, you can see that the data isn't 'continuous', so I first divided them into individual 'segments' ( 4 of them in this example), and then applied a filter to each 'segment'. Someone suggested that I use a Savitzky-Golay filter. The full, run-able code is below:
import scipy as sc
import scipy.signal
import numpy as np
import matplotlib.pyplot as plt
# Sample Data
ydata = np.array([1,0,1,2,1,2,1,0,1,1,2,2,0,0,1,0,1,0,1,2,7,6,8,6,8,6,6,8,6,6,8,6,6,7,6,5,5,6,6, 10,11,12,13,12,11,10,10,11,10,12,11,10,10,10,10,12,12,10,10,17,16,15,17,16, 17,16,18,19,18,17,16,16,16,16,16,15,16])
xdata = np.array([1,2,3,1,5,4,7,8,6,10,11,12,13,10,12,13,17,16,19,18,21,19,23,21,25,20,26,27,28,26,26,26,29,30,30,29,30,32,33, 1,2,3,1,5,4,7,8,6,10,11,12,13,10,12,13,17,16,19,18,21,19,23,21,25,20,26,27,28,26,26,26,29,30,30,29,30,32])
# Used a diff array to find where there is a big change in Y.
# If there's a big change in Y, then there must be a change of 'segment'.
diffy = np.diff(ydata)
# Create empty numpy arrays to append values into
filteredX = np.array([])
filteredY = np.array([])
# Chose 3 to be the value indicating the change in Y
index = np.where(diffy >3)
# Loop through the array
start = 0
for i in range (0, (index[0].size +1) ):
# Check if last segment is reached
if i == index[0].size:
print xdata[start:]
partSize = xdata[start:].size
# Window length must be an odd integer
if partSize % 2 == 0:
partSize = partSize - 1
filteredDataX = sc.signal.savgol_filter(xdata[start:], partSize, 3)
filteredDataY = sc.signal.savgol_filter(ydata[start:], partSize, 3)
filteredX = np.append(filteredX, filteredDataX)
filteredY = np.append(filteredY, filteredDataY)
else:
print xdata[start:index[0][i]]
partSize = xdata[start:index[0][i]].size
if partSize % 2 == 0:
partSize = partSize - 1
filteredDataX = sc.signal.savgol_filter(xdata[start:index[0][i]], partSize, 3)
filteredDataY = sc.signal.savgol_filter(ydata[start:index[0][i]], partSize, 3)
start = index[0][i]
filteredX = np.append(filteredX, filteredDataX)
filteredY = np.append(filteredY, filteredDataY)
# Plots
plt.plot(xdata,ydata, 'bo', label = 'Input Data')
plt.plot(filteredX, filteredY, 'ro', label = 'Filtered Data')
plt.xlabel('X')
plt.ylabel('Y')
plt.title('Result')
plt.legend()
plt.show()
This is my result:
When each point is connected, the result looks as follows.
I have played around with the order, but it seems like a third order gave the best result.
I have also tried these filters, among a few others:
scipy.signal.medfilt
scipy.ndimage.filters.uniform_filter1d
But so far none of the filters I have tried were close to what I really wanted. What is the best way to filter data such as this? Looking forward to your help.

One way to get something looking close to your ideal would be clustering + linear regression.
Note that you have to provide the number of clusters and I also cheated a bit in scaling up y before clustering.
import numpy as np
from scipy import cluster, stats
ydata = np.array([1,0,1,2,1,2,1,0,1,1,2,2,0,0,1,0,1,0,1,2,7,6,8,6,8,6,6,8,6,6,8,6,6,7,6,5,5,6,6, 10,11,12,13,12,11,10,10,11,10,12,11,10,10,10,10,12,12,10,10,17,16,15,17,16, 17,16,18,19,18,17,16,16,16,16,16,15,16])
xdata = np.array([1,2,3,1,5,4,7,8,6,10,11,12,13,10,12,13,17,16,19,18,21,19,23,21,25,20,26,27,28,26,26,26,29,30,30,29,30,32,33, 1,2,3,1,5,4,7,8,6,10,11,12,13,10,12,13,17,16,19,18,21,19,23,21,25,20,26,27,28,26,26,26,29,30,30,29,30,32])
def split_to_lines(x, y, k):
yo = np.empty_like(y, dtype=float)
# get the cluster centers and the labels for each point
centers, map_ = cluster.vq.kmeans2(np.array((x, y * 2)).T.astype(float), k)
# for each cluster, use the labels to select the points belonging to
# the cluster and do a linear regression
for i in range(k):
slope, interc, *_ = stats.linregress(x[map_==i], y[map_==i])
# use the regression parameters to construct y values on the
# best fit line
yo[map_==i] = x[map_==i] * slope + interc
return yo
import pylab
pylab.plot(xdata, ydata, 'or')
pylab.plot(xdata, split_to_lines(xdata, ydata, 4), 'ob')
pylab.show()

How to do the loop along a 3D vector with a known length by python

I have done a point filter programme in a 3D plane, but I need to do a loop along a known 3D normal vector with a known length. Many thanks for the help.
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
point = sta[10] #starting points
normal = axe[10] #normal vector
d = -point.dot(normal)
# create x,y
xx, yy = np.meshgrid(np.linspace(-3.,-2.,101), np.linspace(-11.,-10.,101))
# calculate corresponding z
z = (-normal[0] * xx - normal[1] * yy - d) * 1. /normal[2]
f=[]
for i in xrange(len(xx)-1):
for j in xrange(len(xx)-1):
if (xx[i][j]-sta[10][0])**2 + (yy[i][j]-sta[10][1])**2 + (z[i][j]-sta[10][2])**2 > float(rad[0])**2:
xx[i][j]=NaN
yy[i][j]=NaN
z[i][j]=NaN

Since you're using meshgrid and xx, yy and z have the same shape, numpy's broadcasting policy will automatically do what you need. Try this:
invalid = (xx-sta[10,0])**2 + (yy-sta[10,1])**2 + (z-sta[10,2])**2 > float(rad[0])**2
xx[invalid]=np.NaN
yy[invalid]=np.NaN
z[invalid]=np.NaN
It creates a boolean mask invalid which contains True for all entries that satisfy the condition. You can then use this mask to set the corresponding values to NaN.
Note that you can use tuples to index numpy arrays. I.e. myArray[a][b] is equivalent to myArray[a, b].
Also note that I assumed you excluded the last entries by accident. If it was on purpose that you used xrange(len(xx)-1) rather than xrange(len(xx)), it is getting a bit uglier and you have to do it like this:
invalid = (xx[:-1,:-1]-sta[10,0])**2 + (yy[:-1,:-1]-sta[10,1])**2 + (z[:-1,:-1]-sta[10,2])**2 > float(rad[0])**2
xx[:-1,:-1][invalid]=np.NaN
yy[:-1,:-1][invalid]=np.NaN
z[:-1,:-1][invalid]=np.NaN