Develop Reference

Spatial Encoding (sum of elements within a specific region in a numpy array) [duplicate]

I have data array, with shape 100x100. I want to divide it into 5x5 blocks, and each block has 20x20 grids. The value of each block I want is the sum of all values in it.
Is there a more elegant way to accomplish it?
x = np.arange(100)
y = np.arange(100)
X, Y = np.meshgrid(x, y)
Z = np.cos(X)*np.sin(Y)
Z_new = np.zeros((5, 5))
for i in range(5):
for j in range(5):
Z_new[i, j] = np.sum(Z[i*20:20+i*20, j*20:20+j*20])
This is based on index, how if based on x?
x = np.linspace(0, 1, 100)
y = np.linspace(0, 1, 100)
X, Y = np.meshgrid(x, y)
Z = np.cos(X)*np.sin(Y)
x_new = np.linspace(0, 1, 15)
y_new = np.linspace(0, 1, 15)
Z_new?

Simply reshape splitting each of those two axes into two each with shape (5,20) to form a 4D array and then sum reduce along the axes having the lengths 20, like so -
Z_new = Z.reshape(5,20,5,20).sum(axis=(1,3))
Functionally the same, but potentially faster option with np.einsum -
Z_new = np.einsum('ijkl->ik',Z.reshape(5,20,5,20))
Generic block size
Extending to a generic case -
H,W = 5,5 # block-size
m,n = Z.shape
Z_new = Z.reshape(H,m//H,W,n//W).sum(axis=(1,3))
With einsum that becomes -
Z_new = np.einsum('ijkl->ik',Z.reshape(H,m//H,W,n//W))
To compute average/mean across blocks, use mean instead of sum method.
Generic block size and reduction operation
Extending to use reduction operations that have ufuncs supporting multiple axes parameter with axis for reductions, it would be -
def blockwise_reduction(a, height, width, reduction_func=np.sum):
m,n = a.shape
a4D = a.reshape(height,m//height,width,n//width)
return reduction_func(a4D,axis=(1,3))
Thus, to solve our specific case, it would be :
blockwise_reduction(Z, height=5, width=5)
and for a block-wise average computation, it would be -
blockwise_reduction(Z, height=5, width=5, reduction_func=np.mean)

You can do following.
t = np.eye(5).repeat(20, axis=1)
Z_new = t.dot(Z).dot(t.T)
This is correct because Z_new[i, j] = t[i, k] * Z[k, l] * t[j, l]
Also this seems faster than Divakar's solution.

Such a problem is a very good candidate for a function like scipy.ndimage.measurements.sum since it allows "grouping" and "labelling" terms. You will have what you want with something like:
labels = [[20*(y//5) + x//5 for x in range(100)] for y in range(100)]
s = scipy.ndimage.measurements.sum(Z, labels, range(400))
(Not tested, but that is the idea).

How to create a multi-dimensional grid in python

I have seen similar questions but none that need the format of the output array of shape (numpoints, dim)
Here is a simple example of what I have for dim=2
import numpy as np
bounds = [0.5, 0.5]
n = [10,10]
dim = 2
x = np.linspace(-bounds[0], bounds[0], n[0])
y = np.linspace(-bounds[1], bounds[1], n[1])
X, Y = np.meshgrid(x, y)
s = X.shape
data = np.zeros((n[0]*n[1],dim))
# convert mesh into point vector for which the model can be evaluated
c = 0
for i in range(s[0]):
for j in range(s[1]):
data[c,0] = X[i,j]
data[c,1] = Y[i,j]
c = c+1;
plt.scatter(data[:,0], data[:,1])
Is there a faster/better way of doing this so that the data are arranged in this way? I want a general method that could work for any dim.
Edit: Suggested answer does not work.

Yeah, that can be vectorized with
axis_coords = np.meshgrid(x, y, indexing='xy')
data = np.hstack([c.reshape(-1, 1) for c in axis_coords])
c.reshape(-1, 1) just reshapes c from (HxW to (H*W)x1) so that it can be stacked horizontally.
Note - if you're looking to generalize to more dims you probably want to switch to indexing='ij' so it's arranged by (row, column, dim2, dim3, ...) rather than (column, row, dim2, dim3, ...) since in numpy rows are considered the 0'th dimension and columns the 1st.

I managed to solve my problem with this function that is general enough for any dim:
def get_grid_of_points(n, *args):
ls = [np.linspace(-i,i,n) for i in args]
mesh_ls = np.meshgrid(*ls)
all_mesh = [np.reshape(x, [-1]) for x in mesh_ls]
grid_points = np.stack(all_mesh, axis=1)
return grid_points
get_grid_of_points(10, 0.5, 0.5)

Fast way to calculate min distance between two numpy arrays of 3D points

I would like to know if there is a fast way to calculate Euclidian distance between all points of a 3D numpy array (A [N,3]) to all points of a second 3D numpy array (B [M,3]).
I should then get an array C which would be [N, M] with all distances from points of array A to points of array B to then use np.min() along specified axis to get all minimum distances from points of set A to points of set B.
This is the way I have done the implementation so far :
distances = np.repeat(9999, len(A))
for i, point in enumerate(A):
min_distance = np.min(np.sqrt(np.sum(np.square(point - B), axis=1)))
distances[i] = min_distance
Is there any way to get rid of the for loop...?
Thanks in advance :)

If the scipy method doesn't work or if you do have other reasons, here is a numpy way-
import numpy as np
x = np.random.random((200, 3))
y = np.random.random((100,3))
x = x.reshape((-1, 1, 3)) # [200x1x3]
y = np.expand_dims(y, axis=0) # [1x100x3]
y = y.repeat(x.shape[0], axis=0) # [200x100x3]
distance = np.linalg.norm(y-x, axis=2) # Difference is [200x100x3] and norm results in [200x100]

import numpy as np
# arrays with xyz coordinates of all points
a = np.asarray([[xa1,ya1,za1],...,[xan,yan,zan]])
b = np.asarray([[xb1,yb1,zb1],...,[xbn,ybn,zbn]])
# reshaping to be able to calculate the distance matrix
a_reshaped = a.reshape(a.shape[0], 1, 3)
b_reshaped = b.reshape(1, b.shape[0], 3)
"""calculation of all distances between all points - creates a
len(a) x len(b) matrix"""
distance = np.sqrt(np.sum((a_reshaped - b_reshaped)**2, axis=2))

How to smooth z values and reduce its variation in python?

I have a 3D point cloud file containing x, y,z values. My ultimate goal is to cluster the point cloud. But the point cloud had some noise and I removed the noise using PCA. Now I want to smoothen the z values and cluster it based on the x and y.
I know the smoothing can be done using "gridfit" in matlab, (gridfit.m) but I am looking for a solution that can be done in Python.
https://au.mathworks.com/matlabcentral/fileexchange/8998-surface-fitting-using-gridfit
Is there a method that works similar to gridfit for reducing the variation of z?

this function is open source, it take some time to convert it but it is not impossible.
You just need theses functions:
def histc(X, bins):
map_to_bins = numpy.digitize(X,bins)
r = numpy.zeros(bins.shape)
for i in map_to_bins:
r[i-1] += 1
return r, map_to_bins
def sparse(i, j, v, m, n):
"""
Create and compressing a matrix that have many zeros
Parameters:
i: 1-D array representing the index 1 values
Size n1
j: 1-D array representing the index 2 values
Size n1
v: 1-D array representing the values
Size n1
m: integer representing x size of the matrix
n: integer representing y size of the matrix
Returns:
s: 2-D array
Matrix full of zeros excepting values v at indexes i, j
"""
return scipy.sparse.csr_matrix((v, (i, j)), shape=(m, n))
def repmat(A, reps):
"""
Repeting A list according to list dimension
Parameters:
A: array_like
The input array.
reps: array_like
The number of repetitions of A along each axis.
Returns:
c: ndarray
The tiled output array.
"""
return numpy.tile(A, numpy.array(reps))
def meshgrid(a, b):
"""
Return coordinate matrices from coordinate vectors.
Make N-D coordinate arrays for vectorized evaluations
of N-D scalar/vector fields over N-D grids,
given one-dimensional coordinate arrays a, b
Parameters:
a: array_like
1-D arrays representing the x coordinates of a grid.
b: array_like
1-D arrays representing the y coordinates of a grid.
Returns:
c: 2-D arrays representing the x * ycoordinates of grid.
"""
return numpy.meshgrid(a, b)
Otherwise you can use griddata and apply a blur on it by considering the result as an grayscale image but don't forget to cut off edges to avoid sharp edges:
You have the mathematical description here.
Don't forget to checkout source code here.
Here is the default parameter code in matlab to translate:
function [zgrid,xgrid,ygrid] = gridfit(x,y,z,xnodes,ynodes,varargin)
params.smoothness = 1;
params.interp = 'triangle';
params.regularizer = 'gradient';
params.solver = 'backslash';
params.maxiter = [];
params.extend = 'warning';
params.tilesize = inf;
params.overlap = 0.20;
params.mask = [];
params.autoscale = 'on';
params.xscale = 1;
params.yscale = 1;
params = check_params(params);
xmin = min(x);
xmax = max(x);
ymin = min(y);
ymax = max(y);
xnodes=xnodes(:);
ynodes=ynodes(:);
dx = diff(xnodes);
dy = diff(ynodes);
nx = length(xnodes);
ny = length(ynodes);
ngrid = nx*ny;
params.xscale = mean(dx);
params.yscale = mean(dy);
n = length(x);
[junk,indx] = histc(x,xnodes);
[junk,indy] = histc(y,ynodes);
k=(indx==nx);
indx(k)=indx(k)-1;
k=(indy==ny);
indy(k)=indy(k)-1;
ind = indy + ny*(indx-1);
tx = min(1,max(0,(x - xnodes(indx))./dx(indx)));
ty = min(1,max(0,(y - ynodes(indy))./dy(indy)));
A = sparse(
repmat((1:n)',1,4),
[ind,ind+1,ind+ny,ind+ny+1],
[(1-tx).*(1-ty), (1-tx).*ty, tx.*(1-ty), tx.*ty],
n,
ngrid
);
rhs = z;
smoothparam = params.smoothness;
xyRelativeStiffness = [1;1];
[i,j] = meshgrid(1:nx,2:(ny-1));
ind = j(:) + ny*(i(:)-1);
dy1 = dy(j(:)-1)/params.yscale;
dy2 = dy(j(:))/params.yscale;
Areg = sparse(
repmat(ind,1,3),
[ind-1,ind,ind+1],
xyRelativeStiffness(2)*[-2./(dy1.*(dy1+dy2)),
2./(dy1.*dy2),
-2./(dy2.*(dy1+dy2))],
ngrid,
ngrid
);
[i,j] = meshgrid(1:nx,2:(ny-1));
ind = j(:) + ny*(i(:)-1);
dy1 = dy(j(:)-1)/params.yscale;
dy2 = dy(j(:))/params.yscale;
Areg = sparse(repmat(ind,1,3),[ind-1,ind,ind+1], ...
xyRelativeStiffness(2)*[-2./(dy1.*(dy1+dy2)), ...
2./(dy1.*dy2), -2./(dy2.*(dy1+dy2))],ngrid,ngrid);
[i,j] = meshgrid(2:(nx-1),1:ny);
ind = j(:) + ny*(i(:)-1);
dx1 = dx(i(:)-1)/params.xscale;
dx2 = dx(i(:))/params.xscale;
Areg = [Areg;sparse(repmat(ind,1,3),[ind-ny,ind,ind+ny], ...
xyRelativeStiffness(1)*[-2./(dx1.*(dx1+dx2)), ...
2./(dx1.*dx2), -2./(dx2.*(dx1+dx2))],ngrid,ngrid)];
nreg = size(Areg,1);
NA = norm(A,1);
NR = norm(Areg,1);
A = [A;Areg*(smoothparam*NA/NR)];
rhs = [rhs;zeros(nreg,1)];
zgrid = reshape((A'*A)\(A'*rhs),ny,nx);

Create a matrix using values from a tuple with numpy

I'm trying to create a matrix with values based on x,y values I have stored in a tuple. I use a loop to iterate over the tuple and perform a simple calculation on the data:
import numpy as np
# Trying to fit quadratic equation to the measured dots
N = 6
num_of_params = 3
# x values
x = (1,4,3,5,2,6)
# y values
y = (3.96, 24.96,14.15,39.8,7.07,59.4)
# X is a matrix N * 3 with the x values to the power of {0,1,2}
X = np.zeros((N,3))
Y = np.zeros((N,1))
print X,"\n\n",Y
for i in range(len(x)):
for p in range(num_of_params):
X[i][p] = x[i]**(num_of_params - p - 1)
Y[i] = y[i]
print "\n\n"
print X,"\n\n",Y
Is this can be achieved in an easier way? I'm looking for some way to init the matrix like X = np.zeros((N,3), read_values_from = x)
Is it possible? Is there another simple way?
Python 2.7

Extend array version of x to 2D with a singleton dim (dim with length=1) along the second one using np.newaxis/None. This lets us leverage NumPy broadcasting to get the 2D output in a vectorized manner. Similar philosophy for y.
Hence, the implementation would be -
X = np.asarray(x)[:,None]**(num_of_params - np.arange(num_of_params) - 1)
Y = np.asarray(y)[:,None]
Or use the built-in outer method for np.power to get X that takes care of the array conversion under the hoods -
X = np.power.outer(x, num_of_params - np.arange(num_of_params) - 1)
Alternatively, for Y, use np.expand_dims -
Y = np.expand_dims(y,1)