I have seen similar questions but none that need the format of the output array of shape (numpoints, dim)
Here is a simple example of what I have for dim=2
import numpy as np
bounds = [0.5, 0.5]
n = [10,10]
dim = 2
x = np.linspace(-bounds[0], bounds[0], n[0])
y = np.linspace(-bounds[1], bounds[1], n[1])
X, Y = np.meshgrid(x, y)
s = X.shape
data = np.zeros((n[0]*n[1],dim))
# convert mesh into point vector for which the model can be evaluated
c = 0
for i in range(s[0]):
for j in range(s[1]):
data[c,0] = X[i,j]
data[c,1] = Y[i,j]
c = c+1;
plt.scatter(data[:,0], data[:,1])
Is there a faster/better way of doing this so that the data are arranged in this way? I want a general method that could work for any dim.
Edit: Suggested answer does not work.
Yeah, that can be vectorized with
axis_coords = np.meshgrid(x, y, indexing='xy')
data = np.hstack([c.reshape(-1, 1) for c in axis_coords])
c.reshape(-1, 1) just reshapes c from (HxW to (H*W)x1) so that it can be stacked horizontally.
Note - if you're looking to generalize to more dims you probably want to switch to indexing='ij' so it's arranged by (row, column, dim2, dim3, ...) rather than (column, row, dim2, dim3, ...) since in numpy rows are considered the 0'th dimension and columns the 1st.
I managed to solve my problem with this function that is general enough for any dim:
def get_grid_of_points(n, *args):
ls = [np.linspace(-i,i,n) for i in args]
mesh_ls = np.meshgrid(*ls)
all_mesh = [np.reshape(x, [-1]) for x in mesh_ls]
grid_points = np.stack(all_mesh, axis=1)
return grid_points
get_grid_of_points(10, 0.5, 0.5)
Related
I have data array, with shape 100x100. I want to divide it into 5x5 blocks, and each block has 20x20 grids. The value of each block I want is the sum of all values in it.
Is there a more elegant way to accomplish it?
x = np.arange(100)
y = np.arange(100)
X, Y = np.meshgrid(x, y)
Z = np.cos(X)*np.sin(Y)
Z_new = np.zeros((5, 5))
for i in range(5):
for j in range(5):
Z_new[i, j] = np.sum(Z[i*20:20+i*20, j*20:20+j*20])
This is based on index, how if based on x?
x = np.linspace(0, 1, 100)
y = np.linspace(0, 1, 100)
X, Y = np.meshgrid(x, y)
Z = np.cos(X)*np.sin(Y)
x_new = np.linspace(0, 1, 15)
y_new = np.linspace(0, 1, 15)
Z_new?
Simply reshape splitting each of those two axes into two each with shape (5,20) to form a 4D array and then sum reduce along the axes having the lengths 20, like so -
Z_new = Z.reshape(5,20,5,20).sum(axis=(1,3))
Functionally the same, but potentially faster option with np.einsum -
Z_new = np.einsum('ijkl->ik',Z.reshape(5,20,5,20))
Generic block size
Extending to a generic case -
H,W = 5,5 # block-size
m,n = Z.shape
Z_new = Z.reshape(H,m//H,W,n//W).sum(axis=(1,3))
With einsum that becomes -
Z_new = np.einsum('ijkl->ik',Z.reshape(H,m//H,W,n//W))
To compute average/mean across blocks, use mean instead of sum method.
Generic block size and reduction operation
Extending to use reduction operations that have ufuncs supporting multiple axes parameter with axis for reductions, it would be -
def blockwise_reduction(a, height, width, reduction_func=np.sum):
m,n = a.shape
a4D = a.reshape(height,m//height,width,n//width)
return reduction_func(a4D,axis=(1,3))
Thus, to solve our specific case, it would be :
blockwise_reduction(Z, height=5, width=5)
and for a block-wise average computation, it would be -
blockwise_reduction(Z, height=5, width=5, reduction_func=np.mean)
You can do following.
t = np.eye(5).repeat(20, axis=1)
Z_new = t.dot(Z).dot(t.T)
This is correct because Z_new[i, j] = t[i, k] * Z[k, l] * t[j, l]
Also this seems faster than Divakar's solution.
Such a problem is a very good candidate for a function like scipy.ndimage.measurements.sum since it allows "grouping" and "labelling" terms. You will have what you want with something like:
labels = [[20*(y//5) + x//5 for x in range(100)] for y in range(100)]
s = scipy.ndimage.measurements.sum(Z, labels, range(400))
(Not tested, but that is the idea).
I'm looking to plot the value of a function defined by a matrix over a grid of values.
Let S be an invertable 2x2 matrix and let x be a 2-dimensional vector. How can vectorize the evaluation of x#S#x over a two dimensional grid?
Here is how I currently do it. It works, but takes a beat to perform the computation since the grid is so fine.
#Initialize Matrix
S = np.zeros(shape = (2,2))
while np.linalg.matrix_rank(S)<S.shape[1]:
S = np.random.randint(-5,5+1, size = (2,2))
X,Y = [j.ravel() for j in np.meshgrid(np.linspace(-2,2,1001),np.linspace(-2,2,1001))]
Z = np.zeros_like(X)
for i,v in enumerate(zip(X,Y)):
v = np.array(v)
Z[i] = v#S#v
n = int(np.sqrt(X.size))
Z = Z.reshape(n,n)
X = X.reshape(n,n)
Y = Y.reshape(n,n)
plt.contour(X,Y,Z)
Simplest would be with stacking those X,Y into a 2-column 2D array and then using np.einsum to replace the loopy matrix-multiplications -
p = np.column_stack((X,Y)) # or np.stack((X,Y)).T
Zout = np.einsum('ij,jk,ik->i',p,S,p,optimize=True)
I've written some beginner code to calculate the co-efficients of a simple linear model using the normal equation.
# Modules
import numpy as np
# Loading data set
X, y = np.loadtxt('ex1data3.txt', delimiter=',', unpack=True)
data = np.genfromtxt('ex1data3.txt', delimiter=',')
def normalEquation(X, y):
m = int(np.size(data[:, 1]))
# This is the feature / parameter (2x2) vector that will
# contain my minimized values
theta = []
# I create a bias_vector to add to my newly created X vector
bias_vector = np.ones((m, 1))
# I need to reshape my original X(m,) vector so that I can
# manipulate it with my bias_vector; they need to share the same
# dimensions.
X = np.reshape(X, (m, 1))
# I combine these two vectors together to get a (m, 2) matrix
X = np.append(bias_vector, X, axis=1)
# Normal Equation:
# theta = inv(X^T * X) * X^T * y
# For convenience I create a new, tranposed X matrix
X_transpose = np.transpose(X)
# Calculating theta
theta = np.linalg.inv(X_transpose.dot(X))
theta = theta.dot(X_transpose)
theta = theta.dot(y)
return theta
p = normalEquation(X, y)
print(p)
Using the small data set found here:
http://www.lauradhamilton.com/tutorial-linear-regression-with-octave
I get the co-efficients: [-0.34390603; 0.2124426 ] using the above code instead of: [24.9660; 3.3058]. Could anyone help clarify where I am going wrong?
You can implement normal equation like below:
import numpy as np
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)
X_b = np.c_[np.ones((100, 1)), X] # add x0 = 1 to each instance
theta_best = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)
X_new = np.array([[0], [2]])
X_new_b = np.c_[np.ones((2, 1)), X_new] # add x0 = 1 to each instance
y_predict = X_new_b.dot(theta_best)
y_predict
This assumes X is an m by n+1 dimensional matrix where x_0 always = 1 and y is a m-dimensional vector.
import numpy as np
step1 = np.dot(X.T, X)
step2 = np.linalg.pinv(step1)
step3 = np.dot(step2, X.T)
theta = np.dot(step3, y) # if y is m x 1. If 1xm, then use y.T
Your implementation is correct. You've only swapped X and y (look closely how they define x and y), that's why you get a different result.
The call normalEquation(y, X) gives [ 24.96601443 3.30576144] as it should.
Here is the normal equation in one line:
theta = np.dot(np.linalg.inv(np.dot(X.T,X)),np.dot(X.T,Y))
I have data array, with shape 100x100. I want to divide it into 5x5 blocks, and each block has 20x20 grids. The value of each block I want is the sum of all values in it.
Is there a more elegant way to accomplish it?
x = np.arange(100)
y = np.arange(100)
X, Y = np.meshgrid(x, y)
Z = np.cos(X)*np.sin(Y)
Z_new = np.zeros((5, 5))
for i in range(5):
for j in range(5):
Z_new[i, j] = np.sum(Z[i*20:20+i*20, j*20:20+j*20])
This is based on index, how if based on x?
x = np.linspace(0, 1, 100)
y = np.linspace(0, 1, 100)
X, Y = np.meshgrid(x, y)
Z = np.cos(X)*np.sin(Y)
x_new = np.linspace(0, 1, 15)
y_new = np.linspace(0, 1, 15)
Z_new?
Simply reshape splitting each of those two axes into two each with shape (5,20) to form a 4D array and then sum reduce along the axes having the lengths 20, like so -
Z_new = Z.reshape(5,20,5,20).sum(axis=(1,3))
Functionally the same, but potentially faster option with np.einsum -
Z_new = np.einsum('ijkl->ik',Z.reshape(5,20,5,20))
Generic block size
Extending to a generic case -
H,W = 5,5 # block-size
m,n = Z.shape
Z_new = Z.reshape(H,m//H,W,n//W).sum(axis=(1,3))
With einsum that becomes -
Z_new = np.einsum('ijkl->ik',Z.reshape(H,m//H,W,n//W))
To compute average/mean across blocks, use mean instead of sum method.
Generic block size and reduction operation
Extending to use reduction operations that have ufuncs supporting multiple axes parameter with axis for reductions, it would be -
def blockwise_reduction(a, height, width, reduction_func=np.sum):
m,n = a.shape
a4D = a.reshape(height,m//height,width,n//width)
return reduction_func(a4D,axis=(1,3))
Thus, to solve our specific case, it would be :
blockwise_reduction(Z, height=5, width=5)
and for a block-wise average computation, it would be -
blockwise_reduction(Z, height=5, width=5, reduction_func=np.mean)
You can do following.
t = np.eye(5).repeat(20, axis=1)
Z_new = t.dot(Z).dot(t.T)
This is correct because Z_new[i, j] = t[i, k] * Z[k, l] * t[j, l]
Also this seems faster than Divakar's solution.
Such a problem is a very good candidate for a function like scipy.ndimage.measurements.sum since it allows "grouping" and "labelling" terms. You will have what you want with something like:
labels = [[20*(y//5) + x//5 for x in range(100)] for y in range(100)]
s = scipy.ndimage.measurements.sum(Z, labels, range(400))
(Not tested, but that is the idea).
So I have an array of values of x (in increasing order) and the corresponding y values. Numpy's interp function takes in the X value, and the x and y arrays. How do I get the value of X for a given value of Y?? Eg if y = 0, x = ?.
Cheers!
code:
j = (((1840/(2*pi))**0.5)*exp(phi)) - 1.0 #y axis, phi is a 1-D array
t = linspace(0, 40, 100) #x axis
a = interp(<x-value>,t,j) # this gives me the corresponding y value!
So what should I do to get the x value for a given y value!!
y_interp = np.interp(x_interp, x, y) yields an interpolation of the function y_interp = f(x_interp) based on a previous interpolation y = f(x), where x.size = y.size, x_interp.size = y_interp.size.
If you want x as a function of y, you have to construct the inverse function. As #NPE indicated, you have to make sure that x and y are always increasing. An easy way to check this is to use
np.all(np.diff(x) > 0)
np.all(np.diff(y) > 0)
Now finding the inverse function is actually very simple: you have to reverse the roles of x and y (since we deal with interpolations).
With my notations: x_value = np.interp(y_value, x, y).
With your notations: x_value = interp(y_value, j, t)
When using Numpy make sure you're using a Numpy array: np.array([])
#x data array
x_data = np.array([1,2,3,4])
#y data array
y_data = np.array([1,3,2,1])
#the known max y value is 3
y_max = 3
# sort the arrays
order = y_data.argsort()
y_data = y_data[order]
x_data = x_data[order]
# call the interpolation function but reverse the array datasets to find the corresponding x value
x = np.interp(y_max, y_data, x_data, left=None, right=None, period=None)
For this example the result is x = 2 when max y = 3. The maximum is (2,3).
Furthermore you can append data to a numpy array: x_data = np.append(x_data, appendedValue)