The sample code is as below.
I want to get dataNew(h, w, length) according to data(h, w, c) and ind(h, w). Here length < c, it means dataNew is sliced from data.
Here, length and ind[i, j] is made sure to suit the c value.
I have realize it through for loops, but I wnat the python way. Please help, thanks!
import numpy as np
h, w, c = 3, 4, 5
data = np.arange(60).reshape((h, w, c))
print(data)
length = 3
ind = np.random.randint(0, 3, 12).reshape(h, w)
print(ind)
dataNew = np.empty((h, w, length), np.int16)
for i in range(h):
for j in range(w):
st = ind[i, j]
dataNew[i, j] = data[i, j][st : st + length]
print(dataNew)
We can leverage np.lib.stride_tricks.as_strided based scikit-image's view_as_windows to get sliding windows. More info on use of as_strided based view_as_windows.
from skimage.util.shape import view_as_windows
# Get all sliding windows along the last axis
w = view_as_windows(data,(1,1,length))[...,0,0,:]
# Index into windows with start indices and slice out singleton dims
out = np.take_along_axis(w,ind[...,None,None],axis=-1)[...,0]
Last step is basically using advanced-indexing into the windows with those start indices. This could be made a bit simpler and might be easier to understand. So, alternatively, we could do -
m,n = ind.shape
I,J = np.ogrid[:m,:n]
out = w[I,J,ind]
One way would be creating an indexing array using broadcasting and use np.take_along_axis to index the array:
ix = ind[...,None] + np.arange(length)
np.take_along_axis(data, ix, -1)
Related
I am attempting to write a program which constructs a matrix and performs a singular value decomposition on it. I am evaluating the function ax^2 +bx + 1 on a grid. I then make a uniform meshgrid of a and b. The rows of the matrix correspond to different quadratic coefficients, while each column corresponds to a grid point at which the function is evaluated.
The matlab code is here:
% Collect data
x = linspace(-1,1,100);
[a,b] = meshgrid(0:0.1:1,0:0.1:1);
D=zeros(numel(x),numel(a));
sz = size(D)
% Build “Dose” matrix
for i=1:numel(a)
D(:,i) = a(i)*x.^2+b(i)*x+1;
end
% Do the SVD:
[U,S,V]=svd(D,'econ');
D_reconstructed = U*S*V';
plot(diag(S))
scatter3(a(:),b(:),V(:,1))
This is my attempt at a solution:
import numpy as np
import matplotlib.pyplot as plt
x = np.linspace(-1, 1, 100)
def f(x, a, b):
return a*x*x + b*x + 1
a, b = np.mgrid[0:1:0.1,0:1:0.1]
#a = b = np.arange(0,1,0.01)
D = np.zeros((x.size, a.size))
for i in range(a.size):
D[i] = a[i]*x*x +b[i]*x +1
U, S, V = np.linalg.svd(D)
plt.plot(np.diag(S))
fig = plt.figure()
ax = plt.axes(projection="3d")
ax.scatter(a, b, V[0])
but I always get broadcasting errors which I am not sure how to fix.
Firstly, in MATLAB you're assigning to D(:,i), but in python you're assigning to D[i]. The latter is equivalent to D[i, ...] which is in your case D[i, :]. Instead you seem to need D[:, i].
Secondly, in MATLAB using a linear index into a 2d array (namely a and b) will give you flattened views. If you do that with numpy you get slices of an array instead, just as I mentioned with D[i].
You can do away with the loop with broadcasting and getting your desired 2d array by .ravelling (or reshaping) your a and b arrays:
x = np.linspace(-1, 1, 100)[:, None] # inject trailing singleton for broadcasting
a, b = np.mgrid[0:1:0.1, 0:1:0.1]
D = a.ravel() * x**2 + b.ravel() * x + 1
The way this works is that x has shape (100, 1) after we inject a trailing singleton (in MATLAB trailing singletons are implied, in numpy leading ones), and both a.ravel() and b.ravel() have shape (10*10,) which is compatible with (1, 10*10), making broadcasting possible into shape (100, 10*10). You could also replace the calls to ravel with
a, b = np.mgrid[...].reshape(2, -1)
which is a trick I sometimes use, but this is harder to read if you're unfamiliar with the pattern.
Side note: it's better to use example data where dimensions end up being of different size so that you notice if something ends up being transposed.
need to read an image as an array and for each pixel select 7*7 neighbor pixels then reshape it and put as a first row of training set:
import numpy as np
from scipy import misc
face1=misc.imread('face1.jpg')
face1 dimensions are (288, 352, 3) , need to find 7*7 neighbor pixels for every pixel , so 49*3 color then reshape it as a (1,147) array and stack it into an array for all pixels , i took the following approach:
X_training=np.zeros([1,147] ,dtype=np.uint8)
for i in range(3, face1.shape[0]-3):
for j in range(3, face1.shape[1]-3):
block=face1[i-3:i+4,j-3:j+4]
pxl=np.reshape(block,(1,147))
X_training=np.vstack((pxl,X_training))
resulting X_training shape is (97572, 147)
and as last row contains all zeros then:
a = len(X_training)-1
X_training = X_training[:a]
above code works well for one picture but with Wall time: 5min 19s i have 2000 images, so it will take ages to do it for all the images. I am looking for a faster way to iterate over every pixel and do the above task.
Edit:
this is what i mean by neighbor pixels , for every pixel face1[i-3 : i+4 ,j-3:j+4]
An efficient way is to use stride_tricks to create a 2d rolling window over the image, then flatten it out:
import numpy as np
face1 = np.arange(288*352*3).reshape(288, 352, 3) # toy data
n = 7 # neighborhood size
h, w, d = face1.shape
s = face1.strides
tmp = np.lib.stride_tricks.as_strided(face1, strides=s[:2] + s,
shape=(h - n + 1, w - n + 1, n, n, d))
X_training = tmp.reshape(-1, n**2 * d)
X_training = X_training[::-1] # to get the rows into same order as in the question
tmp is a 5D view into the image, where tmp[x, y, :, :, c] is equivalent to the neigborhood face1[x:x+n, y:y+n, c] in color channel c.
The following is < 1s on my laptop:
import scipy as sp
im = sp.rand(300, 300, 3)
size = 3
ij = sp.meshgrid(range(size, im.shape[0]-size), range(size, im.shape[1]-size))
i = ij[0].T.flatten()
j = ij[1].T.flatten()
N = len(i)
L = (2*size + 1)**2
X_training = sp.empty(shape=[N, 3*L])
for pixel in range(N):
si = (slice(i[pixel]-size, i[pixel]+size+1))
sj = (slice(j[pixel]-size, j[pixel]+size+1))
X_training[pixel, :] = im[si, sj, :].flatten()
X_training = X_training[-1::-1, :]
I'm always a bit sad when I can't think of one-line vectorized version, but at least it's faster for you.
Using scikit-image:
import numpy as np
from skimage import util
image = np.random.random((288, 352, 3))
windows = util.view_as_windows(image, (7, 7, 3))
out = windows.reshape(-1, 7 * 7 * 3)
Through some help here, I have come up with a function that seems to apply the sobel derivative to an image in the X direction F(x,y) = F(x+1,y) - F(x,y)
I can't use any OpenCV functions and I need the 2D output array to be 1 column shorter than the 2D input array.
However, I can't figure out why this is still not returning an output array that is 1 column shorter. Can someone spot the issue and/or tell me if this is on the right track? Thanks much.
output = input[:-1,:]
r,c = input.shape
for i in range(0, r - 1):
output[i] = np.abs(input[i+1] - input[i])
return output
You can use numpy's diff() function. See its doc here.
And a code snippet to illustrate its use:
import numpy as np
a = np.ones([5, 4])
b = np.diff(a, axis=1)
the result b is a (5, 3) array full of zeros.
If you want to keep your loop, you can do:
r,c = input.shape
output = np.zeros([r-1, c])
for i in range(0, r - 1):
output[i] = np.abs(input[i+1] - input[i])
print output
Edit: 'mathematical' x corresponds to the second axis (vertical), and y to the first axis (horizontal). So to obtain F(x+1, y) - F(x, y), you must do:
r,c = input.shape
output = np.zeros([r, c -1])
for j in range(0, c - 1):
output[:, j] = np.abs(input[:, j+1] - input[:, j])
print output
I'm not sure if you mean to create the output array like that. You're creating output as a reference to a subarray of input, so if you modify input you also modify output or vice versa. See:
Numpy array assignment with copy
That said, running the code snippet you provided with input = np.ones([5,5]), and printing output.shape after the return, I get (4,5), which seems to be what you want?
Is there an easy/build-in way to get the element-wise maximum of two (or ideally more) sparse matrices? I.e. a sparse equivalent of np.maximum.
This did the trick:
def maximum (A, B):
BisBigger = A-B
BisBigger.data = np.where(BisBigger.data < 0, 1, 0)
return A - A.multiply(BisBigger) + B.multiply(BisBigger)
No, there's no built-in way to do this in scipy.sparse. The easy solution is
np.maximum(X.A, Y.A)
but this is obviously going to be very memory-intensive when the matrices have large dimensions and it might crash your machine. A memory-efficient (but by no means fast) solution is
# convert to COO, if necessary
X = X.tocoo()
Y = Y.tocoo()
Xdict = dict(((i, j), v) for i, j, v in zip(X.row, X.col, X.data))
Ydict = dict(((i, j), v) for i, j, v in zip(Y.row, Y.col, Y.data))
keys = list(set(Xdict.iterkeys()).union(Ydict.iterkeys()))
XmaxY = [max(Xdict.get((i, j), 0), Ydict.get((i, j), 0)) for i, j in keys]
XmaxY = coo_matrix((XmaxY, zip(*keys)))
Note that this uses pure Python instead of vectorized idioms. You can try shaving some of the running time off by vectorizing parts of it.
Here's another memory-efficient solution that should be a bit quicker than larsmans'. It's based on finding the set of unique indices for the nonzero elements in the two arrays using code from Jaime's excellent answer here.
import numpy as np
from scipy import sparse
def sparsemax(X, Y):
# the indices of all non-zero elements in both arrays
idx = np.hstack((X.nonzero(), Y.nonzero()))
# find the set of unique non-zero indices
idx = tuple(unique_rows(idx.T).T)
# take the element-wise max over only these indices
X[idx] = np.maximum(X[idx].A, Y[idx].A)
return X
def unique_rows(a):
void_type = np.dtype((np.void, a.dtype.itemsize * a.shape[1]))
b = np.ascontiguousarray(a).view(void_type)
idx = np.unique(b, return_index=True)[1]
return a[idx]
Testing:
def setup(n=1000, fmt='csr'):
return sparse.rand(n, n, format=fmt), sparse.rand(n, n, format=fmt)
X, Y = setup()
Z = sparsemax(X, Y)
print np.all(Z.A == np.maximum(X.A, Y.A))
# True
%%timeit X, Y = setup()
sparsemax(X, Y)
# 100 loops, best of 3: 4.92 ms per loop
The latest scipy (13.0) defines element-wise booleans for sparse matricies. So:
BisBigger = B>A
A - A.multiply(BisBigger) + B.multiply(BisBigger)
np.maximum does not (yet) work because it uses np.where, which is still trying to get the truth value of an array.
Curiously B>A returns a boolean dtype, while B>=A is float64.
Here is a function that returns a sparse matrix that is element-wise maximum of two sparse matrices. It implements the answer by hpaulj:
def sparse_max(A, B):
"""
Return the element-wise maximum of sparse matrices `A` and `B`.
"""
AgtB = (A > B).astype(int)
M = AgtB.multiply(A - B) + B
return M
Testing:
A = sparse.csr_matrix(np.random.randint(-9,10, 25).reshape((5,5)))
B = sparse.csr_matrix(np.random.randint(-9,10, 25).reshape((5,5)))
M = sparse_max(A, B)
M2 = sparse_max(B, A)
# Test symmetry:
print((M.A == M2.A).all())
# Test that M is larger or equal to A and B, element-wise:
print((M.A >= A.A).all())
print((M.A >= B.A).all())
from scipy import sparse
from numpy import array
I = array([0,3,1,0])
J = array([0,3,1,2])
V = array([4,5,7,9])
A = sparse.coo_matrix((V,(I,J)),shape=(4,4))
A.data.max()
9
If you haven't already, you should try out ipython, you could have saved your self time my making your spare matrix A then simply typing A. then tab, this will print a list of methods that you can call on A. From this you would see A.data gives you the non-zero entries as an array and hence you just want the maximum of this.
I am trying to implement a 3D DFT but I am running into some trouble. What I believe I should do is to just do 3 consecutive 1D DFTs, one in each direction. Assuming that the 1D DFT is correct, can you see what is wrong with this code:
def dft3d(self, real3d, img3d, nx, ny, nz, dir):
#Transform depth
for i in range(nx):
for j in range(ny):
real = numpy.zeros(nz)
img = numpy.zeros(nz)
for k in range(nz):
real[k] = real3d[i][j][k]
img[k] = img3d[i][j][k]
self.dft(real, img, nz, 1) #This was indented too much. It should work now.
for k in range(nz):
real3d[i][j][k] = real[k]
img3d[i][j][k] = img[k]
#Transform cols
for k in range(nz):
for i in range(nx):
real = numpy.zeros(ny)
img = numpy.zeros(ny)
for j in range(ny):
real[j] = real3d[i][j][k]
img[j] = img3d[i][j][k]
self.dft(real, img, ny, 1)
for j in range(ny):
real3d[i][j][k] = real[j]
img3d[i][j][k] = img[j]
#Transform rows
for j in range(ny):
for k in range(nz):
real = numpy.zeros(nx)
img = numpy.zeros(nx)
for i in range(nx):
real[i] = real3d[i][j][k]
img[i] = img3d[i][j][k]
self.dft(real, img, nx, 1)
for i in range(nx):
real3d[i][j][k] = real[i]
img3d[i][j][k] = img[i]
I know there are built in versions of this in python, but I can't use those. I'm just testing my algorithm in python so I can compare results of my algorithm and the built in ones. As far as I could tell it worked fine for both 1D and 2D transforms, but once I expanded it to 3D the results no longer match. Does anyone know what is wrong?
The first instance of self.dft is indented too far.
Other than that, I see nothing wrong from the code provided.
As a side note, if you are using numpy as your code suggests, you can simplify your code significantly even without resorting to the built-in DFT/FFT.
For example, you can index a 3D numpy array like data3D[i, j, k]. You can slice by doing data3D[:, j, k], data3D[i, :, k], data3D[:, :, k], etc., instead of assigning individual elements one at a time within a for loop.