I have a 1d ndarray A of shape (n,) and a 2d ndarray E of shape (n,m). I am trying to preform the following calculation (the circle-dot denotes element wise multiplication):
I have written it using with a for loop, but this block of code is called thousands of times, and I was hoping there was a way to accomplish this with broadcasting or numpy functions. The following is my for loop solution I'm trying to rewrite:
def fun(E, A):
X = E * A[:,np.newaxis]
R = np.zeros(E.shape[-1])
for ii in xrange(len(E)-1):
for jj in xrange(ii+1, len(E)):
R += X[ii] * X[jj]
return R
Any help would be appreciated.
Current approach, but still not working:
def fun1(E, A):
X = E * A[:,np.newaxis]
R = np.zeros(E.shape[-1])
for ii in xrange(len(E)-1):
for jj in xrange(ii+1, len(E)):
R += X[ii] * X[jj]
return R
def fun2(E, A):
n = E.shape[0]
m = E.shape[1]
A_ = np.triu(A[1:] * A[:-1].reshape(-1,1))
E_ = E[1:] * E[:-1]
R = np.sum((A_.reshape(n-1, 1, n-1) * E_.T).transpose(0,2,1).reshape(n-1*n-1,m), axis=0)
return R
A = np.arange(4,9)
E = np.arange(20).reshape((5,4))
print fun1(E,A)
print fun2(E,A)
Now, this should work:
def fun3(E,A):
n,m = E.shape
n_ = n - 1
X = E * A[:, np.newaxis]
a = (X[:-1].reshape(n_, 1, m) * X[1:])
b = np.tril(np.ones((m, n_, n_))).T
R = np.sum((a*b).reshape(n_*n_, m), axis=0)
return R
Last function was only based on the given formula. This is instead based on fun and tested with your added test case.
Hope this works for you!
Related
I have some constraints of the form of
A_{i,j,k} = r_{i,j}B_{i,j,k}
A is a nxmxp matrix, as is B. r is an nxm matrix.
I would like to vectorize this in Python somehow, as efficiently as possible. Right now, I am making r into nxmxp matrix by saying r_{i,j,k} = r_{i,j} for all 1 <= k <= p. Then I call np.multiply on r and B. This seems inefficient. Any ideas welcome, thanks.
def ndHadamardProduct(r, n, m, p): #r is a n x m matrix, p is an int
rnew = np.zeros(n, m, p)
B = np.zeros(n, m, p)
for i in range(n):
for j in range(m):
for k in range(p):
r[i, j, k] = r[i, j]
B[i, j, k] = random.uniform(0, 1)
return np.multiply(r, B)
Add an extra dimension with np.newaxis and then broadcasting takes care of the repetition for you.
import numpy as np
r = np.random.random((3,4))
b = np.random.random((3,4,5))
a = r[:,:,np.newaxis] * b
I wrote the following code to do multiplication of matrix permutations and I was wondering if it can be written in a numpy style, such that I can get rid of the two for loops:
Z = np.empty([new_d, X.shape[1]])
Z = np.ndarray(shape=(new_d, X.shape[1]))
Z = np.concatenate((X, X**2))
res = []
for i in range(0, d):
for j in range(i+1, d):
res.append(np.array(X.T[:,i]* X.T[:,j]))
Z = np.concatenate((Z, res))
while: X shape is (7, 1000), d = 7, new_d=35
any suggestion ?
Approach #1
We could use np.triu_indices to get those pair-wise permutation-indices and then simply perform elementwise multiplicatons of row-indexed arrays -
r,c = np.triu_indices(d,1)
res = X[r]*X[c]
Approach #2
For memory efficiency and hence performance especially on large arrays, we are better off slicing the input array and run a single loop with each iteration working on chunks of data, like so -
n = d-1
idx = np.concatenate(( [0], np.arange(n,0,-1).cumsum() ))
start, stop = idx[:-1], idx[1:]
L = n*(n+1)//2
res_out = np.empty((L,X.shape[1]), dtype=X.dtype)
for i,(s0,s1) in enumerate(zip(start,stop)):
res_out[s0:s1] = X[i] * X[i+1:]
To get Z directly and thus avoid all those concatenations, we could modify the earlier posted approach, like so -
n = d-1
N = len(X)
idx = 2*N + np.concatenate(( [0], np.arange(n,0,-1).cumsum() ))
start, stop = idx[:-1], idx[1:]
L = n*(n+1)//2
Z_out = np.empty((2*N + L,X.shape[1]), dtype=X.dtype)
Z_out[:N] = X
Z_out[N:2*N] = X**2
for i,(s0,s1) in enumerate(zip(start,stop)):
Z_out[s0:s1] = X[i] * X[i+1:]
I implemented Conjugate Gradient in python by looking into the Wikipedia reference - https://en.wikipedia.org/wiki/Conjugate_gradient_method
The implementation should solve for
ax = b
my application inputs goes as below,
a = <400x400 sparse matrix of type '<class 'numpy.float64'>'
with 1920 stored elements in Compressed Sparse Row format>
b = vector of shape (400, ) and dtype = float64
x = vector of random numbers of shape (400, )
Here is my implementation -
def ConjGrad(a, b, x):
r = (b - np.dot(np.array(a), x));
p = r;
rsold = np.dot(r.T, r);
for i in range(len(b)):
a_p = np.dot(a, p);
alpha = rsold / np.dot(p.T, a_p);
x = x + (alpha * p);
r = r - (alpha * a_p);
rsnew = np.dot(r.T, r);
if (np.sqrt(rsnew) < (10 ** -5)):
break;
p = r + ((rsnew / rsold) * p);
rsold = rsnew;
return p
When i call the above CG function, i get an error within the function for the line -
r = (b - np.dot(np.array(a), x));
The error goes like this -
NotImplementedError: subtracting a sparse matrix from a nonzero scalar is
not supported
At run time, below are the properties of variables within the CG function -
np.dot(np.array(a), x).shape
(400,)
b.shape
(400,)
I wonder why the subtraction is not happenning???
I tested the same function with the sample input arguments below and it worked fine.
a = np.array([[3, 2, -1], [2, -1, 1], [-1, 1, -1]]) # 3X3 symmetric matrix
b = (np.array([1, -2, 0])[np.newaxis]).T # 3X1 matrix
x = (np.array([0, 1, 2])[np.newaxis]).T
Can someone please tell me why its not working for a sparse matrix?
When multiplying a sparsa matrix by a array you should not use: np.dot(np.array(a), x)) but a.dot(x). See the documentation below:
https://docs.scipy.org/doc/scipy/reference/sparse.html
Follows a correct routine:
def conjGrad(A,x,b,tol,N):
r = b - A.dot(x)
p = r.copy()
for i in range(N):
Ap = A.dot(p)
alpha = np.dot(p,r)/np.dot(p,Ap)
x = x + alpha*p
r = b - A.dot(x)
if np.sqrt(np.sum((r**2))) < tol:
print('Itr:', i)
break
else:
beta = -np.dot(r,Ap)/np.dot(p,Ap)
p = r + beta*p
return x
I have a matrix A (shape (P,Q)) and I need to select a set of rows I such that A[I, :] is square and invertible. Is there an algorithm that I don't know to compute it?
More in general, I need to split A in as many non-singular matrices with shape (*, Q) as possible. Is there an algorithm to compute this?
A (pointer to a) numpy implementation would be appreciated.
My current (greedy and brute-force) implementation is the following:
def independent_columns_index(A, tol=1e-05):
"""
Found here: http://stackoverflow.com/questions/13312498/how-to-find-degenerate-rows-columns-in-a-covariance-matrix
"""
A = np.array(A)
Q, R = np.linalg.qr(A)
independent = np.where(np.abs(R.diagonal()) > tol)[0]
return independent
def independent_rows_index(A, tol=1e-05):
return independent_columns_index(A.T)
def n_independent_rows_indices(A, n, tol=1e-05):
if n > A.shape[1]:
return
independent = np.empty(0, dtype=int)
next_unchecked_row = 0
need_more = n
while A.shape[0] - next_unchecked_row > need_more:
need_more = n - len(independent)
first_row = next_unchecked_row
last_row = next_unchecked_row + need_more
next_unchecked_row = last_row
indices = np.append(independent, np.arange(first_row, last_row))
subA = A[indices, :]
ret = independent_rows_index(subA)
independent = indices[ret]
if len(independent) == n:
yield independent
independent = np.empty(0, dtype=int)
def test():
A = np.random.randint(0, 100, [43, 5])
k = 20
A[-k:, :] = A[:k, :]
np.random.shuffle(A)
A[1, :] = A[0, :]
for ind in n_independent_rows_indices(A, 5):
R = A[ind, :]
print(ind)
print(np.linalg.matrix_rank(R))
if __name__ == '__main__':
test()
EDIT: question corrected after Amit's comment. Added my naive algorithm after John's comment.
EDIT: improved solution:
def n_independent_rows_indices(A, n):
if n > A.shape[1]:
return
not_used = np.arange(A.shape[0])
while True:
try:
A_ = np.array(A[not_used, :][:3*n, :]).T
_, _, P = sp.linalg.qr(A_, pivoting=True)
now_used = not_used[P[:n]]
if len(now_used) < n:
return
not_used = np.delete(not_used, P[:n], axis=0)
yield now_used
except:
return
I tried to copy one array, says A (2-D) to another array, says B (3-D) which have following shape
A is m * n array and B is m * n * p array
I tried the following code but it is very slow, like 1 sec/frame
for r in range (0, h):
for c in range (0, w):
x = random.randint(0, 20)
B[r, c, x] = A[r, c]
I also read some websites about fancy indexing but I still don't know how to apply it in mine.
I propose a solution using array indices. M,N,P are each (m,n) index arrays, specifying the m*n elements of B that will receive data from A.
def indexing(A, p):
m,n = A.shape
B = np.zeros((m,n,p), dtype=int)
P = np.random.randint(0, p, (m,n))
M, N = np.indices(A.shape)
B[M,N,P] = A
return B
For comparision, the original loop, and the solution using shuffle
def looping(A, p):
m, n = A.shape
B = np.zeros((m,n,p), dtype=int)
for r in range (m):
for c in range (n):
x = np.random.randint(0, p)
B[r, c, x] = A[r, c]
return B
def shuffling(A, p):
m, n = A.shape
B = np.zeros((m,n,p), dtype=int)
B[:,:,0] = A
map(np.random.shuffle, B.reshape(m*n,p))
return B
for m,n,p = 1000,1000,20, timings are:
looping: 1.16 s
shuffling: 10 s
indexing: 271 ms
for small m,n, looping is fastest. My indexing solution takes more time to setup, but the actual assignment is fast. The shuffling solution has as many iterations as the original.
The M,N arrays don't have to be full. They can be column and row arrays, respectively
M = np.arange(m)[:,None]
N = np.arange(n)[None,:]
or
M,N = np.ogrid[:m,:n]
This shaves off some time, more so for small test cases than a large one.
A repeatable version:
def indexing(A, p, B=None):
m, n = A.shape
if B is None:
B = np.zeros((m,n,p), dtype=int)
for r in range (m):
for c in range (n):
x = np.random.randint(0, p)
B[r, c, x] = A[r, c]
return B
indexing(A,p,indexing(A,p))
If A isn't the same size as the 1st 2 dim of B the index ranges will have to be changed. A doesn't have to be a 2D array either:
B[[0,0,2],[1,1,0],[3,4,5]] = [10,11,12]
Assuming that h=m, w=n and x=p, this should give you the same as you have in your example:
B[:,:,0]=A
map(np.random.shuffle, B.reshape(h*w,p))
Note also, I'm assuming the answer to NPE's question in comments is 'yes'