Develop Reference

Issues with generating the following block matrix using horizontal stacking and vertical stacking

I am trying to generate the following block matrix consisting of submatrices A and B, and N is a positive integer. So far, my code is as follows:
C_lower = B
for j in range(0,N):
for i in range(0,N-j):
col = np.linalg.matrix_power(A,i) # B
C = np.hstack(np.vstack((C_lower,col)))
However, it seems like my code is not working because the loop continues forever. Any suggestions?
Similarly, I'm also having issues with constructing the following block diagonal matrices:
I tried using block_diag from scipy, but there is no way I can repeat Q as many times as N is equal to (i.e., N = 50 in my case). I had to do block_diag(Q,Q,Q,Q,Q,Q,Q.......) in order to get the block diagonal matrix I want.

Here's the answer to your first question. There are a number of issues in your code. This is a better way of achieving what you want:
C = np.zeros((N, N, A.shape[0], B.shape[1]))
for i in range(N):
for j in range(i + 1):
C[i, j] = np.linalg.matrix_power(A, i - j) # B
Similarly for your second question:
Q_ = np.zeros((N, N, *Q.shape))
for i in range(N):
Q_[i, i] = Q

Combination of matrix elements giving non-zero value (PYTHON)

I have to evaluate the following expression, given two quite large matrices A,B and a very complicated function F:
The mathematical expression
I was thinking if there is an efficient way in order to first find those indices i,j that will give a non-zero element after the multiplication of the matrices, so that I avoid the quite slow 'for loops'.
Current working code
# Starting with 4 random matrices
A = np.random.randint(0,2,size=(50,50))
B = np.random.randint(0,2,size=(50,50))
C = np.random.randint(0,2,size=(50,50))
D = np.random.randint(0,2,size=(50,50))
indices []
for i in range(A.shape[0]):
for j in range(A.shape[0]):
if A[i,j] != 0:
for k in range(B.shape[1]):
if B[j,k] != 0:
for l in range(C.shape[1]):
if A[i,j]*B[j,k]*C[k,l]*D[l,i]!=0:
indices.append((i,j,k,l))
print indices
As you can see, in order to get the indices I need I have to use nested loops (= huge computational time).

My guess would be NO: you cannot avoid the for-loops. In order to find all the indices ij you need to loop through all the elements which defeats the purpose of this check. Therefore, you should go ahead and use simple array elementwise multiplication and dot product in numpy - it should be quite fast with for loops taken care by numpy.
However, if you plan on using a Python loop then the answer is YES, you can avoid them by using numpy, using the following pseudo-code (=hand-waving):
i, j = np.indices((N, M)) # CAREFUL: you may need to swap i<->j or N<->M
fs = F(i, j, z) # array of values of function F
# for a given z over the index grid
R = np.dot(A*fs, B) # summation over j
# return R # if necessary do a summation over i: np.sum(R, axis=...)
If the issue is that computing fs = F(i, j, z) is a very slow operation, then you will have to identify elements of A that are zero using two loops built-in into numpy (so they are quite fast):
good = np.nonzero(A) # hidden double loop (for 2D data)
fs = np.zeros_like(A)
fs[good] = F(i[good], j[good], z) # compute F only where A != 0

how to init a array with each element holding the value different from its neighbours

I have a matrix or a multiple array written in python, each element in the array is an integer ranged from 0 to 7, how would I randomly initalize this matrix or multiple array, so that for each element holds a value, which is different from the values of its 4 neighbours(left,right, top, bottom)? can it be implemented in numpy?

You can write your own matrix initializer.
Go through the array[i][j] for each i, j pick a random number between 0 and 7.
If the number equals to either left element: array[i][j-1] or to the upper one: array[i-1][j] regenerate it once again.
You have 2/7 probability to encounter such a bad case, and 4/49 to make it twice in a row, 8/343 for 3 in a row, etc.. the probability dropes down very quickly.
The average case complexity for n elements in a matrix would be O(n).

A simpler problem that might get you started is to do the same for a 1d array. A pure-python solution would look like:
def sample_1d(n, upper):
x = [random.randrange(upper)]
for i in range(1, n)"
xi = random.randrange(upper - 1)
if xi >= x:
xi += 1
x.append(xi)
return x
You can vectorize this as:
def sample_1d_v(n, upper):
x = np.empty(n)
x[0] = 0
x[1:] = np.cumsum(np.random.randint(1, upper, size=n-1)) % upper
x += np.random.randint(upper)
return
The trick here is noting that if there is adjacent values must be different, then the difference between their values is uniformly distributed in [1, upper)

Vectorize sparse sum without scipy.sparse

I am trying to do spatial derivatives and almost managed to get all the loops out of my code, but when I try to sum everything up at the end I have a problem.
I have a set of N~=250k nodes. I have found indices i,j of node pairs with i.size=j.size=~7.5M that are within a certain search distance, originally coming from np.triu_indices(n,1) and passed through a series of boolean masks to wash out nodes not influencing each other. Now I want to sum up the influences on each node from the other nodes.
I currently have this:
def sparseSum(a,i,j,n):
return np.array([np.sum(a[np.logical_or(i==k,j==k)],axis=0) for k in range(n)])
This is very slow. What I would like is something vectorized. If I had scipy I could do
def sparseSum(a,i,j,n):
sp=scipy.sparse.csr_matrix((a,(i,j)),shape=(n,n))+ scipy.sparse.csr_matrix((a,(j,i)),shape=(n,n))
return np.sum(sp, axis=0)
But I'm doing this all within an Abaqus implementation that doesn't include scipy. Is there any way to do this numpy-only?

Approach #1 : Here's an approach making use of matrix-multiplication and broadcasting -
K = np.arange(n)[:,None]
mask = (i == K) | (j == K)
out = np.dot(mask,a)
Approach #2 : For cases with a small number of columns, we can use np.bincount for such bin-based summing along each column, like so -
def sparseSum(a,i,j,n):
if len(a.shape)==1:
out=np.bincount(i,a,minlength=n)+np.bincount(j,a)
else:
ncols = a.shape[1]
out = np.empty((n,ncols))
for k in range(ncols):
out[:,k] = np.bincount(i,a[:,k],minlength=n) + np.bincount(j,a[:,k])
return out

Here's not a turn-key solution but one that adds columns of a sparse matrix. It essentially computes and utilises the csc representation
def sparse_col_sums(i, j, a, N):
order = np.lexsort(j, i)
io, jo, ao = i[order], j[order], a[order]
col_bnds = io.searchsorted(np.arange(N))
return np.add.reduceat(ao, col_bnds)

Remove/set the non-zero diagonal elements of a sparse matrix in scipy

Say I would like to remove the diagonal from a scipy.sparse.csr_matrix. Is there an efficient way of doing so? I saw that in the sparsetools module there are C functions to return the diagonal.
Based on other SO answers here and here my current approach is the following:
def csr_setdiag_val(csr, value=0):
"""Set all diagonal nonzero elements
(elements currently in the sparsity pattern)
to the given value. Useful to set to 0 mostly.
"""
if csr.format != "csr":
raise ValueError('Matrix given must be of CSR format.')
csr.sort_indices()
pointer = csr.indptr
indices = csr.indices
data = csr.data
for i in range(min(csr.shape)):
ind = indices[pointer[i]: pointer[i + 1]]
j = ind.searchsorted(i)
# matrix has only elements up until diagonal (in row i)
if j == len(ind):
continue
j += pointer[i]
# in case matrix has only elements after diagonal (in row i)
if indices[j] == i:
data[j] = value
which I then follow with
csr.eliminate_zeros()
Is that the best I can do without writing my own Cython code?

Based on #hpaulj's comment, I created an IPython Notebook which can be seen on nbviewer. This shows that out of all methods mentioned the following is the fastest (assume that mat is a sparse CSR matrix):
mat - scipy.sparse.dia_matrix((mat.diagonal()[scipy.newaxis, :], [0]), shape=(one_dim, one_dim))