Develop Reference

possibly speed up matrix multiplications in loop

I asked a question here with the details: https://math.stackexchange.com/questions/4381785/possibly-speed-up-matrix-multiplications
In short, I am trying to create a P x N matrix, X, with typical element: \sum_{j,k;j,k \neq i} w_{jp} A_{jk} Y_{kp}, where w is P x N, A is N x N and Y is P x N. See the link above for a markup version of that formula.
I'm providing a mwe here to see how I can correct the code (the calculations seem correct, just incomplete see below) and more importantly speed this up however possible:
w = np.array([[2,1],[3,7]])
A = np.array([[2,1],[9,-1]])
Y = np.array([[6,2],[11,8]])
N=w.shape[1]
P=w.shape[0]
X = np.zeros((P, N))
for p in range(P) :
for i in range(N-1):
for j in range(N-1):
X[p,i] = np.delete(w,i,1)[i,p]*np.delete(np.delete(A,i,0),i,1)[i,j]*np.delete(Y.T,i,0)[j,p]
The output looks like:
array([[ -2. , 0. ],
[-56. , 0.]])
If we set (i,p) = to the (1,1) element of X_{ip}, the value can be understood using the formula provided above:
sum_{j,k;j,k \neq i} w_{j1} A_{jk} Y_{k1} = w_12 A_22 Y_12 = 1 * -1 * 2 = -2 as it is in the output.
the (1,2) element of X_{ip} should be:
sum_{j,k;j,k \neq i} w_{j2} A_{jk} Y_{k2} = w_22 A_22 Y_22 = 7 * -1 * 8 = -56 as it is in the output.
But I am not getting the correct answer for the final column of X because my range is to (N-1) not N because I received an IndexError out of bounds when it is N. More importantly, here N=P=2, but I have large N and P and the code, as is, takes a very long time to run. Any suggestions would be greatly appreciated.

Since the delete functions depend only on i, I factored them out, and reordered the loops. Also corrected the w1 index order.
In [274]: w = np.array([[2,1],[3,7]])
...: A = np.array([[2,1],[9,-1]])
...: Y = np.array([[6,2],[11,8]])
...: N=w.shape[1]
...: P=w.shape[0]
...: X = np.zeros((P, N))
...: for i in range(N-1):
...: print('i',i)
...: w1 = np.delete(w,i,1)
...: a1 = np.delete(np.delete(A,i,0),i,1)
...: y1 = np.delete(Y.T,i,0)
...: print(w1.shape, a1.shape, y1.shape)
...: print(w1#a1#y1)
...: print(np.einsum('ij,jk,li->i',w1,a1,y1))
...: for p in range(P):
...: for j in range(N-1):
...: X[p,i] = w1[p,i]*a1*y1[j,p]
...:
i 0
(2, 1) (1, 1) (1, 2)
[[ -2 -8]
[-14 -56]]
[ -2 -56]
In [275]: X
Out[275]:
array([[ -2., 0.],
[-56., 0.]])
Your [-2,-56] are the diagonal of w1#a1#y1, or the einsum. The 0's are from the original np.zeros because i is only on range(1).
This should be faster because the delete is not repeated unnecessarily. np.delete is still relatively expensive, but I haven't tried to figure out exactly what you are doing.
Didn't your question initially have (2,3) and (3,3) arrays? That, or something a bit larger, may be more general and informative.
edit
I think this is closer to the math expressions:
def foo(w,A,Y):
P, N = w.shape
X = np.zeros((P, N))
for i in range(N):
#w1 = np.delete(w,i,1)
a1 = np.delete(A,i,1)
y1 = np.delete(Y.T,i,0)
print(w1.shape, a1.shape, y1.shape)
for p in range(P) :
for j in range(N-1):
X[p,i] += w[p,i]*a1[i,j]*y1[j,p]
return X
we can get rid of the j loop with:
def foo1(w,A,Y):
P, N = w.shape
X = np.zeros((P, N))
for i in range(N):
a1 = np.delete(A,i,1)
y1 = np.delete(Y.T,i,0)
print(w1.shape, a1.shape, y1.shape)
for p in range(P) :
X[p,i] = w[p,i]*np.dot(a1[i,:],y1[:,p])
return X
with
...: w3 = np.array([[2,1,0],[3,7,0.5]])
...: A3 = np.array([[2,1,0],[9,0,8],[1,2,5]])
...: Y3 = np.array([[6,2,-1],[11,8,-7]])
...: w2 = np.array([[2,1],[3,7]])
...: A2 = np.array([[2,1],[9,-1]])
...: Y2 = np.array([[6,2],[11,8]])
both produce
In [372]: foo(w2,A2,Y2)
(2, 1) (2, 1) (1, 2)
...
Out[372]:
array([[ 4., 54.],
[ 24., 693.]])
In [373]: foo(w3,A3,Y3)
(2, 1) (3, 2) (2, 2)
...
Out[373]:
array([[ 4. , 46. , 0. ],
[ 24. , 301. , 13.5]])
and after more fiddling:
def foo4(w,A,Y):
P, N = w.shape
X = np.zeros((P, N))
for i in range(N):
a1 = np.delete(A,i,1)
y1 = np.delete(Y.T,i,0)
X[:,i] = np.einsum('j,jp->p',a1[i,:],y1)
# X[:,i] = a1[i,:]#y1
return X*w
I suspect it is possible to do w*(A#Y.T) and then subtract an array that involves A[:,i] and Y[:,i], but haven't figured out that array.

Protection against "index 0 is out of bounds for axis 0 with size 0" error in Python

I have a code in which I get a specific distribution of points on the graph of the function tan()
limited from the bottom and top by straight lines:
import matplotlib.pyplot as plt
import numpy as np
import sys
import itertools
import multiprocessing
import tqdm
ic = range(1,10)
jc = range(1,10)
paramlist = list(itertools.product(ic,jc))
def func(params):
ic = params[0]
jc = params[1]
fig = plt.figure(1, figsize=(10,6))
x_all = np.linspace(0, 10*np.pi, 10000, endpoint=False)
x_above = x_all[ (-0.01)*ic*x_all < np.tan(x_all) ]
x = x_above[ np.tan(x_above) < 0.01*jc*x_above ]
y = np.tan(x)
y2 = 0.01*jc*x
y3 = (-0.01)*ic*x
y_up = np.diff(y) > 0
y_diff = np.where( y_up, np.diff(y), 0 )
x_diff = np.where( y_up, np.diff(x), 0 )
diffs = np.sqrt( x_diff**2 + y_diff**2 )
length = diffs.sum()
numbers = [2,4,6,8,10,12,14,16,18,20]
p2 = []
for d in range(len(numbers)):
cumlenth = np.cumsum(diffs)
s = np.abs(np.diff(np.sign(cumlenth-numbers[d]))).astype(bool)
c = np.argwhere(s)[0][0]
p = x[c], y[c]
p2.append(p)
p3 = sorted(p2, key=lambda x: x[0])
x_max = p3[len(p3)-1][0]
p4 = sorted(p2, key=lambda x: x[1])
y_min = p4[0][1]
y_max = p4[len(p3)-1][1]
for b in range(len(p2)):
plt.scatter( p2[b][0], p2[b][1], color="crimson", s=8)
plt.plot(x, np.tan(x))
plt.plot(x, y2)
plt.plot(x, y3)
ax = plt.gca()
ax.set_xlim([0, x_max+0.5])
ax.set_ylim([y_min-0.5, y_max+0.5])
plt.savefig('C:\\Users\\tkp\\Desktop\\wykresy_4\\i='+str(ic)+'_j='+str(jc)+'.png', bbox_inches='tight')
plt.show()
if __name__ == '__main__':
p = multiprocessing.Pool(4)
for params in tqdm.tqdm(p.imap_unordered(func, paramlist), total=len(paramlist)):
#pass
sys.stdout.write('\r'+ str(params))
sys.stdout.flush()
p.close()
p.join()
Where, for example, I receive plot:
The problem is that if I set the range in x_all = np.linspace(0, 10*np.pi, 10000, endpoint=False) too small, I get the error index 0 is out of bounds for axis 0 with size 0. How can I protect yourself against this? Or maybe in this case I can set a variable range in the "linspace" function?

Where does this error occur? That's a fundamental piece of information - for us, but especially for you!
#edison says it's in the argwhere expression. I'll try to recreate that step, starting with a guess as to what diffs looks like:
In [8]: x = np.ones(5)*.1
In [9]: x
Out[9]: array([0.1, 0.1, 0.1, 0.1, 0.1])
In [10]: s = np.cumsum(x)
In [11]: s
Out[11]: array([0.1, 0.2, 0.3, 0.4, 0.5])
In [12]: s-1
Out[12]: array([-0.9, -0.8, -0.7, -0.6, -0.5])
In [13]: np.sign(s-1)
Out[13]: array([-1., -1., -1., -1., -1.])
In [14]: np.diff(np.sign(s-1))
Out[14]: array([0., 0., 0., 0.])
In [15]: np.abs(np.diff(np.sign(s-1)))
Out[15]: array([0., 0., 0., 0.])
In [16]: np.abs(np.diff(np.sign(s-1))).astype(bool)
Out[16]: array([False, False, False, False])
Regardless of the details to this point, it's a good guess that s is an array with just False. where finds the True elements in that array; there are none.
In [17]: np.where(_)
Out[17]: (array([], dtype=int64),)
argwhere is the transpose of this - one column for each dimension, and one row for each found item.
In [18]: np.argwhere(_)
Out[18]: array([], shape=(0, 2), dtype=int64)
In [19]: _[0]
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-19-aa79beb95eae> in <module>
----> 1 _[0]
IndexError: index 0 is out of bounds for axis 0 with size 0
So your first line of defense is to check the shape of the returned array:
c = np.argwhere(s)
if c.shape[0]>0:
c = c[0,0]
p = x[c], y[c]
else:
# what do you want to do if non of `s` are true?
You can work backwards from there, taking care to ensure that the diffs or numbers are correct, and always find a valid c. But regardless, when using where or argwhere, be careful about assuming it has found a given number of items.

Looking up index of value in numpy 3D arrays

import numpy as np
# The 3D arrays have the axis: Z, X, Y
arr_keys = np.random.rand(20, 5, 5)
arr_vals = np.random.rand(20, 5, 5)
arr_idx = np.random.rand(5, 5)
For each grid cell in arr_idx, I want to look up the Z-position of the value closest to it in arr_keys (but with the same X, Y location) and return the value at the corresponding position in arr_vals array. Is there a way to do this without using nested for loops?
So, if the value at X=0, Y=0 for arr_idx is 0.5, I want to find the number closest to it at X=0, Y=0, Z ranges from 0 to 10
in arr_keys, and then I want to use the Z position of that number (lets call it Z_prime) to find the value in arr_vals (Z_prime, X=0, Y=0)

This is the type of problem for which np.take_along_axis was created:
# shape (20, 5, 5)
diff = np.abs(arr_idx - arr_keys)
# argmin(..., keepdims=True) doesn't exist yet - this emulates it
# shape (1, 5, 5)
inds = np.expand_dims(np.argmin(diff, axis=0), axis=0)
# shape (1, 5, 5)
res = np.take_along_axis(arr_vals, inds, axis=0)
# shape (5, 5)
res = res.squeeze(axis=0)

I think #xnx's answer is pretty good. Mine is longer but I'll post it anyway ;).
Also, a note: NumPy is made to handle large multi-dimensional arrays efficiently by vectorizing the operations. So I'd suggest avoiding for loops as much as possible. Whatever the task you're looking for, there is (usually) a way to do it while avoiding loops.
arr_keys = np.split(arr_keys, 20)
arr_keys = np.stack(arr_keys, axis=-1)[0]
arr_vals = np.split(arr_vals, 20)
arr_vals = np.stack(arr_vals, axis=-1)[0]
arr_idx = np.expand_dims(arr_idx, axis=-1)
difference = np.abs(arr_keys - arr_idx)
minimum = np.argmin(difference, axis=-1)
result = np.take_along_axis(arr_vals, np.expand_dims(minimum, axis=-1), axis=-1)
result = np.squeeze(result, axis=-1)

I think this might work: roll the axes into the correct orientation, find the index of the value of the (absolute) minimum for each of the 5x5 X,Y values and take the corresponding Z-values from arr_vals:
idx = np.argmin(np.abs(np.rollaxis(arr_keys,0,3) - arr_idx[:,:,None]), axis=2)
i,j = np.ogrid[:5,:5]
arr_vals[idx[i,j],i,j]
To test this, try the (3,2,2) case:
In [15]: arr_keys
Out[15]:
array([[[ 0.19681533, 0.26897784],
[ 0.60469711, 0.09273087]],
[[ 0.04961604, 0.3460404 ],
[ 0.88406912, 0.41284309]],
[[ 0.46298201, 0.33809574],
[ 0.99604152, 0.4836324 ]]])
In [16]: arr_vals
Out[16]:
array([[[ 0.88865681, 0.88287688],
[ 0.3128103 , 0.24188022]],
[[ 0.23947227, 0.57913325],
[ 0.85768064, 0.91701097]],
[[ 0.78105669, 0.84144339],
[ 0.81071981, 0.69217687]]])
In [17]: arr_idx
Out[17]:
array([[[ 0.31352609],
[ 0.75462329]],
[[ 0.44445286],
[ 0.97086161]]])
gives:
array([[ 0.88865681, 0.57913325],
[ 0.3128103 , 0.69217687]])

A little verbose than the already posted solution but easier to understand.
import numpy as np
# The 3D arrays have the axis: Z, X, Y
arr_keys = np.random.rand(20, 5, 5)
arr_vals = np.random.rand(20, 5, 5)
arr_idx = np.random.rand(5, 5)
arr_idx = arr_idx[np.newaxis, :, :]
dist = np.abs(arr_idx - arr_keys)
dist_ind = np.argmin(dist, axis=0)
x = np.arange(0, 5, 1)
y = np.arange(0, 5, 1)
xx, yy = np.meshgrid(x, y)
res = arr_vals[dist_ind, yy, xx]

What is the most efficient way to compute a Kronecker Product in TensorFlow?

I am interested in implementing this paper on Kronecker Recurrent Units in TensorFlow.
This involves the computation of a Kronecker Product. TensorFlow does not have an operation for Kronecker Products. I am looking for an efficient and robust way to compute this.
Does this exist, or would I need to define a TensorFlow op manually?

If you will read the math definition of conv2d_transpose and see what Kronecker product calculates, you will see that with the appropriate size of stides for conv2d_tranpose (width, height of the second matrix), it does the same thing.
Moreover you even have batching of Kronecker's product out of the box with conv2d_transpose.
Here is an example of you which calculates the Kronecker's product for matrices from wiki.
import tensorflow as tf
a = [[1, 2], [3, 4]]
b = [[0, 5], [6, 7]]
i, k, s = len(a), len(b), len(b)
o = s * (i - 1) + k
a_tf = tf.reshape(tf.constant(a, dtype=tf.float32), [1, i, i, 1])
b_tf = tf.reshape(tf.constant(b, dtype=tf.float32), [k, k, 1, 1])
res = tf.squeeze(tf.nn.conv2d_transpose(a_tf, b_tf, (1, o, o, 1), [1, s, s, 1], "VALID"))
with tf.Session() as sess:
print sess.run(res)
Notice that in the case of a non-square matrix, you will need to calulcate more dimensions in the lines:
i, k, s = len(a), len(b), len(b)
o = s * (i - 1) + k
and use them properly as your strides/outputs arguments.

TensorFlow 1.7+ provides the function kronecker_product in tf.contrib.kfac.utils.kronecker_product:
a = tf.eye(3)
b = tf.constant([[1., 2.], [3., 4.]])
kron = tf.contrib.kfac.utils.kronecker_product(a, b)
tf.Session().run(kron)
Output:
array([[1., 2., 0., 0., 0., 0.],
[3., 4., 0., 0., 0., 0.],
[0., 0., 1., 2., 0., 0.],
[0., 0., 3., 4., 0., 0.],
[0., 0., 0., 0., 1., 2.],
[0., 0., 0., 0., 3., 4.]], dtype=float32)

Here's the utility I use for this. See kronecker_test for example of usage
def fix_shape(tf_shape):
return tuple(int(dim) for dim in tf_shape)
def concat_blocks(blocks, validate_dims=True):
"""Takes 2d grid of blocks representing matrices and concatenates to single
matrix (aka ArrayFlatten)"""
if validate_dims:
col_dims = np.array([[int(b.shape[1]) for b in row] for row in blocks])
col_sums = col_dims.sum(1)
assert (col_sums[0] == col_sums).all()
row_dims = np.array([[int(b.shape[0]) for b in row] for row in blocks])
row_sums = row_dims.sum(0)
assert (row_sums[0] == row_sums).all()
block_rows = [tf.concat(row, axis=1) for row in blocks]
return tf.concat(block_rows, axis=0)
def chunks(l, n):
"""Yield successive n-sized chunks from l."""
for i in range(0, len(l), n):
yield l[i:i + n]
from tensorflow.python.framework import ops
original_shape_func = ops.set_shapes_for_outputs
def disable_shape_inference():
ops.set_shapes_for_outputs = lambda _: _
def enable_shape_inference():
ops.set_shapes_for_outputs = original_shape_func
def kronecker(A, B, do_shape_inference=True):
"""Kronecker product of A,B.
turn_off_shape_inference: if True, makes 10x10 kron go 2.4 sec -> 0.9 sec
"""
Arows, Acols = fix_shape(A.shape)
Brows, Bcols = fix_shape(B.shape)
Crows, Ccols = Arows*Brows, Acols*Bcols
temp = tf.reshape(A, [-1, 1, 1])*tf.expand_dims(B, 0)
Bshape = tf.constant((Brows, Bcols))
# turn off shape inference
if not do_shape_inference:
disable_shape_inference()
# [1, n, m] => [n, m]
slices = [tf.reshape(s, Bshape) for s in tf.split(temp, Crows)]
# import pdb; pdb.set_trace()
grid = list(chunks(slices, Acols))
assert len(grid) == Arows
result = concat_blocks(grid, validate_dims=do_shape_inference)
if not do_shape_inference:
enable_shape_inference()
result.set_shape((Arows*Brows, Acols*Bcols))
return result
def kronecker_test():
A0 = [[1,2],[3,4]]
B0 = [[6,7],[8,9]]
A = tf.constant(A0)
B = tf.constant(B0)
C = kronecker(A, B)
sess = tf.Session()
C0 = sess.run(C)
Ct = [[6, 7, 12, 14], [8, 9, 16, 18], [18, 21, 24, 28], [24, 27, 32, 36]]
Cnp = np.kron(A0, B0)
check_equal(C0, Ct)
check_equal(C0, Cnp)

Try the following solution, see if it works for you:
def tf_kron(a,b):
a_shape = [a.shape[0].value,a.shape[1].value]
b_shape = [b.shape[0].value,b.shape[1].value]
return tf.reshape(tf.reshape(a,[a_shape[0],1,a_shape[1],1])*tf.reshape(b,[1,b_shape[0],1,b_shape[1]]),[a_shape[0]*b_shape[0],a_shape[1]*b_shape[1]])

How about something like this:
def kron(x, y):
"""Computes the Kronecker product of two matrices.
Args:
x: A matrix (or batch thereof) of size m x n.
y: A matrix (or batch thereof) of size p x q.
Returns:
z: Kronecker product of matrices x and y of size mp x nq
"""
with tf.name_scope('kron'):
x = tf.convert_to_tensor(x, dtype_hint=tf.float32)
y = tf.convert_to_tensor(y, dtype_hint=x.dtype)
def _maybe_expand(x):
xs = tf.pad(
tf.shape(x),
paddings=[[tf.maximum(2 - tf.rank(x), 0), 0]],
constant_values=1)
x = tf.reshape(x, xs)
_, mx, nx = tf.split(xs, num_or_size_splits=[-1, 1, 1])
return x, mx, nx
x, mx, nx = _maybe_expand(x)
y, my, ny = _maybe_expand(y)
x = x[..., :, tf.newaxis, :, tf.newaxis]
y = y[..., tf.newaxis, :, tf.newaxis, :]
z = x * y
bz = tf.shape(z)[:-4]
z = tf.reshape(z, tf.concat([bz, mx * my, nx * ny], axis=0))
return z
This solution:
supports batches
supports broadcasting
works in xla
clearly shows the relationship between numpy broadcasting and kronecker products.

Converting a list of points to a numpy 2D array

I'm using genfromtxt to import essentially a 2D array that has all its values listed in a text file of the form (x's and y's are integers):
x1 y1 z1
x2 y2 z2
: : :
I'm using the for loop below but I'm pretty sure there must be a one line way to do it. What would be a more efficient way to do this conversion?
raw = genfromtxt(file,skip_header = 6)
xrange = ( raw[:,0].min() , raw[:,0].max() )
yrange = ( raw[:,1].min() , raw[:,1].max() )
Z = zeros(( xrange[1] - xrange[0] +1 , yrange[1] - yrange[0] +1 ))
for row in raw:
Z[ row[0]-xrange[0] , row[1]-yrange[0] ] = row[2]

You can replace the for loop with the following:
xidx = (raw[:,0]-xrange[0]).astype(int)
yidx = (raw[:,1]-yrange[0]).astype(int)
Z[xidx, yidx] = raw[:,2]

To import a matrix from a file you can just split the lines and then convert to int.
[[int(i) for i in j.split()] for j in open('myfile').readlines()]
of course, I'm supposing your file contains only the matrix.
At the end, you can convert this 2-D array to numpy.

You may try something like this:
>>> Z = zeros((3, 3))
>>> test = array([[0, 1, 2], [1, 1, 6], [2, 0, 4]])
>>> Z[test[:, 0:2].T.tolist()]
array([ 0., 0., 0.])
>>> Z[test[:, 0:2].T.tolist()] = test[:, 2]
>>> Z
array([[ 0., 2., 0.],
[ 0., 6., 0.],
[ 4., 0., 0.]])
In your case:
Z[(raw[:, 0:2] - minimum(raw[:, 0:2], axis=0)).T.tolist()] = raw[:, 2]

You could also go with numpy.searchsorted which will also allow for non-equally spaced / float data:
raw = genfromtxt(file,skip_header = 6)
xvalues = numpy.sorted(set(raw[:,0]))
xidx = numpy.searchsorted(xvalues, raw[:,0])
yvalues = numpy.sorted(set(raw[:,1]))
yidx = numpy.searchsorted(yvalues, raw[:,1])
Z = numpy.zeros((len(xvalues), len(yvalues)))
Z[xidx, yidx] = raw[:,2]
Otherwise, I would be following Simon's answer.