Say that I have a matrix:
X = [
[0, 0, 0, 0, 0],
[0, 0, 1, 0, 0],
[0, 0, 0, 0, 0],
[2, 0, 0, 3, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 4, 0, 0, 0]
]
but it is stored in a sparse format:
# [row, column, value]
X_sparse = [
[1, 2, 1],
[3, 0, 2],
[3, 3, 3]
[6, 1, 4],
]
I also have a vector
b = [1,2,3,4,5]
I would like to pass X_sparse to TensorFlow as data and multiply it with b (i.e. np.dot(X, b)). What is the correct way to do this in TensorFlow? Additionally, this is a toy example, while in real-life X has many rows and columns, so I'd rather not want to work with it converted to non-sparse format.
I don't see an easy way to do so without using a dense tensor representation (tf.sparse_tensor_to_dense()). Tensorflow is however built for operations on huge matrices, so in many cases it shouldn't be a problem...
A quick example:
import tensorflow as tf
import numpy as np
X_sparse_np = np.array([
[1, 2, 1],
[3, 0, 2],
[3, 3, 3],
[6, 1, 4],
])
b_np = np.array([1,2,3,4,5])
b = tf.constant(b_np)
# Converting your input data into a SparseTensor:
indices = X_sparse_np[:, :2].reshape((-1, 2))
values = X_sparse_np[:, 2]
dense_shape = [7, 5]
X_sparse = tf.SparseTensorValue(indices, values, dense_shape)
# Performing your operation on its dense representation:
res = tf.sparse_tensor_to_dense(X_sparse) * b
# opt. Re-obtaining the sparse representation of your result:
sparse_res = tf.SparseTensor(indices, tf.gather_nd(res, indices), dense_shape)
with tf.Session() as sess:
print(sess.run(res))
# [[ 0 0 0 0 0]
# [ 0 0 3 0 0]
# [ 0 0 0 0 0]
# [ 2 0 0 12 0]
# [ 0 0 0 0 0]
# [ 0 0 0 0 0]
# [ 0 8 0 0 0]]
print(sess.run(sparse_res))
# SparseTensorValue(
# indices=array([[1, 2],
# [3, 0],
# [3, 3],
# [6, 1]]),
# values=array([ 3, 2, 12, 8]),
# dense_shape=array([7, 5]))
Probably there was no better option back then, there is a function for this now - doing the matrix multiplication after converting to dense makes the whole thing kind of pointless...
import tensorflow as tf
import numpy as np
A_vals = np.array([
[1, 2, 1],
[3, 0, 2],
[3, 3, 3],
[6, 1, 4],
])
A_sparse = tf.SparseTensor(A_vals[:,:2], tf.cast(A_vals[:,2], tf.float32), [7, 5])
b = tf.constant([1,2,3,4,5], tf.float32)
# sparse-dense matrix multiplication
res = tf.sparse.sparse_dense_matmul(A_sparse, b[:,None])
Related
I want to create a Cartesian product of two numpy so that the first numpy will be the rows and the second will be the columns.
For example get these two numpys:
a = np.array([0,0])
b = np.array([0,1,2,3])
The expected result should be 2d numpy like this:
[[0 0 0]
[0 0 1]
[0 0 2]
[0 0 3]]
The following code does not produce the requested result:
a = np.array([0,0])
b = np.array([0,1,2,3])
_new_ = []
for idx in range(len(a)):
for i in a:
newArr = np.append(a[idx], b)
_new_.append(newArr)
print(np.stack(_new_))
What needs to be changed to produce the desired result?
You can use np.tile with np.column_stack
np.column_stack([np.tile(a, (len(b), 1)), b])
array([[0, 0, 0],
[0, 0, 1],
[0, 0, 2],
[0, 0, 3]])
If you have a a as 2D array
a = np.array([[0, 0], [1, 1]])
b = np.array([0,1,2,3])
np.c_[np.tile(a, (len(b), 1)), np.repeat(b, len(a), axis=0)]
array([[0, 0, 0],
[1, 1, 0],
[0, 0, 1],
[1, 1, 1],
[0, 0, 2],
[1, 1, 2],
[0, 0, 3],
[1, 1, 3]])
I would like to multyply the following matrices (using numpy) in the most efficient way.
This is the code for the matrixes:
a = np.array([[1, 5], [2, 6], [3, 7], [4, 8]])
m = np.array([[1, 0, 0, 1], [1, 0, 1, 0], [0, 1, 0, 1], [0, 1, 1, 1]])
These are the matrixes visualized better:
a:
[[1 5]
[2 6]
[3 7]
[4 8]]
m:
[[1 0 0 1]
[1 0 1 0]
[0 1 0 1]
[0 1 1 1]]
I want to multiply a by (the first column of matrix m), like this
a m[:,0] x0
[[1 5] [[1] [[1 5]
[2 6] * [1] = [2 6]
[3 7] [0] [0 0]
[4 8]] [0]] [0 0]
And then I want to multiply a by (the second column of matrix m), like this
a * m[:,1] = x1
And then 3rd and 4th column
a * m[:,2] = x2
a * m[:,3] = x3
And finally, I want to put the resulting matrices x0,x1,x2,x3 all in one matrix.
X = [x0, x1, x2, x3]
The size X in this example is 4 x 8.
The final result in this example is:
X =
[[[1 5 0 0 0 0 1 5]
[2 6 0 0 2 6 0 0]
[0 0 3 7 0 0 3 7]
[0 0 4 8 4 8 4 8]]
I would like to know how to do this with build-in functions of numpy, and using generators, instead of using 2 for loops, if it is possible.
This is just an example. In reality the matrixes have large dimensions and it is important that the multiplications are done as fast as possible.
Thank you
You may achieve it with broadcast and reshape
arr_out = (a[:,None] * m[...,None]).reshape(4,8)
Out[176]:
array([[1, 5, 0, 0, 0, 0, 1, 5],
[2, 6, 0, 0, 2, 6, 0, 0],
[0, 0, 3, 7, 0, 0, 3, 7],
[0, 0, 4, 8, 4, 8, 4, 8]])
You could transpose and expand the dimensions of m, to get the wanted result:
m.T[...,None] * a
array([[[1, 5],
[2, 6],
[0, 0],
[0, 0]],
[[0, 0],
[0, 0],
[3, 7],
[4, 8]],
...
If you want to stack the arrays horizontally to end up with a 2D array, use np.hstack:
np.hstack(m.T[...,None] * a)
array([[1, 5, 0, 0, 0, 0, 1, 5],
[2, 6, 0, 0, 2, 6, 0, 0],
[0, 0, 3, 7, 0, 0, 3, 7],
[0, 0, 4, 8, 4, 8, 4, 8]])
Or reshaping as:
(a[:,None] * m[...,None]).reshape(m.shape[0], -1)
This is the answer that I was looking for. Thank you Yatu and hpaulj.
X = m.T[...,None] * a
for i in range(4):
reshaped = np.hstack(X[i,:,:])
reshaped_simpler = np.hstack(X)
print (reshaped_simpler)
I got the rest of the answer from the following link:
reshape numpy 3D array to 2D
I rearranged the for loop because I got a Warning regarding the generators going to be depricated in future versions of Numpy.
After some vectorized calculations, I get a sparse block matrix with all my results stacked in blocks of same size.
>>> A = [[1, 1],
... [1, 1]]
>>> B = [[2, 2],
... [2, 2]]
>>> C = [[3, 3],
... [3, 3]]
>>> results = scipy.sparse.block_diag(A, B, C)
>>> print(results.toarray())
[[1 1 0 0 0 0]
[1 1 0 0 0 0]
[0 0 2 2 0 0]
[0 0 2 2 0 0]
[0 0 0 0 3 3]
[0 0 0 0 3 3]]
How can I get back these arrays A,B,C in an efficient way, if necessery by providing their shape (2,2)?
In [177]: >>> A = [[1, 1],
...: ... [1, 1]]
...: >>> B = [[2, 2],
...: ... [2, 2]]
...: >>> C = [[3, 3],
...: ... [3, 3]]
...: >>> results = sparse.block_diag([A, B, C])
...:
In [178]: results
Out[178]:
<6x6 sparse matrix of type '<class 'numpy.int64'>'
with 12 stored elements in COOrdinate format>
block_diag does not preserve the inputs; rather it creates coo format matrix, representing the whole matrix, not the pieces.
In [194]: results.data
Out[194]: array([1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3], dtype=int64)
In [195]: results.row
Out[195]: array([0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5], dtype=int32)
In [196]: results.col
Out[196]: array([0, 1, 0, 1, 2, 3, 2, 3, 4, 5, 4, 5], dtype=int32)
In [179]: results.A
Out[179]:
array([[1, 1, 0, 0, 0, 0],
[1, 1, 0, 0, 0, 0],
[0, 0, 2, 2, 0, 0],
[0, 0, 2, 2, 0, 0],
[0, 0, 0, 0, 3, 3],
[0, 0, 0, 0, 3, 3]], dtype=int64)
block_diag pass the arrays to sparse.bmat. That in turn makes a coo matrix from each, and then merges the coo attributes into 3 arrays, which are inputs to the global sparse matrix.
There is another sparse format bsr that may preserve the blocks (until conversion to csr for calculation), but I'll have to experiment to see that's the case.
Let's make a bsr from that results coo:
In [186]: bresults = sparse.bsr_matrix(results)
In [187]: bresults
Out[187]:
<6x6 sparse matrix of type '<class 'numpy.int64'>'
with 12 stored elements (blocksize = 2x2) in Block Sparse Row format>
In [188]: bresults.blocksize
Out[188]: (2, 2)
In [189]: bresults.data
Out[189]:
array([[[1, 1],
[1, 1]],
[[2, 2],
[2, 2]],
[[3, 3],
[3, 3]]], dtype=int64)
So it deduces that there are blocks, just as you desired.
In [191]: bresults.indices
Out[191]: array([0, 1, 2], dtype=int32)
In [192]: bresults.indptr
Out[192]: array([0, 1, 2, 3], dtype=int32)
So it's a csr like storage, but with the data arranged in blocks.
It may be possible to construct this from your A,B,C without the block_diag intermediary, but I'd have to look at the docs more.
that's a funny little problem.
I don't think there is a function that solves this in one line, but there's a way to do it programmatically.
Check out what res.data prints out, I use it here.
This works when shapes are all the same.
from scipy.sparse import block_diag
a = [[1, 2, 4],
[3, 4, 4]]
b = [[2, 2, 1],
[2, 2, 1]]
c = [[3, 3, 6],
[3, 3, 6]]
res = block_diag((a, b, c))
def goBack(res, shape):
s = shape[0]*shape[1]
num = int(len(res.data)/s)
for i in range (num):
mat = res.data[i*s:(i+1)*s].reshape(shape)
print(mat)
goBack(res, [2,3])
Output:
[[1 2 4]
[3 4 4]]
[[2 2 1]
[2 2 1]]
[[3 3 6]
[3 3 6]]
Edit:
Okay, this does not work when any of the elements of the provided matrices is zero, as then it would not be counted in res.data.
Also, forget it, the link provided by cleb should help you.
I have the following operations which uses numpy.where:
mat = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=np.int32)
index = np.array([[1,0,0],[0,1,0],[0,0,1]])
mat[np.where(index>0)] = 100
print(mat)
How to implement the equivalent in TensorFlow?
mat = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=np.int32)
index = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]])
tf_mat = tf.constant(mat)
tf_index = tf.constant(index)
indi = tf.where(tf_index>0)
tf_mat[indi] = -1 <===== not allowed
Assuming that what you want is to create a new tensor with some replaced elements, and not update a variable, you could do something like this:
import numpy as np
import tensorflow as tf
mat = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=np.int32)
index = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]])
tf_mat = tf.constant(mat)
tf_index = tf.constant(index)
tf_mat = tf.where(tf_index > 0, -tf.ones_like(tf_mat), tf_mat)
with tf.Session() as sess:
print(sess.run(tf_mat))
Output:
[[-1 2 3]
[ 4 -1 6]
[ 7 8 -1]]
You can get indexes by tf.where, then you can either run the index, or use tf.gather to collect data from the origin array, or use tf.scatter_update to update origin data, tf.scatter_nd_update for multi-dimension update.
mat = tf.Variable([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=tf.int32)
index = tf.Variable([[1,0,0],[0,1,0],[0,0,1]])
idx = tf.where(index>0)
tf.scatter_nd_update(mat, idx, /*values you want*/)
note that update values should be the same first dimension size with idx.
see https://www.tensorflow.org/api_guides/python
I have a matrix with some zero
x=np.array([[1,2,3,0],[4,0,5,0],[7,0,0,0],[0,9,8,0]])
>>> x
array([[1, 2, 3, 0],
[4, 0, 5, 0],
[7, 0, 0, 0],
[0, 9, 8, 0]])
And want to random value into only a position which is not zero. I can get the (row, col) position as tuple from np.where
pos = np.where(x!=0)
>>> (array([0, 0, 0, 1, 1, 2, 3, 3], dtype=int64), array([0, 1, 2, 0, 2, 0, 1, 2], dtype=int64))
Is there a way to use np.random (or something else) for the matrix x at position from posonly without changing where is zero?
# pseudocode
new_x = np.rand(x, at pos)
I assume you want to replace non-zero value with random integer number.
You can use the combination of numpy.place and numpy.random.randint functions.
>>> x=np.array([[1,2,3,0],[4,0,5,0],[7,0,0,0],[0,9,8,0]])
>>> x
array([[1, 2, 3, 0],
[4, 0, 5, 0],
[7, 0, 0, 0],
[0, 9, 8, 0]])
>>> lower_bound, upper_bound = 1, 5 # random function boundary
>>> np.place(x, x!=0, np.random.randint(lower_bound, upper_bound, np.count_nonzero(x)))
>>> x
array([[2, 2, 3, 0],
[1, 0, 3, 0],
[2, 0, 0, 0],
[0, 4, 3, 0]])
well you can use x.nonzero() which gives you all indices of array with nonzero values
and then then you just need to put random values at those indices
nz_indices = x.nonzero()
for i,j in zip(nz_indices[0],nz_indices[1]):
x[i][j] = np.random.randint(1500) #random number till 1500
you can find more about randint() here >> randint docs
How about something simple like this:
import numpy as np
x = np.array([[1, 2, 3, 0], [4, 0, 5, 0], [7, 0, 0, 0], [0, 9, 8, 0]])
w = x != 0
x[w] = np.random.randint(10, size=x.shape)[w]
print(x)
[[2 2 2 0]
[0 0 4 0]
[1 0 0 0]
[0 3 1 0]]
You could also do
x = np.random.randint(1, 10, size=x.shape) * (x != 0)
Just index with np.nonzero
i = np.nonzero(x)
x[i] = np.random.randint(1, 10, i[0].size)
Note for reference that np.nonzero(x) <=> np.where(x) <=> np.where(x != 0)