I have a sparse tensor that I'm building up from a collection of indices and values. I'm trying to implement some code to take a full row slice. Although this functionality doesn't appear to be directly supported in TensorFlow, it seems that there are work arounds that can return the indices and values for a specified row as follows:
def sparse_slice(self, indices, values, needed_row_ids):
needed_row_ids = tf.reshape(needed_row_ids, [1, -1])
num_rows = tf.shape(indices)[0]
partitions = tf.cast(tf.reduce_any(tf.equal(tf.reshape(indices[:, 0], [-1, 1]), needed_row_ids), 1),
tf.int32)
rows_to_gather = tf.dynamic_partition(tf.range(num_rows), partitions, 2)[1]
slice_indices = tf.gather(indices, rows_to_gather)
slice_values = tf.gather(values, rows_to_gather)
return slice_indices, slice_values
Then called directly like so on a sparse 4x4 matrix where I am interested in accessing all of the elements in row 3:
with tf.Session().as_default():
indices = tf.constant([[0, 0], [1, 0], [2, 0], [2, 1], [3, 0], [3, 3]])
values = tf.constant([10, 19, 1, 1, 6, 5], dtype=tf.int64)
needed_row_ids = tf.constant([3])
slice_indices, slice_values = self.sparse_slice(indices, values, needed_row_ids)
print('indicies: {} and rows: {}'.format(slice_indices.eval(), slice_values.eval()))
Which outputs the following:
indicies: [[3 0]
[3 3]] and rows: [6 5]
So far so good, I then figure I can use this information to construct a 1x4 dense tensor with the values at the indexes and 0s for the missing columns.
dense_representation = tf.sparse_to_dense(sparse_values=slice_values, sparse_indices=slice_indices,
output_shape=(1,4))
However the moment I run the tensor in a session.
sess = tf.Session()
sess.run(dense_representation)
I receive the following exception:
InvalidArgumentError (see above for traceback): indices[0] = [3,0] is out of bounds: need 0 <= index < [1,4]
[[Node: SparseToDense = SparseToDense[T=DT_INT64, Tindices=DT_INT32, validate_indices=true, _device="/job:localhost/replica:0/task:0/cpu:0"](Gather_2, SparseToDense/output_shape, Gather_3, SparseToDense/default_value)]]
I'm not quite sure what I'm doing wrong or if this has something to do with the output_shape not being properly formed. Essentially I'd like to stuff this all back into a 1 x 4 vector. I haven't been able to find any good examples online for how to do this. Any help would be appreciated.
Related
I am searching for a way to do some batchwise indexing for tensors.
If I have a variable Q of size 1000, I can get the elements I want by
Q[index], where index is a vector of the wanted elements.
Now I would like to do the same for more dimensional tensors.
So suppose Q is of shape n x m and I have a index matrix of shape n x p.
My goal is to get for each of the n rows the specific p elements out of the m elements.
But Q[index] is not working for this situation.
Do you have any thoughts how to handle this?
You can seem to be a simple application of torch.gather which doesn't require any additional reshaping of the data or index tensor:
>>> Q = torch.rand(5, 4)
tensor([[0.8462, 0.3064, 0.2549, 0.2149],
[0.6801, 0.5483, 0.5522, 0.6852],
[0.1587, 0.4144, 0.8843, 0.6108],
[0.5265, 0.8269, 0.8417, 0.6623],
[0.8549, 0.6437, 0.4282, 0.2792]])
>>> index
tensor([[0, 1, 2],
[2, 3, 1],
[0, 1, 2],
[2, 2, 2],
[1, 1, 2]])
The following gather operation applied on dim=1 return a tensor out, such that:
out[i, j] = Q[i, index[i,j]]
This is done with the following call of torch.Tensor.gather on Q:
>>> Q.gather(dim=1, index=index)
tensor([[0.8462, 0.3064, 0.2549],
[0.5522, 0.6852, 0.5483],
[0.1587, 0.4144, 0.8843],
[0.8417, 0.8417, 0.8417],
[0.6437, 0.6437, 0.4282]])
I have a 2D pytorch tensor of shape n by m. I want to index the second dimension using a list of indices (which could be done with torch.gather) then then also set new values to the result of the indexing.
Example:
data = torch.tensor([[0,1,2], [3,4,5], [6,7,8]]) # shape (3,3)
indices = torch.tensor([1,2,1], dtype=torch.long).unsqueeze(-1) # shape (3,1)
# data tensor:
# tensor([[0, 1, 2],
# [3, 4, 5],
# [6, 7, 8]])
I want to select the specified indices per row (which would be [1,5,7] but then also set these values to another number - e.g. 42
I can select the desired columns row wise by doing:
data.gather(1, indices)
tensor([[1],
[5],
[7]])
data.gather(1, indices)[:] = 42 # **This does NOT work**, since the result of gather
# does not use the same storage as the original tensor
which is fine, but I would like to change these values now, and have the change also affect the data tensor.
I can do what I want to achieve using this, but it seems to be very un-pythonic:
max_index = torch.max(indices)
for i in range(0, max_index + 1):
mask = (indices == i).nonzero(as_tuple=True)[0]
data[mask, i] = 42
print(data)
# tensor([[ 0, 42, 2],
# [ 3, 4, 42],
# [ 6, 42, 8]])
Any hints on how to do that more elegantly?
What you are looking for is torch.scatter_ with the value option.
Tensor.scatter_(dim, index, src, reduce=None) → Tensor
Writes all values from the tensor src into self at the indices specified in the index tensor. For each value in src, its output index is specified by
its index in src for dimension != dim and by the corresponding value
in index for dimension = dim.
With 2D tensors as input and dim=1, the operation is:
self[i][index[i][j]] = src[i][j]
No mention of the value parameter though...
With value=42, and dim=1, this will have the following effect on data:
data[i][index[i][j]] = 42
Here applied in-place:
>>> data.scatter_(index=indices, dim=1, value=42)
>>> data
tensor([[ 0, 42, 2],
[ 3, 4, 42],
[ 6, 42, 8]])
I'm trying to compute the cosine similarity between 350k sentences using tensorflow.
My sentences are first vectorisd using sklearn:
doc = df['text']
vec = TfidfVectorizer(binary=False,norm='l2',use_idf=False,smooth_idf=False,lowercase=True,stop_words='english',min_df=1,max_df=1.0,max_features=None,ngram_range=(1, 1))
X = vec.fit_transform(doc)
print(X.shape)
print(type(X))
This works very well and I get sparse matrix back, I have then tried in two ways to convert my sparse matrix to a dense one.
(1) I tried this:
dense = X.toarray()
This only works with a small amount of data (around 10k sentences), but then fails on the actual computation.
(2) I have been trying to convert the output X this way, but get the same error message when doing the first step K:
K = tf.convert_to_tensor(X, dtype=None, dtype_hint=None, name=None)
Y = tf.sparse.to_dense(K, default_value=None, validate_indices=True, name=None)
Any tips/ tricks to solve this mystery would be greatly appreciated. Also happy to consider batching my computations if that should be more efficient in terms of size?
You need to make a TensorFlow sparse matrix from your SciPy one. Since your matrix seems to be in CSR format, you can do it as follows:
import numpy as np
import scipy.sparse
import tensorflow as tf
def sparse_csr_to_tf(csr_mat):
indptr = tf.constant(csr_mat.indptr, dtype=tf.int64)
elems_per_row = indptr[1:] - indptr[:-1]
i = tf.repeat(tf.range(csr_mat.shape[0], dtype=tf.int64), elems_per_row)
j = tf.constant(csr_mat.indices, dtype=tf.int64)
indices = np.stack([i, j], axis=-1)
data = tf.constant(csr_mat.data)
return tf.sparse.SparseTensor(indices, data, csr_mat.shape)
# Test
m = scipy.sparse.csr_matrix([
[0, 0, 1, 0],
[0, 0, 0, 0],
[2, 0, 3, 4],
], dtype=np.float32)
tf_mat = sparse_csr_to_tf(m)
tf.print(tf.sparse.to_dense(tf_mat))
# [[0 0 1 0]
# [0 0 0 0]
# [2 0 3 4]]
I am trying to use numpy.vectorize to iterate over a (2x5) matrix which contains two vectors representing the x- and y-values of coordinates. The coordinates (x- and y-value) are to be fed to a function returning a (1x1) vector for each iteration. So that in the end, the result should be a (1x5) vector. My problem is that instead of iterating through each element I want the algorithm to iterate through both vectors simultaneously, so it picks up the x- and y-values of the coordinates in parallel to feed it to the function.
data = np.transpose(np.array([[1, 2], [1, 3], [2, 1], [1, -1], [2, -1]]))
th_ = np.array([[1, 1]])
th0_ = -2
def positive(x, th = th_, th0 = th0_):
if signed_dist(x, th, th0)[0][0] > 0:
return np.array([[1]])
elif signed_dist(x, th, th0)[0][0] == 0:
return np.array([[0]])
else:
return np.array([[-1]])
positive_numpy = np.vectorize(positive)
results = positive_numpy(data)
Reading the numpy documentation did not really help and I want to avoid large workarounds in favor of computation timing. Thankful for any suggestion!
This is a bit of a guess, but looks like your code can be simplified to
data = np.array([[1, 2], [1, 3], [2, 1], [1, -1], [2, -1]]) # (5,2) array
th_ = np.array([[1, 1]])
th0_ = -2
alist = [signed_dist(x, th_, th0_) for x in data]
arr = np.array(alist) # (5,?,?) array
arr = arr[:,0,0] # (5,) array
arr[arr>0] = 1
(Keras 1.0.7, Tensorflow r0.10)
I am trying to implement my own activation function:
# Custom activation function (Radial Basis Function - RBF)
l2_norm = lambda a, b: K.sqrt(((a - b) ** 2).sum())
def rbf(x, gamma=1.0):
return K.exp(-1 * gamma * l2_norm(x[0], x[1]) ** 2)
Here is the relevant section of my model where I specify my custom activation function:
model = Sequential()
# Some other layers go here
model.add(Dense(n_classes, activation=rbf))
I get the following error:
/raid/home/user/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/tensor_shape.pyc in assert_has_rank(self, rank)
619 """
620 if self.ndims not in (None, rank):
--> 621 raise ValueError("Shape %s must have rank %d" % (self, rank))
622
623 def with_rank(self, rank):
ValueError: Shape (?, 12) must have rank 1
The error occurs on the line return K.exp(-1 * gamma * l2_norm(x[0], x[1]) ** 2) when trying to slice x (which has shape (?, 12)) into x[0] and x[1].
Why does the Tensorflow slice method throw this error?
As the error says, the shape (?, 12) is not of rank 1. Tensor rank (sometimes referred to as order or degree or n-dimension) is the number of dimensions of the tensor. For example, the following tensor (defined as a Python list) has a rank of 2:
t = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
In TensorFlow, the slicing operation is slightly less flexible than in python numpy. The number of dimensions specified in the slice must be equal to the rank of the tensor.
tf.slice(input, begin, size, name=None)
This operation extracts a slice of size size from a tensor input starting at the location specified by begin. The slice size is represented as a tensor shape, where size[i] is the number of elements of the 'i'th dimension of input that you want to slice. The starting location (begin) for the slice is represented as an offset in each dimension of input. In other words, begin[i] is the offset into the 'i'th dimension of input that you want to slice from.
begin is zero-based; size is one-based. If size[i] is -1, all remaining elements in dimension i are included in the slice.
In short, the operator requires that the begin and size vectors — which define the subtensor to be sliced — have the same length as the number of dimensions in input. For example, to slice a 3-D tensor, you must pass a vector (or list) of three numbers as the second and third arguments of tf.slice().
For example:
# 'input' is [[[1, 1, 1], [2, 2, 2]],
# [[3, 3, 3], [4, 4, 4]],
# [[5, 5, 5], [6, 6, 6]]]
tf.slice(input, [1, 0, 0], [1, 1, 3]) ==> [[[3, 3, 3]]]
tf.slice(input, [1, 0, 0], [1, 2, 3]) ==> [[[3, 3, 3],
[4, 4, 4]]]
tf.slice(input, [1, 0, 0], [2, 1, 3]) ==> [[[3, 3, 3]],
[[5, 5, 5]]]
So, you can modify your rbf() function as follows and it should work:
def rbf(x, gamma=1.0):
return K.exp(-1 * gamma * l2_norm(x[0, :], x[1, :]) ** 2)
N.B. I couldn't reproduce this problem with the latest version of TensorFlow. I suspect you're using one of the release candidates of TensorFlow 0.10, or an earlier version, because in the released version it is possible to write the following:
x = tf.placeholder(tf.float32, shape=[None, 12])
print(x[0].get_shape()) # ==> (12,)
print(x[1].get_shape()) # ==> (12,)
In older versions of TensorFlow, you had to specify every dimension of the slice explicitly when using the tf.slice() op or the [] Python slice operator. If you can't upgrade to the latest version (which we'd generally recommend!), the following modified version of your rbf() function should work:
def rbf(x, gamma=1.0):
return K.exp(-1 * gamma * l2_norm(x[0, :], x[1, :]) ** 2)