(Keras 1.0.7, Tensorflow r0.10)
I am trying to implement my own activation function:
# Custom activation function (Radial Basis Function - RBF)
l2_norm = lambda a, b: K.sqrt(((a - b) ** 2).sum())
def rbf(x, gamma=1.0):
return K.exp(-1 * gamma * l2_norm(x[0], x[1]) ** 2)
Here is the relevant section of my model where I specify my custom activation function:
model = Sequential()
# Some other layers go here
model.add(Dense(n_classes, activation=rbf))
I get the following error:
/raid/home/user/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/tensor_shape.pyc in assert_has_rank(self, rank)
619 """
620 if self.ndims not in (None, rank):
--> 621 raise ValueError("Shape %s must have rank %d" % (self, rank))
622
623 def with_rank(self, rank):
ValueError: Shape (?, 12) must have rank 1
The error occurs on the line return K.exp(-1 * gamma * l2_norm(x[0], x[1]) ** 2) when trying to slice x (which has shape (?, 12)) into x[0] and x[1].
Why does the Tensorflow slice method throw this error?
As the error says, the shape (?, 12) is not of rank 1. Tensor rank (sometimes referred to as order or degree or n-dimension) is the number of dimensions of the tensor. For example, the following tensor (defined as a Python list) has a rank of 2:
t = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
In TensorFlow, the slicing operation is slightly less flexible than in python numpy. The number of dimensions specified in the slice must be equal to the rank of the tensor.
tf.slice(input, begin, size, name=None)
This operation extracts a slice of size size from a tensor input starting at the location specified by begin. The slice size is represented as a tensor shape, where size[i] is the number of elements of the 'i'th dimension of input that you want to slice. The starting location (begin) for the slice is represented as an offset in each dimension of input. In other words, begin[i] is the offset into the 'i'th dimension of input that you want to slice from.
begin is zero-based; size is one-based. If size[i] is -1, all remaining elements in dimension i are included in the slice.
In short, the operator requires that the begin and size vectors — which define the subtensor to be sliced — have the same length as the number of dimensions in input. For example, to slice a 3-D tensor, you must pass a vector (or list) of three numbers as the second and third arguments of tf.slice().
For example:
# 'input' is [[[1, 1, 1], [2, 2, 2]],
# [[3, 3, 3], [4, 4, 4]],
# [[5, 5, 5], [6, 6, 6]]]
tf.slice(input, [1, 0, 0], [1, 1, 3]) ==> [[[3, 3, 3]]]
tf.slice(input, [1, 0, 0], [1, 2, 3]) ==> [[[3, 3, 3],
[4, 4, 4]]]
tf.slice(input, [1, 0, 0], [2, 1, 3]) ==> [[[3, 3, 3]],
[[5, 5, 5]]]
So, you can modify your rbf() function as follows and it should work:
def rbf(x, gamma=1.0):
return K.exp(-1 * gamma * l2_norm(x[0, :], x[1, :]) ** 2)
N.B. I couldn't reproduce this problem with the latest version of TensorFlow. I suspect you're using one of the release candidates of TensorFlow 0.10, or an earlier version, because in the released version it is possible to write the following:
x = tf.placeholder(tf.float32, shape=[None, 12])
print(x[0].get_shape()) # ==> (12,)
print(x[1].get_shape()) # ==> (12,)
In older versions of TensorFlow, you had to specify every dimension of the slice explicitly when using the tf.slice() op or the [] Python slice operator. If you can't upgrade to the latest version (which we'd generally recommend!), the following modified version of your rbf() function should work:
def rbf(x, gamma=1.0):
return K.exp(-1 * gamma * l2_norm(x[0, :], x[1, :]) ** 2)
Related
I was recently given task (during exam, not funny) to create function returning cumulative sum along given dimension (input: 2d array), without use of np.cumsum ofc; to be honest i find this quite hard to even start with.
function should look like this:
def cumsum_2d(array : np.ndarray, dim : int = 0) -> np.ndarray:
and then result is supposed to be compared with result from actual np.cumsum
I would be grateful for even basic outline or general idea what to do.
Here is another approach that doesn't use ufunc.accumulate or functools.reduce.
It works by inserting an extra dimension, broadcasting the array along that dimension, and then doing a sum where it only considers indices less than or equal to the current index along the summation dimension.
It's morally similar to a brute-force approach where you make a bunch of copies of the array, set the elements you don't want to zero, and then doing the sum.
import numpy as np
def cumsum_2d(array: np.ndarray, dim: int = 0):
# Make sure the dim argument is positive
dim = dim % array.ndim
# Calculate the new shape with an extra copy of dim
shape_new = list(array.shape)
shape_new.insert(dim + 1, array.shape[dim])
# Insert the new dimension and broadcast the array along that dimension
array = np.broadcast_to(np.expand_dims(array, dim + 1), shape_new)
# Save the indices of the array
indices = np.indices(array.shape)
# Sum along the requested dimension, considering only the elements less than the current index
return np.sum(array, axis=dim, where=indices[dim] <= indices[dim + 1])
a = np.random.random((4, 5))
assert np.array_equal(cumsum_2d(a, 1), np.cumsum(a, 1))
assert np.array_equal(cumsum_2d(a, 0), np.cumsum(a, 0))
assert np.array_equal(cumsum_2d(a, -1), np.cumsum(a, -1))
assert np.array_equal(cumsum_2d(a, -2), np.cumsum(a, -2))
Note that this function should work for arrays of any rank, not just two-dimensional ones.
This approach is fairly "from scratch". It does use functools.reduce(), which I assume must be permitted.
import functools
import numpy as np
def cumsum_2d(array: np.ndarray, dim: int = 0) -> np.ndarray:
if not isinstance(dim, int) or not 0 <= dim <= 1:
raise ValueError('"dim": expected integer 0 or 1, got {dim}.')
elif not array.ndim == 2:
raise ValueError(
f"{array.ndim} dimensional array not allowed - 2 dimensional arrays expected."
)
array = array.T if dim == 1 else array
result = [
functools.reduce(lambda x, y: x + y, array[: i + 1]) for i in range(len(array))
]
result = np.array(result)
result = result.T if dim == 1 else result
return result
a = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
dim = 1
print(f"For dim = {dim} and a= \n{a}:")
print(f"...got: \n{cumsum_2d(a, dim)}")
print(f"...expected: \n{np.cumsum(a, dim)}")
This has the result:
# For dim = 1 and a=
# [[1 2 3]
# [4 5 6]
# [7 8 9]]:
# ...got:
# [[ 1 3 6]
# [ 4 9 15]
# [ 7 15 24]]
# ...expected:
# [[ 1 3 6]
# [ 4 9 15]
# [ 7 15 24]]
Trying with dim = 1 raises ValueError per the function definition - this mimics the AxisError raised by np.cumsum under similar circumstances:
ValueError: "dim": expected integer 0 or 1, got 2.
Lastly, trying with a non 2-D array also raises an customised ValueError as programmed, ensuring the user doesn't get any silently passed unexpected behaviour.
b = np.array([[[1, 2, 3], [1, 2, 3]], [[4, 5, 6], [1, 2, 3]], [[7, 8, 9], [1, 2, 3]]])
cumsum_2d(b, dim)
Result:
ValueError: 3 dimensional array not allowed - 2 dimensional arrays expected.
I am searching for a way to do some batchwise indexing for tensors.
If I have a variable Q of size 1000, I can get the elements I want by
Q[index], where index is a vector of the wanted elements.
Now I would like to do the same for more dimensional tensors.
So suppose Q is of shape n x m and I have a index matrix of shape n x p.
My goal is to get for each of the n rows the specific p elements out of the m elements.
But Q[index] is not working for this situation.
Do you have any thoughts how to handle this?
You can seem to be a simple application of torch.gather which doesn't require any additional reshaping of the data or index tensor:
>>> Q = torch.rand(5, 4)
tensor([[0.8462, 0.3064, 0.2549, 0.2149],
[0.6801, 0.5483, 0.5522, 0.6852],
[0.1587, 0.4144, 0.8843, 0.6108],
[0.5265, 0.8269, 0.8417, 0.6623],
[0.8549, 0.6437, 0.4282, 0.2792]])
>>> index
tensor([[0, 1, 2],
[2, 3, 1],
[0, 1, 2],
[2, 2, 2],
[1, 1, 2]])
The following gather operation applied on dim=1 return a tensor out, such that:
out[i, j] = Q[i, index[i,j]]
This is done with the following call of torch.Tensor.gather on Q:
>>> Q.gather(dim=1, index=index)
tensor([[0.8462, 0.3064, 0.2549],
[0.5522, 0.6852, 0.5483],
[0.1587, 0.4144, 0.8843],
[0.8417, 0.8417, 0.8417],
[0.6437, 0.6437, 0.4282]])
I have a 2D pytorch tensor of shape n by m. I want to index the second dimension using a list of indices (which could be done with torch.gather) then then also set new values to the result of the indexing.
Example:
data = torch.tensor([[0,1,2], [3,4,5], [6,7,8]]) # shape (3,3)
indices = torch.tensor([1,2,1], dtype=torch.long).unsqueeze(-1) # shape (3,1)
# data tensor:
# tensor([[0, 1, 2],
# [3, 4, 5],
# [6, 7, 8]])
I want to select the specified indices per row (which would be [1,5,7] but then also set these values to another number - e.g. 42
I can select the desired columns row wise by doing:
data.gather(1, indices)
tensor([[1],
[5],
[7]])
data.gather(1, indices)[:] = 42 # **This does NOT work**, since the result of gather
# does not use the same storage as the original tensor
which is fine, but I would like to change these values now, and have the change also affect the data tensor.
I can do what I want to achieve using this, but it seems to be very un-pythonic:
max_index = torch.max(indices)
for i in range(0, max_index + 1):
mask = (indices == i).nonzero(as_tuple=True)[0]
data[mask, i] = 42
print(data)
# tensor([[ 0, 42, 2],
# [ 3, 4, 42],
# [ 6, 42, 8]])
Any hints on how to do that more elegantly?
What you are looking for is torch.scatter_ with the value option.
Tensor.scatter_(dim, index, src, reduce=None) → Tensor
Writes all values from the tensor src into self at the indices specified in the index tensor. For each value in src, its output index is specified by
its index in src for dimension != dim and by the corresponding value
in index for dimension = dim.
With 2D tensors as input and dim=1, the operation is:
self[i][index[i][j]] = src[i][j]
No mention of the value parameter though...
With value=42, and dim=1, this will have the following effect on data:
data[i][index[i][j]] = 42
Here applied in-place:
>>> data.scatter_(index=indices, dim=1, value=42)
>>> data
tensor([[ 0, 42, 2],
[ 3, 4, 42],
[ 6, 42, 8]])
My question is about the tensor operation in Tensorflow.
Let's say:
import tensorflow as tf
import numpy as np
a = tf.Variable(np.random.random([10, 3, 3]))
b = tf.Variable(np.random.random([10, 3, 3]))
def some_function(m,n):
# just as an example
return tf.add(m, n)
This works in Tensorflow but it requires to know the dimension in advanced. However, it is very likely that the first dimension of the Tensor is None.
c = []
for i in range(10):
c.append(some_function(a[i], b[i]))
c = tf.stack(c)
So I wonder if there is a zip-like function in Tensorflow? Then we can do:
# TypeError: zip argument #1 must support iteration
c = []
for i, j in zip(a,b):
c.append(some_function(i,j))
c = tf.stack(c)
Maybe we can use some function like tf.map_fn or tf.scan? But I am not sure. Really thank you, guys.
Tensor objects are not iterable, which explains why your third code sample fails. So, to answer your question, there is no zip-like function in TensorFlow.
You can indeed use tf.map_fn to apply a function to a sequence of tensors. The problem you pose in your example code can be solved in the following fashion:
def some_function(tensor):
return tf.reduce_sum(tensor)
c = tf.stack([a, b], axis=1)
d = tf.map_fn(some_function, c, dtype=tf.float32)
yields a Tensor d whose value is [20., 6., 6.].
You can use tf.transpose like this
>>> a = tf.constant([1, 2, 3])
>>> b = tf.constant([4, 5, 6])
>>> tf.transpose([a, b])
<tf.Tensor: shape=(3, 2), dtype=int32, numpy=
array([[1, 4],
[2, 5],
[3, 6]], dtype=int32)>
For those of you using JavsScript, this is #bachr's answer in tensorflow.js (node):
const a = tf.tensor([1, 3, 5, 7])
const b = tf.tensor([2, 4, 6, 8])
const zip = tf.transpose(tf.stack([a, b]))
zip.print()
// Tensor
// [[1, 2],
// [3, 4],
// [5, 6],
// [7, 8]]
I have a sparse tensor that I'm building up from a collection of indices and values. I'm trying to implement some code to take a full row slice. Although this functionality doesn't appear to be directly supported in TensorFlow, it seems that there are work arounds that can return the indices and values for a specified row as follows:
def sparse_slice(self, indices, values, needed_row_ids):
needed_row_ids = tf.reshape(needed_row_ids, [1, -1])
num_rows = tf.shape(indices)[0]
partitions = tf.cast(tf.reduce_any(tf.equal(tf.reshape(indices[:, 0], [-1, 1]), needed_row_ids), 1),
tf.int32)
rows_to_gather = tf.dynamic_partition(tf.range(num_rows), partitions, 2)[1]
slice_indices = tf.gather(indices, rows_to_gather)
slice_values = tf.gather(values, rows_to_gather)
return slice_indices, slice_values
Then called directly like so on a sparse 4x4 matrix where I am interested in accessing all of the elements in row 3:
with tf.Session().as_default():
indices = tf.constant([[0, 0], [1, 0], [2, 0], [2, 1], [3, 0], [3, 3]])
values = tf.constant([10, 19, 1, 1, 6, 5], dtype=tf.int64)
needed_row_ids = tf.constant([3])
slice_indices, slice_values = self.sparse_slice(indices, values, needed_row_ids)
print('indicies: {} and rows: {}'.format(slice_indices.eval(), slice_values.eval()))
Which outputs the following:
indicies: [[3 0]
[3 3]] and rows: [6 5]
So far so good, I then figure I can use this information to construct a 1x4 dense tensor with the values at the indexes and 0s for the missing columns.
dense_representation = tf.sparse_to_dense(sparse_values=slice_values, sparse_indices=slice_indices,
output_shape=(1,4))
However the moment I run the tensor in a session.
sess = tf.Session()
sess.run(dense_representation)
I receive the following exception:
InvalidArgumentError (see above for traceback): indices[0] = [3,0] is out of bounds: need 0 <= index < [1,4]
[[Node: SparseToDense = SparseToDense[T=DT_INT64, Tindices=DT_INT32, validate_indices=true, _device="/job:localhost/replica:0/task:0/cpu:0"](Gather_2, SparseToDense/output_shape, Gather_3, SparseToDense/default_value)]]
I'm not quite sure what I'm doing wrong or if this has something to do with the output_shape not being properly formed. Essentially I'd like to stuff this all back into a 1 x 4 vector. I haven't been able to find any good examples online for how to do this. Any help would be appreciated.