So the pseucode of thing i want is:
splitted_outputs = [tf.split(output, rate, axis=0) for output in outputs]
where outputs is Tensor of shape (512, ?, 128), and splitted_outputs is list of lists of Tensors or Tensor with 3 dimensions. So i can iterate such tensor tensorflow.
I've tried to use tf.map_fn:
splitted_outputs = tf.map_fn(
lambda output: tf.split(output, rate, axis=0),
outputs,
dtype=list
)
but it's not possible cause list is not legal tf dtype.
You can use tf.unstack on outputs to get a list of "subtensors", then use tf.split on each of those:
splitted_outputs = [tf.split(output, rate, axis=0) for output in tf.unstack(outputs, axis=0)]
Note that tf.unstack can only be used like that when the size of the given axis is known, or otherwise you would need to provide a num parameter.
Related
Imagine that you have a batch of two lists of embeddings of the same dimension 4. Each sequence has a different length. You want to compute a function of the two sequences for each pair in the batch. To this end, you write something like
import tensorflow as tf
# suppose the batch size is 1 for simplicity
a = tf.random.normal((1, 100, 4))
b = tf.random.normal((1, 57, 4))
batch = tf.ragged.stack((a, b), axis=1)
This works okay, yielding a tensor with shape batch.shape == TensorShape([1, None, None, 4]). Now, why is the following an IndexError?
batch[0][0]
I am trying to resize an image by a factor during graph construction when the image size is unknown:
H, W, _ = img.get_shape()
scale = tf.random_uniform([1], minval=1, maxval=1.5, dtype=tf.float32, seed=None, name=None)
Out of these I need to magically compute a size which would translate to the following:
tf.image.resize_images(tf.expand_dims(img, 0), [H*scale, W*scale])
Which returns:
ValueError: 'size' must be a 1-D int32 Tensor
Any help is greatly appreciated. Thank you.
Your scale has shape (1,), so [H*scale, W*scale] has shape (1, 2). To fix it, just generate a scalar random instead of a 1-element vector:
scale = tf.random_uniform([], minval=1, maxval=1.5, dtype=tf.float32, seed=None, name=None)
Note the shape is an empty list, meaning you want a scalar.
In addition to that, you have to fix the data type of the size parameter, like this:
tf.image.resize_images(tf.expand_dims(img, 0), tf.cast([H*scale, W*scale], tf.int32))
Your problem is that you're mixing the python list [H*scale, W*scale] with tensors. By default, tensorflow will convert lists like this to tensorflow constants as appropriate. But in this case, your list contains tensors and you're ending up with a nested set of tensors that aren't 1D.
To avoid confusion in cases like this stop using python constructs such as the list and convert your height and width into tensorflow constructs explicitly and check their shape before proceeding.
x = tf.concat((scale*W, scale*H), axis=0)
print(x)
Tensor("concat_3:0", shape=(2,), dtype=float32)
Doing so shows us that we now have a 1D tensor as required. But it's float32 (at least it was in the simple test case I set up), so let's cast that to an int:
x = tf.cast(x, tf.int32)
Now you're ready to go
tf.image.resize_images(tf.expand_dims(img, 0), size=x)
No error should occur there.
I have several tensors:
logits: This tensor contains the final prediction scores.
tf.Tensor 'MemN2N_1/MatMul_3:0' shape=(?, 18230) dtype=float32
The final prediction is computed as predicted_op = tf.argmax(logits, 1, name="predict_op")
Now I want to restrict the prediction into some specific columns. The following two tensors contains column indices I want to choose from.
self._stories is of type
tf.Tensor 'stories:0' shape=(?, 12, 110) dtype=int32
self._queries is of type
tf.Tensor 'queries:0' shape=(?, 110) dtype=int32
Here the 110 columns are the index numbers I want to restrict logits into. For example, if logits = [[10,20,30,40,50], [10,20,30,40,50]..] and self._stories = [[[1,4,...], [1,2,4,...],...], [[0,4,...],[2,4...],...]...] and self._queries = [[1,4...],[2,4,...],...] then logits should look like [[20,30,50],[10,30,50]...]
How can I do this kind of index filtering in tensorflow?
Did you try out tf.equal? This compares two tensors and creates a new tensor containing True where equal, and False where not equal.
With this bool-tensor you feed tf.select, which choses from one tensor or another, element-wise, depending on the bool-value you created in step one.
Didn't look deeply into the particular shapes you provided, but with those two ops you can create the kind of flow you are asking for.
Try tf.gather.
row_indices = [1]
row = tf.gather(tf.constant([[1, 2],[3, 4]]), row_indices)
tf.Session().run(row) # returns [[3, 4]]
You can remove the leading dimension of size 1 using tf.squeeze:
row = tf.squeeze(row, squeeze_dims=0)
You could use
tf.slice(input_, begin, size, name=None).
See the TensorFlow documentation for more details:
https://www.tensorflow.org/api_docs/python/tf/slice
I am using dynamic_rnn to process MNIST data:
# LSTM Cell
lstm = rnn_cell.LSTMCell(num_units=200,
forget_bias=1.0,
initializer=tf.random_normal)
# Initial state
istate = lstm.zero_state(batch_size, "float")
# Get lstm cell output
output, states = rnn.dynamic_rnn(lstm, X, initial_state=istate)
# Output at last time point T
output_at_T = output[:, 27, :]
Full code: http://pastebin.com/bhf9MgMe
The input to the lstm is (batch_size, sequence_length, input_size)
As a result the dimensions of output_at_T is (batch_size, sequence_length, num_units) where num_units=200.
I need to get the last output along the sequence_length dimension. In the code above, this is hardcoded as 27. However, I do not know the sequence_length in advance as it can change from batch to batch in my application.
I tried:
output_at_T = output[:, -1, :]
but it says negative indexing is not implemented yet, and I tried using a placeholder variable as well as a constant (into which I could ideally feed the sequence_length for a particular batch); neither worked.
Any way to implement something like this in tensorflow atm?
Have you noticed that there are two outputs from dynamic_rnn?
Output 1, let's call it h, has all outputs at each time steps (i.e. h_1, h_2, etc),
Output 2, final_state, has two elements: the cell_state, and the last output for each element of the batch (as long as you input the sequence length to dynamic_rnn).
So from:
h, final_state= tf.dynamic_rnn( ..., sequence_length=[batch_size_vector], ... )
the last state for each element in the batch is:
final_state.h
Note that this includes the case when the length of the sequence is different for each element of the batch, as we are using the sequence_length argument.
This is what gather_nd is for!
def extract_axis_1(data, ind):
"""
Get specified elements along the first axis of tensor.
:param data: Tensorflow tensor that will be subsetted.
:param ind: Indices to take (one for each element along axis 0 of data).
:return: Subsetted tensor.
"""
batch_range = tf.range(tf.shape(data)[0])
indices = tf.stack([batch_range, ind], axis=1)
res = tf.gather_nd(data, indices)
return res
In your case (assuming sequence_length is a 1-D tensor with the length of each axis 0 element):
output = extract_axis_1(output, sequence_length - 1)
Now output is a tensor of dimension [batch_size, num_cells].
output[:, -1, :]
works with Tensorflow 1.x now!!
Most answers cover it thoroughly, but this code snip might help understand what's really being returned by the dynamic_rnn layer
=> Tuple of (outputs, final_output_state).
So for an input with max sequence length of T time steps outputs is of the shape [Batch_size, T, num_inputs] (given time_major=False; default value) and it contains the output state at each timestep h1, h2.....hT.
And final_output_state is of the shape [Batch_size,num_inputs] and has the final cell state cT and output state hT of each batch sequence.
But since the dynamic_rnn is being used my guess is your sequence lengths vary for each batch.
import tensorflow as tf
import numpy as np
from tensorflow.contrib import rnn
tf.reset_default_graph()
# Create input data
X = np.random.randn(2, 10, 8)
# The second example is of length 6
X[1,6:] = 0
X_lengths = [10, 6]
cell = tf.nn.rnn_cell.LSTMCell(num_units=64, state_is_tuple=True)
outputs, states = tf.nn.dynamic_rnn(cell=cell,
dtype=tf.float64,
sequence_length=X_lengths,
inputs=X)
result = tf.contrib.learn.run_n({"outputs": outputs, "states":states},
n=1,
feed_dict=None)
assert result[0]["outputs"].shape == (2, 10, 64)
print result[0]["outputs"].shape
print result[0]["states"].h.shape
# the final outputs state and states returned must be equal for each
# sequence
assert(result[0]["outputs"][0][-1]==result[0]["states"].h[0]).all()
assert(result[0]["outputs"][-1][5]==result[0]["states"].h[-1]).all()
assert(result[0]["outputs"][-1][-1]==result[0]["states"].h[-1]).all()
The final assertion will fail as the final state for the 2nd sequence is at 6th time step ie. the index 5 and the rest of the outputs from [6:9] are all 0s in the 2nd timestep
I am new to Stackoverflow and cannot comment yet so I am writing this new answer. #VM_AI, the last index is tf.shape(output)[1] - 1.
So, reusing your answer:
# Let's first fetch the last index of seq length
# last_index would have a scalar value
last_index = tf.shape(output)[1] - 1
# Then let's reshape the output to [sequence_length,batch_size,num_units]
# for convenience
output_rs = tf.transpose(output,[1,0,2])
# Last state of all batches
last_state = tf.nn.embedding_lookup(output_rs,last_index)
This works for me.
You should be able to access the shape of your output tensor using tf.shape(output). The tf.shape() function will return a 1d tensor containing the sizes of the output tensor. In your example, this would be (batch_size, sequence_length, num_units)
You should then be able to extract the value of output_at_T as output[:, tf.shape(output)[1], :]
There is a function in TensorFlow tf.shape that allows you to get the symbolic interpretation of shape rather than None being returned by output._shape[1]. And after fetching the last index you can lookup by using tf.nn.embedding_lookup, which is recommended especially when the data to be fetched is high as this does parallel lookup 32 by default.
# Let's first fetch the last index of seq length
# last_index would have a scalar value
last_index = tf.shape(output)[1]
# Then let's reshape the output to [sequence_length,batch_size,num_units]
# for convenience
output_rs = tf.transpose(output,[1,0,2])
# Last state of all batches
last_state = tf.nn.embedding_lookup(output_rs,last_index)
This should work.
Just to clarify what #Benoit Steiner said. His solution would not work as tf.shape would return symbolic interpretation of the shape value, and such cannot be used for slicing tensors i.e., direct indexing
Now the max pooling function in tensorflow is
tf.nn.max_pool(value, ksize, strides, padding, name=None)
Returns:
A Tensor with type tf.float32. The max pooled output tensor.
I would like to have an extend version of max_pool, like
tf.nn.top_k_pool(value, ksize, strides, padding, k=1, name=None)
Performs the top k pooling on the input.
Args:
value: A 4-D Tensor with shape [batch, height, width, channels] and type tf.float32.
ksize: A list of ints that has length >= 4. The size of the window for each dimension of the input tensor.
strides: A list of ints that has length >= 4. The stride of the sliding window for each dimension of the input tensor.
padding: A string, either 'VALID' or 'SAME'. The padding algorithm.
k: 0-D int32 Tensor. Number of top elements to look in each pool.
name: Optional name for the operation.
Returns:
A Tensor with type tf.float32. The max pooled output tensor. There will be an additional dimension saving the top k values.
I know that I can expend the tensorflow operation following https://www.tensorflow.org/versions/r0.7/how_tos/adding_an_op/index.html
I would like to know if there is an easier way to achieve that.
Here is a function to using top_kto take the max k activations of the channels. You can modify it to fit your purpose:
def make_sparse_layer(inp_x,k, batch_size=None):
in_shape = tf.shape(inp_x)
d = inp_x.get_shape().as_list()[-1]
matrix_in = tf.reshape(inp_x, [-1,d])
values, indices = tf.nn.top_k(matrix_in, k=k, sorted=False)
out = []
vals = tf.unpack(values, axis=0, num=batch_size)
inds = tf.unpack(indices, axis=0, num=batch_size)
for i, idx in enumerate(inds):
out.append(tf.sparse_tensor_to_dense(tf.SparseTensor(tf.reshape(tf.cast(idx,tf.int64),[-1,1]),vals[i], [d]), validate_indices=False ))
shaped_out = tf.reshape(tf.pack(out), in_shape)
return shaped_out
Your best bet is probably the TopK op: https://www.tensorflow.org/versions/r0.7/api_docs/python/nn.html#top_k
usd tf.reshape(), tf.matrix_transpose(), tf.nn.top_k(sorted=False), and the 'data_format' argument in tf.nn.conv2d(), see http://www.infocool.net/kb/OtherCloud/201703/318346.html for details