I'm trying to implement a loop that iterates over the rows of a tensor, retrieve the indices in each row, use them to gather vectors from another tensor and finally combine those vector in a new tensor.
The problem is that each row may contain a different number of indices (e.g. [[-1,-1,1,4,-1], [3,-1,-1,-1,-1]] first row indices: [1, 4]; second row indices [3]).
The problem rises when I use tf.while_loop or tf.scan. With the first one I don't understand how to stack all the gathered tensors together. The second one, instead, wants all the outputs to have the same shape (seems like i cannot tell that all the outputs have a general shape of [None, 10]).
Does anybody ever tried something similar?
I'm attaching the code for the while_loop:
i = tf.constant(0)
def body(i, merging):
i += 1
print('i', i)
i_row = tf.gather(dense, [i])
i_indices = tf.where(i_row > 0)[:, 1]
i_vecs = tf.gather(embeddings_ph, i_indices)
return i, i_vecs
tf.while_loop(lambda i, merging : tf.less(i, 2), body,
loop_vars=[i,merging],
shape_invariants=[i.get_shape(),
tf.TensorShape((None, 3))],
name='vecs_gathering')
What its missing here is to stack all the while_loop outputs (i_vec for each i) together in a new tensors.
Ok, got the inspiration from the rnn implementation. I modified my code as follows and now it works perfectly:
def body(i, outputs):
i_row = tf.gather(dense, [i])
i_indices = tf.where(i_row > 0)[:, 1]
i_vecs = tf.gather(embeddings_ph, i_indices)
outputs = outputs.write(i, i_vecs)
i += 1
return i, outputs
outputs = tf.TensorArray(dtype=tf.float32, infer_shape=False, size=1,
dynamic_size=True)
_, outputs = tf.while_loop(lambda i, *_: tf.less(i, 3), body,[0,outputs])
outputs = outputs.concat()
I want also to stress the fact that you MUST reassign the value of the TensorArray when you perform a write (otherwise tf will complain a lot about the fact you are not using the array you declared)
Related
Using Tensorflow in Python, I want to use the shape of a placeholder for the bound of a for-loop. However, when I try to do this, I get the error: 'Tensor' object cannot be interpreted as an integer. This shape is not a constant value across the data so we cannot use a tf.constant. How can we solve this problem?
Here is a part of our code:
def Model(M, v_a, weights, biases, d, N):
c = multilayer_perceptron((tf.slice(M, [0, 0], ([d, 1]))), v_a, weights,
biases)
for i in range(1, N):
c = tf.concat([c, multilayer_perceptron((tf.slice(M, [0, i], [d, 1])),
v_a, weights, biases)], axis=0)
alpha = tf.nn.softmax(tf.reshape(c, [-1]))
v_ns = tf.matmul(M, tf.reshape(alpha, [N, 1]))
layer_3 = tf.add(tf.matmul(weights['W4'], v_ns), biases['b2'])
v_ms = tf.nn.tanh(layer_3)
layer_4 = tf.add(tf.matmul(weights['W5'], v_ms), biases['b3'])
pred = tf.nn.softmax(tf.reshape(layer_4, [-1]))
return pred
M = tf.placeholder(tf.float32)
I need N to be an integer, but it needs to be equal to the number of columns of the placeholder M, which is not constant across training examples.
EDIT:
Seems I didn't understand the question correctly in first instance. I'll leave the original answer because it's related and just in case someone finds it useful.
In any case, if you want to use a dimension of a tensor as the number of iterations in a loop, then the value of the dimension must be known statically (cannot be None). You would simply do something like this:
for i in range(my_tensor.shape[i_dim].value):
# loop body...
Where i_dim is the dimension you want to iterate. Again, if you don't know the size of the tensor dimension, you would need to resort to tf.while_loop:
def body(dim, i):
# loop body...
return dim, i + 1
cond = lambda dim, i: dim > i
tf.while_loop(cond, body, [tf.shape(my_tensor)[i_dim], 0])
How or whether you can do that at all depends on how much you know about the tensor at the time of looping. If the shape is fully known, you can simply do:
for dim in my_tensor.shape.as_list():
# loop body...
Here, dim will be regular Python integers for known dimensions. However, if there are unknown dimensions they will be read as None. If you don't know the exact dimensions, but you know the rank (i.e. the number of dimensions) of the tensor, you can do something like this:
shape = tf.shape(my_tensor)
for i in range(my_tensor.shape.ndims):
dim = shape[i]
# loop body...
In this case, dim will be TensorFlow values holding the tensor dimensions, so you would only be able to use it to compute other tensors, but all the dimensions are guaranteed to be defined.
Finally, if you don't even know the number of dimensions in the tensor, you will not be able to make a regular loop with the shapes. If anything, you could use tf.while_loop to do something like what you need:
def body(shape, i):
dim = shape[i]
# loop body...
return shape, i + 1
cond = lambda shape, i: tf.shape(shape)[0] > i
tf.while_loop(cond, body, [tf.shape(my_tensor), 0])
Here's the first part of my function that wants to generate text from a trained LSTM and a word embedding of dim 50. The problem comes when I try to set row i of X equal to the embedding vector y_embed. However, that problem only comes up on the third iteration of the for loop. That's strange to me, because I'd expect every row of X to have the same shape.
def generate_text(my_model, length):
ix = np.random.randint(VOCAB_SIZE) #start generating by some
random index
y_word = [reverse_dictionary[ix]] #get the word with that index
y_embed = w2vec[ix] #get the embedding vector
print(y_embed.shape)
X = np.zeros((1, length, EMBED_DIM)) #make our numpy array
print(X[0,2].shape)
for i in range(length): #however many words we want
print("i is "+str(i))
X[0, i] = y_embed #current row of X is current word embedding
y_embed = my_model.predict(X[:, :i+1, :])[0]
#input what we've generated so far, model.predict gives us a list, take the first one
#we'll add it to our input on the next loop iteration
y_word.append(vec2w(y_embed)) #lookup the word by its embedding
The for loop works for its first two iterations, and then throws this error when i=2:
X[0, i] = y_embed #current row of X is current word embedding
ValueError: could not broadcast input array from shape (2,50) into shape (50)
So that's why I have it print the shape of y_embed and X[0,2] beforehand, and the console prints:
(50,)
(50,)
So as far as I can tell, they DO have the same shape. I'm still pretty new to numpy, so maybe it's something obvious, but I can't figure this one out. I should add that I'm using Keras, and model.predict expects a 3D tensor, which is why X is defined the way it is. I also tried setting X[0,i,:] = y_embed but that produced the same error at the same time.
X = np.zeros((1, length, EMBED_DIM))
X is 3d.
X[0, i]
selects on first 2 dir, so it is (EMBED_DIM,), which according to the error is (50,).
The error thinks y_embed is (2,50), 2 columns of 50. Apparently it was created by the last iteration.
my_model.predict(X[:, :i+1, :])[0]
With i==1, it is giving predict X[:,:2,:], a (2,50) array. I don't know what predict does, but I don't think it's a coincidence that the output has the same shape as the input.
New to TensorFlow,Now I need to use the value in placeholder before feeding, something like:
tensor = tf.placeholder(tf.float32, [None, 3])
Mat_items = tf.Variable(tf.random_normal([10,10]))
Mat_users = tf.Variable(tf.random_normal([10,10]))
item_index = tensor[:, 0]
user_index = tensor[:, 1]
rating = tensor[:, 2]
Val = Mat_items[item_index, :]-Mat_users[user_index, :]
while tensor is a placeholder with N rows 3 cols, first col and second col is the index to Mat_items and Mat_users respectively and Mat_items Mat_users are the Variables needed to be indexed.
Running it will absolutely throw an error because item_index, user_index both are Tensor not numeric before feeding.
So I wonder whether Tensorflow could realize this demand?
Any suggestion will be appreciate!:)
=========================================================================
Addition to my question:
Val depends on certain columns in Tensor just like first column and second columns. So when i create my graph, i code
Val = Mat_items[item_index, :]-Mat_users[user_index, :]
item_index and user_index are the slice of tensor, and both of them are type tensor too.It will throw error.I dont know how to realize this demand in TensorFlow.
=========================================================================
Have found a solution:
tensor = tf.placeholder(tf.float32, [None, 3])
Mat_items = tf.Variable(tf.random_normal([10,10]))
Mat_users = tf.Variable(tf.random_normal([10,10]))
for each in range(batch_number):
item_ind, user_ind = tensor[each, 2], tensor[each, 1]
rating = tensor[each, 1]
Val = Mat_item[item_ind, 0]*Mat_user[user_ind, 0]*rating
Code above seems to work while building the gragh cost too much time even with a litte dataset(batch-size about 1000 and only one batch is feeded for testing), It will roughly cost 78 second to build the graph, I are not sure if it's normal?
Your question seems vague, but I guess your question is how to place values into placeholder. If you want to obtain the output from Val, you have to supplement tensor with an input, as it does not contain any values by default. Val also depends on tensor in order to compute its output. Your input has to be of the same size as tensor. (In this case, lets assume your input is random noise input_tensor = numpy.random.uniform(-1, 1, size =(None, 3)) where None is a value you have to specify.
So after you begin a session, execute
output = sess.run(Val, feed_dict = {tensor: input_tensor})
and output will be your result
Good afternoon.
I continue to have issues with updating random elements in tensorflow by index.
I want to randomly choose indices (half of all, for instance), and then set to zero elements correspond to that indices.
Here's the problematic part:
with tf.variable_scope("foo", reuse=True):
temp_var = tf.get_variable("W")
size_2a = tf.get_variable("b")
s1 = tf.shape(temp_var).eval()[0]
s2 = tf.shape(size_2a).eval()[0]
row_indices = tf.random_uniform(dtype=tf.int32, minval=0, maxval = s1 - 1, shape=[s1]).eval()
col_indices = tf.random_uniform(dtype=tf.int32, minval=0, maxval = s2 - 1, shape=[s2]).eval()
ones_mask = tf.ones([s1,s2])
# turn 'ones_mask' into 1d variable since "scatter_update" supports linear indexing only
ones_flat = tf.Variable(tf.reshape(ones_mask, [-1]))
# no automatic promotion, so make updates float32 to match ones_mask
updates = tf.zeros(shape=(s1,), dtype=tf.float32)
# get linear indices
linear_indices = row_indices*s2 + tf.reshape(col_indices,s1*s2)
ones_flat = tf.scatter_update(ones_flat, linear_indices/2, updates)
#I want to set to zero only half of all elements,that's why linear_indices/2
# convert back into original shape
ones_mask = tf.reshape(ones_flat, ones_mask.get_shape())
It gives me ValueError: Cannot reshape a tensor with 10 elements to shape [784,10] (7840 elements) for 'foo_1/Reshape_1' (op: 'Reshape') with input shapes: [10], [2]., but I don't know how to be here without reshaping (I tried to reshape to both s1 and s2, no use)
I have already read these topics:Update values of a matrix variable in tensorflow, advanced indexing (feed_dict doesn't seem to work in my case), python numpy ValueError: operands could not be broadcast together with shapes and practically everything on the subject on stackoverflow =(
I have trained per-pixel models on many images and want to evaluate them on new images.
What I'd like to do is for each image of shape (N, M, 3), apply a function in this fashion:
myfunc(array[i, j, :], i, j)
# Takes (3,1) input and indices
def myfunc(input, i, j):
ret1, ret2 = model[i,j].predict(input)
# returns a single float value
return ret1[1]
where i, j are indices, where myfunc will look up the correct model parameters to apply. If it helps, model can be a numpy array of objects, with the same dimensions as the original input's first two dimensions (N, M)
I was looking at ufuncs and vectorize and wasn't really sure if they did what I wanted. Is there a provided interface for doing this, or will I have to loop through the array myself (ugly and possibly slower as it is in python).
Alternatively, what about applying the same function to each value?
e.g.
myfunc(array[i, j, :])
# Takes (3,1) input
def myfunc(input):
ret1, ret2 = model.predict(input)
# returns a single float value
return ret1[1]