Develop Reference

Why tf.argmax() is returning wrong indexes?

I am trying to get the maximum indexes of logits using tf.argmax() function. My code is show below:
import tensorflow as tf
import numpy as np
logits = tf.random_uniform([1,3,3,21], maxval=255, dtype=tf.float32, seed=0)
logits = logits / tf.norm(logits)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
sess.run(tf.local_variables_initializer())
logits_eval = sess.run(logits)
logits_argmax_np = np.argmax(logits_eval, axis=-1)
ypredT = tf.argmax(logits, axis=-1)
logits_argmax_tf = ypredT.eval()
I can get the right indexes using np.argmax(), but I don't whytf.argmax() is returning wrong indexes.
Thanks a lot in advance.
Edit: I am using tensorflow 1.13

The issue here is that each time you call sess.run you are instigating a separate execution of the session, from the bottom up. Since there are random numbers generated this will not produce the same result in each run, hence your argmax for each run is different. But they are doing the same thing.
To see this you can get both argmaxs from the same session execution, using square brackets to get both the ypredT tensor and the logits tensors from the same sess.run:
# tensorflow graph
logits = tf.random_uniform([1,3,3,21], maxval=255, dtype=tf.float32, seed=0)
logits = logits / tf.norm(logits)
ypredT = tf.argmax(logits, axis=-1) # tensorflow argmax
# run session
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
logits_eval, logits_argmax_tf = sess.run([logits, ypredT])
# after session has closed
logits_argmax_np = np.argmax(logits_eval, axis=-1) # get numpy argmax
print(logits_argmax_np)
print(logits_argmax_tf)
output:
[[[14 0 4]
[ 0 8 0]
[10 12 3]]]
[[[14 0 4]
[ 0 8 0]
[10 12 3]]]

Input to tensorflow in_top_k should be rank 1 or rank 2?

I try to experiment with in_top_k function to see what exactly this function is doing. But I found some really confusing behavior.
First I coded as follows
import numpy as np
import tensorflow as tf
target = tf.constant(np.random.randint(2, size=30).reshape(30,-1), dtype=tf.int32, name="target")
pred = tf.constant(np.random.rand(30,1), dtype=tf.float32, name="pred")
result = tf.nn.in_top_k(pred, target, 1)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
targetVal = target.eval()
predVal = pred.eval()
resultVal = result.eval()
Then it generates the following error:
ValueError: Shape must be rank 1 but is rank 2 for 'in_top_k/InTopKV2' (op: 'InTopKV2') with input shapes: [30,1], [30,1], [].
Then I changed my code to
import numpy as np
import tensorflow as tf
target = tf.constant(np.random.randint(2, size=30), dtype=tf.int32, name="target")
pred = tf.constant(np.random.rand(30,1).reshape(-1), dtype=tf.float32, name="pred")
result = tf.nn.in_top_k(pred, target, 1)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
targetVal = target.eval()
predVal = pred.eval()
resultVal = result.eval()
But now the error becomes
ValueError: Shape must be rank 2 but is rank 1 for 'in_top_k/InTopKV2' (op: 'InTopKV2') with input shapes: [30], [30], [].
So should the input be rank 1 or rank 2?

For in_top_k, the targets need to be rank 1 (class indices) and the predictions rank 2 (scores for each class). This can be seen from the docs easily.
This means that the two error messages actually complain about different inputs each time (targets the first time and predictions the second time), which funnily enough isn't mentioned in the messages at all... Either way, the following snippet should be more like it:
import numpy as np
import tensorflow as tf
target = tf.constant(np.random.randint(2, size=30), dtype=tf.int32, name="target")
pred = tf.constant(np.random.rand(30,1), dtype=tf.float32, name="pred")
result = tf.nn.in_top_k(pred, target, 1)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
targetVal = target.eval()
predVal = pred.eval()
resultVal = result.eval()
Here, we basically combine the "best of both snippets": Predictions from the first one and targets from the second one. However, the way I understand the docs, even for binary classification we need two values for the predictions, one for each class. So something like
import numpy as np
import tensorflow as tf
target = tf.constant(np.random.randint(2, size=30), dtype=tf.int32, name="target")
pred = tf.constant(np.random.rand(30,1), dtype=tf.float32, name="pred")
pred = tf.concat((1-pred, pred), axis=1)
result = tf.nn.in_top_k(pred, target, 1)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
targetVal = target.eval()
predVal = pred.eval()
resultVal = result.eval()

How to save and restore a lstm trained model in Tensorflow using Saver?

I have saved a trained LSTM model and I want to restore the prediction to use it in testing. I was trying to follow this post. But I am getting errors. Here is what I tried:
x = tf.placeholder('tf.float32', [None, input_vec_size, 1])
y = tf.placeholder('tf.float32')
def recurrent_neural_network(x):
layer = {'weights': tf.Variable(tf.random_normal([n_hidden, n_classes])),
'biases': tf.Variable(tf.random_normal([n_classes]))}
x = tf.transpose(x, [1, 0, 2])
x = tf.reshape(x, [-1, 1])
x = tf.split(x, input_vec_size, 0)
lstm_cell = rnn.BasicLSTMCell(n_hidden, state_is_tuple=True)
outputs, states = rnn.static_rnn(lstm_cell, x, dtype=tf.float32)
output = tf.add(tf.matmul(outputs[-1], layer['weights']), layer['biases'])
return output
def train_neural_network(x):
prediction = recurrent_neural_network(x)
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=prediction, labels=y))
optimizer = tf.train.AdamOptimizer(learning_rate=0.001).minimize(cost)
saver = tf.train.Saver()
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
Training ...
saver.save(sess, os.path.join(os.getcwd(), 'my_test_model'))
After that, in the training phase I am trying
def test_neural_network(input_data):
with tf.Session() as sess:
#sess.run(tf.global_variables_initializer())
new_saver = tf.train.import_meta_graph('my_test_model.meta')
new_saver.restore(sess, tf.train.latest_checkpoint('./'))
prediction = tf.get_default_graph().get_tensor_by_name("prediction:0")
Calculate features from input_data ...
result = sess.run(tf.argmax(prediction.eval(feed_dict={x: features}), 1))
But this throws the following error:
KeyError: "The name 'prediction:0' refers to a Tensor which does not exist. The operation, 'prediction', does not exist in the graph."
Then I tried adding :
tf.add_to_collection('prediction', prediction) before saving and replacing by prediction = tf.get_collection('prediction')[0] after restoring. But this gives me the following error:
InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'Placeholder_2' with dtype float and shape [?,34,1]
[[Node: Placeholder_2 = Placeholderdtype=DT_FLOAT, shape=[?,34,1], _device="/job:localhost/replica:0/task:0/cpu:0"]]
I know for the first error, I am supposed to assign a name in order to restore but prediction is not a tensorflow variable. I went through few previous posts and articles but unable to come up with a working solution. So, my questions are:
Am I doing something conceptually wrong? If so, what?
If not, is there an implementation error? And how do I solve it?
Thanks.

I could save my trained model at last and so posting an answer in case anyone comes across this question. I did not get a solution for the exact problem but I could build and save my model using tflearn. In order to train and store:
model = tflearn.DNN(lstm_model(n_classes, input_vec_size))
model.fit(train_x, train_y, validation_set=(test_x, test_y), n_epoch=20,
show_metric=True, snapshot_epoch=True, run_id='lstm_model')
model.save("../Models/lstm_model")
And later, to restore:
model.load(filepath+"lstm_model")
This seemed to be a far easier way to work with the model, and provides a compact and novel way to do the same task which I posted in the question.

Tensorflow: No gradients provided for any variable

I am new to tensorflow and I am building a network but failing to compute/apply the gradients for it. I get the error:
ValueError: No gradients provided for any variable: ((None, tensorflow.python.ops.variables.Variable object at 0x1025436d0), ... (None, tensorflow.python.ops.variables.Variable object at 0x10800b590))
I tried using a tensorboard graph to see if there`s was something that made it impossible to trace the graph and get the gradients but I could not see anything.
Here`s part of the code:
sess = tf.Session()
X = tf.placeholder(type, [batch_size,feature_size])
W = tf.Variable(tf.random_normal([feature_size, elements_size * dictionary_size]), name="W")
target_probabilties = tf.placeholder(type, [batch_size * elements_size, dictionary_size])
lstm = tf.nn.rnn_cell.BasicLSTMCell(lstm_hidden_size)
stacked_lstm = tf.nn.rnn_cell.MultiRNNCell([lstm] * number_of_layers)
initial_state = state = stacked_lstm.zero_state(batch_size, type)
output, state = stacked_lstm(X, state)
pred = tf.matmul(output,W)
pred = tf.reshape(pred, (batch_size * elements_size, dictionary_size))
# instead of calculating this, I will calculate the difference between the target_W and the current W
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(target_probabilties, pred)
cost = tf.reduce_mean(cross_entropy)
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
sess.run(optimizer, feed_dict={X:my_input, target_probabilties:target_prob})
I will appreciate any help on figuring this out.

I always have the tf.nn.softmax_cross_entropy_with_logits() used so that I have the logits as first argument and the labels as second. Can you try this?

Cannot feed value of shape (500,) for Tensor 'x_17:0', which has shape '(?, 500)'

I'm just learning TensorFlow, so sorry if this is obvious. I've checked the documentation and experimented quite a bit and I just can't seem to get this to work.
def train_network():
OUT_DIMS = 1
FIN_SIZE = 500
x = tf.placeholder(tf.float32, [OUT_DIMS, FIN_SIZE], name="x")
w = tf.Variable(tf.zeros([FIN_SIZE, OUT_DIMS]), name="w")
b = tf.Variable(tf.zeros([OUT_DIMS]), name="b")
y = tf.tanh(tf.matmul(x, w) + b)
yhat = tf.placeholder(tf.float32, [None, OUT_DIMS])
cross_entropy = -tf.reduce_sum(yhat*tf.log(y))
train_step = tf.train.GradientDescentOptimizer(0.01).minimize(cross_entropy)
# Launch the model
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
for this_x, this_y in yield_financials():
sess.run(train_step, feed_dict={x: this_x,
yhat: this_y})
print(end=".")
sys.stdout.flush()
yield_financials() outputs an numpy array of 500 numbers and the number that I want it to guess. I've tried shuffling OUT_DIMS and FIN_SIZE around, I tried accumulating them into batches to more closely match what the tutorial looked like, I tried setting OUT_DIMS to 0, removing it entirely, and I tried replacing None with other numbers, but have not made any progress.

Try
this_x = np.reshape(this_x,(1, FIN_SIZE))
sess.run(train_step, feed_dict={x: this_x,
yhat: this_y})

I had the same problem and I solved this problem.I hope that it's helpful for u.
firstly,I transformed load data into :
train_data = np.genfromtxt(train_data1, delimiter=',')
train_label = np.transpose(train_label1, delimiter=',')
test_data = np.genfromtxt(test_data1, delimiter=',')
test_label = np.transpose(test_label1, delimiter=',')
Then,transformed trX, trY, teX, teY data into:
# convert the data
trX, trY, teX, teY = train_data,train_label, test_data, test_label
temp = trY.shape
trY = trY.reshape(temp[0], 1)
trY = np.concatenate((1-trY, trY), axis=1)
temp = teY.shape
teY = teY.reshape(temp[0], 1)
teY = np.concatenate((1-teY, teY), axis=1)
Finally,I transformed launching the graph in a session into:
with tf.Session() as sess:
# you need to initialize all variables
tf.initialize_all_variables().run()
for i in range(100):
sess.run(train_op, feed_dict={X: trX, Y: trY})
print(i, np.mean(np.argmax(teY, axis=1) == sess.run(predict_op, feed_dict={X: teX})))
That's all.