Develop Reference

TensorFlow Tensor handled differently in numpy argmax vs keras argmax

Why does a TensorFlow tensor behave differently in math functions in Numpy than it behaves in math functions in Keras?
Numpy arrays seem to act normally when put in the same situation as the TensorFlow Tensor.
This example shows that a numpy matrix is handled correctly under numpy functions and keras functions.
import numpy as np
from keras import backend as K
arr = np.random.rand(19, 19, 5, 80)
np_argmax = np.argmax(arr, axis=-1)
np_max = np.max(arr, axis=-1)
k_argmax = K.argmax(arr, axis=-1)
k_max = K.max(arr, axis=-1)
print('np_argmax shape: ', np_argmax.shape)
print('np_max shape: ', np_max.shape)
print('k_argmax shape: ', k_argmax.shape)
print('k_max shape: ', k_max.shape)
Outputs this (as expected)
np_argmax shape: (19, 19, 5)
np_max shape: (19, 19, 5)
k_argmax shape: (19, 19, 5)
k_max shape: (19, 19, 5)
As opposed to this example
import numpy as np
from keras import backend as K
import tensorflow as tf
arr = tf.constant(np.random.rand(19, 19, 5, 80))
np_argmax = np.argmax(arr, axis=-1)
np_max = np.max(arr, axis=-1)
k_argmax = K.argmax(arr, axis=-1)
k_max = K.max(arr, axis=-1)
print('np_argmax shape: ', np_argmax.shape)
print('np_max shape: ', np_max.shape)
print('k_argmax shape: ', k_argmax.shape)
print('k_max shape: ', k_max.shape)
which outputs
np_argmax shape: ()
np_max shape: (19, 19, 5, 80)
k_argmax shape: (19, 19, 5)
k_max shape: (19, 19, 5)

You need to execute/run code (say under a TF session) to have tensors evaluated. Until then, the shapes of tensors are not evaluated.
TF docs say:
Each element in the Tensor has the same data type, and the data type is always known. The shape (that is, the number of dimensions it has and the size of each dimension) might be only partially known. Most operations produce tensors of fully-known shapes if the shapes of their inputs are also fully known, but in some cases it's only possible to find the shape of a tensor at graph execution time.

Why don't you try the following code for the 2nd example:
import numpy as np
from keras import backend as K
import tensorflow as tf
arr = tf.constant(np.random.rand(19, 19, 5, 80))
with tf.Session() as sess:
arr = sess.run(arr)
np_argmax = np.argmax(arr, axis=-1)
np_max = np.max(arr, axis=-1)
k_argmax = K.argmax(arr, axis=-1)
k_max = K.max(arr, axis=-1)
print('np_argmax shape: ', np_argmax.shape)
print('np_max shape: ', np_max.shape)
print('k_argmax shape: ', k_argmax.shape)
print('k_max shape: ', k_max.shape)
After arr = tf.constant(np.random.rand(19, 19, 5, 80)), the type of arr is tf.Tensor, but after running arr = sess.run(arr) its type will be changed to numpy.ndarray.

Converting Tensor to a SparseTensor for ctc_loss [duplicate]

This question already has answers here:
Sparse Tensor (matrix) from a dense Tensor Tensorflow
(5 answers)
Closed 5 years ago.
Is there a way to convert a dense tensor into a sparse tensor? Apparently, Tensorflow's Estimator.fit doesn't accept SparseTensors as labels. One reason I would like to pass SparseTensors into Tensorflow's Estimator.fit is to be able to use tensorflow ctc_loss. Here's the code:
import dataset_utils
import tensorflow as tf
import numpy as np
from tensorflow.contrib import grid_rnn, learn, layers, framework
def grid_rnn_fn(features, labels, mode):
input_layer = tf.reshape(features["x"], [-1, 48, 1596])
indices = tf.where(tf.not_equal(labels, tf.constant(0, dtype=tf.int32)))
values = tf.gather_nd(labels, indices)
sparse_labels = tf.SparseTensor(indices, values, dense_shape=tf.shape(labels, out_type=tf.int64))
cell_fw = grid_rnn.Grid2LSTMCell(num_units=128)
cell_bw = grid_rnn.Grid2LSTMCell(num_units=128)
bidirectional_grid_rnn = tf.nn.bidirectional_dynamic_rnn(cell_fw, cell_bw, input_layer, dtype=tf.float32)
outputs = tf.reshape(bidirectional_grid_rnn[0], [-1, 256])
W = tf.Variable(tf.truncated_normal([256,
80],
stddev=0.1, dtype=tf.float32), name='W')
b = tf.Variable(tf.constant(0., dtype=tf.float32, shape=[80], name='b'))
logits = tf.matmul(outputs, W) + b
logits = tf.reshape(logits, [tf.shape(input_layer)[0], -1, 80])
logits = tf.transpose(logits, (1, 0, 2))
loss = None
train_op = None
if mode != learn.ModeKeys.INFER:
#Error occurs here
loss = tf.nn.ctc_loss(inputs=logits, labels=sparse_labels, sequence_length=320)
... # returning ModelFnOps
def main(_):
image_paths, labels = dataset_utils.read_dataset_list('../test/dummy_labels_file.txt')
data_dir = "../test/dummy_data/"
images = dataset_utils.read_images(data_dir=data_dir, image_paths=image_paths, image_extension='png')
print('Done reading images')
images = dataset_utils.resize(images, (1596, 48))
images = dataset_utils.transpose(images)
labels = dataset_utils.encode(labels)
x_train, x_test, y_train, y_test = dataset_utils.split(features=images, test_size=0.5, labels=labels)
train_input_fn = tf.estimator.inputs.numpy_input_fn(
x={"x": np.array(x_train)},
y=np.array(y_train),
num_epochs=1,
shuffle=True,
batch_size=1
)
classifier = learn.Estimator(model_fn=grid_rnn_fn, model_dir="/tmp/grid_rnn_ocr_model")
classifier.fit(input_fn=train_input_fn)
UPDATE:
It turns out, this solution from here converts the dense tensor into a sparse one:
indices = tf.where(tf.not_equal(labels, tf.constant(0, dtype=tf.int32)))
values = tf.gather_nd(labels, indices)
sparse_labels = tf.SparseTensor(indices, values, dense_shape=tf.shape(labels, out_type=tf.int64))
However, I encounter this error now raised by ctc_loss:
ValueError: Shape must be rank 1 but is rank 0 for 'CTCLoss' (op: 'CTCLoss') with input shapes: [?,?,80], [?,2], [?], [].
I have this code that converts dense labels to sparse:
def convert_to_sparse(labels, dtype=np.int32):
indices = []
values = []
for n, seq in enumerate(labels):
indices.extend(zip([n] * len(seq), range(len(seq))))
values.extend(seq)
indices = np.asarray(indices, dtype=dtype)
values = np.asarray(values, dtype=dtype)
shape = np.asarray([len(labels), np.asarray(indices).max(0)[1] + 1], dtype=dtype)
return indices, values, shape
I converted y_train to sparse labels, and place the values inside a SparseTensor:
sparse_y_train = convert_to_sparse(y_train)
print(tf.SparseTensor(
indices=sparse_y_train[0],
values=sparse_y_train[1],
dense_shape=sparse_y_train
))
And compared it to the SparseTensor created inside the grid_rnn_fn:
indices = tf.where(tf.not_equal(labels, tf.constant(0, dtype=tf.int32)))
values = tf.gather_nd(labels, indices)
sparse_labels = tf.SparseTensor(indices, values, dense_shape=tf.shape(labels, out_type=tf.int64))
Here's what I got:
For sparse_y_train:
SparseTensor(indices=Tensor("SparseTensor/indices:0", shape=(33, 2), dtype=int64), values=Tensor("SparseTensor/values:0", shape=(33,), dtype=int32), dense_shape=Tensor("SparseTensor/dense_shape:0", shape=(2,), dtype=int64))
For sparse_labels:
SparseTensor(indices=Tensor("Where:0", shape=(?, 2), dtype=int64), values=Tensor("GatherNd:0", shape=(?,), dtype=int32), dense_shape=Tensor("Shape:0", shape=(2,), dtype=int64))
Which leads me to think that ctc_loss can't seem to handle SparseTensors as labels with dynamic shapes.

Yes. It is possible to convert a tensor to a sparse tensor and back:
Let sparse be a sparse tensor and dense be a dense tensor.
From sparse to dense:
dense = tf.sparse_to_dense(sparse.indices, sparse.shape, sparse.values)
From dense to sparse:
zero = tf.constant(0, dtype=tf.float32)
where = tf.not_equal(dense, zero)
indices = tf.where(where)
values = tf.gather_nd(dense, indices)
sparse = tf.SparseTensor(indices, values, dense.shape)

CTC loss Tensorflow, No valid path found

I'm trying to use ctc_loss in order to predict some labels, but it's the first time I use this function and I got an error.
def bi_rnn(x_bi):
# Prepare data shape to match `rnn` function requirements
# Current data input shape: (batch_size, timesteps, n_input)
# Required shape: 'timesteps' tensors list of shape (batch_size, num_input)
# Unstack to get a list of 'timesteps' tensors of shape (batch_size, num_input)
x_bi_shape = x_bi.get_shape().as_list()
x_bi = tf.unstack(x_bi, x_bi_shape[2], 2)
# Define lstm cells with tensorflow
# Forward direction cell
lstm_fw_cell = rnn.BasicLSTMCell(num_neurons, forget_bias=1.0)
# Backward direction cell
lstm_bw_cell = rnn.BasicLSTMCell(num_neurons, forget_bias=1.0)
bi_outputs, _, _ = rnn.static_bidirectional_rnn(lstm_fw_cell, lstm_bw_cell, x_bi, dtype=tf.float32)
w_bi = init_weights([num_neurons * 2, num_classes])
b_bi = init_bias([num_classes])
bi_outputs = tf.squeeze(input=bi_outputs, axis=1)
return tf.matmul(bi_outputs, w_bi) + b_bi
def sparse_tuple_from(sequences, dtype=np.int32):
"""Create a sparse representention of x.
Args:
sequences: a list of lists of type dtype where each element is a sequence
Returns:
A tuple with (indices, values, shape)
"""
indices = []
values = []
for n, seq in enumerate(sequences):
indices.extend(zip([n] * len(seq), range(len(seq))))
values.extend(seq)
indices = np.asarray(indices, dtype=np.int64)
values = np.asarray(values, dtype=dtype)
shape = np.asarray([len(sequences), np.asarray(indices).max(0)[1] + 1], dtype=np.int64)
return indices, values, shape
bi_output_1 = bi_rnn(vectors)
targets = tf.sparse_placeholder(tf.int32, name="targets_sparse_tensor")
bi_output_1_shape = bi_output_1.get_shape().as_list()
global_step = tf.Variable(0, trainable=False)
bi_output_1 = tf.reshape(bi_output_1, [bi_output_1_shape[0], batch_size, bi_output_1_shape[1]])
# 1d array of size [batch_size]
seq_len = tf.placeholder(tf.int32, [None])
loss = tf.nn.ctc_loss(labels=targets, inputs=bi_output_1, sequence_length=seq_len)
cost = tf.reduce_mean(loss)
optimizer = tf.train.MomentumOptimizer(learning_rate=0.001, momentum=0.9).minimize(cost, global_step=global_step)
here is how I train the net:
with tf.Session() as sess:
sess.run(init)
for i in range(1, epochs):
loss = 0
start = 0
for cont in range(int(n_samples / batch_size)):
batch_x, batch_y = feature_set.next_batch_training_set(batch_size, start)
batch_y = sparse_tuple_from(batch_y)
seq_len_training = np.ones(1) * 7
_, c = sess.run([optimizer, cost], feed_dict={x: batch_x, targets: batch_y, seq_len: seq_len_training})
loss += c
print('Epoch:', i, ' loss:', loss)
with:
batch_size = 1
bi_output_1 = Tensor("Reshape:0", shape=(25, 1, 37), dtype=float32)
batch_x has shape : (1, 20, 100, 1)
batch_y has shape : (1, 7) (before the function sparse_tuple_from())
I got this error:
2017-11-23 18:50:14.033220: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\util\ctc\ctc_loss_calculator.cc:144] No valid path found.
and I don't know how to solve this. I tried to follow this example, but I got no help.

sess.run() causing training to be slow

I am training a CNN and I believe my use of sess.run() is causing my training to be very slow.
In essence, I am use the mnist data set...
from tensorflow.examples.tutorials.mnist import input_data
...
...
features = input_data.read_data_sets("/tmp/data/", one_hot=True)
The issue is, the first layer of the CNN must accept the images in the form of [batch_size, 28, 28, 1], which means I must convert each image before feeding it to the CNN.
I do the following with my script...
x = tf.placeholder(tf.float32, [None, 28, 28, 1])
y = tf.placeholder(tf.float32, [None, 10])
...
...
with tf.Session() as sess:
for epoch in range(25):
total_batch = int(features.train.num_examples/500)
avg_cost = 0
for i in range(total_batch):
batch_xs, batch_ys = features.train.next_batch(10)
# Notice this line.
_, c = sess.run([train_op, loss], feed_dict={x:sess.run(tf.reshape(batch_xs, [10, 28, 28, 1])), y:batch_ys})
avg_cost += c / total_batch
if (epoch + 1) % 1 == 0:
print("Epoch:", '%04d' % (epoch + 1), "cost=", "{:.9f}".format(avg_cost))
Notice the commented line. I am taking the first batch from the training set and I am reshaping to the proper format [batch_size, 28, 28, 1]. I have to call sess.run() every single time and I believe this is the cause for the training to be so slow.
How can I prevent this. I tried reformatting the data in another script using numpy, but it still gave me issues because I cannot feed the numpy array without running sess.run(). Can someone show me how to format the data outside of the training session? Maybe I can format the data in another script and load it into the one containing my CNN?

You should definitely not have the inner sess.run() on new ops at each iteration (though I'm not sure how much it really slows you down). You should do one of these:
have a placeholder of the same shape as your input, e.g. [None, 28*28*1], followed by a tf.reshape([None, 28, 28, 1]), at the beginning of your network (instead of your tf.placeholder([None, 28, 28, 1]))
OR
Keep your neural network, and reformat using numpy reshape instead of tensorflow: _, c = sess.run([train_op, loss], feed_dict={x:batch_xs.reshape( [-1, 28, 28, 1]), y:batch_ys})
It probably also works if you just write _, c = sess.run([train_op, loss], feed_dict={x:tf.reshape(batch_xs, [10, 28, 28, 1]), y:batch_ys}) but you should not do that, since it creates a new op in your graph at each iteration.

One other thing you can do is reshape all the inputs in the beginning itself and then feed it to the placeholder.
import math
import numpy as np
x = tf.placeholder(tf.float32, [None, 28, 28, 1])
y = tf.placeholder(tf.float32, [None, 10])
...
...
with tf.Session() as sess:
X_train=mnist.train.images.reshape(-1,28,28,1)
y_train=mnist.train.labels
train_indicies = np.arange(X_train.shape[0])
num_epochs = 25 // number of epochs
batch_size = 50
total_batch = int(math.ceil(X_train.shape[0]/batch_size))
for epoch in range(25):
for i in np.arange(total_batch):
start_idx = (i*batch_size)%X_train.shape[0]
idx = train_indicies[start_idx:start_idx+batch_size]
_, c = sess.run([train_op, loss], feed_dict={x:X_train[idx,:], y:y_train[idx]})
avg_cost += c / total_batch
if (epoch + 1) % 1 == 0:
print("Epoch:", '%04d' % (epoch + 1), "cost=", "{:.9f}".format(avg_cost))
since we will not be able to use mnist.train.next_batch, we will need to manually calculate and increment the indices.
Hope this works :)

Tensorflow placeholder declaration

I'm trying to convert a tutorial from Keras to TF.
I'm getting the following error:
Traceback (most recent call last):
File "/Users/spicyramen/Documents/Development/google/python/machine_learning/deep_learning/exercise1_tf.py", line 64, in <module>
sess.run(train_step, feed_dict=train_data)
File "/Library/Python/2.7/site-packages/tensorflow/python/client/session.py", line 789, in run
run_metadata_ptr)
File "/Library/Python/2.7/site-packages/tensorflow/python/client/session.py", line 975, in _run
% (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (768,) for Tensor u'Placeholder_1:0', which has shape '(?, 1)'
This seems to be related in how I'm passing the target labels and how my placeholder value is declared.
When I return the labels I have this:
>>> dataset[:, 8].shape
(768,)
>>> dataset[:, 0:8].shape
(768, 8)
Code
import tensorflow as tf
import numpy as np
print("Tensorflow version: " + tf.__version__)
tf.set_random_seed(0)
FILENAME = 'pima-indians-diabetes.csv'
_LEARNING_RATE = 0.003
_NUM_FEATURES = 8
_NUM_LABELS = 1
_NUM_EPOCHS = 150
_BATCH_SIZE = 10
def import_data(filename):
if filename:
dataset = np.loadtxt(filename, delimiter=",")
return dataset[:, 0:8], dataset[:, 8]
# create placeholder. Dataset contains _NUM_FEATURES features:
X = tf.placeholder(tf.float32, [None, _NUM_FEATURES])
Y_ = tf.placeholder(tf.float32,[None, _NUM_LABELS]) # Placeholder for correct answers
# weights and biases
W = tf.Variable(tf.random_normal([_NUM_FEATURES, _NUM_LABELS],
mean=0,
stddev=0.1,
name='weights'))
b = tf.Variable(tf.random_normal([1, _NUM_LABELS],
mean=0,
stddev=0.1,
name='bias'))
# activation function
Y = tf.nn.relu(tf.matmul(X, W) + b, name='activation')
# cost function i.e. sigmoid_cross_entropy_with_logits
cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=Y_, logits=Y, name='loss_function')
optimizer = tf.train.AdamOptimizer(_LEARNING_RATE) # Formal derivation
train_step = optimizer.minimize(cross_entropy)
# cost function i.e. RMSE
# cross_entropy = tf.nn.l2_loss(Y - Y_, name="squared_error_cost")
# optimizer = tf.train.GradientDescentOptimizer(_LEARNING_RATE)
# train_step = optimizer.minimize(cross_entropy)
is_correct = tf.equal(tf.argmax(Y, 1), tf.argmax(Y_, 1))
accuracy = tf.reduce_mean(tf.cast(is_correct, tf.float32))
# init
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)
for i in range(_NUM_EPOCHS):
# data
batch_X, batch_Y = import_data(FILENAME)
# train
train_data = {X: batch_X, Y_: batch_Y}
sess.run(train_step, feed_dict=train_data)
a, c = sess.run([accuracy, cross_entropy], feed_dict=train_data)
print(str(i) + ": accuracy:" + str(a) + " loss: " + str(c))

This is your problem here:
>>> dataset[:, 8].shape
(768,)
TensorFlow is expecting an array of shape (768,1) and not (768,) as the error references:
Cannot feed value of shape (768,) for Tensor u'Placeholder_1:0', which has shape '(?, 1)'
The difference between the two shapes is somewhat small and Numpy would normally broadcast these for you in many circumstances, but TF won't. See the difference between those two shapes in this question with a great answer.
Luckily in your case the solution is very simple. You can use np.expand_dims() to turn your (768,) vector into a (768,1) vector, as demonstrated here:
>>> np.array([5,5,5]).shape
(3,)
>>> np.expand_dims(np.array([5,5,5]), axis=1).shape
(3, 1)
In your import_data function, simply change the return line to
return dataset[:, 0:8], np.expand_dims(dataset[:, 8], axis=1)
Edit: I like the above because np.expand_dims is a little more explicit, but there's another way that is equally simple, and others might think it's more clear---just depends on what you're used to. Figured I'd include it for completeness. The difference between a (N,) and (N,1) array is the first holds values in a 0-dimensional array np.array([5, 5, 5]) while the second holds values in a 1-dimensional array np.array([[5],[5],[5]]). You can turn your 0-d array into a 1-d array by adding a bracket around it; but then it's a row and not a column, so it needs to be transposed. So here's the two suggested ways together; B is the new suggestion, C is the above suggestion:
>>> A = np.array([5,5,5])
>>> B = np.array([A]).T
>>> C = np.expand_dims(A, axis=1)
>>> A; A.shape
array([5, 5, 5])
(3,)
>>> B; B.shape
array([[5],
[5],
[5]])
(3, 1)
>>> C; C.shape
array([[5],
[5],
[5]])
(3, 1)
Edit2: Also TensorFlow itself has a tf.expand_dims() function.