Develop Reference

Dropout and BatchNormalization layers throw TypeError: Incompatible types: <dtype: 'variant'> vs. int32. Value is 1, model works without them

When using custom estimators in Tensorflow 2, when the model contains BatchNorm or Dropout layers, tf fails while building the graph with the following error. It works just fine when I comment out the Dropout and BatchNorm layers.
The model I use is a simple CNN model with two conv blocks and dense layer at the end:
def build_conv_block(x: Model, filter_map_count: int, name: str):
x = Conv2D(filter_map_count, (3, 3), name=f'{name}_conv_2d')(x)
x = BatchNormalization(name=f'{name}_bn')(x) <------- Error when not commented out
x = ReLU(name=f'{name}_relu')(x)
x = MaxPool2D((2, 2), name=f'{name}_max_pool_2d')(x)
x = Dropout(0.25, name=f'{name}_dropout')(x) <------- Error when not commented out
return x
def get_model(params):
input_image = Input(shape=params.input_shape)
x = build_conv_block(input_image, filter_map_count=64, name='layer_1')
x = build_conv_block(x, filter_map_count=128, name='layer_2')
x = Flatten(name='flatten_conv')(x)
output_pred = Dense(10, activation='softmax', name='output')(x)
model = Model(inputs=input_image, outputs=output_pred)
model.optimizer = Adam(learning_rate=params.learning_rate)
return model
I have a standard train_op in the model_fn that takes mnist images and labels as input and the class as output:
# Calculate gradients
with tf.GradientTape() as tape:
y_pred = model(features, training=training)
loss = tf.losses.categorical_crossentropy(labels, y_pred)
if mode == tf.estimator.ModeKeys.TRAIN:
gradients = tape.gradient(loss, model.trainable_variables)
train_op = model.optimizer.apply_gradients(zip(gradients, model.trainable_variables))
return tf.estimator.EstimatorSpec(mode, loss=loss, train_op=train_op)
Here's the traceback of the error I get:
Traceback (most recent call last):
File "F:/Projects/python/my_project/train.py", line 38, in <module>
tf.estimator.train_and_evaluate(estimator, train_spec=train_spec, eval_spec=eval_spec)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 473, in train_and_evaluate
return executor.run()
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 613, in run
return self.run_local()
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 714, in run_local
saving_listeners=saving_listeners)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1160, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1190, in _train_model_default
features, labels, ModeKeys.TRAIN, self.config)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1148, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "F:\Projects\python\my_project\model.py", line 62, in model_fn
gradients = tape.gradient(loss, model.trainable_variables)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\eager\backprop.py", line 1014, in gradient
unconnected_gradients=unconnected_gradients)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\eager\imperative_grad.py", line 76, in imperative_grad
compat.as_str(unconnected_gradients.value))
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\eager\backprop.py", line 138, in _gradient_function
return grad_fn(mock_op, *out_grads)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\ops\cond_v2.py", line 120, in _IfGrad
true_graph, grads, util.unique_grad_fn_name(true_graph.name))
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\ops\cond_v2.py", line 395, in _create_grad_func
func_graph=_CondGradFuncGraph(name, func_graph))
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\framework\func_graph.py", line 915, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\ops\cond_v2.py", line 394, in <lambda>
lambda: _grad_fn(func_graph, grads), [], {},
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\ops\cond_v2.py", line 373, in _grad_fn
src_graph=func_graph)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\ops\gradients_util.py", line 550, in _GradientsHelper
gradient_uid)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\ops\gradients_util.py", line 175, in _DefaultGradYs
constant_op.constant(1, dtype=y.dtype, name="grad_ys_%d" % i)))
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\framework\constant_op.py", line 227, in constant
allow_broadcast=True)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\framework\constant_op.py", line 265, in _constant_impl
allow_broadcast=allow_broadcast))
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\framework\tensor_util.py", line 484, in make_tensor_proto
(dtype, nparray.dtype, values))
TypeError: Incompatible types: <dtype: 'variant'> vs. int32. Value is 1
It looks similar to the error mentioned in TF Issue #31894, but it doesn't seem to solve this problem. The TypeError does not tell much about where and why the error is happening and directly googling it does not help.

Although it may not be too obvious from the TypeError variant vs int32, if we carefully check the logs, we can see that the error occurs when finding gradients:
File "F:\Projects\python\my_project\model.py", line 62, in model_fn
gradients = tape.gradient(loss, model.trainable_variables)
Also, it should be noted that we get the same error even if one of them is present. So, if we try and analyze the common attributes in BatchNormalization and Dropout layer, both may seem to not come under the core layers, but when we look carefully, only those two layers in the model have a different train/test phase i.e. dropout doesn't zero out the values in test phase and batch norm uses a moving mean and variance during test phase.
Now the problem is narrowed down to using any layer that has a different train/test phase. This happens because tensorflow identifies if training mode is on or not using training parameter passed to the model.
This problem can be solved by using
y_pred = model(features, training=True)
when finding the gradients i.e. for the training phase and by using
y_pred = model(features, training=False)
otherwise i.e. for predict and eval phases.
Linked: Errors where moving mean is not updating is also reported, which can be solved by adding the same attribute.

Value error Tensor must be the same graph as tensor, but dimensions seem to be fine

I am attempting to write a CNN in python using tensorflow 1.13.1. For some reason, even after simplifying my model to only a single affine layer I am getting a dimensions error. Here is the relevant code:
tf.reset_default_graph()
X = tf.placeholder(tf.float32, [None, X_SHAPE[1], X_SHAPE[2], 1]) # X_SHAPE is the shape of the input image types I am workig with
y = tf.placeholder(tf.int64, [None])
is_training = tf.placeholder(tf.bool)
def my_model(X, y, is_training):
output = X
output = tf.reshape(output, [-1, output.shape[1] * output.shape[2] *
output.shape[3]])
output = tf.layers.dense(output, 2) # makes the error
output = tf.contrib.layers.batch_norm(output)
return output
y_out = my_model(X, y, is_training)
total_loss = tf.losses.softmax_cross_entropy(tf.one_hot(y, 2), logits=y_out)
mean_loss = tf.reduce_mean(total_loss)
optimizer = tf.train.RMSPropOptimizer(1e-3)
# batch normalization in tensorflow requires this extra dependency
extra_update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(extra_update_ops):
train_step = optimizer.minimize(mean_loss)
sess = tf.Session()
sess.run(tf.global_variables_initializer())
print('Training')
run_model(sess,y_out,mean_loss,X_train,y_train,8,64,100,train_step,True)
print('Validation')
run_model(sess,y_out,mean_loss,X_val,y_val,1,64)
The error I get is the following:
Traceback (most recent call last):
File "C:/Users/t8484200/Documents/fanta/dicom_snippet.py", line 189, in <module>
y_out = my_model(X, y, is_training)
File "C:/Users/t8484200/Documents/fanta/dicom_snippet.py", line 181, in my_model
output = tf.layers.dense(output, 2)
File "C:\Users\t8484200\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "C:\Users\t8484200\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\layers\core.py", line 188, in dense
return layer.apply(inputs)
File "C:\Users\t8484200\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 1227, in apply
return self.__call__(inputs, *args, **kwargs)
File "C:\Users\t8484200\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\layers\base.py", line 530, in __call__
outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
File "C:\Users\t8484200\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 554, in __call__
outputs = self.call(inputs, *args, **kwargs)
File "C:\Users\t8484200\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\keras\layers\core.py", line 975, in call
outputs = gen_math_ops.mat_mul(inputs, self.kernel)
File "C:\Users\t8484200\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 5629, in mat_mul
name=name)
File "C:\Users\t8484200\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 350, in _apply_op_helper
g = ops._get_graph_from_inputs(_Flatten(keywords.values()))
File "C:\Users\t8484200\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 5713, in _get_graph_from_inputs
_assert_same_graph(original_graph_element, graph_element)
File "C:\Users\t8484200\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 5649, in _assert_same_graph
original_item))
ValueError: Tensor("dense/kernel:0", shape=(262144, 2), dtype=float32_ref) must be from the same graph as Tensor("Reshape:0", shape=(?, 262144), dtype=float32).
But dimensions seem to be fine, so I would really appreciate your help on this one!

You are not getting a dimension error. The dimensions are just mentioned as part of the information about the two relevant tensors. I ran this code with tf 1.13.1 and it works for me.
I was able to get the same error for example when I replace the first 4 lines with: (same lines in different order)
X = tf.placeholder(tf.float32, [None, X_SHAPE[1], X_SHAPE[2], 1]) # X_SHAPE is the shape of the input image types I am workig with
y = tf.placeholder(tf.int64, [None])
is_training = tf.placeholder(tf.bool)
tf.reset_default_graph()
The reason is that X is created in the existing graph, and then a new graph is created by the reset command. Then the dense tensor is created in the new graph, but using X from the old graph which is not allowed. (It looks like the reshape command is in the old graph, maybe because it does not create new variables.)
So you need to check where you are resetting the graph compared to where you are defining your placeholders.

conv2d in custom Keras loss function

I am trying to implement a custom loss function in Keras with TF backend based on the Laplacian of two images.
def blur_loss(y_true, y_pred):
#weighting of blur loss
alpha = 1
mae = losses.mean_absolute_error(y_true, y_pred)
lapKernel = K.constant([0, 1, 0, 1, -4, 1, 0, 1, 0],shape = [3, 3])
trueLap = K.conv2d(y_true, lapKernel)
predLap = K.conv2d(y_pred, lapKernel)
trueBlur = K.var(trueLap)
predBlur = K.var(predLap)
blurLoss = alpha * K.abs(trueBlur - predBlur)
loss = (1-alpha) * mae + alpha * blurLoss
return loss
When I try to compile the model I get this error
Traceback (most recent call last):
File "kitti_train.py", line 65, in <module>
model.compile(loss='mean_absolute_error', optimizer='adam', metrics=[blur_loss])
File "/home/ubuntu/.virtualenvs/dl4cv/lib/python3.5/site-packages/keras/engine/training.py", line 924, in compile
handle_metrics(output_metrics)
File "/home/ubuntu/.virtualenvs/dl4cv/lib/python3.5/site-packages/keras/engine/training.py", line 921, in handle_metrics
mask=masks[i])
File "/home/ubuntu/.virtualenvs/dl4cv/lib/python3.5/site-packages/keras/engine/training.py", line 450, in weighted
score_array = fn(y_true, y_pred)
File "/home/ubuntu/prednet/blur_loss.py", line 14, in blur_loss
trueLap = K.conv2d(y_true, lapKernel)
File "/home/ubuntu/.virtualenvs/dl4cv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 3164, in conv2d
data_format='NHWC')
File "/home/ubuntu/.virtualenvs/dl4cv/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 655, in convolution
num_spatial_dims, strides, dilation_rate)
File "/home/ubuntu/.virtualenvs/dl4cv/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 483, in _get_strides_and_dilation_rate
(len(dilation_rate), num_spatial_dims))
ValueError: len(dilation_rate)=2 but should be 0
After reading other questions, my understanding is that this problem stems from the compilation using placeholder tensors for y_true and y_pred. I've tried checking if the inputs are placeholders and replacing them with zero tensors, but this gives me other errors.
How do I use a convolution (the image processing function, not a layer) in my loss function without getting these errors?

The problem here was a misunderstanding of the conv2d function which is not simply a 2-dimensional convolution. It is a batched 2-d convolution of multiple channels. So while you might expect a *2d function to accept 2-dimensional tensors, the input should actually 4 dimensions (batch_size, height, width, channels) and the filter should also be 4 dimensions (filter_height, filter_width, input_channels, output_channels). Details can be found in the TF docs

Keras customized image preprocessing function incurs Value Error "output array is read-only"

I want to use some customized image preprocessing function along with ImageDataGenerator function in Keras. For example, my customized function looks like this:
def customizedDataAugmentation(x):
choice = np.random.choice(np.arange(1, 4), p=[0.3, 0.3, 0.4])
if choice==1:
x = exposure.adjust_gamma(x, np.random.uniform(0.5,1.5))
elif choice==2:
ix = Image.fromarray(np.uint8(x))
blurI = ix.filter(ImageFilter.GaussianBlur(np.random.uniform(0.1,2.5)))
x = np.asanyarray(blurI)
return x
And the way to use it is like:
self.train_datagen = image.ImageDataGenerator(
rescale=1./255,
zoom_range=0.15,
height_shift_range=0.1,
horizontal_flip=True,
preprocessing_function=customizedDataAugmentation
)
However, when I start training, it jumps out this error:
Traceback (most recent call last):
File "/home/joseph/miniconda3/envs/py27/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/home/joseph/miniconda3/envs/py27/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/joseph/miniconda3/envs/py27/lib/python2.7/site-packages/keras/utils/data_utils.py", line 560, in data_generator_task
generator_output = next(self._generator)
File "/home/joseph/miniconda3/envs/py27/lib/python2.7/site-packages/keras/preprocessing/image.py", line 1039, in next
x = self.image_data_generator.standardize(x)
File "/home/joseph/miniconda3/envs/py27/lib/python2.7/site-packages/keras/preprocessing/image.py", line 494, in standardize
x *= self.rescale
ValueError: output array is read-only
self.image_data_generator.standardize(x) is the function that calls the customized function. The definition looks something like this:
def standardize(self, x):
if self.preprocessing_function:
x = self.preprocessing_function(x)
if self.rescale:
x *= self.rescale
....
If I don't call my customized function I wouldn't have this error though.
Anyone knows what's happening?

When I ran into this error, I found that my numpy array wasn't writable, which you can check with
print(x.flags)
You can make the array writable with
x.setflags(write=1)
before returning it.
See also: np arrays being immutable - "assignment destination is read-only"

I have downgraded keras version from 2.3.0 to 2.2.0 and it solve this error

Shaping data for linear regression with TFlearn

I'm trying to expand the tflearn example for linear regression by increasing the number of columns to 21.
from trafficdata import X,Y
import tflearn
print(X.shape) #(1054, 21)
print(Y.shape) #(1054,)
# Linear Regression graph
input_ = tflearn.input_data(shape=[None,21])
linear = tflearn.single_unit(input_)
regression = tflearn.regression(linear, optimizer='sgd', loss='mean_square',
metric='R2', learning_rate=0.01)
m = tflearn.DNN(regression)
m.fit(X, Y, n_epoch=1000, show_metric=True, snapshot_epoch=False)
print("\nRegression result:")
print("Y = " + str(m.get_weights(linear.W)) +
"*X + " + str(m.get_weights(linear.b)))
However, tflearn complains:
Traceback (most recent call last):
File "linearregression.py", line 16, in <module>
m.fit(X, Y, n_epoch=1000, show_metric=True, snapshot_epoch=False)
File "/usr/local/lib/python3.5/dist-packages/tflearn/models/dnn.py", line 216, in fit
callbacks=callbacks)
File "/usr/local/lib/python3.5/dist-packages/tflearn/helpers/trainer.py", line 339, in fit
show_metric)
File "/usr/local/lib/python3.5/dist-packages/tflearn/helpers/trainer.py", line 818, in _train
feed_batch)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 789, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 975, in _run
% (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (64,) for Tensor 'TargetsData/Y:0', which has shape '(21,)'
I found the shape (64, ) comes from the default batch size of tflearn.regression().
Do I need to transform the labels (Y)? In what way?
Thanks!

I tried to do the same. I made these changes to get it to work
# linear = tflearn.single_unit(input_)
linear = tflearn.fully_connected(input_, 1, activation='linear')
My guess is that with features >1 you cannot use tflearn.single_unit(). You can add additional fully_connected layers, but the last one must have only 1 neuron because Y.shape=(?,1)

You have 21 features. Therefore, you cannot use linear = tflearn.single_unit(input_)
Instead try this: linear = tflearn.fully_connected(input_, 21, activation='linear')
The error you get is because your labels, i.e., Y has a shape of (1054,).
You have to first preprocess it.
Try using the code given below before # linear regression graph:
Y = np.expand_dims(Y,-1)
Now before regression = tflearn.regression(linear, optimizer='sgd', loss='mean_square',metric='R2', learning_rate=0.01), type the below code:
linear = tflearn.fully_connected(linear, 1, activation='linear')