Develop Reference

Dropout and BatchNormalization layers throw TypeError: Incompatible types: <dtype: 'variant'> vs. int32. Value is 1, model works without them

When using custom estimators in Tensorflow 2, when the model contains BatchNorm or Dropout layers, tf fails while building the graph with the following error. It works just fine when I comment out the Dropout and BatchNorm layers.
The model I use is a simple CNN model with two conv blocks and dense layer at the end:
def build_conv_block(x: Model, filter_map_count: int, name: str):
x = Conv2D(filter_map_count, (3, 3), name=f'{name}_conv_2d')(x)
x = BatchNormalization(name=f'{name}_bn')(x) <------- Error when not commented out
x = ReLU(name=f'{name}_relu')(x)
x = MaxPool2D((2, 2), name=f'{name}_max_pool_2d')(x)
x = Dropout(0.25, name=f'{name}_dropout')(x) <------- Error when not commented out
return x
def get_model(params):
input_image = Input(shape=params.input_shape)
x = build_conv_block(input_image, filter_map_count=64, name='layer_1')
x = build_conv_block(x, filter_map_count=128, name='layer_2')
x = Flatten(name='flatten_conv')(x)
output_pred = Dense(10, activation='softmax', name='output')(x)
model = Model(inputs=input_image, outputs=output_pred)
model.optimizer = Adam(learning_rate=params.learning_rate)
return model
I have a standard train_op in the model_fn that takes mnist images and labels as input and the class as output:
# Calculate gradients
with tf.GradientTape() as tape:
y_pred = model(features, training=training)
loss = tf.losses.categorical_crossentropy(labels, y_pred)
if mode == tf.estimator.ModeKeys.TRAIN:
gradients = tape.gradient(loss, model.trainable_variables)
train_op = model.optimizer.apply_gradients(zip(gradients, model.trainable_variables))
return tf.estimator.EstimatorSpec(mode, loss=loss, train_op=train_op)
Here's the traceback of the error I get:
Traceback (most recent call last):
File "F:/Projects/python/my_project/train.py", line 38, in <module>
tf.estimator.train_and_evaluate(estimator, train_spec=train_spec, eval_spec=eval_spec)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 473, in train_and_evaluate
return executor.run()
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 613, in run
return self.run_local()
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 714, in run_local
saving_listeners=saving_listeners)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1160, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1190, in _train_model_default
features, labels, ModeKeys.TRAIN, self.config)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1148, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "F:\Projects\python\my_project\model.py", line 62, in model_fn
gradients = tape.gradient(loss, model.trainable_variables)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\eager\backprop.py", line 1014, in gradient
unconnected_gradients=unconnected_gradients)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\eager\imperative_grad.py", line 76, in imperative_grad
compat.as_str(unconnected_gradients.value))
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\eager\backprop.py", line 138, in _gradient_function
return grad_fn(mock_op, *out_grads)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\ops\cond_v2.py", line 120, in _IfGrad
true_graph, grads, util.unique_grad_fn_name(true_graph.name))
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\ops\cond_v2.py", line 395, in _create_grad_func
func_graph=_CondGradFuncGraph(name, func_graph))
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\framework\func_graph.py", line 915, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\ops\cond_v2.py", line 394, in <lambda>
lambda: _grad_fn(func_graph, grads), [], {},
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\ops\cond_v2.py", line 373, in _grad_fn
src_graph=func_graph)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\ops\gradients_util.py", line 550, in _GradientsHelper
gradient_uid)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\ops\gradients_util.py", line 175, in _DefaultGradYs
constant_op.constant(1, dtype=y.dtype, name="grad_ys_%d" % i)))
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\framework\constant_op.py", line 227, in constant
allow_broadcast=True)
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\framework\constant_op.py", line 265, in _constant_impl
allow_broadcast=allow_broadcast))
File "F:\Python\envs\tf2\lib\site-packages\tensorflow_core\python\framework\tensor_util.py", line 484, in make_tensor_proto
(dtype, nparray.dtype, values))
TypeError: Incompatible types: <dtype: 'variant'> vs. int32. Value is 1
It looks similar to the error mentioned in TF Issue #31894, but it doesn't seem to solve this problem. The TypeError does not tell much about where and why the error is happening and directly googling it does not help.

Although it may not be too obvious from the TypeError variant vs int32, if we carefully check the logs, we can see that the error occurs when finding gradients:
File "F:\Projects\python\my_project\model.py", line 62, in model_fn
gradients = tape.gradient(loss, model.trainable_variables)
Also, it should be noted that we get the same error even if one of them is present. So, if we try and analyze the common attributes in BatchNormalization and Dropout layer, both may seem to not come under the core layers, but when we look carefully, only those two layers in the model have a different train/test phase i.e. dropout doesn't zero out the values in test phase and batch norm uses a moving mean and variance during test phase.
Now the problem is narrowed down to using any layer that has a different train/test phase. This happens because tensorflow identifies if training mode is on or not using training parameter passed to the model.
This problem can be solved by using
y_pred = model(features, training=True)
when finding the gradients i.e. for the training phase and by using
y_pred = model(features, training=False)
otherwise i.e. for predict and eval phases.
Linked: Errors where moving mean is not updating is also reported, which can be solved by adding the same attribute.

AttributeError: 'ProgbarLogger' object has no attribute 'log_values'

I have tried to run this segmentation model using spyder.
When I run data.py, I obtain this message
File "C:/Users/achaire/Downloads/Compressed/ultrasound-nerve-segmentation-master/ultrasound-nerve-segmentation-master/data.py", line 19, in create_train_data
imgs = np.ndarray((total, image_rows, image_cols), dtype=np.uint8)
TypeError: 'float' object cannot be interpreted as an integer
The code is:
**14** def create_train_data():
**15** train_data_path = os.path.join(data_path, 'train')
**16** images = os.listdir(train_data_path)
**17** total = len(images) / 2
**18** imgs = np.ndarray((total, image_rows, image_cols), dtype=np.uint8)
**19** imgs_mask = np.ndarray((total, image_rows, image_cols), dtype=np.uint8)
I have replaced line 17 by total = int(len(images) / 2) and np.uint8 by float in lines 18 and 19. The problem is solved.
When I run train.py I have this message
runfile('C:/Users/achaire/Downloads/Compressed/ultrasound-nerve-segmentation-master/ultrasound-nerve-segmentation-master/train.py', wdir='C:/Users/achaire/Downloads/Compressed/ultrasound-nerve-segmentation-master/ultrasound-nerve-segmentation-master')
------------------------------
Loading and preprocessing train data...
------------------------------
------------------------------
Creating and compiling model...
------------------------------
C:\Users\achaire\Anaconda3\lib\site-packages\numpy\core\fromnumeric.py:3118: RuntimeWarning: Mean of empty slice.
out=out, **kwargs)
C:\Users\hamdi\Anaconda3\lib\site-packages\numpy\core\_methods.py:140: RuntimeWarning: Degrees of freedom <= 0 for slice
keepdims=keepdims)
------------------------------
Fitting model...
------------------------------
C:/Users/achaire/Downloads/Compressed/ultrasound-nerve-segmentation-master/ultrasound-nerve-segmentation-master/train.py:119: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.
callbacks=[model_checkpoint])
Train on 0 samples, validate on 0 samples
Epoch 1/20
Traceback (most recent call last):
File "<ipython-input-15-f713d62eb4dc>", line 1, in <module>
runfile('C:/Users/achaire/Downloads/Compressed/ultrasound-nerve-segmentation-master/ultrasound-nerve-segmentation-master/train.py', wdir='C:/Users/achaire/Downloads/Compressed/ultrasound-nerve-segmentation-master/ultrasound-nerve-segmentation-master')
File "C:\Users\achaire\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile
execfile(filename, namespace)
File "C:\Users\achaire\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/achaire/Downloads/Compressed/ultrasound-nerve-segmentation-master/ultrasound-nerve-segmentation-master/train.py", line 153, in <module>
train_and_predict()
File "C:/Users/achaire/Downloads/Compressed/ultrasound-nerve-segmentation-master/ultrasound-nerve-segmentation-master/train.py", line 119, in train_and_predict
callbacks=[model_checkpoint])
File "C:\Users\achaire\Anaconda3\lib\site-packages\keras\engine\training.py", line 1039, in fit
validation_steps=validation_steps)
File "C:\Users\achaire\Anaconda3\lib\site-packages\keras\engine\training_arrays.py", line 217, in fit_loop
callbacks.on_epoch_end(epoch, epoch_logs)
File "C:\Users\achaire\Anaconda3\lib\site-packages\keras\callbacks.py", line 79, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "C:\Users\achaire\Anaconda3\lib\site-packages\keras\callbacks.py", line 338, in on_epoch_end
self.progbar.update(self.seen, self.log_values)
AttributeError: 'ProgbarLogger' object has no attribute 'log_values'
I have the last version of anaconda, keras and python ...

Answer 1:
If the error only occurs when you use smaller datasets, you’re very likely using datasets small enough to not have a single sample in the validation set.
Thus it cannot calculate a validation loss.
Answer 2:
I up-voted the previous answer as it gave me the insight to verify the data and inputs to the fit_generator function and find out what the root cause of the issue actually was. In summary, in cases where my dataset was small, I calculated validation_steps and steps_per_epoch which turned out to be zero (0) which caused the error.
I suppose the better longer-term answer, perhaps for the Keras team, is to cause an error/exception in fit_generator when these values are zero, which would probably lead to a better understanding about how to address this issue.
Answer 3:
The error occurs to us because we forgot to set validation_data in fit() method, while used 'callbacks': [keras.callbacks.EarlyStopping(monitor='val_loss', patience=1)],
Code causing error is:
self.model.fit(
x=x_train,
y=y_train,
callbacks=[keras.callbacks.EarlyStopping(monitor='val_loss', patience=1)],
verbose=True)
Adding validation_data=(self.x_validate, self.y_validate), in fit() fixed:
self.model.fit(
x=x_train,
y=y_train,
callbacks=[keras.callbacks.EarlyStopping(monitor='val_loss', patience=1)],
validation_data=(x_validate, y_validate),
verbose=True)
Answer 4:
This error occurs due to the smaller dataset, to resolve this, increase the train times and split the train set in 80:20.
Reference: https://inneka.com/ml/kr/keras-early-stopping-callback-error-val_loss-metric-not-available/

I got the same error and I fixed it by setting verbose = 0 in while training.

I had this same problem. If the problem is during your fit_generator, then there is a high chance that you have some tiny bug in your data_generator code. Debug your fit_generator.

How compute weighted accuracy for multi-class classification?

I do multi-class classification on unbalanced classes. I'm using SGDClassifier(), GradientBoostingClassifier(), RandomForestClassifier(), and LogisticRegression()with class_weight='balanced'. To compare the results. it is required to compute the accuracy. I tried the following way to compute weighted accuracy:
n_samples = len(y_train)
weights_cof = float(n_samples)/(n_classes*np.bincount(data[target_label].as_matrix().astype(int))[1:])
sample_weights = np.ones((n_samples,n_classes)) * weights_cof
print accuracy_score(y_test, y_pred, sample_weight=sample_weights)
y_train is a binary array. So sample_weights has the same shape as y_train (n_samples, n_classes). When I run the script, I received the following error:
Update:
Traceback (most recent call last):
File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.3.2\helpers\pydev\pydevd.py", line 1596, in <module>
globals = debugger.run(setup['file'], None, None, is_module)
File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.3.2\helpers\pydev\pydevd.py", line 974, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "D:/Destiny/DestinyScripts/MainLocationAware.py", line 424, in <module>
predict_country(featuresDF, score, featuresLabel, country_sample_size, 'gbc')
File "D:/Destiny/DestinyScripts/MainLocationAware.py", line 313, in predict_country
print accuracy_score(y_test, y_pred, sample_weight=sample_weights)
File "C:\ProgramData\Anaconda2\lib\site-packages\sklearn\metrics\classification.py", line 183, in accuracy_score
return _weighted_sum(score, sample_weight, normalize)
File "C:\ProgramData\Anaconda2\lib\site-packages\sklearn\metrics\classification.py", line 108, in _weighted_sum
return np.average(sample_score, weights=sample_weight)
File "C:\ProgramData\Anaconda2\lib\site-packages\numpy\lib\function_base.py", line 1124, in average
"Axis must be specified when shapes of a and weights "
TypeError: Axis must be specified when shapes of a and weights differ.

The error would seem to suggest that the shape of your sample_weights and your y_test/y_pred arrays differ. Basically the method creates a boolean array with y_test == y_pred and passes that along with sample_weights to np.average. One of the first checks in that method is to ensure that the entered array and the weights are the same shape, which apparently in this case they are not.
Update
Your comment "sample_weights, y_test, and y_pred have the same shape (n_samples, n_classes)" exposes the issue. According to the documentation for accuracy_score, y_pred and y_true (in your case y_test and y_pred) should be 1 dimensional. Are you perhaps using one hot encoded labels? If so you should convert them to single value labels and then try the accuracy score again.

ValueError: Found input variables with inconsistent numbers of samples: [75, 1]

Can somebody tell me what kind of error is this and how to solve this (as picture given)
and here is the error message
Traceback (most recent call last):
File "ep5pipeline.py", line 57, in <module>
print (accuracy_score(y_test,predictions, normalize = False))
File "//anaconda/lib/python3.6/site-packages/sklearn/metrics/classification.py", line 172, in accuracy_score
y_type, y_true, y_pred = _check_targets(y_true, y_pred)
File "//anaconda/lib/python3.6/site-packages/sklearn/metrics/classification.py", line 72, in _check_targets
check_consistent_length(y_true, y_pred)
File "//anaconda/lib/python3.6/site-packages/sklearn/utils/validation.py", line 181, in check_consistent_length
" samples: %r" % [int(l) for l in lengths])
ValueError: Found input variables with inconsistent numbers of samples: [75, 1]

In function predict(), return predictions should out of the "for" iteration.
Besides, like Neil said, better to post sources code.

Inserting data into regression network in keras

I am currently struggling to understand how i should train my regression network using keras. I am not sure how I should pass my input data to the network.
Both the input data and the output data is stored as a list of numpy arrays.
Each input numpy array is a matrix which has (400 rows, x columns)
Each output numpy array is a matrix which has (x number of rows, 13 columns)
So input dimension is 400 and output is 13.
But how do I pass each of these sets within the list to the training?
# Multilayer Perceptron
model = Sequential() # Feedforward
model.add(Dense(3, input_dim=400))
model.add(Activation('tanh'))
model.add(Dense(1))
model.compile('sgd', 'mse')
Just by parsing data into gives me this error message :
Traceback (most recent call last):
File "tensorflow_datapreprocess_mfcc_extraction_rnn.py", line 167, in <module>
model.fit(train_set_data,train_set_output,verbose=1)
File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 620, in fit
sample_weight=sample_weight)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1034, in fit
batch_size=batch_size)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 961, in _standardize_user_data
exception_prefix='model input')
File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 51, in standardize_input_data
'...')
Exception: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 arrays but instead got the following list of 270 arrays: [array([[ -1.52587891e-04, 3.05175781e-05, -1.52587891e-04,
-5.18798828e-04, 3.05175781e-05, -3.96728516e-04,
1.52587891e-04, 3.35693359e-04, -9.15527344e-05,
3.3...