Develop Reference

Tensoflow: Unable to restore custom object of type _tf_keras_metric

I have trained a model in Google Colab and trying to load the model in my local machine to use on the web app. But as soon as I load_model() it shows this error:
Unable to restore custom object of type _tf_keras_metric. Please make sure that any custom layers are included in the `custom_objects` arg when calling `load_model()` and make sure that all layers implement `get_config` and `from_config`.
I have used CategoricalAccuracy as a metric in the model and I tried to pass it as a custom object but I get the same error.
Code used in Web App
def load_text_model():
model = keras.models.load_model("." + url_for('static', filename='models/text_model'), custom_objects={'CategoricalAccuracy': tf.keras.metrics.CategoricalAccuracy(name="accuracy")})
return model
Extra information if needed: I am making a multi-class text classification model. I have used bert english uncased model from thub as preprocessor and encoder. Below are the metrics I have used. If more code or context is required, I will provide it.
Code used while training model
def balanced_recall(y_true, y_pred):
"""This function calculates the balanced recall metric
recall = TP / (TP + FN)
"""
recall_by_class = 0
# iterate over each predicted class to get class-specific metric
for i in range(y_pred.shape[1]):
y_pred_class = y_pred[:, i]
y_true_class = y_true[:, i]
true_positives = K.sum(K.round(K.clip(y_true_class * y_pred_class, 0, 1)))
possible_positives = K.sum(K.round(K.clip(y_true_class, 0, 1)))
recall = true_positives / (possible_positives + K.epsilon())
recall_by_class = recall_by_class + recall
return recall_by_class / y_pred.shape[1]
def balanced_precision(y_true, y_pred):
"""This function calculates the balanced precision metric
precision = TP / (TP + FP)
"""
precision_by_class = 0
# iterate over each predicted class to get class-specific metric
for i in range(y_pred.shape[1]):
y_pred_class = y_pred[:, i]
y_true_class = y_true[:, i]
true_positives = K.sum(K.round(K.clip(y_true_class * y_pred_class, 0, 1)))
predicted_positives = K.sum(K.round(K.clip(y_pred_class, 0, 1)))
precision = true_positives / (predicted_positives + K.epsilon())
precision_by_class = precision_by_class + precision
# return average balanced metric for each class
return precision_by_class / y_pred.shape[1]
def balanced_f1_score(y_true, y_pred):
"""This function calculates the F1 score metric"""
precision = balanced_precision(y_true, y_pred)
recall = balanced_recall(y_true, y_pred)
return 2 * ((precision * recall) / (precision + recall + K.epsilon()))
i = tf.keras.layers.Input(shape=(), dtype=tf.string, name='text')
x = preprocessor(i)
x = encoder(x)
x = tf.keras.layers.Dropout(0.2, name="dropout")(x['default'])
x = tf.keras.layers.Dense(num_classes, activation='softmax', name="output")(x)
model = tf.keras.Model(i, x)
METRICS = [
tf.keras.metrics.CategoricalAccuracy(name="accuracy"),
balanced_recall,
balanced_precision,
balanced_f1_score
]

Custom Loss function, Keras \\ ValueError: No gradients

I'm attempting to wrap my Keras neural network in a class object. I have implemented the below outside of a class setting, but I want to make this more object-friendly.
To summarize, my model calls function sequential_model which creates a sequential model. Within the compile step, I have defined my own loss function weighted_categorical_crossentropy which I want the sequential model to implement.
However, when I run the code below I get the following error: ValueError: No gradients provided for any variable:
I suspect the issue is with how I'm defining the weighted_categorical_crossentropy function for its use by sequential.
Again, I was able to get this work in a non-object oriented way. Any help will be much appreciated.
from tensorflow.keras import Sequential, backend as K
class MyNetwork():
def __init__(self, file, n_output=4, n_hidden=20, epochs=3,
dropout=0.10, batch_size=64, metrics = ['categorical_accuracy'],
optimizer = 'rmsprop', activation = 'softmax'):
[...] //Other Class attributes
def model(self):
self.model = self.sequential_model(False)
self.model.summary()
def sequential_model(self, val):
K.clear_session()
if val == False:
self.epochs = 3
regressor = Sequential()
#regressor.run_eagerly = True
regressor.add(LSTM(units = self.n_hidden, dropout=self.dropout, return_sequences = True, input_shape = (self.X.shape[1], self.X.shape[2])))
regressor.add(LSTM(units = self.n_hidden, dropout=self.dropout, return_sequences = True))
regressor.add(Dense(units = self.n_output, activation=self.activation))
self.weights = np.array([0.025,0.225,0.78,0.020])
regressor.compile(optimizer = self.optimizer, loss = self.weighted_categorical_crossentropy(self.weights), metrics = [self.metrics])
regressor.fit(self.X, self.Y*1.0,batch_size=self.batch_size, epochs=self.epochs, verbose=1, validation_data=(self.Xval, self.Yval*1.0))
return regressor
def weighted_categorical_crossentropy(self, weights):
weights = K.variable(weights)
def loss(y_true, y_pred):
y_pred /= K.sum(y_pred, axis=-1, keepdims=True)
y_pred = K.clip(y_pred, K.epsilon(), 1 - K.epsilon())
loss = y_true * K.log(y_pred) * weights
loss = -K.sum(loss, -1)
return loss

There are several problems with above code, but the most noticeable one is you don't return the loss from weighted_categorical_crossentropy. It should look more like:
def weighted_categorical_crossentropy(self, weights):
weights = K.variable(weights)
def loss(y_true, y_pred):
y_pred /= K.sum(y_pred, axis=-1, keepdims=True)
y_pred = K.clip(y_pred, K.epsilon(), 1 - K.epsilon())
loss = y_true * K.log(y_pred) * weights
loss = -K.sum(loss, -1)
return loss
return loss # Return the callable function!
The error is ValueError: No gradients provided for any variable because the loss method doesn't return anything, it returns None! If you try to fit a method with loss=None, the model will have no way of computing gradients and therefore it will throw the same exact error.
Next up is the that you are using return_sequences = True in the layer right before a non-recurrent layer. This causes the Dense layer to be called on mis-shaped data, that's appropriate only for recurrent layers. Don't use it like that.
If you have a good reason for using the return_sequences = True, then you must add Dense layer like:
model.add(keras.layers.TimeDistributed(keras.layers.Dense(...)))
This will cause the Dense layer to act on output sequence on every time step separately. This also means that your y_true must be of appropriate shape.
There could be other problems with the custom loss function that you defined, but I can not deduce the input/output shapes, so you will have to run it and add see if it works. There will probably be matrix multiplication shape mismatch.
Last but not least, think about using the Sub-classing API. Could it make any of your operations easier to write?
Thanks for reading and I'll update this answer once I have that info. Cheers.

Use tf.metrics in Keras?

I'm especially interested in specificity_at_sensitivity. Looking through the Keras docs:
from keras import metrics
model.compile(loss='mean_squared_error',
optimizer='sgd',
metrics=[metrics.mae, metrics.categorical_accuracy])
But it looks like the metrics list must have functions of arity 2, accepting (y_true, y_pred) and returning a single tensor value.
EDIT: Currently here is how I do things:
from sklearn.metrics import confusion_matrix
predictions = model.predict(x_test)
y_test = np.argmax(y_test, axis=-1)
predictions = np.argmax(predictions, axis=-1)
c = confusion_matrix(y_test, predictions)
print('Confusion matrix:\n', c)
print('sensitivity', c[0, 0] / (c[0, 1] + c[0, 0]))
print('specificity', c[1, 1] / (c[1, 1] + c[1, 0]))
The disadvantage of this approach, is I only get the output I care about when training has finished. Would prefer to get metrics every 10 epochs or so.

I've found a related issue on github, and it seems that tf.metrics are still not supported by Keras models. However, in case you are very interested in using tf.metrics.specificity_at_sensitivity, I would suggest the following workaround (inspired by BogdanRuzh's solution):
def specificity_at_sensitivity(sensitivity, **kwargs):
def metric(labels, predictions):
# any tensorflow metric
value, update_op = tf.metrics.specificity_at_sensitivity(labels, predictions, sensitivity, **kwargs)
# find all variables created for this metric
metric_vars = [i for i in tf.local_variables() if 'specificity_at_sensitivity' in i.name.split('/')[2]]
# Add metric variables to GLOBAL_VARIABLES collection.
# They will be initialized for new session.
for v in metric_vars:
tf.add_to_collection(tf.GraphKeys.GLOBAL_VARIABLES, v)
# force to update metric values
with tf.control_dependencies([update_op]):
value = tf.identity(value)
return value
return metric
model.compile(loss='mean_squared_error',
optimizer='sgd',
metrics=[metrics.mae,
metrics.categorical_accuracy,
specificity_at_sensitivity(0.5)])
UPDATE:
You can use model.evaluate to retrieve the metrics after training.

I don't think there is a strict limit to only two incoming arguments, in metrics.py the function is just three incoming arguments, but k selects the default value of 5.
def sparse_top_k_categorical_accuracy(y_true, y_pred, k=5):
return K.mean(K.in_top_k(y_pred, K.cast(K.max(y_true, axis=-1), 'int32'), k), axis=-1)

Make a custom loss function in keras

Hi I have been trying to make a custom loss function in keras for dice_error_coefficient. It has its implementations in tensorboard and I tried using the same function in keras with tensorflow but it keeps returning a NoneType when I used model.train_on_batch or model.fit where as it gives proper values when used in metrics in the model. Can please someone help me out with what should i do? I have tried following libraries like Keras-FCN by ahundt where he has used custom loss functions but none of it seems to work. The target and output in the code are y_true and y_pred respectively as used in the losses.py file in keras.
def dice_hard_coe(target, output, threshold=0.5, axis=[1,2], smooth=1e-5):
"""References
-----------
- `Wiki-Dice <https://en.wikipedia.org/wiki/Sørensen–Dice_coefficient>`_
"""
output = tf.cast(output > threshold, dtype=tf.float32)
target = tf.cast(target > threshold, dtype=tf.float32)
inse = tf.reduce_sum(tf.multiply(output, target), axis=axis)
l = tf.reduce_sum(output, axis=axis)
r = tf.reduce_sum(target, axis=axis)
hard_dice = (2. * inse + smooth) / (l + r + smooth)
hard_dice = tf.reduce_mean(hard_dice)
return hard_dice

There are two steps in implementing a parameterized custom loss function in Keras. First, writing a method for the coefficient/metric. Second, writing a wrapper function to format things the way Keras needs them to be.
It's actually quite a bit cleaner to use the Keras backend instead of tensorflow directly for simple custom loss functions like DICE. Here's an example of the coefficient implemented that way:
import keras.backend as K
def dice_coef(y_true, y_pred, smooth, thresh):
y_pred = y_pred > thresh
y_true_f = K.flatten(y_true)
y_pred_f = K.flatten(y_pred)
intersection = K.sum(y_true_f * y_pred_f)
return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)
Now for the tricky part. Keras loss functions must only take (y_true, y_pred) as parameters. So we need a separate function that returns another function.
def dice_loss(smooth, thresh):
def dice(y_true, y_pred)
return -dice_coef(y_true, y_pred, smooth, thresh)
return dice
Finally, you can use it as follows in Keras compile.
# build model
model = my_model()
# get the loss function
model_dice = dice_loss(smooth=1e-5, thresh=0.5)
# compile model
model.compile(loss=model_dice)

According to the documentation, you can use a custom loss function like this:
Any callable with the signature loss_fn(y_true, y_pred) that returns an array of losses (one of sample in the input batch) can be passed to compile() as a loss. Note that sample weighting is automatically supported for any such loss.
As a simple example:
def my_loss_fn(y_true, y_pred):
squared_difference = tf.square(y_true - y_pred)
return tf.reduce_mean(squared_difference, axis=-1) # Note the `axis=-1`
model.compile(optimizer='adam', loss=my_loss_fn)
Complete example:
import tensorflow as tf
import numpy as np
def my_loss_fn(y_true, y_pred):
squared_difference = tf.square(y_true - y_pred)
return tf.reduce_mean(squared_difference, axis=-1) # Note the `axis=-1`
model = tf.keras.Sequential([
tf.keras.layers.Dense(8, activation='relu'),
tf.keras.layers.Dense(16, activation='relu'),
tf.keras.layers.Dense(1)])
model.compile(optimizer='adam', loss=my_loss_fn)
x = np.random.rand(1000)
y = x**2
history = model.fit(x, y, epochs=10)

In addition, you can extend an existing loss function by inheriting from it. For example masking the BinaryCrossEntropy:
class MaskedBinaryCrossentropy(tf.keras.losses.BinaryCrossentropy):
def call(self, y_true, y_pred):
mask = y_true != -1
y_true = y_true[mask]
y_pred = y_pred[mask]
return super().call(y_true, y_pred)
A good starting point is the custom log guide: https://www.tensorflow.org/guide/keras/train_and_evaluate#custom_losses

How to use F1 Score with Keras model?

For some reason I get error message when trying to specify f1 score with Keras model:
model.compile(optimizer='adam', loss='mse', metrics=['accuracy', 'f1_score'])
I get this error:
ValueError: Unknown metric function:f1_score
After providing 'f1_score' function in the same file where I use 'model.compile' like this:
def f1_score(y_true, y_pred):
# Count positive samples.
c1 = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
c2 = K.sum(K.round(K.clip(y_pred, 0, 1)))
c3 = K.sum(K.round(K.clip(y_true, 0, 1)))
# If there are no true samples, fix the F1 score at 0.
if c3 == 0:
return 0
# How many selected items are relevant?
precision = c1 / c2
# How many relevant items are selected?
recall = c1 / c3
# Calculate f1_score
f1_score = 2 * (precision * recall) / (precision + recall)
return f1_score
model.compile(optimizer='adam', loss='mse', metrics=['accuracy', f1_score])
Model compiles all right and can be saved to a file:
model.save(model_path) # works ok
Yet loading it in another program, :
from keras import models
model = models.load_model(model_path)
fails with an error:
ValueError: Unknown metric function:f1_score
Specifying 'f1_score' in the same file this time does not help, Keras does not see it. What's wrong? How to use F1 Score with Keras model?

When you load the model, you have to supply that metric as part of the custom_objects bag.
Try it like this:
from keras import models
model = models.load_model(model_path, custom_objects= {'f1_score': f1_score})
Where f1_score is the function that you passed through compile.

For your implementation of f1_score to work I had to switch y_true and y_pred in the function declaration.
P.S.: for those who asked: K = keras.backend

change:
metrics=['accuracy', f1_score]
to:
metrics=[f1_score]