i have a time series dataset with multiple binary outputs per instance. For example the test labels for an instance may be [0 1 0 0 0 1 1 0 1 0].
Instead of calculating the accuracy for both class 0 and class 1 i would like to calculate accuracy for only class 1.
Right now i have defined:
metrics=[recall_m])
where:
from keras import backend as K
def recall_m(y_true, y_pred):
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
recall = true_positives / (possible_positives + K.epsilon())
return recall
How could i change this to calculate class-wise accuracy for a specific class?
Related
Whats the correct and optimized way to use multiple loss functions with Python and Keras?
I'm playing with this article:
https://medium.com/#polanitzer/predicting-the-israeli-lottery-results-for-the-november-29-2022-game-using-an-artificial-191489eb2c10
The author use the standard 'mse' as loss, and I wanna add some more custom functions, those functions they do not have mathematical accuracy, but are the result of observations only.
Let's say I have 5 situations:
Previous results will not happen again
The sum of results need be between V and W
From an game numbered from 00 to 99 I wanna limit the predicted numbers from 00 to 49 occurency times in X (numbers in this range cannot exceed X times)
From an game numbered from 00 to 99 I wanna limit the predicted numbers from 50
to 99 occurency times in Y (numbers in this range cannot exceed Y)
I wanna keep the default mse function
My approach started from:
Situation 1
def fn_never_repeat(y_true, y_pred, previous_data):
# Compute the difference score between the predicted outputs and the previous data
diff = K.mean(K.square(y_pred - previous_data), axis=-1)
# Return the weighted sum of the difference scores
return diff
Situation 2
def fn_sum_values(y_true, y_pred):
# Calculate the sum of the predicted numbers
predicted_sum = K.sum(y_pred)
# Set the minimum value to 133 and maximum value to 249
X = 133
Y = 249
# Calculate the loss based on the deviation from the desired range (X, Y)
loss = K.maximum(X - predicted_sum, 0) + K.maximum(predicted_sum - Y, 0)
return loss
Situation 3 and 4
(Using 10 as example for both quadrant)
def fn_quadrant(y_true, y_pred):
count_0_to_49 = K.sum(K.cast(K.less(y_pred, 50), 'float'))
count_50_to_99 = K.sum(K.cast(K.greater_equal(y_pred, 49), 'float'))
penalty = 0
if count_0_to_49 > 10:
penalty += K.square(count_0_to_49 - 10)
if count_50_to_99 > 10:
penalty += K.square(count_50_to_99 - 10)
return K.mean(K.square(y_true - y_pred)) + penalty
Situation 5
def fn_combined_loss(y_true, y_pred):
fn_never_repeat = fn_never_repeat(y_true, y_pred, previous_data)
fn_sum_values = fn_sum_values(y_true, y_pred)
fn_quadrant = fn_quadrant(y_true, y_pred)
mse = K.mean(K.square(y_true - y_pred))
return 0.25 * fn_never_repeat + 0.25 * fn_sum_values + 0.25 * fn_quadrant + 0.25 * mse
Calling it with:
model.compile(optimizer=Adam(learning_rate=0.0001), loss='fn_combined_loss', metrics=['accuracy'], custom_objects={'fn_combined_loss': fn_combined_loss})
And error occours after:
model.fit(x=x_train, y=y_train, batch_size=number_of_batch, epochs=Nepochs, verbose=1, callbacks=[callbacks_list])
I'm stuck on this error:
ValueError: Unknown loss function: combined_loss_fn. Please ensure
this object is passed to the custom_objects argument. See
https://www.tensorflow.org/guide/keras/save_and_serialize#registering_the_custom_object
for details.
The paremeters from lowest times or X times something can happen or cant exceed will come from stored procedure in database. And since the value is provided based on registered results, although it is not an absolute mathematical statistic, it will be dynamically informed.
I'm not sure if functions are correctly, since it is not compiling yet, you guys can point any mistaken if you seen something wrong too.
Thanks in advice!!!
With these few changes I think your training should work
Compile your model with the actual loss function, not as string
model.compile(optimizer=Adam(learning_rate=0.0001),loss=fn_combined_loss,metrics=['accuracy'])
In the combined loss rename variables, do not use the same name as the loss functions have
def fn_combined_loss(y_true, y_pred):
fn_never_repeat_ = fn_never_repeat(y_true, y_pred)
fn_sum_values_ = fn_sum_values(y_true, y_pred)
fn_quadrant_ = fn_quadrant(y_true, y_pred)
mse = K.mean(K.square(y_true - y_pred))
return 0.25 * fn_never_repeat_ + 0.25 * fn_sum_values_ + 0.25 * fn_quadrant_ + 0.25 * mse
In fn_quadrant change the constants to float
def fn_quadrant(y_true, y_pred):
count_0_to_49 = K.sum(K.cast(K.less(y_pred, 50), 'float'))
count_50_to_99 = K.sum(K.cast(K.greater_equal(y_pred, 49), 'float'))
penalty = 0.
if count_0_to_49 > 10:
penalty += K.square(count_0_to_49 - 10.)
if count_50_to_99 > 10:
penalty += K.square(count_50_to_99 - 10.)
return K.mean(K.square(y_true - y_pred)) + penalty
I have trained a model in Google Colab and trying to load the model in my local machine to use on the web app. But as soon as I load_model() it shows this error:
Unable to restore custom object of type _tf_keras_metric. Please make sure that any custom layers are included in the `custom_objects` arg when calling `load_model()` and make sure that all layers implement `get_config` and `from_config`.
I have used CategoricalAccuracy as a metric in the model and I tried to pass it as a custom object but I get the same error.
Code used in Web App
def load_text_model():
model = keras.models.load_model("." + url_for('static', filename='models/text_model'), custom_objects={'CategoricalAccuracy': tf.keras.metrics.CategoricalAccuracy(name="accuracy")})
return model
Extra information if needed: I am making a multi-class text classification model. I have used bert english uncased model from thub as preprocessor and encoder. Below are the metrics I have used. If more code or context is required, I will provide it.
Code used while training model
def balanced_recall(y_true, y_pred):
"""This function calculates the balanced recall metric
recall = TP / (TP + FN)
"""
recall_by_class = 0
# iterate over each predicted class to get class-specific metric
for i in range(y_pred.shape[1]):
y_pred_class = y_pred[:, i]
y_true_class = y_true[:, i]
true_positives = K.sum(K.round(K.clip(y_true_class * y_pred_class, 0, 1)))
possible_positives = K.sum(K.round(K.clip(y_true_class, 0, 1)))
recall = true_positives / (possible_positives + K.epsilon())
recall_by_class = recall_by_class + recall
return recall_by_class / y_pred.shape[1]
def balanced_precision(y_true, y_pred):
"""This function calculates the balanced precision metric
precision = TP / (TP + FP)
"""
precision_by_class = 0
# iterate over each predicted class to get class-specific metric
for i in range(y_pred.shape[1]):
y_pred_class = y_pred[:, i]
y_true_class = y_true[:, i]
true_positives = K.sum(K.round(K.clip(y_true_class * y_pred_class, 0, 1)))
predicted_positives = K.sum(K.round(K.clip(y_pred_class, 0, 1)))
precision = true_positives / (predicted_positives + K.epsilon())
precision_by_class = precision_by_class + precision
# return average balanced metric for each class
return precision_by_class / y_pred.shape[1]
def balanced_f1_score(y_true, y_pred):
"""This function calculates the F1 score metric"""
precision = balanced_precision(y_true, y_pred)
recall = balanced_recall(y_true, y_pred)
return 2 * ((precision * recall) / (precision + recall + K.epsilon()))
i = tf.keras.layers.Input(shape=(), dtype=tf.string, name='text')
x = preprocessor(i)
x = encoder(x)
x = tf.keras.layers.Dropout(0.2, name="dropout")(x['default'])
x = tf.keras.layers.Dense(num_classes, activation='softmax', name="output")(x)
model = tf.keras.Model(i, x)
METRICS = [
tf.keras.metrics.CategoricalAccuracy(name="accuracy"),
balanced_recall,
balanced_precision,
balanced_f1_score
]
I am trying to create two custom functions f1_metric and auc_metric in Keras. The f1_metric works, but the auc not, and I receive different errors. Here is my code:
def f1_metric(y_true, y_pred):
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
precision = true_positives / (predicted_positives + K.epsilon())
recall = true_positives / (possible_positives + K.epsilon())
f1_val = 2*(precision*recall)/(precision+recall+K.epsilon())
return f1_val
def auc_metric(y_true, yhat):
yhat = yhat[:, 1]
# calculate roc curves
fpr, tpr, thresholds = roc_curve(y_true, yhat)
auc=auc(tpr,fpr)
return auc
Here are my compile and fit codes:
opt = SGD(lr=0.01,momentum=0.9)
model.compile(loss='binary_crossentropy', optimizer=opt,metrics=['accuracy']) #
ca = SnapshotEnsemble(n_epochs, n_cycles, 0.01)
# fit model
history=model.fit(trainX, trainy, validation_data=(testX, testy), epochs=n_epochs,
verbose='auto', callbacks=[ca],batch_size=32)
Any Idea how to solve it? I want to define a similar function as I did for f1_metric.
From comments
Approximates the AUC (Area under the curve) of the ROC or PR curves.
tf.keras.metrics.AUC(
num_thresholds=200, curve='ROC',
summation_method='interpolation', name=None, dtype=None,
thresholds=None, multi_label=False, num_labels=None, label_weights=None,
from_logits=False
)
For more details you can refer here
(paraphrased from Captain Trojan)
Assume that y_true and y_pred are in [-1,1]. I want a weighted mean-square-error loss function, in which the loss for samples that are positive in the y_true and negative in y_pred or vice versa are weighted by exp(alpha). Here is my code:
import keras.backend as K
alpha = 1.0
def custom_loss(y_true, y_pred):
se = K.square(y_pred-y_true)
true_label = K.less_equal(y_true,0.0)
pred_label = K.less_equal(y_pred,0.0)
return K.mean(se * K.exp(alpha*K.cast(K.not_equal(true_label,pred_label), tf.float32)))
And here is a plot of this loss function. Different curves are for different values for y_true.
I want to know:
Whether this is a valid loss function, since it is not differentiable in 0?
Is my code correct?
I suggest you this type of loss function to handle imbalance dataset
def focal_loss(y_true, y_pred):
gamma = 2.0, alpha = 0.25
pt_1 = tf.where(tf.equal(y_true, 1), y_pred, tf.ones_like(y_pred))
pt_0 = tf.where(tf.equal(y_true, 0), y_pred, tf.zeros_like(y_pred))
return -K.sum(alpha * K.pow(1. - pt_1, gamma) * K.log(pt_1))-K.sum((1-alpha) * K.pow(pt_0, gamma) * K.log(1. - pt_0))
from this source
I found this function here
How to calculate F1 Macro in Keras? but i am not sure how i can write the same way for specificity? I am using tensorflow backend for keras.
def recall(y_true, y_pred):
"""Recall metric.
Only computes a batch-wise average of recall.
Computes the recall, a metric for multi-label classification of
how many relevant items are selected.
"""
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
recall = true_positives / (possible_positives + K.epsilon())
return recall
I tried this solution but it gives error,
def compute_binary_specificity(y_pred, y_true):
"""Compute the confusion matrix for a set of predictions.
Returns
-------
out : the specificity
"""
TN = np.logical_and(K.eval(y_true) == 0, K.eval(y_pred) == 0)
FP = np.logical_and(K.eval(y_true) == 0, K.eval(y_pred) == 1)
# as Keras Tensors
TN = K.sum(K.variable(TN))
FP = K.sum(K.variable(FP))
specificity = TN / (TN + FP + K.epsilon())
return specificity
Error: InvalidArgumentError: You must feed a value for placeholder tensor 'dense_95_input' with dtype float and shape [?,140]
[[Node: dense_95_input = Placeholderdtype=DT_FLOAT, shape=[?,140], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
and points here
---> TN = np.logical_and(K.eval(y_true) == 0, K.eval(y_pred) == 0)