When defining a custom loss function for a classification problem, is there a way to access particular elements of y_true and y_pred?
Use-case: multi-label classification problem where I wanna penalize the model extra if I predict a false positive for class 5 i.e. y_true[5] == 0 but y_pred[5] == 1
I'm defining the loss like:
def loss(y_true, y_pred):
wt = 10 if (y_true[5]==0 and y_pred[5]==1) else 1
return wt * binary_crossentropy(y_true, y_pred)
I also tried to check if K.gather(y_true, 5) == 0 but that doesn't seem to do it.
My batch size is > 1 (256) and i'm using fit_generator - if that makes any difference. Thanks!
Is there a way to access particular elements of y_true and y_pred?
The indexing of Keras tensors works similarly to the indexing of numpy arrays. The only difference is that the result is a Keras tensor. Therefore, you should use Keras operations subsequently.
Possible implementation of your loss function
For example, here is how your loss function might be implemented:
def loss(y_true, y_pred):
a = K.equal(y_true[:, 5], 0)
b = K.greater(y_pred[:, 5], 0.5)
condition = K.cast(a, 'float') * K.cast(b, 'float')
wt = 10 * condition + (1 - condition)
return K.mean(wt[:, None] * K.binary_crossentropy(y_true, y_pred), axis=-1)
NOTE: Not tested.
Related
I am using the Image segmentation guide by fchollet to perform semantic segmentation. I have attempted modifying the guide to suit my dataset by labelling the 8-bit img mask values into 1 and 2 like in the Oxford Pets dataset. (which will be subtracted to 0 and 1 in class OxfordPets(keras.utils.Sequence):)
Question is how do I get the IoU metric of a single class (e.g 1)?
I have tried different metrics suggested by Stack Overflow but most of suggest using MeanIoU which I tried but I have gotten nan loss as a result. Here is an example of a mask after using autocontrast.
PIL.ImageOps.autocontrast(load_img(val_target_img_paths[i]))
The model seems to train well but the accuracy was decreasing over time.
Also, can someone help explain how the metric score can be calculated from y_true and y_pred? I don't quite fully understand when the label value is used in the IoU metric calculation.
I had a similar problem back then. I used jaccard_distance_loss and dice_metric. They are based on IoU. My task was a binary segmentation, so I guess you might have to modify the code in case you want to use it for a multi-label classification problem.
from keras import backend as K
def jaccard_distance_loss(y_true, y_pred, smooth=100):
"""
Jaccard = (|X & Y|)/ (|X|+ |Y| - |X & Y|)
= sum(|A*B|)/(sum(|A|)+sum(|B|)-sum(|A*B|))
The jaccard distance loss is usefull for unbalanced datasets. This has been
shifted so it converges on 0 and is smoothed to avoid exploding or disapearing
gradient.
Ref: https://en.wikipedia.org/wiki/Jaccard_index
#url: https://gist.github.com/wassname/f1452b748efcbeb4cb9b1d059dce6f96
#author: wassname
"""
intersection = K.sum(K.sum(K.abs(y_true * y_pred), axis=-1))
sum_ = K.sum(K.sum(K.abs(y_true) + K.abs(y_pred), axis=-1))
jac = (intersection + smooth) / (sum_ - intersection + smooth)
return (1 - jac) * smooth
def dice_metric(y_pred, y_true):
intersection = K.sum(K.sum(K.abs(y_true * y_pred), axis=-1))
union = K.sum(K.sum(K.abs(y_true) + K.abs(y_pred), axis=-1))
# if y_pred.sum() == 0 and y_pred.sum() == 0:
# return 1.0
return 2*intersection / union
# Example
size = 10
y_true = np.zeros(shape=(size,size))
y_true[3:6,3:6] = 1
y_pred = np.zeros(shape=(size,size))
y_pred[3:5,3:5] = 1
loss = jaccard_distance_loss(y_true,y_pred)
metric = dice_metric(y_pred,y_true)
print(f"loss: {loss}")
print(f"dice_metric: {metric}")
loss: 4.587155963302747
dice_metric: 0.6153846153846154
You can use the tf.keras.metrics.IoU method, and you can find its documentation here: https://www.tensorflow.org/api_docs/python/tf/keras/metrics/IoU
And how you can use it is shown here:
import tensorflow as tf
y_true = [0, 1]
y_pred = [0.5, 1.0] # they must be the same shape
# target_class_ids indicates the class/classes you want to calculate IoU on
loss = tf.keras.metrics.IoU(num_classes=2, target_class_ids=[1])
loss.update_state(y_true, y_pred)
print(loss.result().numpy())
And as you can see in the documentation, IoU is calculated by:
true_positives / (true_positives + false_positives + false_negatives)
I'm new to Keras and neural networks in general. I'm trying to implement a custom loss function based on mean squared error for a multi-layer autoencoder to be used in anomaly detection. Basically the approach I'm going for is from here https://www.jstage.jst.go.jp/article/ipsjjip/27/0/27_335/_pdf
Unfortunately I don't have the reputation to post images as I'm also new to SO but the formula is on page 2, section 3 as Lprop
The intuition here is that I don't want the autoencoder to update weights for data points that return errors above the ap percentile of losses. This way it learns to reconstruct the inliers in the dataset while struggling with the outliers, hence detecting them as anomalous.
Here's some code I've tried and the compiled model
import keras.backend as K
c = 70.0
def mean_squared_errorx(y_true, y_pred):
es = K.square(y_pred - y_true)
const = np.percentile(es, c)
w = K.cast(K.less(const, K.mean(K.square(y_pred - y_true), axis=-1)), dtype = "float32")
return w * K.mean(K.square(y_pred - y_true), axis=-1)
#'mean_squared_error'
autoencoder.compile(optimizer=adam, loss=mean_squared_errorx)
autoencoder.fit(train, train,
epochs=num_epochs,
batch_size=round(len(train)/50),
shuffle=True,
validation_data=(train, train),
verbose = 0)
encoded_d = encoder.predict(train)
decoded_pred = decoder.predict(encoded_d)
The idea is to get the K.less to return a bool for each error, and then to convert it to a float to serve as a weight in the return statement.
I know the np.percentile part probably won't work on a Tensor but don't know how else to accomplish the percentile ranking.
With that code I'm getting this error message
InvalidArgumentError: Incompatible shapes: [37,21] vs. [37]
[[{{node loss_25/dense_104_loss/Less}}]]
where in this case the batch size is 37 and the number of features is 21. I appreciate any feedback on this or other parts of the code - thanks!
Found a potential workaround if anybody is working on something similar
import keras.backend as K
def mean_squared_error_w(y_true, y_pred):
mses = K.mean(K.square(y_pred - y_true), axis = -1)
std_of_mses = K.std(mses)
const = K.mean(mses, axis = -1) + (std_of_mses * 0.5)
mask = K.cast(K.less(K.mean(K.square(y_pred - y_true), axis=-1), const), dtype = "float32")
return mask * K.mean(K.square(y_pred - y_true), axis=-1)
I believe this will create a tensor of bools for all of the values where the error
is larger than a threshold value, defined by the mean of the batch MSEs plus half a standard deviation (if the errors were normally distributed this should correspond to about the 70th percentile of the data as the cutoff). It converts the bools to the weights 0 or 1 as a mask which is then applied to the output MSE loss
I need to create a custom loss function in Keras and depending on the result of the conditional return two different loss values. I am having trouble getting the if statement to run properly.
I need to do something similar to this:
def custom_loss(y_true, y_pred):
sees = tf.Session()
const = 2
if (sees.run(tf.keras.backend.less(y_pred, y_true))): #i.e. y_pred - y_true < 0
return const * mean_squared_error(y_true, y_pred)
else:
return mean_squared_error(y_true, y_pred)
I keep getting tensor errors (see below) when trying to run this. Any help/advice will be appreciated!
InvalidArgumentError: You must feed a value for placeholder tensor 'dense_63_target' with dtype float and shape [?,?]
[[Node: dense_63_target = Placeholder[dtype=DT_FLOAT, shape=[?,?], _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
You should instead multiply simply by a mask in order to get your desired function
import keras.backend as K
def custom_1loss(y_true, y_pred):
const = 2
mask = K.less(y_pred, y_true) #i.e. y_pred - y_true < 0
return (const - 1) * mask * mean_squared_error(y_true, y_pred) + mean_squared_error(y_true, y_pred)
which has the same desired output as when y_pred is an under-prediction, another MSE term is added. You may have to cast the mask to an integer tensor - I do not remember what specific types - but it would be a minor change.
Also as unsolicited advice to your approach in general. I think you would get better results with a different approach to loss.
import keras.backend as K
def custom_loss2(y_true, y_pred):
beta = 0.1
return mean_squared_error(y_true, y_pred) + beta*K.mean(y_true - y_pred)
observe the difference in gradient behavior:
https://www.desmos.com/calculator/uubwgdhpi6
the second loss function I show you shifts the moment of the local minimum to be a minor over prediction rather than an under prediction (based on what you want). The loss function you give still locally optimizes to mean 0 but with different strength gradients. This will most likely result in simply a slower convergence to the same result as MSE rather than desiring a model that would rather over-predict then under predict. I hope this makes sense.
I am having a hard time with calculating cross entropy in tensorflow. In particular, I am using the function:
tf.nn.softmax_cross_entropy_with_logits()
Using what is seemingly simple code, I can only get it to return a zero
import tensorflow as tf
import numpy as np
sess = tf.InteractiveSession()
a = tf.placeholder(tf.float32, shape =[None, 1])
b = tf.placeholder(tf.float32, shape = [None, 1])
sess.run(tf.global_variables_initializer())
c = tf.nn.softmax_cross_entropy_with_logits(
logits=b, labels=a
).eval(feed_dict={b:np.array([[0.45]]), a:np.array([[0.2]])})
print c
returns
0
My understanding of cross entropy is as follows:
H(p,q) = p(x)*log(q(x))
Where p(x) is the true probability of event x and q(x) is the predicted probability of event x.
There if input any two numbers for p(x) and q(x) are used such that
0<p(x)<1 AND 0<q(x)<1
there should be a nonzero cross entropy. I am expecting that I am using tensorflow incorrectly. Thanks in advance for any help.
In addition to Don's answer (+1), this answer written by mrry may interest you, as it gives the formula to calculate the cross entropy in TensorFlow:
An alternative way to write:
xent = tf.nn.softmax_cross_entropy_with_logits(logits, labels)
...would be:
softmax = tf.nn.softmax(logits)
xent = -tf.reduce_sum(labels * tf.log(softmax), 1)
However, this alternative would be (i) less numerically stable (since
the softmax may compute much larger values) and (ii) less efficient
(since some redundant computation would happen in the backprop). For
real uses, we recommend that you use
tf.nn.softmax_cross_entropy_with_logits().
Like they say, you can't spell "softmax_cross_entropy_with_logits" without "softmax". Softmax of [0.45] is [1], and log(1) is 0.
Measures the probability error in discrete classification tasks in which the
classes are mutually exclusive (each entry is in exactly one class). For
example, each CIFAR-10 image is labeled with one and only one label: an image
can be a dog or a truck, but not both.
NOTE: While the classes are mutually exclusive, their probabilities
need not be. All that is required is that each row of labels is
a valid probability distribution. If they are not, the computation of the
gradient will be incorrect.
If using exclusive labels (wherein one and only
one class is true at a time), see sparse_softmax_cross_entropy_with_logits.
WARNING: This op expects unscaled logits, since it performs a softmax
on logits internally for efficiency. Do not call this op with the
output of softmax, as it will produce incorrect results.
logits and labels must have the same shape [batch_size, num_classes]
and the same dtype (either float16, float32, or float64).
Here is an implementation in Tensorflow 2.0 in case somebody else (me probably) needs it in the future.
#tf.function
def cross_entropy(x, y, epsilon = 1e-9):
return -2 * tf.reduce_mean(y * tf.math.log(x + epsilon), -1) / tf.math.log(2.)
x = tf.constant([
[1.0,0],
[0.5,0.5],
[.75,.25]
]
,dtype=tf.float32)
with tf.GradientTape() as tape:
tape.watch(x)
y = entropy(x, x)
tf.print(y)
tf.print(tape.gradient(y, x))
Output
[-0 1 0.811278105]
[[-1.44269502 29.8973541]
[-0.442695022 -0.442695022]
[-1.02765751 0.557305]]
I am trying to implement a custom objective function in keras frame.
Respectively a weighted average function that takes the two arguments tensors y_true and y_pred ; the weights information is derived from y_true tensor.
Is there a weighted average function in tensorflow ?
Or any other suggestions on how to implement this kind of loss function ?
My function would look something like this:
function(y_true,y_pred)
A=(y_true-y_pred)**2
w - derivable from y_true, tensor of same shape as y_true
return average(A, weights=w) <-- a scalar
y_true and y_pred are 3D tensors.
you can use one of the existing objectives (also called loss) on keras from here.
you may also implement your own custom function loss:
from keras import backend as K
def my_loss(y_true, y_pred):
return K.mean(K.square(y_pred - y_true), axis=-1)
# Let's train the model using RMSprop
model.compile(loss=my_loss, optimizer='SGD', metrics=['accuracy'])
notice the K module, its the keras backend you should use to fully utilize keras performance, dont do something like this unless you dont care from performance issues:
def my_bad_and_slow_loss(y_true, y_pred):
return sum((y_pred - y_true) ** 2, axis=-1)
for your specific case, please write your desired objective function if you need help to write it.
Update
you can try this to provide weights - W as loss function:
def my_loss(y_true, y_pred):
W = np.arange(9) / 9. # some example W
return K.mean(K.pow(y_true - y_pred, 2) * W)