I need to create a custom loss function in Keras and depending on the result of the conditional return two different loss values. I am having trouble getting the if statement to run properly.
I need to do something similar to this:
def custom_loss(y_true, y_pred):
sees = tf.Session()
const = 2
if (sees.run(tf.keras.backend.less(y_pred, y_true))): #i.e. y_pred - y_true < 0
return const * mean_squared_error(y_true, y_pred)
else:
return mean_squared_error(y_true, y_pred)
I keep getting tensor errors (see below) when trying to run this. Any help/advice will be appreciated!
InvalidArgumentError: You must feed a value for placeholder tensor 'dense_63_target' with dtype float and shape [?,?]
[[Node: dense_63_target = Placeholder[dtype=DT_FLOAT, shape=[?,?], _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
You should instead multiply simply by a mask in order to get your desired function
import keras.backend as K
def custom_1loss(y_true, y_pred):
const = 2
mask = K.less(y_pred, y_true) #i.e. y_pred - y_true < 0
return (const - 1) * mask * mean_squared_error(y_true, y_pred) + mean_squared_error(y_true, y_pred)
which has the same desired output as when y_pred is an under-prediction, another MSE term is added. You may have to cast the mask to an integer tensor - I do not remember what specific types - but it would be a minor change.
Also as unsolicited advice to your approach in general. I think you would get better results with a different approach to loss.
import keras.backend as K
def custom_loss2(y_true, y_pred):
beta = 0.1
return mean_squared_error(y_true, y_pred) + beta*K.mean(y_true - y_pred)
observe the difference in gradient behavior:
https://www.desmos.com/calculator/uubwgdhpi6
the second loss function I show you shifts the moment of the local minimum to be a minor over prediction rather than an under prediction (based on what you want). The loss function you give still locally optimizes to mean 0 but with different strength gradients. This will most likely result in simply a slower convergence to the same result as MSE rather than desiring a model that would rather over-predict then under predict. I hope this makes sense.
Related
I have a dataset about 400 variables and 5 target columns. In many of the rows, only a few of the Y values are present, i.e. I have some unknown (NaNs) in the targets. I'm applying a custom loss function through TF to make sure that loss is only applied to predictions of Y values where there is a Y value to compare to.
def nan_friendly_loss(y_true, y_pred):
y_true = tf.cast(y_true, tf.float32)
y_pred = tf.cast(y_pred, tf.float32)
valids = tf.math.is_finite(
y_true
)
#Only use y's that aren't NaN.
y_true = y_true[valids] #tf.print(y_true)
y_pred = y_pred[valids] #tf.print(y_pred)
return = K.sum(K.square(y_pred-y_true))
For instance, If y_true is [1,2,Nan,NaN,5] and y_pred is [2,2,3,4,3], the loss would be 5 (1+0+0+0+4).
Now I'm trying to get a grasp on how well the model is performing on each of the targets. How can I make an accuracy function (mse for instance) so that, for each target Y, it skips the row when y_true is NaN when calculating average accuracies for the dataset? So far, the built in functions are just getting nan results, leading me to believe that it is unable to disregard NaN values in y_true.
For example, the Ys below
y_true = [[1,2,Nan,NaN,5],[3,4,5,6,7]]
y_pred = [[2,2,3,4,4],[3,4,3,5,5]],
should give the following result:
mse_accuracies == [0.5,0,4,1,2.5]
When we need to use a loss function (or metric) other than the ones available , we can construct our own custom function and pass to model.compile.
So define the custom loss function as below. nan_friendly_loss is somewhat similar but you are defining sum instead of mean -
def custom_metric(y_true, y_pred):
y_true = tf.cast(y_true, tf.float32)
y_pred = tf.cast(y_pred, tf.float32)
valids = tf.math.is_finite(y_true)
#Only use y's that aren't NaN.
y_true = y_true[valids] #tf.print(y_true)
y_pred = y_pred[valids] #tf.print(y_pred)
return K.mean(K.square(y_pred - y_true)
Modify your compile statement as below -
model.compile(optimizer='rmsprop',
loss = nan_friendly_loss,
metrics= custom_metric)
Now, your model's custom_metric will serve your requirement.
Hope this answers your question. Happy Learning.
I have my Neural network int TF2 and for that I want to make my own metric. In my function I iterate throw each tensor value and canlculate new value into output_list. That I will stack as my new y_pred and throw it into mean_absolute_error. Compilaction is OK, but in first iteration I get error in the title. What am I doing wrong?
#tf.function
def custom_metric_mae( y_true , y_pred ):
output_list=tf.TensorArray(dtype=tf.float32, size=tf.shape(y_pred))
for i in range(223):
dphi = abs(y_true[i][0]-y_pred[i][0])
if(dphi > 0.5):
output_list.write(i,1 - dphi)
else:
output_list.write(i,dphi)
y_PredChanged = output_list.stack()
return tf.metrics.mean_absolute_error(y_true , y_PredChanged)
My model:
model = keras.Sequential([
keras.layers.Flatten(input_shape=(32,32)),
keras.layers.Dense(64,activation="relu"),
keras.layers.Dense(32,activation="relu"),
keras.layers.Dense(16,activation="relu"),
keras.layers.Dense(1, activation='linear')
])
model.compile(optimizer="adam",loss = "mean_absolute_error",metrics=[custom_metric_mae])
From the documentation of Keras' custom metrics:
The function would need to take (y_true, y_pred) as arguments and return a single tensor value.
tf.metrics.mean_absolute_error returns two values: the actual MAE and an update_op that, once evaluated, updates the running values and returns the MAE value. I'm not entirely sure whether this is compatible with Keras, I'd suggest to replace it with keras.metrics.mae instead.
Do note that when using Keras's mae, you need to average the final result (the function is applied to the last dimension of y_true and y_pred and the result has shape y_true.shape[:-1] which, in general, won't be one value). To do so, use tf.math.reduce_mean:
return tf.math.reduce_mean(keras.metrics.mae(y_true , y_PredChanged))
When defining a custom loss function for a classification problem, is there a way to access particular elements of y_true and y_pred?
Use-case: multi-label classification problem where I wanna penalize the model extra if I predict a false positive for class 5 i.e. y_true[5] == 0 but y_pred[5] == 1
I'm defining the loss like:
def loss(y_true, y_pred):
wt = 10 if (y_true[5]==0 and y_pred[5]==1) else 1
return wt * binary_crossentropy(y_true, y_pred)
I also tried to check if K.gather(y_true, 5) == 0 but that doesn't seem to do it.
My batch size is > 1 (256) and i'm using fit_generator - if that makes any difference. Thanks!
Is there a way to access particular elements of y_true and y_pred?
The indexing of Keras tensors works similarly to the indexing of numpy arrays. The only difference is that the result is a Keras tensor. Therefore, you should use Keras operations subsequently.
Possible implementation of your loss function
For example, here is how your loss function might be implemented:
def loss(y_true, y_pred):
a = K.equal(y_true[:, 5], 0)
b = K.greater(y_pred[:, 5], 0.5)
condition = K.cast(a, 'float') * K.cast(b, 'float')
wt = 10 * condition + (1 - condition)
return K.mean(wt[:, None] * K.binary_crossentropy(y_true, y_pred), axis=-1)
NOTE: Not tested.
I'm new to Keras and neural networks in general. I'm trying to implement a custom loss function based on mean squared error for a multi-layer autoencoder to be used in anomaly detection. Basically the approach I'm going for is from here https://www.jstage.jst.go.jp/article/ipsjjip/27/0/27_335/_pdf
Unfortunately I don't have the reputation to post images as I'm also new to SO but the formula is on page 2, section 3 as Lprop
The intuition here is that I don't want the autoencoder to update weights for data points that return errors above the ap percentile of losses. This way it learns to reconstruct the inliers in the dataset while struggling with the outliers, hence detecting them as anomalous.
Here's some code I've tried and the compiled model
import keras.backend as K
c = 70.0
def mean_squared_errorx(y_true, y_pred):
es = K.square(y_pred - y_true)
const = np.percentile(es, c)
w = K.cast(K.less(const, K.mean(K.square(y_pred - y_true), axis=-1)), dtype = "float32")
return w * K.mean(K.square(y_pred - y_true), axis=-1)
#'mean_squared_error'
autoencoder.compile(optimizer=adam, loss=mean_squared_errorx)
autoencoder.fit(train, train,
epochs=num_epochs,
batch_size=round(len(train)/50),
shuffle=True,
validation_data=(train, train),
verbose = 0)
encoded_d = encoder.predict(train)
decoded_pred = decoder.predict(encoded_d)
The idea is to get the K.less to return a bool for each error, and then to convert it to a float to serve as a weight in the return statement.
I know the np.percentile part probably won't work on a Tensor but don't know how else to accomplish the percentile ranking.
With that code I'm getting this error message
InvalidArgumentError: Incompatible shapes: [37,21] vs. [37]
[[{{node loss_25/dense_104_loss/Less}}]]
where in this case the batch size is 37 and the number of features is 21. I appreciate any feedback on this or other parts of the code - thanks!
Found a potential workaround if anybody is working on something similar
import keras.backend as K
def mean_squared_error_w(y_true, y_pred):
mses = K.mean(K.square(y_pred - y_true), axis = -1)
std_of_mses = K.std(mses)
const = K.mean(mses, axis = -1) + (std_of_mses * 0.5)
mask = K.cast(K.less(K.mean(K.square(y_pred - y_true), axis=-1), const), dtype = "float32")
return mask * K.mean(K.square(y_pred - y_true), axis=-1)
I believe this will create a tensor of bools for all of the values where the error
is larger than a threshold value, defined by the mean of the batch MSEs plus half a standard deviation (if the errors were normally distributed this should correspond to about the 70th percentile of the data as the cutoff). It converts the bools to the weights 0 or 1 as a mask which is then applied to the output MSE loss
I am trying to implement a custom objective function in keras frame.
Respectively a weighted average function that takes the two arguments tensors y_true and y_pred ; the weights information is derived from y_true tensor.
Is there a weighted average function in tensorflow ?
Or any other suggestions on how to implement this kind of loss function ?
My function would look something like this:
function(y_true,y_pred)
A=(y_true-y_pred)**2
w - derivable from y_true, tensor of same shape as y_true
return average(A, weights=w) <-- a scalar
y_true and y_pred are 3D tensors.
you can use one of the existing objectives (also called loss) on keras from here.
you may also implement your own custom function loss:
from keras import backend as K
def my_loss(y_true, y_pred):
return K.mean(K.square(y_pred - y_true), axis=-1)
# Let's train the model using RMSprop
model.compile(loss=my_loss, optimizer='SGD', metrics=['accuracy'])
notice the K module, its the keras backend you should use to fully utilize keras performance, dont do something like this unless you dont care from performance issues:
def my_bad_and_slow_loss(y_true, y_pred):
return sum((y_pred - y_true) ** 2, axis=-1)
for your specific case, please write your desired objective function if you need help to write it.
Update
you can try this to provide weights - W as loss function:
def my_loss(y_true, y_pred):
W = np.arange(9) / 9. # some example W
return K.mean(K.pow(y_true - y_pred, 2) * W)