I'm trying to write a pixel-wise weighted loss function for my model written in Keras but in TensorFlow 2.0 it seems that it is not possible anymore, i.e. it is not possible to have a loss function with other inputs than y_true and y_pred
I used to write it as follows:
from tensorflow.keras.layers import Input, Conv2D
from tensorflow.keras.optimizers import Adam
from tensorflow.keras import backend as K
def my_keras_model():
input = Input((256,256,1), name='input')
weight = Input((256,256,1), name='weights')
c1 = Conv2D(16, (3, 3), activation='relu', kernel_initializer='glorot_uniform', padding='same')(input)
outputs = Conv2D(1, (1, 1), activation='sigmoid')(c1)
model=Model(input=[input,weight], output=outputs)
model.compile(optimizer=Adam(learning_rate=0.001, name='adam'), loss=my_weighted_loss(weight))
return model
def my_weighted_loss(weight):
def loss(y_true, y_pred):
return K.mean(weight * K.binary_crossentropy(y_true, y_pred), axis=-1)
return loss
Any idea of how to do it in TF 2?
One "hacky" way of implementing this would be adding the original input to the output, and writing your own loss function. This way you can do
weight = y_true[...,0]
y_true = y_true[...,1:]
I would also love to hear a better answer :)
Actually it is possible to implement weight maps and do it computation inside the model.
Since it is binary cross_entropy
model=Model(inputs=[image,weight,mask], outputs=outputs)
Define your model in such a way that incase if your using tf dataset.
output_types=((tf.float32,tf.float32,tf.float32),tf.float32)
output_shapes=(([1024,1024,1],[1024,1024,1],[1024,1024,1]),[1024,1024,1])
Now compute the loss function inside the model
bce = y_true * K.log(y_pred+epsilon) + (1-y_true) * K.log(1-y_pred+epsilon) #you have values of y_true also
Here model output would be this computed loss.
Incase if you need a computation out of your model. Just use a Lambda layer for the weights.
weights_out=layers.Lambda(lambda x:x)(weights)
and then output this layer also from your model. So model would have 2 outputs to compute the loss in the form of a tuple and this way also pixelwise weighted loss can be calculated.
model=Model(inputs=[image,weights,mask], outputs=[outputs,weighted_out])
Related
I am interested in using a neural network to estimate the parameters of a linear regression. To do this I am creating a network that makes two-parameter prediction, and I am trying to write a custom loss function that will determine how well the two parameters do as a slope and intercept in a logistic regression model, using a third dataset as a predictor in the logistic regression.
So I have a matrix of predictors X, with dimensions 10,000 by 20, and a binary outcome variable y. Additionally, I have a 10,000 observations linear_predictor that I want to use to use in the custom loss function evaluate the two outputs of the model.
import numpy as np
from tensorflow.keras import Model, Input
from tensorflow.keras import Model, Input
from tensorflow.keras.layers import Dense
import tensorflow as tf
# create some dummy data
X = np.random.rand(10_000, 20)
y = (np.random.rand(10_000) > 0.8).astype(int)
linear_predictor = np.random.rand(10_000)
# define custom loss function
def CustomLoss(y_true, y_pred, input_):
y_estim = y_pred[:,0]*input_ + y_pred[:,1]
y_estim = tf.gather(y_pred, 0, axis=1)*input_ + tf.gather(y_pred, 1, axis=1)
return tf.keras.losses.BinaryCrossentropy(from_logits=True)(y_true, y_estim)
# create inputs to model
lp_input = Input(shape=linear_predictor.shape)
X_input = Input(shape=X.shape)
y_input = Input(shape=y.shape)
# create network
hidden1 = Dense(32, activation='relu')(X_input)
hidden2 = Dense(8, activation='relu')(hidden1)
output = Dense(2, activation='linear')(hidden2)
model = Model([y_input, X_input, lp_input], output)
# add loss function
model.add_loss(CustomLoss(y_input, output, lp_input))
# fit model
model.fit(x=X_input, y=y_input, epochs=3)
However, I am unable to get the CustomLoss function to work. Something is going wrong with subsetting the model's two-parameter output to get one parameter to use as a scalar as the slope and another to use as the intercept.
The error I am getting is:
ValueError: Exception encountered when calling layer "tf.math.multiply_1" (type TFOpLambda).
Dimensions must be equal, but are 2 and 10000 for '{{node tf.math.multiply_1/Mul}} = Mul[T=DT_FLOAT](
Placeholder, Placeholder_1)' with input shapes: [?,2], [?,10000].
Call arguments received by layer "tf.math.multiply_1" (type TFOpLambda):
• x=tf.Tensor(shape=(None, 2), dtype=float32)
• y=tf.Tensor(shape=(None, 10000), dtype=float32)
• name=None
This suggests that the variable y_pred is not being subset, even though I have tried using the method recommended here with numpy-like indexing (y_pred[:1]) as well as the gather_nd method here, among others.
I think this should be possible, any help is appreciated.
Is there a way to have two loss functions in Keras in which the second loss function takes the output from the first loss function?
I am working on a Neural Network with Keras and I want to add another custom function to the Loss term inside the model.compile() to regularize and somehow penalize it, which is the form:
model.compile(loss_1='mean_squared_error', optimizer=Adam(lr=learning_rate), metrics=['mae'])
I would like to add another loss function as a sum of the predicted values from the Loss_1 outputs so that I can tell the Neural Network to minimize the sum of the predicted values from the Loss_1 model. How can I do that (loss_2)?
Something like:
model.compile(loss_1='mean_squared_error', loss_2= np.sum(****PREDICTED_OUTPUT_FROM_LOSS_FUNCTION_1****), optimizer=Adam(lr=learning_rate), metrics=['mae'])
how can this be implemented?
You should define a custom loss function
def custom_loss_function(y_true, y_pred):
squared_difference = tf.square(y_true - y_pred)
absolute_difference = tf.abs(y_true - y_pred)
loss = tf.reduce_mean(squared_difference, axis=-1) + tf.reduce_mean(absolute_difference, axis=-1)
return loss
model.compile(optimizer='adam', loss=custom_loss_function)
I believe that would solve your problem
I am trying to understand how keras/tensorflow work.
In this example I'm working with, an LSTM network with a defined loss function.
I want to print the values in y_pred and loss variables in this example, however a standard print() function will not print the actual numeric values.
When I try print() function, I get the following output: Tensor("loss_13/dense_14_loss/strided_slice:0", shape=(), dtype=float32)
import tensorflow as tf
from tensorflow.keras import Sequential, backend as K
from tensorflow.keras.layers import Dense, LSTM, Dropout
from tensorflow.keras.losses import categorical_crossentropy
regressor = Sequential()
regressor.add(LSTM(units = 10, dropout=0.10, return_sequences = True, input_shape = (X.shape[1], X.shape[2])))
regressor.add(Dense(units = 4, activation='softmax'))
regressor.compile(optimizer = optimizer, loss = weight_fx(np.array([0.005,0.20,0.79,0.005])), metrics = ['categorical_accuracy'])
def weight_fx(weights):
weights = K.variable(weights)
def loss(y_true, y_pred):
y_pred /= K.sum(y_pred, axis=-1, keepdims=True)
print(y_pred)
loss = y_true * K.log(y_pred) * weights
return loss
return loss
Try doing it like this:
import tensorflow as tf
from tensorflow.keras import Sequential, backend as K
from tensorflow.keras.layers import Dense, LSTM, Dropout
from tensorflow.keras.losses import categorical_crossentropy
import numpy as np
X = tf.ones((10,10,10))
y = tf.ones((10,1))
def weight_fx(weights):
weights = K.variable(weights)
def loss(y_true, y_pred):
y_pred /= K.sum(y_pred, axis=-1, keepdims=True)
tf.print(y_pred)
loss = y_true * K.log(y_pred) * weights
return loss
return loss
regressor = Sequential()
regressor.add(LSTM(units = 10, dropout=0.10, return_sequences = True,
input_shape = (X.shape[1], X.shape[2])))
regressor.add(Dense(units = 4, activation='softmax'))
regressor.compile(optimizer = 'adam', loss = weight_fx(np.array([0.005,0.20,0.79,0.005])), metrics = ['categorical_accuracy'])
regressor.fit(X,y)
Q: Why do you see Tensor("loss_13/dense_14_loss/strided_slice:0", shape=(), dtype=float32)?
A: Tensorflow expects that the loss function is going to get called very often, so it is paramount to optimize it as much as possible. Tensorflow has a way of doing this, called 'tracing'. This basically means passing in a 'detector' variable that 'experiences' all operations in the function and remembers them. Then, based on these experiences, Tensorflow builds a separate so-called 'graph' function that is way faster and lacks ability to call many common functions with side-effect in python. Like print(). What you see there is a detector or a 'tracer'. It only runs once.
Then how do I debug?
Several ways for doing this. If you want to print-debug, use tf.print. In my experience this sometimes works, and sometimes doesn't. In case when it doesn't, and you still see the detector variable only, use the model.run_eagerly = True or pass it as an argument in model.compile. Even if you do not use tf.print and set run_eagerly, python's built-in print will still work (Try this).
Last but not least, you can wrap all your side-effect functions in a tf.py_function. This requires a bit more code and a sample copy-and-paste code can be seen here.
Also, Make sure to first define the function and then use it in the model.compile, especially if you are using Jupyter notebook. A buggy old declaration might still persist in memory and will probably ruin your day.
Did this help?
I haven't tried this yet but you should always use:
tf.print(value)
Instead of just normal
print(value)
Tensorflow has implemented this function specifically for this. Hopefully, this helps!
After going through some Stack questions and the Keras documentation, I manage to write some code trying to evaluate the gradient of the output of a neural network w.r.t its inputs, the purpose being a simple exercise of approximating a bivariate function (f(x,y) = x^2+y^2) using as loss the difference between analytical and automatic differentiation.
Combining answers from two questions (Keras custom loss function: Accessing current input pattern
and Getting gradient of model output w.r.t weights using Keras
), I came up with this:
import tensorflow as tf
from keras import backend as K
from keras.models import Model
from keras.layers import Dense, Activation, Input
def custom_loss(input_tensor):
outputTensor = model.output
listOfVariableTensors = model.input
gradients = K.gradients(outputTensor, listOfVariableTensors)
sess = tf.InteractiveSession()
sess.run(tf.initialize_all_variables())
evaluated_gradients = sess.run(gradients,feed_dict={model.input:input_tensor})
grad_pred = K.add(evaluated_gradients[0], evaluated_gradients[1])
grad_true = k.add(K.scalar_mul(2, model.input[0][0]), K.scalar_mul(2, model.input[0][1]))
return K.square(K.subtract(grad_pred, grad_true))
input_tensor = Input(shape=(2,))
hidden = Dense(10, activation='relu')(input_tensor)
out = Dense(1, activation='sigmoid')(hidden)
model = Model(input_tensor, out)
model.compile(loss=custom_loss_wrapper(input_tensor), optimizer='adam')
Which yields the error: TypeError: The value of a feed cannot be a tf.Tensor object. because of feed_dict={model.input:input_tensor}. I understand the error, I just don't know how to fix it.
From what I gathered, I can't simply pass input data into the loss function, it must be a tensor. I realized Keras would 'understand' it when I call input_tensor. This all just leads me to think I'm doing things the wrong way, trying to evaluate the gradient like that. Would really appreciate some enlightenment.
I don't really understand why you want this loss function, but I will provide an answer anyway. Also, there is no need to evaluate the gradient within the function (in fact, you would be "disconnecting" the computational graph). The loss function could be implemented as follows:
from keras import backend as K
from keras.models import Model
from keras.layers import Dense, Input
def custom_loss(input_tensor, output_tensor):
def loss(y_true, y_pred):
gradients = K.gradients(output_tensor, input_tensor)
grad_pred = K.sum(gradients, axis=-1)
grad_true = K.sum(2*input_tensor, axis=-1)
return K.square(grad_pred - grad_true)
return loss
input_tensor = Input(shape=(2,))
hidden = Dense(10, activation='relu')(input_tensor)
output_tensor = Dense(1, activation='sigmoid')(hidden)
model = Model(input_tensor, output_tensor)
model.compile(loss=custom_loss(input_tensor, output_tensor), optimizer='adam')
A Keras loss must have y_true and y_pred as inputs. You can try adding your input object as both x and y during the fit:
def custom_loss(y_true,y_pred):
...
return K.square(K.subtract(grad_true, grad_pred))
...
model.compile(loss=custom_loss, optimizer='adam')
model.fit(X, X, ...)
This way, y_true will be the batch being processed at each iteration from the input X, while y_pred will be the output of the model for that particular batch.
I'm trying to implement this: https://arxiv.org/abs/1706.03741 utility/preference learning approach in keras. The problem:
xTrain: 2 input sequences ,yTrain: binary value
The DNN only maps (single) input vectors to numeric values
To be more precise:
DNN: f(x)=y
t1 and t2 are sequences of vectors
Loss-function: cross_entropy(sigmoid(sum_t1(f(x_t1))-sum_t2(f(x_t2))),1)
The question: How do i implement that ?
My current approach is to use a TimeDistributed network with flattened output. The input is a concatenation of both sequences. (WIP code below) But it think this is a really bad way and there has to be something better ?
model = Sequential()
model.add(TimeDistributed(Masking(mask_value=0.), input_shape(2000,inputDims)))
model.add(TimeDistributed(Dense(64)))
model.add(LeakyReLU(alpha=0.01))
model.add(TimeDistributed(Dense(64)))
model.add(LeakyReLU(alpha=0.01))
model.add(TimeDistributed(Dense(1, activation='sigmoid')))
model.add(Flatten())
model.compile(loss=customLoss,
optimizer=Adam(),
metrics=['mse'])
def customLoss(y_true,y_pred):
y_pred = K.reshape(y_pred, shape=[2,1000])
y_pred = K.sum(y_pred, axis=1)
y_pred = K.sigmoid(y_pred[0]-y_pred[1])
y_pred = K.clip(y_pred, epsilon, 1.0 - epsilon)
loss = K.binary_crossentropy(y_pred, 1)
return loss