For some reason I get error message when trying to specify f1 score with Keras model:
model.compile(optimizer='adam', loss='mse', metrics=['accuracy', 'f1_score'])
I get this error:
ValueError: Unknown metric function:f1_score
After providing 'f1_score' function in the same file where I use 'model.compile' like this:
def f1_score(y_true, y_pred):
# Count positive samples.
c1 = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
c2 = K.sum(K.round(K.clip(y_pred, 0, 1)))
c3 = K.sum(K.round(K.clip(y_true, 0, 1)))
# If there are no true samples, fix the F1 score at 0.
if c3 == 0:
return 0
# How many selected items are relevant?
precision = c1 / c2
# How many relevant items are selected?
recall = c1 / c3
# Calculate f1_score
f1_score = 2 * (precision * recall) / (precision + recall)
return f1_score
model.compile(optimizer='adam', loss='mse', metrics=['accuracy', f1_score])
Model compiles all right and can be saved to a file:
model.save(model_path) # works ok
Yet loading it in another program, :
from keras import models
model = models.load_model(model_path)
fails with an error:
ValueError: Unknown metric function:f1_score
Specifying 'f1_score' in the same file this time does not help, Keras does not see it. What's wrong? How to use F1 Score with Keras model?
When you load the model, you have to supply that metric as part of the custom_objects bag.
Try it like this:
from keras import models
model = models.load_model(model_path, custom_objects= {'f1_score': f1_score})
Where f1_score is the function that you passed through compile.
For your implementation of f1_score to work I had to switch y_true and y_pred in the function declaration.
P.S.: for those who asked: K = keras.backend
change:
metrics=['accuracy', f1_score]
to:
metrics=[f1_score]
Related
I'm having trouble implementing a custom loss function into a Neural Network I'm building in TensorFlow. I want use one of my features as part of the loss function, so I've tried using model.add_loss instead of giving loss a value in the model.compile function.
My data looks like this:
import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
feature_df = np.array([600,9])
training, test, = feature_df[:350,:], feature_df[350:,:]
x_train = training[:,[0,1,2,3,4,5,6]]
y_train = training[:,8]
loss_inp_train = training[:,[6]]
x_test = test[:,[0,1,2,3,4,5,6]]
y_test = test[:,8]
loss_inp_test = test[:,[6]]
I want to use a custom loss function because its not necessarily the mse I'm interested in minimizing, I want to minimize the profitability of this model, which depends if y_true and y_pred fall above or below loss_inp_train
I've tried creating a loss function that looks like this
def custom_loss(y_pred, y_true,inp):
loss = 0
if (y_pred < inp):
if y_true < inp:
loss = loss + .9
else:
loss = loss - 1
else:
if y_true > inp:
loss = loss + .9
else:
loss = loss - 1
loss = loss*-1
return(loss)
And the Model
model = tf.keras.Sequential([
normalize,
layers.Dense(18),
layers.Dense(1)
])
model.add_loss(profit_loss(y_pred,y_train,loss_inp_train))
model.compile(loss = None,
optimizer = tf.optimizers.Adam())
I'm having trouble feeding the loss function the output of the model. I'm still new to TensorFlow, whenever I've accessed predicted values its after the training using model.predict, but obviously I don't have a fitted model yet. How do I reference both a feature of the training data and y_true, y_pred in a function?
Probably the best way to do this is to define a custom loss. Unfortunately I'm not sure how to handle nested if statements like you have. Probably with a combination of K.switch. I can try to give you a partial solutions taking in consideration only the presence of a single if statement. Let's take the following simplified code:
loss = 0
if (y_pred < inp):
loss = # assignment 1
else:
loss = # assignment 2
In this case the loss function could be converted into this:
def profit_loss(inp):
def loss_function(y_true, y_pred):
loss = 0
condition = K.greater(y_pred - inp, 0)
loss1 = # assignment 1 if y_pred < inp
loss2 = # assignment 2 if y_pred >= inp
loss = K.switch(condition, loss2, loss1)
return - K.sum(loss)
return loss_function
model.compile(optimizer = tf.optimizers.Adam(), loss=profit_loss(inp))
This way y_true and y_pred are automatically handled and you just have to feed the inp argument.
Hope this helps getting you closer to solving the problem.
I am trying to apply AdaBoost to a Keras model. The thing is that I have to use a custom loss function (unscaled deviance) which works well when I am using Keras into the RandomizedSearchCV of sklearn, but when I try to use AdaBoostRegressor i get :
ValueError: y should be a 1d array, got an array of shape (64501, 2) instead.
This error is coming from the fact that in my custom loss function, I use three arguments. However Keras only accepts two parameters (y_true, y_pred), so I bypassed this by passing a tuple with two values instead of y_true, like this :
#Loss function
def deviance(data, y_pred):
y_true = data[:, 0]
d = data[:, 1]
lnY = KB.log(y_true)
bool1 = KB.equal(y_true, 0)
zeros = KB.zeros_like(y_true)
lnY = KB.switch(bool1, zeros, lnY)
lnYp = KB.log(y_pred)
bool2 = KB.equal(y_pred, 0)
zeross = KB.zeros_like(y_pred)
lnYp = KB.switch(bool2, zeross, lnYp)
loss = 2 * d * (y_true * lnY - y_true * lnYp[:, 0] - y_true + y_pred[:, 0])
return loss
So the program takes the tuple (call it 'feed') and unpacks it in order to calculate the unscaled deviance. I then use it like this and it works :
grid = RandomizedSearchCV(pipeline, cv = cv, param_distributions=param_grid, verbose=2, n_iter = 40) #plus de folds pourraient augmenter la variance
grid.fit(data, feed)
But now, I want to use this keras model in an AdaboostRegressor by using :
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.pipeline import Pipeline
from sklearn.model_selection import KFold
def baseline_model2(dropout = 0.2, kernel_initializer = 'glorot_uniform', nn1 = 15, nn2 = 10, lr = 0.001, act1 = "relu"):
with tf.device('/gpu:0'):
# create model
#building model
model = keras.Sequential()
model.add(Dense(nn1, input_dim = 21, activation = act1, kernel_initializer=kernel_initializer))
model.add(Dropout(dropout))
#model.add(Dense(2, activation = "exponential"))
model.add(Dense(nn2, activation = act1))
model.add(Dense(1, activation = "exponential", kernel_initializer=kernel_initializer))
optimizer = keras.optimizers.adagrad(lr=lr)
model.compile(loss=deviance, optimizer=optimizer, metrics = [deviance, "mean_squared_error"])
return model
clf = KerasRegressor(build_fn=baseline_model2)
from sklearn.ensemble import AdaBoostRegressor
boostedNN = AdaBoostRegressor(base_estimator=clf)
boostedNN.fit(data, feed)
It is this part that gives me the ValueError. I know it comes from the fact that I am supplying a tuple to the algorithm, but I am searching for a way to bypass this, because Keras will need that tuple in order to evaluate the deviance correctly. I think editing a line or two in skl library could do the trick but I didn't manage to find the right place to do it by myself.
So my question is, if I have something like:
model = Model(inputs = input, outputs = [y1,y2])
model.compile(loss = my_loss ...)
I have only seen my_loss as a dictionary of independent losses and, then, the final loss is defined as the sum of those. But, can I define in a multitask model a loss function that take all the predicted/true values and then I can multiply them (for instance)?
This is the loss I am trying to define:
def my_loss(y_true1, y_true2, y_pred1, y_pred2):
final_loss = binary_crossentropy(y_true1, y_pred1) + y_true1 * categorical_crossentropy(y_true2, y_pred2)
return final_loss
Usually, your paramaters are y_true, y_pred in the loss function, where y_pred is either y1 or y2. But now I need both to compute the loss, so how can I define this loss function and pass all the parameters to the function: y_true1, y_true2, y_pred1, y_pred2.
My current model that I want to change its loss:
x = Input(shape=(n, ))
shared = Dense(32)(x)
sub1 = Dense(16)(shared)
sub2 = Dense(16)(shared)
y1 = Dense(1)(sub1, activation='sigmoid')
y2 = Dense(4)(sub2, activation='softmax')
model = Model(inputs = input, outputs = [y1,y2])
model.compile(loss = ['binary_crossentropy', 'categorical_crossentropy'] ...) #THIS LINE I WANT TO CHANGE IT
Thanks!
I'm not sure if I'm understanding correctly, but I'll try.
The loss function must contain both the predicted and the actual data -- it's a way to measure the error between what your model is predicting and the true data. However, the predicted and actual data do not need to be one-dimensional. You can make y_pred a tensor that contains both y_pred1 and y_pred2. Likewise, y_true can be a tensor that contains both y_true1 and y_true2.
As far as I know, loss functions should return a single number. That's why loss functions often have a mean or a sum to add up all of the losses for individual data points.
Here's an example of mean square error that will work for more than 1D:
import keras.backend as K
def my_loss(y_true, y_pred):
# this example is mean squared error
# works if if y_pred and y_true are greater than 1D
return K.mean(K.square(y_pred - y_true))
Here's another example of a loss function that I think is closer to your question (although I cannot comment on whether or not it's a good loss function):
def my_loss(y_true, y_pred):
# calculate mean(abs(y_pred1*y_pred2 - y_true1*ytrue2))
# this will work for 2D inputs of y_pred and y_true
return K.mean(K.abs(K.prod(y_pred, axis = 1) - K.prod(y_true, axis = 1)))
Update:
You can concatenate two outputs into a single tensor with keras.layers.Concatenate. That way you can still have a loss function with only two arguments.
In the model you wrote above, the y1 output shape is (None, 1) and the y2 output shape is (None, 4). Here's an example of how you could write your model so that the output is a single tensor that concatenates y1 and y1 into a shape of (None, 5):
from keras import Model
from keras.layers import Input, Dense
from keras.layers import Concatenate
input_layer = Input(shape=(n, ))
shared = Dense(32)(input_layer)
sub1 = Dense(16)(shared)
sub2 = Dense(16)(shared)
y1 = Dense(1, activation='sigmoid')(sub1)
y2 = Dense(4, activation='softmax')(sub2)
mergedOutput = Concatenate()([y1, y2])
Below, I show an example for how you could rewrite your loss function. I wasn't sure which of the 5 columns of the output to call y_true1 vs. y_true2, so I guessed that y_true1 was column 1 and y_true2 was the remaining 4 columns. The same column structure would apply to y_pred1 and y_pred2.
from keras import losses
def my_loss(y_true, y_pred):
final_loss = (losses.binary_crossentropy(y_true[:, 0], y_pred[:, 0]) +
y_true[:, 0] *
losses.categorical_crossentropy(y_true[:, 1:], y_pred[:,1:]))
return final_loss
Finally, you can compile the model without any major changes from normal:
model.compile(optimizer='adam', loss=my_loss)
I am currently experimenting with generative adversarial networks in Keras.
As proposed in this paper, I want to use the historical averaging loss function. Meaning that I want to penalize the change of the network weights.
I am not sure how to implement it in a clever way.
I was implementing the custom loss function according to the answer to this post.
def historical_averaging_wrapper(current_weights, prev_weights):
def historical_averaging(y_true, y_pred):
diff = 0
for i in range(len(current_weights)):
diff += abs(np.sum(current_weights[i]) + np.sum(prev_weights[i]))
return K.binary_crossentropy(y_true, y_pred) + diff
return historical_averaging
The weights of the network are penalized, and the weights are changing after each batch of data.
My first idea was to update the loss function after each batch.
Roughly like this:
prev_weights = model.get_weights()
for i in range(len(data)/batch_len):
current_weights = model.get_weights()
model.compile(loss=historical_averaging_wrapper(current_weights, prev_weights), optimizer='adam')
model.fit(training_data[i*batch_size:(i+1)*batch_size], training_labels[i*batch_size:(i+1)*batch_size], epochs=1, batch_size=batch_size)
prev_weights = current_weights
Is this reasonable? That approach seems to be a bit "messy" in my opinion.
Is there another possibility to do this in a "smarter" way?
Like maybe updating the loss function in a data generator and use fit_generator()?
Thanks in advance.
Loss functions are operations on the graph using tensors.
You can define additional tensors in the loss function to hold previous values. This is an example:
import tensorflow as tf
import tensorflow.keras.backend as K
keras = tf.keras
class HistoricalAvgLoss(object):
def __init__(self, model):
# create tensors (initialized to zero) to hold the previous value of the
# weights
self.prev_weights = []
for w in model.get_weights():
self.prev_weights.append(K.variable(np.zeros(w.shape)))
def loss(self, y_true, y_pred):
err = keras.losses.mean_squared_error(y_true, y_pred)
werr = [K.mean(K.abs(c - p)) for c, p in zip(model.get_weights(), self.prev_weights)]
self.prev_weights = K.in_train_phase(
[K.update(p, c) for c, p in zip(model.get_weights(), self.prev_weights)],
self.prev_weights
)
return K.in_train_phase(err + K.sum(werr), err)
The variable prev_weights holds the previous values. Note that we added a K.update operation after the weight errors are calculated.
A sample model for testing:
model = keras.models.Sequential([
keras.layers.Input(shape=(4,)),
keras.layers.Dense(8),
keras.layers.Dense(4),
keras.layers.Dense(1),
])
loss_obj = HistoricalAvgLoss(model)
model.compile('adam', loss_obj.loss)
model.summary()
Some test data and objective function:
import numpy as np
def test_fn(x):
return x[0]*x[1] + 2.0 * x[1]**2 + x[2]/x[3] + 3.0 * x[3]
X = np.random.rand(1000, 4)
y = np.apply_along_axis(test_fn, 1, X)
hist = model.fit(X, y, validation_split=0.25, epochs=10)
The model losses decrease over time, in my test.
I have a Lstm model for sequence prediction,which is shown here:
def create_model(max_sequence_len, total_words):
input_len = max_sequence_len - 1
model = keras.models.Sequential()
model.add(layers.Embedding(total_words, 50, input_length=input_len))
model.add(layers.LSTM(50, input_shape=predictors[:1].shape))
model.add(layers.Dropout(0.2))
model.add(layers.Dense(activation='softmax', units = total_words))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'], lr=0.01)
return model
model_sb = create_model(max_sequence_len, total_words)
history = model_sb.fit(X_train, y_train, epochs = 20 , shuffle = True, validation_split=0.3, )
and it works well but I want to take 2 output from my model who are the output with most probability in softmax dense layer.
for take them I can use this code:
predicted = model_sb.predict(test_sequence, verbose=1)
And then by this code find the first n high probability output:
y_sum = predicted.sum(axis=0)
ind = np.argpartition(y_sum, -n)[-n:]
ind[np.argsort(y_sum[ind])]
But I need to know the accuracy of my model if the output be one of these n output (with "or" condition)
Is there any package which help me?
I mean I don't want to evaluate my model with just one most probability output, I want to evaluate accuracy and loss by 2 high probability result.
This is called top-k accuracy, with k = 2 in your case. Keras already has an implementation of this accuracy:
from keras.metrics import top_k_categorical_accuracy
def my_acc(y_true, y_pred):
return top_k_categorical_accuracy(y_true, y_pred, k=2)
Then you pass this custom metric to your model:
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=[my_acc])