could you help me to understand how to implement a callback function which determines the performance of a model on the test and validation data?
I got a bit confused reading this from a nice block entry:
len(self.model.validation_data) == 3,
because validation_data[0] ==> train_x (that you input in
model.fit()),
validation_data[1] ==> train_y,
validation_data[2]=sample_weight.
As you can see, the blogger mentions that validation_data is here the training data. From the keyword "validation_data" which is also used in model.fit I would assume that self.model.validation_data returns the validation data and something like self.model.x would return the training data (as model.fit(x= ...,) is the input for the training data)
Could someone shed light on this?
Thanks in advance for your help
edit:
I checked with dir(self) if there would be something like an x for the training data. But indeed, there is only validation_data. Could someone explain to me how I can differentiate between test and validation?
Is validation_data always the training data used in
def on_train_end(self, logs={}):
and validation_data becomes the actual validation set when using ?
def on_epoch_end(self, epoch, logs={}):
edit edit:
in the same block the author talks about self.model.training_data
he found but could not retrieve. I searched for this but apparently, it was removed.
So my question may be more on point: How can you load the used training data at the end of an epoch in a callback.
The answer is quite simple.
Before you do model.fit() you can stick anything to the model.
I just sticked X_train and Y_train to it. You can here stick the validation or test data to it, whatever you want.
model.X_train=X_train
model.Y_train=Y_train
Then, you use those values with the sklearn library to calculate any performance (accuracy, f1, kappa, ... )
First, add variables to self to enable later appending each epoch result to the variable
class yourowncallbackname(Callback):
def_on_train_begin(self,logs={}):
self.val_f1=[] #define the variable to collect results
self.val_kappa=[]
then define what happens on the end of each epoch in the same class "yourowncallbackname" (on epoch end works best). Here you use the data sticked to the model (e.g.
model.x_train) to get the results from model.predict.
def on_epoch_end(self, epoch, logs={}):
#LOAD DATA
train_predict= (np.asarray(self.model.predict(self.model.X_train)))
train_true=self.model.Y_train_jan
val_predict = np.asarray(self.model.predict(self.validation_data[0]))
val_true = self.validation_data[1]
Maybe, depending on what you want to predict you have to round the predictions with .round
val_predict = (np.asarray(self.model.predict(self.model.X_train))).round()
Then calculate any performance you like on the predicted data (same as defined in on_train_begin) for e.g. test and validation
_train_f1= f1_score(train_true, train_predict, labels=...)
_train_k= cohen_kappa_score(train_true.ravel(),train_predict,labels=...)
_val_f1= f1_score(val_true, val_predict, labels=...)
_val_k= cohen_kappa_score(val_true.ravel(), val_predict, labels=...)
To collect them, stick them now to self. This will stick to the callback itself which you can later call again.
self.train_f1.append(_train_f1)
self.train_kappa.append(_train_k)
self.val_f1.append(_val_f1)
self.val_kappa.append(_val_k)
This callback you have to initialize now wherever you want to fit your model
callbackmetric=yourowncallbackname()
Then you can put it into the fit:
history=model.fit(...
callbacks=[callbackmetric])
If you now want to use the appended results per epoch you can call them as you would call a class
Yourcallbackresults.val_f1=callbackmetric.val_f1
Yourcallbackresults.train_f1=callbackmetric.train_f1
For completion, I will post the total callback class again. You just have to addapt the correct perfroamnce calculation with al its parameters etc.:
class yourowncallbackname(Callback):
def on_train_begin(self, logs={}):
self.val_f1 = []
self.val_k = []
self.train_f1 = []
self.train_k = []
def on_epoch_end(self, epoch, logs={}):
#LOAD DATA
train_predict= (np.asarray(self.model.predict(self.model.X_train)))
train_true=self.model.Y_train_jan
val_predict = np.asarray(self.model.predict(self.validation_data[0]))
val_true = self.validation_data[1]
#CALC. PERFORMANCE
_train_f1= f1_score(...)
_train_k= cohen_kappa_score(...)
_val_f1= f1_score(...)
_val_k= cohen_kappa_score(...)
self.train_f1.append(_train_f1)
self.train_k.append(_train_k)
self.val_f1.append(_val_f1)
self.val_k.append(_val_k)
print (" val_f1: %f val_k: %f" %(_val_f1, _val_k))
return
Related
I've been joining this hackathon and playing with keras callbacks and neural network, may I know if there is a way to monitor not only loss or val_loss but BOTH of them to avoid overfitting either the test or train set?
e.g: can i put a function for the monitor field instead of just one field name?
If I want to monitor val_loss to pick the lowest but I also want a second criteria to pick the minimum difference between val_loss and loss.
I have an answer to a problem that is pretty similar to this, here.
Basically, it is not possible to monitor multiple metrics with keras callbacks. However, you could define a custom callback (see the documentation for more info) that can access the logs at each epoch and do some operations.
Let's say if you want to monitor loss and val_loss you can do something like this:
import tensorflow as tf
from tensorflow import keras
class CombineCallback(tf.keras.callbacks.Callback):
def __init__(self, **kargs):
super(CombineCallback, self).__init__(**kargs)
def on_epoch_end(self, epoch, logs={}):
logs['combine_metric'] = logs['val_loss'] + logs['loss']
Side note: the most important thing in my opinion is to monitor the validation loss. Train loss of course will keep dropping, so it is not really that meaningful to observe. If you really want to monitor them both I suggest you add a multiplicative factor and give more weight to validation loss. In this case:
class CombineCallback(tf.keras.callbacks.Callback):
def __init__(self, **kargs):
super(CombineCallback, self).__init__(**kargs)
def on_epoch_end(self, epoch, logs={}):
factor = 0.8
logs['combine_metric'] = factor * logs['val_loss'] + (1-factor) * logs['loss']
Then, if you only want to monitor this new metric during the training, you can use it like this:
model.fit(
...
callbacks=[CombineCallback()],
)
Instead, if you also want to stop the training using the new metric, you should combine the new callback with the early stopping callback:
combined_cb = CombineCallback()
early_stopping_cb = keras.callbacks.EarlyStopping(monitor="combine_metric")
model.fit(
...
callbacks=[combined_cb, early_stopping_cb],
)
Be sure to get the CombinedCallback before the early stopping callback in the callbacks list.
Moreover, you can draw more inspiration here.
You can choose between two approaches:
Create a custom metric to record the metric you want, by subclassing tf.keras.metrics.Metric. See https://www.tensorflow.org/api_docs/python/tf/keras/metrics/Metric for an example.
You can then use your metric in standard callbacks e.g. EarlyStopping()
Create a custom callback to do the calculation (and take the action) you want, by subclassing tf.keras.callbacks.CallBack. See https://www.tensorflow.org/guide/keras/custom_callback for how to do this.
Below is a Keras custom callback that should do the job. The callback monitors both the training loss and the validation loss. The form of the callback is
callbacks=[SOMT(model, train_thold, valid_thold)] where:
model is the name of your complied model
train_thold is a float. It is the value of accuracy (in Percent) that must be
achieved by the model in order to conditionally stop training
valid_threshold is a float. It is the value of validation accuracy (in Percent)
that must be achieved by the model in order to conditionally stop training
Note to stop training BOTH the train_thold and valid_thold must be exceeded in the SAME epoch.
If you want to stop training based soley on the training accuracy set the valid_thold to 0.0.
Similarly if you want to stop training on just validation accuracy set train_thold= 0.0.
Note if both thresholds are not achieved in the same epoch training will continue until the value of epochs specified in model.fit is reached.
For example lets take the case that you want to stop training when the
training accuracy has reached or exceeded 95 % and the validation accuracy has achieved at least 85%
then the code would be callbacks=[SOMT(my_model, .95, .85)]
class SOMT(keras.callbacks.Callback):
def __init__(self, model, train_thold, valid_thold):
super(SOMT, self).__init__()
self.model=model
self.train_thold=train_thold
self.valid_thold=valid_thold
def on_train_begin(self, logs=None):
print('Starting Training - training will halt if training accuracy achieves or exceeds ', self.train_thold)
print ('and validation accuracy meets or exceeds ', self.valid_thold)
msg='{0:^8s}{1:^12s}{2:^12s}{3:^12s}{4:^12s}{5:^12s}'.format('Epoch', 'Train Acc', 'Train Loss','Valid Acc','Valid_Loss','Duration')
print (msg)
def on_train_batch_end(self, batch, logs=None):
acc=logs.get('accuracy')* 100 # get training accuracy
loss=logs.get('loss')
msg='{0:1s}processed batch {1:4s} training accuracy= {2:8.3f} loss: {3:8.5f}'.format(' ', str(batch), acc, loss)
print(msg, '\r', end='') # prints over on the same line to show running batch count
def on_epoch_begin(self,epoch, logs=None):
self.now= time.time()
def on_epoch_end(self,epoch, logs=None):
later=time.time()
duration=later-self.now
tacc=logs.get('accuracy')
vacc=logs.get('val_accuracy')
tr_loss=logs.get('loss')
v_loss=logs.get('val_loss')
ep=epoch+1
print(f'{ep:^8.0f} {tacc:^12.2f}{tr_loss:^12.4f}{vacc:^12.2f}{v_loss:^12.4f}{duration:^12.2f}')
if tacc>= self.train_thold and vacc>= self.valid_thold:
print( f'\ntraining accuracy and validation accuracy reached the thresholds on epoch {epoch + 1}' )
self.model.stop_training = True # stop training
In PyTorch, I want to evaluate my model on the validation set every eval_step during training, and I wrote code like this:
def tune(model, loader_train, loader_dev, optimizer, epochs, eval_step):
for epoch in range(epochs):
for step,x in enumerate(loader_train):
optimizer.zero_grad()
loss = model(x)
loss.backward()
optimizer.step()
if step % eval_step == 0:
model.eval()
test(model, loader_dev)
model.train()
When eval_step = int(len(loader_train)/2) and eval_step = int(len(loader_train)/8), they lead to quite different metric result after training through one whole epoch (which means the second output for the former differs the eighth output for the latter).
Could anyone explain why?
The length of loader_train is 20000 (it depends on batch size), and here is my testing script:
def test(model, loader_dev):
preds = []
labels = []
for step,x in enumerate(loader_dev):
preds.append(model(x).view(-1))
labels.apend(x['label'].view(-1))
metric = cal_metric(preds, labels)
logger.info(metric)
I think you probably set 'shffule=True' in your dataloader. Even though you fix 'random seed', dataloader in torch will generate different results if you use another dataloader while using current dataloader. In the scenario you describe, it may cause your model get data input in different order and then result in different metric result.
I would like to implement a model checkpoint callback based on balanced accuracy score. For this, I implemented following class:
class BalAccScore(keras.callbacks.Callback):
def __init__(self, validation_data=None):
super(BalAccScore, self).__init__()
self.validation_data = validation_data
def on_train_begin(self, logs={}):
self.balanced_accuracy = []
def on_epoch_end(self, epoch, logs={}):
y_predict = tf.argmax(self.model.predict(self.validation_data[0]), axis=1)
y_true = tf.argmax(self.validation_data[1], axis=1)
balacc = balanced_accuracy_score(y_true, y_predict)
self.balanced_accuracy.append(round(balacc,6))
logs["val_bal_acc"] = balacc
keys = list(logs.keys())
print("\n ------ validation balanced accuracy score: %f ------\n" %balacc)
I then define following callbacks
balAccScore = BalAccScore(validation_data=(X_2, y_2))
mc = ModelCheckpoint(filepath=callback_path, monitor="val_bal_acc", verbose=1, save_best_only=True, save_freq='epoch')
model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=['val_bal_acc'])
history = model.fit(X_1, y_1, epochs = 5, batch_size = 512,
callbacks=[balAccScore, mc],
validation_data = (X_2, y_2)
)
I then get the error
ValueError: Unknown metric function: val_bal_acc
despite the fact that I find it under history when using for example accuracy instead, i.e. by setting metrics=["acc"] when compiling instead. In which case, I get the to be expected warning:
WARNING:tensorflow:Can save best model only with val_bal_acc available, skipping.
but otherwise the model runs perfectly. Not sure why it is not running otherwise.
you should just remove the quotations in compile :
model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=[val_bal_acc])
or at least this how it works in R
You're getting that error because you're not passing the balanced_accuracy_score as a value for the metrics argument when compiling the model. The string 'val_bal_acc' that you passed in the metrics argument when compiling the model doesn't work because it's not a known metric. You can access metrics by their string name, only for those metrics already implemented in tf.keras.metrics. If you want to monitor the validation balanced accuracy during training you should implement a custom metric class (you can look here how to do it) and then pass it to the metrics argument. Once you've done this you can monitor your custom metric using the name you gave to it with the prefix 'val', if you want to monitor it during the validation time. There's no need for a supplementary custom callback as you did, the logs are updated automatically once you've defined the metric. For this particular case, you can find some implementation of this metric in the answers to this question.
If you prefer a callback approach instead you don't need to define a custom metric but take advantage of the already logged metrics. You can find an implementation of that in my answer here.
I have a generator function which generates tuples of (inputs, targets) on which my model is trained using the fit_generator() method in Keras.
My dataset is divided into 9 equal parts. I wish to perform a leave-one-out cross validation on the dataset using the fit_generator() method and keep the learned parameters of the previous training intact.
My question is that will calling fit_generator() multiple times on the model make it re-learn its learned parameters on the previous train and validation sets from scratch or will it keep those learned parameters intact leading to improvement of accuracy?
After a little digging I found that the fit() method in Keras retains the learned parameters as over here Calling "fit" multiple times in Keras but I'm not sure if the same happens for fit_generator() and if it does can it be used for cross-validation of data.
The pseudo-code I'm thinking of implementing to achieve the cross-validation is as follows:
class DatasetGenerator(Sequence):
def __init__(validation_id, mode):
#Some code
def __getitem__():
#The generator function
#Some code
return (inputs, targets)
for id in range(9):
train_set = DatasetGenerator(id, 'train')
#train_set contains all 8 parts leaving the id part out for validation.
validation_set = DatasetGenerator(id, 'val')
#val_set contains the id part.
history = model.fit_generator(train_set, epochs = 10, steps_per_epoch = 24000, validation_data = val_set, validation_steps = 3000)
print('History Dict:', history.history)
results = model.evaluate_generator(test_set, steps=steps)
print('Test loss, acc:', results)
Will the model keep the learned parameters intact and improve upon them for each iteration of the for loop?
fit and fit_generator behave the same in that regard, calling them again will resume training from the previously trained weights.
Also note that what you are trying to do is not cross-validation, as to do real cross-validation, you train one model for each fold, and the models are completely independent, not continued from training of the previous fold.
As far as I know it will keep the previous trained params. Also, I think what you are trying to do can be done by modifying the on_epoch_end() method of Sequence. Could be something like this:
class DatasetGenerator(Sequence):
def __init__(self, id, mode):
self.id = id
self.mode = mode
self.current_epoch=0
#some code
def __getitem__(self, idx):
id = self.id
#Some code
return (inputs, targets)
def on_epoch_end():
self.current_epoch += 1
if self.current_epoch % 10 == 0:
self.id += 1
I would like to estimate different sets of metrics on train and validation set in keras. Namely, I want to be able to estimate AUC on validation set but not on the train set. I see that there is a way to predict labels within my own callback, but as to my understanding there is already one evaluation per validation cycle for the loss and metrics provided in model.compile(..., metrics=[]), and my validation set is not that small, I want to avoid running score prediction twice.
Is there way to do one of following:
request keras to evaluate a specific metric on val set only
trick the keras by returning None before evaluation on the train
set using any phase flag
retrieve pre-evaluated probability scores on the validation set
from within a callback function
or any other tricks?
Try:
class CustomLossHistory(keras.callbacks.Callback):
def __init__(self, loss, validation_set):
self.loss = loss
self.validation_set = validation_set # validation_set = x, y
def on_train_begin(self, logs={}):
self.losses = []
def on_batch_end(self, batch, logs={}):
current_loss_val = loss(self.validation[1],
self.model.predict(validation[0])
self.losses.append(current_loss_value)
# You could also print it out here.
auc_callback = CustomLossHistory(sklearn.metrics.roc_auc_score, val_set)
model.fit(...., callbacks=[auc_callback])
aucs = auc_callback.losses