Develop Reference

Saving accuracy and loss with callback on colab

So im trying to train a model on colab, and it is going to take me roughly 70-72 hr of continues running. I have a free account, so i get kicked due to over-use or inactivity pretty frequently, which means I cant just dump history in a pickle file.
history = model.fit_generator(custom_generator(train_csv_list,batch_size), steps_per_epoch=len(train_csv_list[:13400])//(batch_size), epochs=1000, verbose=1, callbacks=[stop_training], validation_data=(x_valid,y_valid))
I found the CSVLogger in callback method and added it to my callback as below. But it wont create model_history_log.csv for some reason. I don't get any error or warning. What part am i doing wrong ?
My goal is to only save accuracy and loss, throughout the training process
class stop_(Callback):
def on_epoch_end(self, epoch, logs={}):
model.save(Path("/content/drive/MyDrive/.../model" +str(int(epoch))))
CSVLogger("/content/drive/MyDrive/.../model_history_log.csv", append=True)
if(logs.get('accuracy') > ACCURACY_THRESHOLD):
print("\nReached %2.2f%% accuracy, so stopping training!!" %(ACCURACY_THRESHOLD*100))
self.model.stop_training = True
stop_training = stop_()
Also since im saving the model at every epoch, does the model save this information ? so far i havent found anything, and i doubt it saves accuracy, loss, val accuracy,etc

Think you want to write your callback as follows
class STOP(tf.keras.callbacks.Callback):
def __init__ (self, model, csv_path, model_save_dir, epochs, acc_thld): # initialization of the callback
# model is your compiled model
# csv_path is path where csv file will be stored
# model_save_dir is path to directory where model files will be saved
# number of epochs you set in model.fit
self.model=model
self.csv_path=csv_path
self.model_save_dir=model_save_dir
self.epochs=epochs
self.acc_thld=acc_thld
self.acc_list=[] # create empty list to store accuracy
self.loss_list=[] # create empty list to store loss
self.epoch_list=[] # create empty list to store the epoch
def on_epoch_end(self, epoch, logs=None): # method runs on the end of each epoch
savestr='_' + str(epoch+1) + '.h5' # model will be save as an .h5 file with name _epoch.h5
save_path=os.path.join(self.model_save_dir, savestr)
acc= logs.get('accuracy') #get the accuracy for this epoch
loss=logs.get('loss') # get the loss for this epoch
self.model.save (save_path) # save the model
self.acc_list.append(logs.get('accuracy'))
self.loss_list.append(logs.get('loss'))
self.epoch_list.append(epoch + 1)
if acc > self.acc_thld or epoch+1 ==epochs: # see of acc >thld or if this was the last epoch
self.model.stop_training = True # stop training
Eseries=pd.Series(self.epoch_list, name='Epoch')
Accseries =pd.Series(self.acc_list, name='accuracy')
Lseries=pd.Series(self.loss_list, name='loss')
df=pd.concat([Eseries, Lseries, Accseries], axis=1) # create a dataframe with columns epoch loss accuracy
df.to_csv(self.csv_path, index=False) # convert dataframe to a csv file and save it
if acc > self.acc_thld:
print ('\nTraining halted on epoch ', epoch + 1, ' when accuracy exceeded the threshhold')
then before you run model.fit use code
epochs=20 # set number of epoch for model.fit and the callback
sdir=r'C:\Temp\stooges' # set directory where save model files and the csv file will be stored
acc_thld=.98 # set accuracy threshold
csv_path=os.path.join(sdir, 'traindata.csv') # name your csv file to be saved in sdir
callbacks=STOP(model, csv_path, sdir, epochs, acc_thld) # instantiate the callback
Remember in model.fit set callbacks = callbacks. I tested this on a simple dataset. It ran for only 3 epochs before the accuracy exceeded the threshold of .98. So since it ran for 3 epoch it created 3 save model files in the sdir labeled as
_1.h5
_2.h5
_3.h5
It also created the csv file labelled as traindata.csv. The csv file content was
Epoch loss accuracy
1 8.086007 .817778
2 6.911876 .974444
3 6.129871 .987778

Progress bar not shown during training, Python

I'm trying to train a 2D Unet, for the segmentation task.
I execute this line of code:
model.fit(training_generator, epochs = params["nEpoches"],
validation_data=validation_generator, verbose = 1, use_multiprocessing = True, workers = 6, callbacks=[callbacks_list,csv_logger])
Where
training_generator = Istance of DataGenerator(x_training, y_train_flat, **params), with the image and the masks array as parameters of this class.
epochs = 2
validation_generator = Istance of DataGenerator(x_validation, y_validation_flat, **params), with validation data.
callbacks_list = checkPoint = ModelCheckpoint(filepath, monitor='val_loss', verbose=1, save_best_only=False, mode='min', period=1)
callbacks_list = checkPoint
With the verbose=1 parameter I think I should see a progress bar showing the training status for each epoch, but the only thing I see is Epoch 1/2, without any bar. So I can't say if the training process is going on or if it's stucked somewhere.

According to Tensorflow documentation,
steps_per_epoch:-
Integer or None. Total number of steps (batches of samples) before
declaring one epoch finished and starting the next epoch. When
training with input tensors such as TensorFlow data tensors, the
default None is equal to the number of samples in your dataset divided
by the batch size, or 1 if that cannot be determined. If x is a
tf.data dataset, and 'steps_per_epoch' is None, the epoch will run
until the input dataset is exhausted. When passing an infinitely
repeating dataset, you must specify the steps_per_epoch argument.
validation_steps:-
Only relevant if validation_data is provided and is a tf.data dataset.
Total number of steps (batches of samples) to draw before stopping
when performing validation at the end of every epoch. If
'validation_steps' is None, validation will run until the
validation_data dataset is exhausted. In the case of an infinitely
repeated dataset, it will run into an infinite loop. If
'validation_steps' is specified and only part of the dataset will be
consumed, the evaluation will start from the beginning of the dataset
at each epoch. This ensures that the same validation samples are used
every time.
In your case, training progress is going on, as rightly mentioned by #Kaveh, it does not know how much steps it should have for one epoch and ran into an infinite loop. Check your batch size and add steps_per_epoch and validation_steps to the model.fit() as shown below will resolve your issue.
model.fit(training_generator,
steps_per_epoch = len(training_generator) // training_generator.batch_size,
epochs = params["nEpoches"],
validation_data=validation_generator,
validation_steps=len(validation_generator) // validation_generator.batch_size,
verbose = 1,
use_multiprocessing = True, workers = 6, callbacks=[callbacks_list,csv_logger])
For more information you can refer here

Has Tensorflow training output style changed?

I have been working on my deep learning model for a while. Today, when I started the model training, I noticed only a fraction of my dataset is being trained and the size of data used in each epoch changes with the batch size.
print(mixture_train_shaped.shape)
print(clear_train_shaped.shape)
model.fit(mixture_train_shaped, clear_train_shaped,
validation_split=0.2,
epochs=40,
batch_size=32,
shuffle=True,
verbose=1
)
When I run this code, this is what I see.
(51226, 129, 8, 4)
(51226, 129, 1, 1)
Epoch 1/40
1281/1281 [===========]
Epoch 2/40
1281/1281 [===========]
In my previous training outputs, the model would use the entire set in one epoch. On the above example though, the training set has 40,980 sample and each epoch trains only 40,980/32=1281. In a way, every epoch trains a single batch.
Train on 47 samples, validate on 6 samples
Epoch 1/5000
47/47 [==========]
I haven't changed the code. Is every epoch still using the entire training set or has it changed?

In the previous versions of Colab, the training set size was shown. With this update, the batch numbers are shown in the progress bar but no change in how many items have been trained for the model.

Keras fit_generator(datagen.flow(X_train)) function takes less pictures than X_train have?

I am using keras fit_generator(datagen.flow()) function for training of my inception model, I am so confused about the number of images it is taking on every epoch. Can anyone please help me telling this How it is working. My code is below.
I am using this keras documentation.
from keras.preprocessing.image import ImageDataGenerator
datagen = ImageDataGenerator(rotation_range = 15, horizontal_flip = True)
# Fitting the model with
history = inc_model.fit_generator(datagen.flow(X_train, train_labels, batch_size=10), epochs=20, validation_data = (X_test, test_labels), callbacks=None)
Now my total images in X_train is 4676. However, everytime I run this history line, I get
Epoch 1/20
936/936 [========================] - 167s 179ms/step - loss: 1.4236 - acc: 0.3853 - val_loss: 1.0858 - val_acc: 0.5641
Why is it not taking whole of my X_train images?
Also, if I change batch_size from 10 to lets say 15 it start taking more less images such as
Epoch 1/20
436/436
Thank you.

The 936 and 436 actually refer to batches of samples per epoch. You set your batch size to 10 and 15, so in each case the model is trained on 936 X 10 and 436 X 15 samples per epoch. The samples is even more than your original training set, since you use the ImageDataGenerator which creates additional training instances by applying transformations to existing ones.

What does nb_epoch in neural network stands for?

i'm currently beginning to discover Keras library for deap learning, it seems that in the training phase a centain number of epoch is chosen, but i don't know on which assumption is this choice based on.
In the Mnist dataset the number of epochs chosen is 4 :
model.fit(X_train, Y_train,
batch_size=128, nb_epoch=4,
show_accuracy=True, verbose=1,
validation_data=(X_test, Y_test))
Could someone tell me why and how do we choose a correct number of epochs ?

Starting Keras 2.0, nb_epoch argument has been renamed to epochs everywhere.
Neural networks are trained iteratively, making multiple passes over entire dataset. Each pass over entire dataset is referred to as epoch.
There are two possible ways to choose an optimum number of epochs:
1) Set epochs to a large number, and stop training when validation accuracy or loss stop improving: so-called early stopping
from keras.callbacks import EarlyStopping
early_stopping = EarlyStopping(monitor='val_loss', patience=4, mode='auto')
model.fit(X_train, Y_train,
batch_size=128, epochs=500,
show_accuracy=True, verbose=1,
validation_data=(X_test, Y_test),callbacks = [early_stopping])
2) Consider number of epochs as a hyperparameter and select the best value based on a set of trials (runs) on a grid of epochs values

it seems you might be using old version of keras ,nb_epoch refers to number of epochs which has been replaced by epoch
if you look here you will see that it has been deprecated.
One epoch means that you have trained all dataset(all records) once,if you have 384 records,one epoch means that you have trained your model for all on all 384 records.
Batch size means the data you model uses on single iteration,in that case,128 batch size means that at once,your model takes 128 and do some a single forward pass and backward pass(backpropation)[This is called one iteration]
.it
To break it down with this example,one iteration,your model takes 128 records[1st batch] from your whole 384 to be trained and do a forward pass and backward pass(back propagation).
on second batch,it takes from 129 to 256 records and do another iteration.
then 3rd batch,from 256 to 384 and performs the 3rd iteration.
In this case,we say that it has completed one epoch.
the number of epoch tells the model the number it has to repeat all those processes above then stops.
There is no correct way to choose a number of epoch,its something that is done by experimenting,usually when the model stops to learn(loss is not going down anymore) you usually decrease the learning rate,if it doesn't go down after that and the results seems to be more or less as you expected then you select at that epoch where the model stopped to learn
I hope it helps

In neural networks, an epoch is equivalent to training the network using each data once.
The number of epochs, nb_epoch, is hence how many times you re-use your data during training.