I am using Keras to train a network. Let's say that after 20 epochs I want to stop the training to check if everything is fine, then continue form the 21st epoch. Does calling the model.fit method for a second time reinitialize the already trained weights ?
Does calling the model.fit method for a second time reinitialize the already trained weights ?
No, it will use the preexisting weights your model had and perform updates on them. This means you can do consecutive calls to fit if you want to and manage it properly.
This is true also because in Keras you are also able to save a model (with the save and load_model methods), load it back, and call fit on it. For more info on that check this question.
Another option you got is to use the train_on_batch method instead:
train_on_batch(self, x, y, sample_weight=None, class_weight=None)
Runs a single gradient update on a single batch of data.
This way I think you may have more control in between the updates of you model, where you can check if everything is fine with the training, and then continue to the next gradient update.
Related
I have several seperate training datasets that when read as one crash the kernel in jupyter. So I did a workaround and read them separately in a sequence and call fit() on the same model-object.
To get accuracy metrics I am only grabbing the final history-object, but does this also represent all previous fit()-calls?
By default history is initialized as a callback every time anew when you call fit. This is unless you provide some alternative. One way to do so is to pass the model's history from one fit() call to the next fit() as a callback:
model.fit(x, y, batch_size, epochs, callbacks=[model.history])
this way the new values will be appended to the previously accumulated values, so you'd get statistics over multiple runs of fit().
If you need something more special - save and process history objects from each fit or write a custom callback with memory.
I have been using tf.reset_default_graph() to test out several graphs/models. However, now I would like to train the same model/graph, on different datasets. Therefore, I would like to reset the weights of the created model without having to delete and then create the whole model again. Please note that, I am going to train the model on 20 different datasets. Therefore, maybe resetting the weights only is a simpler operation than deleting the existing model and creating a new one for each dataset. Please correct me if I'm wrong.
Based on this GitHub issue, there is no single function to do it, but the offered workaround is to save your initial weights, then use it to re-initialize them alter (rather than randomly re-initializing them each time).
Initially,
Winit = weights.get_weights() # do this once before training
Later,
weights.set_weights(Winit) # call each time to reset weights
I'm using the Keras plot_model() function to visualize my machine learning models. Apart from having the issue that the first node of my output is always simply a very large number, there is another thing annoying me about this function: It does not provide a very elaborate output. For example, I would like to be able to see more information about the used loss function, the batch size, the number of epochs, the used optimizer, etc...
Is there any way I can retrieve this information from a model I previously saved to the disk and loaded again with the model_from_json() function?
How about TensorBoardCallback? It will create interactive graphs that you can explore based on your model if you use Tensorflow as your backend.
You just need add it as a callback to your fit function and make sure write_graph=True is set (which it is by default). If you want a shortcut you can directly invoke its methods instead of passing as a callback:
tensorboard = TensorboardCallback()
tensorboard.set_model(model) # your model here, will write graph etc
tensorboard.on_train_end() # will close the writer
Then just run tensorboard --logdir=./logs to start the server.
I saw a sample of code (too big to paste here) where the author used model.train_on_batch(in, out) instead of model.fit(in, out). The official documentation of Keras says:
Single gradient update over one batch of samples.
But I don't get it. Is it the same as fit(), but instead of doing many feed-forward and backprop steps, it does it once? Or am I wrong?
Yes, train_on_batch trains using a single batch only and once.
While fit trains many batches for many epochs. (Each batch causes an update in weights).
The idea of using train_on_batch is probably to do more things yourself between each batch.
It is used when we want to understand and do some custom changes after each batch training.
A more precide use case is with the GANs.
You have to update discriminator but during update the GAN network you have to keep the discriminator untrainable. so you first train the discriminator and then train the gan keeping discriminator untrainable.
see this for more understanding:
https://medium.com/datadriveninvestor/generative-adversarial-network-gan-using-keras-ce1c05cfdfd3
The method fit of the model train the model for one pass through the data you gave it, however because of the limitations in memory (especially GPU memory), we can't train on a big number of samples at once, so we need to divide this data into small piece called mini-batches (or just batchs). The methode fit of keras models will do this data dividing for you and pass through all the data you gave it.
However, sometimes we need more complicated training procedure we want for example to randomly select new samples to put in the batch buffer each epoch (e.g. GAN training and Siamese CNNs training ...), in this cases we don't use the fancy an simple fit method but instead we use the train_on_batch method. To use this methode we generate a batch of inputs and a batch of outputs(labels) in each iteration and pass it to this method and it will train the model on the whole samples in the batch at once and gives us the loss and other metrics calculated with respect to the batch samples.
Is it possible to continue training a Keras estimator with all the hyperparameters (including decreasing learning rate) and weights saved from previous epochs, as one does in scikit-learn with the warm_start parameter? Something like this:
estimator = KerasRegressor(build_fn=create_model, epochs=20, batch_size=40, warm_start=True)
Specifically, warm start should do this:
warm_start : bool, optional, default False When set to True, reuse the
solution of the previous call to fit as initialization, otherwise,
just erase the previous solution.
Is there anything like that in Keras?
Yes - it's possible. But rather cumbersome. You need to use train_on_batch function which keeps all model parameters (also optimizer ones).
This is cumbersome because you need to divide your dataset to batches on your own and you are also losing the possibility to apply Callbacks and to use automatic progbar. I hope that in new Keras version this option would be added to a fit method.