Is it possible to continue training a Keras estimator with all the hyperparameters (including decreasing learning rate) and weights saved from previous epochs, as one does in scikit-learn with the warm_start parameter? Something like this:
estimator = KerasRegressor(build_fn=create_model, epochs=20, batch_size=40, warm_start=True)
Specifically, warm start should do this:
warm_start : bool, optional, default False When set to True, reuse the
solution of the previous call to fit as initialization, otherwise,
just erase the previous solution.
Is there anything like that in Keras?
Yes - it's possible. But rather cumbersome. You need to use train_on_batch function which keeps all model parameters (also optimizer ones).
This is cumbersome because you need to divide your dataset to batches on your own and you are also losing the possibility to apply Callbacks and to use automatic progbar. I hope that in new Keras version this option would be added to a fit method.
Related
So I have this model written by the subclassing API, the call signature looks like call(x, training), where training argument is needed to differentiate between training and non-training when doing batchnorm and dropout. How do I make the model forward pass know I am in training mode or eval mode when I use model.fit?
Thanks!
Actually, in the documentation https://www.tensorflow.org/beta/guide/keras/custom_layers_and_models, it says "Some layers, in particular the BatchNormalization layer and the Dropout layer, have different behaviors during training and inference. For such layers, it is standard practice to expose a training (boolean) argument in the call method.
By exposing this argument in call, you enable the built-in training and evaluation loops (e.g. fit) to correctly use the layer in training and inference." So I think the training argument is passed in automatically by keras. I tried to remove the default value for training argument and no errors were thrown, so it is very likely keras built-in loop did the thing.
As far as i know, there is no argument for this. Model.fit simply trains the model on whatever training data provided, and at the end of each epoch evaluates the training on either provided validation data, OR by the use of validation_split.
I know with pytorch you can turn off training by calling eval() on your model.
Also you can set requires_grad=False.
How can you ensure that a TensorFlow element is not modified during training?
If you don't want to train certain Variables in TensorFlow you can achieve this behaviour by adding trainable=False to Variables.
I am currently using lightgbm library on python.
model = lgb.LGBMRegressor(objective='regression', metric='rmse', boosting_type='rf', max_depth=20, num_leaves=20,
learning_rate=0.1, feature_fraction=0.8, feature_fraction_seed=10, bagging_fraction=0.8,
bagging_freq=1, bagging_seed=10, verbosity=True,
lambda_l2=0.000001, lambda_l1=0.00001, max_bin=200)
model.fit(X_train, np.log1p(y_train), num_iteration=1000, eval_set=[(X_val, np.log1p(y_val) )], eval_metric="rmse", verbose=True, early_stopping_rounds=20)
The problem is that, the training stop after 100 iterations. I did not find on the documentation a step parameters, so I was wondering how can I increase the number of iteration? I do not want to use the lgb.train() function. In fact, I want also to use a Grid Search function from sklearn (and get the model then) so I need to use the fit function. Dos anyone know how to solve my problem ?
EDIT : It seems possible with the num_iterations parameters, but we need the last version of lightgbm. My former version did not work because of that !
n_estimators is the parameter of the model constructor in sklearn API, that controls the number of trees to be built. You can also set it via model.set_params().
You have set early_stopping_rounds=20, which means that training will stop if you get no improvement for the validation data in the last 20 iterations. This is used to avoid over-fitting, but may stop training early if set too low.
I am using Keras to train a network. Let's say that after 20 epochs I want to stop the training to check if everything is fine, then continue form the 21st epoch. Does calling the model.fit method for a second time reinitialize the already trained weights ?
Does calling the model.fit method for a second time reinitialize the already trained weights ?
No, it will use the preexisting weights your model had and perform updates on them. This means you can do consecutive calls to fit if you want to and manage it properly.
This is true also because in Keras you are also able to save a model (with the save and load_model methods), load it back, and call fit on it. For more info on that check this question.
Another option you got is to use the train_on_batch method instead:
train_on_batch(self, x, y, sample_weight=None, class_weight=None)
Runs a single gradient update on a single batch of data.
This way I think you may have more control in between the updates of you model, where you can check if everything is fine with the training, and then continue to the next gradient update.
I would like to train a GAN in Keras. My final target is BEGAN, but I'm starting with the simplest one. Understanding how to freeze weights properly is necessary here and that's what I'm struggling with.
During the generator training time the discriminator weights might not be updated. I would like to freeze and unfreeze discriminator alternately for training generator and discriminator alternately. The problem is that setting trainable parameter to false on discriminator model or even on its' weights doesn't stop model to train (and weights to update). On the other hand when I compile the model after setting trainable to False the weights become unfreezable. I can't compile the model after each iteration because that negates the idea of whole training.
Because of that problem it seems that many Keras implementations are bugged or they work because of some non-intuitive trick in old version or something.
I've tried this example code a couple months ago and it worked:
https://github.com/fchollet/keras/blob/master/examples/mnist_acgan.py
It's not the simplest form of GAN, but as far as I remembered, it's not too difficult to remove the classification loss and turn the model into a GAN.
You don't need to turn on/off the discriminator's trainable property and recompile. Simply create and compile two model objects, one with trainable=True (discriminator in the code) and another one with trainable=False (combined in the code).
When you're updating the discriminator, call discriminator.train_on_batch(). When you're updating the generator, call combined.train_on_batch().
Can you use tf.stop_gradient to conditionally freeze weights?
Maybe your adversarial net(generator plus discriminator) are wrote in 'Model'.
However, even you set the d.trainable=False, the independent d net are set non-trainable, but the d in the whole adversarial net is still trainable.
You can use the d_on_g.summary() before then after set d.trainable=False and you would know What I mean(pay attention to the trainable variables).