Please add some context to explain the code sections (or check that you have not incorrectly formatted all of your question as code)
include_top=False,
weights='imagenet')
x = base_model.output
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = se_model.input,outputs=prediction_layer)
return model
When you download the base model, it downloads its architecture along with its weights. It's not possible to train these high-level models from single PC's. That's why we download the pre-trained models from the internet and then fine tune them by re-training them with our own dataset.
# Fine-tune from this layer onwards
for layer in model.layers[:fine_tune_at]:
layer.trainable=False
for layer in model.layers[fine_tune_at:]:
layer.trainable=True
As you can see above, you are freezing weights of pre-trained models' initial layers as you don't want to mess up with that and only need to change the last output layer based on your I/O.
Related
I'm currently developing a model using Keras + Tensorflow in order to determine the temperature range of a set of proteins. What I first did was create a pre-trained model that converts the proteins into embeddings and then predicts its respective temperature.
What I want to do now is incorporate this pre=trained model to a new model which can use this given model and respective weights as input. Then fit on a new dataset and predict once again. The following code for the new top model is:
UPDATED CODE
'Load Pretrained Model'
loaded_model = keras.models.load_model('pretrained_model')
#Freeze all model layer weights
loaded_model.trainable = False
input1 = np.expand_dims(x_train['input1'],1)
input2 = np.expand_dims(x_train['input2'], 1)
input3 = x_train['input3']
#Redefine Input Layers for ANN
input1 = Input(shape = (input1.shape[1],), name = "input1")
input2 = Input(shape = (input2.shape[1],), name = "input2")
input3 = Input(shape = (input3.shape[1],), name = "input2")
base_inputs = [input1, input2, input3]
x = loaded_model(base_inputs, training = False)
x = Dense(64, activation = "relu", kernel_regularizer=regularizers.l2(0.01))(x)
output = Dense(1, activation = "sigmoid")(x)
top_model = Model(inputs = base_inputs, outputs = output)
# Compile the Model
top_model.compile(loss='mse', optimizer = Adam(lr = 0.0001), metrics = ['mse'])
This is not working correctly and I'm not sure on how to get this up and running. I'm struggling a bit to get this and come across this error quite often:
AttributeError: 'Dense' object has no attribute 'shape'
Any thoughts?
Could you please try to use the initialize the inputs using the keras layers and try?
You have initialized the input shapes using Numpy.
But, If I am right unfortunately the dense layer which you have imported from keras does not support this ('Dense' object has no attribute 'op').
Kindly note that, 'Input' is a keras layer
Could you try to as specified in the following link to initialize the keras inputs (https://keras.io/guides/functional_api/)?
As an Example,
input1 = keras.Input(shape=(1,))
input2 = keras.Input(shape=(1,))
input3 = keras.Input(shape=(1,))
It totally depends on your machine learning architecture whether to make layers trainable or not. In Case of transfer learning, You can just use the trained weights from a pre-trained model and train your new network using the trained weights acquired from that model. In this case, you have to freeze the layers of the pre-trained model. Hence trainable = False. You use these weights in the mathematical calculation of the hidden layers you will use in your custom architecture.
But from your code snippet, I could predict that you are not using any hidden layers like LSTM, RNN or any other cells for your sequential data. Also, you are trying to provide the initialized Numpy inputs to a pre-trained model. I don't think whether it is a right way to do so. From the code snippet, you are making the layer not trainable, but you are then trying to train the model.
Also, If I am right, I think that you have to train with new set of data using the Pre-trained model right? If so, then kindly look at the following link(https://keras.io/guides/transfer_learning/).
Considering your problem, I could suggest that transfer learning approach would be a feasible solution. In transfer learning, you could use the trained model from one domain of a set of data to train similar kind of problem using other set of data. To clearly understand the how to make the layers trainable and freeze it, and fine-tuning can be understood in the following link(https://keras.io/guides/transfer_learning/)
Concerning the Attribute error, it is recommended to have a look at the following link (https://keras.io/guides/functional_api/)
At first, you to initialize the input node for keras along with the shape of the inputs with respect to the data you will feed to the train the model. An Example is shown below as follows,
inputs = keras.Input(shape=(784,))
or it can be something like as follows, if you are providing the image data
img_inputs = keras.Input(shape=(32, 32, 3))
The dense layer expects the input should be in a specific shape, which you can find according to your data. If you are not sure about it, please analyse the data at first. It will give you lot of information to proceed further.
I followed a blog on how to implement a vgg16-model from scratch and want to do the same with the pretrained model from Keras. I looked up some other blogs but can't find a fitting solution I think. My task is to classify integrated circuit images into defect or non defects.
I have seen on a paper that they used pretrained imagenet model of vgg16 for fabric defect detection, where they freezed the first seven layers and fine tuned the last nine for their own problem.
(Source: https://journals.sagepub.com/doi/full/10.1177/1558925019897396)
I have already seen examples on how to freeze all layers except the fully connected layers, but how can I try the example with freezing first x layers and fine tune the others for my problem?
The VGG16 is fairly easy to implement from scratch but for models like resnet or xception it is getting a little trickier.
It is not necessary to implement a model from scratch to freeze a few layers. You can do this on pre-trained models as well. In keras, you'd use trainable = False.
For example, let's say you want to use the pre-trained Xception model from keras and want to freeze the first x layers:
#In your includes
from keras.applications import Xception
#Since you're using the model for a different task, you'd want to remove the top
base_model = Xception(weights='imagenet', include_top=False)
#Freeze layers 0 to x
for layer in base_model.layers[0:x]:
layer.trainable = False
#To see all the layers in detail and to check trainable parameters
base_model.summary()
Ideally you'd want to add another layer on top of this model with the output as your classes. For more details, you can check this keras guide: https://keras.io/guides/transfer_learning/
A lot of times the pre-trained weights can be very useful in other classification tasks but in case you want to train a model from scratch on your dataset, you can load the model without the imagenet weights. Or better, load the weights but don't freeze any layers. This will retrain every layer taking imagenet weights as an initialization.
I hope I've answered your question.
I want to add new layers to a pre-trained model, using Tensorflow and Keras. The problem is, those new layers are not to be added on top of the model, but at the start. I want to create a triple-siamese model, which takes 3 different inputs and gives 3 different outputs, using the pre-trained network as the core of the model. For that, I need to insert 3 new input layers at the beginning of the model.
The default path would be to just chain the layers and the model, but this method treats the pre-trained model as a new layer (when a new model with the new inputs and the pre-trained model is created, the new model only contains4 layers, the 3 input layers, and the whole pre-trained model):
input_1 = tf.keras.layers.Input(shape = (224,224,3))
input_2 = tf.keras.layers.Input(shape = (224,224,3))
input_3 = tf.keras.layers.Input(shape = (224,224,3))
output_1 = pre_trained_model(input_1)
output_2 = pre_trained_model(input_2)
output_3 = pre_trained_model(input_3)
new_model = tf.keras.Model([input_1, input_2, input_3], [output_1, output_2, output_3])
new_model has only 4 layers, due to the Keras API considering the pre_trained_model a layer.
I know that the above option works, as I have seen in many code samples, but I wonder if there is a better option for this. It feels awkward for me, because the access to inner layers of the final model will be messed up, not to mention the fact that the model will have an extra input layer after the added 3 input layers (the input layer from the pre trained model is still intact, and is totally unnecessary).
No, this does not add layers, you are making a multi-input multi-output model, where each siamese branch shares weights. There is no other API in Keras to do this, so this is your only option.
And you can always access the layers of the inner model through the pre_trained_model variable, so there is nothing lost.
I am trying to use pre-trained InceptionV3 model. However, I want to remove initial five layers and add my custom layers. How can I do that? I tried model.layers.pop(0), but that alone will not solve the problem.
Edit:
tf.keras does not help either as mentioned in the first answer:
model.layers.pop() doesn't work in the same way in tf.keras as it doesn in Keras. In tf.keras, model.layers is a view of the model. You can't remove the layers but what you can do is define the layer for which you want the output. For example,
base_model = InceptionV3(shape=shape, weights="imagenet", include_top=True)
# you don't want the last five layers:
base_model_output = base_model.layers[-6].output
# new layers
outputs = Dense(....)(base_model_output)
model = Model(base_model.input, outputs)
Since the first few layers starting from the input are changed, then the pretrained weights cannot be used. So, the architecture can be directly taken from here and modified accordingly instead of trying complex surgeries.
https://github.com/keras-team/keras-applications/blob/master/keras_applications/inception_v3.py
Considering the example of Image classification on ImageNet, How to update the pre-trained model using the new data points.
I have loaded the pre-trained model. I have a new data point that is quite different from the distribution of the original data on which the model was previously trained. So, I would like to update/fine-tune the model with the help of new data point. How to go about doing it? Can anyone help me out in doing it? I am using pytorch 0.4.0 for implementation, running on GPU Tesla K40C.
If you don't want to change the output of the classifier (i.e. the number of classes), then you can simply continue training the model with new example images, assuming that they are reshaped to the same shape that the pretrained model accepts.
On the other hand, if you want to change the number of classes in a pre-trained model, then you can replace the last fully connected layer with a new one and train only this specific layer on new samples. Here's a sample code for this case from PyTorch's autograd mechanics notes:
model = torchvision.models.resnet18(pretrained=True)
for param in model.parameters():
param.requires_grad = False
# Replace the last fully-connected layer
# Parameters of newly constructed modules have requires_grad=True by default
model.fc = nn.Linear(512, 100)
# Optimize only the classifier
optimizer = optim.SGD(model.fc.parameters(), lr=1e-2, momentum=0.9)