Develop Reference

How to get the encoder part of a neural network in PyTorch?

I want to apply transfer learning (initiate the encoder of my custom net with the weights from a pre-trained encoder of UNet or ResNet). So the queation is:
Given an instance of UNet or ResNet in Pytorch, how to extract the encoder part of a ResNet or UNet in PyTorch?
This blog shows a way to do it, but it requirs me to have the class of UNet or ResNet at first which is not practical for me. Because the instance of UNet or ResNet is obtained by a function like: net = get_resnet(depth=34), I can only get the instance of UNet or ResNet but I can not get the class of them.

For ResNet the final layer is simply self.fc, so if you instantiate a ResNet model, you can redefine self.fc to your preferred classification task, keeping the rest of the model intact, including pretrained weights if applicable.
For UNet, this is a little trickier, because the decoder consists of upsampling layers as well as the output layer, but the same can be done replacing self.up1, self.up2, self.up3, self.up4, and self.outc.
Be sure to replace the layers after loading weights.

How can I pretrain for a multi-input DNN for Keras

I'm currently developing a model using Keras + Tensorflow in order to determine the temperature range of a set of proteins. What I first did was create a pre-trained model that converts the proteins into embeddings and then predicts its respective temperature.
What I want to do now is incorporate this pre=trained model to a new model which can use this given model and respective weights as input. Then fit on a new dataset and predict once again. The following code for the new top model is:
UPDATED CODE
'Load Pretrained Model'
loaded_model = keras.models.load_model('pretrained_model')
#Freeze all model layer weights
loaded_model.trainable = False
input1 = np.expand_dims(x_train['input1'],1)
input2 = np.expand_dims(x_train['input2'], 1)
input3 = x_train['input3']
#Redefine Input Layers for ANN
input1 = Input(shape = (input1.shape[1],), name = "input1")
input2 = Input(shape = (input2.shape[1],), name = "input2")
input3 = Input(shape = (input3.shape[1],), name = "input2")
base_inputs = [input1, input2, input3]
x = loaded_model(base_inputs, training = False)
x = Dense(64, activation = "relu", kernel_regularizer=regularizers.l2(0.01))(x)
output = Dense(1, activation = "sigmoid")(x)
top_model = Model(inputs = base_inputs, outputs = output)
# Compile the Model
top_model.compile(loss='mse', optimizer = Adam(lr = 0.0001), metrics = ['mse'])
This is not working correctly and I'm not sure on how to get this up and running. I'm struggling a bit to get this and come across this error quite often:
AttributeError: 'Dense' object has no attribute 'shape'
Any thoughts?

Could you please try to use the initialize the inputs using the keras layers and try?
You have initialized the input shapes using Numpy.
But, If I am right unfortunately the dense layer which you have imported from keras does not support this ('Dense' object has no attribute 'op').
Kindly note that, 'Input' is a keras layer
Could you try to as specified in the following link to initialize the keras inputs (https://keras.io/guides/functional_api/)?
As an Example,
input1 = keras.Input(shape=(1,))
input2 = keras.Input(shape=(1,))
input3 = keras.Input(shape=(1,))
It totally depends on your machine learning architecture whether to make layers trainable or not. In Case of transfer learning, You can just use the trained weights from a pre-trained model and train your new network using the trained weights acquired from that model. In this case, you have to freeze the layers of the pre-trained model. Hence trainable = False. You use these weights in the mathematical calculation of the hidden layers you will use in your custom architecture.
But from your code snippet, I could predict that you are not using any hidden layers like LSTM, RNN or any other cells for your sequential data. Also, you are trying to provide the initialized Numpy inputs to a pre-trained model. I don't think whether it is a right way to do so. From the code snippet, you are making the layer not trainable, but you are then trying to train the model.
Also, If I am right, I think that you have to train with new set of data using the Pre-trained model right? If so, then kindly look at the following link(https://keras.io/guides/transfer_learning/).

Considering your problem, I could suggest that transfer learning approach would be a feasible solution. In transfer learning, you could use the trained model from one domain of a set of data to train similar kind of problem using other set of data. To clearly understand the how to make the layers trainable and freeze it, and fine-tuning can be understood in the following link(https://keras.io/guides/transfer_learning/)
Concerning the Attribute error, it is recommended to have a look at the following link (https://keras.io/guides/functional_api/)
At first, you to initialize the input node for keras along with the shape of the inputs with respect to the data you will feed to the train the model. An Example is shown below as follows,
inputs = keras.Input(shape=(784,))
or it can be something like as follows, if you are providing the image data
img_inputs = keras.Input(shape=(32, 32, 3))
The dense layer expects the input should be in a specific shape, which you can find according to your data. If you are not sure about it, please analyse the data at first. It will give you lot of information to proceed further.

How to convert Pytorch model to ONNX model?

I have a model that contains two-part encoder and decoder.
Each of them is a PyTorch model.
I inference them as below.
features = encoder(input)
output = decoder(features)
and I want to convert them into one ONNX model instead of two ONNX models.
How can I do it?

In general you should build a new class which inherits nn.Module that combines the multiple models. In this case you have a simple feed-forward network so we can use the nn.Sequential convenience class.
E.g.
model = torch.nn.Sequential([encoder, decoder])
# inference example
model.eval()
output = model(input)
# save model to onnx file ...

How to remove (pop) initial layers of Keras InceptionV3 pre-trained model?

I am trying to use pre-trained InceptionV3 model. However, I want to remove initial five layers and add my custom layers. How can I do that? I tried model.layers.pop(0), but that alone will not solve the problem.
Edit:
tf.keras does not help either as mentioned in the first answer:

model.layers.pop() doesn't work in the same way in tf.keras as it doesn in Keras. In tf.keras, model.layers is a view of the model. You can't remove the layers but what you can do is define the layer for which you want the output. For example,
base_model = InceptionV3(shape=shape, weights="imagenet", include_top=True)
# you don't want the last five layers:
base_model_output = base_model.layers[-6].output
# new layers
outputs = Dense(....)(base_model_output)
model = Model(base_model.input, outputs)

Since the first few layers starting from the input are changed, then the pretrained weights cannot be used. So, the architecture can be directly taken from here and modified accordingly instead of trying complex surgeries.
https://github.com/keras-team/keras-applications/blob/master/keras_applications/inception_v3.py

how to add tanh to one embedding layer in keras

I wanted to add one tanh layer to embedding layer with keras functional api:
x=layers.Embedding(vocab_size, 8, input_length=max_length)(input)
output=keras.activations.tanh(x)
model = Model(inputs=input, outputs=output)
model.compile(optimizer='rmsprop',loss='categorical_crossentropy',metrics=['accuracy'])
model.fit(data, labels)
but system told me I must use keras layers ,not tensor. I searched a lot keras tutorials. There is only one way to solve this problem:
model.add(Activation('tanh'))
but it is Sequential model which I don't want to use.Is there some ways to solve this with functional api?

With the functional api it's almost the same as the Sequential model:
output = Activation('tanh')(x)