Develop Reference

Feed keras model input to the output layer

So I am building a keras sequential model in which the last output layer is an Upsampling2D layer & I need to feed the input image to that output layer to do a simple operation and return the output, any ideas?
EDIT :
The model mentioned before is the generator of a GAN model in which I need to add the input image to the output of the generator before feeding it to the discriminator

1.You can define a backbone model using inputs of pre-trained model and the outputs of the last layer before the output layer of pre-trained model
2.Base on that backbone model, defined new model have that new skip connection and the output layer as same as pre-trained model
3.Set the weights of output layer in new model to equal to weights of output layer in pre-trained model, using: new_model.layers[-1].set_weights(pre_model.layers[-1].get_weights())
Here is one good article about Adding Layers to the middle of a pre-trained network whithout invalidating the weights

So for the future reference, I solved it by using lambda layers as follow :
# z is the input I needed to use later on with the generator output to perform a certain function
generated_image = self.generator(z)
generated_image_modified=tf.keras.layers.Concatenate()([generated_image,z])
# with x[...,variable_you_need_range] you can access the input we just concatenated in your train loop
lambd = tf.keras.layers.Lambda(lambda x: your_function(x[...,2:5],x[...,:2]))(generated_image_modified)
full_model = self.discriminator(lambd)
self.combined = Model(z,outputs = full_model)

Accessing intermediate tensors of a Keras Model that were not explicitly exposed as layers in TF 2.0

Is it possible to access pre-activation tensors in a Keras Model? For example, given this model:
import tensorflow as tf
image_ = tf.keras.Input(shape=[224, 224, 3], batch_size=1)
vgg19 = tf.keras.applications.VGG19(include_top=False, weights='imagenet', input_tensor=image_, input_shape=image_.shape[1:], pooling=None)
the usual way to access layers is:
intermediate_layer_model = tf.keras.models.Model(inputs=image_, outputs=[vgg19.get_layer('block1_conv2').output])
intermediate_layer_model.summary()
This gives the ReLU outputs for a layer, while I would like the ReLU inputs. I tried doing this:
graph = tf.function(vgg19, [tf.TensorSpec.from_tensor(image_)]).get_concrete_function().graph
outputs = [graph.get_tensor_by_name(tname) for tname in [
'vgg19/block4_conv3/BiasAdd:0',
'vgg19/block4_conv4/BiasAdd:0',
'vgg19/block5_conv1/BiasAdd:0'
]]
intermediate_layer_model = tf.keras.models.Model(inputs=image_, outputs=outputs)
intermediate_layer_model.summary()
but I get the error
ValueError: Unknown graph. Aborting.
The only workaround I've found is to edit the model file to manually expose the intermediates, turning every layer like this:
x = layers.Conv2D(256, (3, 3), activation="relu", padding="same", name="block3_conv1")(x)
into 2 layers where the 1st one can be accessed before activations:
x = layers.Conv2D(256, (3, 3), activation=None, padding="same", name="block3_conv1")(x)
x = layers.ReLU(name="block3_conv1_relu")(x)
Is there a way to acces pre-activation tensors in a Model without essentially editing Tensorflow 2 source code, or reverting to Tensorflow 1 which had full flexibility accessing intermediates?

There is a way to access pre-activation layers for pretrained Keras models using TF version 2.7.0. Here's how to access two intermediate pre-activation outputs from VGG19 in a single forward pass.
Initialize VGG19 model. We can omit top layers to avoid loading unnecessary parameters into memory.
vgg19 = tf.keras.applications.VGG19(
include_top=False,
weights="imagenet"
)
This is the important part: Create a deepcopy of the intermediate layer form which you like to have the features, change the activation of the conv layers to linear (i.e. no activation), rename the layer (otherwise two layers in the model will have the same name which will raise errors) and finally pass the output of the previous through the copied conv layer.
# for more intermediate features wrap a loop around it to avoid copy paste
b5c4_layer = deepcopy(vgg19.get_layer("block5_conv4"))
b5c4_layer.activation = tf.keras.activations.linear
b5c4_layer._name = b5c4_layer.name + str("_preact")
b5c4_preact_output = b5c4_layer(vgg19.get_layer("block5_conv3").output)
b2c2_layer = deepcopy(vgg19.get_layer("block2_conv2"))
b2c2_layer.activation = tf.keras.activations.linear
b2c2_layer._name = b2c2_layer.name + str("_preact")
b2c2_preact_output = b2c2_layer(vgg19.get_layer("block2_conv1").output)
Finally, get the outputs and check if they equal post-activation outputs when we apply ReLU-activation.
vgg19_features = Model(vgg19.input, [b2c2_preact_output, b5c4_preact_output])
vgg19_features_control = Model(vgg19.input, [vgg19.get_layer("block2_conv2").output, vgg19.get_layer("block5_conv4").output])
b2c2_preact, b5c4_preact = vgg19_features(tf.keras.applications.vgg19.preprocess_input(img))
b2c2, b5c4 = vgg19_features_control(tf.keras.applications.vgg19.preprocess_input(img))
print(np.allclose(tf.keras.activations.relu(b2c2_preact).numpy(),b2c2.numpy()))
print(np.allclose(tf.keras.activations.relu(b5c4_preact).numpy(),b5c4.numpy()))
True
True
Here's a visualization similar to Fig. 6 of Wang et al. to see the effect in the feature space.
Input image

To get output of each layer. You have to define a keras function and evaluate it for each layer.
Please refer the code as shown below
from tensorflow.keras import backend as K
inp = model.input # input
outputs = [layer.output for layer in model.layers] # all layer outputs
functors = [K.function([inp], [out]) for out in outputs] # evaluation functions
For more details on this please refer SO Answer.

ResNet always predicting one class

I am trying to do transfer learning in Keras + Tensorflow on a selected subset of Places-205 dataset, containing only 27 categories. I am using InceptionV3, DenseNet121 and ResNet50, pre-trained on ImageNet, and add a couple of extra layers to adapt to my classes. If the model is ResNet, I add Flatten + Dense for classfication, and if it is DenseNet or Inceptionv3, I add Global Avg Pool + Dense (relu) + Dense (classification).
This is the code snippet:
x = base_model.output
if FLAGS.model in 'resnet50':
x = Flatten(name="flatten")(x)
else:
x = GlobalAveragePooling2D()(x)
# Let's add a fully-connected layer
x = Dense(1024, activation = 'relu')(x)
# And a logistic layer
predictions = Dense(classes, activation = 'softmax')(x)
For DenseNet and Inceptionv3 the training is ok, and the validation accuracy hits 70%, but for ResNet the validation accuracy stays fixed at 0.0369/0.037 (which is 1/27, my number of classes). It seems like it always predicts one class, but it's weird because its training progresses ok and the unspecific model code is exactly the same as for DenseNet and InceptionV3, which do work as expected.
Do you have any idea why it happens?
Thanks a lot!

I had a similar issue as you #Ciprian Andrei Focsaneanu, and what I have found to have worked was to make the previous layers (before the fully connected layers) trainable, as the filters/features of the ResNet50 were not suitable for my application.
Strangely enough, I also trained the VGG16 models, which was initially on the same images (imagenet) but its filters worked for my application, but I digress.
Here's the link to a page that inspired me to do this: https://datascience.stackexchange.com/questions/16840/multi-class-neural-net-always-predicting-1-class-after-optimization
Hope this helps!

Error when trying to output previous to last layer in Keras

I am trying to output the previous to last Dense layer in a keras model. I first load the model architecture and the weights:
base_model = applications.ResNet50(weights = None,
include_top = False,
input_shape = (image_size[0], image_size[1], nb_channels))
top_model = Sequential()
top_model.add(Flatten(input_shape=base_model.output_shape[1:]))
top_model.add(Dense(1024, init = 'glorot_uniform', activation='relu', name = 'last_layer_1024'))
top_model.add(Dropout(0.5))
top_model.add(Dense(nb_classes, activation = 'softmax', name = 'softmax_layer'))
top_model_tensor = top_model(base_model.output)
model = Model(inputs = base_model.input, outputs = top_model_tensor)
model.load_weights(weights_path)
Then I remove the last Dense layer by doing this:
model.layers[-1].pop()
#model.outputs = [model.layers[-1].layers[-1].output]
#model.layers[-1].layers[-1].outbound_nodes = []
If I uncomment the commented lines, I get this error: InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'flatten_1_input' with dtype float. If I keep them commented, the last dense layer is NOT effectively removed (by that I mean that when I call predict on model, I still get the output of the last dense layer). How can I solve this issue?
Also, if there is a different method to get the model to output the previous to last dense layer, I can take that as an answer too (instead of trying to fix this way of doing it).
Another solution that does not work is to just cut the long model after you load weights by simply doing this:
short_top_model = Model(top_model.input, top_model.get_layer('last_layer_1024').output)
You get the following error:
RuntimeError: Graph disconnected: cannot obtain value for tensor Tensor("flatten_1_input:0", shape=(?, 1, 1, 2048), dtype=float32, device=/device:GPU:2) at layer "flatten_1_input". The following previous layers were accessed without issue: []

Trying to cut models, change their inputs/outputs etc. sounds not what keras expects from users.
You should just create another model that follows the same path but ends earlier:
#do this "before" creating "top_model_tensor".
short_top = Model(
top_model.input,
top_model.get_layer('last_layer_1024').output
)
top_model_out = top_model(base_model.output)
short_top_out = short_top(base_model.output)
model = Model(base_model.input,top_model_out)
short_model = Model(base_model.input,short_top_out)
Choose which one to use depending on the expected results. Training one updates the other.

A shorter version of the above answer.
#again create connection between two model
feature_vec_model = Model(
top_model.input,
top_model.get_layer('last_layer_1024').output
)
feature_vec_model_output = feature_vec_model(base_model.output)
#Connection created
# Define final connected model & load pretrained weights
complete_feature_vec_model = Model(base_model.input,feature_vec_model_output)
complete_feature_vec_model.load_weights("path_to_model")

How can I know if weights were loaded in Keras Model?

I am using keras for tensorflow 1.4. For some unknown reason, I continue failing when I try to download the weights (vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5') using:
base_model = VGG16(weights='imagenet', include_top=False)
so I manually downloaded the weights from the link cited in the script.
https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5
Since the VGG16's weight cannot be set to the path of the weights (it has to be imagenet or None), I used the code lines:
base_model = VGG16(weights=None, include_top=False)
path = os.path.join('weights','vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5')
base_model.load_weights(path)
It is the same than base_model = VGG16(weights='imagenet', include_top=False) ?
How can I check that the weights and outputs are correct?

You can access the weights in an .h5 file as:
w1 = h5py.File('someFile.h5')
allKeys = w1.keys()
first_layer_wts = w1[allKeys[0]][:] # assuming first layer has weights
After you load the weights in a Keras model, then you can access them as:
w2 = model.get_weights()
Now check if first_layer_wts and the corresponding weights in w2 are equal. This may need some tweaking. For example, if the layer is convolutional, then you have two weight arrays (one for kernel and another for bias). You can access them both as:
first_layer_wts_kernel = w1[allKeys[0]]['kernel'][:]
first_layer_wts_bias = w1[allKeys[0]]['bias'][:]
You can figure this out by always checking the available keys.