How to store the flatten result of a CNN? - python

I have the following convolutional neural network to apply to images:
classifier = Sequential()
classifier.add(Convolution2D(128, (3, 3), input_shape = (128, 128, 3), activation = 'relu'))
classifier.add(MaxPooling2D(pool_size = (2, 2)))
classifier.add(Convolution2D(64, (3, 3), activation = 'relu'))
classifier.add(MaxPooling2D(pool_size = (2, 2)))
classifier.add(Flatten())
After applying the convolutional and maxpooling layers, I flatten the results and want to store only that result (later I want to work with this result using unsupervised methods). How do I do that? The only examples I have continue the proccess to fit the model and I never store the flatten layers.

This is covered in the Keras documentation for pretrained models. See the examples about feature extraction, https://keras.io/applications/#extract-features-with-vgg16
Once you have your model, you just do:
features = model.predict(x)

Related

Saving Layer Values of CNN- Keras

I have created the following simple CNN in Keras (borrowed from a DeepLizard tutorial).
model = Sequential([
Conv2D(filters = 10, kernel_size = (3, 3), activation = 'relu', padding = 'same', input_shape = (320, 320, 3)),
MaxPool2D(pool_size = (2, 2), strides = 2),
Conv2D(filters = 10, kernel_size = (3, 3), activation = 'relu', padding = 'same'),
MaxPool2D(pool_size = (2, 2), strides = 2),
Flatten(),
Dense(20, activation = 'softmax'),
])
model.summary()
model.compile(optimizer = Adam(lr = 0.0001), loss = 'categorical_crossentropy', metrics =['accuracy'])
model.fit(x = train_batches, validation_data = valid_batches, epochs = 10, verbose = 2)
predictions = model.predict(x = test_batches, verbose = 0)
As you can see, I am saving the predictions generated by the model to a dataframe named "predictions". But I am also interested in saving the outputs for each of the MaxPool2D layers, the Conv2D layer, and the flatten layer as well. Is there a way that I can save the outputs of those layers to dataframes/lists as well? Is there a functionality for this in Keras?
Thank you!
You can use model.get_layer() function to extract any layer of your model. Visit the documentation here: https://keras.io/api/models/model/#getlayer-method
Thank you for your responses. They led me in the right direction. Here is the solution I ended up utilizing. I recreated the model, but configured the predictions to output the desired layer (in this case, "conv2d", the first convolutional layer). This produces a 4-D array as an output, where the 1st dimension corresponds to the input, the 2nd and 3rd dimensions are the two dimensions of a filter's outputted feature map, and the 4th dimension corresponds to the n-filters being used in that layer (in this case, the 4th dimension would be '10'). My next quest is to find a way to split that 4 dimensional array into separate 3-dimensional arrays, where each new array corresponds to a distinct filter. In this case, I would be looking for 10 3-dimensional arrays, one for each of the filters used in the first convolutional layer.
from keras.models import Model
model = Sequential([
Conv2D(filters = 10, kernel_size = (3, 3), activation = 'relu', padding = 'same', input_shape = (320, 320, 3)),
MaxPool2D(pool_size = (2, 2), strides = 2),
Conv2D(filters = 10, kernel_size = (3, 3), activation = 'relu', padding = 'same'),
MaxPool2D(pool_size = (2, 2), strides = 2),
Flatten(),
Dense(20, activation = 'softmax'),
])
layer_name = 'conv2d'
intermediate_layer_model = Model(inputs=model.input, outputs=model.get_layer(layer_name).output)
intermediate_output = intermediate_layer_model.predict(valid_batches)

Combining CNN and bidirectional LSTM

I am trying to combine CNN and LSTM for image classification.
I tried the following code and I am getting an error. I have 4 classes on which I want to train and test.
Following is the code:
from keras.models import Sequential
from keras.layers import LSTM,Conv2D,MaxPooling2D,Dense,Dropout,Input,Bidirectional,Softmax,TimeDistributed
input_shape = (200,300,3)
Model = Sequential()
Model.add(TimeDistributed(Conv2D(
filters=16, kernel_size=(12, 16), activation='relu', input_shape=input_shape)))
Model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2),strides=2)))
Model.add(TimeDistributed(Conv2D(
filters=24, kernel_size=(8, 12), activation='relu')))
Model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2),strides=2)))
Model.add(TimeDistributed(Conv2D(
filters=32, kernel_size=(5, 7), activation='relu')))
Model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2),strides=2)))
Model.add(Bidirectional(LSTM((10),return_sequences=True)))
Model.add(Dense(64,activation='relu'))
Model.add(Dropout(0.5))
Model.add(Softmax(4))
Model.compile(loss='sparse_categorical_crossentropy',optimizer='adam')
Model.build(input_shape)
I am getting the following error:
"Input tensor must be of rank 3, 4 or 5 but was {}.".format(n + 2))
ValueError: Input tensor must be of rank 3, 4 or 5 but was 2.
I found a lot of problems in the code:
your data are in 4D so simple Conv2D are ok, TimeDistributed is not needed
your output is 2D so set return_sequences=False in the last LSTM cell
your last layers are very messy: no need to put a dropout between a layer output and an activation
you need categorical_crossentropy and not sparse_categorical_crossentropy because your target is one-hot encoded
LSTM expects 3D data. So you need to pass from 4D (the output of convolutions) to 3D. There are two possibilities you can adopt: 1) make a reshape (batch_size, H, W * channel); 2) (batch_size, W, H * channel). In this way, u have 3D data to use inside your LSTM
here a full model example:
def ReshapeLayer(x):
shape = x.shape
# 1 possibility: H,W*channel
reshape = Reshape((shape[1],shape[2]*shape[3]))(x)
# 2 possibility: W,H*channel
# transpose = Permute((2,1,3))(x)
# reshape = Reshape((shape[1],shape[2]*shape[3]))(transpose)
return reshape
model = Sequential()
model.add(Conv2D(filters=16, kernel_size=(12, 16), activation='relu', input_shape=input_shape))
model.add(MaxPooling2D(pool_size=(2, 2),strides=2))
model.add(Conv2D(filters=24, kernel_size=(8, 12), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2),strides=2))
model.add(Conv2D(filters=32, kernel_size=(5, 7), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2),strides=2))
model.add(Lambda(ReshapeLayer)) # <========== pass from 4D to 3D
model.add(Bidirectional(LSTM(10, activation='relu', return_sequences=False)))
model.add(Dense(nclasses,activation='softmax'))
model.compile(loss='categorical_crossentropy',optimizer='adam')
model.summary()
here the running notebook

Configuration of CNN model for recognition of sequential data - Architecture of the top of the CNN - Parallel Layers

I am trying to configure a network for character recognition of sequential data like license plates.
Now I would like to use the architecture which is noted in Table 3 in Deep Automatic Licence Plate Recognition system (link: http://www.ee.iisc.ac.in/people/faculty/soma.biswas/Papers/jain_icgvip2016_alpr.pdf).
The architecture the authors presented is this one:
The first layers are very common, but where I was stumbling was the top (the part in the red frame) of the architecture. They mention 11 parallel layers and I am really unsure how to get this in Python. I coded this architecture but it does not seem to be right to me.
model = Sequential()
model.add(Conv2D(64, kernel_size=(5, 5), input_shape = (32, 96, 3), activation = "relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(128, kernel_size=(3, 3), activation = "relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(256, kernel_size=(3, 3), activation = "relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(1024, activation = "relu"))
model.add(Dense(11*37, activation="Softmax"))
model.add(keras.layers.Reshape((11, 37)))
Could someone help? How do I have to code the top to get an equal architecture like the authors?
The code below can build the architecture described in the image.
import tensorflow as tf
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Conv2D, Flatten, MaxPooling2D, Dense, Input, Reshape, Concatenate, Dropout
def create_model(input_shape = (32, 96, 1)):
input_img = Input(shape=input_shape)
'''
Add the ST Layer here.
'''
model = Conv2D(64, kernel_size=(5, 5), input_shape = input_shape, activation = "relu")(input_img)
model = MaxPooling2D(pool_size=(2, 2))(model)
model = Dropout(0.25)(model)
model = Conv2D(128, kernel_size=(3, 3), input_shape = input_shape, activation = "relu")(model)
model = MaxPooling2D(pool_size=(2, 2))(model)
model = Dropout(0.25)(model)
model = Conv2D(256, kernel_size=(3, 3), input_shape = input_shape, activation = "relu")(model)
model = MaxPooling2D(pool_size=(2, 2))(model)
model = Dropout(0.25)(model)
model = Flatten()(model)
backbone = Dense(1024, activation="relu")(model)
branches = []
for i in range(11):
branches.append(backbone)
branches[i] = Dense(37, activation = "softmax", name="branch_"+str(i))(branches[i])
output = Concatenate(axis=1)(branches)
output = Reshape((11, 37))(output)
model = Model(input_img, output)
return model
From my understanding, your implementation is almost correct. The authors train 11 individual classifiers taking as input the output from the Fully Connected Layer. Here, you can think of "parallel" as "independent".
However, you cannot apply the Softmax activation right after the Fully Connected Layer. Since all the classifiers are independent, we want each of them to output a probability for each possible character. Putting things differently, we want the sum of the outputs of each classifier to be 1. Hence, the correct implementation would be:
...
model.add(Dense(1024, activation = "relu"))
# Feeding every neuron with the previous layer's output
model.add(Dense(11*37))
model.add(keras.layers.Reshape((11, 37)))
model.add(keras.activations.softmax(x, axis=1))

Specific Dropout in Keras

I would like to train an autoencoder by using only specific PARTS of a layer (the layer named FEATURES in the autoencoder example at the bottom of this question).
In my case, NOK pictures for a new product are very rare, but needed for training. The aim is generate NOK pictures from OK pictures (all examples I found did the opposite). The idea is to force learning OK-picture structure in features[0:n-x] and learning NOK-picture structure (maybe from a similiar product) in features[n-x:n] in order to use the NOK-features as parameters to generate NOK-pictures from OK-pictures.
Two ideas came to my mind using a non-random dropout
(1) keras.layers.Dropout(rate, noise_shape=None, seed=None) has the noise_shape argument, but I am not sure if it helps me as it only describes the shape. It would be perfect to be able to provide a mask consisting of {0,1} to apply on the layer in order to switch on/off specific nodes
(2) creating a custom layer (named MaskLayer below) which performs masking specific nodes of the layer e.g. as a tuple of {0,1}.
I have read this, but I do not think it applies (generate a layer by concatenating layers which can be freezed separately).
def autoEncGenerate0( imgSizeX=28, imgSizeY=28, imgDepth=1): ####:
''' keras blog autoencoder'''
input_img = Input(shape=(imgSizeX, imgSizeY, imgDepth))
x = Conv2D(32, (3, 3), activation='relu', padding='same')(input_img)
x = MaxPooling2D((4, 4), padding='same')(x)
x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)
encoded0 = MaxPooling2D((8, 8), padding='same', name="FEATURES")(x)
encoded1 = MaskLayer(mask)(encoded0) # TO BE DONE (B2) masking layer parts
x = Conv2D(32, (3, 3), activation='relu', padding='same')(encoded1)
x = UpSampling2D((8, 8))(x)
x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((4, 4))(x)
decoded = Conv2D( imgDepth, (3, 3), activation='sigmoid', padding='same')(x)
autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')
return( autoencoder)
Thanks for hints.
There is trainable attribute that each instance of tf.keras.layer.Layer has which disables training of the variables of that layer. UpSampling2D doesn't have any variables so you CAN'T train it. What you want is to train the variables of the convolutional layer that comes before that upsampling layer.
You could do it like this:
# define architecture here
autoencoder = Model(input_img, decoded)
layers_names = [l.name for l in autoencoder.layers]
trainable_layer_index = layers_names.index('FEATURES') - 1
for i in range(len(autoencoder.layers)):
if i != trainable_layer_index:
autoencoder.layers[i].trainable = False
# compile here
NOTE that you compile the model AFTER you set layers to trainable/non-trainable.

Extracting last layers of keras model as a submodel

Say we have a convolutional neural network M. I can extract features from images by using
extractor = Model(M.inputs, M.get_layer('last_conv').output)
features = extractor.predict(X)
How can I get the model that will predict classes using features?
I can't use the following lines because it requires the input of the model to be a placeholder.
predictor = Model([M.get_layer('next_layer').input], M.outputs)
pred = predictor.predict(features)
I also can't use K.function because later I want to use it as part of another model, so I will be appliyng predictor to tf.tensor, not np.array.
This is not the nicest solution, but it works:
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Dense, Dropout, Flatten
def cnn():
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
activation='relu',
input_shape=(28, 28, 1), name='l_01'))
model.add(Conv2D(64, (3, 3), activation='relu', name='l_02'))
model.add(MaxPooling2D(pool_size=(2, 2), name='l_03'))
model.add(Dropout(0.25, name='l_04'))
model.add(Flatten(name='l_05'))
model.add(Dense(128, activation='relu', name='l_06'))
model.add(Dropout(0.5, name='l_07'))
model.add(Dense(10, activation='softmax', name='l_08'))
return model
def predictor(input_shape):
model = Sequential()
model.add(Flatten(name='l_05', input_shape=(12, 12, 64)))
model.add(Dense(128, activation='relu', name='l_06'))
model.add(Dropout(0.5, name='l_07'))
model.add(Dense(10, activation='softmax', name='l_08'))
return model
cnn_model = cnn()
cnn_model.save('/tmp/cnn_model.h5')
predictor_model = predictor(cnn_model.output.shape)
predictor_model.load_weights('/tmp/cnn_model.h5', by_name=True)
Every layer in the model is indexed. So if you know which layers you need, you could loop through them, copying them into a new model. This operation should copy the weights inside the layer as well.
Here's a model (from Oli Blum's answer):
model = Sequential()
# add some layers
model.add(Conv2D(32, kernel_size=(3, 3),
activation='relu',
input_shape=(28, 28, 1), name='l_01'))
model.add(Conv2D(64, (3, 3), activation='relu', name='l_02'))
model.add(MaxPooling2D(pool_size=(2, 2), name='l_03'))
model.add(Dropout(0.25, name='l_04'))
model.add(Flatten(name='l_05'))
model.add(Dense(128, activation='relu', name='l_06'))
model.add(Dropout(0.5, name='l_07'))
model.add(Dense(10, activation='softmax', name='l_08'))
Say you wanted the last three layers:
def extract_layers(main_model, starting_layer_ix, ending_layer_ix):
# create an empty model
new_model = Sequential()
for ix in range(starting_layer_ix, ending_layer_ix + 1):
curr_layer = main_model.get_layer(index=ix)
# copy this layer over to the new model
new_model.add(curr_layer)
return new_model
It depends on what you want to do.
If you are going to throw away the feature extractor afterwards
If you plan on training the feature extractor later
If you are going to use the extracted features but you don't intend on training the model used to generate them, you could use the predict method to get the features as you did:
features = extractor.predict(X)
then save its output to a file (np.save or cPickle or whatever).
After that you could use that new dataset as the input to a new model.
If you plan on training the feature extractor later you'll need to stack the two networks as seen here with vgg as feature extractor https://github.com/fchollet/keras/issues/4576:
img_width, img_height = 150, 150
vgg16_model = VGG16(include_top=False, weights='imagenet')
input = Input(batch_shape=vgg16_model.output_shape)
x = GlobalAveragePooling2D()(input)
x = Dense(256, activation='relu')(x)
x = Dropout(0.5)(x)
predict = Dense(1, activation='sigmoid')(x)
top_model = Model(input, predict)
top_model.load_weights(os.path.join(data_path, 'VGG16Classifier.hdf5'))
input = Input(shape=(3, img_width, img_height))
x = vgg16_model(input)
predict = top_model(x)
model = Model(input, predict)
PS: This example uses channels first ordering. If you are using tensorflow you should change the shape to shape=(img_width, img_height,3 )

Categories

Resources