I am beginning to build CNN models using Keras.
I have built a CNN with a fairly accurate results using the following architecture.
classifier = Sequential()
classifier.add(Convolution2D(32, (3,3), input_shape = (64, 64, 3), activation='relu'))
classifier.add(MaxPool2D(pool_size = (2,2)))
classifier.add(Convolution2D(32, (3,3), activation='relu'))
classifier.add(MaxPool2D(pool_size = (2,2)))
classifier.add(Convolution2D(32, (3,3), activation='relu'))
classifier.add(MaxPool2D(pool_size = (2,2)))
classifier.add(Convolution2D(32, (3,3), activation='relu'))
classifier.add(MaxPool2D(pool_size = (2,2)))
classifier.add(Flatten())
classifier.add(Dense(units=128, activation='relu'))
classifier.add(Dropout(rate = 0.25))
classifier.add(Dense(units=128, activation='relu'))
classifier.add(Dropout(rate = 0.25))
classifier.add(Dense(units=1, activation='sigmoid'))
classifier.compile(optimizer = 'sgd', loss = 'binary_crossentropy', metrics=['accuracy'])
What I want to do is to run my images through the model, but only the convolutional steps. I am interested in the output of the Flattening process (i.e. get the features from the convolutional steps).
Can someone help me how I can get it in Keras?
Thanks in advance
Here is one solution. If you are interested in the output of layer 'max_pooling2d_4' (You can get the layer name by classifier.summary(), but I suggest you to put names for each layer by e.g. classifier.add(MaxPool2D(pool_size=(2,2), name='pool1'))):
layer_dict = dict([(layer.name, layer) for layer in classifier.layers])
# input tensor
input_tensor = classifier.input
# output tensor of the given layer
layer_output = layer_dict['max_pooling2d_4'].output
# get the output with respect to the input
func = K.function([input_tensor], [layer_output])
# test image: [64, 64, 3]
image = np.ones((64,64,3))
# get activation for the test image
activation = func([image[np.newaxis, :, :, :]])
Related
I am trying to use CNN for feature extraction and XGboost for classification of a image data. I researched and found that it could be done by extracting the data after the convolution layers. I found some source code for similar problem and tried on my own.
model = Sequential()
model.add(Conv2D(32, kernel_size=(3,3), strides=(1,1), padding='same', activation="relu", input_shape = data.shape[1:]))
model.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))
model.add(Conv2D(64, kernel_size=(3,3), padding='same', strides=(1,1), activation="relu") )
model.add(MaxPool2D(pool_size=(2,2), strides=(2,2))) #max pool window 2x2
model.add(Conv2D(128, kernel_size=(3,3), padding='same', strides=(1,1), activation="relu"))
model.add(MaxPool2D(pool_size=(2,2), strides=(2,2))) #max pool window 2x2
model.add(Conv2D(256, kernel_size=(3,3), padding='same', strides=(1,1), activation="relu"))
model.add(MaxPool2D(pool_size=(2,2), strides=(2,2))) #max pool window 2x2
model.add(Flatten())
model.add(Dense(128, activation="relu", name='firstDenseLayer'))
model.add(Dense(1, activation="sigmoid"))
# model.summary()
# print(model)
model.compile(loss="binary_crossentropy", optimizer="adam", metrics=["accuracy"])
model.fit(data, label, batch_size=16, epochs=10, validation_data=(val_data, val_label))
Below i accessed the dense layer named "firstDenseLayer".
import xgboost as xgb
from keras.models import Model
layerName = 'firstDenseLayer'
intermediate_layer_model = Model(inputs=model.input,
outputs=model.get_layer(layerName).output)
intermediate_output = intermediate_layer_model.predict(data)
from xgboost import XGBClassifier
xgbmodel = XGBClassifier(objective='multi:softmax', num_class= 2)
xgbmodel.fit(intermediate_output, label)
xgbmodel.score(intermediate_output, label)
As i am new in this, i have several confusions.
How the data is being flowed. After i extract the features of the pictures via convolution layers, how do i actually access the data from there?
What is this line of code doing? What data is it extracting?
intermediate_output = intermediate_layer_model.predict(data)
When i omit(keep commented out) the below line,
model.fit(data, label, batch_size=16, epochs=10, validation_data=(val_data, val_label))
from the first snippet and run the XGboost model directly the XGboost gives low accuracy and when i don't it gives higher accuracy. Why is it being like that?
Kindly help me out. I am stuck with this for quite a while. I am just trying to access the extracted features data from the last convolution layer and use that data to do classification using XGboost. As i tried to follow the method that i found from online, i am not sure if it is the the only way of doing it. If there is another way kindly let me know.
The model.fit(...) line does what you would expect, it trains the convnet defined by model on some data and labels. Your classifier yielding lower accuracy when you're using randomly initialized weights (i.e. without running fit) is not surprising.
intermediate_layer_model is constructed as a keras model whose output is the dense layer just before the output of model. Note the name parameter given to the dense layer in the construction of model.
You could just as easily give a name to one of the Conv2D layers and access it the same way. Alternatively, you could store the layer in a python variable, i.e. instead of
model.add(Conv2D(256, kernel_size=(3,3), padding='same', strides=(1,1), activation="relu"))
in the model construction it could say
last_conv_layer = Conv2D(256, kernel_size=(3,3), padding='same', strides=(1,1), activation="relu")
model.add(last_conv_layer)
Then for the intermediate_layer_model you put
intermediate_layer_model = Model(inputs=model.input, outputs=last_conv_layer.output)
I am trying to configure a network for character recognition of sequential data like license plates.
Now I would like to use the architecture which is noted in Table 3 in Deep Automatic Licence Plate Recognition system (link: http://www.ee.iisc.ac.in/people/faculty/soma.biswas/Papers/jain_icgvip2016_alpr.pdf).
The architecture the authors presented is this one:
The first layers are very common, but where I was stumbling was the top (the part in the red frame) of the architecture. They mention 11 parallel layers and I am really unsure how to get this in Python. I coded this architecture but it does not seem to be right to me.
model = Sequential()
model.add(Conv2D(64, kernel_size=(5, 5), input_shape = (32, 96, 3), activation = "relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(128, kernel_size=(3, 3), activation = "relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(256, kernel_size=(3, 3), activation = "relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(1024, activation = "relu"))
model.add(Dense(11*37, activation="Softmax"))
model.add(keras.layers.Reshape((11, 37)))
Could someone help? How do I have to code the top to get an equal architecture like the authors?
The code below can build the architecture described in the image.
import tensorflow as tf
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Conv2D, Flatten, MaxPooling2D, Dense, Input, Reshape, Concatenate, Dropout
def create_model(input_shape = (32, 96, 1)):
input_img = Input(shape=input_shape)
'''
Add the ST Layer here.
'''
model = Conv2D(64, kernel_size=(5, 5), input_shape = input_shape, activation = "relu")(input_img)
model = MaxPooling2D(pool_size=(2, 2))(model)
model = Dropout(0.25)(model)
model = Conv2D(128, kernel_size=(3, 3), input_shape = input_shape, activation = "relu")(model)
model = MaxPooling2D(pool_size=(2, 2))(model)
model = Dropout(0.25)(model)
model = Conv2D(256, kernel_size=(3, 3), input_shape = input_shape, activation = "relu")(model)
model = MaxPooling2D(pool_size=(2, 2))(model)
model = Dropout(0.25)(model)
model = Flatten()(model)
backbone = Dense(1024, activation="relu")(model)
branches = []
for i in range(11):
branches.append(backbone)
branches[i] = Dense(37, activation = "softmax", name="branch_"+str(i))(branches[i])
output = Concatenate(axis=1)(branches)
output = Reshape((11, 37))(output)
model = Model(input_img, output)
return model
From my understanding, your implementation is almost correct. The authors train 11 individual classifiers taking as input the output from the Fully Connected Layer. Here, you can think of "parallel" as "independent".
However, you cannot apply the Softmax activation right after the Fully Connected Layer. Since all the classifiers are independent, we want each of them to output a probability for each possible character. Putting things differently, we want the sum of the outputs of each classifier to be 1. Hence, the correct implementation would be:
...
model.add(Dense(1024, activation = "relu"))
# Feeding every neuron with the previous layer's output
model.add(Dense(11*37))
model.add(keras.layers.Reshape((11, 37)))
model.add(keras.activations.softmax(x, axis=1))
I'm using this code:
import keras
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout, Flatten,\
Conv2D, MaxPooling2D
from keras.layers.normalization import BatchNormalization
import numpy as np
np.random.seed(1000)
# (3) Create a sequential model
model = Sequential()
# 1st Convolutional Layer
model.add(Conv2D(kernel_size=96, filters=(11, 11), input_shape=(64,64,3), activation='relu', strides=(4,4), padding='valid'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation before passing it to the next layer
model.add(BatchNormalization())
# 2nd Convolutional Layer
model.add(Conv2D(256, 11, 11, activation='relu', strides=(1,1), padding='valid'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())
# 3rd Convolutional Layer
model.add(Conv2D(384, 3, 3, activation='relu', strides=(1,1), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())
# 4th Convolutional Layer
model.add(Conv2D(384, 3, 3, activation='relu', strides=(1,1), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())
# 5th Convolutional Layer
model.add(Conv2D(256, 3, 3, activation='relu', strides=(1,1), padding='valid'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())
# Passing it to a dense layer
model.add(Flatten())
# 1st Dense Layer
model.add(Dense(4096, input_shape=(224*224*3,)))
model.add(Activation('relu'))
# Add Dropout to prevent overfitting
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())
# 2nd Dense Layer
model.add(Dense(4096))
model.add(Activation('relu'))
# Add Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())
# 3rd Dense Layer
model.add(Dense(1000))
model.add(Activation('relu'))
# Add Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())
output_node=109
# Output Layer
model.add(Dense(output_node.shape, activation='softmax'))
model.summary()
# (4) Compile
model.compile(loss='categorical_crossentropy', optimizer='adam',\
metrics=['accuracy'])
#Fitting dataset
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(rescale = 1./255,
shear_range = 0.2,
zoom_range = 0.2,
horizontal_flip = True)
test_datagen = ImageDataGenerator(rescale = 1./255)
training_set = train_datagen.flow_from_directory('dataset/training_set',
target_size = (64, 64),
batch_size = 32,
class_mode = 'categorical')
test_set = test_datagen.flow_from_directory('dataset/test_set',
target_size = (64, 64),
batch_size = 32,
class_mode = 'categorical')
#steps_per_epoch = number of images in training set / batch size (which is 55839/32)
#validation_steps = number of images in test set / batch size (which is 18739/32)
classifier.fit_generator(
training_set,
steps_per_epoch=55839/32,
epochs=5,
validation_data=test_set,
validation_steps=18739/32)
And I'm getting this error:
TypeError: only size-1 arrays can be converted to Python scalars
I've tried looking up this solution: Keras Model giving TypeError: only size-1 arrays can be converted to Python scalars
But, as you can see I have used the .shape method in my output layer and still it doesn't work. I don't see where an array is being created which needs to be a size 1 array in the line
model.add(Conv2D(kernel_size=96, filters=(11, 11), input_shape=(64,64,3), activation='relu', strides=(4,4), padding='valid'))
because that's where the error is being triggered.
EDIT: I tried to set an integral value for 'filters' as suggested by #TavoGLC as:
model.add(Conv2D(filters=11, kernel_size=96, input_shape=(224,224,3), activation='relu', strides=(4,4), padding='valid', data_format='channels_last'))
and I added a data_format='channels_last' to overcome a negative values problem. That made this line of code run properly, but then the 2nd Convolutional layer started giving me problems.
# 2nd Convolutional Layer
model.add(Conv2D(filters=11, kernel_size=256, strides=(1,1), padding='valid', activation='relu'))
Error:
ValueError: Negative dimension size caused by subtracting 256 from 16 for 'conv2d_77/convolution' (op: 'Conv2D') with input shapes: [?,33,16,5], [256,256,33,11].
Again, I've tried the solutions given here: Negative dimension size caused by subtracting 3 from 1 for 'conv2d_2/convolution'
Nothing just seems to work.
Change these:
filter - use a single integer (number of output filters for the convolution).
kernel_size - use the smaller size as the kernel has to move in the input shape(The shape could decrease for deeper layer so you have to understand the shape of the layer input to get the size)
other convolution layers - you have to use tuple (like Conv2D(256, (11, 11))) else it will be considered as another variable, follow the previous procedure on filter and kernel_size for all Conv2D layers.
for output shape use
output_node=109
# Output Layer
model.add(Dense(output_node, activation='softmax'))
I'm using 3d images as my input and my output...
model = Sequential()
#add model layers
model.add(Convolution3D(64, kernel_size=3, activation="relu", input_shape=(240, 240, 155, 1)))
model.add(Convolution3D(32, kernel_size=3, activation='relu'))
model.add(Dense(10, activation='softmax'))
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
But instead of making the final layer a Dense with softmax, I want the output to be a 3D image of the same dimensions as the input.
What would I have to do to upsample?
There are many ways of upsampling - you can look in github for simple implementation of seg-net or U-net (or seg-u-net).
Say we have a convolutional neural network M. I can extract features from images by using
extractor = Model(M.inputs, M.get_layer('last_conv').output)
features = extractor.predict(X)
How can I get the model that will predict classes using features?
I can't use the following lines because it requires the input of the model to be a placeholder.
predictor = Model([M.get_layer('next_layer').input], M.outputs)
pred = predictor.predict(features)
I also can't use K.function because later I want to use it as part of another model, so I will be appliyng predictor to tf.tensor, not np.array.
This is not the nicest solution, but it works:
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Dense, Dropout, Flatten
def cnn():
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
activation='relu',
input_shape=(28, 28, 1), name='l_01'))
model.add(Conv2D(64, (3, 3), activation='relu', name='l_02'))
model.add(MaxPooling2D(pool_size=(2, 2), name='l_03'))
model.add(Dropout(0.25, name='l_04'))
model.add(Flatten(name='l_05'))
model.add(Dense(128, activation='relu', name='l_06'))
model.add(Dropout(0.5, name='l_07'))
model.add(Dense(10, activation='softmax', name='l_08'))
return model
def predictor(input_shape):
model = Sequential()
model.add(Flatten(name='l_05', input_shape=(12, 12, 64)))
model.add(Dense(128, activation='relu', name='l_06'))
model.add(Dropout(0.5, name='l_07'))
model.add(Dense(10, activation='softmax', name='l_08'))
return model
cnn_model = cnn()
cnn_model.save('/tmp/cnn_model.h5')
predictor_model = predictor(cnn_model.output.shape)
predictor_model.load_weights('/tmp/cnn_model.h5', by_name=True)
Every layer in the model is indexed. So if you know which layers you need, you could loop through them, copying them into a new model. This operation should copy the weights inside the layer as well.
Here's a model (from Oli Blum's answer):
model = Sequential()
# add some layers
model.add(Conv2D(32, kernel_size=(3, 3),
activation='relu',
input_shape=(28, 28, 1), name='l_01'))
model.add(Conv2D(64, (3, 3), activation='relu', name='l_02'))
model.add(MaxPooling2D(pool_size=(2, 2), name='l_03'))
model.add(Dropout(0.25, name='l_04'))
model.add(Flatten(name='l_05'))
model.add(Dense(128, activation='relu', name='l_06'))
model.add(Dropout(0.5, name='l_07'))
model.add(Dense(10, activation='softmax', name='l_08'))
Say you wanted the last three layers:
def extract_layers(main_model, starting_layer_ix, ending_layer_ix):
# create an empty model
new_model = Sequential()
for ix in range(starting_layer_ix, ending_layer_ix + 1):
curr_layer = main_model.get_layer(index=ix)
# copy this layer over to the new model
new_model.add(curr_layer)
return new_model
It depends on what you want to do.
If you are going to throw away the feature extractor afterwards
If you plan on training the feature extractor later
If you are going to use the extracted features but you don't intend on training the model used to generate them, you could use the predict method to get the features as you did:
features = extractor.predict(X)
then save its output to a file (np.save or cPickle or whatever).
After that you could use that new dataset as the input to a new model.
If you plan on training the feature extractor later you'll need to stack the two networks as seen here with vgg as feature extractor https://github.com/fchollet/keras/issues/4576:
img_width, img_height = 150, 150
vgg16_model = VGG16(include_top=False, weights='imagenet')
input = Input(batch_shape=vgg16_model.output_shape)
x = GlobalAveragePooling2D()(input)
x = Dense(256, activation='relu')(x)
x = Dropout(0.5)(x)
predict = Dense(1, activation='sigmoid')(x)
top_model = Model(input, predict)
top_model.load_weights(os.path.join(data_path, 'VGG16Classifier.hdf5'))
input = Input(shape=(3, img_width, img_height))
x = vgg16_model(input)
predict = top_model(x)
model = Model(input, predict)
PS: This example uses channels first ordering. If you are using tensorflow you should change the shape to shape=(img_width, img_height,3 )