I am working on image dataset
I first resized all images
im_size1 = 128
im_size2 = 128
i = 0
for f, breed in tqdm(df_train.values):
if type(cv2.imread('train/{}.jpeg'.format(f)))==type(None):
continue
else:
img = cv2.imread('train/{}.jpeg'.format(f))
label = one_hot_labels[i]
x_train.append(cv2.resize(img, (im_size1, imt_size2)))
y_train.append(label)
i += 1
np.save('x_train2',x_train)
np.save('y_train2',y_train)
print('Done')
On the second run i passed Histogram of Gradient Feature
Note: Model was same in both cases. Ran on different projects, got different results
i = 0
for f, breed in tqdm(df_train.values):
if type(cv2.imread('train/{}.jpeg'.format(f)))==type(None):
continue
else:
img = cv2.imread('train/{}.jpeg'.format(f))
label = one_hot_labels[i]
resizedImage = cv2.resize(img, (im_size1, im_size2))
hog_vec, hog_vis = feature.hog(resizedImage, visualize=True)
resizedImageVec = cv2.resize(hog_vec, (im_size1, im_size2))
x_train.append(resizedImageVec)
#np.concatenate(x_train, hog_vec[:])
y_train.append(label)
i += 1
np.save('x_train2hog',x_train)
np.save('y_train2hog',y_train)
print('Done')
Then configured the model like so
base_model = ResNet50(weights = None, include_top=False, input_shape=(im_size1, im_size2, 3))
# Add a new top layer
x = base_model.output
x = Flatten()(x)
x = Dropout(0.2)(x)
x = Dense(32, activation='relu')(x)
x = Dense(16, activation='relu')(x)
predictions = Dense(num_class, activation='softmax')(x)
# This is the model we will train
model = Model(inputs=base_model.input, outputs=predictions)
# First: train only the top layers (which were randomly initialized)
#for layer in base_model.layers:
# layer.trainable = False
model.compile(loss='categorical_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])
callbacks_list = [keras.callbacks.EarlyStopping(monitor='val_acc', verbose=1)]
model.summary()
Now my professor has asked me this.
"Train last layer using images, then train last layer using features (different representation of images). Concatenate these two layers, and then train one more layer."
Please guide me how can i accomplish this
Have a look at the Merge layers from Keras API
keras.layers.Concatenate(axis=-1)
and the functional version:
keras.layers.concatenate(inputs, axis=-1)
Related
I'm creating an Ensemble of Vgg19, DenseNet, and EfficientNetB1.
The code is as follows:
IMAGE_SIZE = (224,224,3)
import tensorflow as tf
vgg19 = tf.keras.applications.vgg19.VGG19(
input_shape=IMAGE_SIZE, weights='imagenet', include_top=False)
for layer in vgg19.layers:
layer._name = layer._name + str('_19')
layer.trainable = False
effnetb1 =tf.keras.applications.efficientnet.EfficientNetB1(
include_top=False, weights='imagenet', input_shape=IMAGE_SIZE)
for layer in effnetb1.layers:
layer._name = layer._name + str('_B1')
layer.trainable=False
densenet=tf.keras.applications.densenet.DenseNet121(
include_top=False, weights="imagenet", input_shape=IMAGE_SIZE)
for layer in densenet.layers:
layer._name = layer._name + str('_Dense')
layer.trainable=False
from keras.layers import Input, Flatten, Concatenate, Dense, Average, Dropout
inp = Input(IMAGE_SIZE)
vgg19_x = Flatten()(vgg19(inp))
vgg19_x = Dense(256, activation='relu')(vgg19_x)
effnet_x = Flatten()(effnetb1(inp))
effnet_x = Dense(256, activation='relu')(effnet_x)
densenet_x = Flatten()(densenet(inp))
densenet_x = Dense(256, activation='relu')(densenet_x)
from keras.models import Model
x = Concatenate()([vgg19_x, effnet_x, densenet_x])
x = Dense(128, activation='relu')(x)
x = Dropout(0.30)(x)
x = Dense(64, activation='relu')(x)
out = Dense(2, activation='softmax')(x)
model = Model(inputs = inp, outputs = out)
model.compile(
loss='categorical_crossentropy',
optimizer=tf.keras.optimizers.Adam(
learning_rate=0.0005,
name="Adam"),
metrics=['accuracy']
)
model.summary()
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
checkpointer = ModelCheckpoint(filepath="/content/drive/MyDrive/ensemble/ensemble-weights.hdf5", verbose=1, save_best_only=True)
r = model.fit(
training_set,
validation_data=test_set,
epochs=30,
steps_per_epoch=len(training_set),
validation_steps=len(test_set),
callbacks = [checkpointer]
)
The code runs fine and the training is successfully taking place when I'm not using the callback. But when I use a ModelCheckpoint, I get the following error after 1st epoch:
ValueError: The target structure is of type `<class 'keras.engine.keras_tensor.KerasTensor'>`
KerasTensor(type_spec=TensorSpec(shape=(None, 224, 224, 3), dtype=tf.float32, name='input_5'), name=...
However, the input structure is a sequence (<class 'list'>) of length 0.
[]
nest cannot guarantee that it is safe to map one to the other.
Can anyone tell me what's wrong here? Also, is it because I'm concatenating three models?
Your help will be appreciated. Thank you!
I also ran into this issue while trying to implement a nested model (which is what would be constructed here after you create the concatenated model).
The issue seems to be that Keras cannot handle the inputs and outputs of nested models in newer tensorflow versions(tf 2.0 and above). Depending on the version you are on, you might want to either explicitly refer the input/output of the nested model you are using. In tf2.6, what seems to work is to define separate models for each part - ie - the common layers added after concatenation should also be wrapped in a model like below (taken from here):
#Make GradCAM heatmap following the Keras tutorial.
last_conv_layer = model.layers[-4].layers[-1]
last_conv_layer_model = keras.Model(model.layers[-4].inputs, last_conv_layer.output)
# Second, we create a model that maps the activations of the last conv
# layer to the final class predictions
classifier_input = keras.Input(shape=last_conv_layer.output.shape[1:])
x = classifier_input
for layer in model.layers[-3:]:
x = layer(x)
classifier_model = keras.Model(classifier_input, x)
#Preparing the image with the preprocessing layers
preprocess_layers = keras.Model(model.inputs, model.layers[-5].output)
img_array = preprocess_layers(prepared_image)
# Then, we compute the gradient of the top predicted class for our input image
# with respect to the activations of the last conv layer
with tf.GradientTape() as tape:
# Compute activations of the last conv layer and make the tape watch it
last_conv_layer_output = last_conv_layer_model(img_array)
tape.watch(last_conv_layer_output)
# Compute class predictions
preds = classifier_model(last_conv_layer_output)
top_pred_index = tf.argmax(preds[0])
top_class_channel = preds[:, top_pred_index]
# This is the gradient of the top predicted class with regard to
# the output feature map of the last conv layer
grads = tape.gradient(top_class_channel, last_conv_layer_output)
You can also check the following github issues (they are not very related, but deal with a similar problem) - issue1, issue2, issue3
I'm new to Keras. With transfer learning following an online tutorial, I tried to train a model for my custom dataset and below is my code. Now how should I perform classification given an image?
Here is the training code:
def build_finetune_model(base_model, dropout, fc_layers, num_classes):
for layer in base_model.layers:
layer.trainable = False
x = base_model.output
x = Flatten()(x)
for fc in fc_layers:
# New FC layer, random init
x = Dense(fc, activation='relu')(x)
x = Dropout(dropout)(x)
# New softmax layer
predictions = Dense(num_classes, activation='softmax')(x)
finetune_model = Model(inputs=base_model.input, outputs=predictions)
return finetune_model
train_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)
train_generator = train_datagen.flow_from_directory(TRAIN_DIR,
target_size=(HEIGHT, WIDTH),
batch_size=BATCH_SIZE)
base_model = MobileNetV2(weights='imagenet', include_top=False, input_shape=(HEIGHT, WIDTH, 3))
root=TRAIN_DIR
class_list = [ item for item in os.listdir(root) if os.path.isdir(os.path.join(root, item)) ]
print (class_list)
FC_LAYERS = [1024, 1024]
dropout = 0.5
finetune_model = build_finetune_model(base_model, dropout=dropout, fc_layers=FC_LAYERS, num_classes=len(class_list))
adam = Adam(lr=0.00001)
finetune_model.compile(adam, loss='categorical_crossentropy', metrics=['accuracy'])
filepath="./checkpoints/" + "MobileNetV2_{epoch:02d}_{acc:.2f}" +"_model_weights.h5"
checkpoint = ModelCheckpoint(filepath, monitor=["acc"], verbose=1, mode='max', save_weights_only=True)
callbacks_list = [checkpoint]
history = finetune_model.fit_generator(train_generator, epochs=NUM_EPOCHS, workers=8,
steps_per_epoch=steps_per_epoch,
shuffle=True, callbacks=callbacks_list)
And here is my classify code, which only returns an array of float numbers!
if __name__ == '__main__':
model = load_model('ResNet50_model_weights.h5')
img_path = 'test.jpg'
img = image.load_img(img_path, target_size=(300, 300))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
print('Input image shape:', x.shape)
results = model.predict(x)
print('Predicted:', results)
Your models' last layer is softmax, which means that it outputs probabilities for each class between 0 and 1. In order to tell which class the input image is predicted to be, take the argmax of results. argmax returns the index of the maximum element in the array.
predicted_class_label = np.argmax(results)
print("network predicted input image is class {}".format(predicted_class_label))
the def get_vgg_twoeyes() is the definition of my model.
I have loaded a pretrained model,which is trained on the same computer,then I want to fine-tune the model. Before retrain the model, I set the model.trainable false that assure the weights of model are fixed. Before my training, the weights are the same to the saved weights. I found the output of model.fit is different from the output of model.predict.
As I assume that the model.fit with same weights as model.predict should output the same result because model.trainable is false that means the model.fit behave the model.predict.
def get_vgg_twoeyes(optimizer='adam', model_type='VGG16',fc1_size=1024,
fc2_size=512, fc3_size=256):
kern_init = initializers.glorot_normal()
img_input = Input(shape=(36, 60, 3), name='img_input')
headpose_input = Input(shape=(2,), name='headpose_input')
# create the base pre-trained model
if model_type == 'VGG19':
base_model = VGG19(input_tensor=img_input, weights='imagenet', include_top=False)
elif model_type == 'VGG16':
base_model = VGG16(input_tensor=img_input, weights='imagenet', include_top=False)
else:
raise Exception('Unknown model type in get_vgg_twoeyes')
# add a global spatial average pooling layer
x = base_model.output
x = GlobalAveragePooling2D()(x)
# let's add a fully-connected layer
x = Dense(fc1_size, kernel_initializer=kern_init)(x)
x = concatenate([x, headpose_input])
x = BatchNormalization()(x)
x = Activation('relu')(x)
x = Dense(fc2_size, kernel_initializer=kern_init)(x)
x = BatchNormalization()(x)
x = Activation('relu')(x)
gaze_predictions = Dense(2, kernel_initializer=kern_init, name='pred_gaze')(x)
# this is the model we will train
model = Model(inputs=[img_input, headpose_input], outputs=gaze_predictions)
model.compile(optimizer=optimizer, loss=angle_loss, metrics=['accuracy', accuracy_angle])
return model
# fine-tune the model
models=load_model(model_path + "15Fold" + prefix + ''+str(i) +
suffix + ".h5",custom_objects={'accuracy_angle':accuracy_angle,
'angle_loss': angle_loss}))
model.trainable=False
adam = Adam(lr=0.0001, beta_1=0.9, beta_2=0.95)
model.compile(optimizer=adam, loss=angle_loss, metrics= ['accuracy', accuracy_angle])
model.fit({'img_input':cal_images,'headpose_input':cal_headposes},
cal_gazes,shuffle=False,batch_size=32,epochs=1,callbacks=
[losshistory()])
predgaze=model.predict({'img_input': cal_images, 'headpose_input':
cal_headposes}, batch_size=2,verbose=1)
You have to probably compile the model again after setting model.trainable=False. Else, you can manually freeze the layers individually like
for l in model.layers:
l.trainable=False
This is my first time creating a multi-input keras model, and I am not sure if I am doing something wrong as my training accuracy is not rising above 50 percent. If I use a single-input model, I can get to 90 percent plus pretty easily. This model uses the same input but different sizes of the input and then takes the max of the softmax prediction as the final prediction. So, I would assume it would be easier to train the model.
Here is the code:
def full_model_2(model_key):
base_model, preprocess = basemodel(model_key) #base model is just an imported imagenet model
base_model2, preprocess = basemodel(model_key)
for layer in base_model.layers:
layer.name = layer.name + "dup"
#first model
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(2048, activation='relu')(x)
x = BatchNormalization()(x)
x = Dense(512, activation='relu')(x)
x = BatchNormalization()(x)
predictions_x = Dense(classes, activation='softmax')(x)
#second model
y = base_model2.output
y = GlobalAveragePooling2D()(y)
y = Dense(2048, activation='relu')(y)
y = BatchNormalization()(y)
y = Dense(512, activation='relu')(y)
y = BatchNormalization()(y)
predictions_y = Dense(classes, activation='softmax')(y)
#predictions = Average()([predictions_x, predictions_y])
predictions = Maximum()([predictions_x, predictions_y])
model = Model(inputs= [base_model.input,base_model2.input], outputs=predictions)
for layer in model.layers[:-10]:
layer.trainable = False
return model, preprocess
My generator looks something like this (take two generators and creates two inputs)
def train_generator():
train_gen, _ = create_generators(batch_size, preprocess, img_size)
train_gen2, _ = create_generators(batch_size, preprocess, img_size2)
print(train_gen.seed)
print(train_gen2.seed)
while True:
X1i = train_gen.next()
X2i = train_gen2.next()
yield [X1i[0], X2i[0]], X1i[1]
Here is the original generator function:
train_datagen = ImageDataGenerator(preprocessing_function = preprocess)
train_generator = train_datagen.flow_from_directory(
train_data_dir,
shuffle = True,
seed = seed,
target_size = (img_size, img_size),
batch_size = batch_size,
class_mode = 'categorical')
return train_generator
model.compile(optimizer= r_optimizer,
loss='categorical_crossentropy',
metrics = ['accuracy'])
I have plotted the inputs from the generators, and I am indeed getting the same images but different sizes, so the input from generators seems to be okay.
I am thinking my model setup is off.
Thank you for your suggestions.
I'm new to ConvNets and Python and want to implement the following:
I want to use the pretrained vgg16 model and add 3 fully connected layers after it with an L2-Normalization at the end.
So Data->VGG16->FC (1x4096)->FC (1x4096)->FC (1x3)->L2-Norm->Output
The first and second FC get an array 1x4096 the last FC gets an array 1x3 where the L2-Norm is performed.
Can anyone give me a hint how to do that ?
I found that I can load the model like that :
model_vgg19 = models.vgg19(pretrained=True)
But how can I add the FCs and the L2-Norm after that ? And how can I get Test-Data through the model ?
I'm quoting an example mentioned in Keras#3465
In Keras framework, if you mention include_top = False while loading your pre-trained model it will not include the final classification layer. You can add your custom FC layers at the end as shown in the example below:
#load vgg16 without dense layer and with theano dim ordering
base_model = VGG16(weights = 'imagenet', include_top = False, input_shape = (3,224,224))
#number of classes in your dataset e.g. 20
num_classes = 20
x = Flatten()(base_model.output)
x = Dense(4096, activation='relu')(x)
x = Dropout(0.5)(x)
x = BatchNormalization()(x)
predictions = Dense(num_classes, activation = 'softmax')(x)
#create graph of your new model
head_model = Model(input = base_model.input, output = predictions)
#compile the model
head_model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
head_model.summary()
.
.
.
#train your model on data
head_model.fit(x, y, batch_size = batch_size, verbose = 1)