I am trying to train a convolutional neural network but I get a quite high number of false positive classified objects. I am using two classes, each 10.000 images with quite obvious differences. I would expect a rather easy task for a CNN, also I used some hand crafted features with a random forest classifier before which worked quite well.
This is the model I am using:
def build(width, height, depth, classes):
model = Sequential()
inputShape = (height, width, depth)
chanDim = -1
# if we are using "channels first", update the input shape
# and channels dimension
if K.image_data_format() == "channels_first":
inputShape = (depth, height, width)
chanDim = 1
# CONV => RELU => POOL layer set
model.add(Conv2D(32, (3, 3), padding="same",
input_shape=inputShape))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
# (CONV => RELU) * 2 => POOL layer set
model.add(Conv2D(64, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(Conv2D(64, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
# (CONV => RELU) * 3 => POOL layer set
model.add(Conv2D(128, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(Conv2D(128, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(Conv2D(128, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
# first (and only) set of FC => RELU layers
model.add(Flatten())
model.add(Dense(512))
model.add(Activation("relu"))
model.add(BatchNormalization())
model.add(Dropout(0.25))
# softmax classifier
model.add(Dense(classes))
model.add(Activation("softmax"))
# return the constructed network architecture
return model
After applying data augmentation, training and validation loss look better but still I get to many false positives.
Here is a screenshot of some example images from a validation set (screenshot), green marked are correct classified, the rest are false positives. Any suggestion, how to improve my model?
Edit:
I add also the code for pre-processing the images:
import matplotlib
matplotlib.use("Agg")
from smallvggnet import SmallVGGNet
from sklearn.preprocessing import LabelBinarizer
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.optimizers import SGD
from imutils import paths
import matplotlib.pyplot as plt
import numpy as np
import argparse
import random
import pickle
import cv2
import os
from keras.utils.np_utils import to_categorical
# initialize the data and labels
print("[INFO] loading images...")
data = []
labels = []
# grab the image paths and randomly shuffle them
imagePaths = sorted(list(paths.list_images("C:/06112020_hyphae/all/")))
random.seed(42)
random.shuffle(imagePaths)
# loop over the input images
for imagePath in imagePaths:
image = cv2.imread(imagePath)
image = cv2.resize(image, (350, 150))
data.append(image)
label = imagePath.split(os.path.sep)[-2].split('/')[-1]
if label == 'pos':
label = 1
else:
label = 0
labels.append(label)
# scale the raw pixel intensities to the range [0, 1]
data = np.array(data, dtype="float") / 255.0
labels = np.array(labels)
# partition the data into training and testing splits using 75% of
# the data for training and the remaining 25% for testing
(trainX, testX, trainY, testY) = train_test_split(data, labels, test_size=0.25, random_state=42)
unique, counts = np.unique(trainY, return_counts=True)
print (dict(zip(unique, counts)))
trainY = to_categorical(trainY)
testY = to_categorical(testY)
# construct the image generator for data augmentation
#aug = ImageDataGenerator(rotation_range=30, width_shift_range=0.1, height_shift_range=0.1, zoom_range=0.2, horizontal_flip=True, fill_mode="nearest")
aug = ImageDataGenerator()
# initialize our VGG-like Convolutional Neural Network
model = SmallVGGNet.build(width=350, height=150, depth=3,
classes=2)
# initialize our initial learning rate, # of epochs to train for,
# and batch size
INIT_LR = 0.01
EPOCHS = 20
BS = 32
# initialize the model and optimizer (you'll want to use
# binary_crossentropy for 2-class classification)
print("[INFO] training network...")
opt = SGD(lr=INIT_LR, decay=INIT_LR / EPOCHS)
model.compile(loss="binary_crossentropy", optimizer=opt,
metrics=["accuracy"])
# train the network
H = model.fit(x=aug.flow(trainX, trainY, batch_size=BS),
validation_data=(testX, testY), steps_per_epoch=len(trainX) // BS,
epochs=EPOCHS)
Related
So I'm trying to make a faceID with Tensorflow using CNN type, i only gave to my data 15 images of my face(is a simple project), and i want it to predict if it is my face or is not my face, like true or false, the problem is that I don't understand what is the error in my code. this is
ValueError: logits and labels must have the same shape ((None, 2) vs (None, 1))
and here is my code
# Face ID project, using CNN tensorflow
from tensorflow.keras.preprocessing.image import img_to_array
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D, BatchNormalization, Activation
from tensorflow.keras import backend as K
import numpy as np
import cv2
import glob
# Preparing the data and parameters
epochs = 10
lr = 1e-3
batch_size = 64
img_dims = (96,96,3)
data = []
labels = []
image_files = glob.glob("C:/Users/berna/Desktop/Programming/AI_ML_DL/Projects/FaceID/Data/*")
for img in image_files:
image = cv2.imread(img)
image = cv2.resize(image, (img_dims[0], img_dims[1]))
image = img_to_array(image)
data.append(image)
if img == img:
label = 1
else:
label = 0
labels.append([label])
# Preproccesing the data (convert arrays)
data = np.array(data, dtype="float32") / 255.0
labels = np.array(labels)
X = data
y = labels
def build(width, height, depth, classes):
model = Sequential()
inputShape = height, width, depth
chanDim = -1
if K.image_data_format() == "channels_first":
inputShape = depth, height, width
chanDim = 1
# Creating the model
model.add(Conv2D(32, (3,3), padding="same", input_shape=inputShape))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(MaxPooling2D(pool_size=(3,3)))
model.add(Dropout(0.25))
model.add(Conv2D(64, (3,3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(Conv2D(64, (3,3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.25))
model.add(Conv2D(128, (3,3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(Conv2D(128, (3,3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(1024))
model.add(Activation("relu"))
model.add(BatchNormalization())
model.add(Dropout(0.5))
model.add(Dense(classes))
model.add(Activation("sigmoid"))
return model
# Build the model call
model = build(width=img_dims[0], height=img_dims[1], depth=img_dims[2], classes=2)
# compile the model
opt = Adam(lr=lr, decay=lr/epochs)
model.compile(loss="binary_crossentropy",
optimizer=opt,
metrics=['accuracy'])
# fitting the model
H = model.fit(X, y, batch_size=batch_size,
epochs=epochs, verbose=1)
model.save('faceid.model')
As stated by #Frightera, replacing:
model.add(Dense(classes)) by model.add(Dense(1)) should work.
Your label is an integer but your last layer output a 2d array.
I'm a newbie to deep learning (and machine learning), and I created a python script that uses TensorFlow/Keras to identify flowers into different groups using this dataset. Here is my code: (I'm doing this on Kaggle)
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import LabelEncoder
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.optimizers import Adam, SGD
from keras.utils import to_categorical
from keras.layers import Conv2D, Dropout, Dense, Flatten
import matplotlib.pyplot as plt
import cv2
import os
NUM_CLASSES = 5
IMG_SIZE = 150
DAISY = '../input/flowers-recognition/flowers/daisy'
DANDELION = '../input/flowers-recognition/flowers/dandelion'
ROSE = '../input/flowers-recognition/flowers/rose'
SUNFLOWER = '../input/flowers-recognition/flowers/sunflower'
TULIP = '../input/flowers-recognition/flowers/tulip'
x = []
y = []
def train_data_gen(DIR, ID):
for img in os.listdir(DIR):
try:
path = DIR + '/' + img
img = plt.imread(path)
img = cv2.resize(img,(IMG_SIZE,IMG_SIZE))
x.append(img)
y.append(ID)
except:
None
train_data_gen(DAISY, 0)
train_data_gen(DANDELION, 1)
train_data_gen(ROSE, 2)
train_data_gen(SUNFLOWER, 3)
train_data_gen(TULIP, 4)
x = np.array(x)
y = to_categorical(y,num_classes = 5)
x_train,x_test,y_train,y_test = train_test_split(x, y, test_size = 0.15)
x_train,x_val,y_train,y_val = train_test_split(x_train, y_train, test_size = 0.15)
datagen = ImageDataGenerator(
featurewise_center=False,
samplewise_center=False,
samplewise_std_normalization=False,
rotation_range=60,
zoom_range = 0.1,
width_shift_range=0.1,
height_shift_range=0.1,
shear_range=0.1,
fill_mode = "reflect"
)
datagen.fit(x_train)
model = Sequential()
model.add(Conv2D(64, kernel_size=(3, 3), strides=2, activation='relu', input_shape=(IMG_SIZE, IMG_SIZE, 3)))
model.add(Dropout(0.5))
model.add(Conv2D(128, kernel_size=(3, 3), strides=2, activation='relu'))
model.add(Dropout(0.5))
model.add(Conv2D(128, kernel_size=(3, 3), strides=2, activation='relu'))
model.add(Dropout(0.5))
model.add(Conv2D(128, kernel_size=(3, 3), strides=2, activation='relu'))
model.add(Dropout(0.5))
model.add(Flatten())
model.add(Dense(1024, activation='relu'))
model.add(Dense(5, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit_generator(datagen.flow(x_train,y_train,batch_size=16), epochs=10, steps_per_epoch=x_train.shape[0]//16, validation_data=(x_val, y_val), verbose=1)
I know that the epochs is 10 here, but later on, I set it to 100. That was taking a really like time, so I stopped it on the 63rd epoch. Even then, it was floating around 60% accuracy, which is way too low. What could be something that I can change to make it more accurate? Would it be that my CNN is too small for this? Or is there something wrong with my data? I'm really new to this so I can't specify my question any further than "Why is my model performing badly?"
Thank you all in advance for the constructive feedback.
You model uses too many Dropout layers. model.add(Dropout(0.5)) effectively drops 50% of your neurons of your incoming layer and on top of that you have 4 of these. You are most probably underfitting.
After each Conv2D layer add a keras.layers.MaxPooling2D layer.
First try removing all the Dropout layers. In that case you will run the risk of overfitting - but if you don't overfit then there is no point of Dropout layers. If you do overfit, experiment with just 1 dropout layer with 20% dropout rate and gradually increase that to 50% and then maybe add another dropout of 20% and continue.
This question already has answers here:
Multiple images input to the same CNN using Conv3d in keras
(2 answers)
Closed 3 years ago.
I have a dataset of 15 class with 460 images all. I want to enter every 8 sequences of images at the same time to the same CNN structure. I use conv3d to do that, but I'm confusing with input shape, it returns error.
This is my model:
IMAGE_DIMS = (8, 460, 60, 60, 3)
data = []
labels = []
# loading images...
imagePaths = "dataset\\path"
listing = os.listdir(imagePaths)
for imagePath in listing:
image_fold = os.listdir(imagePaths + "\\" + imagePath)
for file in image_fold:
im = (imagePaths + "\\" + imagePath + "\\" + file)
image = cv2.imread(im)
image = cv2.resize(image, (IMAGE_DIMS[2], IMAGE_DIMS[3]))
image = img_to_array(image)
data.append(image)
label= imagePath.split(os.path.sep)[-1]
labels.append(label)
# scale the raw pixel intensities to the range [0, 1]
data = np.array(data, dtype="float") / 255.0
labels = np.array(labels)
# binarize the labels
lb = LabelBinarizer()
labels = lb.fit_transform(labels)
(trainX, testX, trainY, testY) = train_test_split(data, labels, test_size=0.2, random_state=42)
model = Sequential()
sample= IMAGE_DIMS[0]
frame=IMAGE_DIMS[1]
height = IMAGE_DIMS[2]
width=IMAGE_DIMS[3]
channels=IMAGE_DIMS[4]
classes=len(lb.classes_)
inputShape = (sample, frame, height, width, channels)
chanDim = -1
if K.image_data_format() == "channels_first":
inputShape = (sample, frame, channels, height, width)
chanDim = 1
model.add(Conv3D(32, (3, 3, 3), padding="same", batch_input_shape=inputShape))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(MaxPooling3D(pool_size=(2, 2, 2), padding="same", data_format="channels_last"))
model.add(Dropout(0.25))
model.add(Conv3D(64, (3, 3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(MaxPooling3D(pool_size=(2, 2, 2), padding="same", data_format="channels_last"))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128))
model.add(Activation("relu"))
model.add(BatchNormalization())
model.add(Dropout(0.5))
# softmax classifier
model.add(Dense(classes))
model.add(Activation("softmax"))
model.summary()
opt = Adam(lr=INIT_LR, decay=INIT_LR / EPOCHS)
model.compile(loss="categorical_crossentropy", optimizer= opt, metrics=["accuracy"])
H = model.fit(trainX, trainY, batch_size=BS, epochs=EPOCHS, verbose=1,validation_data (testX,testY))
and this is my model summary:
But I get the following error:
ValueError: Error when checking input: expected conv3d_1_input to have 5 dimensions, but got array with shape (368, 60, 60, 3)
How can I fix the error, can anyone please help me, I will be thankful for any help. I know the problem with the input shape, the compiler refer to the model.fit step. I thing trainX, testX, trainY, testY must be in 5-dim, but I cannot able to that.
If I understand correctly, you would like to fit your model with 8 images which is called actually batch. So when you call the method model.fit() set batch_size = 8. Another point that, I think, you confused is about the input shape. If you would like to fit images to the network, your input shape is the height x width of the image and the number of channels which is in your case RGB. So, the set input_shape = (3, 60, 60). Please be aware of that the network structure does not includes the total number of images in it. Because the NN structure does not need to know what is the training number. When you fit the training images to the network it will just take a batch of it and does the training job. Lastly, Instead of using 3D convolution layer, you need to use 2D. Think it as a 2D frame that moves over the training image and it does the movement for each channel. Therefore, the frame size need to has a 2D shape, set it (x, x). This frame is called kernel in documents.
The following code just an sample and has not been tested. I hope it helps to understand the structure:
model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=(3, 60, 60)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(number_of_classes))
model.add(Activation('softmax'))
I'm building a keras model of convolution neural network for predicting the correct class and classify the tested objects. the model have the conv2D, activation, maxpooling, dropout, flatten, dense layers. after that I training the network on large dataset, but it take a very long time for training, it may reach to 3,4 days, What I need is to reduce the time required to training the network, Is there any way to do that in python?
I have tried to optimize the learning rate by using the LR_Finder class as follow:
from LR_Finder import LRFinder
lr_finder = LRFinder(min_lr=1e-5,max_lr=1e-2, steps_per_epoch=np.ceil(len(trainX) // BS), epochs=100)
But this also did not give me any reduction about the time required.
This is the code of my model:
class SmallerVGGNet:
#staticmethod
def build(width, height, depth, classes):
# initialize the model along with the input shape to be
# "channels last" and the channels dimension itself
model = Sequential()
inputShape = (height, width, depth)
chanDim = -1
# if we are using "channels first", update the input shape
# and channels dimension
if K.image_data_format() == "channels_first":
inputShape = (depth, height, width)
chanDim = 1
# CONV => RELU => POOL
model.add(Conv2D(32, (3, 3), padding="same",
input_shape=inputShape))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(MaxPooling2D(pool_size=(3, 3)))
model.add(Dropout(0.25))
# (CONV => RELU) * 2 => POOL
model.add(Conv2D(64, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(Conv2D(64, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
# (CONV => RELU) * 2 => POOL
model.add(Conv2D(128, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(Conv2D(128, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=chanDim))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
# first (and only) set of FC => RELU layers
model.add(Flatten())
model.add(Dense(1024))
model.add(Activation("relu"))
model.add(BatchNormalization())
model.add(Dropout(0.5))
# softmax classifier
model.add(Dense(classes))
model.add(Activation("softmax"))
# return the constructed network architecture
return model
and after that I trained the model as following code:
EPOCHS = 100
INIT_LR = 1e-3
BS = 32
IMAGE_DIMS = (96, 96, 3)
data = []
labels = []
# grab the image paths and randomly shuffle them
imagePaths = sorted(list(paths.list_images("Dataset")))
random.seed(42)
random.shuffle(imagePaths)
# loop over the input images
for imagePath in imagePaths:
# load the image, pre-process it, and store it in the data list
image = cv2.imread(imagePath)
image = cv2.resize(image, (IMAGE_DIMS[1], IMAGE_DIMS[0]))
image = img_to_array(image)
data.append(image)
label = imagePath.split(os.path.sep)[-2]
labels.append(label)
# scale the raw pixel intensities to the range [0, 1]
data = np.array(data, dtype="float") / 255.0
labels = np.array(labels)
print("[INFO] data matrix: {:.2f}MB".format(data.nbytes / (1024 * 1000.0)))
# binarize the labels
lb = LabelBinarizer()
labels = lb.fit_transform(labels)
# partition the data into training and testing splits using 80% of
# the data for training and the remaining 20% for testing
(trainX, testX, trainY, testY) = train_test_split(data,
labels, test_size=0.2, random_state=42)
# construct the image generator for data augmentation
aug = ImageDataGenerator(rotation_range=25, width_shift_range=0.1,
height_shift_range=0.1, shear_range=0.2, zoom_range=0.2,
horizontal_flip=True, fill_mode="nearest")
# initialize the model
model = SmallerVGGNet.build(width=IMAGE_DIMS[1], height=IMAGE_DIMS[0],
depth=IMAGE_DIMS[2], classes=len(lb.classes_))
opt = Adam(lr=INIT_LR, decay=INIT_LR / EPOCHS)
model.compile(loss="categorical_crossentropy", optimizer= opt,
metrics=["accuracy"])
print("model compiled in few minutes successfully ^_^")
# train the network
H = model.fit_generator(aug.flow(trainX, trainY, batch_size=BS),
validation_data=(testX, testY), steps_per_epoch=len(trainX) // BS,
epochs=EPOCHS, verbose=1)
According to this code,I expected the output required some minutes or may be a few hours, but when it reach to training in model.fit_generator step, the actual time required is about many hours for every epoch and it requires some days to train all the network or it may be crash and stop working. Is there any way to reduce the training time?
set use_multiprocessing=True and workers>1 when you call fit_generator because the default is to execute the generator on the main thread only
I am having an issue with Keras where evaluate function gives different training loss (way higher) and accuracy(way lower) value as compared to the value that I get during training. I am aware that this question has already been asked at several places (here, here), but I think my issue is different and still not answered in those forums.
Explanation of the Task
It is supposed to be a very simple task. All I am doing is to overfit to my own dataset of 256 images (29x29x3) with 256 output classes (one for each image).
Dataset
Case 1
x_train = All the pixel values in the image = i where i goes from 0 to 255.
y_train = i
Case 2
x_train = Centre 5*5 patch of the pixel values in the image = i where i goes from 0 to 255. All the other pixel values are same for all the images.
y_train = i
This gives me 256 images in total for the training data in each case. (It would be more clear if you just have a look at the code)
Here is my code to reproduce the issue -
from __future__ import print_function
import os
import keras
from keras.datasets import mnist
from keras.models import Sequential, load_model
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D, Activation
from keras.layers.normalization import BatchNormalization
from keras.callbacks import ModelCheckpoint, LearningRateScheduler, Callback
from keras import backend as K
from keras.regularizers import l2
import matplotlib.pyplot as plt
import PIL.Image
import numpy as np
from IPython.display import clear_output
# The GPU id to use, usually either "0" or "1"
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="1"
# To suppress the warnings
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
## Hyperparamters
batch_size = 256
num_classes = 256
l2_reg=0.0
epochs = 500
## input image dimensions
img_rows, img_cols = 29, 29
## Train Image (I took a random image from ImageNet)
train_img_name = 'n01871265_279.JPEG'
ret = PIL.Image.open(train_img_name) #Opening the image
ret = ret.resize((img_rows, img_cols)) #Resizing the image
img = np.asarray(ret, dtype=np.uint8).astype(np.float32) #Converting it to numpy array
print(img.shape) # (29, 29, 3)
## Creating the training data
#############################
x_train = np.zeros((256, img_rows, img_cols, 3))
y_train = np.zeros((256,), dtype=int)
for i in range(len(y_train)):
temp_img = np.copy(img)
## Case1 of dataset
# temp_img[:, :, :] = i # changing all the pixel values
## Case2 of dataset
temp_img[12:16, 12:16, :] = i # changing the centre block of 5*5 pixels
x_train[i, :, :, :] = temp_img
y_train[i] = i
##############################
## Common stuff in Keras
if K.image_data_format() == 'channels_first':
print('Channels First')
x_train = x_train.reshape(x_train.shape[0], 3, img_rows, img_cols)
input_shape = (3, img_rows, img_cols)
else:
print('Channels Last')
x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 3)
input_shape = (img_rows, img_cols, 3)
## Normalizing the pixel values
x_train = x_train.astype('float32')
x_train /= 255
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
## convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
## Model definition
def model_toy(mom):
model = Sequential()
model.add( Conv2D(filters=64, kernel_size=(7, 7), strides=(1,1), input_shape=input_shape, kernel_regularizer=l2(l2_reg)) )
model.add(Activation('relu'))
model.add(BatchNormalization(momentum=mom, epsilon=0.00001))
#Default parameters kept same as PyTorch
#Meaning of PyTorch momentum is different from Keras momentum.
# PyTorch mom = 0.1 is same as Keras mom = 0.9
model.add( Conv2D(filters=128, kernel_size=(7, 7), strides=(1, 1), kernel_regularizer=l2(l2_reg)))
model.add(Activation('relu'))
model.add(BatchNormalization(momentum=mom, epsilon=0.00001))
model.add(Conv2D(filters=256, kernel_size=(5, 5), strides=(1, 1), kernel_regularizer=l2(l2_reg)))
model.add(Activation('relu'))
model.add(BatchNormalization(momentum=mom, epsilon=0.00001))
model.add(Conv2D(filters=512, kernel_size=(5, 5), strides=(1, 1), kernel_regularizer=l2(l2_reg)))
model.add(Activation('relu'))
model.add(BatchNormalization(momentum=mom, epsilon=0.00001))
model.add(Conv2D(filters=1024, kernel_size=(5, 5), strides=(1, 1), kernel_regularizer=l2(l2_reg)))
model.add(Activation('relu'))
model.add(BatchNormalization(momentum=mom, epsilon=0.00001))
model.add( Conv2D( filters=2048, kernel_size=(3, 3), strides=(1, 1), kernel_regularizer=l2(l2_reg) ) )
model.add(Activation('relu'))
model.add(BatchNormalization(momentum=mom, epsilon=0.00001))
model.add(Conv2D(filters=4096, kernel_size=(3, 3), strides=(1, 1), kernel_regularizer=l2(l2_reg)))
model.add(Activation('relu'))
model.add(BatchNormalization(momentum=mom, epsilon=0.00001))
# Passing it to a dense layer
model.add(Flatten())
model.add(Dense(1024, kernel_regularizer=l2(l2_reg)))
model.add(Activation('relu'))
model.add(BatchNormalization(momentum=mom, epsilon=0.00001))
# Output Layer
model.add(Dense(num_classes, kernel_regularizer=l2(l2_reg)))
model.add(Activation('softmax'))
return model
mom = 0.9 #0
model = model_toy(mom)
model.summary()
model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.Adam(lr=0.001),
#optimizer=keras.optimizers.SGD(lr=0.01, momentum=0.9, decay=0.0, nesterov=True),
metrics=['accuracy'])
history = model.fit(x_train, y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
shuffle=True,
)
print('Training results')
print('-------------------------------------------')
score = model.evaluate(x_train, y_train, verbose=1)
print('Training loss:', score[0])
print('Training accuracy:', score[1])
print('-------------------------------------------')
Small Note - I was able to successfully do this task in PyTorch. It is just that my actual task requires me to have a Keras model. That's why I have changed the default values of the BatchNorm layer (the root cause of the issue) according to the ones I used to train PyTorch model.
Here is the image that I used in my code.
Here are the results of training.
Case1 of the dataset
Case2 of the dataset
If you look at these two files, you would be able to notice the discrepancies in the training loss during training vs inference.
(I have set my batch size to be equal to the size of my training data so as to avoid some the reasons BatchNorm generally creates problems as mentioned here)
Next, I looked at the source code of the Keras to see if there is any way I can make the BatchNorm layer use the batch statistics instead of the running mean and variance.
Here is the update formula that Keras (backend - TF) uses to update the running mean and variance.
#running_stat -= (1 - momentum) * (running_stat - batch_stat)
So if I set the momentum value to be 0, it would mean that the value assigned to the runing_stat would always be equal to batch_stat during the training phase. Thus, the value it will use during inference mode will also be same (close) as batch/dataset statistics.
Here are the results for this little experiment with the same issue still occurring.
Case1 of the dataset
Case2 of the dataset
Programming Environment - Python-3.5.2, tensorflow-1.10.0, keras-2.2.4
I tried the same thing with tensorflow-1.12.0, keras-2.2.2 as well but it still did not solve the issue.