Develop Reference

Validation Accuracy Stuck, Accuracy low

I want to create a machine learning model with Tensorflow which detects flowers. I went in the nature and took pictures of 4 different species (~600 per class, one class got 700).
I load these images with Tensorflow Train Generator:
train_datagen = ImageDataGenerator(rescale=1./255,
shear_range=0.2,
zoom_range=0.15,
brightness_range=[0.7, 1.4],
fill_mode='nearest',
vertical_flip=True,
horizontal_flip=True,
rotation_range=15,
width_shift_range=0.1,
height_shift_range=0.1,
validation_split=0.2)
train_generator = train_datagen.flow_from_directory(
pfad,
target_size=(imageShape[0],imageShape[1]),
batch_size=batchSize,
class_mode='categorical',
subset='training',
seed=1,
shuffle=False,
#save_to_dir=r'G:\test'
)
validation_generator = train_datagen.flow_from_directory(
pfad,
target_size=(imageShape[0],imageShape[1]),
batch_size=batchSize,
shuffle=False,
seed=1,
class_mode='categorical',
subset='validation')
Then I am creating a simple model looking like this:
model = tf.keras.Sequential([
keras.layers.Conv2D(128, (3,3), activation='relu', input_shape=(imageShape[0], imageShape[1],3)),
keras.layers.MaxPooling2D(2,2),
keras.layers.Dropout(0.5),
keras.layers.Conv2D(256, (3,3), activation='relu'),
keras.layers.MaxPooling2D(2,2),
keras.layers.Conv2D(512, (3,3), activation='relu'),
keras.layers.MaxPooling2D(2,2),
keras.layers.Flatten(),
keras.layers.Dense(280, activation='relu'),
keras.layers.Dense(4, activation='softmax')
])
opt = tf.keras.optimizers.SGD(learning_rate=0.001,decay=1e-5)
model.compile(loss='categorical_crossentropy',
optimizer= opt,
metrics=['accuracy'])
And want to start the training process (CPU):
history=model.fit(
train_generator,
steps_per_epoch = train_generator.samples // batchSize,
validation_data = validation_generator,
validation_steps = validation_generator.samples // batchSize,
epochs = 200,callbacks=[checkpoint,early,tensorboard],workers=-1)
The result should be that my validation Accuracy improves, but it starts with 0.3375 and stays at this level the whole training process. Validation loss (1.3737) decreases by 0.001. Accuracy start with 0.15 but increases.
Why is my validation accuracy stuck?
Am I using the right loss? Or do I build my model wrong? Is my Tensorflow Train Generator hot encoding the labels?
Thanks

I solved the problem by using RMSprop() without any parameters.
So I changed from:
opt = tf.keras.optimizers.SGD(learning_rate=0.001,decay=1e-5)
model.compile(loss='categorical_crossentropy',optimizer= opt, metrics=['accuracy'])
to:
opt = tf.keras.optimizers.RMSprop()
model.compile(loss='categorical_crossentropy',
optimizer= opt,
metrics=['accuracy'])

This is a similar example, except that for 4 categorical classes, the below is binary. You may want to change the loss to categorical cross entropy, class_mode from binary to categorical in the train and test generators and final dense layer activation to softmax. I am still able to use model.fit_generator()
image_dataGen = ImageDataGenerator(rotation_range=20,
width_shift_range=0.2,height_shift_range=0.2,shear_range=0.1,
zoom_range=0.1,fill_mode='nearest',horizontal_flip=True,
vertical_flip=True,rescale=1/255)
train_images = image_dataGen.flow_from_directory(train_path,target_size = image_shape[:2],
color_mode = 'rgb',class_mode = 'binary')
test_images = image_dataGen.flow_from_directory(test_path,target_size = image_shape[:2],
color_mode = 'rgb',class_mode = 'binary',
shuffle = False)
model = Sequential()
model.add(Conv2D(filters = 32, kernel_size = (3,3),input_shape = image_shape,activation = 'relu'))
model.add(MaxPool2D(pool_size = (2,2)))
model.add(Conv2D(filters = 48, kernel_size = (3,3),input_shape = image_shape,activation = 'relu'))
model.add(MaxPool2D(pool_size = (2,2)))
model.add(Flatten())
model.add(Dense(units = 128,activation = 'relu'))
model.add(Dropout(0.5))
model.add(Dense(units = 1, activation = 'sigmoid'))
model.compile(loss = 'binary_crossentropy',metrics = ['accuracy'], optimizer = 'adam')
results = model.fit_generator(train_images, epochs = 10, callbacks = [early_stop],
validation_data = test_images)

Maybe your learning rate is too high.
Use learning rate = 0.000001 and if that does not work then try another optimizer like Adam.

use model.fit_generator() instead of model.fit() Also below points could be helpful.
In order to use .flow_from_directory, you must organize the images in sub-directories. This is an absolute requirement, otherwise the method won't work. The directories should only contain images of one class, so one folder per class of images. Also could you check if the path for the training data and test data is correct ? They cannot point to the same location. I have used the ImageGenerator class for classification problem. You can also try changing the optimizer to 'Adam'
Structure Needed:
Image Data Folder
Class 1
0.jpg
1.jpg
...
Class 2
0.jpg
1.jpg
...
...
Class n

How can i predict images(outside dataset) using VGG16 with keras after training?

I have trained the VGG16 net using keras with my own dataset, which has 10 classes. so i modified the activation layer with 10 classes.
Here is the code
TRAIN_DIR = "D:\\Dataset\\training"
VALIDATION_DIR = "D:\\Dataset\\validation"
part 2
from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img
IMAGE_WIDTH = 300
IMAGE_HEIGHT = 300
BATCH_SIZE = 16
part 3
train_datagen = ImageDataGenerator(rescale=1./255,
rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest'
)
validation_datagen = ImageDataGenerator(rescale=1./255,
)
train_generator = train_datagen.flow_from_directory(TRAIN_DIR,
target_size=(IMAGE_WIDTH, IMAGE_HEIGHT),
batch_size = BATCH_SIZE,
shuffle=True, # By shuffling the images we add some randomness and prevent overfitting
class_mode="categorical")
validation_generator = validation_datagen.flow_from_directory(VALIDATION_DIR,
target_size=(IMAGE_WIDTH, IMAGE_HEIGHT),
batch_size = BATCH_SIZE,
shuffle=True,
class_mode="categorical")
part 4
training_samples = 1097
validation_samples = 272
total_steps = training_samples // BATCH_SIZE
loading the VGG16
#VGG16 network with pretrained weights is used
from keras.applications import vgg16
model = vgg16.VGG16(weights='imagenet', include_top=False, input_shape=(IMAGE_WIDTH, IMAGE_HEIGHT, 3), pooling="max")
for layer in model.layers[:-5]:
layer.trainable = False
for layer in model.layers:
print(layer, layer.trainable)
part 5
from keras.layers import Dense, GlobalAveragePooling2D, Dropout
from keras.models import Model, Sequential
# Although this part can be done also with the functional API, I found that for this simple models, this becomes more intuitive
transfer_model = Sequential()
for layer in model.layers:
transfer_model.add(layer)
transfer_model.add(Dense(512, activation="relu"))
transfer_model.add(Dropout(0.5))
transfer_model.add(Dense(10, activation="softmax"))
part 6
# Adam optimizer and learning rate 0.0001
from keras import optimizers
adam = optimizers.Adam(lr=0.0001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.00001)
transfer_model.compile(loss="categorical_crossentropy",
optimizer=adam,
metrics=["accuracy"])
finally the training
model_history = transfer_model.fit_generator(train_generator, steps_per_epoch=training_samples // BATCH_SIZE,
epochs=25,
validation_data=validation_generator,
validation_steps=validation_samples // BATCH_SIZE)
part 7, using some random images from the internet to predict
test_path = "D:\\Dataset\\predict\\"
test_datagen = ImageDataGenerator(rescale=1./255,
rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest'
)
test_generator = test_datagen.flow_from_directory(test_path,
target_size=(IMAGE_WIDTH, IMAGE_HEIGHT),
batch_size = 50,
class_mode="categorical")
enter code here
in this part i am trying to predict, but getting this kind of numbers without getting the actual prediction result which i want as images
pred = model.predict_generator(test_generator, steps=1)
print(pred)
the result is like this, but i want these to be real images, but cannot figure out how.

You can't output images from your network and it's not really clear how you imagine this would work - the images are the input and the output is a list of numbers with one value per class. You can interpret these numbers as the probability the image contains an object of that class.
You could then find the most likely class (using e.g. the argmax function) and show an image from this class if you liked - but this would have to be done separately.
Note - you are running the prediction using the original model not the transfer_model:
pred = model.predict_generator(test_generator, steps=1)
You should use the trained transfer model to get the class predictions and this will be in the form of a vector of 10 probabilities, one value for each class.

Why is the model not learning with pretrained vgg16 in keras?

I am using the pre-trained VGG 16 model available with Keras and applying it on the SVHN dataset which is a dataset of 10 classes of number 0 - 10. The network is not learning and has been stuck at 0.17 accuracy. There is something that I am doing incorrectly but I am unable to recognise it. The way I am running my training is as follows:
import tensorflow.keras as keras
## DEFINE THE MODEL ##
vgg16 = keras.applications.vgg16.VGG16()
model = keras.Sequential()
for layer in vgg16.layers:
model.add(layer)
model.layers.pop()
for layer in model.layers:
layer.trainable = False
model.add(keras.layers.Dense(10, activation = "softmax"))
## START THE TRAINING ##
train_optimizer_rmsProp = keras.optimizers.RMSprop(lr=0.0001)
model.compile(loss="categorical_crossentropy", optimizer=train_optimizer_rmsProp, metrics=['accuracy'])
batch_size = 128*1
data_generator = keras.preprocessing.image.ImageDataGenerator(
rescale = 1./255
)
train_generator = data_generator.flow_from_directory(
'training',
target_size=(224, 224),
batch_size=batch_size,
color_mode='rgb',
class_mode='categorical'
)
validation_generator = data_generator.flow_from_directory(
'validate',
target_size=(224, 224),
batch_size=batch_size,
color_mode='rgb',
class_mode='categorical')
history = model.fit_generator(
train_generator,
validation_data = validation_generator,
validation_steps = math.ceil(val_split_length / batch_size),
epochs = 15,
steps_per_epoch = math.ceil(num_train_samples / batch_size),
use_multiprocessing = True,
workers = 8,
callbacks = model_callbacks,
verbose = 2
)
What is it that I am doing wrong? Is there something that I am missing? I was expecting a very high accuracy since it is carrying weights from imagenet but it is stuck at 0.17 accuracy from the first epoch.

I assume you're upsampling the 32x32 MNIST-like images to fit the VGG16 input, what you should actually do in this case is to remove all the dense layers, this way you can input any image size as in convolutional layers the weights are agnostic to the image size.
You can do this like:
vgg16 = keras.applications.vgg16.VGG16(include_top=False, input_shape=(32, 32))
Which I consider should be the default behaviour of the constructor.
When you upsample the image, best case scenario you're basically blurring it, in this case you have to consider that a single pixel of the original image corresponds to 7 pixels of the upsampled one, while VGG16's filters are 3 pixels wide, so in other words you're losing the image's features.
It is not necessary to add 3 dense layers at the end like the original VGG16, you can try with the same layer you have in your code.

Keras prints no output, high memory and CPU usage and GPU is not used when using tensorboard Callback

I have a wierd situation in Keras and it really freaks me out.
I am trying to train a CNN using pretrained Inception with additional convolution, global average pool and dense layers. I am using a ImageDataGenerator to load the data.
The data generator is working fine, I have tested that. The model compiles well also. But when I run fit_generator, no output is printed, the CPU is at 100% and memory starts filling up slowly until it overflows. And although I have a GPU and have worked with it in tensorflow (which is the backend here) a number of times, it is completely ignored by Keras.
Considering that maybe batch size could be a problem, I set it to 1 but it did not solve the issue. The images are of size 299x299, which is not that big anyway.
I will post the code below as a reference though it seems to me that nothing is wrong with it:
def get_datagen():
return ImageDataGenerator(rotation_range=30,
width_shift_range=0.2,
height_shift_range=0.2,
horizontal_flip=True,
fill_mode='nearest'
)
# Setup and compile the model.
model = InceptionV3(include_top=False, input_shape=(None, None, 3))
# Set the model layers to be untrainable
for layer in model.layers:
layer.trainable = False
x = model.output
x = Conv2D(120, 5, activation='relu')(x)
x = GlobalAveragePooling2D()(x)
predictions = Activation('softmax')(x)
model_final = Model(inputs=model.inputs, outputs=predictions)
model_final.compile(optimizer='adam', loss='categorical_crossentropy',metrics=['accuracy'])
# Define the dataflow.
train_gen = get_datagen()
val_test_gen = get_datagen()
train_data = train_gen.flow_from_directory(train_folder, target_size=(299, 299), batch_size=1)
val_data = val_test_gen.flow_from_directory(validation_folder, target_size=(299, 299), batch_size=1)
test_data = val_test_gen.flow_from_directory(test_folder, target_size=(299, 299), batch_size=1)
train_size = train_data.n
val_size = val_data.n
test_size = test_data.n
# Define callbacks.
model_checkpoint = ModelCheckpoint('../models/dbc1/', monitor='val_accuracy', verbose=1, save_best_only=True)
early_stopping = EarlyStopping(monitor='val_accuracy', patience=3, verbose=1, mode='max')
tensorboard = TensorBoard(log_dir='../log/dbc1', histogram_freq=1, write_grads=True, )
model_final.fit_generator(train_data, steps_per_epoch=1, epochs=100,
callbacks=[model_checkpoint, early_stopping, tensorboard],
validation_data=val_data, verbose=1)
EDIT
It seems the tensorboard callback was the problem here. When I remove it, everything works. Does anyone know why this is happening?

There's seems to be a problem (possibly related to keras#3358) when using the histogram_freq=1 under certain conditions.
You could try to set histogram_freq=0 and submit an issue at keras repository. You wouldn't have the gradient histograms, but at least you would be able to train:
model.fit(...,
callbacks=[
TensorBoard(log_dir='./logs/', batch_size=batch_size),
...
])
I notice that this problem doesn't happen with all trained models. If InceptionV3 usage is not a requirement, I recommend you switching to another model. So far, I found that the following code (adapted from yours, using VGG19) works on keras==2.1.2, tensorflow==1.4.1:
from keras.applications import VGG19
from keras.applications.vgg19 import preprocess_input
input_shape = (224, 224, 3)
batch_size = 1
model = VGG19(include_top=False, input_shape=input_shape)
for layer in model.layers:
layer.trainable = False
x, y = model.input, model.output
y = Conv2D(2, 5, activation='relu')(y)
y = GlobalAveragePooling2D()(y)
y = Activation('softmax')(y)
model = Model(inputs=model.inputs, outputs=y)
model.compile('adam',
loss='categorical_crossentropy',
metrics=['accuracy'])
g = ImageDataGenerator(rotation_range=30,
width_shift_range=0.2,
height_shift_range=0.2,
horizontal_flip=True,
preprocessing_function=preprocess_input)
train_data = g.flow_from_directory(train_folder,
target_size=input_shape[:2],
batch_size=batch_size)
val_data = g.flow_from_directory(validation_folder,
target_size=input_shape[:2],
batch_size=batch_size)
test_data = g.flow_from_directory(test_folder,
target_size=input_shape[:2],
batch_size=batch_size)
model.fit_generator(train_data, steps_per_epoch=1, epochs=100,
validation_data=val_data, verbose=1,
callbacks=[
ModelCheckpoint('./ckpt.hdf5',
monitor='val_accuracy',
verbose=1,
save_best_only=True),
EarlyStopping(patience=3, verbose=1),
TensorBoard(log_dir='./logs/',
batch_size=batch_size,
histogram_freq=1,
write_grads=True)])

Keras Merge_layer load inputs directly from folders with datagenerator's flow from directory method.

Is there a way to load data from directory into the merge layer with 2 input tensors using datagenerator with flow_from_directory.
x = merge([base_x, base_y], mode='concat', concat_axis=1)
x = Dense(256, activation='relu', name="fc-1")(x)
x = Dropout(0.5)(x)
predictions = Dense(2, activation='softmax', name='predictions')(x)
model = Model(input=[base_x.input, base_y.input], output=predictions)
# note that it is necessary to start with a fully-trained
# classifier, including the top classifier,
# in order to successfully do fine-tuning
# add the model on top of the convolutional base
print('Model loaded.')
# compile the model with a SGD/momentum optimizer
# and a very slow learning rate.
model.compile(loss="categorical_crossentropy",
optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
metrics=['accuracy'])
model.summary()
# prepare data augmentation configuration
train_datagen = ImageDataGenerator(
rescale=1. / 255,
)
test_datagen = ImageDataGenerator(rescale=1. / 255)
train_generator = train_datagen.flow_from_directory(
train_data_dir,
target_size=(img_height, img_width),
batch_size=32,
class_mode='categorical')
validation_generator = test_datagen.flow_from_directory(
validation_data_dir,
target_size=(img_height, img_width),
batch_size=32,
class_mode='categorical')
model.fit_generator(
train_generator,
samples_per_epoch=train_generator.nb_sample,
nb_epoch=150,
validation_data=validation_generator,
nb_val_samples=3000)
The structure of the folders are
train_dir --
--> category 1
--> category 2
Test_dir --
--> category 1
--> category 2
I want to run this model reading the data directly from the directory. But the exception I was getting is,
ValueError: The model expects 2 input arrays, but only received one array. Found: array with shape (32, 224, 224, 3
How to add 2 input arrays directly from the directory .Doesn't want to convert the input a Numpy array .I input should be read from the directory itself.Is there a way to do this ?