Develop Reference

Accuracy and val_accuracy don't change while training

I tried to train my convolutional neural network using tensorflow and keras libraries. But the values of accuracy and val_accuracy didn't change the whole time. There is my neural network code:
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten, Conv2D, MaxPooling2D
import pickle
X = pickle.load(open("X.pickle", "rb"))
y = pickle.load(open("y.pickle", "rb"))
X = X/255.0
model = Sequential()
model.add(Conv2D(64, (3, 3), input_shape=X.shape[1:]))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(64, (3, 3), activation="relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(64, activation="relu"))
model.add(Dense(1, activation="sigmoid"))
model.compile(loss="binary_crossentropy",
optimizer="adam",
metrics=["accuracy"])
model.fit(X, y, batch_size=10, epochs=10, validation_split=0.1)
There is the creation of traning data, features and labels (X - features, y - labels)
def create_training_data():
for category in CATEGORIES:
path = os.path.join(DATADIR, category)
class_num = CATEGORIES.index(category)
for img in os.listdir(path):
try:
img_array = cv2.imread(os.path.join(path, img), cv2.IMREAD_GRAYSCALE)
new_array = cv2.resize(img_array, (IMG_SIZE, IMG_SIZE))
training_data.append([new_array, class_num])
except Exception as e:
pass
create_training_data()
random.shuffle(training_data)
X = []
y = []
for features, label in training_data:
X.append(features)
y.append(label)
X = np.array(X).reshape(-1, IMG_SIZE, IMG_SIZE, 1)
y = np.array(y)
And this is the log of training:
2023-01-15 00:36:42.368335: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Epoch 1/10
70/70 [==============================] - 45s 619ms/step - loss: 0.3039 - accuracy: 0.9627 - val_loss: 0.1211 - val_accuracy: 0.9744
Epoch 2/10
70/70 [==============================] - 42s 600ms/step - loss: 0.1524 - accuracy: 0.9670 - val_loss: 0.1189 - val_accuracy: 0.9744
Epoch 3/10
70/70 [==============================] - 42s 600ms/step - loss: 0.1537 - accuracy: 0.9670 - val_loss: 0.1622 - val_accuracy: 0.9744
Epoch 4/10
70/70 [==============================] - 44s 627ms/step - loss: 0.1563 - accuracy: 0.9670 - val_loss: 0.1464 - val_accuracy: 0.9744
Epoch 5/10
70/70 [==============================] - 42s 604ms/step - loss: 0.1591 - accuracy: 0.9670 - val_loss: 0.1185 - val_accuracy: 0.9744
Epoch 6/10
70/70 [==============================] - 42s 605ms/step - loss: 0.1511 - accuracy: 0.9670 - val_loss: 0.1338 - val_accuracy: 0.9744
Epoch 7/10
70/70 [==============================] - 49s 698ms/step - loss: 0.1623 - accuracy: 0.9670 - val_loss: 0.1188 - val_accuracy: 0.9744
Epoch 8/10
70/70 [==============================] - 50s 709ms/step - loss: 0.1480 - accuracy: 0.9670 - val_loss: 0.1397 - val_accuracy: 0.9744
Epoch 9/10
70/70 [==============================] - 45s 637ms/step - loss: 0.1508 - accuracy: 0.9670 - val_loss: 0.1203 - val_accuracy: 0.9744
Epoch 10/10
70/70 [==============================] - 47s 665ms/step - loss: 0.1716 - accuracy: 0.9670 - val_loss: 0.1238 - val_accuracy: 0.9744
Process finished with exit code 0
What should I do to fix this problem?

There are a couple potential reasons as to why you are facing this:
Your dataset is far too small. If your validation set is tiny, there is a high probability that your model will get the same % of predictions correct/incorrect
There is a great imbalance in your dataset. If one class heavily outweighs another, your model will favor the majority class, and predict it no matter what, as that is what brings the optimal accuracy for the model.
From what I see, there is nothing wrong with your code, rather modifications that need to be made to the dataset itself.

Hmm accuracy and validation accuracy are high even on the first epoch. Try using a lower learning rate in the Adam optimizer say .0002, On the first epoch pay attention to the loss and validation loss as the batches are process. It should start low and gradually increase during the epoch.

Bayesian optimization keras turner Train the model stays stuck with best hyperparameters

I am implementing Bayesian Optimization to find the best hyperparameters for my convolutional neural network (CNN).
After 10 trials, the algorithm found the best parameters with an accuracy of 80%. When I train the model with the best parameters found previously, the model stays stuck at the same accuracy value (0.5) and val_accu (0.5). I don't understand why.
My model builder is defined as below:
def model_builder(hp):
model = Sequential()
#model.add(Input(shape=(50,50,3)))
for i in range(hp.Int('num_blocks', 1,5)):
hp_padding=hp.Choice('padding_'+ str(i), values=['valid', 'same'])
hp_filters=hp.Choice('filters_'+ str(i), values=[32, 64])
model.add(Conv2D(hp_filters, (3, 3), padding=hp_padding, activation='relu', kernel_initializer='he_uniform', input_shape=(50, 50, 3)))
model.add(MaxPooling2D((2, 2)))
model.add(Dropout(hp.Choice('dropout_'+ str(i), values=[0.0, 0.1, 0.2])))
model.add(Flatten())
hp_units = hp.Int('units', min_value=25, max_value=150, step=25)
model.add(Dense(hp_units, activation='relu', kernel_initializer='he_uniform'))
model.add(Dense(2,activation="sigmoid"))
hp_learning_rate = hp.Choice('learning_rate', values=[1e-2, 1e-3])
hp_optimizer=hp.Choice('Optimizer', values=['Adam', 'SGD'])
#hp_optimizer=hp.Choice('Optimizer', values=['Adam'])
if hp_optimizer == 'Adam':
hp_learning_rate = hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-5])
elif hp_optimizer == 'SGD':
hp_learning_rate = hp.Choice('learning_rate', values=[1e-2, 1e-3])
nesterov=True
momentum=0.9
#model.compile(loss=keras.losses.binary_crossentropy, optimizer=tf.keras.optimizers.Adam(learning_rate=hp_learning_rate), metrics=['accuracy'])
model.compile(optimizer=hp_optimizer,loss=keras.losses.binary_crossentropy,metrics=['accuracy'])
return model
To search the best hyperparameters:
tuner_cnn = kt.tuners.BayesianOptimization(
model_builder,
objective='accuracy',
max_trials=10,
directory='.',
project_name='tuning-cnn')
I launch the instance as the following:
tuner_cnn.search(datagen.flow(X_trainRusReshaped, Y_trainRusHot,batch_size=256), steps_per_epoch=len(X_trainRusReshaped) / batch_size, epochs=80, validation_data=(X_testRusReshaped, Y_testRusHot), callbacks=[stop_early])
and after 10 trials, the algorithm shows me this:
best_mlp_hyperparameters = tuner_cnn.get_best_hyperparameters(1)[0]
print("Best Hyper-parameters")
best_mlp_hyperparameters.values
output:
Best Hyper-parameters
{'Optimizer': 'Adam',
'dropout_0': 0.2,
'filters_0': 64,
'learning_rate': 0.001,
'num_blocks': 3,
'padding_0': 'valid',
'units': 100}
Finally, I train our CNN model using the best hyperparameters
model_cnn = Sequential()
for i in range(best_mlp_hyperparameters['num_blocks']):
hp_padding=best_mlp_hyperparameters['padding_'+ str(i)]
hp_filters=best_mlp_hyperparameters['filters_'+ str(i)]
model_cnn.add(Conv2D(hp_filters, (3, 3), padding=hp_padding, activation='relu', kernel_initializer='he_uniform', input_shape=(50, 50, 3)))
model_cnn.add(MaxPooling2D((2, 2)))
model_cnn.add(Dropout(best_mlp_hyperparameters['dropout_'+ str(i)]))
model_cnn.add(Flatten())
model_cnn.add(Dense(best_mlp_hyperparameters['units'], activation='relu', kernel_initializer='he_uniform'))
model_cnn.add(Dense(2,activation="sigmoid"))
model_cnn.compile(optimizer=best_mlp_hyperparameters['Optimizer'],
loss='binary_crossentropy',
metrics=['accuracy'])
print(model_cnn.summary())
#history_cnn= model_cnn.fit(train_x, train_y, epochs=50, batch_size=32, validation_data=(dev_x, dev_y), callbacks=callback)
training=model_cnn.fit_generator(datagen.flow(X_trainRusReshaped,Y_trainRusHot,batch_size=batch_size),steps_per_epoch=len(X_trainRusReshaped) / batch_size, epochs=epochs,validation_data=(X_testRusReshaped, Y_testRusHot), verbose=1, callbacks=[stop_early])
When I am training I am far from the values obtained previously:
119/119 [==============================] - 25s 201ms/step - loss: 14.8185 - accuracy: 0.4982 - val_loss: 0.6932 - val_accuracy: 0.5000
Epoch 2/80
119/119 [==============================] - 21s 174ms/step - loss: 0.7104 - accuracy: 0.4972 - val_loss: 0.6932 - val_accuracy: 0.5000
Epoch 3/80
119/119 [==============================] - 21s 173ms/step - loss: 0.7013 - accuracy: 0.4982 - val_loss: 0.6932 - val_accuracy: 0.5000
Epoch 4/80
119/119 [==============================] - 21s 173ms/step - loss: 0.6977 - accuracy: 0.4990 - val_loss: 0.6932 - val_accuracy: 0.5000
Epoch 5/80
119/119 [==============================] - 21s 174ms/step - loss: 0.6960 - accuracy: 0.4996 - val_loss: 0.6932 - val_accuracy: 0.5000
Epoch 6/80
119/119 [==============================] - 21s 174ms/step - loss: 0.6949 - accuracy: 0.5000 - val_loss: 0.6932 - val_accuracy: 0.5000
Any idea?

Training and Validation Acuracy is not improving using datagen.flow_from_directory in google colab

The CNN model will classify between two class with training samples = 5974 and validation = 1987.
I am using datagen.flow_from_directory and my model will predict from separate test set. I am running the code For 200 epoch in Google Colab, but after 5 epoch, the training and validation accuracy is not improving.
Accuracy
Epoch 45/200
186/186 [==============================] - 138s 744ms/step - loss: 0.6931 - acc: 0.4983 - val_loss: 0.6931 - val_acc: 0.5000
Epoch 46/200
186/186 [==============================] - 137s 737ms/step - loss: 0.6931 - acc: 0.4990 - val_loss: 0.6931 - val_acc: 0.5000
Epoch 47/200
186/186 [==============================] - 142s 761ms/step - loss: 0.6931 - acc: 0.4987 - val_loss: 0.6931 - val_acc: 0.5000
Epoch 48/200
186/186 [==============================] - 140s 752ms/step - loss: 0.6931 - acc: 0.4993 - val_loss: 0.6931 - val_acc: 0.5005
Epoch 49/200
186/186 [==============================] - 139s 745ms/step - loss: 0.6931 - acc: 0.4976 - val_loss: 0.6931 - val_acc: 0.5010
Epoch 50/200
186/186 [==============================] - 143s 768ms/step - loss: 0.6931 - acc: 0.4992 - val_loss: 0.6931 - val_acc: 0.5000
Epoch 51/200
186/186 [==============================] - 140s 755ms/step - loss: 0.6931 - acc: 0.4980 - val_loss: 0.6931 - val_acc: 0.5000
Epoch 52/200
186/186 [==============================] - 141s 758ms/step - loss: 0.6931 - acc: 0.4990 - val_loss: 0.6931 - val_acc: 0.4995
Epoch 53/200
186/186 [==============================] - 141s 759ms/step - loss: 0.6931 - acc: 0.4985 - val_loss: 0.6931 - val_acc: 0.5000
Epoch 54/200
186/186 [==============================] - 143s 771ms/step - loss: 0.6931 - acc: 0.4987 - val_loss: 0.6931 - val_acc: 0.4995
Epoch 55/200
186/186 [==============================] - 143s 771ms/step - loss: 0.6931 - acc: 0.4992 - val_loss: 0.6931 - val_acc: 0.5005
train_data_path = "/content/drive/My Drive/snk_tod/train"
valid_data_path = "/content/drive/My Drive/snk_tod/valid"
test_data_path = "/content/drive/My Drive/snk_tod/test"
img_rows = 100
img_cols = 100
epochs = 200
print(epochs)
batch_size = 32
num_of_train_samples = 5974
num_of_valid_samples = 1987
#Image Generator
train_datagen = ImageDataGenerator(rescale=1. / 255,
rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest')
valid_datagen = ImageDataGenerator(rescale=1. / 255)
test_datagen = ImageDataGenerator(rescale=1. / 255)
train_generator = train_datagen.flow_from_directory(train_data_path,
target_size=(img_rows, img_cols),
batch_size=batch_size,
shuffle=True,
class_mode='categorical')
validation_generator = valid_datagen.flow_from_directory(valid_data_path,
target_size=(img_rows, img_cols),
batch_size=batch_size,
shuffle=True,
class_mode='categorical')
test_generator = test_datagen.flow_from_directory(test_data_path,
target_size=(img_rows, img_cols),
batch_size=batch_size,
shuffle=False,
class_mode='categorical')
model = Sequential()
model.add(Conv2D((32), (3, 3), input_shape=(img_rows, img_cols, 3), kernel_initializer="glorot_uniform", bias_initializer="zeros"))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.5))
model.add(Conv2D((32), (3, 3),kernel_initializer="glorot_uniform", bias_initializer="zeros"))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.5))
model.add(Conv2D((64), (3, 3),kernel_initializer="glorot_uniform", bias_initializer="zeros"))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.5))
model.add(Conv2D((64), (3, 3),kernel_initializer="glorot_uniform", bias_initializer="zeros"))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.5))
model.add(Flatten()) # this converts our 3D feature maps to 1D feature vectors
model.add(Dropout(0.5))
model.add(Dense(512))
model.add(Dense(2))
model.add(Activation('sigmoid'))
model.compile(loss='categorical_crossentropy',
optimizer='adam',
metrics=['acc'])
#Train
history=model.fit_generator(train_generator,
steps_per_epoch=num_of_train_samples // batch_size,
epochs=epochs,
validation_data=validation_generator,
validation_steps=num_of_valid_samples // batch_size)

Tensorflow model validation accuracy not increasing

I have built a tensorflow model and am getting no change in my validation accuracy in different epochs, which makes me believe there is something wrong in my setup. Below is my code.
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense
from keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras import regularizers
import tensorflow as tf
model = Sequential()
model.add(Conv2D(16, (3, 3), input_shape=(299, 299,3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))
model.add(Conv2D(32, (3, 3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))
model.add(Conv2D(64, (3, 3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))
model.add(Conv2D(64, (3, 3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))
# this converts our 3D feature maps to 1D feature vectors
model.add(Flatten())
model.add(Dense(512))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])
batch_size=32
# this is the augmentation configuration we will use for training
train_datagen = ImageDataGenerator(
rescale=1./255,
# shear_range=0.2,
# zoom_range=0.2,
horizontal_flip=True)
# this is the augmentation configuration we will use for testing:
# only rescaling
test_datagen = ImageDataGenerator(rescale=1./255)
# this is a generator that will read pictures found in
# subfolers of 'data/train', and indefinitely generate
# batches of augmented image data
train_generator = train_datagen.flow_from_directory(
'Documents/Training', # this is the target directory
target_size=(299, 299), #all images will be resized to 299
batch_size=batch_size,
class_mode='binary') # since we use binary_crossentropy loss, we need binary labels
# this is a similar generator, for validation data
validation_generator = test_datagen.flow_from_directory(
'Documents/Dev',
target_size=(299, 299),
batch_size=batch_size,
class_mode='binary')
#w1 = tf.Variable(tf.truncated_normal([784, 30], stddev=0.1))
model.fit_generator(
train_generator,
steps_per_epoch=50 // batch_size,
verbose = 1,
epochs=10,
validation_data=validation_generator,
validation_steps=8 // batch_size)
Which when I run produces the following output. Anything I'm missing here as far as my architecture is concerned or data generation steps? I have referenced Tensorflow model accuracy not increasing and accuracy not increasing in tensorflow model to no avail yet.
Epoch 1/10
3/3 [==============================] - 2s 593ms/step - loss: 0.6719 - accuracy: 0.6250 - val_loss: 0.8198 - val_accuracy: 0.5000
Epoch 2/10
3/3 [==============================] - 2s 607ms/step - loss: 0.6521 - accuracy: 0.6667 - val_loss: 0.8518 - val_accuracy: 0.5000
Epoch 3/10
3/3 [==============================] - 2s 609ms/step - loss: 0.6752 - accuracy: 0.6250 - val_loss: 0.7129 - val_accuracy: 0.5000
Epoch 4/10
3/3 [==============================] - 2s 611ms/step - loss: 0.6841 - accuracy: 0.6250 - val_loss: 0.7010 - val_accuracy: 0.5000
Epoch 5/10
3/3 [==============================] - 2s 608ms/step - loss: 0.6977 - accuracy: 0.5417 - val_loss: 0.6551 - val_accuracy: 0.5000
Epoch 6/10
3/3 [==============================] - 2s 607ms/step - loss: 0.6508 - accuracy: 0.7083 - val_loss: 0.5752 - val_accuracy: 0.5000
Epoch 7/10
3/3 [==============================] - 2s 615ms/step - loss: 0.6596 - accuracy: 0.6875 - val_loss: 0.9326 - val_accuracy: 0.5000
Epoch 8/10
3/3 [==============================] - 2s 604ms/step - loss: 0.7022 - accuracy: 0.6458 - val_loss: 0.6976 - val_accuracy: 0.5000
Epoch 9/10
3/3 [==============================] - 2s 591ms/step - loss: 0.6331 - accuracy: 0.7292 - val_loss: 0.9571 - val_accuracy: 0.5000
Epoch 10/10
3/3 [==============================] - 2s 595ms/step - loss: 0.6085 - accuracy: 0.7292 - val_loss: 0.6029 - val_accuracy: 0.5000
Out[24]: <keras.callbacks.callbacks.History at 0x1ee4e3a8f08>

You are setting the training steps per epoch =50//32=1. So do you only have 50 training images? Similarly for validation you have steps = 8//32=0. Do you have only 8 validation images? When you execute the program how many images do the training and validation generators print out they have found? You will need more images than that. Try setting your batch size =1

Loss not decrasing and is very high keras

I'm learning deep learning in keras and I have a problem.
The loss isn't decreasing and it's very high, about 650.
I'm working on MNIST dataset from tensorflow.keras.datasets.mnist
There is no error, just my NN isn't learning.
There is my model:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Flatten
import tensorflow.nn as tfnn
inputdim = 28 * 28
model = Sequential()
model.add(Flatten())
model.add(Dense(inputdim, activation = tfnn.relu))
model.add(Dense(128, activation = tfnn.relu))
model.add(Dense(10, activation = tfnn.softmax))
model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
model.fit(X_train, Y_train, epochs = 4)
and my output:
Epoch 1/4
60000/60000 [==============================] - 32s 527us/sample - loss: 646.0926 - acc: 6.6667e-05
Epoch 2/4
60000/60000 [==============================] - 39s 652us/sample - loss: 646.1003 - acc: 0.0000e+00 - l - ETA: 0s - loss: 646.0983 - acc: 0.0000e
Epoch 3/4
60000/60000 [==============================] - 35s 590us/sample - loss: 646.1003 - acc: 0.0000e+00
Epoch 4/4
60000/60000 [==============================] - 33s 544us/sample - loss: 646.1003 - acc: 0.0000e+00
```

Ok, I added BatchNormalization between lines and changed loss function to 'sparse_categorical_crossentropy'. That's how my NN looks like:
model = Sequential()
model.add(Flatten())
model.add(BatchNormalization(axis = 1, momentum = 0.99))
model.add(Dense(inputdim, activation = tfnn.relu))
model.add(BatchNormalization(axis = 1, momentum = 0.99))
model.add(Dense(128, activation = tfnn.relu))
model.add(BatchNormalization(axis = 1, momentum = 0.99))
model.add(Dense(10, activation = tfnn.softmax))
model.compile(loss = 'sparse_categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
and thats a results:
Epoch 1/4
60000/60000 [==============================] - 68s 1ms/sample - loss: 0.2045 - acc: 0.9374
Epoch 2/4
60000/60000 [==============================] - 55s 916us/sample - loss: 0.1007 - acc: 0.9689
Thanks for your help!

You may try sparse_categorical_crossentropy loss function. Also what is your batch size? and as has already been suggested you may want to increase number of epochs.