Develop Reference

Confusion matrix not making any sense after training a 3 class image classification model

I'm a rookie at machine learning so please bear with me.
I have a model that trains images and classifies them in 3 different classes. I'm trying to get the confusion matrix for the test data, but either I don't understand it or it's not making any sense. When the model is done training after 200 epochs it shows that the accuracy is around 65%:
loss: 1.5386 - accuracy: 0.6583
But then when the confusion matrix is printed like this:
[[23 51 42]
[20 27 25]
[47 69 56]]
Which isn't correct because the "true" results (23, 27 and 56) don't make up for 65% of all the results. If you add up the numbers in the matrix it adds to the amount of test images so I know that part is correct.
I had a tensorflow warning just before printing the confusion matrix that says the following, but I don't really get its meaning:
WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 13 batches). You may need to use the repeat() function when building your dataset
This is my code:
import sys
from matplotlib import pyplot
import numpy as np
from tensorflow.keras.applications.vgg16 import VGG16
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Conv2D, Flatten, Dense, MaxPooling2D, Dropout, GlobalAveragePooling2D
from tensorflow.keras.optimizers import SGD, RMSprop, Adam
from keras.preprocessing.image import ImageDataGenerator
from sklearn.metrics import confusion_matrix
CLASSES = 3
# define cnn model
def define_model():
# load model
base_model = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
#add new classifier layers
x = base_model.output
x = Dropout(0.4)(x)
x = GlobalAveragePooling2D(name='avg_pool')(x)
predictions = Dense(CLASSES, activation='softmax')(x)
model = Model(inputs=base_model.input, outputs=predictions)
# mark loaded layers as not trainable
for layer in base_model.layers:
layer.trainable = False
# compile model
opt = RMSprop(lr=0.0001)
#momentum=0.9)
model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
return model
# plot diagnostic learning curves
def summarize_diagnostics(history):
# plot loss
pyplot.subplot(211)
pyplot.title('Cross Entropy Loss')
pyplot.plot(history.history['loss'], color='blue', label='train')
pyplot.plot(history.history['val_loss'], color='orange', label='test')
# plot accuracy
pyplot.subplot(212)
pyplot.title('Classification Accuracy')
pyplot.plot(history.history['accuracy'], color='blue', label='train')
pyplot.plot(history.history['val_accuracy'], color='orange', label='test')
# save plot to file
filename = sys.argv[0].split('/')[-1]
pyplot.savefig(filename + '_plot.png')
pyplot.close()
# run the test harness for evaluating a model
def run_test_harness():
# define model
model = define_model()
# create data generator
datagen = ImageDataGenerator(featurewise_center=True)
# specify imagenet mean values for centering
datagen.mean = [123.68, 116.779, 103.939]
# prepare iterators
train_it = datagen.flow_from_directory('../datasetMainPruebas3ClasesQuitandoConfusosCV1/train/',
class_mode='categorical', batch_size=32, target_size=(224, 224))
test_it = datagen.flow_from_directory('../datasetMainPruebas3ClasesQuitandoConfusosCV1/test/',
class_mode='categorical', batch_size=32, target_size=(224, 224))
# fit model
history = model.fit_generator(train_it, steps_per_epoch=len(train_it),
validation_data=test_it, validation_steps=len(test_it), epochs=200, verbose=1)
# evaluate model
_, acc = model.evaluate_generator(test_it, steps=len(test_it), verbose=1)
print('> %.3f' % (acc * 100.0))
#confusion matrix
Y_pred = model.predict_generator(test_it, len(test_it) + 1)
y_pred = np.argmax(Y_pred, axis=1)
print('Confusion Matrix')
cm = confusion_matrix(test_it.classes, y_pred)
print(cm)
# learning curves
summarize_diagnostics(history)
# entry point, run the test harness
run_test_harness()
Any help or tip is welcome, thank you
Edit: after checking one of the comments I changed my CM code to the following:
#confusion matrix
all_y_pred = []
all_y_true = []
for i in range(len(test_it)):
x, y = test_it[i]
y_pred = model.predict(x)
all_y_pred.append(y_pred)
all_y_true.append(y)
all_y_pred = np.concatenate(all_y_pred, axis=0)
all_y_true = np.concatenate(all_y_true, axis=0)
print('Confusion Matrix')
cm = confusion_matrix(all_y_true, all_y_pred)
print(cm)
And now the error I get says "Classification metrics can't handle a mix of multilabel-indicator and continuous-multioutput targets"
Any idea why?

How can I match the encoder decoder dimension of deep auto encoder?

I'm coding very simple deep auto encoder with MNIST data set which have three latent space layer.
However, there are problems with the encoder and decoder dimensions.
The exact error message is : ValueError: Error when checking input: expected input_2 to have shape (128,) but got array with shape (32,) at line 60.
(line 60 : decoded_imgs = decoder.predict(encoded_imgs))
and I don't know how to solve it. I will attach my full code below.
Please help. Thanks.
from keras.layers import Input, Dense
from keras.models import Model
import cv2
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import mlab
from matplotlib import pyplot as plt
import sys
np.set_printoptions(threshold=sys.maxsize)
# encoding_dimensions
encoding_dim = 128
encoding_dim2 = 64
encoding_dim3 = 32
# input placeholder
input_img = Input(shape=(784,))
encoded = Dense(128, activation='relu')(input_img)
encoded = Dense(64, activation='relu')(encoded)
encoded = Dense(32, activation='relu')(encoded)
decoded = Dense(64, activation='relu')(encoded)
decoded = Dense(128, activation='relu')(decoded)
decoded = Dense(784, activation='sigmoid')(decoded)
print(encoded.shape)
print(decoded.shape)
autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')
encoder = Model(input_img, encoded)
encoded_input = Input(shape=(encoding_dim,))
decoder_layer = autoencoder.layers[-1]
decoder = Model(encoded_input, decoder_layer(encoded_input))
from keras.datasets import mnist
(x_train, _), (x_test, _) = mnist.load_data()
x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))
print(x_test.shape)
print(x_train.shape)
autoencoder.fit(x_train, x_train,
epochs=1,
batch_size=256,
shuffle=True,
validation_data=(x_test, x_test))
encoded_imgs = encoder.predict(x_test)
decoded_imgs = decoder.predict(encoded_imgs)
n=10
plt.figure(num=2, figsize=(20, 3))
for i in range(n):
# input data
ax = plt.subplot(2, n, i + 1)
plt.imshow(x_test[i].reshape(28, 28))
plt.gray()
ax.get_yaxis().set_visible(False)
ax.get_xaxis().set_visible(False)
# recnstructed data
ax = plt.subplot(2, n, i + 1 + n)
plt.imshow(decoded_imgs[i].reshape(28, 28))
plt.gray()
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
plt.show()

The problem is with the input definition of your decoder model. You set the input to the decoder model to be: encoded_input = Input(shape=(encoding_dim,)) where encoding_dim=128. However, the real input to the decoder is the encoder output, which is of size 32 (the number of neurons in last Dense layer of the encoder). You need to fix the declaration of the input shape in the decoder model, for example:
encoded_input = Input(shape=(32,))
or mode generally:
encoded_input = Input(shape=encoded.shape)

reuse middle layer as input for another model in Keras

I am building a 5-layer autoencoder with Keras. I made the model that maps from input to output which was fine. I built another model that maps from input to the latent coded vector which worked fine. However, I then tried to make a decoding model that maps from the latent coded vector to the output which did not work.
I know that first I should be making an input layer for the decoded model that makes that shape but I can't figure out how to get my coded layers data as an input for the decoded model and to let it map from the coded vector to the final layer.
from keras.layers import Input, Dense
from keras.models import Model
from keras.datasets import mnist
import numpy as np
(x_train, _), (x_test, _) = mnist.load_data()
# Prepare data and normalize
x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
x_train = x_train.reshape(len(x_train), -1)
x_test = x_test.reshape(len(x_test), -1)
input_size = 784
hidden_size = 128
coded_size = 64
x = Input(shape=(input_size,))
hidden_1 = Dense(hidden_size, activation='relu')(x)
coded =Dense(coded_size, activation='relu')(hidden_1)
hidden_2 = Dense(hidden_size, activation='relu')(coded)
r = Dense(input_size, activation='sigmoid')(hidden_2)
autoencoder = Model(inputs=x, outputs=r)
encoder = Model(inputs=x, outputs=coded)
decoder_input = Input(shape=(coded_size,)) # should do this, but don't know how to connect it below
decoder = Model(inputs=coded, output=r)

You can do it like this:
decoder_input = Input(shape=(coded_size,))
next_input = decoder_input
# get the decoder layers and apply them consecutively
for layer in autoencoder.layers[-2:]:
next_input = layer(next_input)
decoder = Model(inputs=decoder_input, outputs=next_input)
As a side note, there is no h in your model. I think it must be replaced by coded.

Newbie: Keras with TensorFlow MNIST code sample: wrong predictions

(Attempt #2, see below for history)
I'm a total newbie with Deep Learning and Keras, and trying to play with the MNIST sample, following the tutorial at https://elitedatascience.com/keras-tutorial-deep-learning-in-python.
With regard to that sample, I just made some minimal adjustments to the code, to make it compatible with the latest API. Then, I added a sample Python script which loads the saved model and a BMP image file (specified as the script first argument), and outputs the class prediction in the form of an exit code.
I trained the model getting accuracy 0.9911, and saved it.
Yet, when testing with some BMP samples I get inconsistent results: only some of the digits are recognized correctly. Probably I'm missing something obvious in preprocessing my data. After Toyo's reply, I changed both the scripts, following https://nextjournal.com/schmudde/ml4a-mnist, so here I quote the newer versions. The older ones are at the bottom.
I also added a sample visualization of the preprocessed array representing an image, and looking at it I can see the digit, 'white on black', as expected (in the predict preprocessing I added color inversion by setting each value equal to 255-value, as my input BMP are 'black on white').
Yet, the issue is still there.
Thanks!
New Scripts
Training:
# numpy with a specific seed for reproducibility
import numpy as np
np.random.seed(123)
# keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Convolution2D, MaxPooling2D
from keras.utils import np_utils
from keras.datasets import mnist
# load pre-shuffled MNIST data into train and test sets
# MNIST: 60k samples of 28x28 grayscale (0-255) images,
# and additional 10k samples for test.
((x_train, y_train), (x_test, y_test)) = mnist.load_data()
n_train, height, width = x_train.shape
n_test, _, _ = x_test.shape
print('x_train shape: {}'.format(x_train.shape))
print('n_train: {}'.format(n_train))
print('height: {}'.format(height))
print('width: {}'.format(width))
print('n_test: {}'.format(n_test))
# we require shape (count, height, width, channel), where count
# is the count of our samples, and channel is the number of channels,
# here equal to 1 as these are grayscale images (for RGB it would be 3).
x_train = x_train.reshape(n_train, height, width, 1).astype('float32')
x_test = x_test.reshape(n_test, height, width, 1).astype('float32')
# normalize from [0, 255] to [0, 1]
x_train /= 255
x_test /= 255
# show image array
print('Sample array: {}\n'.format(x_train[0].shape))
print(x_train[0])
from matplotlib import pyplot as plt
plt.imshow(x_train[0].reshape(28, 28))
plt.show()
# the labels need to be converted into one-hot vectors,
# which are nc-element arrays (nc is the number of classes),
# which are 0 for all classes except 1 for the class the label
# is assigned to.
# convert integer labels into one-hot vectors
n_classes = 10
y_train = np_utils.to_categorical(y_train, n_classes)
y_test = np_utils.to_categorical(y_test, n_classes)
# define model architecture.
model = Sequential()
# number of convolutional filters
n_filters = 32
# convolution filter size
n_conv = 3
# pooling window size
n_pool = 2
# (1) convolution
model.add(Convolution2D(
n_filters,
kernel_size=(n_conv, n_conv),
# we have a 28x28 single channel (grayscale) image
# so the input shape should be (28, 28, 1)
input_shape=(height, width, 1)
))
model.add(Activation('relu'))
model.add(Convolution2D(n_filters, kernel_size=(n_conv, n_conv)))
model.add(Activation('relu'))
# apply pooling to summarize the features extracted thus far
model.add(MaxPooling2D(pool_size=(n_pool, n_pool)))
# (2) connected layers
model.add(Dropout(0.25))
# flatten the data for the 1D layers
model.add(Flatten())
# Dense(n_outputs)
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.5))
# (3) the softmax output layer gives us a probablity for each class
model.add(Dense(n_classes))
model.add(Activation('softmax'))
model.compile(
loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy']
)
# how many examples to look at during each update step
batch_size = 128
# how many times to run through the full set of examples
n_epochs = 10
model.fit(x_train,
y_train,
batch_size=batch_size,
epochs=n_epochs,
validation_data=(x_test, y_test))
# how'd we do?
loss, accuracy = model.evaluate(x_test, y_test, verbose=0)
print('loss:', loss)
print('accuracy:', accuracy)
# save the model. Use load_model(path) to load it later
model.save('/projects/python/mnist/mnist.h5')
Prediction:
# get the path to the image to be recognized from 1st arg
import sys
if len(sys.argv) < 2:
print('Missing image file path')
exit(-1)
# load the pretrained MNIST model
print('Loading model...')
import numpy as np
import keras
model = keras.models.load_model('/projects/python/mnist/mnist.h5')
# load the image ensuring that its size is 28x28
print('Loading image ' + sys.argv[1])
from keras.preprocessing import image
x = image.load_img(sys.argv[1], color_mode='grayscale', target_size=(28, 28))
# convert image to NumPy array
x = image.img_to_array(x, data_format='channels_last')
print('np from image:\n')
print(x.shape)
print(x)
# add a major dimension to represent the number of samples, i.e. 1
x = x.reshape(1, 28, 28, 1)
# convert data type to float32 and normalize 0-255 values to range [0, 1]
x = x.astype('float32')
# invert colors, we need white on black
x = 255 - x
x /= 255
print('np before predicting:\n')
print(x.shape)
print(x)
from matplotlib import pyplot as plt
plt.imshow(x.reshape(28, 28))
plt.show()
# predict
classes = model.predict_classes(x)
print(classes[0])
exit(classes[0])
Old scripts
# numpy with a specific seed for reproducibility
import numpy as np
np.random.seed(123)
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Convolution2D, MaxPooling2D
from keras.utils import np_utils
from keras.datasets import mnist
from matplotlib import pyplot as plt
# load pre-shuffled MNIST data into train and test sets
# 60k samples of 28x28 images
((x_train, y_train), (x_test, y_test)) = mnist.load_data()
print(x_train.shape)
plt.imshow(x_train[0])
input("Press enter to continue...")
# preprocess input data, as Theano requires explicit depth:
# from shape (n, width, height) to (n, depth=1, width, height)
x_train = x_train.reshape(x_train.shape[0], 1, 28, 28)
x_test = x_test.reshape(x_test.shape[0], 1, 28, 28)
# convert our data type to float32 and normalize our data values to the range [0, 1]
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
# convert 1-dimensional class arrays (with values 0-9)
# to 10-dimensional class matrices (with values 0-1)
y_train = np_utils.to_categorical(y_train, 10)
y_test = np_utils.to_categorical(y_test, 10)
# define model architecture
model = Sequential()
# input_shape is the shape of each single parameter:
# here it is depth=1, width=28, height=28
# 32 convolution filters
# 3 rows in each convolution kernel
# 3 cols in each convolution kernel
model.add(Convolution2D(32, (3, 3), activation='relu', input_shape=(1, 28, 28), data_format="channels_first"))
model.add(Convolution2D(32, (3, 3), activation='relu', data_format="channels_first"))
# reduce the model's parameters by sliding a 2x2 pooling filter
# across the previous layer and taking the max of the 4 values in the 2x2 filter
model.add(MaxPooling2D(pool_size=(2,2)))
# regularize the model to prevent overfitting
model.add(Dropout(0.25))
# flatten (make 1-dimensional)
model.add(Flatten())
# dense: 128=output size of the layer
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
# final layer has size 10 for the 10 digits
model.add(Dense(10, activation='softmax'))
# compile model (loss function and optimizer)
model.compile(loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
# fit model on training data
model.fit(x_train, y_train,
batch_size=32, nb_epochs=10, verbose=1)
# evaluate model on test data
score = model.evaluate(x_test, y_test, verbose=0)
# save the model. Use load_model(path) to load it later
model.save('/projects/python/mnist/mnist.h5')
The consuming script:
# get the path to the image to be recognized from 1st arg
import sys
if len(sys.argv) < 2:
print('Missing image file path')
exit(-1)
# load the pretrained MNIST model
print('Loading model...')
import numpy as np
import keras
model = keras.models.load_model('/projects/python/mnist/mnist.h5')
# load the image ensuring that its size is 28x28
print('Loading image ' + sys.argv[1])
from keras.preprocessing import image
x = image.load_img(sys.argv[1], color_mode='grayscale', target_size=(28, 28))
# convert image to NumPy array
x = image.img_to_array(x, data_format='channels_first')
# add a major dimension to represent the number of samples, i.e. 1
x = x.reshape(1, 1, 28, 28)
# convert data type to float32 and normalize 0-255 values to range [0, 1]
x = x.astype('float32')
x /= 255
print(x.shape)
print(x)
# evaluate
classes = model.predict_classes(x)
print(classes[0])
exit(classes[0])

Autoencoder Gridsearch Hyperparameter tuning Keras

My data shape is the same, I just generated here random numbers. In real the datas are float numbers from range -6 to 6, I scaled them as well. The Input layer size and Encoding dimension have to remain the same. When I am training the loss starts and stays at 0.631 all the time. I changed the learning rate manually. I am new to python and do not know to implement to a grid search to this code to find the right parameters. What else can I do to tune my network ?
import numpy as np
from keras.layers import Input, Dense
from keras.models import Model
from keras import optimizers
#Train data
x_train=np.random.rand(2666000)
x_train = (train-min(train))/(max(train)-min(train))
x_train=x_train.reshape(-1,2000)
x_test=[]#empty testing later
#Enc Dimension
encoding_dim=100
#Input shape
input_dim = Input(shape=(2000,))
#Encoding Layer
encoded = Dense(encoding_dim, activation='relu')(input_dim)
#Decoding Layer
decoded = Dense(2000, activation='sigmoid')(encoded)
#Model AE
autoencoder = Model(input_dim, decoded)
#Model Encoder
encoder = Model(input_dim, encoded)
#Encoding
encoded_input = Input(shape=(encoding_dim,))
#Decoding
decoder_layer = autoencoder.layers[-1]
#Model Decoder
decoder = Model(encoded_input, decoder_layer(encoded_input))
optimizers.Adadelta(lr=0.1, rho=0.95, epsilon=None, decay=0.0)
autoencoder.compile(optimizer=optimizer, loss='binary_crossentropy',
metrics=['accuracy'])
#Train and test
autoencoder_train= autoencoder.fit(x_train, x_train,
epochs=epochs, shuffle=False, batch_size=2048)

I suggest adding more hidden layers. If your loss stays the same it means at least one of two things:
Your data is more or less random and there are no relationships to be drawn
Your model is not complex enough to learn meaningful relationships from your data
A rule of thumb for me is that a model should be powerful enough to overfit the data given enough training iterations.
Unfortunately there is a fine line between sufficiently complex and too complex. You have to play around with the number of hidden layers, the number of units in each layer, and the amount of epochs you take to train your network. Since you only have two Dense layers, a good starting point would be to increase model complexity.
If you insist on using a grid search keras has a wrapper for scikit_learn and sklearn has a grid search module. A toy example:
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import GridSearchCV
def create_model():
<return a compiled but untrained keras model>
model = KerasClassifier(build_fn = create_model, batch_size=1000, epochs=10)
#now write out all the parameters you want to try out for the grid search
activation = ['relu', 'tanh', 'sigmoid'...]
learn_rate = [0.1, 0.2, ...]
init = ['unform', 'normal', 'zero', ...]
optimizer = ['SGD', 'Adam' ...]
param_grid = dict(activation=activation, learn_rate=learn_rate, init=init, optimizer=optimizer)
grid = GridSearchCV(estimator=model, param_grid=param_grid)
result = grid.fit(X, y)