Confusion Matrix with convolution Neural Network - python

I'm new to using neural network and machine learning. I've tried to get the result from my Convolution Neural Network in form of confusion Matrix.
Currently, I got some problems in how to make it visually more easier to understand and how to make it work.
batch_size = 60
# Image Processing Stage
train_data = ImageDataGenerator(rescale=1./255, shear_range=0.2, zoom_range=0.2, horizontal_flip=True)
test_data = ImageDataGenerator(rescale=1./255)
training_set = train_data.flow_from_directory('dataset/train_data', target_size=(128, 128), batch_size=batch_size, class_mode='binary')
test_set = test_data.flow_from_directory('dataset/test_data', target_size=(128, 128), batch_size=batch_size, class_mode='binary')
# Starting Convolution Neural Network
start_cnn = Sequential()
start_cnn.add(Conv2D(64, (3, 3), input_shape=(128, 128, 3), activation='relu', kernel_regularizer=l2(0.0001), bias_regularizer=l2(0.0001))) # 3*3*3*32+32
start_cnn.add(Conv2D(64, (3, 3), activation='relu', kernel_regularizer=l2(0.0001), bias_regularizer=l2(0.0001)))
start_cnn.add(MaxPooling2D(pool_size=(2, 2)))
for i in range(0, 2):
start_cnn.add(Conv2D(32, (3, 3), activation='relu', kernel_regularizer=l2(0.0001), bias_regularizer=l2(0.0001)))
start_cnn.add(MaxPooling2D(pool_size=(2, 2)))
for i in range(0, 2):
start_cnn.add(Conv2D(32, (3, 3), activation='relu', kernel_regularizer=l2(0.0001), bias_regularizer=l2(0.0001)))
start_cnn.add(MaxPooling2D(pool_size=(2, 2)))
start_cnn.add(Dropout(0.25))
# Flattening
start_cnn.add(Flatten())
# Step 4 - Full connection
start_cnn.add(Dense(activation="relu", units=32, kernel_regularizer=l2(0.0001), bias_regularizer=l2(0.0001)))
start_cnn.add(Dropout(0.5))
start_cnn.add(Dense(activation="sigmoid", units=1, kernel_regularizer=l2(0.0001), bias_regularizer=l2(0.0001)))
start_cnn.summary()
start_cnn.load_weights("weights.best.hdf5")
# Compiling the CNN
start_cnn.compile(Adam(learning_rate=0.0001), loss='binary_crossentropy', metrics=['accuracy'])
history = start_cnn.fit(training_set, steps_per_epoch=85, epochs=1, validation_data=test_set, verbose=2, shuffle = False) # step per epoch is 5065/batch size callbacks=callbacks_list,
This is a brief on how my code layout is organized. Please, if there is any comment or concern please do tell me, I am new and would like to learn as much as possible.
Here is the confusion matrix code I did
y_pred = start_cnn.predict_generator(test_set, steps = np.ceil(test_set.samples / test_set.batch_size), verbose=0, workers=0)
y_pred = np.where(y_pred>0.5, 1, 0)
print('Confusion Matrix')
print(confusion_matrix(test_set.classes, y_pred))
print('Classification Report')
print(classification_report(test_set.classes, y_pred))
labels = ['melanoma', 'non-melanoma']
cm = confusion_matrix(test_set.classes, y_pred)
fig = plt.figure()
ax = fig.add_subplot(111)
cax = ax.matshow(cm)
plt.title('Confusion matrix')
fig.colorbar(cax)
ax.set_xticklabels([''] + labels)
ax.set_yticklabels([''] + labels)
plt.xlabel('Predicted')
plt.ylabel('True')
plt.show()
As you can see in the image here. The value only appears in the two box on the left side but not on the right side at all. Also, I would like to use plot_confusion_matrix too, but I can't seems to find a way to get it working at all.
enter image description here
Please help!! Thank you for your time

Related

Tensor model accuracy - machine learning based on images

I am trying to build a software that can trade for me based in machine learning.
So I extracted bunch of data, generated charts with some indicators that I use and separate them in BUY, SELL and NOT_DEFINED. and I am using SELL and BUY to train.
Unfortunately the accuracy is not increasing. In certain level I made the accuracy achieve 0.82 but the final model.evaluate(test_set) always gives me a randonmic number around 0.50 . Even the prediction code I build is give me the same result.
Now not even during training the accuracy increase. Can somebody give me ideas how to make some progress?
import tensorflow as tf
from tensorflow import keras
import os
from keras.preprocessing.image import ImageDataGenerator
# Carregando as imagens
train_datagen = ImageDataGenerator(rescale = 1./255,
shear_range = 0.2,
zoom_range = 0.2,
horizontal_flip = True)
test_datagen = ImageDataGenerator(rescale = 1./255)
training_set = train_datagen.flow_from_directory('C:/Users/Adriano/PycharmProjects/machineLearningChatGPT/data/150_reduzido/train',
target_size = (150, 150),
batch_size = 64,
class_mode = 'categorical',
shuffle=False)
test_set = test_datagen.flow_from_directory('C:/Users/Adriano/PycharmProjects/machineLearningChatGPT/data/150_reduzido/validation',
target_size = (150, 150),
batch_size = 64,
class_mode = 'categorical',
shuffle=False)
# Construindo e treinando o modelo
model = keras.Sequential([
keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(150, 150, 3)),
keras.layers.MaxPooling2D((2, 2)),
keras.layers.Conv2D(64, (3, 3), activation='relu'),
keras.layers.MaxPooling2D((2, 2)),
# keras.layers.Conv2D(64, (3, 3), activation='relu'),
# keras.layers.MaxPooling2D((2, 2)),
# keras.layers.Conv2D(128, (3, 3), activation='relu'),
# keras.layers.MaxPooling2D((2, 2)),
keras.layers.Flatten(),
keras.layers.Dense(128, activation='relu'),
keras.layers.Dense(2, activation='softmax')
])
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(training_set, epochs=20)
# Avaliando o modelo
test_loss, test_acc = model.evaluate(test_set)
# Salvando o modelo e os pesos
model.save("redmodelFalse.h5")
model.save_weights("redweightFalse.h5")
I tried lots of things I saw other people doing. changed the number of epochs, batch_size, add shuffle = false in the training set, add layers, change activation and of course the set of images I am using.

Wrong Confusion Matrix Display - CCN Image Keras model

am new to Deep Learning and using CNNs. I am trying to predict the quality of coffee seeds using images where good quality is Coffee AA and the other is Poor Quality. I built the model tried to evaluate its performance by using a Confusion Matrix on a test data(test_ds) but it seems to give me a one column prediction instead of a diagonal form. I don't know where the issue is coming from whether its how i built the model or how am trying to build the confusion matrix. Kindly help. The rest of the code is here https://colab.research.google.com/drive/1cNgAbCy8e4lG-dCepVxIpFx1Wfeof5wb?usp=sharing
Here is part of the Code:
data = tf.keras.preprocessing.image_dataset_from_directory(
"/data/train2 cofi",
shuffle=True,
image_size= (IMAGE_SIZE,IMAGE_SIZE),
batch_size= BATCH_SIZE)
img_height = 256
img_width = 256
batch_size = 32
test_ds = tf.keras.preprocessing.image_dataset_from_directory(
"/data/test2 cofi",
image_size=(img_height, img_width),
batch_size=batch_size)
#preparing the model layers
input_shape = (BATCH_SIZE, IMAGE_SIZE, IMAGE_SIZE, CHANNELS)
n_classes = 1
model = models.Sequential([
resize_and_rescale,
data_augmentation,
layers.Conv2D(32, (3,3), activation='relu', input_shape = input_shape), # 32-layers, (3,3)-filter size,activation function
layers.MaxPooling2D((2, 2)),
layers.Conv2D(64, kernel_size = (3,3), activation='relu'),
layers.MaxPooling2D((2, 2)),
layers.Conv2D(64, (3, 3), activation='relu'),
layers.MaxPooling2D((2, 2)),
layers.Conv2D(64, (3, 3), activation='relu'),
layers.MaxPooling2D((2, 2)),
layers.Conv2D(64, (3, 3), activation='relu'),
layers.MaxPooling2D((2, 2)),
layers.Flatten(),
layers.Dense(64, activation='relu'),
layers.Dense(n_classes, activation='sigmoid') # softmax helps to normalize the probabilities in the dense layer
])
model.build(input_shape=input_shape)
#Training the model
metrics = [TruePositives(name='TruePov'), FalsePositives(name='FalsePov'),
TrueNegatives(name='TrueNeg'), FalseNegatives(name='FalseNeg'),
BinaryAccuracy(name='accuracy'), Precision(name='Precision'), Recall(name='recall'), AUC(name='auc')]
# Using the adam optimizer to track the gradient descent in the training process
model.compile(
optimizer= Adam(learning_rate = 0.01),
loss = BinaryCrossentropy(),
metrics=metrics
)
history = model.fit(
train_ds,
epochs=EPOCHS,
batch_size=BATCH_SIZE,
verbose=1,
validation_data=val_ds
)
# Building a Confusion Matrix
from sklearn.metrics import confusion_matrix, classification_report
import seaborn as sns
sns.set_style('darkgrid')
classes=test_ds.class_names # ordered list of class names
class_names = ['Arabica AA', 'Poor Arabica Quality']
ytrue=[]
for images, label in test_ds:
for e in label:
ytrue.append(classes[e]) # list of class names associated with each image file in test dataset
ypred=[]
errors=0
count=0
preds=model.predict(test_ds, verbose=1) # predict on the test data
for i, p in enumerate(preds):
count +=1
index=np.argmax(p) # get index of prediction with highest probability
klass=classes[index]
ypred.append(klass)
if klass != ytrue[i]:
errors +=1
acc= (count-errors)* 100/count
msg=f'there were {count-errors} correct predictions in {count} tests for an accuracy of {acc:6.2f} % '
print(msg)
ypred=np.array(ypred)
ytrue=np.array(ytrue)
if len(classes)<= 2: # if more than 30 classes plot is not useful to cramed
# create a confusion matrix
cm = confusion_matrix(ytrue, ypred )
length=len(classes)
if length<8:
fig_width=5
fig_height=5
else:
fig_width= int(length * .4)
fig_height= int(length * .4)
plt.figure(figsize=(fig_width, fig_height))
sns.heatmap(cm, annot=True, vmin=0, fmt='g', cmap='Blues', cbar=False)
plt.xticks(np.arange(length)+.3, classes, rotation= 90)
plt.yticks(np.arange(length)+.3, classes, rotation=0)
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.title("Confusion Matrix")
plt.show()
clr = classification_report(ytrue, ypred, target_names=class_namz)
print("Classification Report:\n----------------------\n", clr)
The output of the Confusion Matrix
enter image description here

Getting wrong order of the Confusion Matrix using-CNN(Binary Classification for image dataset)

Am learning how to create CNN for a Binary classification task 0/1. I trained the model and tried to evaluate my model on the test data. My confusion Matrix gives me one column results such as TruePositive and False Negatives only!!
[[100 0] but was expecting something like this >> [[100 0]
[100 0]] [0 100]]
Actual Image of CM
Am not sure where my issue is coming from, whether its the way i structured the CNN layers or how i tried to plot the Confusion Matrix.
Here's look at the Code: Please guide How best can i create a confusion matrix!!
# loading the train data(data)
data = tf.keras.preprocessing.image_dataset_from_directory("data/train cofi",shuffle=True,
image_size= (IMAGE_SIZE,IMAGE_SIZE),
atch_size= BATCH_SIZE)
# the data
input_shape = (BATCH_SIZE, IMAGE_SIZE, IMAGE_SIZE, CHANNELS)
n_classes = 1
model = models.Sequential([
resize_and_rescale,
data_augmentation,
layers.Conv2D(32, (3,3), activation='relu', input_shape = input_shape), # 32-layers, (3,3)-filter size,activation function
layers.MaxPooling2D((2, 2)),
layers.Conv2D(64, kernel_size = (3,3), activation='relu'),
layers.MaxPooling2D((2, 2)),
layers.Conv2D(64, (3, 3), activation='relu'),
layers.MaxPooling2D((2, 2)),
layers.Conv2D(64, (3, 3), activation='relu'),
layers.MaxPooling2D((2, 2)),
layers.Conv2D(64, (3, 3), activation='relu'),
layers.MaxPooling2D((2, 2)),
layers.Flatten(),
layers.Dense(64, activation='relu'),
layers.Dense(n_classes, activation='sigmoid') # softmax helps to normalize the probabilities in the dense layer
])
model.build(input_shape=input_shape)
# training the model on tran-data
model.compile(optimizer= Adam(learning_rate = 0.01),loss = BinaryCrossentropy(),metrics=metrics)
history = model.fit(train_ds, epochs=EPOCHS, batch_size=BATCH_SIZE, verbose=1, validation_data=val_ds)
Plotting the Confusion
from sklearn.metrics import confusion_matrix, classification_report
import seaborn as sns
sns.set_style('darkgrid')
test_3 = tf.keras.preprocessing.image_dataset_from_directory("/data/test2_cofi",
image_size= (IMAGE_SIZE,IMAGE_SIZE),
batch_size= BATCH_SIZE)
class_names = ['Arabica AA', 'Poor Arabica Quality']
classes = test_3.class_names #ordered list of class names
ytrue=[]
for images, label in test_ds:
for e in label:
ytrue.append(classes[e]) # list of class names associated with each image file in test dataset
ypred=[]
errors=0
count=0
preds=model1.predict(test_3, verbose=1) # predict on the test data
I suspect the use of np.argmax(p) below to be the issue in plotting the CM rather supposed to use a threshold like 0.5 If so, how do i change it incase its the issue!!
for i, p in enumerate(preds):
count +=1
index=np.argmax(p) # get index of prediction with highest probability
klass=classes[index]
ypred.append(klass)
if klass != ytrue[i]:
errors +=1
acc= (count-errors)* 100/count
msg=f'there were {count-errors} correct predictions in {count} tests for an accuracy of {acc:6.2f} % '
print(msg)

not sure why my CNN model isn't getting past 0.03 acc

I am using a CNN model to classify the spectrograms of different audio files.
this is the model, I created
input_shape = (128, 216, 1)
def create_model():
model = models.Sequential()
model.add(layers.Conv2D(32, (5, 5), strides = (1,1), name = 'conv0', input_shape = input_shape))
model.add(layers.BatchNormalization(axis = 3, name = 'bn0'))
model.add(layers.Activation('relu'))
model.add(layers.MaxPooling2D((2, 2), name='max_pool'))
model.add(layers.Conv2D(64, (3, 3), strides = (1,1), name='conv1'))
model.add(layers.Activation('relu'))
model.add(layers.AveragePooling2D((3, 3), name='avg_pool'))
model.add(layers.GlobalAveragePooling2D())
model.add(layers.Dense(300, activation='relu', name='rl'))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(50, activation='softmax', name='sm'))
model.compile(loss='categorical_crossentropy',
optimizer='adam',
metrics='acc')
return model
I'm using this to get the spectrograms from a file
for filename in pbar(os.listdir(save_file + r"spec-samples/")):
image_raw = tf.io.read_file(save_file + r"spec-samples/" + filename)
image = tf.image.decode_image(image_raw)
X.append(image)
and this to get the labels from a csv file
def reload_labels():
y = []
for data in tqdm(df.iterrows(), desc='Progress'):
y.append(data[1][1])
return y
when I fit the model via
cp_callback = tf.keras.callbacks.ModelCheckpoint(
checkpoint, monitor='accuracy', verbose=1, save_best_only=True, mode='max')
es_callback = tf.keras.callbacks.EarlyStopping(
monitor='loss', patience=3, verbose=1)
csv_callback = tf.keras.callbacks.CSVLogger(save_file + "logs.csv", separator=',', append=False)
history = model.fit(
X_, y_,
epochs=20,
validation_data=(X_test, y_test),
callbacks=[cp_callback, es_callback, csv_callback])
the highest accuracy I get is 0.026 which is obviously not great
this is an example of one of the spectrograms
spectrogram
what am i doing wrong?
I am a bit confused by your code. If I understand it you are feeding the network with images of spectrograms and I assume binary labels. The top layer of your model is
model.add(layers.Dense(50, activation='sigmoid', name='sm')
where does the value 50 come from? I would expect the last layer to have 1 node.
Also are you sure you have maintained the order of the images X to line up with the labels Y since they are coming from 2 different sources.
If you have 50 categories (classes) then the top layer should use activation softmax

Model get 97% accuracy on train and validation but when use custom predict it get wrong

I use a CNN model to train image classification , it got great accuracy at test and validation (98% and 97%), but when use my image to predict it alway go wrong, here is my code:
BATCH_SIZE = 30
IMG_HEIGHT = 256
IMG_WIDTH = 256
STEPS_PER_EPOCH = np.ceil(image_count/BATCH_SIZE)
train_data_gen = image_generator.flow_from_directory(directory=str(data_dir),
batch_size=BATCH_SIZE,
shuffle=True,
target_size=(IMG_HEIGHT, IMG_WIDTH),
classes = list(CLASS_NAMES))
here is prepare for dataset and data argumentation:
imgDataGen=ImageDataGenerator(
validation_split=0.2,
rescale=1/255,
horizontal_flip=True,
zoom_range=0.3,
rotation_range=15.,
width_shift_range=0.1,
height_shift_range=0.1,
)
prepare data:
train_dataset = imgDataGen.flow_from_directory(
directory=str(data_dir),
target_size = (IMG_HEIGHT, IMG_WIDTH),
classes = list(CLASS_NAMES),
batch_size = BATCH_SIZE,
subset = 'training'
)
val_dataset = imgDataGen.flow_from_directory(
directory=str(data_dir),
target_size = (IMG_HEIGHT, IMG_WIDTH),
classes = list(CLASS_NAMES),
batch_size =BATCH_SIZE,
subset = 'validation'
)
the model
model = Sequential()
model.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(256, 256, 3)))
model.add(MaxPooling2D((2, 2)))
model.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))
model.add(MaxPooling2D((2, 2)))
model.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))
model.add(MaxPooling2D((2, 2)))
model.add(Flatten())
model.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))
model.add(Dense(6, activation='sigmoid'))
complie:
model.compile(loss='binary_crossentropy',
optimizer=keras.optimizers.SGD(learning_rate=0.001,momentum=0.9),
metrics=['acc'])
train
history = model.fit_generator(
train_dataset,
validation_data = val_dataset,
workers=10,
epochs=20,
)
It get pretty high accuracy 98% on test and 97% on validation
but when i try with my code to predict
def prepare(filepath):
IMG_SIZE=256
img_array=cv2.imread(filepath)
new_array= cv2.resize(img_array,(IMG_SIZE,IMG_SIZE))
return new_array.reshape(1,IMG_SIZE,IMG_SIZE,3)
model=tf.keras.models.load_model('trained-model.h5',compile=False)
#np.set_printoptions(formatter={'float_kind':'{:f}'.format})
predict=model.predict([prepare('cat.jpg')])
pred_name = CATEGORIES[np.argmax(predict)]
print(pred_name)
it got wrong, with cat image it go for dog and dog for cat, but sometime it go right, just i think 98% is more accurate than this, if i try 5 image of cats it fail 3 or 4 images
so it because dataset or because of code?
please help, thanks
So in your second code-block you have this:
rescale=1/255
This is for normalizing your image into the range [0;1]. So every image gets rescaled (/normalized) before going through the network. But in you las code-block where you test it on an image you didnt add normalization. Try adding that to your "prepare" function:
def prepare(filepath):
IMG_SIZE = 256
img_array = cv2.imread(filepath)
# add this:
img_array = image_array / 255
new_array = cv2.resize(img_array,(IMG_SIZE,IMG_SIZE))
return new_array.reshape(1,IMG_SIZE,IMG_SIZE,3)

Categories

Resources