Develop Reference

How to store wrong predictions during evaluation on the CNN

During evaluation, I want to store unique ids that are wrongly predicted to do some more processing.
It is a multiclass prediction problem
Here is the code during the evaluation:
outputs = model(imgs)
loss = criterion(outputs, targets) # Prediction error
val_loss += loss.item()
predicted = torch.argmax(outputs, dim=1)
t_predicted +=predicted.cpu().tolist()
total += targets.size(0)
good_answers = (predicted == targets)
correct += good_answers.sum().item()
Knowing that ids is a list of the ids of the images, When I try to get the ids that are wrong:
wrong_ids += ids[~(good_answers.to('cpu'))]
I get this error:
add(): argument 'other' (position 1) must be Tensor

The question contained a tensorflow tag, so I was preparing an answer. After completing my write up, I've found that this tag is removed. However, I believe my answer can give insight into this general question of whether they're using tf or pytorch.
Data
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()
# train set / data
x_train = x_train.astype('float32') / 255
# validation set / data
x_test = x_test.astype('float32') / 255
# train set / target
y_train = tf.keras.utils.to_categorical(y_train, num_classes=10)
# validation set / target
y_test = tf.keras.utils.to_categorical(y_test, num_classes=10)
print(x_train.shape, y_train.shape)
print(x_test.shape, y_test.shape)
Train
import tensorflow as tf
# declare input shape
input = tf.keras.Input(shape=(32,32,3))
# Block 1
x = tf.keras.layers.Conv2D(32, 3, strides=2, activation="relu")(input)
x = tf.keras.layers.MaxPooling2D(3)(x)
# Now that we apply global max pooling.
gap = tf.keras.layers.GlobalMaxPooling2D()(x)
# Finally, we add a classification layer.
output = tf.keras.layers.Dense(10, activation='softmax')(gap)
# bind all
func_model = tf.keras.Model(input, output)
print('\nFunctional API')
func_model.compile(
metrics=['accuracy'],
loss = 'categorical_crossentropy',
optimizer = tf.keras.optimizers.Adam()
)
func_model.fit(x_train, y_train)
Error Prediction
# Predict the values from the validation dataset
y_pred = func_model.predict(x_test)
# Convert predictions classes to one hot vectors
y_pred_classes = np.argmax(y_pred, axis = 1)
y_test = np.argmax(y_test, axis=1)
# Errors are difference between predicted labels and true labels
errors = (y_pred_classes - y_test != 0)
y_pred_classes_errors = y_pred_classes[errors]
y_pred_errors = y_pred[errors]
y_true_errors = y_test[errors]
x_test_errors = x_test[errors]
# Probabilities of the wrong predicted numbers
y_pred_errors_prob = np.max(y_pred_errors, axis = 1)
# Predicted probabilities of the true values in the error set
true_prob_errors = np.diagonal(np.take(y_pred_errors, y_true_errors, axis=1))
# Difference between the probability of the predicted label and the true label
delta_pred_true_errors = y_pred_errors_prob - true_prob_errors
# Sorted list of the delta prob errors
sorted_dela_errors = np.argsort(delta_pred_true_errors)
# Top 6 errors
most_important_errors = sorted_dela_errors[-6:]
Display
import matplotlib.pyplot as plt
def display_errors(errors_index,img_errors,pred_errors, obs_errors):
""" This function shows 6 images with their predicted and real labels"""
n = 0
nrows = 2
ncols = 3
fig, ax = plt.subplots(nrows,ncols,sharex=True,sharey=True)
for row in range(nrows):
for col in range(ncols):
error = errors_index[n]
ax[row,col].imshow((img_errors[error]).reshape((32,32,3)))
ax[row,col].set_title("Predicted label :{}\nTrue label :{}".format(pred_errors[error],obs_errors[error]))
n += 1
# Show the top 6 errors
display_errors(most_important_errors, x_test_errors, y_pred_classes_errors, y_true_errors)

What worked for me is to declare empty list and then to fill them with the predictions
# TO GET ONLY THE WRONGLY LABELED ITEMS
wrong_ids.append(ids[~(predicted == targets ).to('cpu')])
# TO ALSO STORE ALL TESTED LABELS PREDICTED AND TARGETS IN SEPARATE LIST
tested_labels.append(ids.to('cpu'))
pred_labels.append(predicted.to('cpu'))
true_labels.append(targets.to('cpu'))
then after that to convert the resulting list of tensors to one list:
wrongs =[]
for i,j in enumerate(wrong_ids):
for k,l in enumerate(j):
wrongs.append(l.item())
# and so on for the other lists of tensors
To show all tested labels with the predictions and the true labels:
df = pd.DataFrame({'id': ids,
'predicted': pred_label,
'true_label': true_label})
print(df)

How to create a confusion matrices for an image segmentation task?

I'm dealing with binary image segmentation problem. I've successfully compiled and trained the model. Now I'm trying to achieve two goals:
Get a total confusion matrix for a test set (reason: understand proportions of false positives and false negatives)
Get an individual confusion matrix for every image in a test set (reason: find and analyze images that drag model performance down)
As far as I understand, confusion_matrix from scikit-learn package can help with a total confusion matrix, but I can't make it work with my custom data generator. According to docs, here's a code for confusion_matrix:
sklearn.metrics.confusion_matrix(y_true, y_pred, *, labels=None, sample_weight=None, normalize=None)
I don't understand how to retrieve y_true with my custom data generator:
def learn_generator(templates_folder, masks_folder, image_width, batch_size, shuffle=True):
"""Generate individual batches form dataset"""
counter = 0
images_list = os.listdir(templates_folder)
if shuffle:
random.shuffle(images_list)
while True:
templates_pack = np.zeros((batch_size, image_width, image_width, 3)).astype('float')
masks_pack = np.zeros((batch_size, image_width, image_width, 1)).astype('float')
for i in range(counter, counter + batch_size):
template = cv2.imread(templates_folder + '/' + images_list[i]) / 255.
templates_pack[i - counter] = template
mask = cv2.imread(masks_folder + '/' + images_list[i], cv2.IMREAD_GRAYSCALE) / 255.
mask = np.expand_dims(mask, axis=2)
masks_pack[i - counter] = mask
counter += batch_size
if counter + batch_size >= len(images_list):
counter = 0
if shuffle:
random.shuffle(images_list)
yield templates_pack, masks_pack
test_templates_path = "E:/Project/images/all_templates/test"
test_masks_path = "E:/Project/images/all_masks/test"
TEST_SET_SIZE = len(os.listdir(test_templates_path))
IMAGE_WIDTH = 512
BATCH_SIZE = 4
TEST_STEPS = TEST_SET_SIZE / BATCH_SIZE
test_generator = learn_generator(test_templates_path, test_masks_path, IMAGE_WIDTH, batch_size=BATCH_SIZE)
Y_pred = model.predict_generator(test_generator, steps=TEST_STEPS)
y_pred = np.argmax(Y_pred, axis=1)
y_true = ???
As for individual confusion matrices, no ideas at all...
Any help is appreciated.

I suppose it's too late for you, but maybe this could help somebody else :
I did achive this by using the definition of a confusion matrix, by calculation True Positives, True Negatives, False Positives, False negatives.
This code works only for binary segmentation, assuming that "1" is the output for "positive" and "0" for "negative"...
import seaborn as sns
FP = len(np.where(Y_pred - Y_val == 1)[0])
FN = len(np.where(Y_pred - Y_val == -1)[0])
TP = len(np.where(Y_pred + Y_val ==2)[0])
TN = len(np.where(Y_pred + Y_val == 0)[0])
cmat = [[TP, FN], [FP, TN]]
plt.figure(figsize = (6,6))
sns.heatmap(cmat/np.sum(cmat), cmap="Reds", annot=True, fmt = '.2%', square=1, linewidth=2.)
plt.xlabel("predictions")
plt.ylabel("real values")
plt.show()
confusion_matrix

https://www.kite.com/blog/python/image-segmentation-tutorial/
I think you could change the code from this website for two classes and apply it for your use case.
Additionally, I think this answer does provide the implementation of sklearn confusion matrix method Faster method of computing confusion matrix?

Keras model won't use predict correctly

I have a Keras model that I have trained and evaluated and even tested. Now I am trying to use three test images into the model.
I run the images through a preprocessor which is the same one I used to make the training data. I then do the exact same thing to the single images that I did for the testing data. But it gives me an error of
Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 array(s), but instead got the following list of 2 arrays:
I don't know what is wrong with it.
So this is how I test the model successfully.
y_pred = []
y_true = []
for i in range(0, len(test_x1)):
x1 = test_x1[i]
x2 = test_x2[i]
x1 = np.expand_dims(x1, axis=0)
x2 = np.expand_dims(x2, axis=0)
y_true.append(np.argmax(test_y[i]))
pred = model.predict([x1, x2])
y_pred.append(make_binary(pred))
This is the preprocessing method I used for both images
def create_features(file, image_dir, base_model):
img_path = os.path.join(image_dir, file)
img = image.load_img(img_path, target_size=(224, 224))
img = image.img_to_array(img)
x = resnet50.preprocess_input(img)
x = np.array([x])
feature = base_model.predict(x)
return feature
And this is the way I am processing the new images:
IMAGE_DIR = 'Data'
img1 = 'test1.jpg'
img2 = 'test2.jpg'
img3 = 'test3.jpg'
img1_feat = create_features(img1, IMAGE_DIR, model)
img2_feat = create_features(img2, IMAGE_DIR, model)
img3_feat = create_features(img3, IMAGE_DIR, model)
Now when I look at the two features they are the same.
x1 = test_x1[0]
x1 = np.expand_dims(x1, axis=0)
print(x1.shape)
print(type(x1))
print(img1_feat.shape)
print(type(img1_feat))
(1, 1, 1000)
<class 'numpy.ndarray'>
(1, 1, 1000)
<class 'numpy.ndarray'>
And then I try to make a prediction from it
pred1 = model.predict([img1_feat, img2_feat])
But that results in an error.

I figured out what was wrong thanks to #Matias Valdenegro and #Mukul
I was doing this in an Ipython notebook and after a few epochs go around I found out that on occasion the model gets overwritten by an imported resent model from another class.
Thanks to everyone for the help. I didnt think about using the model.summary() as I didnt really think that it has changed.

Explaination of Isolation Forest Algorithm Source Code

I was going through the source code for Isolation Forest Algorithm in sci-kit module. The overall structure of the model is:
read data set
train test split of data
train the data on model
fit the model
test other set of data on the model
predict anomalies
The source code is available in GitHub. In the source code there is a file called iforest.py (path: sklearn > ensemble > iforest.py )
In that file there is a function called score_samples which calculates the anomaly score for each data-point in the data-set.
The code looks somewhat like this :
def score_samples(self, X):
n_samples = X.shape[0]
n_samples_leaf = numpy.zeros((n_samples, self.n_estimators), order="f")
depths = numpy.zeros((n_samples, self.n_estimators), order="f")
if self._max_features == X.shape[1]:
subsample_features = False
else:
subsample_features = True
for i, (tree, features) in enumerate(zip(self.estimators_ self.estimators_features_)):
if subsample_features:
X_subset = X[:, features]
else:
X_subset = X
leaves_index = tree.apply(X_subset)
node_indicator = tree.decision_path(X_subset)
n_samples_leaf[:, i] = tree.tree_.n_node_samples[leaves_index]
depths[:, i] = numpy.ravel(node_indicator.sum(axis=1))
depths[:, i] -= 1
depths += _average_path_length(n_samples_leaf)
scores = 2 ** (-depths.mean(axis=1) / _average_path_length(
self.max_samples_))
# Take the opposite of the scores as bigger is better (here less
# abnormal)
return -scores
I cannot understand what is happening in the for loop. Can someone please help me out with this.

Creating custom data_generator in Keras for fit_generate()

I am trying to train a CNN-LSTM to read a sequence of 6 frames at a time to the CNN (VGG16 without top layer) and give the extracted features to an LSTM in Keras.
The issue is that, since I need to send 6 frames at a time, I need to reshape every 6 frames and add a dimension. Also, since the labels are for every frame, I need to create another variable to get the label of the first frame for every sequence and put it in a new array and then feed both to feed the model (code below).
The issue is that the data gets way too big to use model.fit() and even when trying with it on a small part of the data I get weird horrible results, so I am trying to use model.fit_generator to iterate the input to the model. But since I cannot just directly feed the data I load from the dataset (because I need to reshape and do what I explained in the first paragraph), I am trying to make my own generator. However, things are not going well and I keep getting errors saying 'tuple' is not an iterator. Does anyone know how I can fix the code to make it work?
train_batches = ImageDataGenerator().flow_from_directory(train_path, target_size=(224, 224),
classes=['Bark', 'Bitting', 'Engage', 'Hidden', 'Jump',
'Stand', 'Walk'], batch_size=18156, shuffle=False)
valid_batches = ImageDataGenerator().flow_from_directory(valid_path, target_size=(224, 224),
classes=['Bark', 'Bitting', 'Engage', 'Hidden', 'Jump',
'Stand', 'Walk'], batch_size=6, shuffle=False)
test_batches = ImageDataGenerator().flow_from_directory(test_path, target_size=(224, 224),
classes=['Bark', 'Bitting', 'Engage', 'Hidden', 'Jump',
'Stand','Walk'], batch_size=6, shuffle=False)
def train_gen():
n_frames=6
n_samples=6 #to decide
H=W=224
C = 3
imgs, labels = next(train_batches)
y = np.empty((n_samples, 7))
j = 0
for i in range(n_samples):
y[i] = labels[j]
j +=6
frame_sequence = imgs.reshape(n_samples,n_frames, H,W,C)
return frame_sequence,y
def valid_gen():
v_frames=6
v_samples=1
H=W=224
C = 3
vimgs,vlabels = next(valid_batches)
y2 = np.empty((v_samples, 7))
k = 0
for l in range(v_samples):
y2[l] = vlabels[k]
k +=6
valid_sequence = vimgs.reshape(v_samples,v_frames, H,W,C)
return valid_sequence,y2
def main():
cnn = VGG16(weights='imagenet',
include_top='False', pooling='avg')
cnn.layers.pop()
print(cnn.summary())
cnn.trainable = False
video_input= Input(shape=(None,224,224,3), name='video_input')
print(video_input.shape)
encoded_frame_sequence = TimeDistributed(cnn)(video_input) # the output will be a sequence of vectors
encoded_video = LSTM(256)(encoded_frame_sequence) # the output will be a vector
output = Dense(7, activation='relu')(encoded_video)
video_model = Model(inputs=[video_input], outputs=output)
tr_data = train_gen()
vd_data= valid_gen()
print(video_model.summary())
imgs, labels = next(train_batches)
vimgs,vlabels = next(valid_batches)
print("Training ...")
video_model.compile(Adam(lr=.001), loss='categorical_crossentropy', metrics=['accuracy'])
video_model.fit_generator(tr_data,
steps_per_epoch=1513,
validation_data=vd_data,
validation_steps=431,
epochs=1,
verbose=2)
Is there a mistake in the way I define the generator?

It seems like the way I defined the generators was not correct. As a Keras admin explained to me, the definition has two issues.
Instead of return we need to use the yield
We need a while True loop to make sure it keeps reading
Note that there are few errors on the rest of the code that I dealt with, but since this question is about the generator, I am only posting an answer about that part (There are two generators but they are similar except for the input):
def train_gen():
n_frames=6
n_samples=5 #to decide
H=W=224
C = 3
while True:
imgs, labels = next(train_batches)
#frame_sequence = imgs.reshape(n_samples,n_frames, H,W,C)
y = np.empty((n_samples, 7))
j = 0
#print("labels")
#print(labels)
#print("y")
#print(y.shape)
if len(labels) == n_frames*n_samples:
frame_sequence = imgs.reshape(n_samples,n_frames, H,W,C)
for i in range(n_samples):
y[i] = labels[j]
# print("inside: ")
#print(y[i])
# print(labels[j])
j +=6
yield frame_sequence,y

I think that you should implement a class for the data-generator, I found this link, it might help you. A detailed example of how to use data generators with Keras