I'm building a simple Neural network in Keras, like the following:
# create model
model = Sequential()
model.add(Dense(1000, input_dim=x_train.shape[1], activation='relu'))
model.add(Dense(1, activation='sigmoid'))
# Compile model
model.compile(loss='mean_squared_error', metrics=['accuracy'], optimizer='RMSprop')
# Fit the model
model.fit(x_train, y_train, epochs=20, batch_size=700, verbose=2)
# evaluate the model
scores = model.evaluate(x_test, y_test, verbose=0)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
The shape of the used data is:
x_train = (49972, 601)
y_train = (49972, 1)
My problem is that the network is not converging, the accuracy is fixed on 0.0168, like below:
Epoch 1/20
- 1s - loss: 3.2222 - acc: 0.0174
Epoch 2/20
- 1s - loss: 3.1757 - acc: 0.0187
Epoch 3/20
- 1s - loss: 3.1731 - acc: 0.0212
Epoch 4/20
- 1s - loss: 3.1721 - acc: 0.0220
Epoch 5/20
- 1s - loss: 3.1716 - acc: 0.0225
Epoch 6/20
- 1s - loss: 3.1711 - acc: 0.0235
Epoch 7/20
- 1s - loss: 3.1698 - acc: 0.0245
Epoch 8/20
- 1s - loss: 3.1690 - acc: 0.0251
Epoch 9/20
- 1s - loss: 3.1686 - acc: 0.0257
Epoch 10/20
- 1s - loss: 3.1679 - acc: 0.0261
Epoch 11/20
- 1s - loss: 3.1674 - acc: 0.0267
Epoch 12/20
- 1s - loss: 3.1667 - acc: 0.0277
Epoch 13/20
- 1s - loss: 3.1656 - acc: 0.0285
Epoch 14/20
- 1s - loss: 3.1653 - acc: 0.0288
Epoch 15/20
- 1s - loss: 3.1653 - acc: 0.0291
I used Sklearn library to build the same structure with the same data, and it works perfectly, shown me an accuracy higher than 0.5:
model = Pipeline([
('classifier', MLPClassifier(hidden_layer_sizes=(1000), activation='relu',
max_iter=20, verbose=2, batch_size=700, random_state=0))
])
I'm totally sure that I used the same data for both models, and this is how I prepare it:
def load_data():
le = preprocessing.LabelEncoder()
with open('_DATA_train.txt', 'rb') as fp:
train = pickle.load(fp)
with open('_DATA_test.txt', 'rb') as fp:
test = pickle.load(fp)
x_train = train[:,0:(train.shape[1]-1)]
y_train = train[:,(train.shape[1]-1)]
y_train = le.fit_transform(y_train).reshape([-1,1])
x_test = test[:,0:(test.shape[1]-1)]
y_test = test[:,(test.shape[1]-1)]
y_test = le.fit_transform(y_test).reshape([-1,1])
print(x_train.shape, ' ' , y_train.shape)
print(x_test.shape, ' ' , y_test.shape)
return x_train, y_train, x_test, y_test
What is the problem with the Keras structure?
Edited:
it's a multi-class classification problem: y_training [0 ,1, 2, 3]
For a multiclass problem your labels should be one hot encoded. For example if the options are [0 ,1, 2, 3] and the label is 1 then it should be [0, 1, 0, 0].
Your final layer should be a dense layer with 4 units and an activation of softmax.
model.add(Dense(4, activation='softmax'))
And your loss should be categorical_crossentropy
model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer='RMSprop')
Related
I'm working on a multiclass text classification problem.
After splitting the data to train and validation data frames, I've performed text augmentation to balance the data (only on the train data of course).
I've ended up with a balanced trained data and 44325 samples (of trained data).
Later on I've applied the "clean text" task for getting (i.e. stemming and stuff) on the trained data.
train['text'] = train['text'].apply(clean_text)
X_train = train.iloc[:, :-1]
y_train = train.iloc[:, -1:]
X_test = valid.iloc[:, :-1]
y_test = valid.iloc[:, -1:]
y_test = pd.DataFrame(y_test).reset_index(drop=True)
tokenizer = Tokenizer(num_words=vocab_size, oov_token='<OOV>')
tokenizer.fit_on_texts(X_train['text'])
train_seq = tokenizer.texts_to_sequences(X_train['text'])
train_padded = pad_sequences(train_seq, maxlen=max_length, padding=padding_type, truncating=trunc_type)
validation_seq = tokenizer.texts_to_sequences(X_test['text'])
validation_padded = pad_sequences(validation_seq, maxlen=max_length, padding=padding_type, truncating=trunc_type)
print('Shape of train data tensor:', train_padded.shape)
print('Shape of validation data tensor:', validation_padded.shape)
Output:
Shape of train data tensor: (44325, 200)
Shape of validation data tensor: (5466, 200)
Here's the encoding section:
encode = OneHotEncoder()
training_labels = encode.fit_transform(y_train)
validation_labels = encode.transform(y_test)
training_labels = training_labels.toarray()
validation_labels = validation_labels.toarray()
Model:
model = Sequential()
model.add(Embedding(vocab_size, embedding_dim, input_length=train_padded.shape[1]))
model.add(Conv1D(48, len(GROUPS), activation='relu', padding='valid'))
model.add(GlobalMaxPooling1D())
model.add(Dropout(0.5))
model.add(Flatten())
model.add(Dropout(0.5))
model.add(Dense(len(GROUPS), activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
epochs = 100
batch_size = 32
history = model.fit(train_padded, training_labels, shuffle=True ,
epochs=epochs, batch_size=batch_size,
validation_split=0.2,
validation_data=(validation_padded, validation_labels),
callbacks=[ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, min_lr=0.0001),
EarlyStopping(monitor='val_loss', mode='min', patience=2, verbose=1),
EarlyStopping(monitor='val_accuracy', mode='max', patience=5, verbose=1)])
Model output:
Epoch 1/100
1109/1109 [==============================] - 88s 79ms/step - loss: 1.2021 - accuracy: 0.5235 - val_loss: 0.8374 - val_accuracy: 0.7232
Epoch 2/100
1109/1109 [==============================] - 87s 79ms/step - loss: 0.9505 - accuracy: 0.6645 - val_loss: 0.7488 - val_accuracy: 0.7461
Epoch 3/100
1109/1109 [==============================] - 86s 77ms/step - loss: 0.8378 - accuracy: 0.7058 - val_loss: 0.6686 - val_accuracy: 0.7663
Epoch 4/100
1109/1109 [==============================] - 88s 79ms/step - loss: 0.7391 - accuracy: 0.7382 - val_loss: 0.6134 - val_accuracy: 0.7891
Epoch 5/100
1109/1109 [==============================] - 89s 80ms/step - loss: 0.6763 - accuracy: 0.7546 - val_loss: 0.5832 - val_accuracy: 0.7997
Epoch 6/100
1109/1109 [==============================] - 87s 79ms/step - loss: 0.6185 - accuracy: 0.7760 - val_loss: 0.5529 - val_accuracy: 0.8050
Epoch 7/100
1109/1109 [==============================] - 87s 79ms/step - loss: 0.5737 - accuracy: 0.7912 - val_loss: 0.5311 - val_accuracy: 0.8153
Epoch 8/100
1109/1109 [==============================] - 88s 80ms/step - loss: 0.5226 - accuracy: 0.8080 - val_loss: 0.5268 - val_accuracy: 0.8226
Epoch 9/100
1109/1109 [==============================] - 88s 79ms/step - loss: 0.4955 - accuracy: 0.8171 - val_loss: 0.5142 - val_accuracy: 0.8285
Epoch 10/100
1109/1109 [==============================] - 88s 80ms/step - loss: 0.4665 - accuracy: 0.8265 - val_loss: 0.5035 - val_accuracy: 0.8338
Epoch 11/100
1109/1109 [==============================] - 88s 79ms/step - loss: 0.4410 - accuracy: 0.8348 - val_loss: 0.5082 - val_accuracy: 0.8399
Epoch 12/100
1109/1109 [==============================] - 88s 80ms/step - loss: 0.4190 - accuracy: 0.8407 - val_loss: 0.5160 - val_accuracy: 0.8414
Epoch 00012: early stopping
... and to the last part I'm unsure of:
def evaluate_preds(y_true, y_preds):
"""
Performs evaluation comparison on y_true labels vs. y_pred labels
on a classification.
"""
accuracy = accuracy_score(y_true, y_preds)
precision = precision_score(y_true, y_preds, average='micro')
recall = recall_score(y_true, y_preds, average='micro')
f1 = f1_score(y_true, y_preds, average='micro')
metric_dict = {"accuracy": round(accuracy, 2),
"precision": round(precision, 2),
"recall": round(recall, 2),
"f1": round(f1, 2)}
print(f"Acc: {accuracy * 100:.2f}%")
print(f"Precision: {precision:.2f}")
print(f"Recall: {recall:.2f}")
print(f"F1 score: {f1:.2f}")
return metric_dict
predicted = model.predict(validation_padded)
evaluate_preds(np.argmax(validation_labels, axis=1), np.argmax(predicted, axis=1))
Output:
Acc: 40.16%
Precision: 0.40
Recall: 0.40
F1 score: 0.40
I can't understand what am I doing wrong.
How come accuracy of the last mentioned method is so low compare to val_accuracy?
I run a tensorflow u-net model without dropout (but BN) with a custom metric called "average accuracy". This is literally the section of code. As you can see, datasets must be the same as I do nothing in between fit and evaluate.
model.fit(x=train_ds, epochs=epochs, validation_data=val_ds, shuffle=True,
callbacks=callbacks)
model.evaluate(train_ds)
model.evaluate(val_ds)
train_ds and val_ds are tf.Dataset. And here the output.
...
Epoch 10/10
148/148 [==============================] - 103s 698ms/step - loss: 0.1765 - accuracy: 0.5872 - average_accuracy: 0.9620 - val_loss: 0.5845 - val_accuracy: 0.5788 - val_average_accuracy: 0.5432
148/148 [==============================] - 22s 118ms/step - loss: 0.5056 - accuracy: 0.4540 - average_accuracy: 0.3654
29/29 [==============================] - 5s 122ms/step - loss: 0.5845 - accuracy: 0.5788 - average_accuracy: 0.5432
There is an unbelievable difference between average_accuracy during training (fit) and average_accuracy of evaluate (both on training dataset). I know that BN can have this effect and also that performance changes during training so they will never be equal. But from 96% to 36%?
My custom accuracy is defined here but I doubt it's my personal implementation as it should be somehow close no matter what I did (I think).
Any hint here is useful. I don't know if I should review the custom metric, the dataset, the model. It seems outside all of them.
I tried to continue training after stopping and average_accuracy starts from where it left at more than 90%.
Context of custom metric. I use it for semantic segmentation. So each image has an image of labels as output of WxHx4 (4 are my total number of classes).
It computes the average accuracy, for example, the accuracy of each class separately and then, if they were 4 classes it does sum(accuracies per class) / 4.
Here the main code:
def custom_average_accuracy(y_true, y_pred):
# Mask to remove the labels (y_true) that are zero: ex. [0, 0, 0]
remove_zeros_mask = tf.math.logical_not(tf.math.reduce_all(tf.math.logical_not(tf.cast(y_true, bool)), axis=-1))
y_true = tf.boolean_mask(y_true, remove_zeros_mask)
y_pred = tf.boolean_mask(y_pred, remove_zeros_mask)
num_cls = y_true.shape[-1]
y_pred = tf.math.argmax(y_pred, axis=-1) # ex. [0, 0, 1] -> [2]
y_true = tf.math.argmax(y_true, axis=-1)
accuracies = tf.TensorArray(tf.float32, size=0, dynamic_size=True)
for i in range(0, num_cls):
cls_mask = y_true == i
cls_y_true = tf.boolean_mask(y_true, cls_mask)
if not tf.equal(tf.size(cls_y_true), 0): # Some images don't have all the classes present.
new_acc = _accuracy(y_true=cls_y_true, y_pred=tf.boolean_mask(y_pred, cls_mask))
accuracies = accuracies.write(accuracies.size(), new_acc)
accuracies = accuracies.stack()
return tf.math.reduce_sum(accuracies) / tf.cast(len(accuracies), dtype=accuracies.dtype)
I believe the problem might be on the if not tf.equal(tf.size(cls_y_true), 0) line but I still can't seem were.
More wird information. This is exactly my lines of code:
x_input, y_true = np.concatenate([x for x, y in ds], axis=0), np.concatenate([y for x, y in ds], axis=0)
model.evaluate(x=x_input, y=y_true) # This gets 38% accuracy
model.evaluate(ds) # This gets 55% accuracy
What the hell is going on here? How can those lines of code give a different result?!?!
So now I have that if I don't do the ds = ds.shuffle() the example up (30ish vs 50ish ACC values) are Ok.
I tried to reproduce this behavior but could not find the discrepancies you noted. The only thing I changed was not tf.equal to tf.math.not_equal:
import pathlib
import tensorflow as tf
dataset_url = "https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz"
data_dir = tf.keras.utils.get_file('flower_photos', origin=dataset_url, untar=True)
data_dir = pathlib.Path(data_dir)
num_classes = 5
batch_size = 32
img_height = 180
img_width = 180
val_ds = tf.keras.utils.image_dataset_from_directory(
data_dir,
validation_split=0.2,
subset="validation",
seed=123,
image_size=(img_height, img_width),
batch_size=batch_size)
train_ds = tf.keras.utils.image_dataset_from_directory(
data_dir,
validation_split=0.2,
subset="training",
seed=123,
image_size=(img_height, img_width),
batch_size=batch_size)
def to_categorical(images, labels):
return images, tf.one_hot(labels, num_classes)
train_ds = train_ds.map(to_categorical)
val_ds = val_ds.map(to_categorical)
model = tf.keras.Sequential([
tf.keras.layers.Rescaling(1./255, input_shape=(img_height, img_width, 3)),
tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
tf.keras.layers.MaxPooling2D(),
tf.keras.layers.Dropout(0.3),
tf.keras.layers.Conv2D(32, 3, padding='same', activation='relu'),
tf.keras.layers.MaxPooling2D(),
tf.keras.layers.Dropout(0.3),
tf.keras.layers.Conv2D(64, 3, padding='same', activation='relu'),
tf.keras.layers.MaxPooling2D(),
tf.keras.layers.Dropout(0.3),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dense(num_classes)
])
def _accuracy(y_true, y_pred):
y_true.shape.assert_is_compatible_with(y_pred.shape)
if y_true.dtype != y_pred.dtype:
y_pred = tf.cast(y_pred, y_true.dtype)
reduced_sum = tf.reduce_sum(tf.cast(tf.math.equal(y_true, y_pred), tf.keras.backend.floatx()), axis=-1)
return tf.math.divide_no_nan(reduced_sum, tf.cast(tf.shape(y_pred)[-1], reduced_sum.dtype))
def custom_average_accuracy(y_true, y_pred):
# Mask to remove the labels (y_true) that are zero: ex. [0, 0, 0]
remove_zeros_mask = tf.math.logical_not(tf.math.reduce_all(tf.math.logical_not(tf.cast(y_true, bool)), axis=-1))
y_true = tf.boolean_mask(y_true, remove_zeros_mask)
y_pred = tf.boolean_mask(y_pred, remove_zeros_mask)
num_cls = y_true.shape[-1]
y_pred = tf.math.argmax(y_pred, axis=-1) # ex. [0, 0, 1] -> [2]
y_true = tf.math.argmax(y_true, axis=-1)
accuracies = tf.TensorArray(tf.float32, size=0, dynamic_size=True)
for i in range(0, num_cls):
cls_mask = y_true == i
cls_y_true = tf.boolean_mask(y_true, cls_mask)
if tf.math.not_equal(tf.size(cls_y_true), 0): # Some images don't have all the classes present.
new_acc = _accuracy(y_true=cls_y_true, y_pred=tf.boolean_mask(y_pred, cls_mask))
accuracies = accuracies.write(accuracies.size(), new_acc)
accuracies = accuracies.stack()
return tf.math.reduce_sum(accuracies) / tf.cast(len(accuracies), dtype=accuracies.dtype)
model.compile(optimizer='adam',
loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),
metrics=['accuracy', custom_average_accuracy])
epochs=10
history = model.fit(
train_ds,
validation_data=val_ds,
epochs=epochs)
model.evaluate(train_ds)
model.evaluate(val_ds)
Found 3670 files belonging to 5 classes.
Using 734 files for validation.
Found 3670 files belonging to 5 classes.
Using 2936 files for training.
Epoch 1/10
92/92 [==============================] - 11s 95ms/step - loss: 1.6220 - accuracy: 0.2868 - custom_average_accuracy: 0.2824 - val_loss: 1.2868 - val_accuracy: 0.4946 - val_custom_average_accuracy: 0.4597
Epoch 2/10
92/92 [==============================] - 8s 85ms/step - loss: 1.2131 - accuracy: 0.4785 - custom_average_accuracy: 0.4495 - val_loss: 1.2051 - val_accuracy: 0.4673 - val_custom_average_accuracy: 0.4350
Epoch 3/10
92/92 [==============================] - 8s 84ms/step - loss: 1.0713 - accuracy: 0.5620 - custom_average_accuracy: 0.5404 - val_loss: 1.1070 - val_accuracy: 0.5232 - val_custom_average_accuracy: 0.5003
Epoch 4/10
92/92 [==============================] - 8s 83ms/step - loss: 0.9463 - accuracy: 0.6281 - custom_average_accuracy: 0.6203 - val_loss: 0.9880 - val_accuracy: 0.5967 - val_custom_average_accuracy: 0.5755
Epoch 5/10
92/92 [==============================] - 8s 84ms/step - loss: 0.8400 - accuracy: 0.6771 - custom_average_accuracy: 0.6730 - val_loss: 0.9420 - val_accuracy: 0.6308 - val_custom_average_accuracy: 0.6245
Epoch 6/10
92/92 [==============================] - 8s 83ms/step - loss: 0.7594 - accuracy: 0.7027 - custom_average_accuracy: 0.7004 - val_loss: 0.8972 - val_accuracy: 0.6431 - val_custom_average_accuracy: 0.6328
Epoch 7/10
92/92 [==============================] - 8s 82ms/step - loss: 0.6211 - accuracy: 0.7619 - custom_average_accuracy: 0.7563 - val_loss: 0.8999 - val_accuracy: 0.6431 - val_custom_average_accuracy: 0.6174
Epoch 8/10
92/92 [==============================] - 8s 82ms/step - loss: 0.5108 - accuracy: 0.8116 - custom_average_accuracy: 0.8046 - val_loss: 0.8809 - val_accuracy: 0.6689 - val_custom_average_accuracy: 0.6457
Epoch 9/10
92/92 [==============================] - 8s 83ms/step - loss: 0.3985 - accuracy: 0.8535 - custom_average_accuracy: 0.8534 - val_loss: 0.9364 - val_accuracy: 0.6676 - val_custom_average_accuracy: 0.6539
Epoch 10/10
92/92 [==============================] - 8s 83ms/step - loss: 0.3023 - accuracy: 0.8995 - custom_average_accuracy: 0.9010 - val_loss: 1.0118 - val_accuracy: 0.6662 - val_custom_average_accuracy: 0.6405
92/92 [==============================] - 6s 62ms/step - loss: 0.2038 - accuracy: 0.9363 - custom_average_accuracy: 0.9357
23/23 [==============================] - 2s 50ms/step - loss: 1.0118 - accuracy: 0.6662 - custom_average_accuracy: 0.663
Well, I was using a TensorFlow dataset. I changed to NumPy and now all seems logical and works.
Still, I need to know the reason tf ds didn't work but at least I don't longer have these weird results.
Not tested yet (I would need to get the code back to what it was, probably do it someday) but this might be related.
Dataset description:
(1) X_train: (6000,4) shape
(2) y_train: (6000,4) shape
(3) X_validation: (2000,4) shape
(4) y_validation: (2000,4) shape
(5) X_test: (2000,4) shape
(6) y_test: (2000,4) shape
Relationship between X and Y is shown here
For single label classification, the activation function of the last layer is Softmax and the loss function is categorical_crossentrop.
And I know the mathematical calculation method for the loss function.
And for multi-class multi-label classification problems, the activation function of the last layer is sigmoid, and the loss function is binary_crossentrop.
I want to know how the mathematical calculation method of the loss function works
It would be a great help to me if you let me know.
def MinMaxScaler(data):
numerator = data - np.min(data)
denominator = np.max(data) - np.min(data)
return numerator / (denominator + 1e-5)
kki = pd.read_csv(filename,names=['UE0','UE1','UE2','UE3','selected_UE0','selected_UE1','selected_UE2','selected_UE3'])
print(kki)
def LoadData(file):
xy = np.loadtxt(file, delimiter=',', dtype=np.float32)
print("Data set length:", len(xy))
tr_set_size = int(len(xy) * 0.6)
xy[:, 0:-number_of_UEs] = MinMaxScaler(xy[:, 0:-number_of_UEs]) #number_of_UES : 4
X_train = xy[:tr_set_size, 0: -number_of_UEs] #6000 row
y_train = xy[:tr_set_size, number_of_UEs:number_of_UEs*2]
X_valid = xy[tr_set_size:int((tr_set_size/3) + tr_set_size), 0:-number_of_UEs]
y_valid = xy[tr_set_size:int((tr_set_size/3) + tr_set_size), number_of_UEs:number_of_UEs *2]
X_test = xy[int((tr_set_size/3) + tr_set_size):, 0:-number_of_UEs]
y_test = xy[int((tr_set_size/3) + tr_set_size):, number_of_UEs:number_of_UEs*2]
print("Training X shape:", X_train.shape)
print("Training Y shape:", y_train.shape)
print("validation x shape:", X_valid.shape)
print("validation y shape:", y_valid.shape)
print("Test X shape:", X_test.shape)
print("Test Y shape:", y_test.shape)
return X_train, y_train, X_valid, y_valid, X_test, y_test, tr_set_size
X_train, y_train, X_valid, y_valid, X_test, y_test, tr_set_size = LoadData(filename)
model = Sequential()
model.add(Dense(64,activation='relu', input_shape=(X_train.shape[1],)))
model.add(Dense(46, activation='relu'))
model.add(Dense(24, activation='relu'))
model.add(Dense(12, activation='relu'))
model.add(Dense(4, activation= 'sigmoid'))
model.compile( loss ='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
hist = model.fit(X_train, y_train, epochs=5, batch_size=1, verbose= 1, validation_data=(X_valid, y_valid), callbacks= es)
This is a learning process, and even if epochs are repeated,
Accuracy does not improve.
Epoch 1/10
6000/6000 [==============================] - 14s 2ms/step - loss: 0.2999 - accuracy: 0.5345 - val_loss: 0.1691 - val_accuracy: 0.5465
Epoch 2/10
6000/6000 [==============================] - 14s 2ms/step - loss: 0.1554 - accuracy: 0.4883 - val_loss: 0.1228 - val_accuracy: 0.4710
Epoch 3/10
6000/6000 [==============================] - 14s 2ms/step - loss: 0.1259 - accuracy: 0.4710 - val_loss: 0.0893 - val_accuracy: 0.4910
Epoch 4/10
6000/6000 [==============================] - 13s 2ms/step - loss: 0.1094 - accuracy: 0.4990 - val_loss: 0.0918 - val_accuracy: 0.5540
Epoch 5/10
6000/6000 [==============================] - 13s 2ms/step - loss: 0.0967 - accuracy: 0.5223 - val_loss: 0.0671 - val_accuracy: 0.5405
Epoch 6/10
6000/6000 [==============================] - 13s 2ms/step - loss: 0.0910 - accuracy: 0.5198 - val_loss: 0.0836 - val_accuracy: 0.5380
Epoch 7/10
6000/6000 [==============================] - 13s 2ms/step - loss: 0.0870 - accuracy: 0.5348 - val_loss: 0.0853 - val_accuracy: 0.5775
Epoch 8/10
6000/6000 [==============================] - 13s 2ms/step - loss: 0.0859 - accuracy: 0.5518 - val_loss: 0.0515 - val_accuracy: 0.6520
Epoch 9/10
6000/6000 [==============================] - 13s 2ms/step - loss: 0.0792 - accuracy: 0.5508 - val_loss: 0.0629 - val_accuracy: 0.4350
Epoch 10/10
6000/6000 [==============================] - 13s 2ms/step - loss: 0.0793 - accuracy: 0.5638 - val_loss: 0.0632 - val_accuracy: 0.6270
Mistake 1 -
The shape of y_train, y_validation and y_test should be (6000,), (2000,) and (2000,) respectively.
Mistake 2 -
For multi-class classification, the loss should be categorical_crossentropy and activation should be a softmax. So, change these two lines, like this:
model.add(Dense(4, activation= 'softmax'))
model.compile(loss ='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
Suggestion -
Why are you splitting data by yourself? Use scikit-learn train_test_split. This code will give you proper splits:
from sklearn.model_selection import train_test_split
x, x_test, y, y_test = train_test_split(xtrain, labels, test_size=0.2, train_size=0.8)
x_train, x_validation, y_train, y_validation = train_test_split(x, y, test_size = 0.25, train_size =0.75)
I have a time series imbalanced dataset on which I have to perform binary classification. I cannot split training and test sets randomly or even perform stratify on them. The issue is that while training the model validation accuracy is always 1. I realize this has somethings to do with the train-test split but I may be wrong.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.8, random_state=None, shuffle=False)
from collections import Counter
print(Counter(y))
print(Counter(y_train))
print(Counter(y_test))
Counter({0.0: 55534, 1.0: 10000})
Counter({0.0: 9995, 1.0: 3111})
Counter({0.0: 45539, 1.0: 6889})
model = Sequential()
#First Hidden Layer
model.add(Dense(128, activation='relu', kernel_initializer='random_normal', input_dim=19))
#model.add(Dropout(0.3))
#Second Hidden Layer
model.add(Dense(64, activation='relu', kernel_initializer='random_normal'))
#Output Layer
model.add(Dense(1, activation='sigmoid', kernel_initializer='random_normal'))
history = model.fit(X_train,y_train, batch_size=128, validation_split=0.1, epochs=50)
Train on 11795 samples, validate on 1311 samples
Epoch 1/50
11795/11795 [==============================] - 0s 34us/step - loss: 1.1359 - accuracy: 0.8719 - val_loss: 4.2016e-18 - val_accuracy: 1.0000
Epoch 2/50
11795/11795 [==============================] - 0s 12us/step - loss: 0.1247 - accuracy: 0.9442 - val_loss: 1.0255e-19 - val_accuracy: 1.0000
Epoch 3/50
11795/11795 [==============================] - 0s 13us/step - loss: 0.1177 - accuracy: 0.9462 - val_loss: 3.2516e-23 - val_accuracy: 1.0000
Epoch 4/50
11795/11795 [==============================] - 0s 12us/step - loss: 0.1103 - accuracy: 0.9519 - val_loss: 1.1607e-23 - val_accuracy: 1.0000
Epoch 5/50
11795/11795 [==============================] - 0s 13us/step - loss: 0.0805 - accuracy: 0.9739 - val_loss: 6.2345e-26 - val_accuracy: 1.0000
Appreciate any help on this problem.Thanks
I want to predict 8-characters license plates, so I wrote the below model in Keras:
x = Input(shape=(HEIGHT, WIDTH, CHANNELS))
base_model = InceptionV3(include_top=False, weights='imagenet', input_shape=(HEIGHT, WIDTH, CHANNELS))
base_model.trainable = False
y = base_model(x)
y = Reshape((8, 9 * 256))(y)
y = LSTM(units=20, return_sequences='true')(y)
y = Dropout(0.5)(y)
y = TimeDistributed(Dense(TOTAL_CHARS, activation="softmax", activity_regularizer=regularizers.l2(REGUL_PARAM)))(y)
y = Dropout(0.25)(y)
model = Model(input=x, output=y)
model.compile(loss="categorical_crossentropy", optimizer='rmsprop', metrics=['accuracy'])
I have about 6000 data for training which I augment them with ImageGenerator. My problem is that the loss and accuracy are approximately constant during time:
************************************************************
Epoch: 1
************************************************************
Train on 6869 samples, validate on 1718 samples
Epoch 1/1
6856/6869 [============================>.] - ETA: 0s - loss: 5.4525 - acc: 0.1924Epoch 00001: val_loss improved from 2.17175 to 2.15020, saving model to ./trained_model_V10.hdf5
6869/6869 [==============================] - 25s 4ms/step - loss: 5.4535 - acc: 0.1924 - val_loss: 2.1502 - val_acc: 0.2232
************************************************************
Epoch: 2
************************************************************
Train on 6869 samples, validate on 1718 samples
Epoch 1/1
6848/6869 [============================>.] - ETA: 0s - loss: 5.4543 - acc: 0.1959Epoch 00001: val_loss improved from 2.15020 to 2.11809, saving model to ./trained_model_V10.hdf5
6869/6869 [==============================] - 26s 4ms/step - loss: 5.4537 - acc: 0.1958 - val_loss: 2.1181 - val_acc: 0.2281
************************************************************
Epoch: 3
************************************************************
Train on 6869 samples, validate on 1718 samples
Epoch 1/1
6856/6869 [============================>.] - ETA: 0s - loss: 5.4284 - acc: 0.1977Epoch 00001: val_loss improved from 2.11809 to 2.09679, saving model to ./trained_model_V10.hdf5
6869/6869 [==============================] - 25s 4ms/step - loss: 5.4282 - acc: 0.1978 - val_loss: 2.0968 - val_acc: 0.2304
************************************************************
Epoch: 4
************************************************************
Train on 6869 samples, validate on 1718 samples
Epoch 1/1
6856/6869 [============================>.] - ETA: 0s - loss: 5.4500 - acc: 0.2004Epoch 00001: val_loss did not improve
6869/6869 [==============================] - 25s 4ms/step - loss: 5.4490 - acc: 0.2004 - val_loss: 2.1146 - val_acc: 0.2355
************************************************************
Epoch: 5
************************************************************
Train on 6869 samples, validate on 1718 samples
Epoch 1/1
6848/6869 [============================>.] - ETA: 0s - loss: 5.4399 - acc: 0.2006Epoch 00001: val_loss did not improve
6869/6869 [==============================] - 25s 4ms/step - loss: 5.4374 - acc: 0.2009 - val_loss: 2.1102 - val_acc: 0.2324
************************************************************
Epoch: 6
************************************************************
Train on 6869 samples, validate on 1718 samples
Epoch 1/1
6856/6869 [============================>.] - ETA: 0s - loss: 5.4636 - acc: 0.1977Epoch 00001: val_loss improved from 2.09679 to 2.09076, saving model to ./trained_model_V10.hdf5
6869/6869 [==============================] - 25s 4ms/step - loss: 5.4629 - acc: 0.1978 - val_loss: 2.0908 - val_acc: 0.2341
************************************************************
Now, I am not sure exactly the correctness of my model and I think the problem is my model. Is this the correct way to combine CNN and LSTM?
I also have tried the below mode:
REGUL_PARAM = 0
image = Input(shape=(HEIGHT, WIDTH, CHANNELS))
x = Reshape((8, HEIGHT, int(WIDTH/8), CHANNELS))(image)
y = TimeDistributed(Conv2D(16, (3, 3), activation='relu', padding='same', activity_regularizer=regularizers.l2(REGUL_PARAM)))(x)
y = TimeDistributed(MaxPooling2D((2, 2)))(y)
y = TimeDistributed(Conv2D(32, (3, 3), activation='relu', padding='same', activity_regularizer=regularizers.l2(REGUL_PARAM)))(y)
y = TimeDistributed(MaxPooling2D((2, 2)))(y)
y = TimeDistributed(Conv2D(64, (3, 3), activation='relu', padding='same', activity_regularizer=regularizers.l2(REGUL_PARAM)))(y)
y = Reshape((int(y.shape[1]), int(y.shape[4]*y.shape[3]*y.shape[2])))(y)
y = Bidirectional(LSTM(units=50, return_sequences='true'))(y)
y = TimeDistributed(Dense(64, activity_regularizer=regularizers.l2(REGUL_PARAM), activation='relu'))(y)
y = Dropout(0.25)(y)
y = TimeDistributed(Dense(TOTAL_CHARS, activity_regularizer=regularizers.l2(REGUL_PARAM), activation='softmax'))(y)
y = Dropout(0.25)(y)
model = Model(inputs=image, outputs=y)
the accuracy for this is about 70%, but the point is that I cannot overfit even on a small potion of my data.
Apparently, your model doesn't work well.
You may take a look at this code.
'''Train a recurrent convolutional network on the IMDB sentiment
classification task.
Gets to 0.8498 test accuracy after 2 epochs. 41s/epoch on K520 GPU.
'''
from __future__ import print_function
from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation
from keras.layers import Embedding
from keras.layers import LSTM
from keras.layers import Conv1D, MaxPooling1D
from keras.datasets import imdb
# Embedding
max_features = 20000
maxlen = 100
embedding_size = 128
# Convolution
kernel_size = 5
filters = 64
pool_size = 4
# LSTM
lstm_output_size = 70
# Training
batch_size = 30
epochs = 2
'''
Note:
batch_size is highly sensitive.
Only 2 epochs are needed as the dataset is very small.
'''
print('Loading data...')
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)
print(len(x_train), 'train sequences')
print(len(x_test), 'test sequences')
print('Pad sequences (samples x time)')
x_train = sequence.pad_sequences(x_train, maxlen=maxlen)
x_test = sequence.pad_sequences(x_test, maxlen=maxlen)
print('x_train shape:', x_train.shape)
print('x_test shape:', x_test.shape)
print('Build model...')
model = Sequential()
model.add(Embedding(max_features, embedding_size, input_length=maxlen))
model.add(Dropout(0.25))
model.add(Conv1D(filters,
kernel_size,
padding='valid',
activation='relu',
strides=1))
model.add(MaxPooling1D(pool_size=pool_size))
model.add(LSTM(lstm_output_size))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='adam',
metrics=['accuracy'])
print('Train...')
model.fit(x_train, y_train,
batch_size=batch_size,
epochs=epochs,
validation_data=(x_test, y_test))
score, acc = model.evaluate(x_test, y_test, batch_size=batch_size)
print('Test score:', score)
print('Test accuracy:', acc)