I am using Tensorflow 2.3 and trying to save model checkpoint after n number of epochs. n can be anything but for now trying with 10
Per this thread, I tried save_freq = 'epoch' and period = 10 which works but since period parameter is deprecated, I wanted to try an alternative approach.
HEIGHT = 256
WIDTH = 256
CHANNELS = 3
EPOCHS = 100
BATCH_SIZE = 1
SAVE_PERIOD = 10
n_monet_samples = 21
checkpoint_filepath = "./model_checkpoints/cyclegan_checkpoints.{epoch:03d}"
model_checkpoint_callback = callbacks.ModelCheckpoint(
filepath=checkpoint_filepath,
save_freq=SAVE_PERIOD * (n_monet_samples//BATCH_SIZE)
)
If I use save_freq=SAVE_PERIOD * (n_monet_samples//BATCH_SIZE) for the checkpoint callback definition, I get error
ValueError: Unrecognized save_freq: 210
I am not sure why since per Keras callback code, as long as save_freq is in epochs or in integer, it should be good.
Please suggest.
It does not show any error to me when I tried the same code in same Tensorflow version==2.3:
checkpoint_path = "training_1/cp.ckpt"
checkpoint_dir = os.path.dirname(checkpoint_path)
BATCH_SIZE = 1
SAVE_PERIOD = 10
n_monet_samples = 21
# Create a callback that saves the model's weights
cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,
save_weights_only=True,
verbose=1, save_freq=SAVE_PERIOD * (n_monet_samples//BATCH_SIZE))
# Train the model with the new callback
model.fit(train_images,
train_labels,
epochs=20,
validation_data=(test_images, test_labels),
callbacks=[cp_callback])
Output:
Epoch 1/20
32/32 [==============================] - 0s 14ms/step - loss: 1.1152 - sparse_categorical_accuracy: 0.6890 - val_loss: 0.6934 - val_sparse_categorical_accuracy: 0.7940
Epoch 2/20
32/32 [==============================] - 0s 9ms/step - loss: 0.4154 - sparse_categorical_accuracy: 0.8840 - val_loss: 0.5317 - val_sparse_categorical_accuracy: 0.8330
Epoch 3/20
32/32 [==============================] - 0s 8ms/step - loss: 0.2787 - sparse_categorical_accuracy: 0.9270 - val_loss: 0.4854 - val_sparse_categorical_accuracy: 0.8400
Epoch 4/20
32/32 [==============================] - 0s 8ms/step - loss: 0.2230 - sparse_categorical_accuracy: 0.9420 - val_loss: 0.4525 - val_sparse_categorical_accuracy: 0.8590
Epoch 5/20
32/32 [==============================] - 0s 10ms/step - loss: 0.1549 - sparse_categorical_accuracy: 0.9620 - val_loss: 0.4275 - val_sparse_categorical_accuracy: 0.8650
Epoch 6/20
32/32 [==============================] - 0s 10ms/step - loss: 0.1110 - sparse_categorical_accuracy: 0.9770 - val_loss: 0.4251 - val_sparse_categorical_accuracy: 0.8630
Epoch 7/20
11/32 [=========>....................] - ETA: 0s - loss: 0.0936 - sparse_categorical_accuracy: 0.9886
Epoch 00007: saving model to training_1/cp.ckpt
32/32 [==============================] - 0s 14ms/step - loss: 0.0807 - sparse_categorical_accuracy: 0.9840 - val_loss: 0.4248 - val_sparse_categorical_accuracy: 0.8610
Epoch 8/20
32/32 [==============================] - 0s 10ms/step - loss: 0.0612 - sparse_categorical_accuracy: 0.9950 - val_loss: 0.4058 - val_sparse_categorical_accuracy: 0.8650
Epoch 9/20
32/32 [==============================] - 0s 8ms/step - loss: 0.0489 - sparse_categorical_accuracy: 0.9950 - val_loss: 0.4393 - val_sparse_categorical_accuracy: 0.8610
Epoch 10/20
32/32 [==============================] - 0s 6ms/step - loss: 0.0361 - sparse_categorical_accuracy: 1.0000 - val_loss: 0.4150 - val_sparse_categorical_accuracy: 0.8620
Epoch 11/20
32/32 [==============================] - 0s 10ms/step - loss: 0.0294 - sparse_categorical_accuracy: 1.0000 - val_loss: 0.4090 - val_sparse_categorical_accuracy: 0.8670
Epoch 12/20
32/32 [==============================] - 0s 7ms/step - loss: 0.0272 - sparse_categorical_accuracy: 0.9990 - val_loss: 0.4365 - val_sparse_categorical_accuracy: 0.8600
Epoch 13/20
32/32 [==============================] - 0s 8ms/step - loss: 0.0203 - sparse_categorical_accuracy: 1.0000 - val_loss: 0.4231 - val_sparse_categorical_accuracy: 0.8620
Epoch 14/20
1/32 [..............................] - ETA: 0s - loss: 0.0115 - sparse_categorical_accuracy: 1.0000
Epoch 00014: saving model to training_1/cp.ckpt
32/32 [==============================] - 0s 9ms/step - loss: 0.0164 - sparse_categorical_accuracy: 1.0000 - val_loss: 0.4263 - val_sparse_categorical_accuracy: 0.8650
Epoch 15/20
32/32 [==============================] - 0s 7ms/step - loss: 0.0128 - sparse_categorical_accuracy: 1.0000 - val_loss: 0.4260 - val_sparse_categorical_accuracy: 0.8690
Epoch 16/20
32/32 [==============================] - 0s 7ms/step - loss: 0.0120 - sparse_categorical_accuracy: 1.0000 - val_loss: 0.4194 - val_sparse_categorical_accuracy: 0.8740
Epoch 17/20
32/32 [==============================] - 0s 9ms/step - loss: 0.0110 - sparse_categorical_accuracy: 1.0000 - val_loss: 0.4302 - val_sparse_categorical_accuracy: 0.8710
Epoch 18/20
32/32 [==============================] - 0s 6ms/step - loss: 0.0090 - sparse_categorical_accuracy: 1.0000 - val_loss: 0.4331 - val_sparse_categorical_accuracy: 0.8660
Epoch 19/20
32/32 [==============================] - 0s 7ms/step - loss: 0.0084 - sparse_categorical_accuracy: 1.0000 - val_loss: 0.4320 - val_sparse_categorical_accuracy: 0.8760
Epoch 20/20
16/32 [==============>...............] - ETA: 0s - loss: 0.0074 - sparse_categorical_accuracy: 1.0000
Epoch 00020: saving model to training_1/cp.ckpt
32/32 [==============================] - 0s 13ms/step - loss: 0.0072 - sparse_categorical_accuracy: 1.0000 - val_loss: 0.4280 - val_sparse_categorical_accuracy: 0.8750
<tensorflow.python.keras.callbacks.History at 0x7f90f0082cd0>
As you already know save_freq is equal to 'epoch' or integer. When using 'epoch', the callback saves the model after each epoch. When using integer, the callback saves the model at end of theses many batches(end of these many steps_per_epoch).
As above definition of save_freq, checkpoints saves every after 210 steps.
Please check this for more details on ModelCheckpoint Arguments.
Here is my code and result of the training.
batch_size = 100
epochs = 50
yale_history = yale_classifier.fit(x_train, y_train_oh,batch_size=batch_size,epochs=epochs,validation_data=(x_train,y_train_oh))
Epoch 1/50
20/20 [==============================] - 32s 2s/step - loss: 3.9801 - accuracy: 0.2071 - val_loss: 3.6919 - val_accuracy: 0.0245
Epoch 2/50
20/20 [==============================] - 30s 2s/step - loss: 1.2557 - accuracy: 0.6847 - val_loss: 4.1914 - val_accuracy: 0.0245
Epoch 3/50
20/20 [==============================] - 30s 2s/step - loss: 0.4408 - accuracy: 0.8954 - val_loss: 4.6284 - val_accuracy: 0.0245
Epoch 4/50
20/20 [==============================] - 30s 2s/step - loss: 0.1822 - accuracy: 0.9592 - val_loss: 4.9481 - val_accuracy: 0.0398
Epoch 5/50
20/20 [==============================] - 30s 2s/step - loss: 0.1252 - accuracy: 0.9760 - val_loss: 5.3728 - val_accuracy: 0.0276
Epoch 6/50
20/20 [==============================] - 30s 2s/step - loss: 0.0927 - accuracy: 0.9816 - val_loss: 5.7009 - val_accuracy: 0.0260
Epoch 7/50
20/20 [==============================] - 30s 2s/step - loss: 0.0858 - accuracy: 0.9837 - val_loss: 6.0049 - val_accuracy: 0.0260
Epoch 8/50
20/20 [==============================] - 30s 2s/step - loss: 0.0646 - accuracy: 0.9867 - val_loss: 6.3786 - val_accuracy: 0.0260
Epoch 9/50
20/20 [==============================] - 30s 2s/step - loss: 0.0489 - accuracy: 0.9898 - val_loss: 6.5156 - val_accuracy: 0.0260
You can see that I also used the training data as the validation data. This is weird that the training loss is not the same as the validation loss. Further, when I evaluated it, seem like my model was not trained at all as follow.
yale_classifier.evaluate(x_train, y_train_oh)
62/62 [==============================] - 6s 96ms/step - loss: 7.1123 - accuracy: 0.0260
[7.112329483032227, 0.026020407676696777]
Do you have any recommened to solve this problem ?
I followed this tutorial https://medium.com/the-andela-way/deep-learning-hello-world-e1fc53ea888 to experiment with Keras, the source code is here https://github.com/sirghiny/mnist
However I received a very low score and the training was very short, as if the model is trained on very few samples. Here is the output in terminal:
Epoch 1/5
300/300 [==============================] - 12s 39ms/step - loss: 2.3791 - categorical_accuracy: 0.0899 - val_loss: 2.3104 - val_categorical_accuracy: 0.0528
Epoch 2/5
300/300 [==============================] - 11s 38ms/step - loss: 2.3326 - categorical_accuracy: 0.1060 - val_loss: 2.2920 - val_categorical_accuracy: 0.0864
Epoch 3/5
300/300 [==============================] - 10s 32ms/step - loss: 2.2891 - categorical_accuracy: 0.1315 - val_loss: 2.2742 - val_categorical_accuracy: 0.1571
Epoch 4/5
300/300 [==============================] - 9s 31ms/step - loss: 2.2510 - categorical_accuracy: 0.1576 - val_loss: 2.2569 - val_categorical_accuracy: 0.2367
Epoch 5/5
300/300 [==============================] - 9s 30ms/step - loss: 2.2174 - categorical_accuracy: 0.1889 - val_loss: 2.2397 - val_categorical_accuracy: 0.3133
Evaluating the model...
1250/1250 [==============================] - 2s 2ms/step - loss: 2.2382 - categorical_accuracy: 0.3171
938/938 [==============================] - 2s 2ms/step - loss: 2.2369 - categorical_accuracy: 0.3232
Please tell me what did i do wrong?
You updated your model weight only 1500 times (epochs*number_of_batch).
You might want to increase the epochs or/and reduce the batch_size to perform more weights' update as we see in your logs that the network is still learning.
Additionally, you should find an up-to-date tutorial like this one as TensorFlow changed a lot recently.
I am really new with TensorFlow and model building and training. However, I was following a tutorial and everything went well until at one point I got the following error:
2020-04-29 17:24:35.235550: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
I have no clue what is causing the error. The code I am using is this:
import tensorflow as tf
from tensorflow.keras.optimizers import RMSprop
import keras_preprocessing
from keras_preprocessing import image
from keras.preprocessing.image import ImageDataGenerator
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import os
from PIL import Image
training_datagen = ImageDataGenerator(rescale=1. / 255)
validation_datagen = ImageDataGenerator(rescale=1. / 255)
# Here I am giving the path of the images to train the model
train_dir = r"C:\Users\User\Desktop\Project\Project\dataset\train"
train_gen = training_datagen.flow_from_directory(train_dir, target_size=(150, 150), class_mode="categorical")
val_dir = r"C:\Users\User\Desktop\Project\Project\dataset\validation"
val_gen = training_datagen.flow_from_directory(val_dir, target_size=(150, 150), class_mode="categorical")
# Here I am training the model with individual fruits
train_apple_dir = r"C:\Users\User\Desktop\Project\Project\dataset\train\Apple"
train_banana_dir = r"C:\Users\User\Desktop\Project\Project\dataset\train\Banana"
# printing the number of apples in train dataset
number_apples_train = len(os.listdir(train_apple_dir))
print("total training apple images:", number_apples_train)
number_banana_train = len(os.listdir(train_banana_dir))
print("total training apple images:", number_banana_train)
# Here I am getting the first 10 names of apple images
apple_names = os.listdir(train_apple_dir)
print(apple_names[:10])
# Building the model
model = tf.keras.models.Sequential([
tf.keras.layers.Conv2D(64, (3, 3), activation="relu", input_shape=(150, 150, 3)),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Conv2D(64, (3, 3), activation="relu"),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Conv2D(128, (3, 3), activation="relu"),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Conv2D(128, (3, 3), activation="relu"),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(512, activation="relu"),
tf.keras.layers.Dense(15, activation="softmax")
])
model.summary()
model.compile(loss="categorical_crossentropy", optimizer='rmsprop', metrics=['accuracy'])
fruit_model = model.fit(train_gen, epochs=1, verbose=1, validation_data=val_gen, workers=10)
Full error traceback:
C:\Users\User\anaconda3\envs\project-env\python.exe
C:/Users/User/Desktop/Project/2ndYearProject/fruit_classifier.py
Using TensorFlow backend.
Found 7765 images belonging to 15 classes.
Found 7765 images belonging to 15 classes.
total training apple images: 492
total training apple images: 490
['0_100.jpg', '100_100.jpg', '101_100.jpg', '102_100.jpg', '103_100.jpg', '104_100.jpg', '105_100.jpg', '106_100.jpg', '107_100.jpg', '108_100.jpg']
2020-04-29 17:21:17.562203: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 148, 148, 64) 1792
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 74, 74, 64) 0
_________________________________________________________________
conv2d_1 (Conv2D) (None, 72, 72, 64) 36928
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 36, 36, 64) 0
_________________________________________________________________
conv2d_2 (Conv2D) (None, 34, 34, 128) 73856
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 17, 17, 128) 0
_________________________________________________________________
conv2d_3 (Conv2D) (None, 15, 15, 128) 147584
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 7, 7, 128) 0
_________________________________________________________________
flatten (Flatten) (None, 6272) 0
_________________________________________________________________
dense (Dense) (None, 512) 3211776
_________________________________________________________________
dense_1 (Dense) (None, 15) 7695
=================================================================
Total params: 3,479,631
Trainable params: 3,479,631
Non-trainable params: 0
_________________________________________________________________
WARNING:tensorflow:sample_weight modes were coerced from
...
to
['...']
WARNING:tensorflow:sample_weight modes were coerced from
...
to
['...']
Train for 243 steps, validate for 243 steps
1/243 [..............................] - ETA: 6:49 - loss: 2.7038 - accuracy: 0.1875
2/243 [..............................] - ETA: 4:34 - loss: 3.4685 - accuracy: 0.1406
3/243 [..............................] - ETA: 3:47 - loss: 3.1995 - accuracy: 0.1562
4/243 [..............................] - ETA: 3:26 - loss: 3.0967 - accuracy: 0.1172
5/243 [..............................] - ETA: 3:11 - loss: 3.0149 - accuracy: 0.1187
6/243 [..............................] - ETA: 3:02 - loss: 2.9531 - accuracy: 0.1094
7/243 [..............................] - ETA: 2:54 - loss: 2.9210 - accuracy: 0.0982
8/243 [..............................] - ETA: 2:49 - loss: 2.8854 - accuracy: 0.1094
9/243 [>.............................] - ETA: 2:46 - loss: 2.8430 - accuracy: 0.1181
10/243 [>.............................] - ETA: 2:46 - loss: 2.8042 - accuracy: 0.1312
11/243 [>.............................] - ETA: 2:43 - loss: 2.8623 - accuracy: 0.1307
12/243 [>.............................] - ETA: 2:41 - loss: 2.8341 - accuracy: 0.1354
13/243 [>.............................] - ETA: 2:39 - loss: 2.7907 - accuracy: 0.1611
14/243 [>.............................] - ETA: 2:37 - loss: 2.7307 - accuracy: 0.1763
15/243 [>.............................] - ETA: 2:36 - loss: 2.6982 - accuracy: 0.1917
16/243 [>.............................] - ETA: 2:35 - loss: 2.6755 - accuracy: 0.1855
17/243 [=>............................] - ETA: 2:34 - loss: 2.6318 - accuracy: 0.1930
18/243 [=>............................] - ETA: 2:33 - loss: 2.5757 - accuracy: 0.2083
19/243 [=>............................] - ETA: 2:31 - loss: 2.5932 - accuracy: 0.2122
20/243 [=>............................] - ETA: 2:30 - loss: 2.5570 - accuracy: 0.2234
21/243 [=>............................] - ETA: 2:29 - loss: 2.5350 - accuracy: 0.2321
22/243 [=>............................] - ETA: 2:28 - loss: 2.4881 - accuracy: 0.2500
23/243 [=>............................] - ETA: 2:26 - loss: 2.4220 - accuracy: 0.2704
24/243 [=>............................] - ETA: 2:25 - loss: 2.3835 - accuracy: 0.2799
25/243 [==>...........................] - ETA: 2:24 - loss: 2.3830 - accuracy: 0.2825
26/243 [==>...........................] - ETA: 2:22 - loss: 2.3481 - accuracy: 0.2885
27/243 [==>...........................] - ETA: 2:21 - loss: 2.2924 - accuracy: 0.3090
28/243 [==>...........................] - ETA: 2:20 - loss: 2.2524 - accuracy: 0.3270
29/243 [==>...........................] - ETA: 2:18 - loss: 2.2276 - accuracy: 0.3287
30/243 [==>...........................] - ETA: 2:17 - loss: 2.2073 - accuracy: 0.3344
31/243 [==>...........................] - ETA: 2:16 - loss: 2.1791 - accuracy: 0.3407
32/243 [==>...........................] - ETA: 2:15 - loss: 2.1395 - accuracy: 0.3516
33/243 [===>..........................] - ETA: 2:14 - loss: 2.0994 - accuracy: 0.3636
34/243 [===>..........................] - ETA: 2:13 - loss: 2.0554 - accuracy: 0.3759
35/243 [===>..........................] - ETA: 2:12 - loss: 2.0124 - accuracy: 0.3893
36/243 [===>..........................] - ETA: 2:11 - loss: 1.9763 - accuracy: 0.4002
37/243 [===>..........................] - ETA: 2:10 - loss: 1.9455 - accuracy: 0.4079
38/243 [===>..........................] - ETA: 2:09 - loss: 1.9066 - accuracy: 0.4194
39/243 [===>..........................] - ETA: 2:08 - loss: 1.8678 - accuracy: 0.4311
40/243 [===>..........................] - ETA: 2:07 - loss: 1.8342 - accuracy: 0.4406
41/243 [====>.........................] - ETA: 2:07 - loss: 1.8034 - accuracy: 0.4505
42/243 [====>.........................] - ETA: 2:06 - loss: 1.7719 - accuracy: 0.4591
43/243 [====>.........................] - ETA: 2:05 - loss: 1.7409 - accuracy: 0.4680
44/243 [====>.........................] - ETA: 2:04 - loss: 1.7135 - accuracy: 0.4766
45/243 [====>.........................] - ETA: 2:03 - loss: 1.6833 - accuracy: 0.4829
46/243 [====>.........................] - ETA: 2:02 - loss: 1.6609 - accuracy: 0.4901
47/243 [====>.........................] - ETA: 2:01 - loss: 1.6461 - accuracy: 0.4930
48/243 [====>.........................] - ETA: 2:00 - loss: 1.6200 - accuracy: 0.5003
49/243 [=====>........................] - ETA: 1:59 - loss: 1.6031 - accuracy: 0.5048
50/243 [=====>........................] - ETA: 1:59 - loss: 1.5824 - accuracy: 0.5110
51/243 [=====>........................] - ETA: 1:58 - loss: 1.5569 - accuracy: 0.5188
52/243 [=====>........................] - ETA: 1:57 - loss: 1.5354 - accuracy: 0.5239
53/243 [=====>........................] - ETA: 1:56 - loss: 1.5183 - accuracy: 0.5294
54/243 [=====>........................] - ETA: 1:56 - loss: 1.5067 - accuracy: 0.5323
55/243 [=====>........................] - ETA: 1:55 - loss: 1.4871 - accuracy: 0.5363
56/243 [=====>........................] - ETA: 1:54 - loss: 1.4643 - accuracy: 0.5441
57/243 [======>.......................] - ETA: 1:53 - loss: 1.4433 - accuracy: 0.5494
58/243 [======>.......................] - ETA: 1:53 - loss: 1.4205 - accuracy: 0.5566
59/243 [======>.......................] - ETA: 1:52 - loss: 1.4007 - accuracy: 0.5626
60/243 [======>.......................] - ETA: 1:51 - loss: 1.3805 - accuracy: 0.5689
61/243 [======>.......................] - ETA: 1:50 - loss: 1.3604 - accuracy: 0.5750
62/243 [======>.......................] - ETA: 1:50 - loss: 1.3424 - accuracy: 0.5793
63/243 [======>.......................] - ETA: 1:49 - loss: 1.3232 - accuracy: 0.5855
64/243 [======>.......................] - ETA: 1:48 - loss: 1.3069 - accuracy: 0.5896
65/243 [=======>......................] - ETA: 1:48 - loss: 1.2911 - accuracy: 0.5945
66/243 [=======>......................] - ETA: 1:47 - loss: 1.2776 - accuracy: 0.5983
67/243 [=======>......................] - ETA: 1:46 - loss: 1.2648 - accuracy: 0.6029
68/243 [=======>......................] - ETA: 1:45 - loss: 1.2506 - accuracy: 0.6065
69/243 [=======>......................] - ETA: 1:45 - loss: 1.2385 - accuracy: 0.6099
70/243 [=======>......................] - ETA: 1:44 - loss: 1.2282 - accuracy: 0.6137
71/243 [=======>......................] - ETA: 1:44 - loss: 1.2153 - accuracy: 0.6174
72/243 [=======>......................] - ETA: 1:43 - loss: 1.1989 - accuracy: 0.6228
73/243 [========>.....................] - ETA: 1:42 - loss: 1.1864 - accuracy: 0.6262
74/243 [========>.....................] - ETA: 1:42 - loss: 1.1716 - accuracy: 0.6313
75/243 [========>.....................] - ETA: 1:41 - loss: 1.1584 - accuracy: 0.6346
76/243 [========>.....................] - ETA: 1:40 - loss: 1.1454 - accuracy: 0.6382
77/243 [========>.....................] - ETA: 1:40 - loss: 1.1318 - accuracy: 0.6425
78/243 [========>.....................] - ETA: 1:39 - loss: 1.1181 - accuracy: 0.6471
79/243 [========>.....................] - ETA: 1:38 - loss: 1.1044 - accuracy: 0.6516
80/243 [========>.....................] - ETA: 1:38 - loss: 1.0913 - accuracy: 0.6559
81/243 [=========>....................] - ETA: 1:37 - loss: 1.0794 - accuracy: 0.6598
82/243 [=========>....................] - ETA: 1:36 - loss: 1.0698 - accuracy: 0.6636
83/243 [=========>....................] - ETA: 1:36 - loss: 1.0581 - accuracy: 0.6673
84/243 [=========>....................] - ETA: 1:35 - loss: 1.0458 - accuracy: 0.6713
85/243 [=========>....................] - ETA: 1:34 - loss: 1.0352 - accuracy: 0.6744
86/243 [=========>....................] - ETA: 1:34 - loss: 1.0258 - accuracy: 0.6775
87/243 [=========>....................] - ETA: 1:33 - loss: 1.0338 - accuracy: 0.6780
88/243 [=========>....................] - ETA: 1:32 - loss: 1.0462 - accuracy: 0.6756
89/243 [=========>....................] - ETA: 1:32 - loss: 1.0428 - accuracy: 0.6757
90/243 [==========>...................] - ETA: 1:31 - loss: 1.0373 - accuracy: 0.6772
91/243 [==========>...................] - ETA: 1:30 - loss: 1.0289 - accuracy: 0.6801
92/243 [==========>...................] - ETA: 1:30 - loss: 1.0195 - accuracy: 0.6829
93/243 [==========>...................] - ETA: 1:29 - loss: 1.0091 - accuracy: 0.6860
94/243 [==========>...................] - ETA: 1:28 - loss: 0.9992 - accuracy: 0.6894
95/243 [==========>...................] - ETA: 1:28 - loss: 0.9892 - accuracy: 0.6926
96/243 [==========>...................] - ETA: 1:27 - loss: 0.9792 - accuracy: 0.6959
97/243 [==========>...................] - ETA: 1:26 - loss: 0.9696 - accuracy: 0.6990
98/243 [===========>..................] - ETA: 1:26 - loss: 0.9599 - accuracy: 0.7021
99/243 [===========>..................] - ETA: 1:25 - loss: 0.9503 - accuracy: 0.7051
100/243 [===========>..................] - ETA: 1:25 - loss: 0.9410 - accuracy: 0.7081
101/243 [===========>..................] - ETA: 1:24 - loss: 0.9319 - accuracy: 0.7110
102/243 [===========>..................] - ETA: 1:23 - loss: 0.9229 - accuracy: 0.7138
103/243 [===========>..................] - ETA: 1:23 - loss: 0.9140 - accuracy: 0.7166
104/243 [===========>..................] - ETA: 1:22 - loss: 0.9052 - accuracy: 0.7193
105/243 [===========>..................] - ETA: 1:21 - loss: 0.8969 - accuracy: 0.7217
106/243 [============>.................] - ETA: 1:21 - loss: 0.8900 - accuracy: 0.7238
107/243 [============>.................] - ETA: 1:20 - loss: 0.8842 - accuracy: 0.7260
108/243 [============>.................] - ETA: 1:20 - loss: 0.8769 - accuracy: 0.7283
109/243 [============>.................] - ETA: 1:19 - loss: 0.8694 - accuracy: 0.7305
110/243 [============>.................] - ETA: 1:18 - loss: 0.8619 - accuracy: 0.7327
111/243 [============>.................] - ETA: 1:18 - loss: 0.8544 - accuracy: 0.7351
112/243 [============>.................] - ETA: 1:17 - loss: 0.8468 - accuracy: 0.7375
113/243 [============>.................] - ETA: 1:17 - loss: 0.8393 - accuracy: 0.7398
114/243 [=============>................] - ETA: 1:16 - loss: 0.8320 - accuracy: 0.7421
115/243 [=============>................] - ETA: 1:15 - loss: 0.8249 - accuracy: 0.7443
116/243 [=============>................] - ETA: 1:15 - loss: 0.8178 - accuracy: 0.7466
117/243 [=============>................] - ETA: 1:14 - loss: 0.8109 - accuracy: 0.7487
118/243 [=============>................] - ETA: 1:14 - loss: 0.8041 - accuracy: 0.7509
119/243 [=============>................] - ETA: 1:13 - loss: 0.7975 - accuracy: 0.7530
120/243 [=============>................] - ETA: 1:12 - loss: 0.7940 - accuracy: 0.7545
121/243 [=============>................] - ETA: 1:12 - loss: 0.7913 - accuracy: 0.7558
122/243 [==============>...............] - ETA: 1:11 - loss: 0.7934 - accuracy: 0.7562
123/243 [==============>...............] - ETA: 1:11 - loss: 0.8107 - accuracy: 0.7529
124/243 [==============>...............] - ETA: 1:10 - loss: 0.8141 - accuracy: 0.7528
125/243 [==============>...............] - ETA: 1:09 - loss: 0.8127 - accuracy: 0.7531
126/243 [==============>...............] - ETA: 1:09 - loss: 0.8088 - accuracy: 0.7545
127/243 [==============>...............] - ETA: 1:08 - loss: 0.8031 - accuracy: 0.7565
128/243 [==============>...............] - ETA: 1:08 - loss: 0.7984 - accuracy: 0.7576
129/243 [==============>...............] - ETA: 1:07 - loss: 0.7944 - accuracy: 0.7588
130/243 [===============>..............] - ETA: 1:06 - loss: 0.7885 - accuracy: 0.7607
131/243 [===============>..............] - ETA: 1:06 - loss: 0.7832 - accuracy: 0.7623
132/243 [===============>..............] - ETA: 1:05 - loss: 0.7775 - accuracy: 0.7641
133/243 [===============>..............] - ETA: 1:04 - loss: 0.7724 - accuracy: 0.7656
134/243 [===============>..............] - ETA: 1:04 - loss: 0.7668 - accuracy: 0.7674
135/243 [===============>..............] - ETA: 1:03 - loss: 0.7613 - accuracy: 0.7691
136/243 [===============>..............] - ETA: 1:03 - loss: 0.7557 - accuracy: 0.7708
137/243 [===============>..............] - ETA: 1:02 - loss: 0.7503 - accuracy: 0.7725
138/243 [================>.............] - ETA: 1:01 - loss: 0.7450 - accuracy: 0.7741
139/243 [================>.............] - ETA: 1:01 - loss: 0.7402 - accuracy: 0.7755
140/243 [================>.............] - ETA: 1:00 - loss: 0.7355 - accuracy: 0.7771
141/243 [================>.............] - ETA: 1:00 - loss: 0.7307 - accuracy: 0.7785
142/243 [================>.............] - ETA: 59s - loss: 0.7256 - accuracy: 0.7801
143/243 [================>.............] - ETA: 58s - loss: 0.7206 - accuracy: 0.7816
144/243 [================>.............] - ETA: 58s - loss: 0.7156 - accuracy: 0.7831
145/243 [================>.............] - ETA: 57s - loss: 0.7107 - accuracy: 0.7846
146/243 [=================>............] - ETA: 57s - loss: 0.7058 - accuracy: 0.7861
147/243 [=================>............] - ETA: 56s - loss: 0.7011 - accuracy: 0.7876
148/243 [=================>............] - ETA: 55s - loss: 0.6963 - accuracy: 0.7890
149/243 [=================>............] - ETA: 55s - loss: 0.6917 - accuracy: 0.7904
150/243 [=================>............] - ETA: 54s - loss: 0.6870 - accuracy: 0.7918
151/243 [=================>............] - ETA: 54s - loss: 0.6825 - accuracy: 0.7932
152/243 [=================>............] - ETA: 53s - loss: 0.6780 - accuracy: 0.7946
153/243 [=================>............] - ETA: 52s - loss: 0.6736 - accuracy: 0.7959
154/243 [==================>...........] - ETA: 52s - loss: 0.6696 - accuracy: 0.7970
155/243 [==================>...........] - ETA: 51s - loss: 0.6655 - accuracy: 0.7983
156/243 [==================>...........] - ETA: 51s - loss: 0.6613 - accuracy: 0.7996
157/243 [==================>...........] - ETA: 50s - loss: 0.6571 - accuracy: 0.8009
158/243 [==================>...........] - ETA: 49s - loss: 0.6530 - accuracy: 0.8022
159/243 [==================>...........] - ETA: 49s - loss: 0.6489 - accuracy: 0.8034
160/243 [==================>...........] - ETA: 48s - loss: 0.6450 - accuracy: 0.8047
161/243 [==================>...........] - ETA: 48s - loss: 0.6413 - accuracy: 0.8057
162/243 [===================>..........] - ETA: 47s - loss: 0.6543 - accuracy: 0.8049
163/243 [===================>..........] - ETA: 46s - loss: 0.6824 - accuracy: 0.8035
164/243 [===================>..........] - ETA: 46s - loss: 0.6813 - accuracy: 0.8029
165/243 [===================>..........] - ETA: 45s - loss: 0.6780 - accuracy: 0.8038
166/243 [===================>..........] - ETA: 45s - loss: 0.6743 - accuracy: 0.8049
167/243 [===================>..........] - ETA: 44s - loss: 0.6705 - accuracy: 0.8061
168/243 [===================>..........] - ETA: 43s - loss: 0.6666 - accuracy: 0.8073
169/243 [===================>..........] - ETA: 43s - loss: 0.6627 - accuracy: 0.8084
170/243 [===================>..........] - ETA: 42s - loss: 0.6589 - accuracy: 0.8095
171/243 [====================>.........] - ETA: 42s - loss: 0.6552 - accuracy: 0.8107
172/243 [====================>.........] - ETA: 41s - loss: 0.6515 - accuracy: 0.8118
173/243 [====================>.........] - ETA: 40s - loss: 0.6478 - accuracy: 0.8129
174/243 [====================>.........] - ETA: 40s - loss: 0.6441 - accuracy: 0.8139
175/243 [====================>.........] - ETA: 39s - loss: 0.6405 - accuracy: 0.8150
176/243 [====================>.........] - ETA: 39s - loss: 0.6370 - accuracy: 0.8160
177/243 [====================>.........] - ETA: 38s - loss: 0.6334 - accuracy: 0.8171
178/243 [====================>.........] - ETA: 37s - loss: 0.6304 - accuracy: 0.8178
179/243 [=====================>........] - ETA: 37s - loss: 0.6281 - accuracy: 0.8186
180/243 [=====================>........] - ETA: 36s - loss: 0.6248 - accuracy: 0.8196
181/243 [=====================>........] - ETA: 36s - loss: 0.6215 - accuracy: 0.8206
182/243 [=====================>........] - ETA: 35s - loss: 0.6181 - accuracy: 0.8216
183/243 [=====================>........] - ETA: 34s - loss: 0.6148 - accuracy: 0.8226
184/243 [=====================>........] - ETA: 34s - loss: 0.6114 - accuracy: 0.8235
185/243 [=====================>........] - ETA: 33s - loss: 0.6082 - accuracy: 0.8245
186/243 [=====================>........] - ETA: 33s - loss: 0.6051 - accuracy: 0.8255
187/243 [======================>.......] - ETA: 32s - loss: 0.6018 - accuracy: 0.8264
188/243 [======================>.......] - ETA: 32s - loss: 0.5988 - accuracy: 0.8271
189/243 [======================>.......] - ETA: 31s - loss: 0.5962 - accuracy: 0.8279
190/243 [======================>.......] - ETA: 30s - loss: 0.5946 - accuracy: 0.8281
191/243 [======================>.......] - ETA: 30s - loss: 0.5949 - accuracy: 0.8277
192/243 [======================>.......] - ETA: 29s - loss: 0.6007 - accuracy: 0.8273
193/243 [======================>.......] - ETA: 29s - loss: 0.5988 - accuracy: 0.8279
194/243 [======================>.......] - ETA: 28s - loss: 0.5958 - accuracy: 0.8288
195/243 [=======================>......] - ETA: 27s - loss: 0.5931 - accuracy: 0.8297
196/243 [=======================>......] - ETA: 27s - loss: 0.5901 - accuracy: 0.8305
197/243 [=======================>......] - ETA: 26s - loss: 0.5875 - accuracy: 0.8314
198/243 [=======================>......] - ETA: 26s - loss: 0.5847 - accuracy: 0.8323
199/243 [=======================>......] - ETA: 25s - loss: 0.5818 - accuracy: 0.8331
200/243 [=======================>......] - ETA: 24s - loss: 0.5789 - accuracy: 0.8339
201/243 [=======================>......] - ETA: 24s - loss: 0.5760 - accuracy: 0.8348
202/243 [=======================>......] - ETA: 23s - loss: 0.5733 - accuracy: 0.8356
203/243 [========================>.....] - ETA: 23s - loss: 0.5705 - accuracy: 0.8364
204/243 [========================>.....] - ETA: 22s - loss: 0.5677 - accuracy: 0.8372
205/243 [========================>.....] - ETA: 22s - loss: 0.5650 - accuracy: 0.8380
206/243 [========================>.....] - ETA: 21s - loss: 0.5622 - accuracy: 0.8388
207/243 [========================>.....] - ETA: 20s - loss: 0.5595 - accuracy: 0.8396
208/243 [========================>.....] - ETA: 20s - loss: 0.5568 - accuracy: 0.8403
209/243 [========================>.....] - ETA: 19s - loss: 0.5542 - accuracy: 0.8411
210/243 [========================>.....] - ETA: 19s - loss: 0.5516 - accuracy: 0.8419
211/243 [=========================>....] - ETA: 18s - loss: 0.5490 - accuracy: 0.8426
212/243 [=========================>....] - ETA: 18s - loss: 0.5464 - accuracy: 0.8433
213/243 [=========================>....] - ETA: 17s - loss: 0.5438 - accuracy: 0.8441
214/243 [=========================>....] - ETA: 16s - loss: 0.5413 - accuracy: 0.8448
215/243 [=========================>....] - ETA: 16s - loss: 0.5388 - accuracy: 0.8455
216/243 [=========================>....] - ETA: 15s - loss: 0.5363 - accuracy: 0.8463
217/243 [=========================>....] - ETA: 15s - loss: 0.5338 - accuracy: 0.8470
218/243 [=========================>....] - ETA: 14s - loss: 0.5314 - accuracy: 0.8477
219/243 [==========================>...] - ETA: 13s - loss: 0.5289 - accuracy: 0.8484
220/243 [==========================>...] - ETA: 13s - loss: 0.5265 - accuracy: 0.8491
221/243 [==========================>...] - ETA: 12s - loss: 0.5242 - accuracy: 0.8497
222/243 [==========================>...] - ETA: 12s - loss: 0.5218 - accuracy: 0.8504
223/243 [==========================>...] - ETA: 11s - loss: 0.5195 - accuracy: 0.8511
224/243 [==========================>...] - ETA: 11s - loss: 0.5172 - accuracy: 0.8518
225/243 [==========================>...] - ETA: 10s - loss: 0.5149 - accuracy: 0.8524
226/243 [==========================>...] - ETA: 9s - loss: 0.5126 - accuracy: 0.8531
227/243 [===========================>..] - ETA: 9s - loss: 0.5103 - accuracy: 0.8537
228/243 [===========================>..] - ETA: 8s - loss: 0.5081 - accuracy: 0.8544
229/243 [===========================>..] - ETA: 8s - loss: 0.5059 - accuracy: 0.8550
230/243 [===========================>..] - ETA: 7s - loss: 0.5039 - accuracy: 0.8555
231/243 [===========================>..] - ETA: 6s - loss: 0.5034 - accuracy: 0.8560
232/243 [===========================>..] - ETA: 6s - loss: 0.5079 - accuracy: 0.8554
233/243 [===========================>..] - ETA: 5s - loss: 0.5107 - accuracy: 0.8556
234/243 [===========================>..] - ETA: 5s - loss: 0.5091 - accuracy: 0.8560
235/243 [============================>.] - ETA: 4s - loss: 0.5071 - accuracy: 0.8566
236/243 [============================>.] - ETA: 4s - loss: 0.5050 - accuracy: 0.8572
237/243 [============================>.] - ETA: 3s - loss: 0.5029 - accuracy: 0.8578
238/243 [============================>.] - ETA: 2s - loss: 0.5009 - accuracy: 0.8584
239/243 [============================>.] - ETA: 2s - loss: 0.4988 - accuracy: 0.8590
240/243 [============================>.] - ETA: 1s - loss: 0.4968 - accuracy: 0.8596
241/243 [============================>.] - ETA: 1s - loss: 0.4947 - accuracy: 0.8601
242/243 [============================>.] - ETA: 0s - loss: 0.4927 - accuracy: 0.8607
243/243 [==============================] - 197s 811ms/step - loss: 0.4907 - accuracy: 0.8613 - val_loss: 0.0068 - val_accuracy: 0.9994
2020-04-29 17:24:35.235550: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
Process finished with exit code 0
It seems this is a known issue with ongoing reports of occurrence, even in the most recent versions of TensorFlow. Apparently, it's related to parallelism and/or distribution strategy used for the data generator. One simple workaround is to use only one workers, i.e. workers=1 (which is the default value if not set), when calling the model.fit.
I am doing a Binary classification of IMDB movie review data into Positive or Negative Sentiment.
I have 25K movie reviews and corresponding label.
Preprocessing:
Removed the stop words and split the data into 70:30 training and test. So 17.5K training and 7k test. 17.5k training has been further divided into 14K train and 3.5 k validation dataset as used in keras.model.fit method
Each processed movie review has been converted to TF-IDF vector using Keras text processing module.
Here is my Fully Connected Architecture I used in Keras Dense class
def model_param(self):
""" Method to do deep learning
"""
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation
from keras.optimizers import SGD
from keras import regularizers
self.model = Sequential()
# Dense(64) is a fully-connected layer with 64 hidden units.
# in the first layer, you must specify the expected input data shape:
# here, 20-dimensional vectors.
self.model.add(Dense(32, activation='relu', input_dim=self.x_train_std.shape[1]))
self.model.add(Dropout(0.5))
#self.model.add(Dense(60, activation='relu'))
#self.model.add(Dropout(0.5))
self.model.add(Dense(1, activation='sigmoid'))
sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
self.model.compile(loss='binary_crossentropy',
optimizer=sgd,
metrics=['accuracy'])
def fit(self):
""" Training the deep learning network on the training data
"""
self.model.fit(self.x_train_std, self.y_train,validation_split=0.20,
epochs=50,
batch_size=128)
As you see, I tried first without Dropout and as usual I got training accuracy as 1.0 but validation was poor as overfitting was happening. So I added Dropout to prevent overfitting
However inspite of trying multiple dropout ratio, adding another layer with different no. of units in it, changing learning rate I am still getting overfitting on validation dataset. Gets stuck at 85% while training keeps increasing to 99% and so on. Even changed the Epochs from 10 to 50
What could be going wrong here
Train on 14000 samples, validate on 3500 samples
Epoch 1/50
14000/14000 [==============================] - 0s - loss: 0.5684 - acc: 0.7034 - val_loss: 0.3794 - val_acc: 0.8431
Epoch 2/50
14000/14000 [==============================] - 0s - loss: 0.3630 - acc: 0.8388 - val_loss: 0.3304 - val_acc: 0.8549
Epoch 3/50
14000/14000 [==============================] - 0s - loss: 0.2977 - acc: 0.8749 - val_loss: 0.3271 - val_acc: 0.8591
Epoch 4/50
14000/14000 [==============================] - 0s - loss: 0.2490 - acc: 0.8991 - val_loss: 0.3302 - val_acc: 0.8580
Epoch 5/50
14000/14000 [==============================] - 0s - loss: 0.2251 - acc: 0.9086 - val_loss: 0.3388 - val_acc: 0.8546
Epoch 6/50
14000/14000 [==============================] - 0s - loss: 0.2021 - acc: 0.9189 - val_loss: 0.3532 - val_acc: 0.8523
Epoch 7/50
14000/14000 [==============================] - 0s - loss: 0.1797 - acc: 0.9286 - val_loss: 0.3670 - val_acc: 0.8529
Epoch 8/50
14000/14000 [==============================] - 0s - loss: 0.1611 - acc: 0.9350 - val_loss: 0.3860 - val_acc: 0.8543
Epoch 9/50
14000/14000 [==============================] - 0s - loss: 0.1427 - acc: 0.9437 - val_loss: 0.4077 - val_acc: 0.8529
Epoch 10/50
14000/14000 [==============================] - 0s - loss: 0.1344 - acc: 0.9476 - val_loss: 0.4234 - val_acc: 0.8526
Epoch 11/50
14000/14000 [==============================] - 0s - loss: 0.1222 - acc: 0.9534 - val_loss: 0.4473 - val_acc: 0.8506
Epoch 12/50
14000/14000 [==============================] - 0s - loss: 0.1131 - acc: 0.9546 - val_loss: 0.4718 - val_acc: 0.8497
Epoch 13/50
14000/14000 [==============================] - 0s - loss: 0.1079 - acc: 0.9559 - val_loss: 0.4818 - val_acc: 0.8526
Epoch 14/50
14000/14000 [==============================] - 0s - loss: 0.0954 - acc: 0.9630 - val_loss: 0.5057 - val_acc: 0.8494
Epoch 15/50
14000/14000 [==============================] - 0s - loss: 0.0906 - acc: 0.9636 - val_loss: 0.5229 - val_acc: 0.8557
Epoch 16/50
14000/14000 [==============================] - 0s - loss: 0.0896 - acc: 0.9657 - val_loss: 0.5387 - val_acc: 0.8497
Epoch 17/50
14000/14000 [==============================] - 0s - loss: 0.0816 - acc: 0.9666 - val_loss: 0.5579 - val_acc: 0.8463
Epoch 18/50
14000/14000 [==============================] - 0s - loss: 0.0762 - acc: 0.9709 - val_loss: 0.5704 - val_acc: 0.8491
Epoch 19/50
14000/14000 [==============================] - 0s - loss: 0.0718 - acc: 0.9723 - val_loss: 0.5834 - val_acc: 0.8454
Epoch 20/50
14000/14000 [==============================] - 0s - loss: 0.0633 - acc: 0.9752 - val_loss: 0.6032 - val_acc: 0.8494
Epoch 21/50
14000/14000 [==============================] - 0s - loss: 0.0687 - acc: 0.9724 - val_loss: 0.6181 - val_acc: 0.8480
Epoch 22/50
14000/14000 [==============================] - 0s - loss: 0.0614 - acc: 0.9762 - val_loss: 0.6280 - val_acc: 0.8503
Epoch 23/50
14000/14000 [==============================] - 0s - loss: 0.0620 - acc: 0.9756 - val_loss: 0.6407 - val_acc: 0.8500
Epoch 24/50
14000/14000 [==============================] - 0s - loss: 0.0536 - acc: 0.9794 - val_loss: 0.6563 - val_acc: 0.8511
Epoch 25/50
14000/14000 [==============================] - 0s - loss: 0.0538 - acc: 0.9791 - val_loss: 0.6709 - val_acc: 0.8500
Epoch 26/50
14000/14000 [==============================] - 0s - loss: 0.0507 - acc: 0.9807 - val_loss: 0.6869 - val_acc: 0.8491
Epoch 27/50
14000/14000 [==============================] - 0s - loss: 0.0528 - acc: 0.9794 - val_loss: 0.7002 - val_acc: 0.8483
Epoch 28/50
14000/14000 [==============================] - 0s - loss: 0.0465 - acc: 0.9810 - val_loss: 0.7083 - val_acc: 0.8469
Epoch 29/50
14000/14000 [==============================] - 0s - loss: 0.0504 - acc: 0.9796 - val_loss: 0.7153 - val_acc: 0.8497
Epoch 30/50
14000/14000 [==============================] - 0s - loss: 0.0477 - acc: 0.9819 - val_loss: 0.7232 - val_acc: 0.8480
Epoch 31/50
14000/14000 [==============================] - 0s - loss: 0.0475 - acc: 0.9819 - val_loss: 0.7343 - val_acc: 0.8469
Epoch 32/50
14000/14000 [==============================] - 0s - loss: 0.0459 - acc: 0.9819 - val_loss: 0.7352 - val_acc: 0.8500
Epoch 33/50
14000/14000 [==============================] - 0s - loss: 0.0426 - acc: 0.9807 - val_loss: 0.7429 - val_acc: 0.8511
Epoch 34/50
14000/14000 [==============================] - 0s - loss: 0.0396 - acc: 0.9846 - val_loss: 0.7576 - val_acc: 0.8477
Epoch 35/50
14000/14000 [==============================] - 0s - loss: 0.0420 - acc: 0.9836 - val_loss: 0.7603 - val_acc: 0.8506
Epoch 36/50
14000/14000 [==============================] - 0s - loss: 0.0359 - acc: 0.9856 - val_loss: 0.7683 - val_acc: 0.8497
Epoch 37/50
14000/14000 [==============================] - 0s - loss: 0.0377 - acc: 0.9849 - val_loss: 0.7823 - val_acc: 0.8520
Epoch 38/50
14000/14000 [==============================] - 0s - loss: 0.0352 - acc: 0.9861 - val_loss: 0.7912 - val_acc: 0.8500
Epoch 39/50
14000/14000 [==============================] - 0s - loss: 0.0390 - acc: 0.9845 - val_loss: 0.8025 - val_acc: 0.8489
Epoch 40/50
14000/14000 [==============================] - 0s - loss: 0.0371 - acc: 0.9853 - val_loss: 0.8128 - val_acc: 0.8494
Epoch 41/50
14000/14000 [==============================] - 0s - loss: 0.0367 - acc: 0.9848 - val_loss: 0.8184 - val_acc: 0.8503
Epoch 42/50
14000/14000 [==============================] - 0s - loss: 0.0331 - acc: 0.9871 - val_loss: 0.8264 - val_acc: 0.8500
Epoch 43/50
14000/14000 [==============================] - 0s - loss: 0.0338 - acc: 0.9871 - val_loss: 0.8332 - val_acc: 0.8483
Epoch 44/50