I am new to deep learning however I know that usually, the loss function output would be in digits like 0.4, 0.6 something like that but the loss values I get look a bit different. Could some please tell me what is happening?
In the epoch, we can see the loss numbers are looking like this 4736.9226.
code:
#3d model
from keras.layers import Conv3D, MaxPooling3D, BatchNormalization, Dropout, Dense, Flatten, concatenate
from keras.models import Model
from keras import Input
# 3D Convolutional Model:
input_model=Input(shape=(10,250,250,1))
layer=Conv3D(32,(3,3,3),strides=(1,1,1),activation='relu')(input_model)
layer=MaxPooling3D((2,2,2))(layer)
layer=Conv3D(64,(3,3,3),strides=(1,1,1),activation='relu')(layer)
layer=MaxPooling3D((2,2,2))(layer)
layer=BatchNormalization()(layer)
layer=Flatten()(layer)
layer=Dense(128,activation='relu')(layer)
layer=Dropout(0.1)(layer)
layer=Dense(64,activation='relu')(layer)
layer=Dropout(0.1)(layer)
layer=Dense(32,activation='relu')(layer)
layer=Dropout(0.1)(layer)
layer_output=Dense(2,activation='softmax')(layer)
model_3dConv=Model(input_model,layer_output)
model_3dConv.summary()
epoch:
Epoch 1/10
13/13 [==============================] - 91s 7s/step - loss: 4736.9226 - acc: 0.4918 - val_loss: 387258.4062 - val_acc: 0.5625
Epoch 2/10
13/13 [==============================] - 90s 7s/step - loss: 4021.6621 - acc: 0.5050 - val_loss: 246713.4844 - val_acc: 0.5625
Epoch 3/10
13/13 [==============================] - 89s 7s/step - loss: 3532.2977 - acc: 0.5936 - val_loss: 166724.2500 - val_acc: 0.5625
Whereas if I use 2d model this does not happen.
Update after 10 epoch:
[![Epoch 1/10
13/13 \[==============================\] - 91s 7s/step - loss: 4736.9226 - acc: 0.4918 - val_loss: 387258.4062 - val_acc: 0.5625
Epoch 2/10
13/13 \[==============================\] - 90s 7s/step - loss: 4021.6621 - acc: 0.5050 - val_loss: 246713.4844 - val_acc: 0.5625
Epoch 3/10
13/13 \[==============================\] - 89s 7s/step - loss: 3532.2977 - acc: 0.5936 - val_loss: 166724.2500 - val_acc: 0.5625
Epoch 4/10
13/13 \[==============================\] - 94s 7s/step - loss: 2712.2616 - acc: 0.5445 - val_loss: 112906.0078 - val_acc: 0.5625
Epoch 5/10
13/13 \[==============================\] - 89s 7s/step - loss: 3779.4980 - acc: 0.5557 - val_loss: 75516.1641 - val_acc: 0.5625
Epoch 6/10
13/13 \[==============================\] - 89s 7s/step - loss: 3778.8524 - acc: 0.5036 - val_loss: 53132.3477 - val_acc: 0.6875
Epoch 7/10
13/13 \[==============================\] - 91s 7s/step - loss: 3544.4086 - acc: 0.4869 - val_loss: 36817.3906 - val_acc: 0.6875][1]][1]
There is absolutely no reason the loss function output would be in digits like 0.4, 0.6 something. I'm not sure what loss function your using here but for example the loss function mse can be any positive number from [0, inf). All the matters is your loss/val_loss should be going down each epoch, which it looks like your is doing.
Train for the full 10 epochs and check that val_loss is dropping and hopefully val_acc is going up.
Related
I'm a newbie with deep learning and I try to create a model and I don't really understand the model. add(layers). I m sure that the input shape (it's for recognition). I think the problem is in the Dropout, but I don't understand the value.
Can someone explains to me the
model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation = 'relu', input_shape = (128,128,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation = 'relu'))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Flatten())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dense(6, activation='softmax'))
model.compile(loss='categorical_crossentropy',
optimizer=optimizers.Adam(lr=1e-4), metrics=['acc'])
-------------------------------------------------------
history = model.fit(
train_data,
train_labels,
epochs=30,
validation_data=(test_data, test_labels),
)
and here is the result :
Epoch 15/30
5/5 [==============================] - 0s 34ms/step - loss: 0.3987 - acc: 0.8536 - val_loss: 0.7021 - val_acc: 0.7143
Epoch 16/30
5/5 [==============================] - 0s 31ms/step - loss: 0.3223 - acc: 0.8891 - val_loss: 0.6393 - val_acc: 0.7778
Epoch 17/30
5/5 [==============================] - 0s 32ms/step - loss: 0.3321 - acc: 0.9082 - val_loss: 0.6229 - val_acc: 0.7460
Epoch 18/30
5/5 [==============================] - 0s 31ms/step - loss: 0.2615 - acc: 0.9409 - val_loss: 0.6591 - val_acc: 0.8095
Epoch 19/30
5/5 [==============================] - 0s 32ms/step - loss: 0.2161 - acc: 0.9857 - val_loss: 0.6368 - val_acc: 0.7143
Epoch 20/30
5/5 [==============================] - 0s 33ms/step - loss: 0.1773 - acc: 0.9857 - val_loss: 0.5644 - val_acc: 0.7778
Epoch 21/30
5/5 [==============================] - 0s 32ms/step - loss: 0.1650 - acc: 0.9782 - val_loss: 0.5459 - val_acc: 0.8413
Epoch 22/30
5/5 [==============================] - 0s 31ms/step - loss: 0.1534 - acc: 0.9789 - val_loss: 0.5738 - val_acc: 0.7460
Epoch 23/30
5/5 [==============================] - 0s 32ms/step - loss: 0.1205 - acc: 0.9921 - val_loss: 0.5351 - val_acc: 0.8095
Epoch 24/30
5/5 [==============================] - 0s 32ms/step - loss: 0.0967 - acc: 1.0000 - val_loss: 0.5256 - val_acc: 0.8413
Epoch 25/30
5/5 [==============================] - 0s 32ms/step - loss: 0.0736 - acc: 1.0000 - val_loss: 0.5493 - val_acc: 0.7937
Epoch 26/30
5/5 [==============================] - 0s 32ms/step - loss: 0.0826 - acc: 1.0000 - val_loss: 0.5342 - val_acc: 0.8254
Epoch 27/30
5/5 [==============================] - 0s 32ms/step - loss: 0.0687 - acc: 1.0000 - val_loss: 0.5452 - val_acc: 0.8254
Epoch 28/30
5/5 [==============================] - 0s 32ms/step - loss: 0.0571 - acc: 1.0000 - val_loss: 0.5176 - val_acc: 0.7937
Epoch 29/30
5/5 [==============================] - 0s 32ms/step - loss: 0.0549 - acc: 1.0000 - val_loss: 0.5142 - val_acc: 0.8095
Epoch 30/30
5/5 [==============================] - 0s 32ms/step - loss: 0.0479 - acc: 1.0000 - val_loss: 0.5243 - val_acc: 0.8095
I never depassed the 70% average but on this i have 80% but i think i'm on overfitting.. I evidemently searched on differents docs but i'm lost
Have you try following into your training:
Data Augmentation
Pre-trained Model
Looking at the execution time per epoch, it looks like your data set is pretty small. Also, it's not clear whether there is any class imbalance in your dataset. You probably should try stratified CV training and analysis on the folds results. It won't prevent overfit but it will eventually give you more insight into your model, which generally can help to reduce overfitting. However, preventing overfitting is a general topic, search online to get resources. You can also try this
model.compile(loss='categorical_crossentropy',
optimizer='adam, metrics=['acc'])
-------------------------------------------------------
# src: https://keras.io/api/callbacks/reduce_lr_on_plateau/
# reduce learning rate by a factor of 0.2 if val_loss -
# won't improve within 5 epoch.
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2,
patience=5, min_lr=0.00001)
# src: https://keras.io/api/callbacks/early_stopping/
# stop training if val_loss don't improve within 15 epoch.
early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=15)
history = model.fit(
train_data,
train_labels,
epochs=30,
validation_data=(test_data, test_labels),
callbacks=[reduce_lr, early_stop]
)
You may also find it useful of using ModelCheckpoint or LearningRateScheduler. This doesn't guarantee of no overfit but some approach for that to adopt.
I followed this tutorial https://medium.com/the-andela-way/deep-learning-hello-world-e1fc53ea888 to experiment with Keras, the source code is here https://github.com/sirghiny/mnist
However I received a very low score and the training was very short, as if the model is trained on very few samples. Here is the output in terminal:
Epoch 1/5
300/300 [==============================] - 12s 39ms/step - loss: 2.3791 - categorical_accuracy: 0.0899 - val_loss: 2.3104 - val_categorical_accuracy: 0.0528
Epoch 2/5
300/300 [==============================] - 11s 38ms/step - loss: 2.3326 - categorical_accuracy: 0.1060 - val_loss: 2.2920 - val_categorical_accuracy: 0.0864
Epoch 3/5
300/300 [==============================] - 10s 32ms/step - loss: 2.2891 - categorical_accuracy: 0.1315 - val_loss: 2.2742 - val_categorical_accuracy: 0.1571
Epoch 4/5
300/300 [==============================] - 9s 31ms/step - loss: 2.2510 - categorical_accuracy: 0.1576 - val_loss: 2.2569 - val_categorical_accuracy: 0.2367
Epoch 5/5
300/300 [==============================] - 9s 30ms/step - loss: 2.2174 - categorical_accuracy: 0.1889 - val_loss: 2.2397 - val_categorical_accuracy: 0.3133
Evaluating the model...
1250/1250 [==============================] - 2s 2ms/step - loss: 2.2382 - categorical_accuracy: 0.3171
938/938 [==============================] - 2s 2ms/step - loss: 2.2369 - categorical_accuracy: 0.3232
Please tell me what did i do wrong?
You updated your model weight only 1500 times (epochs*number_of_batch).
You might want to increase the epochs or/and reduce the batch_size to perform more weights' update as we see in your logs that the network is still learning.
Additionally, you should find an up-to-date tutorial like this one as TensorFlow changed a lot recently.
I'm using Keras '2.2.4' and TensorFlow '1.13.1' and training a MobileNetV2 on a dataset of images with 2 classes.
When I fit the model on both train-set and validation-set I get two distinct progress bars and multiple losses.
history = model.fit(
data.flow(train_x,train_y,batch_size=BS),
steps_per_epoch=len(train_x)//BS,
validation_data=(val_x, val_y),
validation_steps=len(val_x)//BS,
epochs=EPOCHS)```
Epoch 1/10
153/153 [==============================] - 0s 1ms/sample - loss: 0.0134 - acc: 1.0000
12/12 [==============================] - 2s 202ms/step - loss: 0.1543 - acc: 0.9859 - val_loss: 0.0130 - val_acc: 1.0000
Epoch 2/10
153/153 [==============================] - 0s 1ms/sample - loss: 0.0026 - acc: 1.0000
12/12 [==============================] - 2s 194ms/step - loss: 0.0590 - acc: 0.9803 - val_loss: 0.0026 - val_acc: 1.0000
While if I don't add validation_data=... to model.fit:
history = model.fit(
data.flow(train_x,train_y,batch_size=BS),
steps_per_epoch=len(train_x)//BS,
epochs=EPOCHS)
Epoch 1/10
12/12 [==============================] - 2s 186ms/step - loss: 0.0404 - acc: 0.9887
Epoch 2/10
12/12 [==============================] - 2s 187ms/step - loss: 0.0189 - acc: 0.9944
Epoch 3/10
12/12 [==============================] - 2s 189ms/step - loss: 0.0137 - acc: 0.9972
I get only one progress bar that obviously won't show validation loss and metrics.
My question is: why? I remember that usually only one progress bar is shown with both train and validation loss and metrics. What is the that second progress bar shown in the first example?
I am learning to write CNNs in Keras on Kaggle using one of the datasets I found there.
The link to my notebook is
https://www.kaggle.com/vj6978/brain-tumor-vimal?scriptVersionId=16814133
The code, the dataset and the ROC curve are available at the link. The ROC curve itself looks as if the model is simply making guesses rather than a learned prediction.
The testing accuracy also seems to peak at around 60% to 70% only which is quiet low. Any help would be appreciated.
Thanks
Vimal James
I believe your last activation should be sigmoid instead of softmax.
UPDATE :
Just forked your kernel on Kaggle and modifying as follows gives better results :
model = Sequential()
model.add(Conv2D(128, (3,3), input_shape = data_set.shape[1:]))
model.add(Activation("relu"))
model.add(AveragePooling2D(pool_size = (2,2)))
model.add(Conv2D(128, (3,3)))
model.add(Activation("relu"))
model.add(AveragePooling2D(pool_size = (2,2)))
model.add(Flatten())
model.add(Dense(64))
model.add(Dense(1))
model.add(Activation("sigmoid")) # Last activation should be sigmoid for binary classification
model.compile(optimizer = "adam", loss = "binary_crossentropy", metrics = ['accuracy'])
This gave the following results :
rain on 204 samples, validate on 23 samples
Epoch 1/15
204/204 [==============================] - 2s 11ms/step - loss: 2.8873 - acc: 0.6373 - val_loss: 0.8000 - val_acc: 0.8261
Epoch 2/15
204/204 [==============================] - 1s 3ms/step - loss: 0.7292 - acc: 0.7206 - val_loss: 0.6363 - val_acc: 0.7391
Epoch 3/15
204/204 [==============================] - 1s 3ms/step - loss: 0.4731 - acc: 0.8088 - val_loss: 0.5417 - val_acc: 0.8261
Epoch 4/15
204/204 [==============================] - 1s 3ms/step - loss: 0.3605 - acc: 0.8775 - val_loss: 0.6820 - val_acc: 0.8696
Epoch 5/15
204/204 [==============================] - 1s 3ms/step - loss: 0.2986 - acc: 0.8529 - val_loss: 0.8356 - val_acc: 0.8696
Epoch 6/15
204/204 [==============================] - 1s 3ms/step - loss: 0.2151 - acc: 0.9020 - val_loss: 0.7592 - val_acc: 0.8696
Epoch 7/15
204/204 [==============================] - 1s 3ms/step - loss: 0.1305 - acc: 0.9657 - val_loss: 1.2486 - val_acc: 0.8696
Epoch 8/15
204/204 [==============================] - 1s 3ms/step - loss: 0.0565 - acc: 0.9853 - val_loss: 1.2668 - val_acc: 0.8696
Epoch 9/15
204/204 [==============================] - 1s 3ms/step - loss: 0.0426 - acc: 0.9853 - val_loss: 1.4674 - val_acc: 0.8696
Epoch 10/15
204/204 [==============================] - 1s 3ms/step - loss: 0.0141 - acc: 1.0000 - val_loss: 1.7379 - val_acc: 0.8696
Epoch 11/15
204/204 [==============================] - 1s 3ms/step - loss: 0.0063 - acc: 1.0000 - val_loss: 1.7232 - val_acc: 0.8696
Epoch 12/15
204/204 [==============================] - 1s 3ms/step - loss: 0.0023 - acc: 1.0000 - val_loss: 1.8291 - val_acc: 0.8696
Epoch 13/15
204/204 [==============================] - 1s 3ms/step - loss: 0.0014 - acc: 1.0000 - val_loss: 1.9164 - val_acc: 0.8696
Epoch 14/15
204/204 [==============================] - 1s 3ms/step - loss: 8.6263e-04 - acc: 1.0000 - val_loss: 1.8946 - val_acc: 0.8696
Epoch 15/15
204/204 [==============================] - 1s 3ms/step - loss: 6.8785e-04 - acc: 1.0000 - val_loss: 1.9596 - val_acc: 0.8696
Test loss: 3.079359292984009
Test accuracy: 0.807692289352417
You are using a softmax activation with a single neuron, this will always produce constant 1.0 output, due to the normalization used in softmax, so it makes no sense. For binary classification you have to use the sigmoid activation with a single output neuron.
The description of my work.
I have multiple execution traces(sequences) of the same application. One execution trace is different from another execution trace and written to the separate CSV file.
Similarly, I have multiple execution traces of different applications.
Now I want to build an LSTM model that will be trained by multiple sequences.
one sequence's updated weight will be preserved for further tuning/fine tuning by another sequence.
Snapshot of the Training Model code as follows:
# design network
model = Sequential()
model.add(LSTM(n_neurons, input_shape=(n_seq, n_features)))
model.add(Dense(n_seq))
for i in range(0,10):
model.compile(loss='mae', optimizer='adam', metrics=['accuracy'])
# Prepare data for LSTM
train_X, train_y, test_X, test_y =
prepare_training_data(each time for seperate sequence)
history = model.fit(train_X, train_y, epochs=nb_epoch,
batch_size=n_batch, validation_data=(test_X, test_y), verbose=2, shuffle=False)
But the problem is after each fit call my LSTM model's network weight is getting reset(from my understanding and observation) instead of retaining the previous weights of the network?
Please correct me if I am wrong anywhere?
Kindly help me to figure out how to propagate the previously trained network weight for fine tuning on the new sequence?
I am attaching my findings below.
Sequence 1:
Train on 448 samples, validate on 128 samples
Epoch 1/10
- 0s - loss: 0.3213 - acc: 0.3795 - val_loss: 0.1987 - val_acc: 0.4219
Epoch 2/10
- 0s - loss: 0.2046 - acc: 0.3460 - val_loss: 0.1980 - val_acc: 0.3047
Epoch 3/10
- 0s - loss: 0.1766 - acc: 0.3192 - val_loss: 0.1874 - val_acc: 0.4141
Epoch 4/10
- 0s - loss: 0.1773 - acc: 0.3571 - val_loss: 0.1867 - val_acc: 0.3047
Epoch 5/10
- 0s - loss: 0.1728 - acc: 0.3192 - val_loss: 0.1840 - val_acc: 0.2266
Epoch 6/10
- 0s - loss: 0.1660 - acc: 0.3571 - val_loss: 0.1843 - val_acc: 0.3203
Epoch 7/10
- 0s - loss: 0.1641 - acc: 0.3304 - val_loss: 0.1834 - val_acc: 0.3828
Epoch 8/10
- 0s - loss: 0.1599 - acc: 0.4330 - val_loss: 0.1835 - val_acc: 0.4219
Epoch 9/10
- 0s - loss: 0.1576 - acc: 0.4241 - val_loss: 0.1834 - val_acc: 0.4531
Epoch 10/10
- 0s - loss: 0.1555 - acc: 0.4487 - val_loss: 0.1829 - val_acc: 0.4766
acc: 47.66%`
Sequence 2:
Train on 512 samples, validate on 160 samples
Epoch 1/10
Epoch 00001: val_acc improved from -inf to 0.45000, saving model to model.h5
- 1s - loss: 0.1473 - acc: 0.3828 - val_loss: 0.1546 - val_acc: 0.4500
Epoch 2/10
Epoch 00002: val_acc did not improve
- 0s - loss: 0.1433 - acc: 0.3867 - val_loss: 0.1553 - val_acc: 0.4250
Epoch 3/10
Epoch 00003: val_acc improved from 0.45000 to 0.45625, saving model to model.h5
- 0s - loss: 0.1432 - acc: 0.4180 - val_loss: 0.1535 - val_acc: 0.4562
Epoch 4/10
Epoch 00004: val_acc improved from 0.45625 to 0.47500, saving model to model.h5
- 0s - loss: 0.1421 - acc: 0.4238 - val_loss: 0.1545 - val_acc: 0.4750
Epoch 5/10
Epoch 00005: val_acc did not improve
- 0s - loss: 0.1413 - acc: 0.4004 - val_loss: 0.1564 - val_acc: 0.4562
Epoch 6/10
Epoch 00006: val_acc improved from 0.47500 to 0.51250, saving model to model.h5
- 0s - loss: 0.1405 - acc: 0.4258 - val_loss: 0.1562 - val_acc: 0.5125
Epoch 7/10
Epoch 00007: val_acc did not improve
- 0s - loss: 0.1394 - acc: 0.4785 - val_loss: 0.1527 - val_acc: 0.512
Epoch 8/10
Epoch 00008: val_acc improved from 0.51250 to 0.52500, saving model to model.h5
- 0s - loss: 0.1375 - acc: 0.4629 - val_loss: 0.1502 - val_acc: 0.5250
Epoch 9/10
Epoch 00009: val_acc did not improve
- 0s - loss: 0.1361 - acc: 0.4551 - val_loss: 0.1484 - val_acc: 0.4688
Epoch 10/10
Epoch 00010: val_acc did not improve
- 0s - loss: 0.1355 - acc: 0.4648 - val_loss: 0.1473 - val_acc: 0.4750
acc: 47.50%
Sequence 3:
Train on 480 samples, validate on 128 samples
Epoch 1/10
Epoch 00001: val_acc improved from -inf to 0.41406, saving model to model.h5
- 1s - loss: 0.1342 - acc: 0.3937 - val_loss: 0.1275 - val_acc: 0.4141
Epoch 2/10
Epoch 00002: val_acc did not improve
- 0s - loss: 0.1363 - acc: 0.4313 - val_loss: 0.1308 - val_acc: 0.3047
Epoch 3/10
Epoch 00003: val_acc improved from 0.41406 to 0.44531, saving model to model.h5
- 0s - loss: 0.1352 - acc: 0.4479 - val_loss: 0.1289 - val_acc: 0.4453
Epoch 4/10
Epoch 00004: val_acc did not improve
- 0s - loss: 0.1324 - acc: 0.4188 - val_loss: 0.1273 - val_acc: 0.3438
Epoch 5/10
Epoch 00005: val_acc did not improve
- 0s - loss: 0.1301 - acc: 0.4333 - val_loss: 0.1253 - val_acc: 0.4453
Epoch 6/10
Epoch 00006: val_acc did not improve
- 0s - loss: 0.1309 - acc: 0.4583 - val_loss: 0.1243 - val_acc: 0.4141
Epoch 7/10
Epoch 00007: val_acc did not improve
- 0s - loss: 0.1338 - acc: 0.4375 - val_loss: 0.1329 - val_acc: 0.4375
Epoch 8/10
Epoch 00008: val_acc did not improve
- 0s - loss: 0.1340 - acc: 0.4479 - val_loss: 0.1235 - val_acc: 0.3906
Epoch 9/10
Epoch 00009: val_acc did not improve
- 0s - loss: 0.1282 - acc: 0.4333 - val_loss: 0.1227 - val_acc: 0.3906
Epoch 10/10
Epoch 00010: val_acc did not improve
- 0s - loss: 0.1295 - acc: 0.4208 - val_loss: 0.1234 - val_acc: 0.2266
acc: 22.66%
A small snapshot of two sequences of the application execution trace:
Each sample is having 8 features, first seven features are taken as inputs and last one is treated as output to be predicted.
sequence 1:
40 626979.9375 1196586.8750 16452.5000 3275.4375 519773.6875 1.6600 20.5535
40 692134.0000 1288955.4375 17689.7500 3352.3125 521722.0000 4.5441 43.7865
40 735489.6250 1336525.0625 17956.4375 3355.8750 522180.3750 3.0677 29.3883
40 779080.3125 1380106.4375 18235.3125 3357.3125 522605.8125 2.3105 19.5920
40 822905.4375 1423345.5625 18507.9375 3360.0000 522896.2500 10.8020 69.7630
40 866268.5625 1466615.0000 18773.5625 3362.8750 523337.1875 3.0905 19.2260
.......
sequence 2:
40 582271.0625 1035435.8750 16294.5000 1256.5000 357175.3750 3.7675 34.1337
40 686667.4375 1193365.5000 18752.4375 1340.9375 361748.1250 3.8250 33.8135
40 735528.9375 1252983.3125 19288.8125 1354.3750 363153.9375 2.7997 25.0650
40 778706.5000 1295276.5625 19533.8125 1355.8125 363278.3750 3.6734 35.2727
40 822147.1250 1340507.5625 19808.3750 1357.1250 363673.7500 3.3200 39.5510
.....
sequence n: