I am doing research in NLP and deep learning with mental health textual data. While training my CNN model my validation loss is lower than training loss but almost around the training loss. The validation and training loss does not go too low instead are stuck on almost 75-80% loss but accuracy achieved is also 76%. What shall i do? What is the exact interpretation of this?
I am training a multi-label image classifier using transfer learning in Keras. During training I asked the model to report out loss and acc in each epoch. In the very last epoch, it says the training acc is ~86% and val acc is pretty much the same. However, when I took the trained model and test on the training data using Sklearn performance matrix, it says the accuracy is 97%.
I am not sure if I am doing some thing wrong or the way of calculating accuracy is different in Keras and Sklearn. Please help
I am new to Lstm and machine learning and I am trying to understand some of it's concepts. Below is the code of my Lstm model.
Lstm model:
model = Sequential()
model.add(Embedding(vocab_size, 50, input_length=max_length-1))
model.add(LSTM(50))
model.add(Dropout(0.1))
model.add(Dense(vocab_size, activation='softmax'))
early_stopping = EarlyStopping(monitor='val_loss', patience=42)
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
history = model.fit(X, y, validation_split=0.2, epochs=500, verbose=2,batch_size = 20)
Below is a sample of my output:
And the train/test accuracy and train/test loss diagrams:
My undersanding (and please correct me if I am wrong) is that val_loss and val_accuracy is the loss and accuracy of the test data. My question is, what is the train accuracy and train loss and how these values are computed?. Thank you.
1. loss and val_loss-
In deep learning, the loss is the value that a neural network is trying to minimize. That is how a neural network learns by adjusting weights and biases in a manner that reduces the loss.
loss and val_loss differ because the former is applied to the train set, and the latter to the test set. As such, the latter is a good indication of how the model performs on unseen data.
2. accuracy and val_accuracy-
Once again, acc is on the training data, and val_acc is on the validation data. It's best to rely on val_acc for a fair representation of model performance because a good neural network will end up fitting the training data at 100%, but would perform poorly on unseen data.
Training should be stopped when val_acc stops increasing, otherwise your model will probably overffit. You can use earlystopping callback to stop training.
3. Why do we need train accuracy and loss?
It's not a meaningful evaluation metric because a neural network with sufficient parameters can essentially memorize the labels of training data and then perform no better than random guessing on previously unseen examples.
However, it can be useful to monitor the accuracy and loss at some fixed interval during training as it may indicate whether the backend is functioning as expected and if the training process needs to be stopped.
Refer here for a detailed explanation about earlystopping.
4. How accuracy and loss are calculated?
Loss and accuracy are calculated as you train, according to the loss and metrics specified in compiling the model. Before you train, you must compile your model to configure the learning process. This allows you to specify the optimizer, loss function, and metrics, which in turn are how the model fit function knows what loss function to use, what metrics to keep track of, etc.
The loss function (like binary cross entropy) documentation can be found here and the metrics (like accuracy) documentation can be found here.
I have built a Keras model and while training, the categorical accuracy metric reaches 0.78.
However after training the model, when I predict the output of same training data when I run the following code:
predicted_labels = model.predict(input_data)
acc = sklearn.metrics.accuracy_score(true_labels, predicted_labels)
the accuracy is 0.39.
To summarize, I don't get same accuracy result for Keras and Sklearn.
There are many ways of measuring accuracy, and sklearn might not be using the same as Keras.
You may take your compiled model and lossAndMetrics = model.evaluate(input_data, true_labels) to see loss and metrics that are surely the same you used for training.
PS: it's not rare to have a bad result for test/validation data if your model is overfitting.
I am detecting objects using CNN and keras
When i test/train model it outputs acc and loss.
I am using MSE loss functions so i understand what loss mean, however what is accuracy and how is it calculated? I have 4000 loss and accuracy 80% which is stupid. It does not detect object 80% correctly. What does it mean and how is it calcualted then?
Thanks for help.