I am training a multi-label image classifier using transfer learning in Keras. During training I asked the model to report out loss and acc in each epoch. In the very last epoch, it says the training acc is ~86% and val acc is pretty much the same. However, when I took the trained model and test on the training data using Sklearn performance matrix, it says the accuracy is 97%.
I am not sure if I am doing some thing wrong or the way of calculating accuracy is different in Keras and Sklearn. Please help
Related
Normally I would use the code torch.save(model.state_dict(), 'test.pth') to save the best model based on the performance of the validation set.
In the training phase, I print the loss and Accuracy in the last epoch and I got Loss:0.38703016219139097 and Accutacy:86.9.
However, When I load the model which I just got from the training phase to print the loss and Accuracy, I would get the same Accuracy and different loss: 0.38702996191978456.
Why would that happen? I try different datasets and neural networks, but get the same result.
If I've understood correctly, at the end of each epoch, you print the training accuracy/loss, and also save the model if it beats the current best model on the validation set. Is that it?
Because if my understanding of the situation is correct, then it is perfectly normal. Your "best" model in regards of the TRAINING accuracy/loss is under no obligation to also be the best in regards of the VALIDATION accuracy/loss. (One of the best examples of this is the overfitting phenomenon)
I am doing research in NLP and deep learning with mental health textual data. While training my CNN model my validation loss is lower than training loss but almost around the training loss. The validation and training loss does not go too low instead are stuck on almost 75-80% loss but accuracy achieved is also 76%. What shall i do? What is the exact interpretation of this?
I have developed parameters for a Keras model and looked at the loss and accuracy over epochs as shown below. What does it mean when your loss and accuracy overlaps for test and training data? What else is apparent from the data in these graphs? This is a classification supervised learning model.
I have built a Keras model and while training, the categorical accuracy metric reaches 0.78.
However after training the model, when I predict the output of same training data when I run the following code:
predicted_labels = model.predict(input_data)
acc = sklearn.metrics.accuracy_score(true_labels, predicted_labels)
the accuracy is 0.39.
To summarize, I don't get same accuracy result for Keras and Sklearn.
There are many ways of measuring accuracy, and sklearn might not be using the same as Keras.
You may take your compiled model and lossAndMetrics = model.evaluate(input_data, true_labels) to see loss and metrics that are surely the same you used for training.
PS: it's not rare to have a bad result for test/validation data if your model is overfitting.
I am detecting objects using CNN and keras
When i test/train model it outputs acc and loss.
I am using MSE loss functions so i understand what loss mean, however what is accuracy and how is it calculated? I have 4000 loss and accuracy 80% which is stupid. It does not detect object 80% correctly. What does it mean and how is it calcualted then?
Thanks for help.