We are pursuing a project wherein when an image is fed into a CNN model, it predicts a number (Regression). But whatever the kernel size, filters have been changed, the validation RMSE stands at 14 only and training RMSE as 3. What could be the possible reason for this, also have tried regularization but of no use.
Related
I'm currently creating a model and while creating it I came with some questions. Does training the same model with the same data multiple times leads to better precision of those objects, since your training it every time? And what could be the issue when sometimes the object gets 90% precision and when I re-run it gets lower precision or even not predicting the right object? Is it because of Tensorflow running on the GPU?
I will guess that you are doing image recognition and that you want to identify images (objects) using a neuronal network made with Keras. You should train it once, but during training you will do several epochs, meaning the algorithm adapts the weights in several rounds (epochs). For each round it goes over all training images. Once trained, you can use the model to identify images/objects.
You can evaluate the accuracy of your trained model over the same training set, but it is better to use a different set (see train_test_split from sklearn for instance).
Training is a stochastic process, meaning that every time you train your network it will be different in the end. Hence, you will get different accurcies. The stochasticity comes from different initial weights or from using stochastic gradient descent methods for instance.
The question does not appear to have anything to do with Keras or TensorFlow but basic understandting of how neuronal networks work. There is no connection to running Tensorflow on the GPU. You will also not get better precision by training with the same objects. If you train your model on a dataset for a very long time (many epochs), you might get into overfitting. Then, the accuracy of your model on this training dataset will be very high, but the model will have low accuracy on other datasets.
A common technique is split your date in train and validation datasets, then repeatedly train your model using EarlyStopping. This will train on the training dataset, then calculate the loss against the validation dataset, and then keep training until no further improvement is seen. You can set a patience parameter to wait for X epochs without an improvement to stop training (and optionally save the best model)
https://machinelearningmastery.com/how-to-stop-training-deep-neural-networks-at-the-right-time-using-early-stopping/
Another trick is image augmentation with ImageDataGenerator which will generate synthetic data for you (rotations, shifts, mirror images, brightness adjusts, noise etc). This can effectively increase the amount of data you have to train with, thus reducing overfitting.
https://machinelearningmastery.com/how-to-configure-image-data-augmentation-when-training-deep-learning-neural-networks/
I'm using Keras and Python to train a MLP Sequential model for classification of two classes. My Training Data has 247 features and I've got 17 Samples of class 1, 922 Samples of class 2. I use Smote Borderline Oversampling Algorithm to balance the dataset. I use Cross Validation with k=4 to validate the performance of precision and recall. For Training on each fold I plot the loss curve over training and validation, to estimate if the model is under- or overfitted.
I trained a model with 3 hidden layers, and reached 95% of precision and 71% for recall. The loss function plot for each fold doesn't seem to be overfitted. (I'm not allowed by stackoverflow to post the image). But the evaluation of this model is worse, than with a model whose training precision and recall is worse.
Is this overfitting? And how can I detect it before evaluation?
Thanks in advance!
You can't detect overfitting simply from the learning curve. The definition of overfitting is when your model does extremely well on the training set, and poorly on your evaluation set, which is exactly what you're reporting.
In this case, I suspect the main problem is the unbalanced dataset. You can verify the spread of both classes in each of your sets (training, validation folds and test set) and see how your model performs on the minority class.
I am using a CNN in TensorFlow with two convolution layers, a single fully-connected layer and a linear layer to predict object sizes. The labels are sizes and the features are images.
To assess the performance of the network, I am using five-fold cross validation. Using TensorBoard I plot the accuracy for both the training set and the cross-validation set.
Both accuracies increase, but the cross-validation accuracy increases more slowly. Thinking the divergence in accuracies is due to the model overfitting, I tried to regularize the weights using L2 regularization. But, this just reduced the training accuracy, while the trend in cross-validation accuracy remained the same. The cross-validation accuracy always remains below 50%.
Can anyone recommend a few methods I might consider to improve the cross-validation accuracy and hence the predictive power of the model? Thank you very much.
without regularization Training accuracy is in gray, cross-validation accuracy is in green.
with regularization Training accuracy is in blue, cross-validation accuracy is in red.
Overfitting has multiple remedies. To name a few:
Regularization: instead of L2 regularization, you can try adding Dropout layers and see how the model fares. Dropout layers deactivate certain neurons during training, forcing the model to rely on the other ones as well.
Data augmentation: there are multiple techniques to augment your training data. You can either generate new images with image processing techniques or make your existing images more "CNN-friendly". Some keywords to search for are data centering and normalization/standardization, zca whitening, traditional image processing such as zoom/crop, invert, color filters, shift/skew, distort and rotate functions as well as NN-based data augmentation techniques.
Model architecture: changing your model architecture will result higher (overfitting) or lower (underfitting) loss of generality. Experiment with the number of layers, size of convolutional kernel and consider using pre-trained networks (transfer learning) such as Inception v3, AlexNet, GoogLeNet, VGG-16, etc.
There are certainly a million other ways but this is a great place to start.
I have trained an image classifier CNN for 50 epochs and achieved 65% accuracy with 64% accuracy on the validation data.
My problem is when using model.predict on a single sample (one image) the network behaves as though it is untrained.
I fed model.predict thousands of images, one at a time and the average classification accuracy was only 46%.
I tried using both model.save_model and saving the json model and weights separately, but there was no difference.
My only thought as to why this is occurring is to do with the BatchNorm layers in my model effecting the consistency of the data.
My model contains six CNN layers, with three max pooling and one final fully connected layer, with a BatchNormalisation layer between every layer (seven in total). I used a batch size of 128 during training, but of course the batch size for the prediction samples is 1. I don't know much about BatchNorm, but I wonder if there is some kind of normalisation that is happening on the training and testing data but not on the predictions?
I have a continuous dataset, for which I built a neural network. Before running the NN, I have scaled the training data and added bias after each layer. I am pretty happy with training result, however, I am facing a problem, when trying to build a predict method.
As I have said, the training data was scaled. The problem is, if I run un-scaled testing data with training weights, whole model will act differently and testing score will of course produce different result.
My question is, how to exclude bias from prediction model? Do I need to remove every weight out manually or are there and good practises around?