Develop Reference

Training Loss Improving but Validation Converges Early

I am creating a CNN using TensorFlow and when training, I find that the training dataset is still improving (i.e. loss still decreasing), while the test/validation dataset has converged and is no longer improving. (Learning Curve Plot attached below)
Does anyone know why this might be the case and how could I possibly fix it, to have the validation loss reduce along with the training? Would be greatly appreciated!
Plot of my models learning curve:

The plot of losses is very typical. Your model appears to be performing very well with very low MSE losss. At this point you have essentially reached the limits of your models performance. One thing which may help is to use an adjustable learning rate. The Keras callback ReduceLROnPlateau can be setup to monitor the validation loss. If the validation loss fails to decrease for a 'patience' number of epochs the learning rate will be reduced by a factor "factor" where factor is a number less than 1. Documentation is here.
You may also want to use the Keras EarlyStopping callback. This callback can be set to monitor validation loss and halt training if it fails to decrease for "patience" number of epochs. If you set restore_best_weights=True it will leave your model with the weights used in the epoch with the lowest validation loss. This will prevent your model from returning an over fit model. My recommended code is shown below
rlronp=f.keras.callbacks.ReduceLROnPlateau(monitor="val_loss", factor=0.5, patience=1)
estop=tf.keras.callbacks.EarlyStopping(monitor="val_loss",patience=3,restore_best_weights=True)
callbacks=[rlronp, estop]
In model.fit include callbacks=callbacks. I suspect neither of the above will provide much improvement. You will probably have to try some changes to your model as well. Adding a Dropout layer may help to some degree to reduce over-fitting as would including regularization. Documentation for that is here.. Of course the standard approach of getting a larger data set may also help but is not always easy to achieve. If you are working with images you could try image augmentation using say the Keras ImageDataGenerator or Tensorflow Image Augmentation layers. Documentation for that is here.. One thing I found which helps for the case of images is to crop your images to just the Region of Interest (ROI). For example if you were doing face recognition cropping the images to just be of the face will help significantly.

this means you're hitting your architecture's limit, training loss will keep decreasing (this is known as overfitting), which will eventually INCREASE validation loss, make changes to the parameters or consider altering your layers (adding, removing, etc.), maybe even look into ways you could alter the dataset.
when this happened to me a while ago I added a LSTM layer to my CNN architecture and also incorporated K-means validation, this is not a walkthrough, you need to figure this out for your specific problem, good luck.

Compute gradients in a custom layer in Keras

I have written a code that computes Choquet pooling in a Custom Layer in Keras. Below the Colab link to the notebook:
https://colab.research.google.com/drive/1lCrUb2Jm680JRnACPxWpxkOSkP_DlHGj
As you can the code crashes in gradient computation, precisely inside the function custom_grad. This is impossible because I'm returning 0 gradients with the same shape as the previous layer.
So I have 2 questions:
Is in Keras (or in Tensorflow) a way to compute gradient between the layer input and its output?
If I have passed a Tensor with the same shape as the previous layer, but filled with 0s, why the code is not working?
Thanks for your attention and I'm waiting for your help.
Thanks in advance

No one is interested in that question.
After several trials, I have found a solution. The problem is that, as posted by Mainak431 in this GitHub repo:
link to diff and non-diff ops in tensorflow
There are differentiable TensorFlow operations and non-differentiable operations. In the Colab notebook, I used, as an example, scatter_nd_update that is non-differentiable.
So I suggest, if you want to create your own Custom Layer in Keras to take a look at the above lists in order to use operations that allow Keras to auto-differentiate for you.
Anyway, I'm working on it to inform as much as possible on that open research topic. I remember that with the neural network the "LEGO-ing" is borderline, and I know for sure that many of you are interested in adding your operations(aggregation or something else) in a deep neural network model.
Special Thanks to Maniak431, I love you <3

How to determine activation, loss, optimizer in keras while making artificial neural network

This is my dataframe
https://drive.google.com/file/d/1qAnyOkp_YayqzZ4i0CwqCTDiYTIOmv6I/view?usp=sharing
I need to find the value of ra, last column of that dataset via the ANN
I have used keras library to make that, here is my code
https://gist.github.com/anonymous/9955247ad7341e5bc119556dead9fc71
But the y_pred variable has set of 0s in output. Am I doing anything wrong with activation function?
I need to predict the ra values with training dataset
P.S: I am a newbie to datascience and just I have started learning it via udemy

You can remove the second hidden layer as simple Ann is enough for this and also we don’t have to use activator at the output layer as it is regression problem.
Please see the sample code https://github.com/naveenkambham/MachineLearningModels/blob/master/NeuralNetwork.py . This is similar to your requirement.

Convolutional Neural Network using TensorFlow

I'm building a CNN model using Tensorflow, without the use of any frontend APIs such as Keras. I'm creating a VGG-16 model and using the pre-trained weights, and want to fine tune the last layers to serve my purpose.
Following the tutorial here, http://cv-tricks.com/tensorflow-tutorial/training-convolutional-neural-network-for-image-classification/
I re-created the training script and modified as per my requirements. However, my training does not happen and the training accuracy is stuck at 50.00% and validation accuracy is forming a pattern repeating the numbers.
Attached is the screenshot of the same.
I have been stuck on this for days now and can't seem to find the error. Any help is appreciated.
The code is pretty long and hence here is the gist file for the same

Your cross entropy is wrong, you are comparing your logits with the softmax of your logits.
This:
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=layer_fc2,
labels=y_pred)
Should be:
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=layer_fc2,
labels=y_true)
Some things to note. I would not train on some data point and then evaluate on the same datapoint. Your training accuracy is probably going to be biased by doing so. Another point to note ist that tf.argmax(tf.softmax(logits)) is the same as tf.argmax(logits).

Python/Pybrain: How can I fix weights of a neural network during training?

I am quite new to neural networks and trying to use pybrain to build and train a network.
I am building my network manually with full connections between all layers (input, two hidden layers, output) and then set some weights to zero using _SetParameters as I don't want connections between some specific nodes.
My problem is that the weights that are zero at the beginning are adapted in the same way as all other weights and therefore no more zero after training the network via backprop. How can I force the "zero-weights" to stay zero through the whole process?
Thanks a lot for your answers.
Fiona

Looks like you'll have to use the Connections module to create your network with the specific connections between nodes. Use the inSliceFrom and outSliceFrom parameters. See this StackOverflow answer for more details.

I am struggling with a similar problem.
So far I am using net._setParameters command to fix the weights after each training step, but there should be a better answer..
It might help for the meantime, I am waiting for the better answer as well :-)