Input nodes in Keras NN - python

I am trying to create an neural network based on the iris dataset. I have an input of four dimensions. X = dataset[:,0:4].astype(float). Then, I create a neural network with four nodes.
model = Sequential()
model.add(Dense(4, input_dim=4, init='normal', activation='relu'))
model.add(Dense(3, init='normal', activation='sigmoid'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
As I understand, I pass each dimension to the separate node. Four dimensions - four nodes. When I create a neural network with 8 input nodes, how does it work? Performance still is the same as with 4 nodes.
model = Sequential()
model.add(Dense(8, input_dim=4, init='normal', activation='relu'))
model.add(Dense(3, init='normal', activation='sigmoid'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

You have an error on your last activation. Use softmax instead of sigmoid and run again.
replace
model.add(Dense(3, init='normal', activation='sigmoid'))
with
model.add(Dense(3, init='normal', activation='softmax'))

To answer your main question of "How does this work?":
From a conceptual standpoint, you are initially creating a fully-connected, or Dense, neural network with 3 layers: an input layer with 4 nodes, a hidden layer with 4 nodes, and an output layer with 3 nodes. Each node in the input layer has a connection to every node in the hidden layer, and same with the hidden to the output layer.
In your second example, you just increased the number of nodes in the hidden layer from 4 to 8. A larger network can be good, as it can be trained to "look" for more things in your data. But too large of a layer and you may overfit; this means the network remembers too much of the training data, when it really just needs a general idea of the training data so it can still recognize slightly different data, which is your testing data.
The reason you may not have seen an increase in performance is likely either overfitting or your activation function; Try a function other than relu in your hidden layer. After trying a few different function combinations, if you don't see any improvement, you are likely overfitting.
Hope this helps.

Related

Keras regression model does not fit 2 labels simulataneously

I am trying to use keras to do nonlinear regression. I have simulated 90000 datasets and labelled them with 2 parameters. My goal is to have a fully connected NN to estimate these two parameters after training. Currently, the model works well for fitting only one label. As a test, I have tried fitting each label independently and this works well, however when I want to fit both simultaneously it fails (i.e., the model predicts 1 label accurately, but not the other. In some instances the second label is off by a factor of 1000 and in other cases it simply reads [0.] ... depending on the activation for my output layer). 1 label is on the order of 1e7 and the other label varies between 0 and 1. I have tried normalizing both labels to lie between 0 and 1 - this didn't help. Each input should be a vector of size 1024 and associated with 2 labels.
Any help or literature suggestions on how to fit multi-labelled data would be much appreciated. Attached below is the code for my model. Thank you.
# Build The Model
model = Sequential()
# The Input Layer :
model.add(Dense(1024, kernel_initializer='normal', input_dim=1024, activation='relu'))
# The Hidden Layers :
model.add(Dense(1024, kernel_initializer='normal',activation='relu'))
model.add(Dense(1024, kernel_initializer='normal',activation='relu'))
model.add(Dense(1024, kernel_initializer='normal',activation='relu'))
# The Output Layer :
model.add(Dense(2, kernel_initializer='normal', activation='relu'))
# Compile the network :
model.compile(loss='MSE', optimizer='adam', metrics=['MSE'])
model.summary()

Counting CNN layers

i created a CNN for a project i am involved with and i need to present it. The issue is, I am not sure about how to count the layers.
Here is my model:
model = Sequential()
model.add(Conv2D(64,(3,3), input_shape = (40,40,2)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(64,(3,3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dense(1600))
model.add(Reshape((40,40)))
model.add(Activation('sigmoid'))
model.compile(loss='MSE',
optimizer='SGD',
metrics=['MAE'])
len(model.layers) returned 12 :
So i used 1 input 10 hidden 1 output layers,
or
i need to count them as a group and say 1 input 2 hidden 1 output?
When calculating the depth of a CNN network, we only consider the layers that have tunable/trainable weights/parameters. In CNN only Convolutional Layers and Fully Connected Layers will have trainable parameters. If you want to label layers consider only Convolutional, Full Connected and Output layers (Conv2D and Dense).
Max Pooling Layers are generally take together with Convolutional Layers as one layer.

How to define a multi-dimensional neural network with keras

I have implemented a simple neural network with keras that takes an input of 50 values and returns a classification of '0' or '1'. I believe the model is expecting an input shape of (50, 1). I'd like to add another 50 data values for each input, but I'd like them to be associated with the original 50 respective inputs. So instead of making the input of shape (100, 1), I guess I'd like to make it of shape (50, 2). I would like the neural network to know from the start that each input feature has two values associated with it, instead of it thinking there are 100 separate input features. Here's what I have so far:
model = Sequential()
model.add(Dense(50, input_dim=50, kernel_initializer='normal', activation='relu'))
model.add(Dense(100, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
Can anyone show me the way the alter this structure to accept my new input shape?

How to extract weights of hidden layers with Keras API

I have a simple sequential neural network which I would like to use to train a classifier. It is made of one input layer, 3 hidden layers and one output layer, as follows:
#sequential type of model
model = Sequential()
#stacking layers with .add
model.add(Dense(len(ytrain), activation='relu', input_dim=100))
model.add(Dropout(0.5))
model.add(Dense(len(ytrain), activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(len(ytrain), activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(len(ytrain), activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(5, activation='softmax'))
How can I extract the weights associated with each hidden layer. The ultimate goal is to then use the activation function to compute the probability of each label to be the correct one.
Hope you understand. Any kind of help is appreciated.
weights = [layer.get_weights() for layer in model.layers]

Keras recursively train model

I have a code that runs like this, more or less
for i in list:
train_lstm(attributes)
and then, further below, I have
def train_lstm(attributes):
model = Sequential()
model.add(Embedding(100, 500,input_length=5))
model.add(LSTM(100))
model.add(Dense(1, activation='sigmoid'))
do the stuff
It's not really a bug, but my question is:
Since LSTM networks "remember", I suppose that it loses all its point if in every for-iteration I reset the network.
Should I then define the model outside the loop and reuse it?
model = Sequential()
model.add(Embedding(100, 500,input_length=5))
model.add(LSTM(100))
model.add(Dense(1, activation='sigmoid'))
for i in list:
train_lst(attributes, model)
I'm using the network for time series prediction. Thanks

Categories

Resources