Instruction of questions:
Instantiate a Sequential model.
Add a Dense layer of 50 neurons with an input shape of 1 neuron.
Add two Dense layers of 50 neurons each and 'relu' activation.
End your model with a Dense layer with a single neuron and no activation.
Below is my code:
# Instantiate a Sequential model
model = Sequential()
# Add a Dense layer with 50 neurons and an input of 1 neuron
model.add(Dense(50, input_shape=(2,), activation='relu'))
# Add two Dense layers with 50 neurons and relu activation
model.add(Dense(____,____=____))
model.____
# End your model with a Dense layer and no activation
model.____
I am confused about the part
model.add(Dense(____,____=____))
In the documentation, the only required parameter for the Dense layer is units, which is the number of neurons. The default activation function is None, so if you want it to be "relu", do activation="relu".
In conclusion, this is that piece of code that creates a Dense layer with 50 neurons and activation as relu:
model.add(Dense(50, activation="relu"))
In model.add(Dense(___,___=___)), you have three blanks. The first one is for mentioning number of neurons, the second one for saying that you want to set some value for activation and the third one is for setting that value as relu.
So you will get model.add(Dense(50,activation='relu'))
More information can be found in the Dense layer documentation.
Related
Let's say I want to code this basic Neural Network Structure in Keras which has 10 units in Input Layer and 3 units in Output layer.
Now if I am using Keras, and give input_shape of more then 10, how it will adjust in it.
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
model = Sequential()
model.add(Dense(10, activation = 'relu', input_shape = (64,)))
model.add(Dense(3, activation = 'sigmoid'))
model.summary()
You see, here input_shape is of size 64, but how will it adjust in model whose first layer has 10 units because for what I have learned that size of input shape/vector should be equal to number of units in the input layer.
Or Am I not implementing this neural network right?
That would not be a problem. The weight matrix of shape (10,64) would be used in input layer. your input has shape 64 and first hidden layer has 10 units giving a output of 3 units. Seems fine to me.
But your input layer itself is 64. So what you are getting is a 3-layer network with a hidden layer of 10 units.
If the shape of your input vector is 64, then you really need to have an input layer with size 64. The input layer of a neural network doesn't perform any computations. It just passes the inputs forward to the first hidden layer. This one, on the other hand, performs the computations for all neurons contained in it (linear combination of input vector and weights, later served as an input to the activation function, which is the ReLU in your case).
In your code, you are building a neural net with 64 input neurons (which again don't perform any computations), 10 neurons in the first (and only) hidden layer and 3 neurons in the output layer.
How the input_dim parameter used in different cases with Dense function in case of Multi Layer Perceptron (MLP) in Keras?
I have tried and seen the below observation in case of Sequential Model:
The input_dim parameter is always passed as part of Dense() function when adding very first hidden layer in the MLP
The input_dim parameter is never passed as part of Dense() function for other hidden layers (e.g. 2nd layer, 3rd layer, 4th layer, etc) or output layer in the MLP
# To create a Sequential model
model_batch_drop_5lyr = Sequential()
# Hidden Layer1
model_batch_drop_5lyr.add(Dense(684, activation='relu', input_dim = input_dim, kernel_initializer=RandomNormal(mean=0.0, stddev=0.025, seed=None)))
model_batch_drop_5lyr.add(BatchNormalization())
model_batch_drop_5lyr.add(Dropout(0.5)) # 50% no. of neurons droped
# Hidden Layer2
model_batch_drop_5lyr.add(Dense(512, activation='relu', kernel_initializer=RandomNormal(mean=0.0, stddev=0.050, seed=None)))
model_batch_drop_5lyr.add(BatchNormalization())
model_batch_drop_5lyr.add(Dropout(0.5)) # 50% no. of neurons droped
# Hidden Layer3
model_batch_drop_5lyr.add(Dense(356, activation='relu', kernel_initializer=RandomNormal(mean=0.0, stddev=0.075, seed=None)))
model_batch_drop_5lyr.add(BatchNormalization())
model_batch_drop_5lyr.add(Dropout(0.5)) # 50% no. of neurons droped
# Hidden Layer4
model_batch_drop_5lyr.add(Dense(228, activation='relu', kernel_initializer=RandomNormal(mean=0.0, stddev=0.125, seed=None)))
model_batch_drop_5lyr.add(BatchNormalization())
model_batch_drop_5lyr.add(Dropout(0.5)) # 50% no. of neurons droped
# Hidden Layer5
model_batch_drop_5lyr.add(Dense(128, activation='relu', kernel_initializer=RandomNormal(mean=0.0, stddev=0.155, seed=None)))
model_batch_drop_5lyr.add(BatchNormalization())
model_batch_drop_5lyr.add(Dropout(0.5)) # 50% no. of neurons droped
# Output Layer
model_batch_drop_5lyr.add(Dense(output_dim, activation='softmax'))
# Printing Summary of the model with 3 hidden layers
print(model_batch_drop_5lyr.summary())
Why do we use input dimension only in hidden layer1. What are other scenario where input_dim can be used for other layers as well?
input_dim is very similar to the input_shape parameter, which is also only typically used in the first layer. Both tells keras what is the dimensionality of the input data to the first layer, from where shape inference can be performed for that and subsequent layers.
input_dim = n is equivalent to input_shape = (n,), and you use in the layer that receives the input because the input is the only unknown to keras, the user needs to specify the expected shape of the input.
What is the difference between the Dropout layer and the dropout and recurrent_droput parameters in keras? Do they all serve the same purpose?
Example:
model.add(Dropout(0.2)) # layer
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2)) # parameters
Yes they have the same functionality, dropout as a parameter is used before linear transformations of that layer (multiplication of weights and addition of bias). Dropout as layer can be used before an activation layer too.
recurrent_dropout also has same functionality but different direction(usually dropouts are between input and output, it is between timestamps)
I'm a bit confused about the number of layers that are used in Keras models. The documentation is rather opaque on the matter.
According to Jason Brownlee the first layer technically consists of two layers, the input layer, specified by input_dim and a hidden layer. See the first questions on his blog.
In all of the Keras documentation the first layer is generally specified as
model.add(Dense(number_of_neurons, input_dim=number_of_cols_in_input, activtion=some_activation_function)).
The most basic model we could make would therefore be:
model = Sequential()
model.add(Dense(1, input_dim = 100, activation = None))
Does this model consist of a single layer, where 100 dimensional input is passed through a single input neuron, or does it consist of two layers, first a 100 dimensional input layer and second a 1 dimensional hidden layer?
Further, if I were to specify a model like this, how many layers does it have?
model = Sequential()
model.add(Dense(32, input_dim = 100, activation = 'sigmoid'))
model.add(Dense(1)))
Is this a model with 1 input layer, 1 hidden layer, and 1 output layer or is this a model with 1 input layer and 1 output layer?
Your first one consists of a 100 neurons input layer connected to one single output neuron
Your second one consists of a 100 neurons input layer, one hidden layer of 32 neurons and one output layer of one single neuron.
You have to think of your first layer as your input layer (with the same number of neurons as the dimenson, so 100 for you) connected to another layer with as many neuron as you specify (1 in your first case, 32 in the second one)
In Keras what is useful is the command
model.summary()
For your first question, the model is :
1 input layer and 1 output layer.
For the second question :
1 input layer
1 hidden layer
1 activation layer (The sigmoid one)
1 output layer
For the input layer, this is abstracted by Keras with the input_dim arg or input_shape, but you can find this layer in :
from keras.layers import Input
Same for the activation layer.
from keras.layers import Activation
# Create a `Sequential` model and add a Dense layer as the first layer.
model = tf.keras.models.Sequential()
model.add(tf.keras.Input(shape=(16,)))
model.add(tf.keras.layers.Dense(32, activation='relu'))
# Now the model will take as input arrays of shape (None, 16)
# and output arrays of shape (None, 32).
# Note that after the first layer, you don't need to specify
# the size of the input anymore:
model.add(tf.keras.layers.Dense(32))
model.output_shape
(None, 32)
model.layers
[<keras.layers.core.dense.Dense at 0x7f494062e950>,
<keras.layers.core.dense.Dense at 0x7f4944048d90>]
model.summary()
Output
it may help you understand clearly
I'm creating a model to classify if the input waverform contains rising edge of SDA of I2C line.
My input has 20000 datapoints and 100 training data.
I've initially found an answer regarding the input in here Keras 1D CNN: How to specify dimension correctly?
However, I'm getting an error in the activation function:
ValueError: Error when checking target: expected activation_1 to have 3 dimensions, but got array with shape (100, 1)
My model is:
model.add(Conv1D(filters=n_filter,
kernel_size=input_filter_length,
strides=1,
activation='relu',
input_shape=(20000,1)))
model.add(BatchNormalization())
model.add(MaxPooling1D(pool_size=4, strides=None))
model.add(Dense(1))
model.add(Activation("sigmoid"))
adam = Adam(lr=learning_rate)
model.compile(optimizer= adam, loss='binary_crossentropy', metrics=['accuracy'])
model.fit(train_data, train_label,
nb_epoch=10,
batch_size=batch_size, shuffle=True)
score = np.asarray(model.evaluate(test_new_data, test_label, batch_size=batch_size))*100.0
I can't determine the problem in here. On why the activation function expects a 3D tensor.
The problem lies in the fact that starting from keras 2.0, a Dense layer applied to a sequence will apply the layer to each time step - so given a sequence it will produce a sequence. So your Dense is actually producing a sequence of 1-element vectors and this causes your problem (as your target is not a sequence).
There are several ways on how to reduce a sequence to a vector and then apply a Dense to it:
GlobalPooling:
You may use GlobalPooling layers like GlobalAveragePooling1D or GlobalMaxPooling1D, eg.:
model.add(Conv1D(filters=n_filter,
kernel_size=input_filter_length,
strides=1,
activation='relu',
input_shape=(20000,1)))
model.add(BatchNormalization())
model.add(GlobalMaxPooling1D(pool_size=4, strides=None))
model.add(Dense(1))
model.add(Activation("sigmoid"))
Flattening:
You might colapse the whole sequence to a single vector using Flatten layer:
model.add(Conv1D(filters=n_filter,
kernel_size=input_filter_length,
strides=1,
activation='relu',
input_shape=(20000,1)))
model.add(BatchNormalization())
model.add(MaxPooling1D(pool_size=4, strides=None))
model.add(Flatten())
model.add(Dense(1))
model.add(Activation("sigmoid"))
RNN Postprocessing:
You could also add a recurrent layer on a top of your sequence and make it to return only the last output:
model.add(Conv1D(filters=n_filter,
kernel_size=input_filter_length,
strides=1,
activation='relu',
input_shape=(20000,1)))
model.add(BatchNormalization())
model.add(MaxPooling1D(pool_size=4, strides=None))
model.add(SimpleRNN(10, return_sequences=False))
model.add(Dense(1))
model.add(Activation("sigmoid"))
Conv1D has its output with 3 dimensions (and it will keep like that until the Dense layer).
Conv output: (BatchSize, Length, Filters)
For the Dense layer to output only one result, you need to add a Flatten() or Reshape((shape)) layer, to make it (BatchSize, Lenght) only.
If you call model.summary(), you will see exactly what shape each layer is outputting. You have to adjust the output to be exactly the same shape as the array you pass as the correct results. The None that appears in those shapes is the batch size and may be ignored.
About your model: I think you need more convolution layers, reducing the number of filters gradually, because condensing so much data in a single Dense layer does not usually bring good results.
About dimensions: keras layers toturial and samples