I was reading this tutorial when a doubt appeared. In the tutorial de dataset has 10 attributes (after the one-hot conversion), but when the model was created, the input layer has more neurons (64) than inputs (10).
Here's the code:
def build_model():
model = keras.Sequential([
layers.Dense(64, activation='relu', input_shape=[len(train_dataset.keys())]),
layers.Dense(64, activation='relu'),
layers.Dense(1)
])
optimizer = tf.keras.optimizers.RMSprop(0.001)
model.compile(loss='mse',
optimizer=optimizer,
metrics=['mae', 'mse'])
return model
I thought that the number of neurons should be equals to the number of entries. Can anyone explain this?
Thanks for your attention
The 64 in your Dense layers specifies the number of neurons of the first and second hidden layers. The input_shape parameter specifies how keras will automatically handle the number of neurons in the input layer, so 10 in your case.
Related
I have a model like this:
model = Sequential()
model.add(Embedding(100, 5, input_length=X.shape[1]))
model.add(LSTM(100))
model.add(Dense(2, activation='softmax'))
print(model.summary())
I was surfing on the internet for enhance the layer like this on below:
Input Layer - 7 feature
Recurrent Layer - 8 hidden unit
Fully Connected Layer - 4 neuron
Output Layer - 2 neuron
Here's my code I've been tried
model = tf.keras.Sequential([
tf.keras.layers.LSTM(feature, input_shape = input_shape),
tf.keras.layers.LSTM(8),
tf.keras.layers.Dense(4, activation=tf.nn.relu)
tf.keras.layers.Dense(2, activation=tf.nn.softmax)
])
But I didn't find anything - how to enhance layer like this?
Thanks in advance
I am trying to make a 3 sequence many-to-many LSTM model, but I am confused about it's implementation in Keras. I searched on internet for examples of many-to-many models, but each website gives different method. That has confused me even more. What is the correct method of those? I want a model like this:
Some of the various methods I found were
Using encoder, decoder
from keras.layers import RepeatVector
from keras.layers import TimeDistributed
model = Sequential()
# encoder layer
model.add(LSTM(100, activation='relu', input_shape=(3, 1)))
# repeat vector
model.add(RepeatVector(3))
# decoder layer
model.add(LSTM(100, activation='relu', return_sequences=True))
model.add(TimeDistributed(Dense(1)))
model.compile(optimizer='adam', loss='mse')
Another with encoder, decoder
from keras.models import Model
from keras.layers import Input, LSTM, Dense
encoder_inputs = Input(shape=(None, 1))
encoder = LSTM(100, return_state=True)
encoder_outputs, state_h, state_c = encoder(encoder_inputs)
encoder_states = [state_h, state_c]
decoder_inputs = Input(shape=(None, 1))
decoder_lstm = LSTM(100, return_sequences=True, return_state=True)
decoder_outputs, _, _ = decoder_lstm(decoder_inputs,
initial_state=encoder_states)
decoder_dense = Dense(num_decoder_tokens, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)
model = Model([encoder_inputs, decoder_inputs], decoder_outputs)
model = Sequential()
model.add(LSTM(100,input_shape=(3,1),return_sequences=True))
model.add(TimeDistributed(Dense(2)))
model.compile(optimizer='adam', loss='mse')
model = Sequential()
model.add(LSTM(100,input_shape=(3,1),return_sequences=True))
model.compile(optimizer='adam', loss='mse')
Which one of these is the correct method? which one will give the model like the one I want?
You have to mention your problem statement first.
1 and 2 are best for neural machine translation problems. While 2 is superior because it is considering return states in LSTM layer. 3 is also a good architecture where logic from input to output is simple. 4 is a very basic architecture becuase nth output in the output array has knowledge about [0 to n-1th input, not later ones] also no fully connected (Dense) layer so even moderate logic cannot be learned here.
Hello I am trying to build a seq2seq model to generate some music.
I really dont know much about it though.
On the internet I have found this model:
def createSeq2Seq():
#seq2seq model
#encoder
model = Sequential()
model.add(LSTM(input_shape = (None, input_dim), units = num_units, activation= 'tanh', return_sequences = True ))
model.add(BatchNormalization())
model.add(Dropout(0.3))
model.add(LSTM(num_units, activation= 'tanh'))
#decoder
model.add(RepeatVector(y_seq_length))
num_layers= 2
for _ in range(num_layers):
model.add(LSTM(num_units, activation= 'tanh', return_sequences = True))
model.add(BatchNormalization())
model.add(Dropout(0.3))
model.add(TimeDistributed(Dense(output_dim, activation= 'softmax')))
return model
My data is a list of pianorolls. A piano roll is a matrix with the columns representing a one-hot encoding of the different possible pitches (49 in my case) with each column representing a time (0,02s in my case). The pianoroll matrix is then only ones and zeros.
I have prepared my training data reshaping my pianoroll songs (putting them all one after the other) into
shape = (something, batchsize, 49). So my input data are all the songs one after the other separeted in blocks of size the batchsize. My training data is then the same input but delayed one batch.
The x_seq_length and y_seq_length are equal to the batch_size. Input_dim = 49
My input and output sequences have the same dimension.
Have I made any mistake in my reasoning? Is the seq2seq model Ive found correct? What does the RepeatVector does?
This is not a seq2seq model. RepeatVector takes the last state of the last encoder LSTM and makes one copy per output token. Then you feed these copies into a "decoder" LSTM, which thus has the same input in every time step.
A proper autoregressive decoder takes its previous outputs as input, i.e., at training time, the input of the decoder is the same as its output, but shifted by one position. This also means that your model misses the embedding layer for the decoder inputs.
How the input_dim parameter used in different cases with Dense function in case of Multi Layer Perceptron (MLP) in Keras?
I have tried and seen the below observation in case of Sequential Model:
The input_dim parameter is always passed as part of Dense() function when adding very first hidden layer in the MLP
The input_dim parameter is never passed as part of Dense() function for other hidden layers (e.g. 2nd layer, 3rd layer, 4th layer, etc) or output layer in the MLP
# To create a Sequential model
model_batch_drop_5lyr = Sequential()
# Hidden Layer1
model_batch_drop_5lyr.add(Dense(684, activation='relu', input_dim = input_dim, kernel_initializer=RandomNormal(mean=0.0, stddev=0.025, seed=None)))
model_batch_drop_5lyr.add(BatchNormalization())
model_batch_drop_5lyr.add(Dropout(0.5)) # 50% no. of neurons droped
# Hidden Layer2
model_batch_drop_5lyr.add(Dense(512, activation='relu', kernel_initializer=RandomNormal(mean=0.0, stddev=0.050, seed=None)))
model_batch_drop_5lyr.add(BatchNormalization())
model_batch_drop_5lyr.add(Dropout(0.5)) # 50% no. of neurons droped
# Hidden Layer3
model_batch_drop_5lyr.add(Dense(356, activation='relu', kernel_initializer=RandomNormal(mean=0.0, stddev=0.075, seed=None)))
model_batch_drop_5lyr.add(BatchNormalization())
model_batch_drop_5lyr.add(Dropout(0.5)) # 50% no. of neurons droped
# Hidden Layer4
model_batch_drop_5lyr.add(Dense(228, activation='relu', kernel_initializer=RandomNormal(mean=0.0, stddev=0.125, seed=None)))
model_batch_drop_5lyr.add(BatchNormalization())
model_batch_drop_5lyr.add(Dropout(0.5)) # 50% no. of neurons droped
# Hidden Layer5
model_batch_drop_5lyr.add(Dense(128, activation='relu', kernel_initializer=RandomNormal(mean=0.0, stddev=0.155, seed=None)))
model_batch_drop_5lyr.add(BatchNormalization())
model_batch_drop_5lyr.add(Dropout(0.5)) # 50% no. of neurons droped
# Output Layer
model_batch_drop_5lyr.add(Dense(output_dim, activation='softmax'))
# Printing Summary of the model with 3 hidden layers
print(model_batch_drop_5lyr.summary())
Why do we use input dimension only in hidden layer1. What are other scenario where input_dim can be used for other layers as well?
input_dim is very similar to the input_shape parameter, which is also only typically used in the first layer. Both tells keras what is the dimensionality of the input data to the first layer, from where shape inference can be performed for that and subsequent layers.
input_dim = n is equivalent to input_shape = (n,), and you use in the layer that receives the input because the input is the only unknown to keras, the user needs to specify the expected shape of the input.
I'm creating a model to classify if the input waverform contains rising edge of SDA of I2C line.
My input has 20000 datapoints and 100 training data.
I've initially found an answer regarding the input in here Keras 1D CNN: How to specify dimension correctly?
However, I'm getting an error in the activation function:
ValueError: Error when checking target: expected activation_1 to have 3 dimensions, but got array with shape (100, 1)
My model is:
model.add(Conv1D(filters=n_filter,
kernel_size=input_filter_length,
strides=1,
activation='relu',
input_shape=(20000,1)))
model.add(BatchNormalization())
model.add(MaxPooling1D(pool_size=4, strides=None))
model.add(Dense(1))
model.add(Activation("sigmoid"))
adam = Adam(lr=learning_rate)
model.compile(optimizer= adam, loss='binary_crossentropy', metrics=['accuracy'])
model.fit(train_data, train_label,
nb_epoch=10,
batch_size=batch_size, shuffle=True)
score = np.asarray(model.evaluate(test_new_data, test_label, batch_size=batch_size))*100.0
I can't determine the problem in here. On why the activation function expects a 3D tensor.
The problem lies in the fact that starting from keras 2.0, a Dense layer applied to a sequence will apply the layer to each time step - so given a sequence it will produce a sequence. So your Dense is actually producing a sequence of 1-element vectors and this causes your problem (as your target is not a sequence).
There are several ways on how to reduce a sequence to a vector and then apply a Dense to it:
GlobalPooling:
You may use GlobalPooling layers like GlobalAveragePooling1D or GlobalMaxPooling1D, eg.:
model.add(Conv1D(filters=n_filter,
kernel_size=input_filter_length,
strides=1,
activation='relu',
input_shape=(20000,1)))
model.add(BatchNormalization())
model.add(GlobalMaxPooling1D(pool_size=4, strides=None))
model.add(Dense(1))
model.add(Activation("sigmoid"))
Flattening:
You might colapse the whole sequence to a single vector using Flatten layer:
model.add(Conv1D(filters=n_filter,
kernel_size=input_filter_length,
strides=1,
activation='relu',
input_shape=(20000,1)))
model.add(BatchNormalization())
model.add(MaxPooling1D(pool_size=4, strides=None))
model.add(Flatten())
model.add(Dense(1))
model.add(Activation("sigmoid"))
RNN Postprocessing:
You could also add a recurrent layer on a top of your sequence and make it to return only the last output:
model.add(Conv1D(filters=n_filter,
kernel_size=input_filter_length,
strides=1,
activation='relu',
input_shape=(20000,1)))
model.add(BatchNormalization())
model.add(MaxPooling1D(pool_size=4, strides=None))
model.add(SimpleRNN(10, return_sequences=False))
model.add(Dense(1))
model.add(Activation("sigmoid"))
Conv1D has its output with 3 dimensions (and it will keep like that until the Dense layer).
Conv output: (BatchSize, Length, Filters)
For the Dense layer to output only one result, you need to add a Flatten() or Reshape((shape)) layer, to make it (BatchSize, Lenght) only.
If you call model.summary(), you will see exactly what shape each layer is outputting. You have to adjust the output to be exactly the same shape as the array you pass as the correct results. The None that appears in those shapes is the batch size and may be ignored.
About your model: I think you need more convolution layers, reducing the number of filters gradually, because condensing so much data in a single Dense layer does not usually bring good results.
About dimensions: keras layers toturial and samples