Develop Reference

Adding additional hidden layer and attention layer to LSTM model

sequence_input = Input(shape=(MAX_LENGTH_SEQUENCE,), dtype='int32')
embedded_sequences = embedding_layer(sequence_input)
l_lstm = Bidirectional(LSTM(10))(embedded_sequences)
preds = Dense(len(macronum), activation='softmax')(l_lstm)
model = Model(sequence_input, preds)
model.compile(loss='categorical_crossentropy',
optimizer='rmsprop',
metrics=['acc'])
I need to add additional hidden layer and an attention layer to the above LSTM model , usually i construct the model in this way:
model = tensorflow.keras.Sequential()
model.add(tensorflow.keras.layers.LSTM(128, dropout=0.3,
recurrent_dropout=0.2,input_shape=(N, K), return_sequences=True))
#model.add(tensorflow.keras.layers.LSTM(128, dropout=0.3,
recurrent_dropout=0.2,input_shape=(N, K), return_sequences=True))
model.add(Attention(name='attention_weight'))
model.add(tensorflow.keras.layers.Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
print(model.summary())
print(_x_train.shape)
model.fit(_x_train, _y_train, epochs=10, batch_size=1)
scores = model.evaluate(_x_test, _y_test, verbose=0)
print("Accuracy: %.2f%%" % (scores[1]*100))
as shown in the second code i normaly stack and add as many layer as i need using
model.add
but for the first code i'm not familiar with this approach,
if i want to add extra LSTM layer and Attention layer to the first code, where should i include them inside the code? what is the right syntax ?

Machine Learning with Keras: Different Validation Loss for the Same Model

I am trying to use keras to train a simple feedforward network. I tried two different methods of what I think is the same network, but one is performing significantly better. The first one and the better performing one is the following:
inputs = keras.Input(shape=(384,))
dense = layers.Dense(64, activation="relu")
x = dense(inputs)
x = layers.Dense(64, activation="relu")(x)
outputs = layers.Dense(384)(x)
model = keras.Model(inputs=inputs, outputs=outputs, name="simple_model")
model.compile(loss='mse',optimizer='Adam')
history = model.fit(X_train,
y_train_tf,
epochs=20,
validation_data=(X_test, y_test),
steps_per_epoch=100,
validation_steps=50)
and it settles on a validation loss of about 0.2. The second model performs much worse:
model = keras.models.Sequential()
model.add(Dense(64, input_shape=(384,), activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dense(384, activation='relu'))
optimizer = tf.keras.optimizers.Adam()
model.compile(loss='mse', optimizer=optimizer)
history = model.fit(X_train,
y_train_tf,
epochs=20,
validation_data=(X_test, y_test),
steps_per_epoch=100,
validation_steps=50)
and this has validation loss of around 5. But when I do model.summary, they look virtually the same. Is there something wrong with the second model?

I am not sure that they are the same since second model has relu activation after last layer (384 units) and first doesn't. This might be the issue since default activation of the Keras dense layer is None.

How can I extract Flatten Layer Output for each epoch?

model = Sequential()
model.add(Conv2D(50, (5,5), activation='relu', input_shape =(5,5,1), kernel_initializer='he_normal'))
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))
model.summary()
# compile the model
model.compile(loss='binary_crossentropy', optimizer= 'adam', metrics=['accuracy'])
model_checkpoint=ModelCheckpoint(r'C:\Users\globo\Desktop\Test_CNN\Results\Kernel5x5\Weights'+'\\'+test+'\model_test{epoch:02d}.h5',save_freq=1,save_weights_only=True)
# fit the model
history = model.fit(X_train, Y_train, epochs=10, batch_size=32, verbose=1, callbacks=[model_checkpoint], shuffle=True, validation_split=0.5)
I'm already extracting weights for each epoch with "ModelCheckpoint", but how can I extract flatten layer output for each epoch and save them?

doing this with sequential models is not feasible at all.
you should use functional API
inp = Input((5,5,1))
x = Conv2D(50, (5,5), activation='relu', kernel_initializer='he_normal')(inp)
xflatten = Flatten()(x)
out = Dense(1, activation='sigmoid')(xflatten)
main_model = Model(inp, out) # this works same as your model
flatten_model = Model(inp, xflatten) # and this only outputs the flatten layer and is not necessary to compile it because we won't train it, it just shows the output of a layer
main_model.compile(loss='binary_crossentropy', optimizer= 'adam', metrics=['accuracy'])
history = main_model.fit(X_train, Y_train, epochs=10, batch_size=32, verbose=1, callbacks=[model_checkpoint], shuffle=True, validation_split=0.5)
to see the flatten layers's output:
flatten_model.predict(X)

get a list of predictions of a neural network

I created a neural network to classify messages. Now I want to collect the predictions into a list in python. How do I do this?
So here is the model:
model = Sequential()
model.add(layers.Dense(500, activation = "relu", input_shape=(7600,)))
# Hidden - Layers
model.add(layers.Dropout(0.4, noise_shape=None, seed=None))
model.add(layers.Dense(300, activation = "relu"))
model.add(layers.Dropout(0.4, noise_shape=None, seed=None))
model.add(layers.Dense(100, activation = "relu"))
model.add(layers.Dropout(0.4, noise_shape=None, seed=None))
model.add(layers.Dense(20, activation = "softmax"))
model.summary()
model.compile(loss="categorical_crossentropy",
optimizer="adam",
metrics=['accuracy'])
model.fit( np.array(vectorized_training), np.array(y_train_neralnet),
batch_size=2000,
epochs=3,
verbose=1,
validation_data=(np.array(vectorized_validation), np.array(y_validation_neralnet)))
Here I tried to print the shape of validation_data that is inside of the model.fit() method but it gives an error.
NameError: name 'validation_data' is not defined

This is what you are looking for:
preds = model.predict(X_test)

pickle Keras ANN

I am trying to use this code to fit an ANN using Keras and then pickle it:
early_stopping = EarlyStopping(monitor='val_loss', patience=4, mode='auto')
model = Sequential()
model.add(Dense(units=40, kernel_initializer='random_uniform', input_dim=x_train.shape[1], activation='relu'))
model.add(Dense(units=1, kernel_initializer='random_uniform', activation='sigmoid', W_regularizer=reg))
model.compile(loss='binary_crossentropy', optimizer='adam')
model.fit(x=x_train, y=y_train, epochs=1, validation_data=(x_val, y_val), callbacks=[early_stopping])
pickle_file_and_path = 'C:/Bla/DLModel20180816.sav'
pickle.dump(model, open(pickle_file_and_path, 'wb'))
Unfortunately, I get:
pickle.dump(model, open(pickle_file_and_path, 'wb'))
TypeError: can't pickle _thread.RLock objects
Any ideas?

The canonical way of storing Keras models is to use the built-in model.save() method which will save the model into a HDF5 file.
Adapting your code:
model = Sequential()
model.add(Dense(units=40, kernel_initializer='random_uniform', input_dim=x_train.shape[1], activation='relu'))
model.add(Dense(units=1, kernel_initializer='random_uniform', activation='sigmoid', W_regularizer=reg))
model.compile(loss='binary_crossentropy', optimizer='adam')
model.fit(x=x_train, y=y_train, epochs=1, validation_data=(x_val, y_val), callbacks=[early_stopping])
# Save the model
model.save('./model.h5')
Afterwards, you can load it as follows:
from keras.models import load_model
model = load_model('./model.h5')
And start retraining or use the model for inference. Note that you can also store the only the weights with the save_weights() method. For the full documentation and more examples see the Keras website.