Develop Reference

how to fix NotImplementedError

I have the following code to run a vgg
checkpoint = tf.keras.callbacks.ModelCheckpoint(checkpoint_filepath+'MAssCalcVGG-noAug.h5',
monitor='val_loss', mode='auto', verbose=1,
save_best_only=True, save_freq='epoch'
)
#add custom fully-connected network on top of the already-trained base network
model = models.Sequential()
model.add(conv_base)
model.add(layers.Flatten())
model.add(layers.Dense(256, activation="relu"))
model.add(layers.Dense(1, activation="sigmoid"))
#freeze convolutional base
conv_base.trainable = False
model.compile(loss="binary_crossentropy",
optimizer=optimizers.Adam(lr=1e-3), # lr = 0.0001
metrics=METRICS)
#train fully-connected added part
history = model.fit(train_generat.flow(train_dataset_split,
train_labels_split,
batch_size=BATCH_SIZE,
shuffle=False),
steps_per_epoch=len(train_dataset_split) // BATCH_SIZE,
epochs=100,
validation_data=valid_generat.flow(valid_dataset_split,
valid_labels_split,
batch_size=BATCH_SIZE,
shuffle=False),
validation_steps=len(valid_labels_split) // BATCH_SIZE,
callbacks=[es, checkpoint, GarbageCollectorCallback()])
#model.save(save(os.path.join(checkpoint_filepath, 'MAssCalcVGG-noAug.h5'))))
model.summary()
but I get this:
NotImplementedError: Layer ModuleWrapper has arguments in `__init__` and therefore must override `get_config`
how can I fix it?
the code was perfectly working before probably I intentionally made a change to cause this.

How can I restart model.fit after x epochs if loss remains high?

I am having trouble sometimes getting a fit on my data, and when I restart fit (with shuffle=true) then I sometimes get a good fit.
See my previous question:
https://datascience.stackexchange.com/questions/62516/why-does-my-model-sometimes-not-learn-well-from-same-data
As a work around, I want to automatically restart the fitting process, if loss is high after x epochs. How can I achieve this?
I assume I would need to use a custom version of EarlyStopping callback? How could I differentiate between ES because of finding a low loss ( < 0.5) so training is finished, or because loss > 0.5 after x epochs so need to restart training?
Here is a simplified structure:
def train_till_good():
while not_finished:
train()
def train():
load_data()
model = VerySimpleNet2();
checkpoint = keras.callbacks.ModelCheckpoint(filepath=images_root + dataset_name + '\\CheckPoint.hdf5')
myOpt = keras.optimizers.Adam(lr=0.001,decay=0.01)
model.compile(optimizer=myOpt, loss='categorical_crossentropy', metrics=['accuracy'])
LRS = CyclicLR(base_lr=0.000005, max_lr=0.0003, step_size=200.)
tensorboard = keras.callbacks.TensorBoard(log_dir='C:\\Tensorflow', histogram_freq=0,write_graph=True, write_images=False)
ES = keras.callbacks.EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=5)
model.fit(train_images, train_labels, shuffle=True, epochs=num_epochs,
callbacks=[checkpoint,
tensorboard,
ES,
LRS],
validation_data = (test_images, test_labels)
)
def VerySimpleNet2():
model = keras.Sequential([
keras.layers.Dense(112, activation=tf.nn.relu, input_shape=(224, 224, 3)),
keras.layers.Dropout(0.4),
keras.layers.Flatten(),
keras.layers.Dense(3, activation=tf.nn.softmax)
])
return model

Using Keras with Ensemble Voting Classifier

I am trying to use EnsembleVoteClassifier from mlxtend library, where my classifiers are ANN, SVM, Logistic Regression. I am pre-fitting the models and calling EnsembleVoteClassifier just for prediction:
ensemble=EnsembleVoteClassifier(clfs=[model_nn, model_logreg],voting='hard',refit=False)
ensemble.fit(X_train,y_train)
y_pred_ensemble = ensemble.predict(X_test)
The problem is with Keras. My code is below:
model_nn = Sequential()
model_nn.add(Dense(20, input_shape=(X_train.shape[1],),
kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=42),
bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=42)))
model_nn.add(Activation('relu'))
model_nn.add(BatchNormalization())
model_nn.add(Dropout(0.5))
model_nn.add(Dense(2, activation='softmax'))
model_nn.compile (loss = 'sparse_categorical_crossentropy', optimizer=k.optimizers.Adam(lr=1e-4))
early_stopping_monitor = EarlyStopping(monitor='val_loss', mode='min', patience=20)
lr_reduce= ReduceLROnPlateau(monitor='val_loss', verbose=1, mode='min', patience=20)
history = model_nn.fit(X_train, y_train, epochs=1000,
class_weight=class_weights,
batch_size=32,
validation_data=(X_val, y_val), verbose = 1,
callbacks=[early_stopping_monitor, lr_reduce])
y_pred_nn = model_nn.predict(X_test)
y_pred_nn = y_pred_nn.argmax(axis=1)
The problem is that the shape of predicted class is (n_samples,2) which produces an error in EnsembleVoteClassifier:
raise ValueError("bad input shape {0}".format(shape))
ValueError: bad input shape (, 2)
Is there any way to pass the pipeline that will take care of the shape problem and output the keras prediction with the same shape as sklearn does?
Thank you.

GRU with random selection data in keras

I use a recurrent network (in special GRU) for predict a time serie with a lenght of 90 occurrences. The type of data is multivariante, and a follow this example.
Multivariante Time Series
Option 1:
I use keras for develop the rnn
n_train_quarter = int(len(serie) * 0.75)
train = values[:n_train_quarter, :]
test = values[n_train_quarter:, :]
X_train, y_train = train[:, :-1], train[:, - 1]
X_test, y_test = test[:, :-1], test[:, - 1]
# All parameter can be changes kernel, activation, optimizer, ...
model = Sequential()
model.add(GRU(64, input_shape=(X_train.shape[1], X_train.shape[2]),return_sequences=True))
model.add(Dropout(0.5))
# n is random
for i in range(n)
model.add(GRU(64,kernel_initializer = 'uniform', return_sequences=True))
model.add(Dropout(0.5))
model.add(Flatten())
model.add(Dense(1))
model.add(Activation('softmax'))
#Compile and fit
model.compile(loss='mean_squared_error', optimizer='SGD')
early_stopping = EarlyStopping(monitor='val_loss', patience=50)
checkpointer = ModelCheckpoint(filepath=Checkpoint_mode, verbose=0, save_weights_only=False, save_best_only=True)
model.fit(X_train, y_train,
batch_size=256,
epochs=64,
validation_split=0.25,
callbacks=[early_stopping, checkpointer],
verbose=0,
shuffle=False)
And the result with less error look like the image (there are various experiment with same result)
Option 2:
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = 0.33, random_state = 42)
# All parameter can be changes kernel, activation, optimizer, ...
model = Sequential()
model.add(GRU(64, input_shape=(X_train.shape[1], X_train.shape[2]),return_sequences=True))
model.add(Dropout(0.5))
# n is random
for i in range(n)
model.add(GRU(64,kernel_initializer = 'uniform', return_sequences=True))
model.add(Dropout(0.5))
model.add(Flatten())
model.add(Dense(1))
model.add(Activation('softmax'))
#Compile and fit
model.compile(loss='mean_squared_error', optimizer='SGD')
early_stopping = EarlyStopping(monitor='val_loss', patience=50)
checkpointer = ModelCheckpoint(filepath=Checkpoint_mode, verbose=0, save_weights_only=False, save_best_only=True)
model.fit(X_train, y_train,
batch_size=256,
epochs=64,
validation_split=0.25,
callbacks=[early_stopping, checkpointer],
verbose=0,
shuffle=False)
And the result with less error print as
Can use "train_test_split" of sklearn with random select for the data?
Why is better the result with secuential data than with random selection data, if GRU is better with secuential data?

Keras earlystopping: print selected epoch

Simple question. I am using Keras earlystopping in the following form:
Earlystop = EarlyStopping(monitor='val_loss', min_delta=0, patience=5, verbose=1, mode='auto')
How can I get Keras to print the selected epoch once the model has been fit? I think you have to use logs but don't quite know how.
Thanks.
Edit:
The full code is very long! Let me add a bit more than I gave. Hopefully it will help.
# Define model
def design_flexiNN(m_type, neurons, shape_timestep, shape_feature, activation, kernel_ini):
model = Sequential()
model.add(Dense(neurons, input_dim=shape_feature, activation = activation, use_bias=True, kernel_initializer=kernel_ini))
model.add(Dense(1, use_bias=True))
model.compile(loss='mae', optimizer='Adam')
return model
# fit model
def fit_flexiNN(m_type, train_X, train_y, epochs, batch_size, test_X, test_y):
history = model.fit(train_X, train_y, epochs=epochs, batch_size=batch_size, callbacks=callbacks_list, validation_data=(test_X, test_y), verbose=0, shuffle=False)
Earlystop = EarlyStopping(monitor='val_loss', min_delta=0, patience=5, verbose=1, mode='auto')
callbacks_list = [Earlystop]
model = design_flexiNN(m_type, neurons, neurons_step, train_X_feature_shape, activation, kernel_ini);
history = fit_flexiNN(m_type, train_X, train_y, ini_epochs, batch_size, test_X, test_y)
I've been able to infer the selected epoch by doing len(history.history['val_loss']) minus 1, but that doesn't work if you have a patience above zero.

Been trying to solve this myself and realised that the len(history.history['val_loss']) method is almost correct. All you need to add is:
len(history.history['val_loss']) - patience
which should give you the epoch number for the selected model (assuming that the model didnt run for the full number of epochs).
A slightly more thorough method would be:
model_loss = history.history["val_loss"]
epoch_chosen = model_loss.index(min(model_loss)) +1
print(epoch_chosen)
Hope this helps!