Develop Reference

Training process stuck in a local minimum for neural network

I'm working on a neural network that's designed to identify patterns in a large dataset, but I'm running into an issue where the training process seems to be stuck in a local minimum. Despite trying a variety of different optimization algorithms and adjusting the learning rate, I can't seem to get the network to converge on a more optimal solution. Here's the code I'm using to train the network:
import numpy as np
import tensorflow as tf
# Load dataset
data = np.load('data.npy')
# Split dataset into training and validation sets
train_data = data[:5000]
val_data = data[5000:]
# Define neural network architecture
model = tf.keras.models.Sequential([
tf.keras.layers.Dense(512, activation='relu', input_shape=(data.shape[1],)),
tf.keras.layers.Dense(256, activation='relu'),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dense(1, activation='sigmoid')
])
# Compile model
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
loss='binary_crossentropy',
metrics=['accuracy'])
# Train model
history = model.fit(train_data[:, :-1], train_data[:, -1],
validation_data=(val_data[:, :-1], val_data[:, -1]),
batch_size=32,
epochs=100,
verbose=1)
I suspect that there might be an issue with the dataset itself, but I've tried normalizing and standardizing the data, as well as applying various preprocessing techniques, with no luck. Any insights or suggestions would be helpful.

Here is one source related to your problems (https://github.com/christianversloot/machine-learning-articles/blob/main/getting-out-of-loss-plateaus-by-adjusting-learning-rates.md). Have a look! May give you some ideas.

Have you tried using different train/val splits? It could be something specific to the split you happen to have selected.

I've created a neural network in Python, does anyone know how to make it predict multiple timeseries simultaneously? Is this possible?

I created a neural network in python that is predicting my time-series very well.
My issue is I want to be able to create a neural network that can predict multiple time series at the same time.
Is this possible and how would I go about it?
This is the code to build the NN for a single time series
nn_model = Sequential()
nn_model.add(Dense(12, input_dim=1, activation='relu'))
nn_model.add(Dense(1))
nn_model.compile(loss='mean_squared_error', optimizer='adam', metrics=['mse', 'mae'])
early_stop = EarlyStopping(monitor='loss', patience=2, verbose=1)
history = nn_model.fit(X_train, y_train, epochs=100, batch_size=1, verbose=1, callbacks=[early_stop], shuffle=False)
Any ideas about how to convert this to run for multiple time series?

Keras simplest conv network not learning anything

I wrote simple code to learn Keras:
from tensorflow import keras
def main():
(x_train, y_train), (x_test, y_test) = keras.datasets.cifar10.load_data()
model = keras.Sequential()
model.add(keras.layers.Conv2D(16, 3, padding='same', activation='relu'))
model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(10, activation='softmax'))
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.fit(x_train, y_train, epochs=4)
model.summary()
if __name__ == '__main__':
main()
But it seems to not learn anything. Not like it should learn much, but should at least decrease loss and increase accuracy a little. But both are stuck the same every epoch.
I had exact same model written in Pytorch and it achieved around 35% accuracy. This in tensorflow + keras is stuck on 10%.
tensorflow-gpu v1.9
What am I missing?

I think the default learning rate is to high for this problem. Try something like
opt=keras.optimizers.Adam(lr=1.e-5)
model.compile(optimizer=opt, loss='sparse_categorical_crossentropy',
metrics=['accuracy'])

I checked the default learning rate used by Adam in both keras and PyTorch, and they both use 1e-3. Therefore, learning rate should not be the issue, assume you use default in both models.
Alternatively, I think this is related to the weight initialization, which is explicitly handled by each layer in keras but not in PyTorch.
Simply changing the training line to the following,
model.fit(x_train/255., y_train, shuffle=True,
validation_data=(x_test/255., y_test), epochs=4)
you should observe both training and validation accuracy reach around 60%.
I am not familiar with PyTorch, but I suggest you initialize the weights in the keras network with those used by the PyTorch network. In this way, you will have a fair comparison.

How do I inspect a sequential model that is created with LSTM in python code?

I have an algorithm written in python, it is timeseries analysis using LSTM. My professor asked me to show the details of the model that is created in the code. How do I inspect the "model" here? Does it have some visualization of the model in the background?
model = Sequential()
model.add(LSTM(50, input_shape=(trainX.shape[1], trainX.shape[2])))
model.add(Dropout(0.2))
model.add(Dense(1))
model.compile(loss='mae', optimizer='adam')
history = model.fit(trainX, trainY, epochs=50, batch_size=72, validation_data=(testX, testY), verbose=0, shuffle=False)

There is a visualization tool in Keras called plot_model. You can use it to save your model as an image where you can see the structure of your model including input and output dimensions.
from keras.utils import plot_model
plot_model(model, to_file='model.png')
You can read more about it here: Keras Visualization

Keras and cross validation

I am currently trying train a regression network using keras. To ensure I proper training I've want to train using crossvalidation.
The Problem is that it seems that keras don't have any functions supporting crossvalidation or do they?
The only solution I seemed to have found is to use scikit test_train_split and run a model.fit for for each k fold manually. Isn't there a already an integrated solutions for this, rather than manually doing it ?

Nope... That seem to be the solution. (Of what I know of.)

There is a scikit learn wrapper for Keras that will help you do this easily: https://keras.io/scikit-learn-api/
I recommend reading Dr. Jason Brownlee's example: https://machinelearningmastery.com/regression-tutorial-keras-deep-learning-library-python/
def baseline_model():
# create model
model = Sequential()
model.add(Dense(13, input_dim=13, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal'))
# Compile model
model.compile(loss='mean_squared_error', optimizer='adam')
return model
estimator = KerasRegressor(build_fn=wider_model, nb_epoch=100, batch_size=5, verbose=0)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(pipeline, X, Y, cv=kfold)