Develop Reference

Train many neural networks and pick best one

I'm working on a classification task, trying to reconstruct a network from paper. In that paper, they are talking about doing a train test split 300 times and training the network each time after they are taking the mean of all predictions from each network for specific input data.
So here's the question: What is the best option for doing that, I've already reconstructed their network and thinking about using a for loop and saving outputs of each network in a data frame but can't get it the right way.
Here's the code :
# Set X and Y for training
X = dum_bll_fsrq.drop(['type2', 'name', 'Type_is_bll', 'Type_is_fsrq'], axis = 1)
Y = dum_bll_fsrq.iloc[:,-2:]
# Train test split
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3, stratify = Y)
# Create model
model_two_neuron = tf.keras.Sequential([
tf.keras.layers.Dense(40, input_shape=(15,)), # input shape required
tf.keras.layers.Dense(2, activation=tf.nn.sigmoid)
])
model_two_neuron.compile(optimizer=tf.keras.optimizers.Adam(),
loss=tf.keras.losses.MeanSquaredError(),
metrics=[tf.keras.metrics.Precision()])
# Train
model_two_neuron.fit(X_train, y_train, epochs=20)

You can use callbacks to save the best weights for each of your models, then evaluate the best results saved by callbacks after training.
Here is a basic example, provided in the Documentation:
model.compile(loss=..., optimizer=...,
metrics=['accuracy'])
EPOCHS = 10
checkpoint_filepath = '/tmp/checkpoint'
model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(
filepath=checkpoint_filepath,
save_weights_only=True,
monitor='val_accuracy',
mode='max',
save_best_only=True)
# Model weights are saved at the end of every epoch, if it's the best seen
# so far.
model.fit(epochs=EPOCHS, callbacks=[model_checkpoint_callback])
# The model weights (that are considered the best) are loaded into the model.
model.load_weights(checkpoint_filepath)

Error checking input: expected embedding_1 input to have shape but got shape

I have successfully created my Keras sequential model and trained it for a while. Now I am trying to make some predictions but it fails even using the same data as used on the training phase.
I am getting this error: {ValueError}Error when checking input: expected embedding_1_input to have shape (2139,) but got array with shape (1,)
However, when checking the input I am trying to use, it says (2139,). I would like to know if anyone knows what this might be
df = pd.read_csv('../../data/parsed-data/data.csv')
df = ModelUtil().remove_entries_based_on_threshold(df, 'Author', 2)
#show_column_distribution(df, 'Author')
y = df.pop('Author')
le = LabelEncoder()
le.fit(y)
encoded_Y = le.transform(y)
tokenizer, padded_sentences, max_sentence_len \
= PortugueseTextualProcessing().convert_corpus_to_number(df)
ModelUtil().save_tokenizer(tokenizer)
vocab_len = len(tokenizer.word_index) + 1
glove_embedding = PortugueseTextualProcessing().load_vector(tokenizer)
embedded_matrix = PortugueseTextualProcessing().build_embedding_matrix(glove_embedding, vocab_len, tokenizer)
cv_scores = []
kfold = StratifiedKFold(n_splits=4, shuffle=True, random_state=7)
models = []
nn = NeuralNetwork()
nn.build_baseline_model(embedded_matrix, max_sentence_len, vocab_len, len(np_utils.to_categorical(encoded_Y)[0]))
# Separate some validation samples
val_data, X, Y = ModelUtil().extract_validation_data(padded_sentences, encoded_Y)
for train_index, test_index in kfold.split(X, Y):
# convert integers to dummy variables (i.e. one hot encoded)
dummy_y = np_utils.to_categorical(Y)
print("TRAIN:", train_index, "TEST:", test_index)
X_train, X_test = X[train_index], X[test_index]
y_train, y_test = dummy_y[train_index], dummy_y[test_index]
nn.train(X_train, y_train, 100)
scores = nn.evaluate_model(X_test, y_test)
cv_scores.append(scores[1] * 100)
models.append(nn)
print("%.2f%% (+/- %.2f%%)" % (np.mean(cv_scores), np.std(cv_scores)))
best_model = models[cv_scores.index(max(cv_scores))]
best_model.save_model()
best_model.predict_entries(X[0])
Method that performs the prediction and model creation
def build_baseline_model(self, emd_matrix, long_sent_size, vocab_len, number_of_classes):
self.model = Sequential()
embedding_layer = Embedding(vocab_len, 100, weights=[emd_matrix], input_length=long_sent_size,
trainable=False)
self.model.add(embedding_layer)
self.model.add(Dropout(0.2))
self.model.add(Flatten())
# softmax performing better than relu
self.model.add(Dense(number_of_classes, activation='softmax'))
self.model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])
return self.model
def predict_entries(self, entry):
predictions = self.model.predict_classes(entry)
# show the inputs and predicted outputs
print("X=%s, Predicted=%s" % (entry, predictions[0]))
return predictions
X[0].shape evaluates to
: (2139,)

In you case you should apply a reshape so you can get an array with a unique element that contains the sentence.
X_reshape = X[0].reshape(1, 2139)
best_model.predict_entries(X_reshape)

keras predicting classes method

So, I have this little project going on about predicting the nba 2019 champion but it seems that my code is not clear enough to make keras understand what I want. I have passed a list of past champions on my dataset and made it the output class to get the current champion.
I'm using a dataset for teams stats from 2014 to 2018 regular seasons and I'm assuming that I should have the 2019 stats to do it. I have made my dataset very well encoded for my NN to understand by providing one hot encoding in every feature I think it's useful.
x = pd.concat([df.drop(['Unnamed: 0','Team','Game','Date','Opponent','LastSeasonChamp'], axis = 1), df_ohc], axis = 1)
y = df['LastSeasonChamp']
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.35)
x_train = tf.keras.utils.normalize(x_train.values, axis = 1)
x_test = tf.keras.utils.normalize(x_test.values, axis = 1)
n_classes = 30
model = tf.keras.Sequential()
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(x_train.shape[1], input_shape = (x_train.shape[0],x_train.shape[1]), activation = tf.nn.relu))
model.add(tf.keras.layers.Dense(np.mean([x_train.shape[1], n_classes], dtype = int), activation = tf.nn.relu))
model.add(tf.keras.layers.Dense(n_classes, activation = tf.nn.softmax))
model.compile(optimizer = 'adagrad' , loss = 'sparse_categorical_crossentropy', metrics=['accuracy'])
model.fit(x_train, y_train.values, epochs = 3)
model.evaluate(x_test, y_test)
model.save('nba_champ_2019')
new_model = tf.keras.models.load_model('nba_champ_2019')
pred = new_model.predict(x_test)
y_pred = to_categorical(pred)
So, I could expect my y_pred to be a column with 0 and 1 and but all I get is a column full of 1.

The to_categorical function is used to convert a list of class IDs to an one-hot matrix. You don't need it here. You should get the output you expect by removing in this case.

Keras LSTM - Validation Loss Increasing From Epoch #1

I'm currently undertaking my first 'real' DL project of (surprise) predicting stock movements. I know that I'm 1000:1 to make anything useful but I'm enjoying it and want to see it through, I've learnt more in my few weeks of attempting this than I have in the prior 6 months of completing MOOC's.
I'm building an LSTM using Keras to currently predict the next 1 step forward and have attempted the task as both classification (up/down/steady) and now as a regression problem. Both result in a similar roadblock in that my validation loss never improves from epoch #1.
I can get the model to overfit such that training loss approaches zero with MSE (or 100% accuracy if classification), but at no stage does the validation loss decrease. This screams overfitting to my untrained eye so I added varying amounts of dropout but all that does is stifle the learning of the model/training accuracy and shows no improvements on the validation accuracy.
I have attempted to change a significant number of hyperparameters - learning rate, optimiser, batchsize, lookback window, #layers, #units, dropout, #samples, etc, also tried with subset of data and subset of features but I just can't get it to work so I'm very thankful for any help.
Code Below (it's not pretty I know):
# Import saved full dataframe ~ 200 features
import feather
df = feather.read_dataframe('df_feathered')
df.set_index('time', inplace=True)
# Difference the dataset to make stationary
df = df.diff(periods=1, axis=0)
# MAKE LARGE SAMPLE FOR TESTING
df_train = df.loc['2017-3-1':'2017-6-30']
df_val = df.loc['2017-7-1':'2017-8-31']
df_test = df.loc['2017-9-1':'2017-9-30']
# Make x_train, x_val sets by dropping target variable
x_train = df_train.drop('close+1', axis=1)
x_val = df_val.drop('close+1', axis=1)
# Scale the training data first then fit the transform to the test set
scaler = StandardScaler()
x_train = scaler.fit_transform(x_train)
x_test = scaler.transform(x_val)
# scaler = MinMaxScaler(feature_range=(0,1))
# x_train = scaler.fit_transform(df_train1)
# x_test = scaler.transform(df_val1)
# Create y_train, y_test, simply target variable for regression
y_train = df_train['close+1']
y_test = df_val['close+1']
# Define Lookback window for LSTM input
sliding_window = 15
# Convert x_train, x_test, y_train, y_test into 3d array (samples,
timesteps, features) for LSTM input
dataXtrain = []
for i in range(len(x_train)-sliding_window-1):
a = x_train[i:(i+sliding_window), 0:(x_train.shape[1])]
dataXtrain.append(a)
dataXtest = []
for i in range(len(x_test)-sliding_window-1):
a = x_test[i:(i+sliding_window), 0:(x_test.shape[1])]
dataXtest.append(a)
dataYtrain = []
for i in range(len(y_train)-sliding_window-1):
dataYtrain.append(y_train[i + sliding_window])
dataYtest = []
for i in range(len(y_test)-sliding_window-1):
dataYtest.append(y_test[i + sliding_window])
# Make data the divisible by a variety of batch_sizes for training
# Started at 1000 to not include replaced NaN values
dataXtrain = np.array(dataXtrain[1000:172008])
dataYtrain = np.array(dataYtrain[1000:172008])
dataXtest = np.array(dataXtest[1000:83944])
dataYtest = np.array(dataYtest[1000:83944])
# Checking input shapes
print('dataXtrain size is: {}'.format((dataXtrain).shape))
print('dataXtest size is: {}'.format((dataXtest).shape))
print('dataYtrain size is: {}'.format((dataYtrain).shape))
print('dataYtest size is: {}'.format((dataYtest).shape))
### ACTUAL LSTM MODEL
batch_size = 256
timesteps = dataXtrain.shape[1]
features = dataXtrain.shape[2]
# Model set-up, stacked 4 layer stateful LSTM
model = Sequential()
model.add(LSTM(512, return_sequences=True, stateful=True,
batch_input_shape=(batch_size, timesteps, features)))
model.add(LSTM(256,stateful=True, return_sequences=True))
model.add(LSTM(256,stateful=True, return_sequences=True))
model.add(LSTM(128,stateful=True))
model.add(Dense(1, activation='linear'))
model.summary()
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.9, patience=5, min_lr=0.000001, verbose=1)
def coeff_determination(y_true, y_pred):
from keras import backend as K
SS_res = K.sum(K.square( y_true-y_pred ))
SS_tot = K.sum(K.square( y_true - K.mean(y_true) ) )
return ( 1 - SS_res/(SS_tot + K.epsilon()) )
model.compile(loss='mse',
optimizer='nadam',
metrics=[coeff_determination,'mse','mae','mape'])
history = model.fit(dataXtrain, dataYtrain,validation_data=(dataXtest, dataYtest),
epochs=100,batch_size=batch_size, shuffle=False, verbose=1, callbacks=[reduce_lr])
score = model.evaluate(dataXtest, dataYtest,batch_size=batch_size, verbose=1)
print(score)
predictions = model.predict(dataXtest, batch_size=batch_size)
print(predictions)
import matplotlib.pyplot as plt
%matplotlib inline
#plt.plot(history.history['mean_squared_error'])
#plt.plot(history.history['val_mean_squared_error'])
plt.plot(history.history['coeff_determination'])
plt.plot(history.history['val_coeff_determination'])
#plt.plot(history.history['mean_absolute_error'])
#plt.plot(history.history['mean_absolute_percentage_error'])
#plt.plot(history.history['val_mean_absolute_percentage_error'])
#plt.title("MSE")
plt.ylabel("R2")
plt.xlabel("epoch")
plt.legend(["train", "val"], loc="best")
plt.show()
plt.plot(history.history["loss"][5:])
plt.plot(history.history["val_loss"][5:])
plt.title("model loss")
plt.ylabel("loss")
plt.xlabel("epoch")
plt.legend(["train", "val"], loc="best")
plt.show()
plt.figure(figsize=(20,8))
plt.plot(dataYtest)
plt.plot(predictions)
plt.title("Prediction")
plt.ylabel("Price")
plt.xlabel("Time")
plt.legend(["Truth", "Prediction"], loc="best")
plt.show()

Maybe you should remember you are predicting sock returns, which it's very likely to predict nothing. So val_loss increasing is not overfitting at all. Instead of adding more dropouts, maybe you should think about adding more layers to increase it's power.

Try to reduce learning rate much (and remove dropouts for now).
Why do you use
shuffle=False
in fit() function?

Keras always predicting the same output

Keras will always predict the same class for every input i give hm. There are currently four classes.
News, Weather, Sport and Economy.
The training set consists of a lot of different texts, where the class is the same as its topic. There are a lot more texts classified as News and Sport, than there are texts for Weather and Economy.
News: 12112 texts
Weather: 1685 texts
Sport: 13669 texts
economy: 1282 texts
I would have expected the model to be biased towards Sport and News, but instead it is completely biased towards Weather with every input beeing classified as Weather with at least 80% confidence.
Just to add to my confusion: While training the annotator will reach accuracy scores from 95% to 100%(sic!). I guess I am doing something really stupid here but I don't know what it is.
This one is how i call my classifier. It runs on python 3 on a Windows pc.
with open('model.json') as json_data:
model_JSON = json.load(json_data)
model_JSON = json.dumps(model_JSON)
model = model_from_json(model_JSON)
model.load_weights('weights.h5')
text = str(text.decode())
encoded = one_hot(text, max_words, split=" ")
tokenizer = Tokenizer(num_words=max_words)
matrix = tokenizer.sequences_to_matrix([encoded], mode='binary')
result = model.predict(matrix)
legende = ["News", "Wetter", "Sport", "Wirtschaft"]
print(str(legende))
print(str(result))
cat = numpy.argmax(result)
return str(legende[cat]).encode()
This one is how I train my classifier. I omitted the part where I fetch the data from a database. This is done on a Linux VM.
I already tried changing the loss and activation around, but nothing happened.
Also I am curently trying to use more epochs but up to now that hasn't helped either.
max_words = 10000
batch_size=32
epochs=15
rows = cursor.fetchall()
X = []
Y = []
# Einlesen der Rows
for row in rows:
X.append(row[5])
Y.append(row[1])
num_classes = len(set(Y))
Y = one_hot("$".join(Y), num_classes, split="$")
for i in range(len(X)):
X[i] = one_hot(str(X[i]), max_words, split=" ")
split = round(len(X) * 0.2)
x_test = np.asarray(X[0:int(split)])
y_test = np.asarray(Y[0:int(split)])
x_train = np.asarray(X[int(split):len(X)])
y_train = np.asarray(Y[int(split):len(X)])
print('x_test shape', x_test.shape)
print('y_test shape', y_test.shape)
print(num_classes, 'classes')
#vektorisieren
tokenizer = Tokenizer(num_words=max_words)
x_train = tokenizer.sequences_to_matrix(x_train, mode='binary')
x_test = tokenizer.sequences_to_matrix(x_test, mode='binary')
#klassenvektor zu binärer klassenmatrix
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
#model erstellen
model = Sequential()
model.add(Dense(512, input_shape=(max_words,)))
model.add(Dropout(0.5))
model.add(Dense(num_classes))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
history = model.fit(x_train, y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_split=0.1
)
score = model.evaluate(x_test, y_test,
batch_size=batch_size,
verbose=1
)
print('Test score', score[0])
print('Test accuracy', score[1])
#write model to json
print("writing model to json")
model_json = model.to_json()
with open("model.json", 'w') as json_file:
json_file.write(model_json)
#save weights as hdf5
print("saving weights to hdf5")
model.save_weights("weights.h5")

Thanks to the tip that #Daniel Möller gave me I found out what the problem was. His tip was to look at how many instances of each Class are contained in your training set.
In my case I found out, that hashing your classes with One_Hot is not smart, as it will sometimes encode multiple classes with the same number. For me One_Hot encoded nearly everything as a 1. This way Keras learned to only predict 1.