Develop Reference

Accuracy is not increasing in images classification when using TensorFlow model

At my job Interview yesterday, I was asked to build a neural network using TesnorFlow in python to classify images from the flowers images dataset.
But even though it should've worked theoretically, for some reason I couldn't increase the accuracy above 20s%.
Python Version: 3.8.13
TensorFlow Versioin: 2.4.1
The data preprocessing methods from the interviewer were given as follows:
# create datase
IMG_SIZE = 160
BATCH_SIZE = 32
AUTOTUNE = tf.data.experimental.AUTOTUNE
def _parse_data(x,y):
image = tf.io.read_file(x)
image = tf.image.decode_jpeg(image, channels=3)
image = tf.cast(image, dtype=tf.float32)
image = tf.math.l2_normalize(image)
image = tf.image.resize(image, (IMG_SIZE, IMG_SIZE))
return image,y
def _input_fn(x,y):
ds = tf.data.Dataset.from_tensor_slices((x,y))
ds = ds.map(_parse_data)
ds = ds.shuffle(buffer_size=data_size)
ds = ds.repeat()
ds = ds.batch(BATCH_SIZE)
ds = ds.prefetch(buffer_size=AUTOTUNE)
return ds
train_ds = _input_fn(x_train, y_train)
validation_ds = _input_fn(x_valid, y_valid)
With both training and validation datasets being
<PrefetchDataset shapes: ((None, 160, 160, 3), (None,)), types:
(tf.float32, tf.int32)>
With the network being as follows:
from tensorflow.keras import datasets, layers, models
model_seq = models.Sequential()
model_seq.add(layers.experimental.preprocessing.RandomFlip("horizontal",input_shape=(IMG_SIZE,IMG_SIZE,3)))
model_seq.add(layers.experimental.preprocessing.RandomRotation(0.2))
model_seq.add(layers.experimental.preprocessing.Rescaling(1./255))
model_seq.add(layers.Conv2D(16, 3, padding='same', activation='relu'))
model_seq.add(layers.MaxPooling2D())
model_seq.add(layers.Conv2D(32, 3, padding='same', activation='relu'))
model_seq.add(layers.MaxPooling2D())
model_seq.add(layers.Conv2D(64, 3, padding='same', activation='relu'))
model_seq.add(layers.MaxPooling2D())
model_seq.add(layers.Dropout(0.2))
model_seq.add(layers.Flatten())
model_seq.add(layers.Dense(128, activation='relu'))
model_seq.add(layers.Dense(len(label_names), activation='softmax'))
model_seq.summary()
The output layer being the only thing that isn't allowed to be change.
model_seq.add(layers.Dense(len(label_names), activation='softmax'))
(Please note I was for some reason asked to use model_seq.add(), and even though it could be triggering for some of you, please ignore it this once :) )
For compiling the model, I used the following:
model_seq.compile(optimizer="Adam",
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
And for fitting the model:
history = model_seq.fit(train_ds,epochs=20,
validation_data = validation_ds,
steps_per_epoch=100,validation_steps=100)
The things I've tried:
Using different Augmentation methods (or removing the whole section
from the network).
Changing the Batch and Image sizes.
Using Dropout layers.
Using early stopping as follows:
callback = tf.keras.callbacks.EarlyStopping(
monitor='val_loss',
min_delta=0, patience=3, verbose=0,
mode='auto',baseline=None,
restore_best_weights=True)
history = model_seq.fit(train_ds,epochs=20,
validation_data = validation_ds,
steps_per_epoch=100,validation_steps=100,
callbacks = [callback])
Yet despite all of the above, I couldn't get any results. Since I couldn't find out what I did wrong exactly, I'm hoping someone here could tell me, so I could learn from this experience.
(Please take into consideration that I wasn't allowed to change the preprocessing functions, with the parameters IMG_SIZE and BATCH_SIZE being the only exception).

“TLDR: If you want to use their preprocessing part and don't change anything go to First Approach. If you want to augment images, Go to Second Approach and use ImageDataGenerator”.
First Approach
As you say: I had to use the preprocessing functions and don't change this and because I don't access your data, I use cifar10 dataset and use your preprocessing part. (only line of reading from file changed).
Because you shouldn't change IMG_SIZE=160 in the preprocessing part, I add this layer : tf.keras.layers.Lambda(lambda image: tf.image.resize(image, (32, 32)))) to the network because working with large images causes a crash.
You don't need a very large network, first check with a small network then step by step add parameters then add layers.
We can get a better result like the below: (On cifar10 with your network I get 10% accuracy like you.)
import tensorflow as tf
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.cifar10.load_data()
# create datase
IMG_SIZE = 160
BATCH_SIZE = 32
data_size = 32
AUTOTUNE = tf.data.experimental.AUTOTUNE
def _parse_data(x,y):
# image = tf.io.read_file(x) <- because don't read from path
image = x
# image = tf.image.decode_jpeg(image, channels=3) <- because don't read from path and don't have jpeg
image = tf.cast(image, dtype=tf.float32)
image = tf.math.l2_normalize(image)
image = tf.image.resize(image, (IMG_SIZE, IMG_SIZE))
return image,y
def _input_fn(x,y):
ds = tf.data.Dataset.from_tensor_slices((x,y))
ds = ds.map(_parse_data)
ds = ds.shuffle(buffer_size=data_size)
ds = ds.repeat()
ds = ds.batch(BATCH_SIZE)
ds = ds.prefetch(buffer_size=AUTOTUNE)
return ds
train_ds = _input_fn(X_train, y_train)
validation_ds = _input_fn(X_test, y_test)
model = tf.keras.Sequential([
tf.keras.Input(shape=(160, 160, 3)),
tf.keras.layers.Lambda(lambda image: tf.image.resize(image, (32, 32))),
tf.keras.layers.Conv2D(filters=32,kernel_size=(3,3),activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(50,activation='relu'),
tf.keras.layers.Dense(10,activation='softmax')
])
model.compile(optimizer="Adam",
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['accuracy'])
history = model.fit(train_ds,epochs=4,
validation_data = validation_ds,
steps_per_epoch=100,validation_steps=100)
Output:
Epoch 1/4
100/100 [==============================] - 8s 38ms/step - loss: 2.2445 - accuracy: 0.1375 - val_loss: 2.1406 - val_accuracy: 0.2138
Epoch 2/4
100/100 [==============================] - 3s 33ms/step - loss: 2.0552 - accuracy: 0.2688 - val_loss: 1.9764 - val_accuracy: 0.3250
Epoch 3/4
100/100 [==============================] - 4s 38ms/step - loss: 1.9468 - accuracy: 0.3022 - val_loss: 1.9014 - val_accuracy: 0.3200
Epoch 4/4
100/100 [==============================] - 4s 36ms/step - loss: 1.8936 - accuracy: 0.3341 - val_loss: 1.8883 - val_accuracy: 0.3419
Second Approach: Augment images with ImageDataGenerator:
import tensorflow as tf
flowers = tf.keras.utils.get_file(
'flower_photos',
'https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz',
untar=True)
data_gen = tf.keras.preprocessing.image.ImageDataGenerator(
rescale=1./255,
rotation_range = 10, # Degree range for random rotations.
horizontal_flip = True, # Randomly flip inputs horizontally.
vertical_flip = True, # Randomly flip inputs vertically.
)
imgs_dataset = data_gen.flow_from_directory(flowers, class_mode='categorical',
target_size=(160, 160), batch_size=32,
shuffle=True)
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(filters=32,kernel_size=(3,3),input_shape=(160, 160, 3),activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(50,activation='relu'),
tf.keras.layers.Dense(5,activation='softmax')
])
model.compile(optimizer='adam', loss='categorical_crossentropy',metrics=['accuracy'])
model.fit(imgs_dataset,epochs=5)
Output:
Found 3670 images belonging to 5 classes.
Epoch 1/5
115/115 [==============================] - 39s 311ms/step - loss: 1.7904 - accuracy: 0.4161
Epoch 2/5
115/115 [==============================] - 27s 236ms/step - loss: 1.0878 - accuracy: 0.5605
Epoch 3/5
115/115 [==============================] - 28s 244ms/step - loss: 1.0252 - accuracy: 0.6005
Epoch 4/5
115/115 [==============================] - 27s 233ms/step - loss: 0.9735 - accuracy: 0.6196
Epoch 5/5
115/115 [==============================] - 29s 248ms/step - loss: 0.9313 - accuracy: 0.6455

Keras - val_accuraccy stuck at 0.0000e+00 in LSTM model [duplicate]

This question already has answers here:
Keras NN regression model gives low loss, and 0 acuracy
(2 answers)
Closed 2 years ago.
Below is my Keras model for predicting cryptocurrency prices. The problem is although the loss and val_loss decreases, the accuracy is stuck at a certain value (2.5840e-04) and doesn't change, and the val_accuracy is stuck at 0.0000e+00. I checked each of my inputs thoroughly, but I couldn't find anything wrong. Is there a problem with my model?
(This is a link to my notebook if you need)
This is my data preparation (ethusd.csv is a standard stock format dataset. you can see it in my notebook in the link above)
DATASET_PATH = "../input/crifier/ethusd.csv"
import math
import matplotlib.pyplot as plt
import tensorflow.keras as keras
import pandas as pd
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import LSTM
from tensorflow.keras.layers import Dropout
from tensorflow.keras.layers import *
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.optimizers import Adam
import tensorflow as tf
from sklearn.utils import shuffle
df=pd.read_csv(DATASET_PATH)
df = df[-100000:-90000]
global WINDOW
global LEN_TRAIN
WINDOW = 400
TRAIN_RATIO = 0.9
LEN_TRAIN = int(LEN_DF*TRAIN_RATIO)
LEN_TEST = int(LEN_DF - LEN_TRAIN)
training_set = df.iloc[:LEN_TRAIN, 1:2].values
test_set = df.iloc[LEN_TRAIN:, 1:2].values
def reshaper(dataset):
sc = MinMaxScaler(feature_range = (0, 1))
dataset_scaled = sc.fit_transform(dataset)
# Creating a data structure with 60 time-steps and 1 output
X_scaled = []
y_scaled = []
for i in range(0, len(dataset)-WINDOW):
X_scaled.append(dataset_scaled[i:i+WINDOW, 0])
y_scaled.append(dataset_scaled[i+WINDOW, 0])
X_scaled, y_scaled = np.array(X_scaled), np.array(y_scaled)
X_scaled = np.reshape(X_scaled, (X_scaled.shape[0], X_scaled.shape[1], 1))
return X_scaled, y_scaled,sc
X_train, y_train, sc_train = reshaper(training_set)
X_train, y_train = shuffle(X_train, y_train)
X_test, y_test, sc_test = reshaper(test_set)
And this is my model
model = Sequential()
#Adding the first LSTM layer and some Dropout regularisation
model.add(LSTM(units = 50, return_sequences = True, input_shape = (X_train.shape[1], 1)))
model.add(Dropout(0.2))
# Adding a second LSTM layer and some Dropout regularisation
model.add(LSTM(units = 50, return_sequences = True))
model.add(Dropout(0.2))
# Adding a third LSTM layer and some Dropout regularisation
model.add(LSTM(units = 50, return_sequences = True))
model.add(Dropout(0.2))
# Adding a fourth LSTM layer and some Dropout regularisation
model.add(LSTM(units = 50))
model.add(Dropout(0.2))
# Adding the output layer
model.add(Dense(units = 1))
# Compiling the RNN
model.compile(optimizer = Adam(learning_rate=0.001), loss = 'mean_squared_error',metrics=['accuracy'])
model.fit(X_train, y_train, validation_split=0.1, epochs = 100, batch_size = 1000, verbose=2)
I keep getting the following accuracy and val_accuracy
Epoch 1/100
8/8 - 3s - loss: 0.0991 - accuracy: 1.2920e-04 - val_loss: 0.0046 - val_accuracy: 0.0000e+00
Epoch 2/100
8/8 - 2s - loss: 0.0217 - accuracy: 2.5840e-04 - val_loss: 0.0068 - val_accuracy: 0.0000e+00
Epoch 3/100
8/8 - 2s - loss: 0.0132 - accuracy: 2.5840e-04 - val_loss: 0.0047 - val_accuracy: 0.0000e+00
.
.
.
Epoch 28/100
8/8 - 2s - loss: 0.0036 - accuracy: 2.5840e-04 - val_loss: 0.0011 - val_accuracy: 0.0000e+00
Epoch 29/100
8/8 - 2s - loss: 0.0034 - accuracy: 2.5840e-04 - val_loss: 0.0012 - val_accuracy: 0.0000e+00
Epoch 30/100
8/8 - 2s - loss: 0.0033 - accuracy: 2.5840e-04 - val_loss: 0.0011 - val_accuracy: 0.0000e+00

Your model is regression model (you are predicting float not category). Accuracy is not applicable here.
When calculating accuracy keras is comparing your output to labels. E.g. it will give you zero accuracy even if your output is say 0.499999, but label is 0.5

Why is my CNN pre trained image classifier overfitting?

I have just started with Computer Vision and in the current task i am classifying images in 4 categories.
Total number of image files=1043
I am using pretrained InceptionV3 and fine tuning it on my dataset.
This is what i have after the epoch:
Epoch 1/5
320/320 [==============================] - 1925s 6s/step - loss: 0.4318 - acc: 0.8526 - val_loss: 1.1202 - val_acc: 0.5557
Epoch 2/5
320/320 [==============================] - 1650s 5s/step - loss: 0.1807 - acc: 0.9446 - val_loss: 1.2694 - val_acc: 0.5436
Epoch 3/5
320/320 [==============================] - 1603s 5s/step - loss: 0.1236 - acc: 0.9572 - val_loss: 1.2597 - val_acc: 0.5546
Epoch 4/5
320/320 [==============================] - 1582s 5s/step - loss: 0.1057 - acc: 0.9671 - val_loss: 1.3845 - val_acc: 0.5457
Epoch 5/5
320/320 [==============================] - 1580s 5s/step - loss: 0.0982 - acc: 0.9700 - val_loss: 1.2771 - val_acc: 0.5572
That is a huge difference. Kindly help me to figure out why is my model not able to generalize as it is fitting quite well on the train data.
my code for reference:-
from keras.utils import to_categorical
from keras.models import Model
from keras.layers import Dense, GlobalAveragePooling2D, Dropout
from keras.applications.inception_v3 import InceptionV3, preprocess_input
CLASSES = 4
# setup model
base_model = InceptionV3(weights='imagenet', include_top=False)
from sklearn.preprocessing import OneHotEncoder
x = base_model.output
x = GlobalAveragePooling2D(name='avg_pool')(x)
x = Dropout(0.4)(x)
predictions = Dense(CLASSES, activation='softmax')(x)
model = Model(inputs=base_model.input, outputs=predictions)
for layer in base_model.layers:
layer.trainable = False
model.compile(optimizer='rmsprop',
loss='categorical_crossentropy',
metrics=['accuracy'])
from sklearn.preprocessing import LabelEncoder
encoder = LabelEncoder()
df['Category']= encoder.fit_transform(df['Category'])
from keras.preprocessing.image import ImageDataGenerator
WIDTH = 299
HEIGHT = 299
BATCH_SIZE = 32
train_datagen = ImageDataGenerator(rescale=1./255,preprocessing_function=preprocess_input)
validation_datagen = ImageDataGenerator(rescale=1./255)
df['Category'] =df['Category'].astype(str)
#dfval['Category'] = dfval['Category'].astype(str)
from sklearn.utils import shuffle
df = shuffle(df)
from sklearn.model_selection import train_test_split
dftrain,dftest = train_test_split(df, test_size = 0.2, random_state = 0)
train_generator = train_datagen.flow_from_dataframe(dftrain,target_size=(HEIGHT, WIDTH),batch_size=BATCH_SIZE,class_mode='categorical', x_col='Path', y_col='Category')
validation_generator = validation_datagen.flow_from_dataframe(dftest,target_size=(HEIGHT, WIDTH),batch_size=BATCH_SIZE,class_mode='categorical', x_col='Path', y_col='Category')
EPOCHS = 5
BATCH_SIZE = 32
STEPS_PER_EPOCH = 320
VALIDATION_STEPS = 64
MODEL_FILE = 'filename.model'
history = model.fit_generator(
train_generator,
epochs=EPOCHS,
steps_per_epoch=STEPS_PER_EPOCH,
validation_data=validation_generator,
validation_steps=VALIDATION_STEPS)
Any help would be appreciated :)

If you don't use preprocess_input in "all" your data, you will get terrible results.
Look at these:
train_datagen = ImageDataGenerator(
preprocessing_function=preprocess_input,
...)
validation_datagen = ImageDataGenerator()
Now, I notice you are using rescale. Since you imported the correct preprocess_input function from the inception code, I really think you should not be using this rescale. The preprocess_input function is supposed to do all the necessary preprocessing. (Not all models were trained with normalized inputs)
But would rescale be a problem if you're applying it to both batasets?
Well... if the trainable=False applied correctly to the BatchNormalization layers, this means that these layers have stored values for mean and variation which will only work well if the data is within the expected range.

Keras - Single pixel classification without ImageDataGenerator

This is a newbie question, but I just can't get the simplest Keras experiment work. I went through a course and its samples work well, so my computer is set up correctly.
I've a few thousand 16x12 images, called "GGYRBGBBW.png", "BBYWBRBBB.png" and so on. The images have a single color with minimal shade differences and are reduced to a single pixel when loading. The first character of the filenames serves as training labels (e.g. green images' names start with 'G'). I need to build and train a simple model that indicates one of the 6 possible colors in the image. (This is a first learning step towards a much more complicated project).
I don't want to use ImageDataGenerator because this whole project will grow beyond that a simple directory-structure categorization can do, and I'll do the image randomization with my own external image generator.
I created a Keras model that looks like this:
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
flatten_1 (Flatten) (None, 3) 0
_________________________________________________________________
dense_1 (Dense) (None, 6) 24
=================================================================
Total params: 24
Trainable params: 24
Non-trainable params: 0
So, pretty simple - input shape is only three values (RGB), flattened, then 6 output neurons for each color category.
When running the experiments, beginning with random initial values, accuracy stays super low, sometimes 0.
210/210 [==============================] - 0s 2ms/step - loss: 7.2430 - acc: 0.2095 - val_loss: 6.7980 - val_acc: 0.2000
Epoch 2/50
210/210 [==============================] - 0s 10us/step - loss: 7.2411 - acc: 0.2095 - val_loss: 9.6617 - val_acc: 0.2000
Epoch 3/50
210/210 [==============================] - 0s 5us/step - loss: 9.9256 - acc: 0.2095 - val_loss: 9.6598 - val_acc: 0.2000
Epoch 4/50
210/210 [==============================] - 0s 5us/step - loss: 9.9236 - acc: 0.2095 - val_loss: 9.6579 - val_acc: 0.2000
Epoch 5/50
210/210 [==============================] - 0s 10us/step - loss: 9.9217 - acc: 0.2095 - val_loss: 9.6560 - val_acc: 0.2000
Epoch 6/50
210/210 [==============================] - 0s 10us/step - loss: 9.9197 - acc: 0.2095 - val_loss: 9.6541 - val_acc: 0.2000
I must be missing something trivial, but since I'm a noob in this, I can't figure out what. Here's the complete source code:
from __future__ import print_function
import random
import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
os.system('cls')
import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras import backend as K
from keras.preprocessing.image import img_to_array
import numpy as np
num_classes = 6
batch_size = 300
epochs = 50
Image_width, Image_height = 1, 1
train_dir = './OneColorTrainingData'
colors="WYBGRO"
def load(numOfPics):
X = []
y = []
print("Reading training images")
allFiles = os.listdir(train_dir)
randomFiles = random.choices(allFiles, k=numOfPics)
for f in randomFiles:
path = os.path.join(train_dir, f)
img = keras.preprocessing.image.load_img(path, grayscale=False, target_size=(Image_width, Image_width))
img = img_to_array(img)
img /= 255
X.append(img)
y.append(colors.index(f[0]))
y = keras.utils.to_categorical(y, num_classes=num_classes)
return X, y
Data, labels = load(batch_size)
print(str(len(Data)) + " training files loaded")
print(labels)
model = Sequential()
model.add(Flatten(input_shape=(Image_height, Image_width, 3)))
model.add(Dense(num_classes, activation=K.tanh))
model.compile(loss=keras.losses.categorical_crossentropy, optimizer=keras.optimizers.Adadelta(), metrics=['accuracy'])
print(model.input_shape)
print(model.summary())
hist = model.fit(np.array(Data), np.array(labels), batch_size=batch_size, epochs=epochs, verbose=1, validation_split=0.3)
score = model.evaluate(np.array(Data), np.array(labels), verbose=0)
print('Test loss: ', score[0])
print('Test accuracy: ', score[1])
Any help would be appreciated.

You have K.tanh as your finaly Dense layer activation. When doing 1-of-many classification we often use softmax instead which will produce a probability distribution over the colour classes:
model.add(Dense(num_classes, activation='softmax'))
Now your target labels will be one-hot vectors [0,0,1,0,0,0] indication which class it is. You can also use sparse_categorical_crossentropy loss and give labels as class integers 2 as your target. It means the same thing in this context.

I'm not positive, but I think you need a hidden layer. You currently have a single affine transformation on your input: h_0 = W_0*x + b_0. This is passed through a K.tanh, so your logits are simply y = tanh(h_0). I have a hunch that for a small enough problem like this you may be able to prove that this cannot converge, but I am not certain.
I would just add a second dense layer, and use softmax for your final output as #nuric suggests:
model = Sequential()
model.add(Flatten(input_shape=(Image_height, Image_width, 3)))
model.add(Dense(10, activation=K.tanh))
model.add(Dense(num_classes, activation='softmax'))

OK, I figured it out. I needed to switch to softmax (thanks all who suggested), and use much more epochs because convergence was random and slow. I didn't think I needed to use more epochs because nothing changed after the first few - but it turned out that once I had 500 instead of 50, the network managed to learn all the colors one by one and hit 100% accuracy (with a single output layer) almost every time. Thank you all for the help!

Keras LSTM + TensorFlow and a number sequence (improve loss)

first of all, I'm running with the following setup:
Running on windows 10
Python 3.6.2
TensorFlow 1.8.0
Keras 2.1.6
I'm trying to predict, or at least guesstimate the following number sequence:
https://codepen.io/anon/pen/RJRPPx (limited to 20,000 for testing), the full sequence contains about one million records.
And here is the code (run.py)
import lstm
import time
import matplotlib.pyplot as plt
def plot_results(predicted_data, true_data):
fig = plt.figure(facecolor='white')
ax = fig.add_subplot(111)
ax.plot(true_data, label='True Data')
plt.plot(predicted_data, label='Prediction')
plt.legend()
plt.show()
def plot_results_multiple(predicted_data, true_data, prediction_len):
fig = plt.figure(facecolor='white')
ax = fig.add_subplot(111)
ax.plot(true_data, label='True Data')
#Pad the list of predictions to shift it in the graph to it's correct start
for i, data in enumerate(predicted_data):
padding = [None for p in range(i * prediction_len)]
plt.plot(padding + data, label='Prediction')
plt.legend()
plt.show()
#Main Run Thread
if __name__=='__main__':
global_start_time = time.time()
epochs = 10
seq_len = 50
print('> Loading data... ')
X_train, y_train, X_test, y_test = lstm.load_data('dice_amplified/primeros_20_mil.csv', seq_len, True)
print('> Data Loaded. Compiling...')
model = lstm.build_model([1, 50, 100, 1])
model.fit(
X_train,
y_train,
batch_size = 512,
nb_epoch=epochs,
validation_split=0.05)
predictions = lstm.predict_sequences_multiple(model, X_test, seq_len, 50)
#predicted = lstm.predict_sequence_full(model, X_test, seq_len)
#predicted = lstm.predict_point_by_point(model, X_test)
print('Training duration (s) : ', time.time() - global_start_time)
plot_results_multiple(predictions, y_test, 50)
I have tried to:
increase and decrease epochs.
increase and decrease batch size.
amplify the data.
The following plot represents:
epochs = 10
batch_size = 512
validation_split = 0.05
As well, as far as I understand, the loss should be decreasing with more epochs? Which doesn't seem to be happening!
Using TensorFlow backend.
> Loading data...
> Data Loaded. Compiling...
> Compilation Time : 0.03000473976135254
Train on 17056 samples, validate on 898 samples
Epoch 1/10
17056/17056 [==============================] - 31s 2ms/step - loss: 29927.0164 - val_loss: 289.8873
Epoch 2/10
17056/17056 [==============================] - 29s 2ms/step - loss: 29920.3513 - val_loss: 290.1069
Epoch 3/10
17056/17056 [==============================] - 29s 2ms/step - loss: 29920.4602 - val_loss: 292.7868
Epoch 4/10
17056/17056 [==============================] - 27s 2ms/step - loss: 29915.0955 - val_loss: 286.7317
Epoch 5/10
17056/17056 [==============================] - 26s 2ms/step - loss: 29913.6961 - val_loss: 298.7889
Epoch 6/10
17056/17056 [==============================] - 26s 2ms/step - loss: 29920.2068 - val_loss: 287.5138
Epoch 7/10
17056/17056 [==============================] - 28s 2ms/step - loss: 29914.0650 - val_loss: 295.2230
Epoch 8/10
17056/17056 [==============================] - 25s 1ms/step - loss: 29912.8860 - val_loss: 295.0592
Epoch 9/10
17056/17056 [==============================] - 28s 2ms/step - loss: 29907.4067 - val_loss: 286.9338
Epoch 10/10
17056/17056 [==============================] - 46s 3ms/step - loss: 29914.6869 - val_loss: 289.3236
Any recommendations? How could I improve it? Thanks!
Lstm.py contents:
import os
import time
import warnings
import numpy as np
from numpy import newaxis
from keras.layers.core import Dense, Activation, Dropout
from keras.layers.recurrent import LSTM
from keras.models import Sequential
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' #Hide messy TensorFlow warnings
warnings.filterwarnings("ignore") #Hide messy Numpy warnings
def load_data(filename, seq_len, normalise_window):
f = open(filename, 'rb').read()
data = f.decode().split('\n')
sequence_length = seq_len + 1
result = []
for index in range(len(data) - sequence_length):
result.append(data[index: index + sequence_length])
if normalise_window:
result = normalise_windows(result)
result = np.array(result)
row = round(0.9 * result.shape[0])
train = result[:int(row), :]
np.random.shuffle(train)
x_train = train[:, :-1]
y_train = train[:, -1]
x_test = result[int(row):, :-1]
y_test = result[int(row):, -1]
x_train = np.reshape(x_train, (x_train.shape[0], x_train.shape[1], 1))
x_test = np.reshape(x_test, (x_test.shape[0], x_test.shape[1], 1))
return [x_train, y_train, x_test, y_test]
def normalise_windows(window_data):
normalised_data = []
for window in window_data:
normalised_window = [((float(p) / float(window[0])) - 1) for p in window]
normalised_data.append(normalised_window)
return normalised_data
def build_model(layers):
model = Sequential()
model.add(LSTM(
input_shape=(layers[1], layers[0]),
output_dim=layers[1],
return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(
layers[2],
return_sequences=False))
model.add(Dropout(0.2))
model.add(Dense(
output_dim=layers[3]))
model.add(Activation("linear"))
start = time.time()
model.compile(loss="mse", optimizer="rmsprop")
print("> Compilation Time : ", time.time() - start)
return model
def predict_point_by_point(model, data):
#Predict each timestep given the last sequence of true data, in effect only predicting 1 step ahead each time
predicted = model.predict(data)
predicted = np.reshape(predicted, (predicted.size,))
return predicted
def predict_sequence_full(model, data, window_size):
#Shift the window by 1 new prediction each time, re-run predictions on new window
curr_frame = data[0]
predicted = []
for i in range(len(data)):
predicted.append(model.predict(curr_frame[newaxis,:,:])[0,0])
curr_frame = curr_frame[1:]
curr_frame = np.insert(curr_frame, [window_size-1], predicted[-1], axis=0)
return predicted
def predict_sequences_multiple(model, data, window_size, prediction_len):
#Predict sequence of 50 steps before shifting prediction run forward by 50 steps
prediction_seqs = []
for i in range(int(len(data)/prediction_len)):
curr_frame = data[i*prediction_len]
predicted = []
for j in range(prediction_len):
predicted.append(model.predict(curr_frame[newaxis,:,:])[0,0])
curr_frame = curr_frame[1:]
curr_frame = np.insert(curr_frame, [window_size-1], predicted[-1], axis=0)
prediction_seqs.append(predicted)
return prediction_seqs
Addendum:
At nuric suggestion I modified the model as following:
def build_model(layers):
model = Sequential()
model.add(LSTM(input_shape=(layers[1], layers[0]), output_dim=layers[1], return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(layers[2], return_sequences=False))
model.add(Dropout(0.2))
model.add(Dense(output_dim=layers[3]))
model.add(Activation("linear"))
model.add(Dense(64, input_dim=50, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(1))
start = time.time()
model.compile(loss="mse", optimizer="rmsprop")
print("> Compilation Time : ", time.time() - start)
return model
Still a bit lost on this one...

Even though you normalise the input, you don't normalise the output. The LSTM by default has a tanh output which means you will have a limited feature space, ie the dense layer won't be able to regress to large numbers.
You have a fixed length numerical input (50,), directly pass that to Dense layers with relu activation and will perform better on regression tasks, something simple like:
model = Sequential()
model.add(Dense(64, input_dim=50, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(1))
For regression it is also preferable to use l2 regularizers instead of Dropout because you are not really feature extracting for classification etc.