Develop Reference

Why is my Tensorflow LSTM Timeseries model returning only one value into the future

I have a tensorflow model for predicting Timeseries values using LSTM, it trains fine but when I ask it to predict some values in time it only gives me the T+1 value,
How can I make it to give me from T+1 to T+n values instead of just T+1.
I thought about giving him back the predicted value to analyse again in a loop, e.g.
We look back at 20 samples for this example
T±0 = now value
T-k = value k steps into the past (known)
T+n = value n steps into the future (unkown at the start)
--- Algorithm
T+1 = model.predict(data from T-20 to T±0)
T+2 = model.predict(data from T-19 to T+1) #using the previously found T+1 value
T+3 = model.predict(data from T-18 to T+2) #using the previously found T+1 and T+2 values
.
.
.
T+n = model.predict(data from T-(k-n) to T+(n-1)) # using the previously found T+1 .. T+(n-1) values
The thing is that T+1 has an mean absolute error of around 0.75%, doesn't the error propagate/compound through the predictions ? If it does it means that if I ask the program to predict T+10 it will have a mean absolute error of 0.75%^10 = ~7.7%, which is not very good in my case. So I'm looking for other ways to predict up to T+n values.
I've looked at few youtube tutorials but each time it seems that their call of model.predict(X) returns multiple values already, and I have no idea about what parameters I could have missed.
Code :
import tensorflow.keras as tfks
import pandas as pd
import numpy as np
def model_training(dataframe,folder_name, window_size=40, epochs=100, batch_size=64):
"""Function to start training the model on given data
Parameters
----------
dataframe : `pandas.DataFrame` The dataframe to train the model on
window_size : `int` The size of the lookback window to use
epochs : `int` The number of epochs to train for
batch_size : `int` The batch size to use for training
Returns
-------
None
"""
dataframe,_ = Process.pre(dataframe) #function to standardize each column of the data
TRAIN_SIZE = 0.7
VAL_SIZE = 0.2
TEST_SIZE = 0.1
#Splitting the data into train, validation and test sets
x,y = dataframe_to_xy(dataframe, window_size) #converts pandas dataframe to numpy array
x_train,y_train = x[:int(len(dataframe)*TRAIN_SIZE)],y[:int(len(dataframe)*TRAIN_SIZE)]
x_val,y_val = x[int(len(dataframe)*TRAIN_SIZE):int(len(dataframe)*(TRAIN_SIZE+VAL_SIZE))],y[int(len(dataframe)*TRAIN_SIZE):int(len(dataframe)*(TRAIN_SIZE+VAL_SIZE))]
x_test,y_test = x[int(len(dataframe)*(TRAIN_SIZE+VAL_SIZE)):],y[int(len(dataframe)*(TRAIN_SIZE+VAL_SIZE)):]
#Creating the model base
model = tfkr.models.Sequential()
model.add(tfkr.layers.InputLayer(input_shape=(window_size, 10)))
model.add(tfkr.layers.LSTM(64))
model.add(tfkr.layers.Dense(8, 'relu'))
model.add(tfkr.layers.Dense(10, 'linear'))
model.summary()
#Compiling and saving the model
cp = tfkr.callbacks.ModelCheckpoint('ai\\models\\'+folder_name+'\\', save_best_only=True)
model.compile(loss=tfkr.losses.MeanSquaredError(), optimizer=tfkr.optimizers.Adam(learning_rate=0.0001), metrics=[tfkr.metrics.RootMeanSquaredError()])
model.fit(x_train, y_train, epochs=epochs, batch_size=batch_size, validation_data=(x_val, y_val), callbacks=[cp])
def predict_data(model,data_pre,window_size,n_future):
'''Function to predict data using the model
Parameters
----------
model : `tensorflow.keras.models.Sequential` The model to use for prediction
data_pre : `pandas.DataFrame` The dataframe to predict on
window_size : `int` The size of the lookback window to use
n_future : `int` Number of values to predict in the future
Returns
-------
data_pred : `pandas.DataFrame` The dataframe containing the predicted values
'''
time_interval = data_pre.index[1] - data_pre.index[0]
#Setting up the dataframe to predict on
data_pre, proc_params = Process.pre(data_pre) #function to standardize each column of the data
data_pred = data_pre.iloc[-window_size:]
data_pred = data_pred.to_numpy().astype('float32')
data_pred = data_pred.reshape(1,window_size,10)
#Predicting the data
data_pred = model.predict(data_pred)
#Converting the data from numpy array to pandas dataframe and doing some formatting + post-processing/reversing standardization
#yada yada pandas dataframe
return data_pred
If there is no way preventing the error propagation, do you have any tips for reducting the error of the model ?
Thanks in advance.

You could let your LSTM return the full sequences, not only the last output, like this:
model = tfkr.models.Sequential()
model.add(tfkr.layers.InputLayer(input_shape=(window_size, 10)))
model.add(tfkr.layers.LSTM(64, return_sequences=True))
model.add(tfkr.layers.Dense(8, 'relu'))
model.add(tfkr.layers.Dense(10, 'linear'))
Each LSTM output would go through the same dense weights (because we did not flatten). Your output would be then of shape
(None, window_size, 10)
This means that for k input time points you would get k output time points.
Downside is, that the first output would be calculated only using the first input. The second output only by using the first two inputs, and so on. So I would suggest to use a bidirectional LSTM, and combine both directions, maybe like this:
model = tfkr.models.Sequential()
model.add(tfkr.layers.InputLayer(input_shape=(window_size, 10)))
model.add(tfkr.layers.Bidirectional(tfkr.layers.LSTM(64, return_sequences=True), merge_mode='sum')
model.add(tfkr.layers.Dense(8, 'relu'))
model.add(tfkr.layers.Dense(10, 'linear'))

One way which you can do this is, lets assume you initially trained your model something like this:
x_train = [
[1, 2, 3, 4, 5, 6, 7],
[2, 3, 4, 5, 6, 7, 8],
]
y_train = [
[8],
[9]
]
And the training was done as such using the model.fit with all appropriate reshaping.
model.fit(x_train, y_train)
Then, you would expect something like this when you try to predict
ans = model.predict(x_train[0])
ans: 8
But lets say you wanted it predict the next value as well, with the expected output as 8 and then 9, or for n expected outputs, you would expect a result such as 8, 9, 10, 11... and so on.
Then I suggest to approach the problem the following way:
def predict_continuous(model, input_arr, num_ahead):
# Your input arr was [1,2,3,4,5,6,7]
# You want the output to go beyond just 8, lets say 8,9,10,11,...
# num_ahead: How much into the future you need predictions.
predictions = []
model_input = input_arr
for i in range(num_ahead):
predictions.append(model.predict(model_input)) # Get the prediction, and subsequent predictions are appended
# Since the LSTM model is trained for a particular input size, we need to keep dynamically changing the input
# so we strip off the first value from the input array, and append the latest prediction as its last value.
model_input = model_input[1:].append(predictions[-1])
return predictions
Now this won't work directly, you will have to do some reshaping, however the core idea remains the same.

Determining the Right Shape for a RNN with a Integer Sequence Target

I am trying to build an RNN based model in Tensorflow that takes a sequence of categorical values as an input, and sequence of categorical values as the output.
For example, if I have sequence of 30 values, the first 25 would be the training data, and the last 5 would be the target. Imagine the data is something like a person pressing keys on a computer keyboard and recording their key presses over time.
I've tried to feed the training data and targets into this model in different shapes, and I always get an error that indicates the data is in the wrong shape.
I've included a code sample that should run and demonstrate what I'm trying to do and the failure I'm seeing.
In the code sample, I've used windows for batches. So if there are 90 values in the sequence, the first 25 values would be the training data for the first batch, and the next 5 values would be the target. This next batch would be the next 30 values (25 training values, 5 target values).
import numpy as np
import tensorflow as tf
from tensorflow import keras
num_categories = 20
data_sequence = np.random.choice(num_categories, 10000)
def create_target(batch):
X = tf.cast(batch[:,:-5][:,:,None], tf.float32)
Y = batch[:,-5:][:,:,None]
return X,Y
def add_windows(data):
data = tf.data.Dataset.from_tensor_slices(data)
return data.window(20, shift=1, drop_remainder=True)
dataset = tf.data.Dataset.from_tensor_slices(data_sequence)
dataset = dataset.window(30, drop_remainder=True)
dataset = dataset.flat_map(lambda x: x.batch(30))
dataset = dataset.batch(5)
dataset = dataset.map(create_target)
model = keras.models.Sequential([
keras.layers.SimpleRNN(20, return_sequences=True),
keras.layers.SimpleRNN(20, return_sequences=True),
keras.layers.TimeDistributed(keras.layers.Dense(num_categories, activation="softmax"))
])
optimizer = keras.optimizers.Adam()
model.compile(loss="sparse_categorical_crossentropy", optimizer=optimizer)
model.fit(dataset, epochs=1)
The error I get when I run the above code is
Node: 'sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits'
logits and labels must have the same first dimension, got logits shape [125,20] and labels shape [25]
I've also tried the following model, but the errors are similar.
model = keras.models.Sequential([
keras.layers.SimpleRNN(20, return_sequences=True),
keras.layers.SimpleRNN(20),
keras.layers.Dense(num_categories, activation="softmax"))
])
Does anybody have any recommendations about what I need to do to get this working?
Thanks.

I figured out the issue. The size of the time dimension needs to be the same for the training data and the target.
If you look at my original example code, the training data has these shapes
X.shape = (1, 25, 1)
Y.shape = (1, 5, 1)
To fix it, the time dimension should be the same.
X.shape = (1, 15, 1)
Y.shape = (1, 15, 1)
Here is the updated function that will let the model train. Note that all I did was update the array sizes so they are equally sized. The value of 15 is used because the original array length is 30.
def create_target(batch):
X = tf.cast(batch[:,:-15][:,:,None], tf.float32)
Y = batch[:,-15:][:,:,None]
return X,Y

Train autoencoder without looping

I have some data that is of shape 10000 x 1440 x 8 where 10000 is the number of days, 1440 the number of minutes and 8 is the number of features.
For each day, ie. each submatrix of size 1440 x 8 I wish to train an autoencoder and extract the weights from the second layer, such that my output will be a matrix output = 10000 x 8
I can do this in a loop with
import numpy as np
from keras.layers import Input, Dense
from keras import regularizers, models, optimizers
data = np.random.random(size=(10000,1440,8))
def AE(y, epochs=100,learning_rate = 1e-4, regularization = 5e-4, epochs=3):
input = Input(shape=(y.shape[1],))
encoded = Dense(1, activation='relu',
kernel_regularizer=regularizers.l2(regularization))(input)
decoded = Dense(y.shape[1], activation='relu',
kernel_regularizer=regularizers.l2(regularization))(encoded)
autoencoder = models.Model(input, decoded)
autoencoder.compile(optimizer=optimizers.Adam(lr=learning_rate), loss='mean_squared_error')
autoencoder.fit(y, y, epochs=epochs, batch_size=10, shuffle=False)
(w1,b1,w2,b2)=autoencoder.get_weights()
return (w1,b1,w2,b2)
lst = []
for i in range(data.shape[0]):
y = data[i]
(_, _, w2, _) = AE(y)
lst.append(w2[0])
output = np.array(lst)
However, this feels very stupid as surely I must be able to just pass the 3D data to the autoencoder and retrieve what I want. However, if I try modify the shape of input to be input = Input(shape=(y.shape[1],y.shape[2]))
I get an error
ValueError: Dimensions must be equal, but are 1440 and 8 for '{{node
mean_squared_error/SquaredDifference}} =
SquaredDifference[T=DT_FLOAT](model_778/dense_1558/Relu,
IteratorGetNext:1)' with input shapes: [?,1440,1440], [?,1440,8].
Any pointers on how to get the shape right?

Simply reshape your your data like so and call the function.
data = data.reshape(data.shape[0]*data.shape[1], -1)
(w1, b1, w2, b2) = AE(data)
print(w2.shape)
Your first layer of the NN is a Dense layer. You can only pass two dimensional data into it. One dimension will be batch size and the other dimension will be the feature vector. When you are using the data in the way you are using it, you are considering each data point independently. Which means that you can join the first two axes together and just pass it on to the NN. However, note that you would still need to modify the code so that you are not passing the entire dataset at once to the NN. You need to split the data into batches and loop over those before passing it on. And honestly, it's the same as what you are doing now. So your looping is not as bad as you think it is for what you are trying to do.
However, also note that you have a time series data and considering each datapoint as an independent point doesn't really make sense. You need an LSTM layer or something to learn the time series encoding.

Keras multi-output data reshape for LSTM model

I have a Keras LSTM model that contains multiple outputs.
The model is defined as follows:
outputs=[]
main_input = Input(shape= (seq_length,feature_cnt), name='main_input')
lstm = LSTM(32,return_sequences=True)(main_input)
for _ in range((output_branches)): #output_branches is the number of output branches of the model
prediction = LSTM(8,return_sequences=False)(lstm)
out = Dense(1)(prediction)
outputs.append(out)
model = Model(inputs=main_input, outputs=outputs)
model.compile(optimizer='rmsprop',loss='mse')
I have a problem when reshaping the output data.
The code for reshaping the output data is:
y=y.reshape((len(y),output_branches,1))
I got the following error:
ValueError: Error when checking model target: the list of Numpy arrays
that you are passing to your model is not the size the model expected.
Expected to see 5 array(s), but instead got the following list of 1
arrays: [array([[[0.29670931],
[0.16652206],
[0.25114482],
[0.36952324],
[0.09429612]],
[[0.16652206],
[0.25114482],
[0.36952324],
[0.09429612],...
How can I correctly reshape the output data?

It depends on how y is structured initially. Here I assume that y is a single-valued label for each sequence in batch.
When there are multiple inputs/outputs model.fit() expects a corresponding list of inputs/outputs to be given. np.split(y, output_branches, axis=-1) in a following fully reproducible example does exactly this - for each batch splits a single list of outputs into a list of separate outputs where each output (in this case) is 1-element list:
import tensorflow as tf
import numpy as np
tf.enable_eager_execution()
batch_size = 100
seq_length = 10
feature_cnt = 5
output_branches = 3
# Say we've got:
# - 100-element batch
# - of 10-element sequences
# - where each element of a sequence is a vector describing 5 features.
X = np.random.random_sample([batch_size, seq_length, feature_cnt])
# Every sequence of a batch is labelled with `output_branches` labels.
y = np.random.random_sample([batch_size, output_branches])
# Here y.shape() == (100, 3)
# Here we split the last axis of y (output_branches) into `output_branches` separate lists.
y = np.split(y, output_branches, axis=-1)
# Here y is not a numpy matrix anymore, but a list of matrices.
# E.g. y[0].shape() == (100, 1); y[1].shape() == (100, 1) etc...
outputs = []
main_input = tf.keras.layers.Input(shape=(seq_length, feature_cnt), name='main_input')
lstm = tf.keras.layers.LSTM(32, return_sequences=True)(main_input)
for _ in range(output_branches):
prediction = tf.keras.layers.LSTM(8, return_sequences=False)(lstm)
out = tf.keras.layers.Dense(1)(prediction)
outputs.append(out)
model = tf.keras.models.Model(inputs=main_input, outputs=outputs)
model.compile(optimizer='rmsprop', loss='mse')
model.fit(X, y)
You might need to play around with axes as you didn't specify how exactly your data look like.
EDIT:
As author is looking for an answer drawing from official sources, it's mentioned here (not explicitly though, it only mentions what the Dataset should yield, hence - what kind of input structure model.fit() expects):
When calling fit with a Dataset object, it should yield either a tuple of lists like ([title_data, body_data, tags_data], [priority_targets, dept_targets]) or a tuple of dictionaries like ({'title': title_data, 'body': body_data, 'tags': tags_data}, {'priority': priority_targets, 'department': dept_targets}).

Since you have an amount of outputs equal to output_branches, your output data must be a list with the same amount of arrays.
Basically, if the output data is in the middle dimension as your reshape suggests:
y = [ y[:,i] for i in range(output_branches)]

How to shape large DataFrame for python's keras LSTM?

I nearly found what I need in the accepted answer here. But had memory issues because the test df provided was only 11 rows.
What I'm trying to predict is using LSTM to forecast 10 days ahead of a Time Series data in a regression model (not classifier!). My dataframe X has around 1500 rows and 2000 features, being of shape (1500, 2000) while the truth values y are just 1500 rows of 1 feature (that can range any value between -1 and 1).
Since LSTM needs 3D vector as an input, I'm really struggling how to reshape the data.
Again, following the example at first paragraph, it crashes for MemoryError when padding values, more specifically at df.cumulative_input_vectors.tolist().
My test (read forecast) is a dataframe of shape (10, 2000).
Due to sensitive data I can't actually share the values/example. How can I help you help me with this?
So, to enable the LSTM to learn from the 1500 rows of y, how should I reshape my x of 1500 rows and 2000 features? Also, how should I reshape my forecast of 10 rows and 2000 features?
They'll undergo -at first because I'm learning LSTM- a simple LSTM model of:
model = Sequential()
model.add(LSTM(50, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(train_X, train_y , epochs=50, batch_size=2, verbose=1)
what I've tried, but when predictin got error:
# A function to make a 3d data of what I understood needed done:
def preprocess_data(stock, seq_len):
amount_of_features = len(stock.columns)
data = stock.values
sequence_length = seq_len #+ 1
result = []
for index in range(len(data) - sequence_length):
result.append(data[index : index + sequence_length])
X_train = np.array(result)
X_train = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], amount_of_features))
return X_train
# creating the train as:
# X == the DF of 1500 rows and 2000 features
window = 10
train_X = preprocess_data(X[::-1], window)

After a while I've managed to understand properly what the dimensions where. Keras expect a 3d array of .shape (totalRows, sequences, totalColumns). The sequences one was the most confusing to me.
That was because when reshaping the df df.reshape(len(df), 1, len(df.columns)) meaning keras would learn for a matrix of 1 line it gave me bad results because I didn't know it's best to scale the data for me MinMaxScaler(-1,1) worked best, but could be (0,1).
What made me understand that was to first use sequence of more than 1 row (or days, since my dataset was a Time Series). Meaning that instead of feeding 1 row of features X results in 1 value of y, I used something like 5 rows of features X results in 1 value of y. as in:
# after scaling the df, resulted in "scaled_dataset"
sequences = 5
result = []
# for loop will walk for each of the 1500 rows
for i in range(0,len(scaled_dataset)):
# every group must have the same length, so if current loop position i + number
# of sequences is higher than df length, breaks
if i+sequences <= len(scaled_dataset):
# this will add into the list as [[R1a,R1b...R1t],[R2a,R2b...R2t],...[R5a,R5b...R5t]]
result.append(scaled_dataset[i:i+sequences].values)
# Converting to array + keras takes float32 better than 64
train_x = np.array(result).astype('float32')
# making the y into same length as X
train_y = np.array(y.tail(train_x.shape[0]).values)
train_x.shape, train_y.shape
'>>> (1495, 5, 2400), (1495,)
Writen in another way the mentality on keras shapes for my problem:
Considering it's a time series, above means that 5 days (rows 0 to 4) of data results in the value y of row 5.
Then, less the first day + the next day after last - still 5 days - (rows 1 to 5) of data results in the value y of row 6.
Then, less the second day + the next day after last - still 5 days - (rows 2 to 6) of data results in the value y of row 7.
It's quite confusing for starters of keras/LSTM, but I hope I could details this for people who might land here.