Develop Reference

How to shape the input of a RNN with multiple features for each target?

I'm trying to learn how to use RNN for time-series predictions and in all the examples I'm seeing out there they use a sequence of prices to predict the following price. In the examples each target (Y_train[n]) is associated to a sequence or matrix composed of the last 30 prices/steps ([X_train[[n-1],[n-2]....,[n-30]).
However in the real world to accurately predict you need more than the sequence of the last 30 prices, you would also need other... should I say features? Like the last 30 values of volume or the last 30 values of a sentiment index.
So my question is:
How do you shape the input of an RNN with two sequences for each target (last 30 prices and last 30 volume values)? This is the example code I'm using with only 1 sequence to use as reference:
import pandas as pd
import numpy as np
from sklearn.preprocessing import MinMaxScaler
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from keras.layers import Dropout
# Dividing Dataset (Test and Train)
train_lim = int(len(df) * 2 / 3)
training_set = df[:train_lim][['Close']]
test_set = df[train_lim:][['Close']]
# Normalizing
sc = MinMaxScaler(feature_range=(0, 1))
training_set_scaled = sc.fit_transform(training_set)
# Shaping Input
X_train = []
y_train = []
X_test = []
for i in range(30, training_set_scaled.size):
X_train.append(training_set_scaled[i - 30:i, 0])
y_train.append(training_set_scaled[i, 0])
X_train, y_train = np.array(X_train), np.array(y_train)
for i in range(30, len(test_set)):
X_test.append(test_set.iloc[i - 30:i, 0])
X_test = np.array(X_test)
# Adding extra dimension ???
X_train = np.reshape(X_train, [X_train.shape[0], X_train.shape[1], 1])
X_test = np.reshape(X_test, [X_test.shape[0], X_test.shape[1], 1])
regressor = Sequential()
# LSTM layer 1
regressor.add(LSTM(units=50, return_sequences=True, input_shape=(X_train.shape[1], 1)))
regressor.add(Dropout(0.2))
# LSTM layer 2,3,4
regressor.add(LSTM(units=50, return_sequences=True))
regressor.add(Dropout(0.2))
regressor.add(LSTM(units=50, return_sequences=True))
regressor.add(Dropout(0.2))
regressor.add(LSTM(units=50, return_sequences=True))
regressor.add(Dropout(0.2))
regressor.add(LSTM(units=50, return_sequences=True))
regressor.add(Dropout(0.2))
# LSTM layer 5
regressor.add(LSTM(units=50))
regressor.add(Dropout(0.2))
# Fully connected layer
regressor.add(Dense(units=1))
# Compiling the RNN
regressor.compile(optimizer='adam', loss='mean_squared_error')
# Fitting the RNN model
regressor.fit(X_train, y_train, epochs=120, batch_size=32)
The dataframe that I'm using is a standard OHLCV with a datetime index so it will look like this:
Datetime Open High Low Close Volume
01/01/2021 102.42 103.33 100.57 101.23 1990
02/01/2021 101.23 105.22 99.45 100.11 1970
... ... ... ... ... ...
01/12/2021 203.22 210.34 199.22 201.11 2600

You can follow exactly the same process, the only difference is that the length of the last dimension of the arrays with the input sequences (X_train and X_test) will be greater than one (as it will be equal to the number of external regressors plus one, where the plus one comes from the fact that the past values of the target are also used as an input).
import pandas as pd
import numpy as np
import yfinance as yf
from sklearn.preprocessing import MinMaxScaler
from keras.models import Sequential
from keras.layers import Dense, LSTM, Dropout
pd.options.mode.chained_assignment = None
# define the target and features
target = ['Close']
features = ['Volume', 'High', 'Low']
# download the data
df = yf.download(tickers=['AAPL'], period='1y')
df = df[features + target]
# split the data
split = int(df.shape[0] * 2 / 3)
df_train = df.iloc[:split, :].copy()
df_test = df.iloc[split:, :].copy()
# scale the data
target_scaler = MinMaxScaler().fit(df_train[target])
df_train[target] = target_scaler.transform(df_train[target])
df_test[target] = target_scaler.transform(df_test[target])
features_scaler = MinMaxScaler().fit(df_train[features])
df_train[features] = features_scaler.transform(df_train[features])
df_test[features] = features_scaler.transform(df_test[features])
# extract the input sequences and output values
sequence_length = 30
X_train, y_train = [], []
for i in range(sequence_length, df_train.shape[0]):
X_train.append(df_train[features + target].iloc[i - sequence_length: i])
y_train.append(df_train[target].iloc[i])
X_train, y_train = np.array(X_train), np.array(y_train)
X_test, y_test = [], []
for i in range(sequence_length, df_test.shape[0]):
X_test.append(df_test[features + target].iloc[i - sequence_length: i])
y_test.append(df_test[target].iloc[i])
X_test, y_test = np.array(X_test), np.array(y_test)
print(X_train.shape)
# (138, 30, 4)
print(X_test.shape)
# (55, 30, 4)
# build and train the model
model = Sequential()
model.add(LSTM(units=50, return_sequences=True, input_shape=X_train.shape[1:]))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50))
model.add(Dropout(0.2))
model.add(Dense(units=1))
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(X_train, y_train, epochs=120, batch_size=32)
model.evaluate(X_test, y_test)
# generate the test set predictions
y_pred = model.predict(X_test)
y_pred = target_scaler.inverse_transform(y_pred)
# plot the test set predictions
df['Predicted Close'] = np.nan
df['Predicted Close'].iloc[- y_pred.shape[0]:] = y_pred.flatten()
df[['Close', 'Predicted Close']].plot()

LSTM integration of user-defined loss function

I would like to integrate a custom loss function for my LSTM in python. The code shows my approach so far.
How would I best implement the loss function shown in the images? How would I handle the constraint <0?
Thanks for any help!
Code
# Importing the libraries
ep=25 #Epochs
bs=32 #Batch-Size
vs=0.2 #Validation-Split
r=ep+1 #Range
# Importing the training set
dataset_train = pd.read_csv(r'C:\Users\Name\Desktop\Recurrent Neural Networks\JPM_train.csv',delimiter =';')
training_set = dataset_train.iloc[:, 1:2].values
# Feature Scaling
from sklearn.preprocessing import MinMaxScaler
sc = MinMaxScaler(feature_range = (0, 1))
training_set_scaled = sc.fit_transform(training_set)
# Creating a data structure with 60 timesteps and 1 output
X_train = []
y_train = []
X_val=[]
y_val=[]
for i in range(60, 1516):
X_train.append(training_set_scaled[i-60:i, 0])
y_train.append(training_set_scaled[i, 0])
X_train, y_train, X_val, y_val = np.array(X_train), np.array(y_train), np.array(X_val), np.array(y_val)
# Reshaping
X_train = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], 1))
# Importing the Keras libraries and packages
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from keras.layers import Dropout
def custom_loss(y_true, y_pred):
if(#HERE):
loss=(predicted_stock_price-real_stock_price)^2
else:
loss=0
return loss
# Initialising the RNN
model = Sequential()
# Adding the first LSTM layer and some Dropout regularisation
model.add(LSTM(units = 50, return_sequences = True, input_shape = (X_train.shape[1], 1)))
model.add(Dropout(0.2))
# Adding a second LSTM layer and some Dropout regularisation
model.add(LSTM(units = 50, return_sequences = True))
model.add(Dropout(0.2))
# Adding a third LSTM layer and some Dropout regularisation
model.add(LSTM(units = 50, return_sequences = True))
model.add(Dropout(0.2))
# Adding a fourth LSTM layer and some Dropout regularisation
model.add(LSTM(units = 50))
model.add(Dropout(0.2))
# Adding the output layer
model.add(Dense(units = 1))
# Compiling the RNN
model.compile(optimizer = 'adam', loss = custom_loss ,metrics=['accuracy'])
# Fitting the RNN to the Training set
history=model.fit(X_train, y_train, epochs = ep, batch_size = bs, validation_split=vs)
# Getting the real stock price of 2017
dataset_test = pd.read_csv(r'C:\Users\Name\Desktop\Recurrent Neural Networks\JPM_test.csv',delimiter =';')
real_stock_price = dataset_test.iloc[:, 1:2].values
dataset_total = pd.concat((dataset_train['Preis'], dataset_test['Preis']), axis = 0)
inputs = dataset_total[len(dataset_total) - len(dataset_test) - 60:].values
inputs = inputs.reshape(-1,1)
inputs = sc.transform(inputs)
X_test = []
for i in range(60, 80):
X_test.append(inputs[i-60:i, 0])
X_test = np.array(X_test)
X_test = np.reshape(X_test, (X_test.shape[0], X_test.shape[1], 1))
predicted_stock_price = model.predict(X_test)
predicted_stock_price = sc.inverse_transform(predicted_stock_price)
history_dict = history.history
print(history_dict.keys())
accuracy = history_dict['accuracy']
validation_accuracy = history_dict['val_accuracy']
loss = history_dict['loss']
validation_loss = history_dict['val_loss']
gs = gridspec.GridSpec(2, 2)
#plt.tight_layout()
#plt.subplots_adjust(hspace=1.0)
fig = plt.figure(figsize=(16,16))
# Visualising the results
ax = plt.subplot(gs[1, :]) # row 1, span all columns
plt.plot(real_stock_price, color = 'red', label = 'Real Google Stock Price')
plt.plot(predicted_stock_price, color = 'blue', label = 'Predicted Google Stock Price')
plt.title('Google Stock Price Prediction')
plt.xlabel('Time')
plt.ylabel('Google Stock Price')
plt.legend()
plt.show()
Only the Custom loss function
def custom_loss(y_true, y_pred):
if(#HERE):
loss=(predicted_stock_price-real_stock_price)^2
else:
loss=0
return loss
Pictures of the targeted loss function
Here is the link to the original text:
https://www.researchgate.net/publication/342094242_Deep_Stock_Predictions

You can use this loss function that calculates the current prediction (t1) minus the previous real_stock_price (t-1) :
def custom_loss(y_true, y_pred):
if((y_true[0]-y_true[1])*(y_pred-y_true[1])):
loss=(y_pred -y_true[0] )^2
else:
loss=0
return loss
I think that the derivatives in the backpropagation will not be affected by this shifting of time.

Keras fit() on ImageDataGenerator() with featurewise_center gives poor validation accuracy

I have a question about using fit() on ImageDataGenerator.
I run MNIST testing successfully with Dense layers , in batches.
Following code works perfectly( Validation Accuracy 98.5%).
load
(X_train, y_train), (X_test, y_test) = mnist.load_data()
# separate data into train and validation
from sklearn.model_selection import train_test_split
# Split the data
valid_per = 0.15
X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=valid_per, shuffle= True)
N1 = X_train.shape[0] # training size
N2 = X_test.shape[0] # test size
N3 = X_valid.shape[0] # valid size
h = X_train.shape[1]
w = X_train.shape[2]
num_pixels = h*w
# reshape N1 samples to num_pixels
#x_train = X_train.reshape(N1, num_pixels).astype('float32') # shape is now (51000,784)
#x_test = X_test.reshape(N2, num_pixels).astype('float32') # shape is now (9000,784)
y_train = np_utils.to_categorical(y_train) #(51000,10): 10000 lables for 10 classes
y_valid = np_utils.to_categorical(y_valid) #(9000,10): 9000 labels for 10 classes
y_test = np_utils.to_categorical(y_test) # (10000,10): 10000 lables for 10 classes
num_classes = y_test.shape[1]
def baseline_model():
# create model
model = Sequential()
# flatten input to (N1,w*h) as fit_generator expects (N1,w*h), but dont' have x,y as inputs(so cant reshape)
model.add(Flatten(input_shape=(h,w,1)))
model.add(Dense(num_pixels, input_dim=num_pixels, kernel_initializer='normal', activation='relu'))
# Define output layer with softmax function
model.add(Dense(num_classes, kernel_initializer='normal', activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
return model
model = baseline_model()
model.summary()
batch_size = 200
epochs = 20
steps_per_epoch_tr = int(N1/ batch_size) # 51000/200
steps_per_epoch_val = int(N3/batch_size)
# reshape to be [samples][width][height][ channel] for ImageData Gnerator->datagen.flow
x_t = X_train.reshape(N1, w, h, 1).astype('float32')
x_v = X_valid.reshape(N3, w, h, 1).astype('float32')
# define data preparation
#datagen = ImageDataGenerator(rescale=1./255,featurewise_center= True,featurewise_std_normalization=True,width_shift_range=0.1,height_shift_range=0.1) # scales x_t
datagen = ImageDataGenerator(rescale=1./255,width_shift_range=0.1,height_shift_range=0.1) # scales x_t
#datagen.fit(x_t)
#datagen.fit(x_v)
train_gen = datagen.flow(x_t, y_train, batch_size=batch_size)
valid_gen = datagen.flow(x_v,y_valid, batch_size=batch_size)
model.fit_generator(train_gen,steps_per_epoch = steps_per_epoch_tr,validation_data = valid_gen,
validation_steps = steps_per_epoch_val,epochs=epochs)
now, if i comment out line 53, and un-comment line 52, 54 and 55, I get validation accuracy of 1%.
so, this gives poor accuracy:
datagen = ImageDataGenerator(rescale=1./255,featurewise_center= True,featurewise_std_normalization=True,width_shift_range=0.1,height_shift_range=0.1) # scales x_t
##datagen = ImageDataGenerator(rescale=1./255,width_shift_range=0.1,height_shift_range=0.1) # scales x_t
datagen.fit(x_t)
datagen.fit(x_v)
If I un-comment line 52, but keep lines 54,55 commented out, accuracy is again 98.5%,
datagen = ImageDataGenerator(rescale=1./255,featurewise_center= True,featurewise_std_normalization=True,width_shift_range=0.1,height_shift_range=0.1) # scales x_t
##datagen = ImageDataGenerator(rescale=1./255,width_shift_range=0.1,height_shift_range=0.1) # scales x_t
#datagen.fit(x_t)
#datagen.fit(x_v)
but as per Keras documentation, we need lines 54 and 55 if we use featurewise_center.
So, I am confused what is going wrong.

You've used both rescaling and feature normalization which is the cause of the problem. Don't use rescaling when doing feature_normalization. This causes all the input values to the network to be negative. Remove, 'rescale=1./255' from ImageDataGenerator.
datagen = ImageDataGenerator(featurewise_center= True,featurewise_std_normalization=True,width_shift_range=0.1,height_shift_range=0.1) # scales x_t
datagen.fit(x_t)
Also, use separate ImageDataGenerators for train and validation since data augmentation is usually done only for training data. And, the mean/std is calculated on the training data and applied on the validation/test data.
Like this:
x_v = (x_v - datagen.mean)/(datagen.std + 1e-6)
datagen_valid = ImageDataGenerator(...)
valid_gen = datagen_valid.flow(x_v, y_valid, batch_size=batch_size)

RNN model is not learning. The predicted and actual values as attached in the graph is a flat line

I have rescaled, normalized, initialized, dropout the time series forecasting RNN model, yet it is not learning. The predicted and actual values as shown in the graph is flat line at zero.I cannot figure out what the problem is with my model. Your input will be greatly appreicated.Here is the entire code
[Predicted and actual value graph[\]\[1\]][1]
# split into train and test sets
train_size = int(len(dataset) * 0.80)
test_size = len(dataset) - train_size
train, test = dataset[0:train_size,:]
dataset[train_size:len(dataset),:]
scaler = MinMaxScaler(feature_range=(-1, 1))
scaler = scaler.fit(train)
train_scaled = scaler.transform(train)
test_scaled = scaler.transform(test)
# convert an array of values into a dataset matrix
def create_dataset(dataset, look_back=1):
dataX, dataY = [], []
for i in range(len(dataset)-look_back-1):
a = dataset[i:(i+look_back), 0]
dataX.append(a)
dataY.append(dataset[i + look_back, 0])
return np.array(dataX), np.array(dataY)
# reshape into X=t and Y=t+1
look_back = 40
train_X, train_y = create_dataset(train_scaled, look_back)
test_X, test_y = create_dataset(test_scaled, look_back)
# reshape input to be [samples, time steps, features]
train_X= np.expand_dims(train_X, axis=1)
test_X = np.expand_dims(test_X, axis=1)
print ('Train_X shape', train_X.shape, 'Test_X.shape', test_X.shape)
model = Sequential()
model.add(SimpleRNN(64, input_shape=(1,window_size),
kernel_initializer = 'random_normal',
bias_initializer= 'random_normal'))
#model.add(Dropout(0.2))
model.add(Dense(32, kernel_initializer = 'random_normal',
bias_initializer= 'random_normal'))
#model.add(Dropout(0.2))
model.add(Dense(1, activation = 'linear'))
optimizer = keras.optimizers.adam(lr=0.1)
model.compile(loss='mse', optimizer=optimizer, metrics=['accuracy'])
#fit network history =model.fit(train_X, train_y, epochs=2,
batch_size=32,
validation_data=(test_X, test_y), verbose=1, shuffle=False)
def predict_and_score(model, X, Y): # Make predictions on the
original
scale of the data. pred =
scaler.inverse_transform(model.predict(X))
# Prepare Y data to also be on the original scale for
interpretability.
orig_data = scaler.inverse_transform([Y]) # Calculate RMSE. score
= math.sqrt(mean_squared_error(orig_data[0], pred[:, 0]))
return(score, pred)
rmse_train, train_predict = predict_and_score(model, train_X,
train_y)
rmse_test, test_predict = predict_and_score(model, test_X,
test_y)
print("Training data score: %.2f RMSE" % rmse_train)
print("Test data score: %.2f RMSE" % rmse_test)
Training data score: 151198.09 RMSE
Test data score: 125302.35 RMSE
[enter image description here][1]

One obvious problem is with your data: Its range is absolutely ridiculous. While NNs can learn 'anything' in theory, I'd say that it's practically impossible to learn to model output which: (1) is zero most of the time and (2) then ranges up to 1e14.
While the (1) may be intrinsic to your domain, (2) seems unnecessary.
Start by downscaling your data to range [0, 1], that should make it at least possible to learn something.

Evaluating Regression Neural Network model's accuracy

I am new to machine learning and created a neural network for regression output. I have ~95000 training examples and ~24000 test examples. I want to know how can I evaluate my model and get train and test errors? How to know the accuracy of this regression model? My Y variable values ranges between 100-200 and X have 9 input features in the dataset.
Here is my code:
import pandas as pd
from keras.layers import Dense, Activation,Dropout
from keras.models import Sequential
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
from matplotlib import pyplot
# Importing the dataset
# Importing the dataset
dataset = pd.read_csv('data2csv.csv')
X = dataset.iloc[:,1:10].values
y = dataset.iloc[:, :1].values
# Splitting the dataset into the Training set and Test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size = 0.2, random_state = 0)
# Feature Scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
# Initialising the ANN
model = Sequential()
# Adding the input layer and the first hidden layer
model.add(Dense(10, activation = 'relu', input_dim = 9))
# Adding the second hidden layer
model.add(Dense(units = 5, activation = 'sigmoid'))
model.add(Dropout(0.2))
# Adding the third hidden layer
model.add(Dense(units = 5, activation = 'relu'))
model.add(Dropout(0.2))
model.add(Dense(units = 5, activation = 'relu'))
model.add(Dense(units = 5, activation = 'relu'))
# Adding the output layer
model.add(Dense(units = 1))
#model.add(Dense(1))
# Compiling the ANN
model.compile(optimizer = 'adam', loss = 'mean_squared_error',metrics=['mae','mse','mape','cosine'])
# Fitting the ANN to the Training set
history=model.fit(X_train, y_train,validation_data=(X_val, y_val) ,batch_size = 1000, epochs = 100)
test_loss = model.evaluate(X_test,y_test)
loss = history.history['loss']
acc = history.history['mean_absolute_error']
val_loss = history.history['val_loss']
val_acc = history.history['val_mean_absolute_error']
mape_loss=history.history['mean_absolute_percentage_error']
cosine_los=history.history['cosine_proximity']
pyplot.plot(history.history['mean_squared_error'])
pyplot.plot(history.history['mean_absolute_error'])
pyplot.plot(history.history['mean_absolute_percentage_error'])
pyplot.plot(history.history['cosine_proximity'])
pyplot.show()
epochs = range(1, len(loss)+1)
plt.plot(epochs, loss, 'ro', label='Training loss')
plt.legend()
plt.show()
y_pred = model.predict(X_test)
plt.plot(y_test, color = 'red', label = 'Real data')
plt.plot(y_pred, color = 'blue', label = 'Predicted data')
plt.title('Prediction')
plt.legend()
plt.show()
[]
My test loss after model.evaluate. Note that here there are 5 loss functions as shown in the code.
1) 84.69654303799824
2) 7.030169963975834
3) 84.69654303799824
4) 5.241855282313331
5) -0.9999999996023872

To evaluate your model you can use evaluate method:
test_loss = model.evaluate(X_test, y_test)
It returns the loss on the given test data computed using the same loss function you used during training (i.e. mean_squared_error).
Further, If you want to get training loss at the end of each epoch you can use History object which is returned by fit method:
history = model.fit(...)
loss = history.history['loss']
The loss is a list containing the loss values of training at the end of each epoch. If you have used validation data when training the model (i.e. model.fit(..., validation_data=(X_val, y_val)) or have used any other metric like mean_absolute_error (i.e. model.compile(..., metrics=['mae'])), you can also access their values:
acc = history.history['mae']
val_loss = history.history['val_loss']
val_acc = history.history['val_mae']
Bonus: To plot the training loss curve:
epochs = range(1, len(loss)+1)
plt.plot(epochs, loss, 'ro', label='Training loss')
plt.legend()
plt.show()

To show validation loss while training:
model.fit(X_train, y_train, batch_size = 1000, epochs = 100, validation_data = (y_train,y_test))
I don't think you can easily get accuracy by plotting, since your input is 9 dimensional, you could plot the predicted y for each feature, just turn off the lines that join the dots i.e. plt.plot(x,y,'k.') note 'k' so no line, but I'm not sure if that will be useful.