I am constructing an LSTM predictor with Keras. My input array is historical price data. I segment the data into window_size blocks, in order to predict prediction length blocks ahead. My data is a list of 4246 floating point numbers. I seperate my data into 4055 arrays each of length 168 in order to predict 24 units ahead.
This gives me an x_train set with dimension (4055,168). I then scale my data and try to fit the data but run into a dimension error.
df = pd.DataFrame(data)
print(f"Len of df: {len(df)}")
min_max_scaler = MinMaxScaler()
H = 24
window_size = 7*H
num_pred_blocks = len(df)-window_size-H+1
x_train = []
y_train = []
for i in range(num_pred_blocks):
x_train_block = df['C'][i:(i + window_size)]
x_train.append(x_train_block)
y_train_block = df['C'][(i + window_size):(i + window_size + H)]
y_train.append(y_train_block)
LEN = int(len(x_train)*window_size)
x_train = min_max_scaler.fit_transform(x_train)
batch_size = 1
def build_model():
model = Sequential()
model.add(LSTM(input_shape=(window_size,batch_size),
return_sequences=True,
units=num_pred_blocks))
model.add(TimeDistributed(Dense(H)))
model.add(Activation("linear"))
model.compile(loss="mse", optimizer="rmsprop")
return model
num_epochs = epochs
model= build_model()
model.fit(x_train, y_train, batch_size = batch_size, epochs = 50)
The error being returned is as such.
ValueError: Error when checking model target: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 array(s), but instead got the following list of 4055 arrays: [array([[0.00630006],
Am I not segmenting correctly? Loading correctly? Should the number of units be different than the number of prediction blocks? I appreciate any help. Thanks.
Edit
The suggestions to convert them to Numpy arrays is correct but MinMixScalar() returns a numpy array. I reshaped the arrays into the proper dimension but now my computer is having CUDA memory error. I consider the problem solved. Thank you.
df = pd.DataFrame(data)
min_max_scaler = MinMaxScaler()
H = prediction_length
window_size = 7*H
num_pred_blocks = len(df)-window_size-H+1
x_train = []
y_train = []
for i in range(num_pred_blocks):
x_train_block = df['C'][i:(i + window_size)].values
x_train.append(x_train_block)
y_train_block = df['C'][(i + window_size):(i + window_size + H)].values
y_train.append(y_train_block)
x_train = min_max_scaler.fit_transform(x_train)
y_train = min_max_scaler.fit_transform(y_train)
x_train = np.reshape(x_train, (len(x_train), 1, window_size))
y_train = np.reshape(y_train, (len(y_train), 1, H))
batch_size = 1
def build_model():
model = Sequential()
model.add(LSTM(batch_input_shape=(batch_size, 1, window_size),
return_sequences=True,
units=100))
model.add(TimeDistributed(Dense(H)))
model.add(Activation("linear"))
model.compile(loss="mse", optimizer="rmsprop")
return model
num_epochs = epochs
model = build_model()
model.fit(x_train, y_train, batch_size = batch_size, epochs = 50)
I don't think you passed the batch size in the model.
input_shape=(window_size,batch_size) is the data dimension. which is correct, but you should use input_shape=(window_size, 1)
If you want to use batch, you have to add another dimension, like this LSTM(n_neurons, batch_input_shape=(n_batch, X.shape[1], X.shape[2])) (Cited from the Keras)
in your case:
def build_model():
model = Sequential()
model.add(LSTM(input_shape=(batch_size, 1, window_size),
return_sequences=True,
units=num_pred_blocks))
model.add(TimeDistributed(Dense(H)))
model.add(Activation("linear"))
model.compile(loss="mse", optimizer="rmsprop")
return model
You also need to use np.shape to change the dimension of the of your data, it should be (batch_dim, data_dim_1, data_dim_2). I use numpy, so numpy.reshape() will work.
First your data should be row-wise, so for each row, you should have a shape of (1, 168), then add the batch dimension, it will be (batch_n, 1, 168).
Hope this help.
That's probably because x_train and y_train were not updated to numpy arrays. Take a closer look at this issue on github.
model = build_model()
x_train, y_train = np.array(x_train), np.array(y_train)
model.fit(x_train, y_train, batch_size = batch_size, epochs = 50)
Related
I am attempting to do a sandbox project using the wine reviews dataset and want to combine both text data as well as some engineered numeric features into the neural network, but I am receiving a value error.
The three sets of features I have are the description (the actual reviews), scaled price, and scaled number of words (length of description). The y target variable I converted into a dichotomous variable representing good or bad reviews turning it into a classification problem.
Whether or not these are the best features to use is not the point, but I am hoping to try to combine NLP with meta or numeric data. When I run the code with just the description it works fine, but adding the additional variables is causing a value error.
y = df['y']
X = df.drop('y', axis=1)
# split up the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)
X_train.head();
description_train = X_train['description']
description_test = X_test['description']
#subsetting the numeric variables
numeric_train = X_train[['scaled_price','scaled_num_words']].to_numpy()
numeric_test = X_test[['scaled_price','scaled_num_words']].to_numpy()
MAX_VOCAB_SIZE = 60000
tokenizer = Tokenizer(num_words=MAX_VOCAB_SIZE)
tokenizer.fit_on_texts(description_train)
sequences_train = tokenizer.texts_to_sequences(description_train)
sequences_test = tokenizer.texts_to_sequences(description_test)
word2idx = tokenizer.word_index
V = len(word2idx)
print('Found %s unique tokens.' % V)
Found 31598 unique tokens.
nlp_train = pad_sequences(sequences_train)
print('Shape of data train tensor:', nlp_train.shape)
Shape of data train tensor: (91944, 136)
# get sequence length
T = nlp_train.shape[1]
nlp_test = pad_sequences(sequences_test, maxlen=T)
print('Shape of data test tensor:', nlp_test.shape)
Shape of data test tensor: (45286, 136)
data_train = np.concatenate((nlp_train,numeric_train), axis=1)
data_test = np.concatenate((nlp_test,numeric_test), axis=1)
# Choosing embedding dimensionality
D = 20
# Hidden state dimensionality
M = 40
nlp_input = Input(shape=(T,),name= 'nlp_input')
meta_input = Input(shape=(2,), name='meta_input')
emb = Embedding(V + 1, D)(nlp_input)
emb = Bidirectional(LSTM(64, return_sequences=True))(emb)
emb = Dropout(0.40)(emb)
emb = Bidirectional(LSTM(128))(emb)
nlp_out = Dropout(0.40)(emb)
x = tf.concat([nlp_out, meta_input], 1)
x = Dense(64, activation='swish')(x)
x = Dropout(0.40)(x)
x = Dense(1, activation='sigmoid')(x)
model = Model(inputs=[nlp_input, meta_input], outputs=[x])
#next, create a custom optimizer
optimizer1 = RMSprop(learning_rate=0.0001)
# Compile and fit
model.compile(
loss='binary_crossentropy',
optimizer='adam',
metrics=['accuracy']
)
print('Training model...')
r = model.fit(
data_train,
y_train,
epochs=5,
validation_data=(data_test, y_test))
I apologize if that was over-kill but wanted to make sure I didn't leave out any relevant clues or information that would potentially be helpful. The error I get from running the code is
ValueError: Layer model expects 2 input(s), but it received 1 input tensors. Inputs received: [<tf.Tensor 'IteratorGetNext:0' shape=(None, 138) dtype=float32>]
How do I resolve that error?
Thank you for posting all your code. These two lines are the problem:
data_train = np.concatenate((nlp_train,numeric_train), axis=1)
data_test = np.concatenate((nlp_test,numeric_test), axis=1)
A numpy array is interpreted as one input regardless of its shape.
Either use tf.data.Dataset and feed your dataset directly to your model:
train_dataset = tf.data.Dataset.from_tensor_slices((nlp_train, numeric_train))
labels = tf.data.Dataset.from_tensor_slices(y_train)
dataset = tf.data.Dataset.zip((train_dataset, train_dataset))
r = model.fit(dataset, epochs=5)
Or just feed your data directly to model.fit() as a list of inputs:
r = model.fit(
[nlp_train, numeric_train],
y_train,
epochs=5,
validation_data=([nlp_test, numeric_test], y_test))
I have a classification problem (0 or 1) with 78 features.
Let's say I have that data in a dataframe with 78 columns and 202 rows, each cell value holding an integer in the range [0, infinity).
Trying to achieve prediction with TensorFlow, when I fit my model I get the following warning:
WARNING:tensorflow:Model was constructed with shape (None, 101, 78) for input Tensor("input_2:0", shape=(None, 101, 78), dtype=float32), but it was called on an input with incompatible shape (101, 78).
I would have thought my shape definition was correct, so why am I getting this warning? Perhaps I'm misunderstanding how to use the framework.
X_train, X_test, y_train, y_test = train_test_split(df_x, series_y, random_state=1, test_size=0.5)
numpy_x_train = X_train.to_numpy()
numpy_y_train = y_train.to_numpy()
numpy_x_test = X_test.to_numpy()
shape_x = len(X_train)
shape_y = len(X_train.columns)
inputs = keras.Input(shape=(shape_x, shape_y))
x = Rescaling(scale=1.0 / 255)(inputs)
num_classes = 1
outputs = layers.Dense(num_classes, activation="softmax")(x)
model = keras.Model(inputs=inputs, outputs=outputs)
processed_data = model(numpy_x_train)
optimiser = keras.optimizers.RMSprop(learning_rate=1e-3)
loss = keras.losses.CategoricalCrossentropy()
model.compile(optimizer=optimiser, loss=loss)
history = model.fit(numpy_x_train, numpy_y_train, batch_size=32, epochs=10)
here a full example based on your problem. you have 2D data so in the input layer you have to specify only the feature dimension and not also the sample dimension. you also are carrying out a binary classification task so the best choice is to use a final dense layer with 1 dimension, a sigmoid activation function, and binary crossentropy as a loss. the predicted class will be 1 if the prob are > 0.5 otherwise it is 0
from tensorflow import keras
import numpy as np
# create dummy data
train_size, test_size = 101, 101
n_features = 78
num_classes = 2 # 0 or 1
numpy_x_train = np.random.uniform(0,256, (train_size,n_features))
numpy_y_train = np.random.randint(0,num_classes,train_size)
numpy_x_test = np.random.uniform(0,256, (test_size,n_features))
numpy_y_test = np.random.randint(0,num_classes,test_size)
# rescaling data
numpy_x_train = numpy_x_train / 255
numpy_x_test = numpy_x_test / 255
# define model
inputs = keras.layers.Input(shape=(numpy_x_train.shape[1],))
outputs = keras.layers.Dense(1, activation="sigmoid")(inputs)
optimiser = keras.optimizers.RMSprop(learning_rate=1e-3)
loss = keras.losses.BinaryCrossentropy()
model = keras.Model(inputs=inputs, outputs=outputs)
model.compile(optimizer=optimiser, loss=loss)
history = model.fit(numpy_x_train, numpy_y_train, batch_size=32, epochs=10)
# get test predictions
test_prob = model.predict(numpy_x_test).ravel()
test_class = (test_prob>0.5)+0 # if prob > 0.5 is 1 else is 0
I am learning how to set up the RNN-LSTM network for prediction. I have created the dataset with one input variable.
x y
1 2.5
2 6
3 8.6
4 11.2
5 13.8
6 16.4
...
By the following python code, I have created the window data, like [x(t-2), x(t-1), x(t)] to predict [y(t)]:
df= pd.read_excel('dataset.xlsx')
# split a univariate dataset into train/test sets
def split_dataset(data):
train, test = data[:-328], data[-328:-6]
return train, test
train, test = split_dataset(df.values)
# scale train and test data
def scale(train, test):
# fit scaler
scaler = MinMaxScaler(feature_range=(0,1))
scaler = scaler.fit(train)
# transform train
#train = train.reshape(train.shape[0], train.shape[1])
train_scaled = scaler.transform(train)
# transform test
#test = test.reshape(test.shape[0], test.shape[1])
test_scaled = scaler.transform(test)
return scaler, train_scaled, test_scaled
scaler, train_scaled, test_scaled = scale(train, test)
def to_supervised(train, n_input, n_out=7):
# flatten data
data = train
X, y = list(), list()
in_start = 0
# step over the entire history one time step at a time
for _ in range(len(data)):
# define the end of the input sequence
in_end = in_start + n_input
out_end = in_end + n_out
# ensure we have enough data for this instance
if out_end <= len(data):
x_input = data[in_start:in_end, 0]
x_input = x_input.reshape((len(x_input), 1))
X.append(x_input)
y.append(data[in_end:out_end, 0])
# move along one time step
in_start += 1
return np.array(X), np.array(y)
train_x, train_y = to_supervised(train_scaled, n_input = 3, n_out = 1)
test_x, test_y = to_supervised(test_scaled, n_input = 3, n_out = 1)
verbose, epochs, batch_size = 0, 20, 16
n_timesteps, n_features, n_outputs = train_x.shape[1], train_x.shape[2], train_y.shape[1]
model = Sequential()
model.add(LSTM(200, return_sequences= False, input_shape = (train_x.shape[1],train_x.shape[2])))
model.add(Dense(1))
model.compile(loss = 'mse', optimizer = 'adam')
history = model.fit(train_x, train_y, epochs=epochs, verbose=verbose, validation_data = (test_x, test_y))
However, I have other questions about this:
Q1: What is the meaning of units in LSTM? [model.add(LSTM(units, ...))]
(I have tried different units for the model, it would be more accurate as units increased.)
Q2: How many layers should I set?
Q3: How can I predict multi-steps ? e.g base on (x(t),x(t-1)) to predict y(t), y(t+1) I have tried to set the n_out = 2 in the to_supervised function, but when I applied the same method, it returned the error
train_x, train_y = to_supervised(train_scaled, n_input = 3, n_out = 2)
test_x, test_y = to_supervised(test_scaled, n_input = 3, n_out = 2)
verbose, epochs, batch_size = 0, 20, 16
n_timesteps, n_features, n_outputs = train_x.shape[1], train_x.shape[2], train_y.shape[1]
model = Sequential()
model.add(LSTM(200, return_sequences= False, input_shape = (train_x.shape[1],train_x.shape[2])))
model.add(Dense(1))
model.compile(loss = 'mse', optimizer = 'adam')
history = model.fit(train_x, train_y, epochs=epochs, verbose=verbose, validation_data = (test_x, test_y))
ValueError: Error when checking target: expected dense_27 to have shape (1,) but got array with shape (2,)
Q3(cont): What should I add or change in the model setting?
Q3(cont): What is the return_sequences ? When should I set True?
Q1. Units in LSTM is the number of neurons in your LSTM layer.
Q2. That depends on your model / data. Try changing them around to see the effect.
Q3. That depends which apporach you take.
Q4. Ideally you'll want to predict a single time step every time.
It is possible to predict several at a time, but in my experience you will get better results like as i have described below
e.g
use y(t-1), y(t) to predict y_hat(t+1)
THEN
use y(t), y_hat(t+1) to predict y_hat(t+2)
Are you sure you're actually using X to predict Y in this case?
how does train x/y and test x/y look like?
Re Q1: It is the number of LSTM cells (=LSTM units), which consist of several neurons themselves but have (in the standard case as given) only one output each. Thus, the number of units corresponds directly to the dimensionality of your output.
I am learning how to set up the RNN-LSTM network for prediction. I have created the dataset with one-variables.
x y
1 2.5
2 6
3 8.6
4 11.2
5 13.8
6 16.4
...
And the relationship of the y(t) = 2.5x(t) + x(t-1) -0.9*x(t-2). And I am trying to set up the RNN-LSTM to learn the pattern, but it occurred the error of my program. My program is like below:
df= pd.read_excel('dataset.xlsx')
def split_dataset(data):
# split into standard weeks
train, test = data[:-328], data[-328:-6]
# restructure into windows of weekly data
train = np.array(np.split(train, len(train)/1))
test = np.array(np.split(test, len(test)/1))
return train, test
verbose, epochs, batch_size = 0, 20, 16
train, test = split_dataset(df.values)
train_x, train_y = train[:,:,0], train[:,:,1]
model = Sequential()
model.add(LSTM(200, return_sequences=True, input_shape = train_x.shape))
model.compile(loss='mse', optimizer='adam')
It occurred the ValueError:
ValueError: Error when checking input: expected lstm_35_input to have 3 dimensions, but got array with shape (8766, 1)
Any experienced DS or pythoner can teach me how to set up the network?
Thanks
For LSTM based RNN, the input should be of 3 dimensions (batch, time, data_point). I assume index of your x variable is its time. In this case, you have to transform your input into batches of some window, say a window of 3, then your input is:
batch # input target
0 x[0:3] y[3]
1 x[1:4] y[4]
2 x[2:5] y[5]
Note: your y's start from t=3 since you are using last 3 time steps to predict the next 4th value. If your y's are already calculated from the last three time steps as you have said, then y's should start from 0 index, i.e. at batch 0 you have y[0] as the target
UPDATE as per below comment
If you want to have multiple sequences, then you can model it as a sequence to sequence problem and will be an N to M mapping, you need five x values to predict three y's:
batch # input target
0 x[0:5] y[3:6]
1 x[1:6] y[4:7]
2 x[2:7] y[5:8]
In current, I have created the data window and it look like work for I mentioned case.
Below is my code:
df= pd.read_excel('dataset.xlsx')
# split a univariate dataset into train/test sets
def split_dataset(data):
train, test = data[:-328], data[-328:-6]
return train, test
train, test = split_dataset(df.values)
# scale train and test data to [-1, 1]
def scale(train, test):
# fit scaler
scaler = MinMaxScaler(feature_range=(0,1))
scaler = scaler.fit(train)
# transform train
#train = train.reshape(train.shape[0], train.shape[1])
train_scaled = scaler.transform(train)
# transform test
#test = test.reshape(test.shape[0], test.shape[1])
test_scaled = scaler.transform(test)
return scaler, train_scaled, test_scaled
scaler, train_scaled, test_scaled = scale(train, test)
def to_supervised(train, n_input, n_out=7):
# flatten data
data = train
X, y = list(), list()
in_start = 0
# step over the entire history one time step at a time
for _ in range(len(data)):
# define the end of the input sequence
in_end = in_start + n_input
out_end = in_end + n_out
# ensure we have enough data for this instance
if out_end <= len(data):
x_input = data[in_start:in_end, 0]
x_input = x_input.reshape((len(x_input), 1))
X.append(x_input)
y.append(data[in_end:out_end, 0])
# move along one time step
in_start += 1
return np.array(X), np.array(y)
train_x, train_y = to_supervised(train_scaled, n_input = 3, n_out = 1)
test_x, test_y = to_supervised(test_scaled, n_input = 3, n_out = 1)
verbose, epochs, batch_size = 0, 20, 16
n_timesteps, n_features, n_outputs = train_x.shape[1], train_x.shape[2], train_y.shape[1]
model = Sequential()
model.add(LSTM(200, return_sequences= False, input_shape = (train_x.shape[1],train_x.shape[2])))
model.add(Dense(1))
model.compile(loss = 'mse', optimizer = 'adam')
history = model.fit(train_x, train_y, epochs=epochs, verbose=verbose, validation_data = (test_x, test_y))
However, I have other questions about this:
Q1: What is the meaning of units in LSTM? [model.add(LSTM(units, ...))]
(I have tried different units for the model, it would be more accurate as units increased.)
Q2: How many layers should I set?
Q3: How can I predict multi-steps ? e.g base on (x(t),x(t-1)) to predict y(t), y(t+1)
I have tried to set the n_out = 2 in the to_supervised function, but when I applied the same method, it returned the error
train_x, train_y = to_supervised(train_scaled, n_input = 3, n_out = 2)
test_x, test_y = to_supervised(test_scaled, n_input = 3, n_out = 2)
verbose, epochs, batch_size = 0, 20, 16
n_timesteps, n_features, n_outputs = train_x.shape[1], train_x.shape[2], train_y.shape[1]
model = Sequential()
model.add(LSTM(200, return_sequences= False, input_shape = (train_x.shape[1],train_x.shape[2])))
model.add(Dense(1))
model.compile(loss = 'mse', optimizer = 'adam')
history = model.fit(train_x, train_y, epochs=epochs, verbose=verbose, validation_data = (test_x, test_y))
ValueError: Error when checking target: expected dense_27 to have shape (1,) but got array with shape (2,)
Q3(cont): What should I add or change in the model setting?
Q3(cont): What is the return_sequences ? When should I set True?
I know there were already some questions in this area, but I couldn't find the answer to my problem.
I have an LSTM (with tflearn) for a regression problem.
I get 3 types of errors, no matter what kind of modifications I do.
import pandas
import tflearn
import tensorflow as tf
from sklearn.cross_validation import train_test_split
csv = pandas.read_csv('something.csv', sep = ',')
X_train, X_test = train_test_split(csv.loc[:,['x1', 'x2',
'x3','x4','x5','x6',
'x7','x8','x9',
'x10']].as_matrix())
Y_train, Y_test = train_test_split(csv.loc[:,['y']].as_matrix())
#create LSTM
g = tflearn.input_data(shape=[None, 1, 10])
g = tflearn.lstm(g, 512, return_seq = True)
g = tflearn.dropout(g, 0.5)
g = tflearn.lstm(g, 512)
g = tflearn.dropout(g, 0.5)
g = tflearn.fully_connected(g, 1, activation='softmax')
g = tflearn.regression(g, optimizer='adam', loss = 'mean_square',
learning_rate=0.001)
model = tflearn.DNN(g)
model.fit(X_train, Y_train, validation_set = (Y_train, Y_test))
n_examples = Y_train.size
def mean_squared_error(y,y_):
return tf.reduce_sum(tf.pow(y_ - y, 2))/(2 * n_examples)
print()
print("\nTest prediction")
print(model.predict(X_test))
print(Y_test)
Y_pred = model.predict(X_test)
print('MSE Test: %.3f' % ( mean_squared_error(Y_test,Y_pred)) )
At the first run when starting new kernel i get
ValueError: Cannot feed value of shape (100, 10) for Tensor 'InputData/X:0', which has shape '(?, 1, 10)'
Then, at the second time
AssertionError: Input dim should be at least 3.
and it refers to the second LSTM layer. I tried to remove the second LSTM an Dropout layers, but then I get
feed_dict[net_inputs[i]] = x
IndexError: list index out of range
If you read this, have a nice day. I you answer it, thanks a lot!!!!
Ok, I solved it. I post it so maybe it helps somebody:
X_train = X_train.reshape([-1,1,10])
X_test = X_test.reshape([-1,1,10])