I'm looking into examples for time series with Keras and found some good examples for cases where the target is one data point, e.e. in the below code:
import numpy as np
import pandas as pd
import datetime as dt
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, LSTM
## data
x_train = np.random.random((100, 10))
x_train = np.reshape(x_train,(100,10,1))
y_train = np.random.random((100, 1))
###training
model = Sequential()
x_shape = 10
y_shape = 1
y_units = 1
model.add(LSTM(units=50, return_sequences=True, input_shape=(x_shape,y_shape)))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50))
model.add(Dropout(0.2))
model.add(Dense(units=y_units)) #Prediction of the next closing price
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(x_train, y_train, epochs=25, batch_size=32)
## inference
model.predict(x_train).shape
(100, 1)
My question is, which should be the data shapes and NN to predict two values, at first glance what makes sense to me is to have:
x_train = np.random.random((100, 10))
x_train = np.reshape(x_train,(100,10,1))
y_train = np.random.random((100, 2))
y_train = np.reshape(y_train,(100,2) )
model = Sequential()
x_shape = 10
y_shape = 2
y_units = 2
model.add(LSTM(units=50, return_sequences=True, input_shape=(x_shape,y_shape)))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50))
model.add(Dropout(0.2))
model.add(Dense(units=y_units)) #Prediction of the next closing price
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(x_train, y_train, epochs=25, batch_size=32)
But then I get the error:
ValueError: Input 0 is incompatible with layer sequential_25: expected shape=(None, None, 2), found shape=(None, 10, 1)
What it has worked is in that same code to change:
y_shape = 1
then I get the rigth output shape, but I dont understand why the shape of y is set to one if it is a two dimentions array
Input data must be with two units also
x_train = tf.random.uniform(shape=(100, 10, 2))
Related
I'm trying to make a binary Classification by combining CNN (con1D) with GRU. my dataset dataset is like that :
X_train shape : (223461, 5)
y_train shape :(223461,)
the X_train is like that and the Y_train is a labels (0,1) like that
first I convert that train dataset :
dataset = X_train.values
dataset=dataset[1:]
dataset = dataset.astype('float32')
dataset
the same for y-train:
dataset_target = y_train.values
dataset_target=dataset_target[1:]
dataset_target = dataset_target.astype('float32')
dataset_target
now the shapes are dataset.shape =(223460, 5)
, dataset_target.shape = (223460,)
than my model structure is :
verbose, epochs, batch_size = 0, 100, 64
n_timesteps, n_features, n_outputs = dataset.shape[0], dataset.shape[1], dataset_target.shape[0]
model = Sequential()
model.add(Conv1D(filters=64, kernel_size=3, activation='relu', input_shape= (n_timesteps,n_features)))
model.add(MaxPooling1D(pool_size=2))
model.add(GRU(64))
model.add(Dropout(0.4))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.4))
model.add(Dense(1, activation='sigmoid'))
opt = Adam(learning_rate=0.01)
model.compile(loss=tf.keras.losses.BinaryCrossentropy(), optimizer=opt , metrics=['accuracy'])
model.summary()
and when I want to fit dataset to my model:
# fit network
model.fit(dataset, dataset_target, epochs=epochs, batch_size=batch_size, verbose=1)
# evaluate model
_, accuracy = model.evaluate(X_test, y_test, batch_size=batch_size, verbose=1)
#accuracy
I get an error Input 0 of layer "sequential" is incompatible with the layer: expected shape=(None, 223460, 5), found shape=(64, 5)
Is the first axis of the dataset (233460 samples) actually time steps, and do you have 5 'channels' of data? In that case, it would help if you slice the dataset along the first axis and then assign to each 'slice' single label, for example, the last value related to the slice from the y_train. In that case, n_timesteps would be the length of the slice, and the shape of the dataset something like (n_samples, n_timesteps, 5). Basically, Conv1D expects each training sample to be 2D, but in you case it's 1D, because the first dimension is just a number of samples.
I might have interpreted the dataset the wrong way. In that case, please clarify how it works so I would fix my suggestion.
Here's the example:
import tensorflow as tf
import tensorflow.keras as keras
from tensorflow.keras.layers import Conv1D, MaxPooling1D, GRU, \
Dropout, Flatten, Dense
from tensorflow.keras import Sequential
from tensorflow.keras.optimizers import Adam
import numpy as np
X_train = np.random.normal(0, 1, (223461, 5))
y_train = np.random.randint(0, 2, 223461)
dataset = X_train[1:]
dataset_target = y_train[1:]
n_timesteps = 10
# Slice dataset and target
dataset = np.stack(np.split(dataset, n_timesteps)[:-1])
dataset_target = np.stack([y[-1] for y in np.split(dataset_target, n_timesteps)[:-1]])
Define and train the model:
def get_model(dataset, n_timesteps):
verbose, epochs, batch_size = 0, 100, 64
n_timesteps, n_features = dataset.shape[1], dataset.shape[2]
model = Sequential()
model.add(Conv1D(filters=64, kernel_size=3, activation='relu', input_shape = (n_timesteps, n_features)))
model.add(MaxPooling1D(pool_size=2))
model.add(GRU(64))
model.add(Dropout(0.4))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.4))
model.add(Dense(1, activation='sigmoid'))
opt = Adam(learning_rate=0.01)
model.compile(loss=tf.keras.losses.BinaryCrossentropy(), optimizer=opt , metrics=['accuracy'])
model.summary()
return model
verbose, epochs, batch_size = 0, 1, 64
model = get_model(dataset, n_timesteps)
model.fit(dataset, dataset_target, epochs=epochs, batch_size=batch_size, verbose=1)
Hope it helps!
I'm trying to learn how to use RNN for time-series predictions and in all the examples I'm seeing out there they use a sequence of prices to predict the following price. In the examples each target (Y_train[n]) is associated to a sequence or matrix composed of the last 30 prices/steps ([X_train[[n-1],[n-2]....,[n-30]).
However in the real world to accurately predict you need more than the sequence of the last 30 prices, you would also need other... should I say features? Like the last 30 values of volume or the last 30 values of a sentiment index.
So my question is:
How do you shape the input of an RNN with two sequences for each target (last 30 prices and last 30 volume values)? This is the example code I'm using with only 1 sequence to use as reference:
import pandas as pd
import numpy as np
from sklearn.preprocessing import MinMaxScaler
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from keras.layers import Dropout
# Dividing Dataset (Test and Train)
train_lim = int(len(df) * 2 / 3)
training_set = df[:train_lim][['Close']]
test_set = df[train_lim:][['Close']]
# Normalizing
sc = MinMaxScaler(feature_range=(0, 1))
training_set_scaled = sc.fit_transform(training_set)
# Shaping Input
X_train = []
y_train = []
X_test = []
for i in range(30, training_set_scaled.size):
X_train.append(training_set_scaled[i - 30:i, 0])
y_train.append(training_set_scaled[i, 0])
X_train, y_train = np.array(X_train), np.array(y_train)
for i in range(30, len(test_set)):
X_test.append(test_set.iloc[i - 30:i, 0])
X_test = np.array(X_test)
# Adding extra dimension ???
X_train = np.reshape(X_train, [X_train.shape[0], X_train.shape[1], 1])
X_test = np.reshape(X_test, [X_test.shape[0], X_test.shape[1], 1])
regressor = Sequential()
# LSTM layer 1
regressor.add(LSTM(units=50, return_sequences=True, input_shape=(X_train.shape[1], 1)))
regressor.add(Dropout(0.2))
# LSTM layer 2,3,4
regressor.add(LSTM(units=50, return_sequences=True))
regressor.add(Dropout(0.2))
regressor.add(LSTM(units=50, return_sequences=True))
regressor.add(Dropout(0.2))
regressor.add(LSTM(units=50, return_sequences=True))
regressor.add(Dropout(0.2))
regressor.add(LSTM(units=50, return_sequences=True))
regressor.add(Dropout(0.2))
# LSTM layer 5
regressor.add(LSTM(units=50))
regressor.add(Dropout(0.2))
# Fully connected layer
regressor.add(Dense(units=1))
# Compiling the RNN
regressor.compile(optimizer='adam', loss='mean_squared_error')
# Fitting the RNN model
regressor.fit(X_train, y_train, epochs=120, batch_size=32)
The dataframe that I'm using is a standard OHLCV with a datetime index so it will look like this:
Datetime Open High Low Close Volume
01/01/2021 102.42 103.33 100.57 101.23 1990
02/01/2021 101.23 105.22 99.45 100.11 1970
... ... ... ... ... ...
01/12/2021 203.22 210.34 199.22 201.11 2600
You can follow exactly the same process, the only difference is that the length of the last dimension of the arrays with the input sequences (X_train and X_test) will be greater than one (as it will be equal to the number of external regressors plus one, where the plus one comes from the fact that the past values of the target are also used as an input).
import pandas as pd
import numpy as np
import yfinance as yf
from sklearn.preprocessing import MinMaxScaler
from keras.models import Sequential
from keras.layers import Dense, LSTM, Dropout
pd.options.mode.chained_assignment = None
# define the target and features
target = ['Close']
features = ['Volume', 'High', 'Low']
# download the data
df = yf.download(tickers=['AAPL'], period='1y')
df = df[features + target]
# split the data
split = int(df.shape[0] * 2 / 3)
df_train = df.iloc[:split, :].copy()
df_test = df.iloc[split:, :].copy()
# scale the data
target_scaler = MinMaxScaler().fit(df_train[target])
df_train[target] = target_scaler.transform(df_train[target])
df_test[target] = target_scaler.transform(df_test[target])
features_scaler = MinMaxScaler().fit(df_train[features])
df_train[features] = features_scaler.transform(df_train[features])
df_test[features] = features_scaler.transform(df_test[features])
# extract the input sequences and output values
sequence_length = 30
X_train, y_train = [], []
for i in range(sequence_length, df_train.shape[0]):
X_train.append(df_train[features + target].iloc[i - sequence_length: i])
y_train.append(df_train[target].iloc[i])
X_train, y_train = np.array(X_train), np.array(y_train)
X_test, y_test = [], []
for i in range(sequence_length, df_test.shape[0]):
X_test.append(df_test[features + target].iloc[i - sequence_length: i])
y_test.append(df_test[target].iloc[i])
X_test, y_test = np.array(X_test), np.array(y_test)
print(X_train.shape)
# (138, 30, 4)
print(X_test.shape)
# (55, 30, 4)
# build and train the model
model = Sequential()
model.add(LSTM(units=50, return_sequences=True, input_shape=X_train.shape[1:]))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50))
model.add(Dropout(0.2))
model.add(Dense(units=1))
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(X_train, y_train, epochs=120, batch_size=32)
model.evaluate(X_test, y_test)
# generate the test set predictions
y_pred = model.predict(X_test)
y_pred = target_scaler.inverse_transform(y_pred)
# plot the test set predictions
df['Predicted Close'] = np.nan
df['Predicted Close'].iloc[- y_pred.shape[0]:] = y_pred.flatten()
df[['Close', 'Predicted Close']].plot()
I'm trying to build a 1D CNN model by processing ECG signals to diagnose sleep apnea.
I am using the sklearn library and encountered an error in train_test_split. Here is my code:
# loading the file
with open("ApneaData.csv") as csvDataFile:
csvReader = csv.reader(csvDataFile)
for line in csvReader:
lis.append(line[0].split()) # create a list of lists
# making a list of all x-variables
for i in range(1, len(lis)):
data.append(list(map(int, lis[i])))
# a list of all y-variables (either 0 or 1)
target = Extract(data) # sleep apn or not
# converting to numpy arrays
data = np.array(data)
target = np.array(target)
# stacking data into 3D
loaded = dstack(data)
change = dstack(target)
trainX, testX, trainy, testy = train_test_split(loaded, change, test_size=0.3)
# the model
verbose, epochs, batch_size = 0, 10, 32
n_timesteps, n_features, n_outputs = trainX.shape[0], trainX.shape[1], trainy.shape[0]
model = Sequential()
model.add(Conv1D(filters=64, kernel_size=3, activation='relu', input_shape=(n_timesteps,n_features)))
model.add(Conv1D(filters=64, kernel_size=3, activation='relu'))
model.add(Dropout(0.5))
model.add(MaxPooling1D(pool_size=2))
model.add(Flatten())
model.add(Dense(100, activation='relu'))
model.add(Dense(n_outputs, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# fitting the model
model.fit(trainX, trainy, epochs=epochs, batch_size=batch_size, verbose=verbose)
# evaluate model
_, accuracy = model.evaluate(testX, testy, batch_size=batch_size, verbose=0)
I get the error:
ValueError: Error when checking input: expected conv1d_15_input to have shape (11627, 6001) but got array with shape (6001, 1)
I don't understand what I'm doing wrong? Any help would be much appreciated.
I think that n_timesteps and n_features should be shape[1] and shape[2], the first dimension is your number of samples
First,
# a list of all y-variables (either 0 or 1)
target = Extract(data) # sleep apn or not
This suggets you're doing a binary classification, and it seems you haven't applied one-hot-encoding. So, you last layer should be sigmoid.
the first dimension denotes number of samples. So,
trainX = tranX.reshape(trainX.shape[0], trainX.shape[1], -1)
(add a third dimension if not there already)
n_timesteps, n_features, n_outputs = trainX.shape[1], trainX.shape[2], 1
Finally, change your model.
model.add(Dense(n_outputs, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
I am experimenting with the use of multi-input to a double CNN.
However, I am getting the Graph disconnected: cannot obtain value for tensor Tensor("embedding_1_input:0", shape=(?, 40), dtype=float32) at layer "embedding_1_input". The following previous layers were accessed without issue: []
I didn't find a fix for it, knowing that there are layers with shape = (?, 50) why is there such dimension "?" ????
from keras.models import Model
from keras.layers import Input, Dot
from keras.layers.core import Reshape
from keras.backend import int_shape
max_words=50
thedata = [' '.join(list(x)) for x in self.input_data]
emb_size = 15
tokenizer = Tokenizer(num_words=max_words)
tokenizer.fit_on_texts(thedata)
self.dictionary = tokenizer.word_index
vocab_size = len(tokenizer.word_index) + 1
vocab_size = 5000
allWordIndices = []
for text in thedata:
wordIndices = self.convert_text_to_index_array(text)
allWordIndices.append(wordIndices)
allWordIndices = np.asarray(allWordIndices)
train_x = tokenizer.sequences_to_matrix(allWordIndices, mode='binary')
train_y = list(map(lambda x: self.c[x], self.output_data))
X_train, X_test, Y_train, Y_test = train_test_split(train_x, train_y, test_size = 0.1, shuffle=True)
vocab_size1 = 50000
input_size = 40
input_shape = (input_size, 1)
# first model
L_branch = Sequential()
L_branch.add(Embedding(vocab_size1, emb_size, input_length=max_words, trainable=True))
L_branch.add(Conv1D(50, activation='relu', kernel_size=10, input_shape=input_shape))
L_branch.add(MaxPooling1D(emb_size))
L_branch.add(Flatten())
L_branch.add(Dense(emb_size, activation='sigmoid'))
# second model
R_branch = Sequential()
R_branch.add(Embedding(vocab_size, output_dim=emb_size, input_length=max_words, trainable=True))
R_branch.add(Dense(emb_size, activation='sigmoid'))
input_in1 = Input(shape=(input_size, 1, 1))
input_in2 = Input(shape=(input_size, 1, 1))
merged = Dot(axes=(1, 2))([L_branch.outputs[0], R_branch.outputs[0]])
print(merged.shape)
#
merged = Reshape((-1, int_shape(merged)[1],))(merged)
latent = Conv1D(50, activation='relu', kernel_size=self.nb_classes, input_shape=(1, input_size, 1))(merged)
latent = MaxPooling1D(self.nb_classes)(latent)
out = Dense(self.nb_classes, activation='sigmoid')(latent)
final_model = Model([input_in2, input_in1], out)
final_model.compile(
loss='mse',
optimizer='adam',
metrics=['accuracy'])
final_model.summary()
final_model.fit(
[X_train, train_x],
Y_train,
batch_size=200,
epochs=5,
verbose=1
# validation_split=0.1
)
I tried adding flatten layer after the second embedding but I got an error. Even with a dropout.
I also tried eliminating the maxpooling from the L_branch model.
Tried not working with the reshape and just expand the input dims since I was getting an error of the second Conv1D saying the layer expects dims = 3 but it was getting dims = 2.
latent = Conv1D(50, activation='relu', kernel_size=nb_classes, input_shape=(1, input_size, 1))(merged)
File "/root/anaconda3/envs/oea/lib/python3.7/site-packages/keras/engine/base_layer.py", line 414, in __call__
self.assert_input_compatibility(inputs)
File "/root/anaconda3/envs/oea/lib/python3.7/site-packages/keras/engine/base_layer.py", line 311, in assert_input_compatibility
str(K.ndim(x)))
ValueError: Input 0 is incompatible with layer conv1d_2: expected ndim=3, found ndim=2`
I also didn't get which input is used in the first model and which input is used in the second?
you have to define your new model in this way: final_model = Model([L_branch.input, R_branch.input], out)
I provide an example of network with a structure similar to yours
vocab_size1 = 5000
vocab_size2 = 50000
input_size1 = 40
input_size2 = 40
max_words=50
emb_size = 15
nb_classes = 10
# first model
L_branch = Sequential()
L_branch.add(Embedding(vocab_size1, emb_size, input_length=input_size1, trainable=True))
L_branch.add(Conv1D(50, activation='relu', kernel_size=10))
L_branch.add(MaxPooling1D())
L_branch.add(Flatten())
L_branch.add(Dense(emb_size, activation='relu'))
# second model
R_branch = Sequential()
R_branch.add(Embedding(vocab_size2, emb_size, input_length=input_size2, trainable=True))
R_branch.add(Flatten())
R_branch.add(Dense(emb_size, activation='relu'))
merged = Concatenate()([L_branch.output, R_branch.output])
latent = Dense(50, activation='relu')(merged)
out = Dense(nb_classes, activation='softmax')(latent)
final_model = Model([L_branch.input, R_branch.input], out)
final_model.compile(
loss='sparse_categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
final_model.summary()
X1 = np.random.randint(0,vocab_size1, (100,input_size1))
X2 = np.random.randint(0,vocab_size2, (100,input_size2))
y = np.random.randint(0,nb_classes, 100)
final_model.fit([X1,X2], y, epochs=10)
I'm new with Keras and I'm trying to build a model for personal use/future learning. I've just started with python and I came up with this code (with help of videos and tutorials). I have a data of 16324 instances, each instance consists of 18 features and 1 dependent variable.
import pandas as pd
import os
import time
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, LSTM, BatchNormalization
from tensorflow.keras.callbacks import TensorBoard, ModelCheckpoint
EPOCHS = 10
BATCH_SIZE = 64
NAME = f"-TEST-{int(time.time())}"
df = pd.read_csv("EntryData.csv", names=['1SH5', '1SHA', '1SA5', '1SAA', '1WH5', '1WHA', '2SA5', '2SAA', '2SH5', '2SHA', '2WA5', '2WAA', '3R1', '3R2', '3R3', '3R4', '3R5', '3R6', 'Target'])
df_val = 14554
validation_df = df[df.index > df_val]
df = df[df.index <= df_val]
train_x = df.drop(columns=['Target'])
train_y = df[['Target']]
validation_x = validation_df.drop(columns=['Target'])
validation_y = validation_df[['Target']]
model = Sequential()
model.add(LSTM(128, input_shape=(train_x.shape[1:]), return_sequences=True))
model.add(Dropout(0.2))
model.add(BatchNormalization())
model.add(LSTM(128, return_sequences=True))
model.add(Dropout(0.1))
model.add(BatchNormalization())
model.add(LSTM(128))
model.add(Dropout(0.2))
model.add(BatchNormalization())
model.add(Dense(32, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(2, activation='softmax'))
opt = tf.keras.optimizers.Adam(lr=0.001, decay=1e-6)
model.compile(loss='sparse_categorical_crossentropy',
optimizer=opt,
metrics=['accuracy'])
tensorboard = TensorBoard(log_dir=f'logs/{NAME}')
filepath = "RNN_Final-{epoch:02d}-{val_acc:.3f}"
checkpoint = ModelCheckpoint("models/{}.model".format(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')) # saves only the best ones
history = model.fit(
train_x, train_y,
batch_size=BATCH_SIZE,
epochs=EPOCHS,
validation_data=(validation_x, validation_y),
callbacks=[tensorboard, checkpoint],)
score = model.evaluate(validation_x, validation_y, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
model.save("models/{}".format(NAME))
In line
model.add(LSTM(128, input_shape=(train_x.shape[1:]), return_sequences=True))
is throwing an error:
ValueError: Input 0 of layer lstm is incompatible with the layer:
expected ndim=3, found ndim=2. Full shape received: [None, 18]
I was searching for solution on this site and on google for few hours now and I was not able to find proper answer for this or I was not able to implement the solution for similar problem.
Thank you for any tips.
An LSTM network expects three dimensional input of this format:
(n_samples, time_steps, features)
There are two main ways this can be a problem.
Your input is 2D
You have stacked (multiple) LSTM layers
1. Your input is 2D
You need to turn your input to 3D.
x = x.reshape(len(x), 1, x.shape[1])
# or
x = np.expand_dims(x, 1)
Then, specify the right input shape in the first layer:
LSTM(64, input_shape=(x.shape[1:]))
2. You have stacked LSTM layers
By default, LSTM layers will not return sequences, i.e., they will return 2D output. This means that the second LSTM layer will not have the 3D input it needs. To address this, you need to set the return_sequences=True:
tf.keras.layers.LSTM(8, return_sequences=True),
tf.keras.layers.LSTM(8)
Here's how to reproduce and solve the 2D input problem:
import tensorflow as tf
import numpy as np
x = np.random.rand(100, 10)
# x = np.expand_dims(x, 1) # uncomment to solve the problem
y = np.random.randint(0, 2, 100)
model = tf.keras.Sequential([
tf.keras.layers.LSTM(8),
tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy'])
history = model.fit(x, y, validation_split=0.1)
Here's how to reproduce and solve the stacked LSTM layers problem:
import tensorflow as tf
import numpy as np
x = np.random.rand(100, 1, 10)
y = np.random.randint(0, 2, 100)
model = tf.keras.Sequential([
tf.keras.layers.LSTM(8), # use return_sequences=True to solve the problem
tf.keras.layers.LSTM(8),
tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy'])
history = model.fit(x, y, validation_split=0.1)