I'm trying to construct a (very simple) Keras model as a baseline for a project. I have a list of 3459 numpy arrays of shape (2, 6, 15) as input, and a list of target values (ints as numpy arrays with shape ()). When I try to train the model I get this error:
"ValueError: Number of samples 2 is less than samples required for specified batch_size 32 and steps 108."
The model so far is extremely simple, but I'm having no luck getting it to train:
input = Input(shape=(2, 6, 15))
x = Dense(64, activation='relu')(input)
x = Dense(64, activation='relu')(x)
output = Dense(1)(x)
model = Model(inputs=input, outputs=output)
model.compile(optimizer='adam', loss='mean_squared_error',
metrics=['accuracy'])
hist = model.fit(
X_train,
y_train,
batch_size=32,
epochs=10,
validation_data=(X_test, y_test),
steps_per_epoch=(len(X_train) // 32),
validation_steps=(len(X_test) // 32))
I'm currently loading the data from pickle files, and I suspect that the issue might be the array structure of the individual training cases. When looking at one of the arrays in the X_train it has a structure [[[...]...], [[...]...]], and I suspect the code is confusing the outer brackets as the batch container, so it's reading a batch size of 2 as input instead. Just a theory, but I don't know how to address that to check for myself.
The error is indeed due to the way your data is being generated/loaded; there are no errors if you train the model on random tensors with the specified shapes.
I'm trying to implement the sliding windows approach and use DNN for the forecasting part. The window length = 24
What I did:
I have x (input) and y (output) in the data set. I kept the "y" value as it is (single array). And on the x-value:
def generate_input(data, sequence_length=1):
x_data = []
for i in range(len(data)-sequence_length+1):
a = data[i:(i+sequence_length)]
x_data.append(a)
return np.array (x_data)
sequence_length = 24
x_train = generate_input(train, sequence_length)
#Shape of X train: (201389, 24)
#Shape of y train: (201412,)
model = Sequential()
model.add(Dense(30,input_shape= (x_train.shape[1],)))
model.add(Dense(20))
model.add(Dropout(0.2))
model.compile(loss="mse", optimizer='rmsprop')
model.summary()
model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs,
validation_split=0.1)
The error message I'm receiving:
Error when checking target: expected dropout_5 to have shape (20,) but got
array with shape (1,)
One more question, how can I use the same approach for multivariate time series? I want to use sequences as input to predict y.
I changed the slicing part to:
x_data.append(data[i:i+sequence_length])
But I received an error:
cannot copy sequence with size 24 to array axis with dimension 4
model.summary() should show you that the output layer in your model is the Dropout layer with a shape of (None, 20). That is probably not what you want. It seems that you are trying to predict a single value. Thus you need to add a Dense(1) layer after. It is also highly unusual to have dropout as an output layer.
Also, x_train and y_train should have the same shape[0].
I've got a 2D numpy matrix (from a DataFrame) of already condensed word vectors (I used a max pooling technique, am trying to compare a logres to a bi-LSTM approach), and I'm not sure how to prepare it to use it in a keras model.
I'm aware of the need of a 3D tensor for the Bi-LSTM model, and have tried googling solutions, but couldn't find a solution that worked.
This is what I have right now:
# Set model parameters
epochs = 4
batch_size = 32
input_shape = (1, 10235, 3072)
# Create the model
model = Sequential()
model.add(Bidirectional(LSTM(64, return_sequences = True, input_shape = input_shape)))
model.add(Dropout(0.5))
model.add(Dense(1, activation = 'sigmoid'))
# Try using different optimizers and different optimizer configs
model.compile('adam', 'binary_crossentropy', metrics = ['accuracy'])
# Fit the training set over the model and correct on the validation set
model.fit(inputs['X_train'], inputs['y_train'],
batch_size = batch_size,
epochs = epochs,
validation_data = [inputs['X_validation'], inputs['y_validation']])
# Get score over the test set
return model.evaluate(inputs['X_test'], inputs['y_test'])
I currently got the following error:
ValueError: Input 0 is incompatible with layer bidirectional_23: expected ndim=3, found ndim=2
The shape of my training data (inputs['X_train']) is (10235, 3072).
Thanks so much!
I've made it work with the suggestion of the reply by doing the following:
Remove return_sequence = True;
Apply the following transformations to the X sets: np.reshape(inputs[dataset], (inputs[dataset].shape[0], inputs[dataset].shape[1], 1))
Change the input shape of the LSTM layer to (10235, 3072, 1) which is the shape of X_train.
I have some hard time to get the dimensions of a LSTM network right.
So I have the following data:
train_data.shape
(25391, 3) # to be read as 25391 timesteps and 3 features
train_labels.shape
(25391, 1) # to be read as 25391 timesteps and 1 feature
So I have thought my input dimension is (1, len(train_data), train_data.shape[1]) as I plan to submit 1 batch. But I get the following error:
Error when checking target: expected lstm_10 to have 2 dimensions, but got array with shape (1, 25391, 1)
Here is the model code:
model = Sequential()
model.add(LSTM(1, # predict one feature and one timestep
batch_input_shape=(1, len(train_data), train_data.shape[1]),
activation='tanh',
return_sequences=False))
model.compile(loss = 'categorical_crossentropy', optimizer='adam', metrics = ['accuracy'])
print(model.summary())
# as 1 sample with len(train_data) time steps and train_data.shape[1] features.
model.fit(x=train_data.values.reshape(1, len(train_data), train_data.shape[1]),
y=train_labels.values.reshape(1, len(train_labels), train_labels.shape[1]),
epochs=1,
verbose=1,
validation_split=0.8,
validation_data=None,
shuffle=False)
How should the input dimensions look like?
The problem is in the target (i.e. labels) shape you provide (i.e. Error when checking target). The output of LSTM layer in your model, which is also the output of the model, has a shape of (None, 1) since you are specifying to only the final output to be returned (i.e. return_sequences=False). In order to have the output of each timestep you need to set return_sequences=True. This way the output shape of LSTM layer would be (None, num_timesteps, num_units) which is consistent with the shape of labels array you provide.
I am trying to train an LSTM recurrent neural network, for sequence classification.
My data has the following formart:
Input: [1,5,2,3,6,2, ...] -> Output: 1
Input: [2,10,4,6,12,4, ...] -> Output: 1
Input: [4,1,7,1,9,2, ...] -> Output: 2
Input: [1,3,5,9,10,20, ...] -> Output: 3
.
.
.
So basically I want to provide a sequence as an input and get an integer as an output.
Each input sequence has length = 2000 float numbers, and I have around 1485 samples for training
The output is just an integer from 1 to 10
This is what I tried to do:
# Get the training numpy 2D array for the input (1485X 2000).
# Each element is an input sequence of length 2000
# eg: [ [1,2,3...], [4,5,6...], ... ]
x_train = get_training_x()
# Get the training numpy 2D array for the outputs (1485 X 1).
# Each element is an integer output for the corresponding input from x_train
# eg: [ 1, 2, 3, ...]
y_train = get_training_y()
# Create the model
model = Sequential()
model.add(LSTM(100, input_shape=(x_train.shape)))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
print(model.summary())
model.fit(x_train, y_train, nb_epoch=3, batch_size=64)
I get the following error:
Error when checking input: expected lstm_1_input to have 3 dimensions, but got array with shape (1485, 2000)
I tried using this instead:
model.add(LSTM(100, input_shape=(1485, 1, 2000)))
But got the another error this time:
ValueError: Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=4
Can anyone explain what is my input shape? and what am I doing wrong?
Thanks
try reshaping your training data to:
x_train=x_train.reshape(x_train.shape[0], 1, x_train.shape[1])
input_shape=(None, x_train.shape[1], 1), where None is the batch size, x_train.shape[1] is the length of each sequence of features, and 1 is each feature length. (Not sure if batch size is necessary for Sequential model).
And then reshape your data into x_train = x_train.reshape(-1, x_train.shape[1], 1).
Given the format of your input and output, you can use parts of the approach taken by one of the official Keras examples. More specifically, since you are not creating a binary classifier, but rather predicting an integer, you can use one-hot encoding to encode y_train using to_categorical().
# Number of elements in each sample
num_vals = x_train.shape[1]
# Convert all samples in y_train to one-hot encoding
y_train = to_categorical(y_train)
# Get number of possible values for model inputs and outputs
num_x_tokens = np.amax(x_train) + 1
num_y_tokens = y_train.shape[1]
model = Sequential()
model.add(Embedding(num_x_tokens, 100))
model.add(LSTM(100))
model.add(Dense(num_y_tokens, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='adam',
metrics=['accuracy'])
model.fit(x_train, y_train,
batch_size=64,
epochs=3)
The num_x_tokens in the code above would be the maximum size of the element in one of your input samples (e.g. if you have two samples [1, 7, 2] and [3, 5, 4] then num_x_tokens is 7). If you use numpy you can find this with np.amax(x_train). Similarly, num_y_tokens is the number of categories you have in y_train.
After training, you can run predictions using the code below. Using np.argmax effectively reverses to_categorical in this configuration.
model_out = model.predict(x_test)
model_out = np.argmax(model_out, axis=1)
You can import to_categorical using from keras.utils import to_categorical, Embedding using from keras.layers import Embedding, and numpy using import numpy as np.
Also, you don't have to do print(model.summary()). model.summary() is enough to print out the summary.
EDIT
If it is the case that the input is of the form [[0.12, 0.31, ...], [0.22, 0.95, ...], ...] (say, generated with x_train = np.random.rand(num_samples, num_vals)) then you can use x_train = np.reshape(x_train, (num_samples, num_vals, 1)) to change the shape of the array to input it into the LSTM layer. The code to train the model in that case would be:
num_samples = x_train.shape[0]
num_vals = x_train.shape[1] # Number of elements in each sample
# Reshape for what LSTM expects
x_train = np.reshape(x_train, (num_samples, num_vals, 1))
y_train = to_categorical(y_train)
# Get number of possible values for model outputs
num_y_tokens = y_train.shape[1]
model = Sequential()
model.add(LSTM(100, input_shape=(num_vals, 1)))
model.add(Dense(num_y_tokens, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='adam',
metrics=['accuracy'])
model.fit(x_train, y_train,
batch_size=64,
epochs=3)
The num_vals is the length of each sample array in x_train. np.reshape(x_train, (num_samples, num_vals, 1)) changes each sample from [0.12, 0.31, ...] form to [[0.12], [0.31], ...] form, which is the shape that LSTM then takes (input_shape=(num_vals, 1)). The extra 1 seems strange in this case, but it is necessary to add an extra dimension to the input for the LSTM since it expects each sample to have at least two dimensions, typically called (timesteps, data_dim), or in this case (num_vals, 1).
To see how else LSTMs are used in Keras you can refer to:
Keras Sequential model guide (has several LSTM examples)
Keras examples (look for *.py files with lstm in their name)