Develop Reference

Keras, shapes are incompatible

I made a keras classification model, and I have inputs with different lengths, so I'm using train_on_batch. And I'm getting a
ValueError: Shapes (1, 5) and (1, 36329, 5) are incompatible .
Each input is a set of 2D points.
X_train.shape >>> (2680,)
X_train[0].shape >>> (36329, 2)
X_train[5].shape >>> (40233, 2)
For the output shape :
y_train.shape >>> (2680, 5)
# y_train[0] >>> array([0, 0, 0, 1, 0])
The full code:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(array, classes_bi, test_size=0.33, random_state=69)
from keras.models import Sequential
from keras.layers import Dense
model = Sequential()
model.add(Dense(10000, activation='LeakyReLU'))
model.add(Dense(1000, activation='LeakyReLU'))
model.add(Dense(100, activation='LeakyReLU'))
model.add(Dense(5, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
epochs=5
for epoch in range(5):
for diag,output in zip(X_train,y_train):
diag = np.expand_dims(diag,axis=0) #add the batch size = 1
output = np.expand_dims(output,axis=0) ##add batch size = 1
#print(diag.shape) >>> (1, 36329, 2)
#print(output.shape) >>> (1, 5)
model.train_on_batch(diag,output)
Error :
ValueError Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_9248/1106303834.py in <module>
6 print(diag.shape)
7 print(output.shape)
----> 8 model.train_on_batch(diag,output)
...
ValueError: in user code:
...
ValueError: Shapes (1, 5) and (1, 36329, 5) are incompatible
I tried to expand the dimsention of output twice to get a shape of (1, 1, 5) with (1, 36329, 5) but it didnt work.

For the model you are building, the dimensions of your training data needs to be constant - it cannot vary from one training example to the other.
When you create a model with Sequential(), the input shape of your model will be defined when you do the training for the first time by calling model.fit or model.train_on_batch.
For example, if your first training batch has dimension (36329, 2), your model will assume that each of your training examples have the dimension (2, ) and this particular batch has 36329 training examples - so you need 36329 labels.
The batch size can change from batch to batch, but the dimension (2, ) needs to maintain.
This might be your case:
I think your problem is because your batch of 2D points contain thousands of examples but only 1 label per batch.
If each of the 36329 training examples in X_train[0] correspond to the same label y_train[0] >>> array([0, 0, 0, 1, 0]), all you need to do is broadcast the label so it has the same number of training examples as the input.
for epoch in range(5):
for diag, output in zip(X_train,y_train):
training_examples = diag.shape[0]
broadcast_arr = np.ones((training_examples, 1))
output = output * broadcast_arr
model.train_on_batch(diag,output)
PS: I wanted to ask a couple of questions by commenting, but I don't have enough reputation to do so, that's why I'm posting this answer as my best understanding of your question.

ValueError: Error when checking input: expected input_1 to have shape (168, 5) but got array with shape (5808, 5)

I'm trying to implement a hybrid LSTM-DNN forecaster with multiple inputs using the code from Hvass-Labs Time Series tutorial #23. Basically I want to forecast day-ahead prices (just a 24 time step into the future for now) of electricity using sequential and non-sequential data. The model I'm using is two sets of inputs feeding an LSTM (for the sequential data) and Dense for the non-sequential data, with their outputs concatenated. It looks like this:
!https://imgur.com/a/x15FfIy
Basically whenever I try to fit the model after one epoch it shows this error:
UPDATE:
ValueError: Error when checking input: expected input_1 to have shape (168, 5) but got array with shape (5808, 5)
The changes I have implemented:
# Chop off x_test_scaled into two parts:
x_test1_scaled = x_test_scaled[:,0:5] # shape is (5808, 5)
x_test2_scaled = x_test_scaled[:,5:12] # shape is (5808, 7)
validation_data = [np.expand_dims(x_test1_scaled, axis=0), np.expand_dims(x_test2_scaled, axis=0)], np.expand_dims(y_test_scaled, axis=0)
I'm confused because I have indeed assigned the generator to the generator in the model.fit_generator, and I'm not passing the x_test1_scaled which does have the shape of (5808, 5). edit:(not validation_data)
%%time
model.fit_generator(generator=generator,
epochs=10,
steps_per_epoch=30,
validation_data=validation_data,
callbacks=callbacks)
If this helps, this is my model:
# first input model
input_1 = Input(shape=((168,5)))
dense_1 = Dense(50)(input_1)
# second input model
input_2 = Input(shape=((168,7)))
lstm_1 = LSTM(units=64, return_sequences=True, input_shape=(None, 7,))(input_2)
# merge input models
merge = concatenate([dense_1, lstm_1])
output = Dense(num_y_signals, activation='sigmoid')(merge)
model = Model(inputs=[input_1, input_2], outputs=output)
# summarize layers
print(model.summary())
EDIT: Cleared this problem, replaced with error on top.
Thus far I've managed everything up to actually fitting the model.
Whenever an epoch finishes however it goes into the error:
ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays: [array([[[0.4 , 0.44444442, 0. , ..., 0.1734707 ,
0.07272629, 0.07110982],
[0.3904762 , 0.43434343, 0.04347826, ..., 0.1740398 ,
0.07282589, 0.06936309],
...
I have tried the solutions from other stackexchange posts of the same error message. They haven't been successful, but I was able to eventually isolate the problem array to that of the validation_data. I just don't know how to "reshape" it into the required 2 array.
The batch generator: I have included the two sets of inputs already. the x_batch_1 and x_batch_2
def batch_generator(batch_size, sequence_length):
"""
Generator function for creating random batches of training-data.
"""
# Infinite loop.
while True:
# Allocate a new array for the batch of input-signals.
x_shape = (batch_size, sequence_length, num_x_signals)
x_batch = np.zeros(shape=x_shape, dtype=np.float16)
# Allocate a new array for the batch of output-signals.
y_shape = (batch_size, sequence_length, num_y_signals)
y_batch = np.zeros(shape=y_shape, dtype=np.float16)
# Fill the batch with random sequences of data.
for i in range(batch_size):
# Get a random start-index.
# This points somewhere into the training-data.
idx = np.random.randint(num_train - sequence_length)
# Copy the sequences of data starting at this index.
x_batch[i] = x_train_scaled[idx:idx+sequence_length]
y_batch[i] = y_train_scaled[idx:idx+sequence_length]
x_batch_1 = x_batch[ :, :, 0:5]
x_batch_2 = x_batch[ :, :, 5:12]
yield ([x_batch_1, x_batch_2], y_batch)
batch_size = 32
sequence_length = 24 * 7
generator = batch_generator(batch_size=batch_size,
sequence_length=sequence_length)
Validation set:
validation_data = np.expand_dims(x_test_scaled, axis=0), np.expand_dims(y_test_scaled, axis=0)
And lastly the model fit:
%%time
model.fit_generator(generator=generator,
epochs=10,
steps_per_epoch=30,
validation_data=validation_data,
callbacks=callbacks)
ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays: [array([[[0.4 , 0.44444442, 0. , ..., 0.1734707 ,
0.07272629, 0.07110982],
[0.3904762 , 0.43434343, 0.04347826, ..., 0.1740398 ,
0.07282589, 0.06936309],
...
The array is the same one as the validation_data. Another thing is that the error creeps up whenever the first epoch finishes which strengthens the case for the problem being the validation_data.

It's because your model need 2 sets of input, x_batch_1, x_batch_2 in your batch_generator. While your validation_data has only one array np.expand_dims(x_test_scaled, axis=0)
You need to make validation_data looks like the batch_generator, probably [np.expand_dims(x_test1_scaled, axis=0), np.expand_dims(x_test2_scaled, axis=0)], np.expand_dims(y_test_scaled, axis=0).
In case of you still don't understand, please provide information about x_test1_scaled, like it's shape or how you load it.

How to fix ValueError (x and y should have the same length) while having multiple outputs?

I'm setting up a model which has an image input (130,130,1) and 3 outputs each containing a (10,1) vector where softmax is applied individually.
(Inspired by J. Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, and
Vinay D. Shet. Multi-digit number recognition from street view imagery using deep convolutional neural networks. CoRR, abs/1312.6082, 2013. URL
http://arxiv.org/abs/1312.6082 , sadly they didn't publish their network).
input = keras.layers.Input(shape=(130,130, 1)
l0 = keras.layers.Conv2D(32, (5, 5), padding="same")(input)
[conv-blocks etc]
l12 = keras.layers.Flatten()(l11)
l13 = keras.layers.Dense(4096, activation="relu")(l12)
l14 = keras.layers.Dense(4096, activation="relu")(l13)
output1 = keras.layers.Dense(10, activation="softmax")(l14)
output2 = keras.layers.Dense(10, activation="softmax")(l14)
output3 = keras.layers.Dense(10, activation="softmax")(l14)
model = keras.models.Model(inputs=input, outputs=[output1, output2, output3])
model.compile(loss=['categorical_crossentropy', 'categorical_crossentropy',
'categorical_crossentropy'],
loss_weights=[1., 1., 1.],
optimizer=optimizer,
metrics=['accuracy'])
train_generator = train_datagen.flow(x_train,
[[y_train[:, 0, :], y_train[:, 1, :], y_train[:, 2, :]],
batch_size=batch_size)
But then i'm getting: ValueError: x (images tensor) and y (labels) should have the same length. Found: x.shape = (1000, 130, 130, 1), y.shape = (3, 1000, 10)
But if I change it to:
[same as before]
train_generator = train_datagen.flow(x_train,
y_train,
batch_size=batch_size)
Then i'm getting: ValueError: Error when checking model target: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 3 array(s)
dimension(x_train) = (1000, 130, 130, 1)
where each single image is (130, 130, 1) and there are 1000 images
dimension(y_train) = (1000, 3, 10)
In the documentation it is stated that it should be like that;
model = Model(inputs=[main_input, auxiliary_input], outputs=
[main_output, auxiliary_output])
However, I don't know how you should be able to have the same length for outputs and inputs?

Thanks to #Djib2011. When I looked up examples in the documentation for passing it in a dictionary, I remarked that all the examples make use of model.fit() and not model.fit_generator().
So I did research and found that there is still a bug (open since 2016!) for ImageDataGenerator with single input and multiple outputs.
Sad story.
So the solution is to use model.fit() instead of model.fit_generator().

Data Structure Discrepancy in Tensorflow/TFLearn

I have two datasets, which is like:
input:
array([[[ 0.99309823],
...
[ 0. ]]])
shape : (1, 2501)
output:
array([[0, 0, 0, ..., 0, 0, 1],
...,
[0, 0, 0, ..., 0, 0, 0]])
shape : (2501, 9)
And I processed it with TFLearn; as
input_layer = tflearn.input_data(shape=[None,2501])
hidden1 = tflearn.fully_connected(input_layer,1205,activation='ReLU', regularizer='L2', weight_decay=0.001)
dropout1 = tflearn.dropout(hidden1,0.8)
hidden2 = tflearn.fully_connected(dropout1,1205,activation='ReLU', regularizer='L2', weight_decay=0.001)
dropout2 = tflearn.dropout(hidden2,0.8)
softmax = tflearn.fully_connected(dropout2,9,activation='softmax')
# Regression with SGD
sgd = tflearn.SGD(learning_rate=0.1,lr_decay=0.96, decay_step=1000)
top_k=tflearn.metrics.Top_k(3)
net = tflearn.regression(softmax,optimizer=sgd,metric=top_k,loss='categorical_crossentropy')
model = tflearn.DNN(net)
model.fit(input,output,n_epoch=10,show_metric=True, run_id='dense_model')
It works but not the way that I want. It's a DNN model. I want that when I enter 0.95, model must give me corresponding prediction for example [0,0,0,0,0,0,0,0,1]. However, when I want to enter 0.95, it says that,
ValueError: Cannot feed value of shape (1,) for Tensor 'InputData/X:0', which has shape '(?, 2501)'
When I tried to understand I realise that I need (1,2501) shaped data to predict for my wrong based model.
What i want is for every element in input, predict corresponding element in output. As you can see, in the instance dataset,
for [0.99309823], corresponding output is [0,0,0,0,0,0,0,0,1]. I want tflearn to train itself like this.
I may have wrong structured data, or model(probably dataset), I explained all the things, I need help I'm really out of my mind.

Your input data should be Nx1 (N = number of samples) dimensional to archive this transformation ([0.99309823] --> [0,0,0,0,0,0,0,0,1] ). According to your input data shape, it looks more likely including 1 sample with 2501 dimensions.
ValueError: Cannot feed value of shape (1,) for Tensor 'InputData/X:0', which has shape '(?, 2501)' This error means that tensorflow expecting you to provide a vector with shape (,2501), but you are feeding the network with a vector with shape (1,).
Example modified code with dummy data:
import numpy as np
import tflearn
#creating dummy data
input_data = np.random.rand(1, 2501)
input_data = np.transpose(input_data) # now shape is (2501,1)
output_data = np.random.randint(8, size=2501)
n_values = 9
output_data = np.eye(n_values)[output_data]
# checking the shapes
print input_data.shape #(2501,1)
print output_data.shape #(2501,9)
input_layer = tflearn.input_data(shape=[None,1]) # now network is expecting ( Nx1 )
hidden1 = tflearn.fully_connected(input_layer,1205,activation='ReLU', regularizer='L2', weight_decay=0.001)
dropout1 = tflearn.dropout(hidden1,0.8)
hidden2 = tflearn.fully_connected(dropout1,1205,activation='ReLU', regularizer='L2', weight_decay=0.001)
dropout2 = tflearn.dropout(hidden2,0.8)
softmax = tflearn.fully_connected(dropout2,9,activation='softmax')
# Regression with SGD
sgd = tflearn.SGD(learning_rate=0.1,lr_decay=0.96, decay_step=1000)
top_k=tflearn.metrics.Top_k(3)
net = tflearn.regression(softmax,optimizer=sgd,metric=top_k,loss='categorical_crossentropy')
model = tflearn.DNN(net)
model.fit(input_data, output_data, n_epoch=10,show_metric=True, run_id='dense_model')

Also my friend warned me about same thing as rcmalli. He says
reshape:
input = tf.reshape(input, (2501,1))
change
input_layer = tflearn.input_data(shape=[None,2501])
to
input_layer = tflearn.input_data(shape=[None, 1])
Variable dimension must be "None". In your wrong case, 2501 is the magnitude(or something else, I translated from another lang., but you got it) of your dataset. 1 is constant input magnitude.

How can I use categorical one-hot labels for training with Keras?

I have inputs that look like this:
[
[1, 2, 3]
[4, 5, 6]
[7, 8, 9]
...]
of shape (1, num_samples, num_features), and labels that look like this:
[
[0, 1]
[1, 0]
[1, 0]
...]
of shape (1, num_samples, 2).
However, when I try to run the following Keras code, I get this error:
ValueError: Error when checking model target: expected dense_1 to have 2 dimensions, but got array with shape (1, 8038, 2). From what I've read, this appears to stem from the fact that my labels are 2D, and not simply integers. Is this correct, and if so, how can I use one-hot labels with Keras?
Here's the code:
num_features = 463
trX = np.random(8038, num_features)
trY = # one-hot array of shape (8038, 2) as described above
def keras_builder(): #generator to build the inputs
while(1):
x = np.reshape(trX, (1,) + np.shape(trX))
y = np.reshape(trY, (1,) + np.shape(trY))
print(np.shape(x)) # (1, 8038, 463)
print(np.shape(y)) # (1, 8038, 2)
yield x, y
model = Sequential()
model.add(LSTM(100, input_dim = num_features))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit_generator(keras_builder(), samples_per_epoch = 1, nb_epoch=3, verbose = 2, nb_worker = 1)
Which promptly throws the error above:
Traceback (most recent call last):
File "file.py", line 35, in <module>
model.fit_generator(keras_builder(), samples_per_epoch = 1, nb_epoch=3, verbose = 2, nb_worker = 1)
...
ValueError: Error when checking model target: expected dense_1 to have 2 dimensions, but got array with shape (1, 8038, 2)
Thank you!

There are a lot of things that do not add up.
I assume that you are trying to solve a sequential classification task, i.e. your data is shaped as (<batch size>, <sequence length>, <feature length>).
In your batch generator you create a batch consisting of one sequence of length 8038 and 463 features per sequence element. You create a matching Y batch to compare against, consisting of one sequence with 8038 elements, each of size 2.
Your problem is that Y does not match up with the output of the last layer. Your Y is 3-dimensional while the output of your model is only 2-dimensional: Y.shape = (1, 8038, 2) does not match dense_1.shape = (1,1). This explains the error message you get.
The solution to this: you need to enable return_sequences=True in the LSTM layer to return a sequence instead of only the last element (effectively removing the time-dimension). This would give an output shape of (1, 8038, 100) at the LSTM layer. Since the Dense layer is not able to handle sequential data you need to apply it to each sequence element individually which is done by wrapping it in a TimeDistributed wrapper. This then gives your model the output shape (1, 8038, 1).
Your model should look like this:
from keras.layers.wrappers import TimeDistributed
model = Sequential()
model.add(LSTM(100, input_dim=num_features, return_sequences=True))
model.add(TimeDistributed(Dense(1, activation='sigmoid')))
This can be easily spotted when examining the summary of the model:
print(model.summary())