Develop Reference

rnn and input shape on a 1d toy dataset with keras

i was trying to implement a simple RNN on a 1 dimensional toy dataset (just one x and one target variable y) to learn but I am having issue with the input shape.
X_train has input shape equal to (848,)
y_train the same
my model is very simple
from keras.layers import Input, SimpleRNN, Dense
def create_rnn_model(train_size):
inputs = Input(shape=( 1,))
rnn = SimpleRNN(units=32, input_shape=( 1,))(inputs)
dense1 = Dense(units=64, activation='relu')(rnn)
dense2 = Dense(units=64, activation='relu')(dense1)
modello = Model(inputs=inputs, outputs=dense2)
return modello
optimizer = tf.optimizers.SGD(learning_rate=0.0001,momentum=0.9)
rnn_model = create_rnn_model(train_size=train_size)
rnn_model.compile(optimizer=optimizer,
loss="mse" , jit_compile=True ]
)
but whenever i try to fit it i get this input shape that doesn't allow me to go further
ValueError: Input 0 of layer "simple_rnn_5" is incompatible with the
layer: expected ndim=3, found ndim=2. Full shape received: (None, 1)
i know it depends on the format that the rnn want but i don't understand how to organize the data correctly.
i was trying without any windows size but i would really like to understand how to set a windows size of 10 for example.

Keras Functional API embedding layer output to LSTM

When passing the output of my embedding layer to the LSTM layer I'm running into a ValueError that I cannot figure out. My model is:
def lstm_mod(self, n_cells,batch_size):
input = tf.keras.Input((self.n_seq, self.n_features))
embedding = tf.keras.layers.Embedding(batch_size,self.n_seq,input_length=self.n_clusters)(input)
x= tf.keras.layers.LSTM(n_cells)(embedding)
out = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(input, out,name="LSTM")
model.compile(loss='mse', optimizer='Adam')
return model
The error is:
ValueError: Input 0 of layer lstm is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: [None, 128, 7, 128]
Given that the dimensions passed to the model input and the embedding layer are consistent through the arguments of the model I'm puzzled by this. Any guidance is appreciated.

Keras adds an additional dimension (None) when you feed your data through your model because it processes your data in batches.
In this line :
input = tf.keras.Input((self.n_seq, self.n_features))
You've defined a 2-dimensional input, and Keras adds a 3rd dimension (the batch), hence expected ndim=3.
However, the data that is being passed to the input layer is 4-dimensional, which means that your actual input data shape is 3-dimensional + the batch dimension, not 2-dimensional + batch.
To fix this you need to either re-shape your 3-D input to 2-D, or add an additional dimension to the input shape.
Print out the values for self.n_seq and self.n_features and find out what is missing from the shape 128, 7, 128 and that should guide you as to what you need to add.

TensorFlow CNN Incompatible Shapes: 4D input shape

I have sample data in the form: Data[n][31][31][5][2] with:
"[n]" being the sample
"[31][31]" being the array of data points
"[5]" being the number of bits within that data point
and "[2]" being one-hot encoding of the bits (eg a bit of 1 would be [1, 0] and a zero [0, 1])
The output is intended to either be a [5][2] or a [10] array of values which is validated against another [5][2] or [10] array.
When trying to build the model, I get the following error:
"ValueError: Shapes (None, 5, 2) and (None, 10) are incompatible"
The model code looks like this:
(with train_m[n][31][31][5][2], tr_m[5][2], check_m[n][31][31][5][2], cr_m[5][2] being training data and expected output followed by validation data and expected output.)
model = Sequential([
Conv2D(num_filters, filter_size, input_shape=(31, 31, 5, 2)),
Flatten(),
Dense(10, activation='relu'),
])
model.compile(
'adam',
loss='categorical_crossentropy',
metrics=['accuracy'],
)
model.summary()
model.fit(
train_m,
tr_m,
epochs=(100),
validation_data=(check_m, cr_m),
verbose=0
)
As the [5][2] outputs are one-hotted, I'm uncertain if they can be made to a [10] matrix while still being interpreted correctly. Further, would there be any way to make the dense layer to a [5][2]?
The full error can be seen here. I felt it would be awfully long to include in rawtext here.
If there's anything more that's needed, please let me know - I'm still very new to working with TensorFlow.

Your label shapes are (5,2) but network output is (10,) so this is confusing. Both output shape and label shape should be the same.
use:
tf.keras.layers.Reshape((5,2))
after the Dense layer. you'll be fine

Why do you have to reshape inputs in Keras/Tensorflow 2?

I have a simple network:
input_layer = Input(1)
inner_layer = Dense(4, activation='relu')(input_layer)
output_layer = Dense(1, activation='linear')(inner_layer)
model = Model(input_layer, output_layer)
optimizer = Adam(learning_rate=0.01)
model.compile(optimizer=optimizer, loss='mse')
Intuitively, inference for input 0 would simply be model.predict(0). However this generates this error: expected input_2 to have 2 dimensions, but got array with shape ()
I understand it expects the input (which is a single number) to be two dimensional, but I don't understand what Tensorflow accepts as valid input. I tried many different combinations of inputs, some work and some don't, it seems quite inconsistent and the warnings/error are usually not useful:
When calling model.predict():
model.predict(0) - Throws
model.predict([0]) - Works
model.predict([[0]]) - Works
When calling model() (I saw here that's needed to get the gradients):
model(0) - Throws
model([0]) - Throws
model([[0]]) - Throws
When using np.reshape:
model(np.reshape(0,[1,1])) - Works
model(np.reshape([0],[1,1])) - Works
model(np.reshape([[0]],[1,1])) - Works
What seems to be working consistently is using numpy's reshape function. it always works both for model.predict() and for model() on all the inputs as long as they're reshaped to a [1,1] shape.
My questions:
What are the guidelines to feeding inputs into tensorflow models in regards to inputs shapes/types?
What does "shape ()" mean?
What does "(None, 1)" mean?
Why does reshape work but [[0]] does not? Both create a 2-dimensional collection.
Why when calling model(0)/model([0])/model([[0]]) does this warning show: WARNING:tensorflow:Model was constructed with shape Tensor("input_1:0", shape=(None, 1), dtype=float32) for input (None, 1), but it was re-called on a Tensor with incompatible shape ()?

The shape of the tensor inputs = tf.keras.layers.Input(1) is (None, 1) (run inputs.get_shape().as_list()). The None means any size that is determined dynamically (batch size). The 1 is the shape of your data point. For example, this is a tensor of shape (3, 1):
[[1], [2], [1]]
This is a tensor of shape (3,)
[1, 2, 1]
If you define a tensor of shape (None, 1) you must feed the data of same shape.
The [[0]] has correct shape (1, 1) and won't throw any error or warning if you pass it as numpy array of expected data type:
import tensorflow as tf
import numpy as np
input_layer = tf.keras.layers.Input(1)
inner_layer = tf.keras.layers.Dense(4, activation='relu')(input_layer)
output_layer = tf.keras.layers.Dense(1, activation='linear')(inner_layer)
model = tf.keras.models.Model(input_layer, output_layer)
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)
model.compile(optimizer=optimizer, loss='mse')
print(model(np.array([[0.]], dtype=np.float32)).numpy()) # [[0.]]
print(model.predict(np.array([[0.], [1]], dtype=np.float32))) # [[0. ]
# [0.08964952]]
np.reshape() works because it automatically converts your list to numpy array. For more about np.reshape refer to the official documentation.
The model.predict() also expects the same shape as model.__call__(), but can perform automatic reshaping (expands dimension on the left, i.e [1] -- > [[1]]).

Error passing multiple inputs into Keras model

I want to train a binary classifier using Keras and my training data is of shape (2000,2,128) and labels of shape (2000,) as Numpy arrays.
The idea is to train such that embeddings together in a single array means they are either same or different, labelled using 0 or 1 respectively.
The training data looks like:
[[[0 1 2 ....128][129.....256]][[1 2 3 ...128][9 9 3 5...]].....]
and the labels looks like [1 1 0 0 1 1 0 0..].
Here is the code:
import keras
from keras.layers import Input, Dense
from keras.models import Model
frst_input = Input(shape=(128,), name='frst_input')
scnd_input = Input(shape=(128,),name='scnd_input')
x = keras.layers.concatenate([frst_input, scnd_input])
x = Dense(128, activation='relu')(x)
x=(Dense(1, activation='softmax'))(x)
model=Model(inputs=[frst_input, scnd_input], outputs=[x])
model.compile(optimizer='rmsprop', loss='binary_crossentropy',
loss_weights=[ 0.2],metrics=['accuracy'])
I am getting the following error while running this code:
ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays: [array([[[ 0.07124118, -0.02316936, -0.12737238, ..., 0.15822273,
0.00129827, -0.02457245],
[ 0.15869428, -0.0570458 , -0.10459555, ..., 0.0968155 ,
0.0183982 , -0.077924...
How can I resolve this issue? Is my code correct to train the classifier using two inputs to classify?

Well, you have two options here:
1) Reshape the training data to (2000, 128*2) and define only one input layer:
X_train = X_train.reshape(-1, 128*2)
inp = Input(shape=(128*2,))
x = Dense(128, activation='relu')(inp)
x = Dense(1, activation='sigmoid'))(x)
model=Model(inputs=[inp], outputs=[x])
2) Define two input layers, as you have already done, and pass a list of two input arrays when calling fit method:
# assuming X_train have a shape of `(2000, 2, 128)` as you suggested
model.fit([X_train[:,0], X_train[:,1]], y_train, ...)
Further, since you are doing binary classification here, you need to use sigmoid as the activation of last layer (i.e. using softmax in this case would always outputs 1, since softmax normalizes the outputs such that their sum equals to one).