Develop Reference

Keras custom data generator giving dimension errors with multi input and multi output( functional api model)

I have written a generator function with Keras, before returning X,y from __getitem__ I have double check the shapes of the X's and Y's and they are alright, but generator is giving dimension mismatch array and warnings.
(Colab Code to reproduce: https://colab.research.google.com/drive/1bSJm44MMDCWDU8IrG2GXKBvXNHCuY70G?usp=sharing)
My training and validation generators are pretty much same as
class ValidGenerator(Sequence):
def __init__(self, df, batch_size=64):
self.batch_size = batch_size
self.df = df
self.indices = self.df.index.tolist()
self.num_classes = num_classes
self.shuffle = shuffle
self.on_epoch_end()
def __len__(self):
return int(len(self.indices) // self.batch_size)
def __getitem__(self, index):
index = self.index[index * self.batch_size:(index + 1) * self.batch_size]
batch = [self.indices[k] for k in index]
X, y = self.__get_data(batch)
return X, y
def on_epoch_end(self):
self.index = np.arange(len(self.indices))
if self.shuffle == True:
np.random.shuffle(self.index)
def __get_data(self, batch):
#some logic is written here
#hat prepares 3 X features and 3 Y outputs
X = [input_array_1,input_array_2,input_array_3]
y = [out_1,out_2,out_3]
#print(len(X))
return X, y
I am return tupple of X,y from which has 3 input features and 3 output features each, so shape of X is (3,32,10,1)
I am using functional api to build model(I have things like concatenation, multi input/output, which isnt possible with sequential) with following structure
When I try to fit the model with generator with following code
train_datagen = TrainGenerator(df=train_df, batch_size=32, num_classes=None, shuffle=True)
valid_datagen = ValidGenerator(df=train_df, batch_size=32, num_classes=None, shuffle=True)
model.fit(train_datagen, epochs=2,verbose=1,callbacks=[checkpoint,es])
I get these warnings and errors, that dont go away
Epoch 1/2
WARNING:tensorflow:Model was constructed with shape (None, 10) for input >Tensor("input_1:0", shape=(None, 10), dtype=float32), but it was called >on an input with incompatible shape (None, None, None).
WARNING:tensorflow:Model was constructed with shape (None, 10) for input
Tensor("input_2:0", shape=(None, 10), dtype=float32), but it was
called on an input with incompatible shape (None, None, None).
WARNING:tensorflow:Model was constructed with shape (None, 10) for
input Tensor("input_3:0", shape=(None, 10), dtype=float32), but it was
called on an input with incompatible shape (None, None, None).
...
...
call
return super(RNN, self).call(inputs, **kwargs)
/home/eduardo/.virtualenvs/kgpu3/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py:975
call
input_spec.assert_input_compatibility(self.input_spec, inputs,
/home/eduardo/.virtualenvs/kgpu3/lib/python3.8/site-packages/tensorflow/python/keras/engine/input_spec.py:176
assert_input_compatibility
raise ValueError('Input ' + str(input_index) + ' of layer ' +
ValueError: Input 0 of layer lstm is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: [None, None, None, 88]
I have rechecked whole code and it isnt possible to have input (None,None,None) like in warning or in error, my input dimension is (3,32,10,1)
Update
I have also tried to write a generator function with python and got exactly same error.
My generator function
def generate_arrays_from_file(batchsize,df):
#print(bat)
inputs = []
targets = []
batchcount = 0
while True:
df3 = df.loc[np.arange(batchcount*batchsize,(batchcount*batchsize)+batchsize)]
#Some pre processing
X = [input_array_1,input_array_2,input_array_3]
y = [out_1,out_2,out_3]
yield X,y
batchcount = batchcount +1
It seems like it is something wrong internally wit keras (may be due to the fact I am using functional API)
Update 2
I also tried to output tuple
X = (input1_X,input2_X,input3_X)
y = (output1_y,output2_y,output3_y)
and also named input/output, but it doesnt work
X = {"input_1": input1_X, "input_2": input2_X,"input_3": input3_X}
y = {"output_1": output1_y, "output_2": output2_y,"output_3": output3_y}
Note about problem formulation:
Changing the individual X features to shape (32,10) instead of (32,10,1) might help to get rid of this error but that is not what I want, it changes my problem(I no longer have 10 time steps with one feature each)

Keras use 'None' for dynamic dimensions.
As you can see on the model.summary() chart - the model expecting shape(None, 10) for all of your inputs, which is two dimensional. With batch dimension - you should feed three dimensional data to the model.
But you are feeding four dimensional data.
I would guess that your model doesn't split your input list by three inputs. Try to change your inputs to tuple:
X = (input_array_1,input_array_2,input_array_3)

In order to resolve this error:
ValueError: Input 0 of layer lstm is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: [None, None, None, 88]
TrainGenerator should be changed in the following way.
Current code:
input1_X = np.array(df3['input1_X'].to_list()).reshape(dlen,pad_len,1)
input2_X = np.array(df3['input2_X'].to_list()).reshape(dlen,pad_len,1)
input3_X = np.array(df3['input3_X'].to_list()).reshape(dlen,pad_len,1)
Should be changed to:
input1_X = np.array(df3['input1_X'].to_list()).reshape(dlen,pad_len)
input2_X = np.array(df3['input2_X'].to_list()).reshape(dlen,pad_len)
input3_X = np.array(df3['input3_X'].to_list()).reshape(dlen,pad_len)
The reason is that each of the 3 Inputs expects a 2-dimensional array, but the generator provides a 3-dimensional one. The expected shape is (batch_size, 10).

I had a similar issue with a custom generator that just had to pass a numpy array of size 10 as input and one single output.
To solve this problem i had to trasform the shape of the 2 vectors passed to the neural network like this:
def slides_generator(integer_list):
# stuff happens
x = np_ts[np_index:np_index+10] # numpy array
y = np_ts[np_index+10] # numpy array
yield tf.convert_to_tensor(x)[np.newaxis, ...], tf.convert_to_tensor(y)[np.newaxis, ...]
doge_gen = slides_generator(integer_list) #next(doge_gen)
basically you need to pass the 2 arrays with shape (None,size),
so in my case were (None,10) and (None,1), and to achieve this i just passed 2 reshaped tensors.
you need the None dimension as the batch size.

Tensorflow construct tensor of shape (N*N*K) based on unknown batch size N

I'm trying to build a custom loss function that needs to use the unknown batch size to construct more tensors.
The batch size is a "dynamic" shape ?. Assuming this is notated as N and the prediction tensor has a shape of N*K where K is known. How do I build a tensor with a shape of N*N*K?
Basically it's multiple "mask matrices" with the same shape as the prediction tensor. So if the batch size is 2, then there should be 2 mask matrices, each of which is N*K.
I first tried to use tf.while_loop, to build up a list of N*K matrices, and then use tf.concat(l, axis=0). but this didn't work because the tf.while_loop either didn't run the side effects to append into l, or if I tried to accumulate l recursively, it also complained that the resulting shape was different from the loop_vars.
Situation is for TF1, but would be interesting a solution in TF2.
Here's an attempt:
predictions_arr = np.array([
[1,2,3,4],
[1,2,3,4],
[1,2,3,4]
], dtype=np.float32)
predictions = tf.constant(predictions_arr, dtype=tf.float32)
predictions_shape = tf.shape(predictions)
batch_size, feature_size = predictions_shape[0], predictions_shape[1]
def mask(i):
zero_vector = tf.zeros((1, feature_size), dtype=tf.float32)
one_vector = tf.ones((1, feature_size), dtype=tf.float32)
# how do I compare j != i?
# seems like it is not possible
# also the dtypes don't work
mask_matrix = tf.map_fn(
lambda j: one_vector if j != i else zero_vector,
tf.range(batch_size, dtype=tf.float32)
)
return mask_matrix
mask(0)
This gives me: ValueError: Tensor conversion requested dtype float32 for Tensor with dtype int32: <tf.Tensor 'strided_slice:0' shape=() dtype=int32>.

Try using tf.shape to get a dynamic shape and variables corresponding to dynamic shape values. Specifically you want to do something like this:
shape = tf.shape(predictions)
n, k = shape[0], shape[1]
# n, k = shape might not work if TF can't figure out the dimension in advance
result = tf.zeros([n, n, k])
EDIT: Following the comments, here's a complete snippet that seems to work (tested in TF 2)
#tf.function
def my_loss_matrix(predictions, i):
"""Generate the i-th loss matrix"""
shape = tf.shape(predictions)
n, k = shape[0], shape[1]
return tf.ones([n, k]) * i
#tf.function
def my_loss(predictions):
shape = tf.shape(predictions)
n, k = shape[0], shape[1]
return tf.map_fn(lambda i: my_loss_matrix(predictions, i), tf.range(n, dtype=predictions.dtype))
predictions = tf.Variable(tf.ones([10,3]))
my_loss(predictions)

Simple tensorflow keras model with single matrix multiply not working

I'm trying to setup a simple tf.keras model in which a vector is fed in as input and the output is the result of a single matrix multiply.
The lines of code to create the model suceed but calling it for a forward pass results in an error.
n_input_nodes = 2
n_output_nodes = 1
x = tf.keras.Input(shape=(n_input_nodes,))
W = tf.ones((n_input_nodes,n_output_nodes), dtype=tf.float32)
y = tf.matmul(x, W)
model = tf.keras.Model(inputs=x, outputs=y)
x_input = tf.constant([10,30.], shape=[1, 2])
output = model(x_input)
The final line (i.e. the forward pass) throws the following error:
ValueError: Argument must be a dense tensor: [array([[1.], [1.]], dtype=float32)] - got shape [1, 2, 1], but wanted [1].
The input is of shape (2,1) and the weight matrix has shape (2,1). Matrix multiply between the two should be a valid multiplication and result in a [1,1] tensor; however, this is not the case.

They require a dense tensor and not a sparse tensor. Consider this shape
W = tf.ones((n_input_nodes,), dtype=tf.float32)
It requires a tensor of shape ( 2, ) which is dense.

Use tf.nn.l2_loss on a collection of differently shaped vectors

I want to calcualte the l2 loss over all my weights and biases in my neural network. Therefor I add all weights and biases to the 'tf.GraphKeys.REGULARIZATION_LOSSES' and want to calculate the l2 loss with the in tensorflow defined function:
W = tf.Variable(tf.truncated_normal([inputDim, outputDim], stddev=0.1), name='W')
b = tf.Variable(tf.ones([outputDim])/10, name='b')
tf.add_to_collection(tf.GraphKeys.REGULARIZATION_LOSSES, W)
tf.add_to_collection(tf.GraphKeys.REGULARIZATION_LOSSES, b)
...
and later in the code:
...
vars = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
l2_loss = tf.nn.l2_loss(vars) * config.L2PENALTY
I get this error using the function on a network with 3 layers and couldn't find a solution to it:
ValueError: Tried to convert 't' to a tensor and failed. Error: Shapes must be equal rank, but are 2 and 1
From merging shape 4 with other shapes. for 'l2_loss/L2Loss/packed' (op: 'Pack') with input shapes: [784,512], [512], [512,256], [256], [256,10], [10].

Although tf.nn.l2_loss receives Tensor as its argument, you passed a list of Tensors to tf.nn.l2_loss. So the error message means l2_loss cannot convert the list to a Tensor.
We should calculate L2 loss like
vars = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
list_l2_loss = []
for v in vars:
list_l2_loss.append(tf.nn.l2_loss(v))
total_l2_loss = tf.add_n(list_l2_loss)
Adding Information
You can add n-D Tensor to a collection by tf.add_to_collection. But tf.GraphKeys.REGULARIZATION_LOSSES is the reserved name for regularizers created by tf.get_variable. So we should you an original name like "MyReguralizers".

How to select shape of tensor for 1-D signal (accelrometer signal) ? I keep getting VlaueError due to tensor shape

I am trying to model CNN for 1-D signal but I am unable to understand rank errors.
My program goes like this:
#Weights
def init_weights(shape):
init_random_dist = tf.truncated_normal(shape, stddev=0.1)
return tf.Variable(init_random_dist)
#Bias
def init_bias(shape):
init_bias = tf.constant(0.1,shape=shape)
return tf.Variable(init_bias)
def conv1d(x,W):
#x is input accelration data and W is corresponding weight
x = tf.cast(x, tf.float32)
tf.nn.conv1d(x,W,stride=1,padding='VALID')
def convolution_layer(input_x,shape):
w = init_weights(shape)
b = init_bias([shape[3]])
return tf.nn.relu(conv1d(input_x,w)+b)
Now placeholders
x = tf.placeholder(tf.float32,shape=[1,1,200,1])
y_true = tf.placeholder(tf.float32,shape=[None,6])
while creating 1st layer using con_layer_1 = convolution_layer(x,shape=[1,20,1,32]) i get rank ValueError which i'm unable to degubg. Error statement is:
ValueError: Shape must be rank 4 but is rank 5 for 'conv1d_20/Conv2D' (op: 'Conv2D') with input shapes: [1,1,1,200,1], [1,1,20,1,32].

The input and weights shapes to the nn.conv1d is not right. The input shape for the nn.conv1d should be of the size : [ batch_size, input_length, input_channels] and the weights matrix should be of size [filter_size, inputs_channels, output_channels]. So you need to change your code to :
def convolution_layer(input_x,shape):
w = init_weights(shape)
b = init_bias([shape[2]])
return tf.nn.relu(conv1d(input_x,w)+b)
x = tf.placeholder(tf.float32,shape=[1,200,1])
y_true = tf.placeholder(tf.float32,shape=[None,6])
con_layer_1 = convolution_layer(x,shape=[20,1,32])
Note: You should try yo use tf.layers API that takes care of the weights assignment and all.