Develop Reference

Deep Learning how to split 5 dimensions timeseries and pass some dimensions through embedding layer

I have an input that is a time series of 5 dimensions:
a = [[8,3],[2] , [4,5],[1], [9,1],[2]...] #total 100 timestamps. For each element, dims 0,1 are numerical data and dim 2 is a numerical encoding of a category. This is per sample, 3200 samples
The category has 3 possible values (0,1,2)
I want to build a NN such that the last dimension (the category) will go through an embedding layer with output size 8, and then will be concatenated back to the first two dims (the numerical data).
So, this will be something like:
input1 = keras.layers.Input(shape=(2,)) #the numerical features
input2 = keras.layers.Input(shape=(1,)) #the encoding of the categories. this part will be embedded to 5 dims
x2 = Embedding(input_dim=1, output_dim = 8)(input2) #apply it to every timestamp and take only dim 3, so [2],[1], [2]
x = concatenate([input1,x2]) #will get 10 dims at each timepoint, still 100 timepoints
x = LSTM(units=24)(x) #the input has 10 dims/features at each timepoint, total 100 timepoints per sample
x = Dense(1, activation='sigmoid')(x)
model = Model(inputs=[input1, input2] , outputs=[x]) #input1 is 1D vec of the width 2 , input2 is 1D vec with the width 1 and it is going through the embedding
model.compile(
loss='binary_crossentropy',
optimizer='adam',
metrics=['acc']
)
How can I do it? (preferably in keras)?
My problem is how to apply the embedding to every time point?
Meaning, if I have 1000 timepoints with 3 dims each, I need to convert it to 1000 timepoints with 8 dims each (The emebedding layer should transform input2 from (1000X1) to (1000X8)

There are a couple of issues you are having here.
First let me give you a working example and explain along the way how to solve your issues.
Imports and Data Generation
import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from tensorflow.keras.models import Model
num_timesteps = 100
max_features_values = [100, 100, 3]
num_observations = 2
input_list = [[[np.random.randint(0, v) for _ in range(num_timesteps)]
for v in max_features_values]
for _ in range(num_observations)]
input_arr = np.array(input_list) # shape (2, 3, 100)
In order to use an embedding we need to the voc_size as input_dimension, as stated in the LSTM documentation.
Embedding and Concatenation
voc_size = len(np.unique(input_arr[:, 2, :])) + 1 # 4
Now we need to create the inputs. Inputs should be of size [None, 2, num_timesteps] and [None, 1, num_timesteps] where the first dimension is the flexible and will be filled with the number of observations we are passing in. Let's use the embedding right after that using the previously calculated voc_size.
inp1 = layers.Input(shape=(2, num_timesteps)) # TensorShape([None, 2, 100])
inp2 = layers.Input(shape=(1, num_timesteps)) # TensorShape([None, 1, 100])
x2 = layers.Embedding(input_dim=voc_size, output_dim=8)(inp2) # TensorShape([None, 1, 100, 8])
x2_reshaped = tf.transpose(tf.squeeze(x2, axis=1), [0, 2, 1]) # TensorShape([None, 8, 100])
This cannot be easily concatenated since all dimensions must match except for the one along the concatenation axis. But the shapes are not matching unfortunately. Therefore we reshape x2. We do so by removing the first dimension and then transposing.
Now we can concatenate without any issue and everything works in a straight forward fashion:
x = layers.concatenate([inp1, x2_reshaped], axis=1)
x = layers.LSTM(32)(x)
x = layers.Dense(1, activation='sigmoid')(x)
model = Model(inputs=[inp1, inp2], outputs=[x])
Check on Dummy Example
inp1_np = input_arr[:, :2, :]
inp2_np = input_arr[:, 2:, :]
model.predict([inp1_np, inp2_np])
# Output
# array([[0.544262 ],
# [0.6157502]], dtype=float32)
#This outputs values between 0 and 1 just as expected.

In case you are not looking for Embeddings the way it's usually used in Keras (positive integers mapping to dense vectors). You might be looking for some sort of unprojection or basis expansion, in which 3 dimensions get mapped (embedded) to 8 and concatenating the result. This can be done using the kernel trick or other methods, but also happens implicitly in neural networks with non-linear applications.
As such, you can do something like this, following a similar format to pythonic833 because it was good (but with timestamps in the middle per the Keras LSTM documentation asking for [batch, timesteps, feature]):
Input generation
import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from tensorflow.keras.models import Model
num_timesteps = 100
num_features = 5
num_observations = 2
input_list = [[[np.random.randint(1, 100) for _ in range(num_features)]
for _ in range(num_timesteps)]
for _ in range(num_observations)]
input_arr = np.array(input_list) # shape (2, 100, 5)
Model construction
Then you can process the inputs:
input1 = layers.Input(shape=(num_timesteps, 2,))
input2 = layers.Input(shape=(num_timesteps, 3))
x2 = layers.Dense(8, activation='relu')(input2)
x = layers.concatenate([input1,x2], axis=2) # This produces tensors of shape (None, 100, 10)
x = layers.LSTM(units=24)(x)
x = layers.Dense(1, activation='sigmoid')(x)
model = Model(inputs=[input1, input2] , outputs=[x])
model.compile(
loss='binary_crossentropy',
optimizer='adam',
metrics=['acc']
)
Results
inp1_np = input_arr[:, :, :2]
inp2_np = input_arr[:, :, 2:]
model.predict([inp1_np, inp2_np])
which produces
array([[0.44117224],
[0.23611131]], dtype=float32)
Other explanations about basis expansion to check out:
https://stats.stackexchange.com/questions/527258/embedding-data-into-a-larger-dimension-space
https://www.reddit.com/r/MachineLearning/comments/2ffejw/why_dont_researchers_use_the_kernel_method_in/

Training data dimensions for semantic segmentation using CNN

I encountered many hardships when trying to fit a CNN (U-Net) to my tif training images in Python.
I have the following structure to my data:
X
0
[Images] (tif, 3-band, 128x128, values ∈ [0, 255])
X_val
0
[Images] (tif, 3-band, 128x128, values ∈ [0, 255])
y
0
[Images] (tif, 1-band, 128x128, values ∈ [0, 255])
y_val
0
[Images] (tif, 1-band, 128x128, values ∈ [0, 255])
Starting with this data, I defined ImageDataGenerators:
import tensorflow as tf
from tensorflow import keras as ks
from matplotlib import pyplot as plt
import numpy as np
bs = 10 # batch size
args_col = {"data_format" : "channels_last",
"brightness_range" : [0.5, 1.5]
}
args_aug = {"rotation_range" : 365,
"width_shift_range" : 0.05,
"height_shift_range" : 0.05,
"horizontal_flip" : True,
"vertical_flip" : True,
"fill_mode" : "constant",
"featurewise_std_normalization" : False,
"featurewise_center" : False
}
args_flow = {"color_mode" : "rgb",
"class_mode" : "sparse",
"batch_size" : bs,
"target_size" : (128, 128),
"seed" : 42
}
# train generator
X_generator = ks.preprocessing.image.ImageDataGenerator(rescale = 1.0/255.0,
**args_aug,
**args_col)
X_gen = X_generator.flow_from_directory(directory = "my/directory/X",
**args_flow)
y_generator = ks.preprocessing.image.ImageDataGenerator(**args_aug,
cval = NoDataValue)
y_gen = y_generator.flow_from_directory(directory = "my/directory/y",
**args_flow, color_mode = "grayscale")
train_generator = zip(X_gen, y_gen)
# val generator
X_val_generator = ks.preprocessing.image.ImageDataGenerator(rescale = 1.0/255.0)
X_val_gen = X_generator.flow_from_directory(directory = "my/directory/X_val"),
**args_flow)
y_val_generator = ks.preprocessing.image.ImageDataGenerator()
y_val_gen = y_generator.flow_from_directory(directory = "my/directory/y_val"),
**args_flow, color_mode = "grayscale")
val_generator = zip(X_val_gen, y_val_gen)
Using this generator, I can create pairs of training images and corresponding masks and visualize them like this:
X, y = next(train_generator)
X_test = X[0][0]
y_test = y[0][0]
plt.subplot(1, 2, 1)
plt.imshow(np.array(X_test))
plt.subplot(1, 2, 2)
plt.imshow(np.array(y_test))
Resulting in:
However, I cannot train a U-Net, as I intended:
When I define a U-Net based on an example from the internet (or basically any other example of a U-Net I've found) as model and then do the following:
model.compile(optimizer = "adam", loss = "sparse_categorical_crossentropy", metrics = ["accuracy"])
model.fit(train_generator, epochs = 5, steps_per_epoch = 10, validation_data = val_generator)
it will fail with the error:
ValueError: Layer model expects 1 input(s), but it received 2 input tensors. Inputs received: [<tf.Tensor 'IteratorGetNext:0' shape=(None, None, None, None) dtype=float32>, <tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=float32>]
I tried other loss functions and other class_mode arguments, but it always failed with some error related to the dimensions of the input data or the data passed between layers. An other example (when setting class_mode = None:
InvalidArgumentError: logits and labels must have the same first dimension, got logits shape [16384,1] and labels shape [49152]
I just started getting into CNNs and Python, so I have no clue what to try further or how to resolve those errors. I was pretty sure I use the correct loss function, which seems to be often the problem when similar errors occur (I have multiple classes, hence the "sparse_categorical_crossentropy").
Any ideas how to solve this and make the data fit the expected CNN input (or the other way round, depending on what the problem is)?
Note:
My ImageDataGenerator outputs a pair of images (X and y) with the following format (I noticed I had to set color_mode to "grayscale" for the masks (y)):
I used keras.layers.Input(shape = (128, 128, 3)) in the example U-Net, since the keras documentation states shape = "A shape tuple (integers), not including the batch size".

I found the answer to this particular problem. Amongst other issues, "class_mode" has to be set to None for this kind of model. With that set, the second array in both X and y is not written by the ImageDataGenerator. As a result, X and y are interpreted as the data and the mask (which is what we want) in the combined ImageDataGenerator. Otherwise, X_val_gen already produces the tuple shown in the screenshot, where the second entry is interpreted as the class, which would make sense in a classification problem with images spread out in various folders each labeled with a class ID.

Simple tensorflow keras model with single matrix multiply not working

I'm trying to setup a simple tf.keras model in which a vector is fed in as input and the output is the result of a single matrix multiply.
The lines of code to create the model suceed but calling it for a forward pass results in an error.
n_input_nodes = 2
n_output_nodes = 1
x = tf.keras.Input(shape=(n_input_nodes,))
W = tf.ones((n_input_nodes,n_output_nodes), dtype=tf.float32)
y = tf.matmul(x, W)
model = tf.keras.Model(inputs=x, outputs=y)
x_input = tf.constant([10,30.], shape=[1, 2])
output = model(x_input)
The final line (i.e. the forward pass) throws the following error:
ValueError: Argument must be a dense tensor: [array([[1.], [1.]], dtype=float32)] - got shape [1, 2, 1], but wanted [1].
The input is of shape (2,1) and the weight matrix has shape (2,1). Matrix multiply between the two should be a valid multiplication and result in a [1,1] tensor; however, this is not the case.

They require a dense tensor and not a sparse tensor. Consider this shape
W = tf.ones((n_input_nodes,), dtype=tf.float32)
It requires a tensor of shape ( 2, ) which is dense.

Shapes of logits and labels are incompatible

The full error message is like this:
ValueError: Shapes (2, 1) and (50, 1) are incompatible
It occurs when my model is trained. The mistake either is in my input_fn:
train_input_fn = tf.estimator.inputs.numpy_input_fn(
x = {"x" : training_data},
y = training_labels,
batch_size = 50,
num_epochs = None,
shuffle = True)
in my logits and loss function:
dense = tf.layers.dense(inputs = pool2_flat, units = 1024, activation = tf.nn.relu)
dropout = tf.layers.dropout(inputs = dense, rate = 0.4, training = mode == tf.estimator.ModeKeys.TRAIN)
logits = tf.layers.dense(inputs = dropout, units = 1)
loss = tf.losses.softmax_cross_entropy(labels = labels, logits = logits)
or in my dataset. I can only print out the shape of my dataset for you to take a look at it.
#shape of the dataset
train_data.shape
(1196,2,1)
train_data[0].shape
(2,1)
#this is the data
train_data[0][0].shape
(1,)
train_data[0][0][0].shape
(20,50,50)
#this is the labels
train_data[0][1].shape
(1,)
The problem seems to be the shape of the logits. They are supposed to be [batch_size, num_classes] in this case [50,1] but are [2,1]. The shape of the labels is correctly [50,1]
I have made a github gist if you want to take a look at the whole code.
https://gist.github.com/hjkhjk1999/38f358a53da84a94bf5a59f44050aad5

In your code, you are stating that the inputs to your model will be feed in batches of 50 samples per batch with one variable. But it looks like your are feeding actually a batch of 2 samples with 1 variable (shape=[2, 1]) despite feeding labels with shape [50, 1].
That's the problem, you are giving 50 'questions' and two 'answers'.
Also, your dataset is shaped in a really weird way. I see you named your github gist 3D Conv. If you are indeed trying to do a 3D convolution you might want to reshape your dataset into a tensor (numpy array) of this shape shape = [samples, width, height, deepth]

unpack(unstack) an input (placeholder) with one None dimension in tensorflow

I am trying to use LSTM with inputs with different time steps (different number of frames). The input to the rnn.static_rnn should be a sequence of tf (not a tf!). So, I should convert my input to sequence. I tried to use tf.unstack and tf.split, but both of them need to know exact size of inputs, while one dimension of my inputs (time steps) is changing by different inputs. following is part of my code:
n_input = 256*256 # data input (img shape: 256*256)
n_steps = None # timesteps
batch_size = 1
# tf Graph input
x = tf.placeholder("float", [ batch_size , n_input,n_steps])
y = tf.placeholder("float", [batch_size, n_classes])
# Permuting batch_size and n_steps
x1 = tf.transpose(x, [2, 1, 0])
x1 = tf.transpose(x1, [0, 2, 1])
x3=tf.unstack(x1,axis=0)
#or x3 = tf.split(x2, ?, 0)
# Define a lstm cell with tensorflow
lstm_cell = rnn.BasicLSTMCell(num_units=n_hidden, forget_bias=1.0)
# Get lstm cell output
outputs, states = rnn.static_rnn(lstm_cell, x3, dtype=tf.float32,sequence_length=None)
I got following error when I am using tf.unstack:
ValueError: Cannot infer num from shape (?, 1, 65536)
Also, there are some discussions here and here, but none of them were useful for me. Any help is appreciated.

As explained in here, tf.unstack does not work if the argument is unspecified and non-inferrable.
In your code, after transpositions, x1 has the shape of [ n_steps, batch_size, n_input] and its value at axis=0 is set to None.