Develop Reference

Deep Learning how to split 5 dimensions timeseries and pass some dimensions through embedding layer

I have an input that is a time series of 5 dimensions:
a = [[8,3],[2] , [4,5],[1], [9,1],[2]...] #total 100 timestamps. For each element, dims 0,1 are numerical data and dim 2 is a numerical encoding of a category. This is per sample, 3200 samples
The category has 3 possible values (0,1,2)
I want to build a NN such that the last dimension (the category) will go through an embedding layer with output size 8, and then will be concatenated back to the first two dims (the numerical data).
So, this will be something like:
input1 = keras.layers.Input(shape=(2,)) #the numerical features
input2 = keras.layers.Input(shape=(1,)) #the encoding of the categories. this part will be embedded to 5 dims
x2 = Embedding(input_dim=1, output_dim = 8)(input2) #apply it to every timestamp and take only dim 3, so [2],[1], [2]
x = concatenate([input1,x2]) #will get 10 dims at each timepoint, still 100 timepoints
x = LSTM(units=24)(x) #the input has 10 dims/features at each timepoint, total 100 timepoints per sample
x = Dense(1, activation='sigmoid')(x)
model = Model(inputs=[input1, input2] , outputs=[x]) #input1 is 1D vec of the width 2 , input2 is 1D vec with the width 1 and it is going through the embedding
model.compile(
loss='binary_crossentropy',
optimizer='adam',
metrics=['acc']
)
How can I do it? (preferably in keras)?
My problem is how to apply the embedding to every time point?
Meaning, if I have 1000 timepoints with 3 dims each, I need to convert it to 1000 timepoints with 8 dims each (The emebedding layer should transform input2 from (1000X1) to (1000X8)

There are a couple of issues you are having here.
First let me give you a working example and explain along the way how to solve your issues.
Imports and Data Generation
import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from tensorflow.keras.models import Model
num_timesteps = 100
max_features_values = [100, 100, 3]
num_observations = 2
input_list = [[[np.random.randint(0, v) for _ in range(num_timesteps)]
for v in max_features_values]
for _ in range(num_observations)]
input_arr = np.array(input_list) # shape (2, 3, 100)
In order to use an embedding we need to the voc_size as input_dimension, as stated in the LSTM documentation.
Embedding and Concatenation
voc_size = len(np.unique(input_arr[:, 2, :])) + 1 # 4
Now we need to create the inputs. Inputs should be of size [None, 2, num_timesteps] and [None, 1, num_timesteps] where the first dimension is the flexible and will be filled with the number of observations we are passing in. Let's use the embedding right after that using the previously calculated voc_size.
inp1 = layers.Input(shape=(2, num_timesteps)) # TensorShape([None, 2, 100])
inp2 = layers.Input(shape=(1, num_timesteps)) # TensorShape([None, 1, 100])
x2 = layers.Embedding(input_dim=voc_size, output_dim=8)(inp2) # TensorShape([None, 1, 100, 8])
x2_reshaped = tf.transpose(tf.squeeze(x2, axis=1), [0, 2, 1]) # TensorShape([None, 8, 100])
This cannot be easily concatenated since all dimensions must match except for the one along the concatenation axis. But the shapes are not matching unfortunately. Therefore we reshape x2. We do so by removing the first dimension and then transposing.
Now we can concatenate without any issue and everything works in a straight forward fashion:
x = layers.concatenate([inp1, x2_reshaped], axis=1)
x = layers.LSTM(32)(x)
x = layers.Dense(1, activation='sigmoid')(x)
model = Model(inputs=[inp1, inp2], outputs=[x])
Check on Dummy Example
inp1_np = input_arr[:, :2, :]
inp2_np = input_arr[:, 2:, :]
model.predict([inp1_np, inp2_np])
# Output
# array([[0.544262 ],
# [0.6157502]], dtype=float32)
#This outputs values between 0 and 1 just as expected.

In case you are not looking for Embeddings the way it's usually used in Keras (positive integers mapping to dense vectors). You might be looking for some sort of unprojection or basis expansion, in which 3 dimensions get mapped (embedded) to 8 and concatenating the result. This can be done using the kernel trick or other methods, but also happens implicitly in neural networks with non-linear applications.
As such, you can do something like this, following a similar format to pythonic833 because it was good (but with timestamps in the middle per the Keras LSTM documentation asking for [batch, timesteps, feature]):
Input generation
import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from tensorflow.keras.models import Model
num_timesteps = 100
num_features = 5
num_observations = 2
input_list = [[[np.random.randint(1, 100) for _ in range(num_features)]
for _ in range(num_timesteps)]
for _ in range(num_observations)]
input_arr = np.array(input_list) # shape (2, 100, 5)
Model construction
Then you can process the inputs:
input1 = layers.Input(shape=(num_timesteps, 2,))
input2 = layers.Input(shape=(num_timesteps, 3))
x2 = layers.Dense(8, activation='relu')(input2)
x = layers.concatenate([input1,x2], axis=2) # This produces tensors of shape (None, 100, 10)
x = layers.LSTM(units=24)(x)
x = layers.Dense(1, activation='sigmoid')(x)
model = Model(inputs=[input1, input2] , outputs=[x])
model.compile(
loss='binary_crossentropy',
optimizer='adam',
metrics=['acc']
)
Results
inp1_np = input_arr[:, :, :2]
inp2_np = input_arr[:, :, 2:]
model.predict([inp1_np, inp2_np])
which produces
array([[0.44117224],
[0.23611131]], dtype=float32)
Other explanations about basis expansion to check out:
https://stats.stackexchange.com/questions/527258/embedding-data-into-a-larger-dimension-space
https://www.reddit.com/r/MachineLearning/comments/2ffejw/why_dont_researchers_use_the_kernel_method_in/

Using two tables as input for multi-output regression with keras, the missing matrix product

In the keras MWE below I'm trying to train a multi-output regression model with 1000 samples having 20 features (X) as input and producing outputs of size 50 (Y). However, I'm missing a step that I fail to wrap my head around and that I miss the word to describe properly. Let me try anyway, and please forgive the mess:
Here, each one of the 50 outputs is characterised by a set of 10 "feature filters" which are here to interact (through e.g. a dot product) with the 20 features to produce the numeric output. I miss a layer that would train a unique weight matrix of size (20, 10) whose sum (or average) subsequently produces the numeric output Y. The idea is that the output reacts to the features in ways that are dictated by those feature filters and that those interactions are consistent across outputs (e.g. high values in one feature filter might lead to a higher reaction to one feature and lower to another one, and those positive/negative relationships are not output-specific but identified for the whole dataset via the common weight matrix of size 10x20).
How could that side-input matrix (10, 50) of output-specific "feature filters" enter the network? My try below consists in (1) a tensordot product for every sample with the side-matrix (i.e. 3D output), which is (2) subsequently flatten to 1D to interact with a small Dense layer. The dense layer is then (3) tiled/repeated so that it stays small and learn weights that apply to all outputs. The tiled dense output is then (4) dimensionally reduced through averaging to fit the output format of (n, 50).
The problem with this approach is the Dense layer is fully connected, when all that is needed is a locally connected weight matrix (10 * 20) that is tiled 50 times. That is, 1 weight/bias per interaction between feature and feature filter, which apply to every outputs. Having that one weight per interaction we can then visualise which interaction are key to match the output (which is not possible if fully connected).
I suspect I need to replace the dense layer by some locally connected or convolutional or separable or some sort of layer that I don't really understand. Any ideas?
import numpy as np
import tensorflow as tf
from tensorflow import keras
## create dummy input/output matrices
XData = np.ones((1000, 20)) ## 1000 samples, 20 features
YData = np.ones((1000, 50)) ## 1000 samples, 50 outputs
filterData = np.ones((10, 50)) ## 10 feature filters, 50 outputs
filterData = tf.cast(filterData, tf.float32) ## needed for tf.math.reduce_mean() below
## input of size (n, 20)
input = keras.Input(XData.shape[1])
## dot product with filterData, out size = (n, 20, 10, 50)
tdot = keras.layers.Lambda(lambda x: tf.tensordot(x, filterData, axes=0))(input)
# flatten for dense layers, out size = (n, 10000)
tflat = keras.layers.Flatten()(tdot)
## learning dense layer, out size = (n, 20*10),
tdense = keras.layers.Dense(XData.shape[1] * filterData.shape[0], activation="linear")(tflat)
## tiling layer that repeats the dense layer for every output
ttile = keras.layers.Lambda(lambda x: keras.backend.repeat(x, filterData.shape[1]))(tdense)
## reduce dimensions through averaging to fit YData, out size = (n, 50)
tmean = keras.layers.Lambda(lambda x: tf.math.reduce_mean(x, axis=(2)))(ttile)
## make the model
model = keras.Model(input, tmean)
model.compile(
optimizer='adam',
loss='mse'
)
history = model.fit(
x = XData,
y = YData,
epochs = 3,
validation_split = 0.3,
verbose = 2,
batch_size=10
)
EDIT
The code below achieves the singular connection, i.e. one weigth per feature/feature_filter interaction (shared throughout outputs), that the dense layer does not allow. It consists in a collection of 20 * 10 = 200 single unit dense layer that are subsequently concatenated, before being tiled 50 times. However learning is very poor and maybe setting that concatenated colection inside a time distributed layer, as suggested by #SoheilStar could help. However the presence of the loop prevents me from using it in the sequential API code given by #SoheilStar. Any help on this?
## create dummy input/output matrices
XData = np.ones((1000, 20)) ## 1000 samples, 20 features
YData = np.zeros((1000, 50)) ## 1000 samples, 50 outputs
filterData = np.ones((10, 50)) ## 10 feature filters, 50 outputs
filterData = tf.cast(filterData, tf.float32) ## needed for tf.math.reduce_mean() below
## input of size (n, 20)
input = keras.Input(XData.shape[1])
## dot product with filterData, out size = (n, 20, 10, 50)
tdot = keras.layers.Lambda(lambda x: tf.tensordot(x, filterData, axes=0))(input)
# flatten for dense layers, out size = (n, 10000)
tflat = keras.layers.Flatten()(tdot)
## singular connection layer, i.e. a concatenated collection of single unit dense layer, out size = (n, 200)
dense_list = [None] * (filterData.shape[0] * XData.shape[1])
for i in range(filterData.shape[0] * XData.shape[1]):
dense_list[i] = keras.layers.Dense(1, activation="linear")(tflat[:,i:(i+1)])
tdense = keras.layers.Concatenate()(dense_list)
## tiling layer that repeats the dense layer for every output
ttile = keras.layers.Lambda(lambda x: keras.backend.repeat(x, filterData.shape[1]))(tdense)
## reduce dimensions through averaging to fit YData, out size = (n, 50)
tmean = keras.layers.Lambda(lambda x: tf.math.reduce_mean(x, axis=(2)))(ttile)
## make the model
model = keras.Model(input, tmean)
EDIT 2
To address the previous problem of having the for loop in the time distributed layer, I defined a custom function to give to the time distributed layer:
## define a custom layer to be used in a time distributed layer with the sequential api
class customLayer(tf.keras.layers.Layer):
def __init__(self):
super().__init__()
self.input_dim = filterData.shape[0] * XData.shape[1]
self.dense_list = [None] * (self.input_dim)
for i in range(self.input_dim):
self.dense_list[i] = keras.layers.Dense(1, activation="linear")
self.concat = keras.layers.Concatenate()
self.flat = keras.layers.Flatten()
def call(self, inputs):
flat_input = self.flat(inputs)
list = [None] * (self.input_dim)
for i in range(self.input_dim):
list[i] = self.dense_list[i](flat_input[:, i:(i+1)])
return self.concat(list)
def compute_output_shape(self, input_dim):
return (None, self.input_dim)
## transpose and time distribute along the first dimension (now the output size)
tdot_ = tf.transpose(tdot, [0, 3, 1, 2])
## call the customLayer inside a time distributed layer
tcustom = tf.keras.layers.TimeDistributed(customLayer())(tdot_)
And this work, technically, but learning is very poor. The proposition of #SoheilStar below, works after changing the last line so that we have instead:
## This layer would try to train its parameters according to each parameter
tdense_ = tf.keras.layers.TimeDistributed(tf.keras.layers.Flatten())(tdot_)
tdense_ = [tf.keras.layers.TimeDistributed(keras.layers.Dense(1, activation="linear"))(tdense_[:, :, i][..., None]) for i in range(XData.shape[1] * filterData.shape[0])]
tdense_ = tf.keras.layers.Concatenate()(tdense_)
Although again learning is poor, but it is probably to be expected with my real data and the small number of weights in presence.

Updated
I am not sure if I got the problem correctly. If you are looking to train the Dense layer simultaneously on the 50 outputs, then you can use TimeDistributed layer like this:
## create dummy input/output matrices
XData = tf.ones((1000, 20)) ## 1000 samples, 20 features
YData = tf.ones((1000, 50)) ## 1000 samples, 50 outputs
filterData = tf.ones((10, 50))
TrData = tf.ones((10, 50), dtype=tf.float32) ## 10 feature filters, 50 outputs
## input of size (n, 20)
input = keras.Input(XData.shape[1])
## dot product with filterData, out size = (n, 20, 10, 50)
tdot = keras.layers.Lambda(lambda x: tf.tensordot(x, filterData, axes=0))(input)
## My modification
## change the order of dimensions in order to use the TimeDistributed layer
tdot_ = tf.transpose(tdot, [0, 3, 1, 2])
## This layer would try to train its parameters according to each output
tdense_ = tf.keras.layers.TimeDistributed(tf.keras.models.Sequential([tf.keras.layers.Flatten(),
tf.keras.layers.Dense(XData.shape[1] * TrData.shape[0]),
tf.keras.layers.Dense(1)]))(tdot_)
## I used the Flatten to squeeze the output and the final shape would be (Batch, 50)
final_output = tf.keras.layers.Flatten()(tdense_)
But if it is not, then why you're not putting another Dense layer with a size of 50 after the tdense? like this:
tdense = keras.layers.Dense(XData.shape[1] * TrData.shape[0], activation="linear")(tflat)
final_output = keras.layers.Dense(50, activation="linear")(tdense)
Update
To address the issue you mentioned about the For loop, I did some modifications:
import numpy as np
import tensorflow as tf
from tensorflow import keras
## create dummy input/output matrices
XData = tf.ones((1000, 20)) ## 1000 samples, 20 features
YData = tf.ones((1000, 50)) ## 1000 samples, 50 outputs
filterData = tf.ones((10, 50))
TrData = tf.ones((10, 50), dtype=tf.float32) ## 10 feature filters, 50 outputs
## input of size (n, 20)
input = keras.Input(XData.shape[1])
## dot product with filterData, out size = (n, 20, 10, 50)
tdot = keras.layers.Lambda(lambda x: tf.tensordot(x, filterData, axes=0))(input)
## My modification
## change the order of dimension in order to use TimeDistributed
tdot_ = tf.transpose(tdot, [0, 3, 1, 2])
## This layer would try to train its parameters according to each parameter
tdense_ = tf.keras.layers.TimeDistributed(tf.keras.layers.Flatten())(tdot_)
tdense_ = [tf.keras.layers.TimeDistributed(keras.layers.Dense(1, activation="linear"))(tdense_[:, :, i][..., None]) for i in range(XData.shape[1] * filterData.shape[0])]
tdense_ = tf.keras.layers.TimeDistributed(tf.keras.layers.Concatenate())(tdense_)
## reduce dimensions through averaging to fit YData, out size = (n, 50)
tmean = keras.layers.Lambda(lambda x: tf.math.reduce_mean(x, axis=(2)))(tdense_)

How does tf.dataset interact with keras.conv1D?

I'm using tf 1.15, i'm trying to make a regression task using a signal.
First of all i load my signals into the pipeline, i have several files, here i simulate the loading using a np.zeros to make the code usable by you.
Every file has this shape (?, 75000, 3), where ? is a random number of elements, 75000 is the number of samples in each element and 3 is the number of signals.
Using the tf.data i unpack them and i get a dataset who output signals with this shape (75000,), and i use them in my keras model.
Everything should be fine until i create the keras model, i copied my input pipeline because during my tests i got different errors using a generic tf.data.dataset or using the dataset built in this way.
import numpy as np
import tensorflow as tf
# called in the dataset pipeline
def my_func(x):
p = np.zeros([86, 75000, 3])
x = p[:,:,0]
y = p[:, :, 1]
z = p[:, :, 2]
return x, y, z
# called in the dataset pipeline
def load_sign(path):
func = tf.compat.v1.numpy_function(my_func, [path], [tf.float64, tf.float64, tf.float64])
return func
# Dataset pipeline
s = [1, 2] # here i have the file paths, i simulate it with numbers
AUTOTUNE = tf.data.experimental.AUTOTUNE
ds = tf.data.Dataset.from_tensor_slices(s)
# ds = ds.map(load_sign, num_parallel_calls=AUTOTUNE)
ds = ds.map(load_sign, num_parallel_calls=AUTOTUNE).unbatch()
itera = tf.data.make_one_shot_iterator(ds)
ABP, ECG, PLETH = itera.get_next()
# Until there everything should be fine
# Here i create my convolutional network
signal = tf.keras.layers.Input(shape=(None,75000), dtype='float32')
x = tf.compat.v1.keras.layers.Conv1D(64, (1), strides=1, padding='same')(signal)
x = tf.keras.layers.Dense(75000)(x)
model = tf.keras.Model(inputs=signal, outputs=x, name='resnet18')
# And finally i try to insert my signal into model
logits = model(PLETH)
I get this error:
ValueError: Input 0 of layer conv1d is incompatible with the layer: its rank is undefined, but the layer requires a defined rank.
Why? And how can i make it works?
Also the input size of my net should be this one according the documentation:
3D tensor with shape: (batch_size, steps, input_dim)
What is the steps? In my case i assume it should be (batch_size, 1, 75000), right?

In Keras or Tensorflow, how to input different sample sizes and pairwise concatenate embeddings within a model?

In follow-up to [this question], a few notes on what we're looking to accomplish:
We have two inputs X and Y of different sample sizes n and m, and a boolean vector z of size nm. Each element of z denotes whether two items in X and Y are some sort of match
We want to use embedding layers (perhaps the same, perhaps different) on X and Y before pairwise concatenating the output of these embedding layers to derive input to an output layer.
Here's one example of what a simple network could look like:
The linked answer and a few other resources helped get as far as this example, which builds a model but throws this error at fit time: ValueError: All input arrays (x) should have the same number of samples. Got array shapes: [(10, 2), (12, 2)].
Recent versions of Keras allow for skipping the dimension check, can this check be skipped in Tensorflow? I'd also be happy to use Keras, but I'm not sure how to perform the reshape and concatenate in Keras in the middle of a model.
Or, is this simply not possible in either? Is the only option to expand and pairwise concatenate prior to input?
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np
k = 2
N = 10
M = 12
x = np.random.randint(2, size = 2 * N).reshape((-1,2))
y = np.random.randint(2, size = 2 * M).reshape((-1,2))
x_rep = np.tile(x, (1, M)).reshape((-1,2))
y_rep = np.tile(y, (N, 1))
xy = np.concatenate((x_rep, y_rep), axis=1)
xy = xy.astype(bool)
z = (xy[:,0] == xy[:,2]) * (xy[:,1] ^ xy[:,3])
print(z[:20])
xy = xy.astype(int)
z = z.astype(int)
first = keras.Input(shape=(k,))
second = keras.Input(shape=(k,))
shared_dense = layers.Dense(k)
first_dense = shared_dense(first)
second_dense = shared_dense(second)
first_tiled = layers.Lambda(tf.tile, arguments={'multiples':[1, M]}, name='first_expanded' )(first_dense) #keras.backend.tile(first_dense, [1, M])
second_tiled = layers.Lambda(tf.tile, arguments={'multiples':[N,1]}, name='second_expanded')(second_dense) #keras.backend.tile(first_dense, [1, M])
first_reshaped = layers.Reshape((k,))(first_tiled)
concatenated = layers.Concatenate()([first_reshaped, second_tiled])
out = layers.Dense(1)(concatenated)
model = keras.Model([first, second], out)
keras.utils.plot_model(model, 'tf_nw.png', show_shapes=True)
model.compile('Adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit([x, y], z)

Import and reshape MNIST data, numpy

I want to reshape the MNIST dataset from shape (70000, 784) to (70000, 28, 28), the following code is tryed, but it gets a TypeError:
TypeError: only integer scalar arrays can be converted to a scalar index
df = pd.read_csv('images.csv', sep=',', header=None)
x_data = np.array(df)
x_data = x_data.reshape(x_data[0], 28, 28)
This works, but is slow
data = np.array(df)
x_data = []
for d in data:
x_data.append(d.reshape(28,28))
x_data = np.array(x_data)
How should this be with numpy.reshape() and without looping?
Manny thanks!

I think, the problem with the second one is because ur using a for loop it can take more time. So i would suggest you can try this
import tensorflow as tf
#load the data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data', validation_size=0)
#considering only first 2 data points
img = mnist.train.images[:2]
x = tf.reshape(img, shape=[-1, 28, 28, 1]) # -1 refers to standard feature which is equivalent to 28*28*1 here
Ideally i got the shape for x as (2, 28, 28, 1). Hope this helps!!

For MNIST dataset, you may use following to convert your dataset into 3D,
train = pd.read_csv("images.csv")
data = data.values.reshape(-1,28,28,1)
assuming you have data as pandas dataframe and first label column is already dropped.

Datasets.fetch_openml returns pair values includes features and target of mnist data.
Then we reshape the a certain row of feature in (28,28) 2-D array.
And as these features are the pixel intensity we can plot this 2-D array to visualise.
pixel_values,targets=datasets.fetch_openml(
'mnist_784',
version=1,
return_X_y=True
)
single_image=pixel_values[1:2].values.reshape(28,28)
plt.imshow(single_image,cmap='gray')