I'm making a model whose output I want to be of dims (A,B).
So I'm making a list of denses (A elements, B outputs each) and I wanted my output to be (No_samples, A,B).
It is a list of A elements with (No_samples,B). The method of having one dense with output AxB does not help because for every row I want to softmax accross only that
I've attempted to use tf.concatenate, tf.reshape, but there is always either an error or the same undesirable output. My difficulty is that in order to proceed I have to do some really weird reshaping and I wish to avoid that by
for i in range(0, A):
outputs.append(Dense(B, activation="softmax")(out))
And I've tried everything below (separately):
outputs = tf.stack(outputs)
outputs = Reshape(self.output_shape)(outputs)
outputs = tf.convert_to_tensor(outputs)
The expected outcome is that the output has shape of (A,?,B) instead of (?, A, B). Is there another method that I can have multiple denses in parralel with the above behaviour?
Simple example with A=3, B=1.
from keras import backend as K
from keras.layers import Concatenate, Dense, Input, Lambda
from keras.models import Model
import numpy as np
def expand_dims(x):
return K.expand_dims(x, axis=-2) #expand (None, 1) to (None, 1, 1)
x = Input((2,))
A = 3
B = 1
y = Lambda(expand_dims)(Dense(B, activation="softmax")(x))
for i in range(0, A-1):
# Concatenate on the newly added dimension
y = Concatenate(axis=-2)([y,Lambda(expand_dims)(Dense(B, activation="softmax")(x))])
model = Model(x, y)
print(model.predict(np.ones((4,2))).shape)
(4, 3, 1) # Output shape is (No_samples, A,B)
Related
I have an input that is a time series of 5 dimensions:
a = [[8,3],[2] , [4,5],[1], [9,1],[2]...] #total 100 timestamps. For each element, dims 0,1 are numerical data and dim 2 is a numerical encoding of a category. This is per sample, 3200 samples
The category has 3 possible values (0,1,2)
I want to build a NN such that the last dimension (the category) will go through an embedding layer with output size 8, and then will be concatenated back to the first two dims (the numerical data).
So, this will be something like:
input1 = keras.layers.Input(shape=(2,)) #the numerical features
input2 = keras.layers.Input(shape=(1,)) #the encoding of the categories. this part will be embedded to 5 dims
x2 = Embedding(input_dim=1, output_dim = 8)(input2) #apply it to every timestamp and take only dim 3, so [2],[1], [2]
x = concatenate([input1,x2]) #will get 10 dims at each timepoint, still 100 timepoints
x = LSTM(units=24)(x) #the input has 10 dims/features at each timepoint, total 100 timepoints per sample
x = Dense(1, activation='sigmoid')(x)
model = Model(inputs=[input1, input2] , outputs=[x]) #input1 is 1D vec of the width 2 , input2 is 1D vec with the width 1 and it is going through the embedding
model.compile(
loss='binary_crossentropy',
optimizer='adam',
metrics=['acc']
)
How can I do it? (preferably in keras)?
My problem is how to apply the embedding to every time point?
Meaning, if I have 1000 timepoints with 3 dims each, I need to convert it to 1000 timepoints with 8 dims each (The emebedding layer should transform input2 from (1000X1) to (1000X8)
There are a couple of issues you are having here.
First let me give you a working example and explain along the way how to solve your issues.
Imports and Data Generation
import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from tensorflow.keras.models import Model
num_timesteps = 100
max_features_values = [100, 100, 3]
num_observations = 2
input_list = [[[np.random.randint(0, v) for _ in range(num_timesteps)]
for v in max_features_values]
for _ in range(num_observations)]
input_arr = np.array(input_list) # shape (2, 3, 100)
In order to use an embedding we need to the voc_size as input_dimension, as stated in the LSTM documentation.
Embedding and Concatenation
voc_size = len(np.unique(input_arr[:, 2, :])) + 1 # 4
Now we need to create the inputs. Inputs should be of size [None, 2, num_timesteps] and [None, 1, num_timesteps] where the first dimension is the flexible and will be filled with the number of observations we are passing in. Let's use the embedding right after that using the previously calculated voc_size.
inp1 = layers.Input(shape=(2, num_timesteps)) # TensorShape([None, 2, 100])
inp2 = layers.Input(shape=(1, num_timesteps)) # TensorShape([None, 1, 100])
x2 = layers.Embedding(input_dim=voc_size, output_dim=8)(inp2) # TensorShape([None, 1, 100, 8])
x2_reshaped = tf.transpose(tf.squeeze(x2, axis=1), [0, 2, 1]) # TensorShape([None, 8, 100])
This cannot be easily concatenated since all dimensions must match except for the one along the concatenation axis. But the shapes are not matching unfortunately. Therefore we reshape x2. We do so by removing the first dimension and then transposing.
Now we can concatenate without any issue and everything works in a straight forward fashion:
x = layers.concatenate([inp1, x2_reshaped], axis=1)
x = layers.LSTM(32)(x)
x = layers.Dense(1, activation='sigmoid')(x)
model = Model(inputs=[inp1, inp2], outputs=[x])
Check on Dummy Example
inp1_np = input_arr[:, :2, :]
inp2_np = input_arr[:, 2:, :]
model.predict([inp1_np, inp2_np])
# Output
# array([[0.544262 ],
# [0.6157502]], dtype=float32)
#This outputs values between 0 and 1 just as expected.
In case you are not looking for Embeddings the way it's usually used in Keras (positive integers mapping to dense vectors). You might be looking for some sort of unprojection or basis expansion, in which 3 dimensions get mapped (embedded) to 8 and concatenating the result. This can be done using the kernel trick or other methods, but also happens implicitly in neural networks with non-linear applications.
As such, you can do something like this, following a similar format to pythonic833 because it was good (but with timestamps in the middle per the Keras LSTM documentation asking for [batch, timesteps, feature]):
Input generation
import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from tensorflow.keras.models import Model
num_timesteps = 100
num_features = 5
num_observations = 2
input_list = [[[np.random.randint(1, 100) for _ in range(num_features)]
for _ in range(num_timesteps)]
for _ in range(num_observations)]
input_arr = np.array(input_list) # shape (2, 100, 5)
Model construction
Then you can process the inputs:
input1 = layers.Input(shape=(num_timesteps, 2,))
input2 = layers.Input(shape=(num_timesteps, 3))
x2 = layers.Dense(8, activation='relu')(input2)
x = layers.concatenate([input1,x2], axis=2) # This produces tensors of shape (None, 100, 10)
x = layers.LSTM(units=24)(x)
x = layers.Dense(1, activation='sigmoid')(x)
model = Model(inputs=[input1, input2] , outputs=[x])
model.compile(
loss='binary_crossentropy',
optimizer='adam',
metrics=['acc']
)
Results
inp1_np = input_arr[:, :, :2]
inp2_np = input_arr[:, :, 2:]
model.predict([inp1_np, inp2_np])
which produces
array([[0.44117224],
[0.23611131]], dtype=float32)
Other explanations about basis expansion to check out:
https://stats.stackexchange.com/questions/527258/embedding-data-into-a-larger-dimension-space
https://www.reddit.com/r/MachineLearning/comments/2ffejw/why_dont_researchers_use_the_kernel_method_in/
Let's say I have a tensor of shape [3000,20,10], I'll call it input... I have another tensor of shape [3000] that contains only indices for the 3rd dimension of the first tensor, I'll call it "indices"
Basically, for every i in 0:2999, I want input[i,20,indices[i]]... So my final output will be of shape [3000,20]
I should be able to easily do this with a for loop, but I would like to find how to do this without a for loop... I tried looking at tf.gather but was honestly too confused to have any success in that direction
Any help would be greatly appreciated
Try this code:
from tensorflow.keras.layers import Conv2D, Input, BatchNormalization, Activation, Add, MaxPooling2D, LSTM, Dense
import tensorflow as tf
input = tf.range(3000*20*10)
input = tf.reshape(input, (3000,20,10))
indices = tf.random.uniform((3000,), 0, 10, dtype=tf.int32)
inds = indices[:, tf.newaxis, tf.newaxis]
inds = tf.repeat(inds, 20, 1)
output = tf.gather(input, inds, batch_dims=2, axis=2)
output = tf.squeeze(output)
import random
for _ in range(100):
i = random.randint(0, 2999)
j = random.randint(0, 19)
print(input[i][j][[indices[i]]]==output[i][j])
In follow-up to [this question], a few notes on what we're looking to accomplish:
We have two inputs X and Y of different sample sizes n and m, and a boolean vector z of size nm. Each element of z denotes whether two items in X and Y are some sort of match
We want to use embedding layers (perhaps the same, perhaps different) on X and Y before pairwise concatenating the output of these embedding layers to derive input to an output layer.
Here's one example of what a simple network could look like:
The linked answer and a few other resources helped get as far as this example, which builds a model but throws this error at fit time: ValueError: All input arrays (x) should have the same number of samples. Got array shapes: [(10, 2), (12, 2)].
Recent versions of Keras allow for skipping the dimension check, can this check be skipped in Tensorflow? I'd also be happy to use Keras, but I'm not sure how to perform the reshape and concatenate in Keras in the middle of a model.
Or, is this simply not possible in either? Is the only option to expand and pairwise concatenate prior to input?
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np
k = 2
N = 10
M = 12
x = np.random.randint(2, size = 2 * N).reshape((-1,2))
y = np.random.randint(2, size = 2 * M).reshape((-1,2))
x_rep = np.tile(x, (1, M)).reshape((-1,2))
y_rep = np.tile(y, (N, 1))
xy = np.concatenate((x_rep, y_rep), axis=1)
xy = xy.astype(bool)
z = (xy[:,0] == xy[:,2]) * (xy[:,1] ^ xy[:,3])
print(z[:20])
xy = xy.astype(int)
z = z.astype(int)
first = keras.Input(shape=(k,))
second = keras.Input(shape=(k,))
shared_dense = layers.Dense(k)
first_dense = shared_dense(first)
second_dense = shared_dense(second)
first_tiled = layers.Lambda(tf.tile, arguments={'multiples':[1, M]}, name='first_expanded' )(first_dense) #keras.backend.tile(first_dense, [1, M])
second_tiled = layers.Lambda(tf.tile, arguments={'multiples':[N,1]}, name='second_expanded')(second_dense) #keras.backend.tile(first_dense, [1, M])
first_reshaped = layers.Reshape((k,))(first_tiled)
concatenated = layers.Concatenate()([first_reshaped, second_tiled])
out = layers.Dense(1)(concatenated)
model = keras.Model([first, second], out)
keras.utils.plot_model(model, 'tf_nw.png', show_shapes=True)
model.compile('Adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit([x, y], z)
I have 2 inputs into a lambda layer one size (2,3,) the other (3,) . The lambda layer should return an output of size 2, however when the multiply layer is executed the following error occurs:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimensions must be equal, but are 2 and 3 for 'multiply_1/mul' (op: 'Mul') with input shapes: [?,2], [?,3].
The relevant code is below and any help would be much appreciated,thanks:
import numpy as np
from keras.models import Model
from keras import backend as K
from keras.engine.topology import Layer
from keras.layers import Dense,Input,concatenate,Lambda,multiply,add
import tensorflow as tf
import time
def weights_Fx(x):
j = x[0][:,0]
k = x[1][0]
y = j - k
return y
def sum_layer(x):
x = tf.reduce_sum(x)
return x
type1_2 = Dense(units=1, activation = 'relu',name = "one")
type1_3 = Dense(units=1,activation = 'relu',name = "two")
in1 = Input(shape=(1,))
in2 = Input(shape=(1,))
n1 = type1_2(in1)
n2 = type1_3(in2)
model = concatenate([n1,n2],axis=-1,name='merge_predicitions')
coords_in = Input(shape=(2,3,))
coords_target = Input(shape=(3,))
model2 = Lambda(weights_Fx,output_shape=(2,),name='weightsFx')([coords_in,coords_target])
model = multiply([model,model2])
model = Lambda(sum_layer)(model)
model = Model(inputs=[in1,in2,coords_in,coords_target],outputs=[model])
The issue was to do with how I was indexing the array. It is important to remember that though the data is of shape (2,3) keras will create a Tensor of shape (None,2,3) therefore to perform the operation as desired the following is needed:
y = x[0][:,:,0]-x[1][:,0]
Furthermore in "sum layer" in order to prevent the rank (number of dimensions) in the tensor being reduced by 1 the following is required:
y = K.sum(x,axis=1,keepdims=True)
Your Lambda layer is not returning output of shape 2 but it is returning output of shape 3.
Model2 shape is (,3) and not (,2) which is giving error in multiply of the model and model2
Take a look at your coords_in and coords_target shape.
I am trying to use the Keras.backend ops to write a function that I will wrap as a Lambda to use in my model.
There are two tensors, X and Y. X is not trainable. Y is trainable.
The python function that is wrapped is:
import keras.backend as K
from keras.activations import softmax
def _attention(inputs):
X, Y = inputs
attention_weight = K.dot(X, K.expand_dims(Y))
attention_weight = K.squeeze(attention_weight, axis=-1)
attention_weight = softmax(attention_weight, axis=-1)
return attention_weight
which I wanted to wrap as:
Y = K.random_normal_variable(shape=(200,), mean=0.0, scale=1.0)
attend = Lambda(_attention)
attention = attend((X,Y))
When I call:
model = Model(inputs=[input], outputs=[attention])
I receive the message
ValueError: Output tensors to a Model must be the output of a TensorFlowLayer(thus holding past layer metadata). Found: Tensor("lambda_2/Softmax:0", shape=(?, ?), dtype=float32)
Do I really need to make a custom layer for the expand_dims, dot product, and squeeze method? I know I could always reshape Y from (dim,) -> (dim,1) but I am still stuck with the squeeze.