I'm a newbie to Keras. I'm playing around Keras to get some intuition and stuck with here.
input_image = tf.keras.Input(shape=(16,16,3))
x = tf.keras.layers.Conv2D(32,(3,3), padding = 'same')(input_image)
model = tf.keras.Model(input_image , x)
model.compile(optimizer='Adam',loss = 'MSE')
inputs = np.random.normal(size = (16,16,3))
outputs = np.random.normal(size = (16,16,32))
model.fit(x = inputs , y =outputs)
I just wanted to see the output shape that model.summary says (None, 16, 16, 32). But now I have two questions. One is the output shape and another is why my code doesn't work. I hope someone tells me what I'm missing. Thanks~
inputs = np.random.normal(size = (1,16,16,3)) #<---- here
outputs = np.random.normal(size = (1,16,16,32)) #<---here
They should be 4D not 3D in shape. You need to give the detail of batch also.
(batch_size, w,h,c) <---- 4D
You are missing batch_size
32,(3,3) from tf.keras.layers.Conv2D(32,(3,3), padding = 'same')(input_image)
You have 32 filters. So the channel depth will be 32. But since you have used the padding='same' so your output will have the same dimension as input. Only differ in depth.
Related
I have an input that is a time series of 5 dimensions:
a = [[8,3],[2] , [4,5],[1], [9,1],[2]...] #total 100 timestamps. For each element, dims 0,1 are numerical data and dim 2 is a numerical encoding of a category. This is per sample, 3200 samples
The category has 3 possible values (0,1,2)
I want to build a NN such that the last dimension (the category) will go through an embedding layer with output size 8, and then will be concatenated back to the first two dims (the numerical data).
So, this will be something like:
input1 = keras.layers.Input(shape=(2,)) #the numerical features
input2 = keras.layers.Input(shape=(1,)) #the encoding of the categories. this part will be embedded to 5 dims
x2 = Embedding(input_dim=1, output_dim = 8)(input2) #apply it to every timestamp and take only dim 3, so [2],[1], [2]
x = concatenate([input1,x2]) #will get 10 dims at each timepoint, still 100 timepoints
x = LSTM(units=24)(x) #the input has 10 dims/features at each timepoint, total 100 timepoints per sample
x = Dense(1, activation='sigmoid')(x)
model = Model(inputs=[input1, input2] , outputs=[x]) #input1 is 1D vec of the width 2 , input2 is 1D vec with the width 1 and it is going through the embedding
model.compile(
loss='binary_crossentropy',
optimizer='adam',
metrics=['acc']
)
How can I do it? (preferably in keras)?
My problem is how to apply the embedding to every time point?
Meaning, if I have 1000 timepoints with 3 dims each, I need to convert it to 1000 timepoints with 8 dims each (The emebedding layer should transform input2 from (1000X1) to (1000X8)
There are a couple of issues you are having here.
First let me give you a working example and explain along the way how to solve your issues.
Imports and Data Generation
import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from tensorflow.keras.models import Model
num_timesteps = 100
max_features_values = [100, 100, 3]
num_observations = 2
input_list = [[[np.random.randint(0, v) for _ in range(num_timesteps)]
for v in max_features_values]
for _ in range(num_observations)]
input_arr = np.array(input_list) # shape (2, 3, 100)
In order to use an embedding we need to the voc_size as input_dimension, as stated in the LSTM documentation.
Embedding and Concatenation
voc_size = len(np.unique(input_arr[:, 2, :])) + 1 # 4
Now we need to create the inputs. Inputs should be of size [None, 2, num_timesteps] and [None, 1, num_timesteps] where the first dimension is the flexible and will be filled with the number of observations we are passing in. Let's use the embedding right after that using the previously calculated voc_size.
inp1 = layers.Input(shape=(2, num_timesteps)) # TensorShape([None, 2, 100])
inp2 = layers.Input(shape=(1, num_timesteps)) # TensorShape([None, 1, 100])
x2 = layers.Embedding(input_dim=voc_size, output_dim=8)(inp2) # TensorShape([None, 1, 100, 8])
x2_reshaped = tf.transpose(tf.squeeze(x2, axis=1), [0, 2, 1]) # TensorShape([None, 8, 100])
This cannot be easily concatenated since all dimensions must match except for the one along the concatenation axis. But the shapes are not matching unfortunately. Therefore we reshape x2. We do so by removing the first dimension and then transposing.
Now we can concatenate without any issue and everything works in a straight forward fashion:
x = layers.concatenate([inp1, x2_reshaped], axis=1)
x = layers.LSTM(32)(x)
x = layers.Dense(1, activation='sigmoid')(x)
model = Model(inputs=[inp1, inp2], outputs=[x])
Check on Dummy Example
inp1_np = input_arr[:, :2, :]
inp2_np = input_arr[:, 2:, :]
model.predict([inp1_np, inp2_np])
# Output
# array([[0.544262 ],
# [0.6157502]], dtype=float32)
#This outputs values between 0 and 1 just as expected.
In case you are not looking for Embeddings the way it's usually used in Keras (positive integers mapping to dense vectors). You might be looking for some sort of unprojection or basis expansion, in which 3 dimensions get mapped (embedded) to 8 and concatenating the result. This can be done using the kernel trick or other methods, but also happens implicitly in neural networks with non-linear applications.
As such, you can do something like this, following a similar format to pythonic833 because it was good (but with timestamps in the middle per the Keras LSTM documentation asking for [batch, timesteps, feature]):
Input generation
import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from tensorflow.keras.models import Model
num_timesteps = 100
num_features = 5
num_observations = 2
input_list = [[[np.random.randint(1, 100) for _ in range(num_features)]
for _ in range(num_timesteps)]
for _ in range(num_observations)]
input_arr = np.array(input_list) # shape (2, 100, 5)
Model construction
Then you can process the inputs:
input1 = layers.Input(shape=(num_timesteps, 2,))
input2 = layers.Input(shape=(num_timesteps, 3))
x2 = layers.Dense(8, activation='relu')(input2)
x = layers.concatenate([input1,x2], axis=2) # This produces tensors of shape (None, 100, 10)
x = layers.LSTM(units=24)(x)
x = layers.Dense(1, activation='sigmoid')(x)
model = Model(inputs=[input1, input2] , outputs=[x])
model.compile(
loss='binary_crossentropy',
optimizer='adam',
metrics=['acc']
)
Results
inp1_np = input_arr[:, :, :2]
inp2_np = input_arr[:, :, 2:]
model.predict([inp1_np, inp2_np])
which produces
array([[0.44117224],
[0.23611131]], dtype=float32)
Other explanations about basis expansion to check out:
https://stats.stackexchange.com/questions/527258/embedding-data-into-a-larger-dimension-space
https://www.reddit.com/r/MachineLearning/comments/2ffejw/why_dont_researchers_use_the_kernel_method_in/
I'm trying to reshape a Tensorflow model's input along the batch dimension. I want to combine some of the batch samples into a time-series so I can feed it into an LSTM layer.
Specifically, I have 1024 samples and I'd like to put them into groups of 64 timesteps with the result being 16 batches of 64 timesteps, each timestep having the original 24 features.
#input tensor is (1024, 24)
inputLayer = Input(shape=(24,))
#I want it to be (16, 64, 24)
reshapedLayer = layers.Reshape([64, 24])(inputLayer)
lstmLayer = layers.LSTM(128, activation='relu')(reshapedLayer)
This compiles but throws a runtime error
tensorflow.python.framework.errors_impl.InvalidArgumentError:
Input to reshape is a tensor with 24576 values, but the requested shape has 1572864
I understand what the error is telling me, but I'm not sure the right way to go about fixing it.
Perhaps this could work for you:
import tensorflow as tf
inputs = tf.keras.layers.Input(shape=(24,))
x = tf.reshape(inputs, (16, 64, 24))
x = tf.keras.layers.LSTM(128, activation='relu')(x)
model = tf.keras.Model(inputs=inputs, outputs=x)
# dummy data
inputs = tf.random.uniform(shape=(1024, 24))
outputs = model(inputs)
Replacing the Reshape layer with tf.reshape.
I am facing some problems in training the following GRU model, which has to be stateful and output the hidden state.
import numpy as np
import tensorflow as tf #2.1.0
from tensorflow import keras
BATCH_SIZE = 1
nfeatures = 3
history = 30 # shapes input array
horizon = 5 # shapes output array
nodes = 32
input_layer = tf.keras.layers.Input(batch_shape=(1,30,3),name="INPUT")
output, state_h = tf.keras.layers.GRU(nodes,
return_sequences=True,
stateful=True,
return_state=True,
batch_input_shape=(1,history,3), name='GRU1')(input_layer)
output_layer = tf.keras.layers.GRU(nodes, activation='tanh', name='GRU2')(output, state_h)
output_dense = tf.keras.layers.Dense(5, name='DENSE')(output_layer)
model = tf.keras.Model(input_layer, [output_dense, state_h])
model.compile(optimizer=tf.keras.optimizers.Adam(clipvalue=2.0),
loss='mse',
metrics=['mean_absolute_error', 'mean_squared_error'])
As I need the model to output the hidden state, I do not use a Sequential model. (I had no problems training a stateful sequential model.)
The features fed to network are of shape np.shape(x)=(30,3) and the target np.shape(y)=(5,).
If I call model.predict(x), where x is a numpy array with the shape mentioned above, it throws an error, as expected, because the input shape doesn't match the expected input. Therefore, I reshape the input array to have an input shape of (1,30,3) by calling np.expand_dims(x,axis=0). After that, it works fine, i.e. I get an output.
The issues I am facing are when I try to train the model. Calling
model.fit(x, y,epochs=1,steps_per_epoch=STEPS_PER_EPOCH)
throws the same error, about the shape of the data
ValueError: Error when checking input: expected input to have 3 dimensions, but got array with shape (30, 3)
Reshapping the data as I did for the prediction didn't help
model.fit(np.expand_dims(x,axis=0), np.expand_dims(y,axis=0),epochs=1,steps_per_epoch=STEPS_PER_EPOCH)
ValueError: The number of samples 1 is not divisible by steps 30. Please change the number of steps to a value that can consume all the samples.
This was a new error, setting the steps_per_epoch=1 threw a new one
ValueError: Error when checking model target: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), for inputs ['DENSE', 'GRU1'] but instead got the following list of 1 arrays: [array([[0.5124772 , 0.51047856, 0.509669 , 0.50830126, 0.5070507 ]],
dtype=float32)]...
Is the format of my data wrong or is the architecture of my layers missing something? I tried adding a Flatten layer after the input, but it didn't make much sense (in my head) and it didn't work either.
Thanks in advance.
Problem here is that the Number of Nodes should be equal to the Output Shape. Changing the value of Nodes from 32 to 5, along with other minor changes, will fix the Error.
Complete working code is shown below:
import numpy as np
import tensorflow as tf #2.1.0
from tensorflow import keras
BATCH_SIZE = 1
nfeatures = 3
history = 30 # shapes input array
horizon = 5 # shapes output array
nodes = 5
x = np.ones(shape = (30,3))
x = np.expand_dims(x, axis = 0)
y = np.ones(shape = (5,))
y = np.expand_dims(y, axis = 0)
print(x.shape) #(1, 30, 3)
print(y.shape) #(1, 5)
input_layer = tf.keras.layers.Input(batch_shape=(1,30,3),name="INPUT")
output, state_h = tf.keras.layers.GRU(nodes,
return_sequences=True,
stateful=True,
return_state=True,
batch_input_shape=(1,history,3), name='GRU1')(input_layer)
output_layer = tf.keras.layers.GRU(nodes, activation='tanh', name='GRU2')(output, state_h)
output_dense = tf.keras.layers.Dense(5, name='DENSE')(output_layer)
model = tf.keras.Model(input_layer, [output_dense, state_h])
model.compile(optimizer=tf.keras.optimizers.Adam(clipvalue=2.0),
loss='mse',
metrics=['mean_absolute_error', 'mean_squared_error'])
STEPS_PER_EPOCH = 1
model.fit(x, y,epochs=1,steps_per_epoch=STEPS_PER_EPOCH)
Output of the above code is:
(1, 30, 3)
(1, 5)
1/1 [==============================] - 0s 3ms/step - loss: 1.8172 - DENSE_loss: 1.1737 - GRU1_loss: 0.6435 - DENSE_mean_absolute_error: 1.0498 - DENSE_mean_squared_error: 1.1737 - GRU1_mean_absolute_error: 0.7157 - GRU1_mean_squared_error: 0.6435
<tensorflow.python.keras.callbacks.History at 0x7f698bf8ac50>
Hope this helps. Happy Learning!
I'm following the TensorFlow Keras tutorial for text generation. The training part works perfectly, but when I try to predict the next token, I get an error.
Here's all the important code:
Making the vocabulary and dataset.
vocab = sorted(set(text))
char2index = { c:i for i, c in enumerate(vocab) }
index2char = np.array(vocab)
chars_to_int = np.array([char2index[c] for c in text])
char_dataset = tf.data.Dataset.from_tensor_slices(chars_to_int)
sequences = char_dataset.batch(seq_length + 1, drop_remainder=True)
def split_input_and_target(sequence):
input_ = sequence[:-1]
target_ = sequence[1:]
return input_, target_
dataset = sequences.map(split_input_and_target)
dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)
Building the model
(important part here is that BATCH_SIZE = 64):
model = tf.keras.Sequential()
model.add(tf.keras.layers.Embedding(len(vocab), EMBEDDING_DIM,
batch_input_shape=[BATCH_SIZE, None]))
# here are a few more layers
model.compile(loss="sparse_categorical_crossentropy", optimizer="adam")
model.fit(dataset, epochs=EPOCHS)
Actually trying to generate text (this one was copied almost directly from the tutorial after I started getting desperate):
num_tokens = 100
seed = "some text"
input_eval = [char2index[c] for c in seed]
input_eval = tf.expand_dims(input_eval, 0)
text_generated = []
model.reset_states()
for i in range(num_tokens):
predictions = model(input_eval)
predictions = tf.squeeze(predictions, 0)
# more stuff
Then, I first get a warning:
WARNING:tensorflow:Model was constructed with shape (64, None) for input Tensor("embedding_14_input:0", shape=(64, None), dtype=float32), but it was called on an input with incompatible shape (1, 9).
Then it gives me an error:
---->3 predictions = model(input_eval)
...
ValueError: Tensor's shape (9, 64, 256) is not compatible with supplied shape [9, 1, 256]
The second number, 64, is my batch size. If I change BATCH_SIZE to 1, everything works and all is fine, but this is obviously not the solution I am hoping for.
(I somehow managed to miss a step in the tutorial despite reading it several times over the past few hours.)
Here's the relevant passage:
To keep this prediction step simple, use a batch size of 1.
Because of the way the RNN state is passed from timestep to timestep, the model only accepts a fixed batch size once built.
To run the model with a different batch_size, we need to rebuild the model and restore the weights from the checkpoint.
tf.train.latest_checkpoint(checkpoint_dir)
model = build_model(vocab_size, embedding_dim, rnn_units, batch_size=1)
model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))
model.build(tf.TensorShape([1, None]))
I hope my silly mistake will help somebody to remember to reload the model in the future!
The full error message is like this:
ValueError: Shapes (2, 1) and (50, 1) are incompatible
It occurs when my model is trained. The mistake either is in my input_fn:
train_input_fn = tf.estimator.inputs.numpy_input_fn(
x = {"x" : training_data},
y = training_labels,
batch_size = 50,
num_epochs = None,
shuffle = True)
in my logits and loss function:
dense = tf.layers.dense(inputs = pool2_flat, units = 1024, activation = tf.nn.relu)
dropout = tf.layers.dropout(inputs = dense, rate = 0.4, training = mode == tf.estimator.ModeKeys.TRAIN)
logits = tf.layers.dense(inputs = dropout, units = 1)
loss = tf.losses.softmax_cross_entropy(labels = labels, logits = logits)
or in my dataset. I can only print out the shape of my dataset for you to take a look at it.
#shape of the dataset
train_data.shape
(1196,2,1)
train_data[0].shape
(2,1)
#this is the data
train_data[0][0].shape
(1,)
train_data[0][0][0].shape
(20,50,50)
#this is the labels
train_data[0][1].shape
(1,)
The problem seems to be the shape of the logits. They are supposed to be [batch_size, num_classes] in this case [50,1] but are [2,1]. The shape of the labels is correctly [50,1]
I have made a github gist if you want to take a look at the whole code.
https://gist.github.com/hjkhjk1999/38f358a53da84a94bf5a59f44050aad5
In your code, you are stating that the inputs to your model will be feed in batches of 50 samples per batch with one variable. But it looks like your are feeding actually a batch of 2 samples with 1 variable (shape=[2, 1]) despite feeding labels with shape [50, 1].
That's the problem, you are giving 50 'questions' and two 'answers'.
Also, your dataset is shaped in a really weird way. I see you named your github gist 3D Conv. If you are indeed trying to do a 3D convolution you might want to reshape your dataset into a tensor (numpy array) of this shape shape = [samples, width, height, deepth]