The full error message is like this:
ValueError: Shapes (2, 1) and (50, 1) are incompatible
It occurs when my model is trained. The mistake either is in my input_fn:
train_input_fn = tf.estimator.inputs.numpy_input_fn(
x = {"x" : training_data},
y = training_labels,
batch_size = 50,
num_epochs = None,
shuffle = True)
in my logits and loss function:
dense = tf.layers.dense(inputs = pool2_flat, units = 1024, activation = tf.nn.relu)
dropout = tf.layers.dropout(inputs = dense, rate = 0.4, training = mode == tf.estimator.ModeKeys.TRAIN)
logits = tf.layers.dense(inputs = dropout, units = 1)
loss = tf.losses.softmax_cross_entropy(labels = labels, logits = logits)
or in my dataset. I can only print out the shape of my dataset for you to take a look at it.
#shape of the dataset
train_data.shape
(1196,2,1)
train_data[0].shape
(2,1)
#this is the data
train_data[0][0].shape
(1,)
train_data[0][0][0].shape
(20,50,50)
#this is the labels
train_data[0][1].shape
(1,)
The problem seems to be the shape of the logits. They are supposed to be [batch_size, num_classes] in this case [50,1] but are [2,1]. The shape of the labels is correctly [50,1]
I have made a github gist if you want to take a look at the whole code.
https://gist.github.com/hjkhjk1999/38f358a53da84a94bf5a59f44050aad5
In your code, you are stating that the inputs to your model will be feed in batches of 50 samples per batch with one variable. But it looks like your are feeding actually a batch of 2 samples with 1 variable (shape=[2, 1]) despite feeding labels with shape [50, 1].
That's the problem, you are giving 50 'questions' and two 'answers'.
Also, your dataset is shaped in a really weird way. I see you named your github gist 3D Conv. If you are indeed trying to do a 3D convolution you might want to reshape your dataset into a tensor (numpy array) of this shape shape = [samples, width, height, deepth]
Related
I'm trying to reshape a Tensorflow model's input along the batch dimension. I want to combine some of the batch samples into a time-series so I can feed it into an LSTM layer.
Specifically, I have 1024 samples and I'd like to put them into groups of 64 timesteps with the result being 16 batches of 64 timesteps, each timestep having the original 24 features.
#input tensor is (1024, 24)
inputLayer = Input(shape=(24,))
#I want it to be (16, 64, 24)
reshapedLayer = layers.Reshape([64, 24])(inputLayer)
lstmLayer = layers.LSTM(128, activation='relu')(reshapedLayer)
This compiles but throws a runtime error
tensorflow.python.framework.errors_impl.InvalidArgumentError:
Input to reshape is a tensor with 24576 values, but the requested shape has 1572864
I understand what the error is telling me, but I'm not sure the right way to go about fixing it.
Perhaps this could work for you:
import tensorflow as tf
inputs = tf.keras.layers.Input(shape=(24,))
x = tf.reshape(inputs, (16, 64, 24))
x = tf.keras.layers.LSTM(128, activation='relu')(x)
model = tf.keras.Model(inputs=inputs, outputs=x)
# dummy data
inputs = tf.random.uniform(shape=(1024, 24))
outputs = model(inputs)
Replacing the Reshape layer with tf.reshape.
I'm a newbie to Keras. I'm playing around Keras to get some intuition and stuck with here.
input_image = tf.keras.Input(shape=(16,16,3))
x = tf.keras.layers.Conv2D(32,(3,3), padding = 'same')(input_image)
model = tf.keras.Model(input_image , x)
model.compile(optimizer='Adam',loss = 'MSE')
inputs = np.random.normal(size = (16,16,3))
outputs = np.random.normal(size = (16,16,32))
model.fit(x = inputs , y =outputs)
I just wanted to see the output shape that model.summary says (None, 16, 16, 32). But now I have two questions. One is the output shape and another is why my code doesn't work. I hope someone tells me what I'm missing. Thanks~
inputs = np.random.normal(size = (1,16,16,3)) #<---- here
outputs = np.random.normal(size = (1,16,16,32)) #<---here
They should be 4D not 3D in shape. You need to give the detail of batch also.
(batch_size, w,h,c) <---- 4D
You are missing batch_size
32,(3,3) from tf.keras.layers.Conv2D(32,(3,3), padding = 'same')(input_image)
You have 32 filters. So the channel depth will be 32. But since you have used the padding='same' so your output will have the same dimension as input. Only differ in depth.
I am facing some problems in training the following GRU model, which has to be stateful and output the hidden state.
import numpy as np
import tensorflow as tf #2.1.0
from tensorflow import keras
BATCH_SIZE = 1
nfeatures = 3
history = 30 # shapes input array
horizon = 5 # shapes output array
nodes = 32
input_layer = tf.keras.layers.Input(batch_shape=(1,30,3),name="INPUT")
output, state_h = tf.keras.layers.GRU(nodes,
return_sequences=True,
stateful=True,
return_state=True,
batch_input_shape=(1,history,3), name='GRU1')(input_layer)
output_layer = tf.keras.layers.GRU(nodes, activation='tanh', name='GRU2')(output, state_h)
output_dense = tf.keras.layers.Dense(5, name='DENSE')(output_layer)
model = tf.keras.Model(input_layer, [output_dense, state_h])
model.compile(optimizer=tf.keras.optimizers.Adam(clipvalue=2.0),
loss='mse',
metrics=['mean_absolute_error', 'mean_squared_error'])
As I need the model to output the hidden state, I do not use a Sequential model. (I had no problems training a stateful sequential model.)
The features fed to network are of shape np.shape(x)=(30,3) and the target np.shape(y)=(5,).
If I call model.predict(x), where x is a numpy array with the shape mentioned above, it throws an error, as expected, because the input shape doesn't match the expected input. Therefore, I reshape the input array to have an input shape of (1,30,3) by calling np.expand_dims(x,axis=0). After that, it works fine, i.e. I get an output.
The issues I am facing are when I try to train the model. Calling
model.fit(x, y,epochs=1,steps_per_epoch=STEPS_PER_EPOCH)
throws the same error, about the shape of the data
ValueError: Error when checking input: expected input to have 3 dimensions, but got array with shape (30, 3)
Reshapping the data as I did for the prediction didn't help
model.fit(np.expand_dims(x,axis=0), np.expand_dims(y,axis=0),epochs=1,steps_per_epoch=STEPS_PER_EPOCH)
ValueError: The number of samples 1 is not divisible by steps 30. Please change the number of steps to a value that can consume all the samples.
This was a new error, setting the steps_per_epoch=1 threw a new one
ValueError: Error when checking model target: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), for inputs ['DENSE', 'GRU1'] but instead got the following list of 1 arrays: [array([[0.5124772 , 0.51047856, 0.509669 , 0.50830126, 0.5070507 ]],
dtype=float32)]...
Is the format of my data wrong or is the architecture of my layers missing something? I tried adding a Flatten layer after the input, but it didn't make much sense (in my head) and it didn't work either.
Thanks in advance.
Problem here is that the Number of Nodes should be equal to the Output Shape. Changing the value of Nodes from 32 to 5, along with other minor changes, will fix the Error.
Complete working code is shown below:
import numpy as np
import tensorflow as tf #2.1.0
from tensorflow import keras
BATCH_SIZE = 1
nfeatures = 3
history = 30 # shapes input array
horizon = 5 # shapes output array
nodes = 5
x = np.ones(shape = (30,3))
x = np.expand_dims(x, axis = 0)
y = np.ones(shape = (5,))
y = np.expand_dims(y, axis = 0)
print(x.shape) #(1, 30, 3)
print(y.shape) #(1, 5)
input_layer = tf.keras.layers.Input(batch_shape=(1,30,3),name="INPUT")
output, state_h = tf.keras.layers.GRU(nodes,
return_sequences=True,
stateful=True,
return_state=True,
batch_input_shape=(1,history,3), name='GRU1')(input_layer)
output_layer = tf.keras.layers.GRU(nodes, activation='tanh', name='GRU2')(output, state_h)
output_dense = tf.keras.layers.Dense(5, name='DENSE')(output_layer)
model = tf.keras.Model(input_layer, [output_dense, state_h])
model.compile(optimizer=tf.keras.optimizers.Adam(clipvalue=2.0),
loss='mse',
metrics=['mean_absolute_error', 'mean_squared_error'])
STEPS_PER_EPOCH = 1
model.fit(x, y,epochs=1,steps_per_epoch=STEPS_PER_EPOCH)
Output of the above code is:
(1, 30, 3)
(1, 5)
1/1 [==============================] - 0s 3ms/step - loss: 1.8172 - DENSE_loss: 1.1737 - GRU1_loss: 0.6435 - DENSE_mean_absolute_error: 1.0498 - DENSE_mean_squared_error: 1.1737 - GRU1_mean_absolute_error: 0.7157 - GRU1_mean_squared_error: 0.6435
<tensorflow.python.keras.callbacks.History at 0x7f698bf8ac50>
Hope this helps. Happy Learning!
Here is what I have tried:
tf.reset_default_graph()
X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])
y = tf.placeholder(tf.float32, [None,n_outputs])
layers = [tf.contrib.rnn.LSTMCell(num_units=n_neurons,
activation=tf.nn.leaky_relu, use_peepholes = True)
for layer in range(n_layers)]
multi_layer_cell = tf.contrib.rnn.MultiRNNCell(layers)
rnn_outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)
tf.summary.histogram("outputs", rnn_outputs)
tf.summary.image("RNN",rnn_outputs)
I am getting the following error:
InvalidArgumentError: Tensor must be 4-D with last dim 1, 3, or 4, not [55413,4,100]
[[Node: RNN_1 = ImageSummary[T=DT_FLOAT, bad_color=Tensor<type: uint8 shape: [4] values: 255 0 0...>, max_images=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](RNN_1/tag, rnn/transpose_1)]]
Kindly, help me get the visualization of the rnn inside the LSTM model that I am trying to run. This will help in understanding what LSTM is doing more accurately.
You can plot each RNN output as an image with one axis being the time and the other axis being the output. Here is an small example:
import tensorflow as tf
import numpy as np
n_steps = 100
n_inputs = 10
n_neurons = 10
n_layers = 3
x = tf.placeholder(tf.float32, [None, n_steps, n_inputs])
layers = [tf.contrib.rnn.LSTMCell(num_units=n_neurons,
activation=tf.nn.leaky_relu, use_peepholes=True)
for layer in range(n_layers)]
multi_layer_cell = tf.contrib.rnn.MultiRNNCell(layers)
rnn_outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, x, dtype=tf.float32)
# Time steps in horizontal axis, outputs in vertical axis, add last dimension for channel
rnn_out_imgs = tf.transpose(rnn_outputs, (0, 2, 1))[..., tf.newaxis]
out_img_sum = tf.summary.image("RNN", rnn_out_imgs, max_outputs=10)
init_op = tf.global_variables_initializer()
with tf.Session() as sess, tf.summary.FileWriter('log') as fw:
sess.run(init_op)
fw.add_summary(sess.run(out_img_sum, feed_dict={x: np.random.rand(10, n_steps, n_inputs)}))
You would get a visualization that could look like this:
Here the brighter pixels would represent a stronger activation, so even if it is hard to tell what exactly is causing what you can at least see if any meaningful patterns arise.
Your RNN output has the wrong shape for tf.summary.image. The tensor should be four-dimensional with the dimensions' sizes given by [batch_size, height, width, channels].
In your code, you're calling tf.summary.image with rnn_outputs, which has shape [55413, 4, 100]. Assuming your images are 55413-by-100 pixels in size and that each pixel contains 4 channels (RGBA), I'd use tf.reshape to reshape rnn_outputs to [1, 55413, 100, 4]. Then you should be able to call tf.summary.image without error.
I don't think I can help you visualize the RNN's operation, but when I was learning about RNNs and LSTMs, I found this article very helpful.
I have two datasets, which is like:
input:
array([[[ 0.99309823],
...
[ 0. ]]])
shape : (1, 2501)
output:
array([[0, 0, 0, ..., 0, 0, 1],
...,
[0, 0, 0, ..., 0, 0, 0]])
shape : (2501, 9)
And I processed it with TFLearn; as
input_layer = tflearn.input_data(shape=[None,2501])
hidden1 = tflearn.fully_connected(input_layer,1205,activation='ReLU', regularizer='L2', weight_decay=0.001)
dropout1 = tflearn.dropout(hidden1,0.8)
hidden2 = tflearn.fully_connected(dropout1,1205,activation='ReLU', regularizer='L2', weight_decay=0.001)
dropout2 = tflearn.dropout(hidden2,0.8)
softmax = tflearn.fully_connected(dropout2,9,activation='softmax')
# Regression with SGD
sgd = tflearn.SGD(learning_rate=0.1,lr_decay=0.96, decay_step=1000)
top_k=tflearn.metrics.Top_k(3)
net = tflearn.regression(softmax,optimizer=sgd,metric=top_k,loss='categorical_crossentropy')
model = tflearn.DNN(net)
model.fit(input,output,n_epoch=10,show_metric=True, run_id='dense_model')
It works but not the way that I want. It's a DNN model. I want that when I enter 0.95, model must give me corresponding prediction for example [0,0,0,0,0,0,0,0,1]. However, when I want to enter 0.95, it says that,
ValueError: Cannot feed value of shape (1,) for Tensor 'InputData/X:0', which has shape '(?, 2501)'
When I tried to understand I realise that I need (1,2501) shaped data to predict for my wrong based model.
What i want is for every element in input, predict corresponding element in output. As you can see, in the instance dataset,
for [0.99309823], corresponding output is [0,0,0,0,0,0,0,0,1]. I want tflearn to train itself like this.
I may have wrong structured data, or model(probably dataset), I explained all the things, I need help I'm really out of my mind.
Your input data should be Nx1 (N = number of samples) dimensional to archive this transformation ([0.99309823] --> [0,0,0,0,0,0,0,0,1] ). According to your input data shape, it looks more likely including 1 sample with 2501 dimensions.
ValueError: Cannot feed value of shape (1,) for Tensor 'InputData/X:0', which has shape '(?, 2501)' This error means that tensorflow expecting you to provide a vector with shape (,2501), but you are feeding the network with a vector with shape (1,).
Example modified code with dummy data:
import numpy as np
import tflearn
#creating dummy data
input_data = np.random.rand(1, 2501)
input_data = np.transpose(input_data) # now shape is (2501,1)
output_data = np.random.randint(8, size=2501)
n_values = 9
output_data = np.eye(n_values)[output_data]
# checking the shapes
print input_data.shape #(2501,1)
print output_data.shape #(2501,9)
input_layer = tflearn.input_data(shape=[None,1]) # now network is expecting ( Nx1 )
hidden1 = tflearn.fully_connected(input_layer,1205,activation='ReLU', regularizer='L2', weight_decay=0.001)
dropout1 = tflearn.dropout(hidden1,0.8)
hidden2 = tflearn.fully_connected(dropout1,1205,activation='ReLU', regularizer='L2', weight_decay=0.001)
dropout2 = tflearn.dropout(hidden2,0.8)
softmax = tflearn.fully_connected(dropout2,9,activation='softmax')
# Regression with SGD
sgd = tflearn.SGD(learning_rate=0.1,lr_decay=0.96, decay_step=1000)
top_k=tflearn.metrics.Top_k(3)
net = tflearn.regression(softmax,optimizer=sgd,metric=top_k,loss='categorical_crossentropy')
model = tflearn.DNN(net)
model.fit(input_data, output_data, n_epoch=10,show_metric=True, run_id='dense_model')
Also my friend warned me about same thing as rcmalli. He says
reshape:
input = tf.reshape(input, (2501,1))
change
input_layer = tflearn.input_data(shape=[None,2501])
to
input_layer = tflearn.input_data(shape=[None, 1])
Variable dimension must be "None". In your wrong case, 2501 is the magnitude(or something else, I translated from another lang., but you got it) of your dataset. 1 is constant input magnitude.