I have an implementation of a Many-to-one RNN with variable sequence length (a sentence classification problem)
I am trying to implement a sampled softmax loss since I have 500 classes and want to speed up the training.
The following are my input parameters shapes
WLast.shape
TensorShape([Dimension(500), Dimension(500)])
bLast.shape
TensorShape([Dimension(500)])
labels.shape
TensorShape([Dimension(None), Dimension(500)])
pred_out.shape
TensorShape([Dimension(None), Dimension(500)])
Pred_out is the last prediction from the RNN
Problem is that when I run:
cost = tf.nn.sampled_softmax_loss(WLast,bLast,labels,pred_out,10,500)
it gives me this error:
InvalidArgumentError: Dimension must be 1 but is 500 for 'sampled_softmax_loss/ComputeAccidentalHits' (op: 'ComputeAccidentalHits') with input shapes: [?,500], [10].
I don't understand, the shapes matches the arguments of the function, does someone know what could I be doing wrong?
Thanks in advance!
I found this implementation: https://github.com/olirice/sampled_softmax_loss
and solved the problem by reshaping the labels
labels = tf.reshape(tf.argmax(labels, 1), [-1,1])
Related
I have made a neural network for a final project but when I run it, gives me "ValueError: Error when checking target: expected dense_3 to have shape (6,) but got array with shape (1,)" I am brand new to coding and very lost. Have attached the link for Colabs. Thank you:)
[https://drive.google.com/file/d/1dcUuTVVDGwxHn2O5qqJk0wgiEf83MslN/view?usp=sharing][1]
Dataset: https://www.kaggle.com/camnugent/california-housing-prices
So I have changed the shape to multiple numbers and every time it tells me to change to 6, when I change to 6 it gives me other numbers. I also ran "print(X_train.shape)" which gave me "(16512, 6)"
model = Sequential()
model.add(Dense(32, input_shape=(6,), activation='relu'))
I would like the neural network to run and predict housing prices.
Housing price prediction is a regression problem since the target label/output is a single (housing price) continuous value. Therefore, final Dense layer of your model should have only one unit/neuron with linear activation function whereas you have 2 unit with sigmoid activation. Furthermore, you should should compile your model with mean_squared_error loss function.
model.add(Dense(1)) # Default activation is Linear
model.compile(Adam(lr=0.05), loss='mean_squared_error')
Hope it will help.
It's recommended to scale your features to be in the same range as features are in different ranges. You may want to check out Why, How and When to Scale your Features.
I'm building a 1D model with TensorFlow for audio but I have a problem with the input shape during the second MaxPool1D in the model.
The problem is here, after this Pooling:
x = Convolution1D(32, 3, activation=relu, padding='valid')(x)
x = MaxPool1D(4)(x)
I get this error:
ValueError: Negative dimension size caused by subtracting 4 from 1 for 'max_pooling1d_5/MaxPool' (op: 'MaxPool') with input shapes: [?,1,1,32].
I tried to reshape x (which is a tensor) but I don't think I'm going in the right way.
In this same model, before that, I have a couple convolutional layers and a maxpooling that are working proporly.
Anyone have suggestions?
Thanks
The number of steps in the input to the MaxPool1D layer is smaller than the pool size.
In the error, it says ...input shapes: [?,1,1,32], which means the output from the Convolution1D layer has shape [1,32]. It needs to be at least 4 steps to be used as input to the MaxPool1D(4) layer, so have a minimum size of [4,32].
You can continue walking this back. For example, the Convolution1D layer will decrease the step size by kernel_size-1=2. This means the input to the Convolution1D layer needs to have at least 4+2=6 steps, meaning a shape of at least [6,?]. Continuing up to the input layer, you'll find the input size is too small.
You'll need to change the architecture to allow the input size, or, if applicable, change the input size.
I am using Keras for building Conv Net for the first time. My layers are as follows:
layers = [
Conv2D(8,kernel_size=(4,4),padding='same',input_shape=( 200, 180,3),kernel_initializer="glorot_normal",data_format="channels_first"),
Activation("relu"),
MaxPooling2D(pool_size=(8,8),padding='same',data_format='channels_first'),
Conv2D(16,(2,2),padding='same',kernel_initializer="glorot_normal"),
Activation("relu"),
MaxPooling2D(pool_size=(4,4),padding='same',data_format='channels_first'),
Conv2D(4,(3,3),padding='same',kernel_initializer="glorot_normal"),
Activation("relu"),
MaxPooling2D(pool_size=(2,2),padding='same',data_format='channels_first'),
Flatten(),
Dense(2,input_shape=(48,)),
Softmax(axis=-1)
]
#Edit, here is the part for compiling the model and fitting it
model = Sequential(layers)
model.compile(optimizer="adam",loss="sparse_categorical_crossentropy"
metrics=["accuracy"])
trainHistory = model.fit(x=X_train,y=Y_train,batch_size=3,epochs=1000)
My labels array is of shape (,2). But when I try to use fit on the model, it gives me the error that softmax_1 expected to have shape (1,). But I have clearly mentioned units of Dense as 2 and softmax returns output of the same dimension as the input.
So where did the 1, came from? I tried to use dummy label array of 1 dimension and it runs. So what am I doing wrong? How do I use 2 dimensional array that I have?
The problem is that you are using sparse_categorical_crossentropy as the loss function. This loss function is used when the given labels (i.e. Y_train) are encoded as integers (i.e. 0, 1, 2, ...). However, If the labels are one-hot encoded, which seems to be the case in your code, you need to use categorical_crossentropy as the loss function instead.
I'm new to Keras and am trying to test out a model I've just trained.
I'm using Tensorflow backend and Python 3.
However, the shape my input has and the shape Keras says it has in an error are completely different. Here's my code:
testnote = np.zeros((3,))
testnote[0] = 70
testnote[1] = 70
print(testnote.shape)
pred = model.predict(testnote)
print(pred)
My consistent output is "(3,)" for the shape of testnote and then an error for my predict line: "ValueError: Error when checking input: expected dense_1_input to have shape (3,) but got array with shape (1,)"
How is it that Keras reads testnote as having shape (1,) when I've just confirmed that the shape is (3,)? Is it using some sort of different standard for what "shape" means? I've tried reshaping and adding brackets and a bunch of other things, but I don't really know what the problem is.
For additional context, the model takes in an array with 3 scalar input (representing pitch, velocity, and instrument class) and outputs an array with 1025 scalar outputs. I am carefully not using the word "dimension" since I think this is where I'm getting confused, and technically both are only 1 dimension. I'm sure there are many problems with my model which I will have to fix following this. However, I'd like to just get this prediction function working so I can understand what my output looks like.
Thanks in advance for any help.
A Keras Model implicitly expects that your data (passed as a np array) has a dimension for the batch size. Currently, your model is interpreting testnote as being 3 examples of shape 1. Try adding the batch dimension to 'testnote' as follows:
testnote = testnote.reshape(1,-1)
This will reshape testnote to shape (1, 3), so that you explicitly define the batch size to be 1.
I am using a dropout layer in my model. As I use temporal data, I want the noise_shape to be the same per timestep -> (batch_size, 1, features).
The problem is if I use a batch size that does not fit into the provided samples, I get an error message. Example: batch_size= 2, samples= 7. In the last iteration, the batch_size (2) is larger than the rest of the samples (1)
The other layers (my case: Masking, Dense, and LSTM) apparently don`t have a problem with that and just use a smaller batch for the last, not fitting, samples.
ConcreteError:
Training data shape is:[23, 300, 34]
batchsize=3
InvalidArgumentError (see above for traceback): Incompatible shapes:
[2,300,34] vs. [3,1,34] [[Node: dropout_18/cond/dropout/mul =
Mul[T=DT_FLOAT,
_device="/job:localhost/replica:0/task:0/device:CPU:0"](dropout_18/cond/dropout/div,
dropout_18/cond/dropout/Floor)]]
Meaning that for the last batch [2,300,34], the batch_size cannot split up into [3,1,34]
As I am still in the parameter tuning phase (does that ever stop :-) ),
Lookback(using LSTMs),
split-percentage of train/val/test,
and batchsize
will still constantly change. All of the mentioned influence the actual length and shape of the Training data.
I could try to always find the next fitting int for batch_size by some calculations. Example, if batch_size=4 and samples=21, I could reduce batch_size to 3. But if the number of training samples are e.g. primes this again would not work. Also If I choose 4, I probably would like to have 4.
Do I think to complex? Is there a simple solution without a lot of exception programming?
Thank you
Thanks to nuric in this post, the answer is quite simple.
The current implementation does adjust the according to the runtime
batch size. From the Dropout layer implementation code:
symbolic_shape = K.shape(inputs) noise_shape = [symbolic_shape[axis]
if shape is None else shape
for axis, shape in enumerate(self.noise_shape)]
So if you give noise_shape=(None, 1, features) the shape will be
(runtime_batchsize, 1, features) following the code above.