I have a problem. I have built a ConvNet. One hidden before the final output the shape of the output of that hidden layer is (None,64,32,32). What I want is to take the element wise average of those 64 channels. I have tried this:
main_inputs=[]
outputs=[]
def convnet(channels,rows,columns):
input=Input(shape=(channels,rows,columns))
main_inputs.append(input)
conv1=Convolution2D(kernel_size=(3,3) ,filters=64, padding="same")(input)
activation1= Activation('relu')(conv1)
conv2=Convolution2D(kernel_size=(3,3), filters=64, padding="same")(activation1)
activation2 = Activation('relu')(conv2)
conv3=Convolution2D(kernel_size=(3,3), filters=64, padding="same")(activation2)
activation3 = Activation('relu')(conv3)
conv4=Convolution2D(kernel_size=(3,3), filters=channels, padding="same")(activation3)
out=keras.layers.Average()(conv4)
activation4 = Activation('linear')(out)
outputs.append(activation4)
print(np.shape(outputs))
model = Model(inputs=main_inputs, outputs=outputs)
return model
But when I am getting an error:
ValueError: A merge layer should be called on a list of inputs
After that instead of the keras.layer.average I tried with the backend documentation:
out=K.mean(conv4,axis=1)
But I am getting this error:
'Tensor' object has no attribute '_keras_history'
Any ideas?
Let's say conv4 is a tensor with shape (batch_size, nb_channels, 32, 32). You can average conv4 over the channels' dimension as follows:
out = Lambda(lambda x: K.mean(x, axis=1))(conv4)
The resulting tensor out will have shape (batch_size, 32, 32). You need to wrap all the backend operations within a Lambda layer, so that the resulting tensors are valid Keras tensors (so that they don't lack some attributes such as _keras_history).
If you want the shape of out to be (batch_size, 1, 32, 32) instead, you can do:
out = Lambda(lambda x: K.mean(x, axis=1)[:, None, :, :])(conv4)
NOTE: Not tested.
Add my few cents to rvinas answer - there's parameter called keepdims which prevent reducing shape of tensor after applying some operation to it.
keepdims: A boolean, whether to keep the dimensions or not. If
keepdims is False, the rank of the tensor is reduced by 1. If keepdims
is True, the reduced dimension is retained with length 1.
out = Lambda(lambda x: K.mean(x, axis=1), keepdims=True)(conv4)
Related
I am trying to tidy up my code by moving from the Keras functional API to the subclassing API. The class I came up with so far is below:
class FeedForwardNN(Model):
def __init__(self, params):
super().__init__()
self.params = params
self.layout = params['layout']
# Define layers
self.dense = Dense(units=params['layout'][1],
activation=params['activation'],
kernel_initializer=params['initializer'])
self.output_layer = Dense(units=params['layout'][-1],
kernel_initializer=params['initializer'])
self.dropout = Dropout(params['dropout'])
self.batch_norm = BatchNormalization()
def call(self, x):
for layer in self.layout[1:-1]:
x = self.dropout(self.dense(x))
if self.params['batch_norm']:
x = self.batch_norm(x)
x = self.output_layer(x)
return x
Where layout is a list of the neurons in each layer (including input and output layers).
However, when fitting the model, the following error is raised:
ValueError: Input 0 of layer "dense" is incompatible with the layer: expected axis -1 of input shape to have value 5, but received input with shape (None, 100)
Call arguments received:
• x=tf.Tensor(shape=(None, 5), dtype=float32)
which seems to occur on the line:
x = self.dropout(self.dense(x))
I checked the shape of the training data X that is passed to the fit() method, and it appears to have the right shape i.e. (number of observations, number of predictors).
Does anyone have an idea of where my mistake is?
The problem is that you are using same self.dense layer over and over again in your for loops
for layer in self.layout[1:-1]:
x = self.dropout(self.dense(x))
After the first loop, x has shape (batch, 100). Then in the second loop, instead of passing this x to the second Dense layer (which you don't seem to have created in the first place), you re-pass it to the first Dense layer, which expects shape (batch, 5), causing the error.
You can create a list of dense layer as follows in __init__
self.denses = [Dense(units=self.layout[i],
activation=params['activation'],
kernel_initializer=params['initializer']) for i in self.layout[1:-1]]
and call them in sequence
for dense_layer in self.denses:
x = self.dropout(dense_layer(x))
Let us assume we have a tensor x with shape (64,100,5,32) which corresponds to (batchSize,Length,Height,Channels). Now I want to apply a 2D conv Layer on each 2D matrix of size (100,5) for each of the 32th channels. So I need to extract 32 slices and process them with the same 2D conv layer (parameters). I dont know how to start with lambda und map_fn (please not use time distributed layer). Finally, I want a tensor with size (64,100,5,32).
Thanks for a short code snipped how do this.
you can simply use a for loops with index slicing (without Lambda layer). here a dummy example:
n_sample = 3
H,W,C = 100,5,32
X = np.random.uniform(0,1, (n_sample,H,W,C))
inp = Input((H,W,C))
convs = []
conv = Conv2D(1, 3, padding='same') # this is always the same for all the slices
for c in range(inp.shape[-1]):
_x = tf.expand_dims(inp[:,:,:,c], -1)
convs.append(conv(_x))
convs = Concatenate()(convs)
model = Model(inp, convs)
model.compile('adam', 'mse')
model.fit(X,X, epochs=2)
I am confused on how to replicate Keras (TensorFlow) convolutions in PyTorch.
In Keras, I can do something like this. (the input size is (256, 237, 1, 21) and the output size is (256, 237, 1, 1024).
import tensorflow as tf
x = tf.random.normal((256,237,1,21))
y = tf.keras.layers.Conv1D(filters=1024, kernel_size=5,padding="same")(x)
print(y.shape)
(256, 237, 1, 1024)
However, in PyTorch, when I try to do the same thing I get a different output size:
import torch.nn as nn
x = torch.randn(256,237,1,21)
m = nn.Conv1d(in_channels=237, out_channels=1024, kernel_size=(1,5))
y = m(x)
print(y.shape)
torch.Size([256, 1024, 1, 17])
I want PyTorch to give me the same output size that Keras does:
This previous question seems to imply that Keras filters are PyTorch's out_channels but thats what I have. I tried to add the padding in PyTorch of padding=(0,503) but that gives me torch.Size([256, 1024, 1, 1023]) but that still not correct. This also takes so much longer than keras does so I feel that I have incorrectly assigned a parameter.
How can I replicate what Keras did with convolution in PyTorch?
In TensorFlow, tf.keras.layers.Conv1D takes in a tensor of shape (batch_shape + (steps, input_dim)). Which means that what is commonly known as channels appears on the last axis. For instance in 2D convolution you would have (batch, height, width, channels). This is different from PyTorch where the channel dimension is right after the batch axis: torch.nn.Conv1d takes in shapes of (batch, channel, length). So you will need to permute two axes.
For torch.nn.Conv1d:
in_channels is the number of channels in the input tensor
out_channels is the number of filters, i.e. the number of channels the output will have
stride the step size of the convolution
padding the zero-padding added to both sides
In PyTorch there is no option for padding='same', you will need to choose padding correctly. Here stride=1, so padding must equal to kernel_size//2 (i.e. padding=2) in order to maintain the length of the tensor.
In your example, since x has a shape of (256, 237, 1, 21), in TensorFlow's terminology it will be considered as an input with:
a batch shape of (256, 237),
steps=1, so the length of your 1D input is 1,
21 input channels.
Whereas in PyTorch, x of shape (256, 237, 1, 21) would be:
batch shape of (256, 237),
1 input channel
a length of 21.
Have kept the input in both examples below (TensorFlow vs. PyTorch) as x.shape=(256, 237, 21) assuming 256 is the batch size, 237 is the length of the input sequence, and 21 is the number of channels (i.e. the input dimension, what I see as the dimension on each timestep).
In TensorFlow:
>>> x = tf.random.normal((256, 237, 21))
>>> m = tf.keras.layers.Conv1D(filters=1024, kernel_size=5, padding="same")
>>> y = m(x)
>>> y.shape
TensorShape([256, 237, 1024])
In PyTorch:
>>> x = torch.randn(256, 237, 21)
>>> m = nn.Conv1d(in_channels=21, out_channels=1024, kernel_size=5, padding=2)
>>> y = m(x.permute(0, 2, 1))
>>> y.permute(0, 2, 1).shape
torch.Size([256, 237, 1024])
So in the latter, you would simply work with x = torch.randn(256, 21, 237)...
PyTorch now has out of the box same convolution operation you can take a look at this link [Same convolution][1]
class InceptionNet(nn.Module):
def __init__(self, in_channels, in_1x1, in_3x3reduce, in_3x3, in_5x5reduce, in_5x5, in_1x1pool):
super(InceptionNet, self).__init__()
self.incep_1 = ConvBlock(in_channels, in_1x1, kernel_size=1, padding='same')
Note a same convolution only supports the default stride value which is 1 anything other won't work.
[1]: https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html
The full error message is like this:
ValueError: Shapes (2, 1) and (50, 1) are incompatible
It occurs when my model is trained. The mistake either is in my input_fn:
train_input_fn = tf.estimator.inputs.numpy_input_fn(
x = {"x" : training_data},
y = training_labels,
batch_size = 50,
num_epochs = None,
shuffle = True)
in my logits and loss function:
dense = tf.layers.dense(inputs = pool2_flat, units = 1024, activation = tf.nn.relu)
dropout = tf.layers.dropout(inputs = dense, rate = 0.4, training = mode == tf.estimator.ModeKeys.TRAIN)
logits = tf.layers.dense(inputs = dropout, units = 1)
loss = tf.losses.softmax_cross_entropy(labels = labels, logits = logits)
or in my dataset. I can only print out the shape of my dataset for you to take a look at it.
#shape of the dataset
train_data.shape
(1196,2,1)
train_data[0].shape
(2,1)
#this is the data
train_data[0][0].shape
(1,)
train_data[0][0][0].shape
(20,50,50)
#this is the labels
train_data[0][1].shape
(1,)
The problem seems to be the shape of the logits. They are supposed to be [batch_size, num_classes] in this case [50,1] but are [2,1]. The shape of the labels is correctly [50,1]
I have made a github gist if you want to take a look at the whole code.
https://gist.github.com/hjkhjk1999/38f358a53da84a94bf5a59f44050aad5
In your code, you are stating that the inputs to your model will be feed in batches of 50 samples per batch with one variable. But it looks like your are feeding actually a batch of 2 samples with 1 variable (shape=[2, 1]) despite feeding labels with shape [50, 1].
That's the problem, you are giving 50 'questions' and two 'answers'.
Also, your dataset is shaped in a really weird way. I see you named your github gist 3D Conv. If you are indeed trying to do a 3D convolution you might want to reshape your dataset into a tensor (numpy array) of this shape shape = [samples, width, height, deepth]
According to this Deep Learning course http://cs231n.github.io/convolutional-networks/#conv, It says that if there is an input x with shape [W,W] (where W = width = height) goes through a Convolutional Layer with filter shape [F,F]and stride S, the Layer will return an output with shape [(W-F)/S +1, (W-F)/S +1]
However, when I'm trying to follow the tutorial of the Tensorflow: https://www.tensorflow.org/versions/r0.11/tutorials/mnist/pros/index.html. There seems to have difference of the function tf.nn.conv2d(inputs, filter, stride)
Whatever how do I change my filter size, conv2d will constantly return me a value with the same shape as the input.
In my case, I am using the MNIST dataset which indicates that every image has size [28,28](ignoring channel_num = 1)
but after I defining the first conv1 layers, I used the conv1.get_shape() to see its output, it gives me [28,28, num_of_filters]
Why is this? I thought the return value should follow the formula above.
Appendix: Code snippet
#reshape x from 2d to 4d
x_image = tf.reshape(x, [-1, 28, 28, 1]) #[num_samples, width, height, channel_num]
## define the shape of weights and bias
w_shape = [5, 5, 1, 32] #patch_w, patch_h, in_channel, output_num(out_channel)
b_shape = [32] #bias only need to be consistent with output_num
## init weights of conv1 layers
W_conv1 = weight_variable(w_shape)
b_conv1 = bias_variable(b_shape)
## first layer x_iamge->conv1/relu->pool1
#Our convolutions uses a stride of one
#and are zero padded
#so that the output is the same size as the input
h_conv1 = tf.nn.relu(
conv2d(x_image, W_conv1) + b_conv1
)
print 'conv1.shape=',h_conv1.get_shape()
## conv1.shape= (?, 28, 28, 32)
## I thought conv1.shape should be (?, (28-5)/1+1, 24 ,32)
h_pool1 = max_pool_2x2(h_conv1) #output 32 num
print 'pool1.shape=',h_pool1.get_shape() ## pool1.shape= (?, 14, 14, 32)
It depends on the padding parameter. 'SAME' will keep the output as WxW (assuming stride=1,) 'VALID' will shrink the size of the output to (W-F+1)x(W-F+1)
Conv2d has a parameter called padding see here
Where if you set padding to "VALID" it will satisfy your formula. It defaults to "SAME" which pads (same as adding a border around) the image filled with zeroes such that the output will remain the same shape as the input.