I am pretty confused when it comes to the shape of a convolutional layer in tensorflow.
kernels = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, "conv1d(_.)?/kernel:0")
Running this line returns me a kernel with 4 dimensions and a bias.
I expected that the kernel would return me [filter_width, filter_height, filter_number] and a 2D matrix with weights. Instead i have a fourth dimensions and no weights at all.
Maybe I should not interchange dense with convolutional layers in my mind. However most of the explanations I find on the internet stay on a simple level without going into the details of tensorflows model.
So most important for me would be getting the interconnected weights of the edges between the layers. Like seen on this picture:
This link regards to my problem: Something I want from Tensorflow
I hope someone can follow my trouble in understanding, otherwise do not hesitate adding comments.
Filters/kernels always have 4 dimensions, which are (width, height, filter_nr, channels). Channels equals to the number of channels in the input image, but for later layers in the network it may be different.
The weights you are asking are for a fully connected (dense) layer, not for a convolutional layer (which is what Conv2D is).
Related
Post-edit: Turns out I got confused while constantly playing with the three functions below.
model.weights
model.get_weights()
model.layer(i).get_weights()
model.layer(i).get_weights() returns two separate arrays (without any tags) which are kernel and bias if bias exists in the model.
model.get_weights() directly returns all the weights without any tags.
model.weights returns weights and a bit of info such as name of the layer it belongs to and its shape. I was using this one for the experiment in the question.
What confused me was simply 1 and 3 above.
Note: I've decided not to delete the question because it received an answer and with the post-edit may it still help someone.
The question was...
After saving a Keras model, when I check the weights, I notice 2 separate biases.
Below is a part of weights listed by names.
conv2d/kernel:0
conv2d/bias:0
kernel ones store a bias array as their 2nd numpy array element which I knew as the original bias of the layer. Then, there is bias ones as well separately.
Which one serves what purpose? What is the difference between them?
The convolution layer (conv2d) has a kernel and a bias term and the dense layer (dense) has also a kernel and a bias term. The bias terms are here to give a new degree of freedom for each layer making the neural net more powerful to predict.
I try to switch from pytorch to tensorflow and since the model now seems to be a fixed thing in tensorflow, i stumble upon a problem when working with Convolutional Neural Networks.
I have a very simple model, just one Conv1D layer and a kernel with size 2.
I want to train it on a small Configuration, say 16 input size and then export the training results on a 32 input size.
How can i access the 3 parameters in this network? (2 kernel, 1 bias) I want to do so to apply them for the higher size case. I struggle because i need to pre-define a input size of the model, this was not the case with pytorch.
Thanks for answering, I've only found outdated answers to this question
model.layers[0].get_weights() yields the weights of the first layer, assuming model is a tf.keras.Model object.
I'm creating a Sequential model in Keras that takes a colour image and convolves it through multiple Conv2D layers of approximately the same size and shape as the top layer (minus the edges sliced off by the convolutions, basically).
My understanding is as follows:
kernel_size indicates the patch size for each convolution's input
filters indicates the layer depth for each convolution's output.
I then do some other stuff after the convolutions, which isn't relevant here.
However, when I tried to compile my model prior to testing it on a little data, I discovered that Tensorflow complains when I try to make kernel_size for a given layer greater than filters for the previous layer. It doesn't actually say that; instead it says
Negative dimension size caused by subtracting 3 from 1 for 'conv2d_2/convolution' (op: 'Conv2D') with input shapes [?,1,1022,1022], [3,3,1022,1]
which isn't exactly informative. However, I noticed that the numbers it puts in correspond to
Negative dimension size caused by subtracting <this layer's kernel_size> from <previous layer's filters> ....
and setting filters to be higher stopped the error.
My question is: why should this be? I thought filters specified depth, and kernel_size specified width. There shouldn't be any need to fit a convolution patch into the thickness of the previous layer. Moreover, this problem does not occur on the first layer, whose channel depth (which I understand to be effectively equivalent to filters) is 3.
Is this a bug, or am I misinterpreting these parameters' meaning, or something else?
Code snippet:
__model = Sequential()
# feature layers
__model.add(Conv2D(input_shape=(3, iX, iY), data_format="channels_first", kernel_size=kernelfilters[0][0],
filters=kernelfilters[0][1], activation=ACTIVATION))
for kernelfilter in kernelfilters:
__model.add(Conv2D(kernel_size=kernelfilter[0], filters=kernelfilter[1], activation=ACTIVATION))
The last line is the one that breaks.
Each kernelfilter in the kernelfilters array is a pair of numbers specifying a kernel_size and a filters value, in that order. iX and iY are the initial image dimensions. ACTIVATION is a constant, currently set to "relu" but I might change it later!
Your premise is wrong, this is not the case in general. This only happens if you fiddle with the image_data_format keras parameter (in ~/.keras/keras.kson) or with the data_format parameter for each layer.
Changing this parameter in an in consistent way (in some layers only) would completely screw up how the data is interpreted, as it changes the position of the channels dimension, which could be interpreter as one of the spatial dimensions. for the TF backend (meaning the input_shape in the top layer should be a tuple of the form (width, height, channels)).
I am working with a CNN implemented in Keras which at some point has a flatten layer. Now, my goal is to allow different input shaped images. So my first conv. layer looks something like:
model.add(Conv2D(...., input_shape=(None, None, 1))
Well in this setup my flatten layer becomes unhappy and tells me to specify the input shape. As such, I am using a GlobalMaxPooling layer currently, which I would like to avoid.
After all, why does the flatten layer bother about the width and height?
Background: I try to train a net for classification (smaller resolution) and afterwards use this net for object detection (higher resolution)
Thanks
It bothers about the shape because you will probably want to connect another layer to it.
And its feature dimension will be the basis for the next layer to create its own weights. A layer can't have a variable size weight matrix, thus, it can't have a variable size feature input.
I am wondering if it is possible how to add a similar to flattened layer for images of variable length.
Say we have an input layer for our CNN as:
input_shape=(1, None, None)
After performing your typical series of convolution/maxpooling layers, can we create a flattened layer, such that the shape is:
output_shape=(None,...)
If not, would someone be able to explain why not?
You can add GlobalMaxPooling2D and GlobalAveragePooling2D.
These will eliminate the spatial dimensions and keep only the channels dimension. Max will take the maximum values, Average will get the mean value.
I don't really know why you can't use a Flatten layer, but in fact you can't with variable dimensions.
I understand why a Dense wouldn't work: it would have a variable number of parameters, which is totally infeasible for backpropagation, weight update and things like that. (PS: Dense layers act only on the last dimension, so that is the only that needs to be fixed).
Examples:
A Dense layer requires the last dimension fixed
A Conv layer can have variable spatial dimensions, but needs fixed channels (otherwise the number of parameters will vary)
A recurrent layer can have variable time steps, but needs fixed features and so on
Also, notice that:
For classification models, you'd need a fixed dimension output, so, how to flatten and still guarantee the correct number of elements in each dimension? It's impossible.
For models with variable output, why would you want to have a fixed dimension in the middle of the model anyway?
If you're going totally custom, you can always use K.reshape() inside a Lambda layer and work with the tensor shapes:
import keras.backend as K
def myReshape(x):
shape = K.shape(x)
batchSize = shape[:1]
newShape = K.variable([-1],dtype='int32')
newShape = K.concatenate([batchSize,newShape])
return K.reshape(x,newShape)
The layer: Lambda(myReshape)
I don't think you can because the compile step uses those dimensions to allocate fixed memory when your model is instanced for training or prediction. Some dimensions need to be known ahead of time, so the matrix dimensions can be allocated.
I understand why you want variable-sized image input, the world is not (226, 226, 3). It depends on your specific goals, but for me, scaling up or windowing to a region of interest using say Single Shot Detection as a preprocessing step may be helpful. You could just start with Keras's ImageDataGenerator to scale all images to a fixed size - then you see how much of a performance gain you get from conditional input sizing or windowing preprocessing.
#mikkola, I have found flatten to be very helpful for TimeDistributed models. You can add flatten after the convolution steps using:
your_model.add(Flatten())