I'm trying to implement the architecture in this paper: https://arxiv.org/pdf/1701.05957.pdf (in the top of page 3)
The problem is that, my dataset contains different size of images, and while trying to apply convolution after transpose with dynamic shapes, the unknown shapes will lead to error:
ValueError: Shape of a new variable (fuse01/weights) must be fully defined, but instead was (3, 3, ?, 10).
This is my code:
import tensorflow as tf
import numpy as np
slim = tf.contrib.slim
def conv(input_batch, nb_kernel, nb_row, nb_col, scope_name, strides=None):
if strides is None:
strides = 1
with slim.arg_scope([slim.conv2d], padding='SAME', stride=strides):
out = slim.conv2d(input_batch, nb_kernel, [nb_row, nb_col], scope=scope_name)
return out
def conv_trans(input_batch, nb_kernel, nb_row, nb_col, name_scope, stride, output_like):
with tf.name_scope(name_scope):
weights = get_weights([nb_row, nb_col, nb_kernel, input_batch.get_shape()[3].value])
out_shape = tf.shape(output_like)
out_shape = [input_batch.get_shape()[0].value, out_shape[1], out_shape[2], nb_kernel]
output = tf.nn.conv2d_transpose(input_batch, weights, out_shape, [1, stride, stride, 1])
return output
def get_weights(shape):
initializer = tf.contrib.layers.xavier_initializer_conv2d(dtype=tf.float32)
variable = tf.Variable(initializer(shape=shape), name='weights')
return variable
a = np.ones([1, 165, 167, 3], np.float32)
x = tf.placeholder(tf.float32, [1, None, None, 3])
net = conv(x, 10, 3, 3, 'conv1')
net = conv(net, 20, 3, 3, 'conv2', strides=2)
skip_01 = net
net = conv(net, 40, 3, 3, 'conv3', strides=2)
skip_02 = net
net = conv(net, 80, 3, 3, 'conv4', strides=2)
skip_03 = net
net = conv(net, 160, 3, 3, 'conv5', strides=2)
up_01 = conv_trans(net, 30, 3, 3, 'test', 2, skip_03) # shape: (?, ?, ?, ?)
fuse_01 = tf.concat([skip_03, up_01], 3) # shape: (1, ?, ?, ?)
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)
print fuse_01
print sess.run(fuse_01, feed_dict={x: a})
fuse_01 = conv(fuse_01, 10, 3, 3, 'fuse01') # this will cause error
Is there any way to get the specific shape of the tensor after applying conv_transpose?
The problem is that, my dataset contains different size of images, and while trying to ....
The easiest approach is just to convert all the images to the same size at the very beginning. You are trying to remove rain/snow from the image, so the size of the image should not matter.
Related
This following code below giving a output of shape (1,1,3) for the shape of xodd is (1,1,2). The given kernel shape is(112, 1, 1).
from torch.nn import functional as F
output = F.conv1d(xodd, kernel, padding=zeros)
How the padding=zeros works?
And also, How can I write an equivalent code in tensorflow so that the output is as same as the above output?
What is padding=zeros?
If we set paddin=zeros, we don't need to add numbers at the right and the left of the tensor.
Padding=0:
from torch.nn import functional as F
import torch
inputs = torch.randn(33, 16, 6) # (minibatch,in_channels,features)
filters = torch.randn(20, 16, 5) # (out_channels, in_channels, kernel_size)
out_tns = F.conv1d(inputs, filters, stride=1, padding=0)
print(out_tns.shape)
# torch.Size([33, 20, 2]) # (minibatch,out_channels,(features-kernel_size+1))
Padding=2:(We want to add two numbers at the right and the left of the tensor)
inputs = torch.randn(33, 16, 6) # (minibatch,in_channels,features)
filters = torch.randn(20, 16, 5) # (out_channels, in_channels, kernel_size)
out_tns = F.conv1d(inputs, filters, stride=1, padding=2)
print(out_tns.shape)
# torch.Size([33, 20, 6]) # (minibatch,out_channels,(features-kernel_size+1+2+2))
How can I write an equivalent code in tensorflow:
import tensorflow as tf
input_shape = (33, 6, 16)
x = tf.random.normal(input_shape)
out_tf = tf.keras.layers.Conv1D(filters = 20,
kernel_size = 5,
strides = 1,
input_shape=input_shape[1:])(x)
print(out_tf.shape)
# TensorShape([33, 2, 20])
# If you want that tensor have shape exactly like pytorch you can transpose
tf.transpose(out_tf, [0, 2, 1]).shape
# TensorShape([33, 20, 2])
I've developed a custom convolutional layer. I can use it inside a model and train it (model.fit works), but model.predict() yields an error!
I will add a simple code to demonstrate how the code is structured.
modelx1 = tf.keras.models.Sequential([tf.keras.Input(shape=(49,)), Dense(1, activation = 'relu')])
class customLayer(tf.keras.layers.Layer):
def __init__(self,n=10):super(customLayer, self).__init__()
def call(self, inputs):
_, Dim0,Dim1, Dim3 = inputs.shape
input_victorized = tf.image.extract_patches(images=inputs, sizes=[-1, 7, 7, 1],
strides=[1, 1, 1, 1],rates=[1, 1, 1, 1], padding='SAME')
input_victorized2 = tf.reshape(input_victorized, [-1,49])
model_output = modelx1(input_victorized2)
out = tf.reshape(model_output,[-1,Dim0,Dim1,Dim3])
return out
The custom layer reshapes the input, then feeds it to 'modelx1' then it reshapes the output.
Here is a simple model where the custom layer is used:
input1 = tf.keras.Input(shape=(28,28,1))
x = Conv2D(filters = 2, kernel_size = 5, activation = 'relu')(input1)
Layeri = customLayer()(x)
xxc = Flatten()(Layeri)
y = Dense(units = 3, activation = 'softmax')(xxc)
model = tf.keras.Model(inputs=input1, outputs=y)
model.summary()
The error appears when I run model.predict:
model.predict(np.ones([100,28,28,1]))
UnimplementedError: Only support ksizes across space.
[[node model_58/custom_layer_9/ExtractImagePatches
(defined at <ipython-input-279-953feb59f882>:7)
]] [Op:__inference_predict_function_14640]
Errors may have originated from an input operation.
Input Source operations connected to node model_58/custom_layer_9/ExtractImagePatches:
In[0] model_58/conv2d_98/Relu (defined at /usr/local/lib/python3.7/dist-packages/keras/backend.py:4867)
I think this should work:-
image = tf.expand_dims(image, 0)
extracted_patches = tf.image.extract_patches(images = image,
sizes = [1, int(0.5 * image_height), int(0.5 * image_width), 1],
strides = [1, int(0.5 * image_height), int(0.5 * image_width), 1],
rates = [1, 1, 1, 1],
padding = "SAME")
And then use tf.reshape to extract these patches
patches = tf.reshape(extracted_patches,
[-1,int(0.5*image_height),int(0.5*image_width),3])
I had a similar error a couple of months back; This Fixed it!
`
At a certain stage in a resnet, I have 6 features per image i.e. each example is of shape 1X8X8X6, I want to involve each feature with 4 constant filters (DWT) of size 1X2X2X1 with a stride of 2 to get 24 features in next layer and the image to become 1X4X4X24. However, I am unable to use tf.nn.conv2d or tf.nn.convolution for this purpose, conv2d says fourth dimension of input be equal to 3rd dimension of the filter, but how can I do this, I tried doing for the first filter but even this doesn't work:
x_in = np.random.randn(1,8,8,6)
kernel_in = np.array([[[[1],[1]],[[1],[1]]]])
kernel_in.shape
x = tf.constant(x_in, dtype=tf.float32)
kernel = tf.constant(kernel_in, dtype=tf.float32)
tf.nn.convolution(x, kernel, strides=[1, 1, 1, 1], padding='VALID')
try in this way
x_in = np.random.randn(1,8,8,6) # [batch, in_height, in_width, in_channels]
kernel_in = np.ones((2,2,6,24)) # [filter_height, filter_width, in_channels, out_channels]
x = tf.constant(x_in, dtype=tf.float32)
kernel = tf.constant(kernel_in, dtype=tf.float32)
tf.nn.conv2d(x, kernel, strides=[1, 2, 2, 1], padding='VALID')
# <tf.Tensor: shape=(1, 4, 4, 24), dtype=float32, numpy=....>
A simple example of how to fill predefined values to filters in a Keras.conv2d layer in TF2:
model = models.Sequential()
# one 3x3 filter
model.add(layers.Conv2D(1, (3, 3), input_shape=(None, None, 1)))
# access to the target layer
layer = model.layers[0]
current_w, current_bias = layer.get_weights() # see the current weights
new_w = tf.constant([[1,2, 3],
[4, 5, 6],
[7, 8, 9]])
new_w = tf.reshape(new_w, custom_w.shape) # fix the shape
new_bias = tf.constant([0])
layer.set_weights([new_w, new_bias])
model.summary()
# let's see ..
tf.print(model.layers[0].get_weights())
According to the tensorflow website, tf.reshape takes a tensor of a certain shape and maps it to a tensor of another shape. I want to map a tensor of size [600, 64] to a tensor of size [-1, 8, 8, 1] (in which the dimension at the -1 position is 600). This doesn't seem to be working though.
I am running this on tensorflow on python 3.6 and although it reshapes to something like [-1, 8, 8], it doesn't reshape to [-1, 8, 8, 1]
import tensorflow as tf
import numpy as np
from sklearn import datasets
from sklearn.preprocessing import LabelBinarizer
# preprocessing method needed
def flatten(array):
temp = []
for j in array:
temp.extend(j)
return temp
# preprocess the data
digits = datasets.load_digits()
images = digits.images
images = [flatten(i) for i in images]
labels = digits.target
labels = LabelBinarizer().fit_transform(labels)
# the stats needed
width = 8
height = 8
alpha = 0.1
num_labels = 10
kernel_length = 3
batch_size = 10
channels = 1
# the tensorflow placeholders and reshaping
X = tf.placeholder(tf.float32, shape = [None, width * height * channels])
# AND NOW HERE IS WHERE THE ERROR STARTS
y_true = tf.placeholder(tf.float32, shape = [None, num_labels])
X = tf.reshape(X, [-1, 8, 8, 1])
# the convolutional model
conv1 = tf.layers.conv2d(X, filters = 32, kernel_size = [kernel_length, kernel_length])
conv2 = tf.layers.conv2d(conv1, filters = 64, kernel_size = [2, 2])
flatten = tf.reshape(X, [-1, 1])
dense1 = tf.layers.dense(flatten, units=50, activation = tf.nn.relu)
y_pred = tf.layers.dense(dense1, units=num_labels, activation = tf.nn.softmax)
# the loss and training functions
loss = tf.losses.mean_squared_error(labels=y_true, predictions=y_pred)
train = tf.train.GradientDescentOptimizer(alpha).minimize(loss)
# initializing the variables and the tf.session
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)
# running the session
for i in range(batch_size):
_, lossVal = sess.run((train, loss), feed_dict = {X:images[:600], y_true: labels[:600]})
print(lossVal)
I keep on getting this error:
ValueError: Cannot feed value of shape (600, 64) for Tensor 'Reshape:0', which has shape '(?, 8, 8, 1)'
And I feel like that should not be the case since 8 * 8 * 1 does equal 64.
images[:600]'s shape is (600, 64), which does not correspond to the placeholder expected shape, (None, 8, 8, 1).
Either reshape your data or change the shape of the placeholder.
Note that the fact that you originally defined the placeholder shape to be (None, 64) is inconsequential as you reshape it a few lines later.
I am trying to import weights saved from a Tensorflow model to PyTorch. So far the results have been very similar. I ran into a snag when the model calls for conv2d with stride=2.
To verify the mismatch, I set up a very simple comparison between TF and PyTorch. First, I compare conv2d with stride=1.
import tensorflow as tf
import numpy as np
import torch
import torch.nn.functional as F
np.random.seed(0)
sess = tf.Session()
# Create random weights and input
weights = torch.empty(3, 3, 3, 8)
torch.nn.init.constant_(weights, 5e-2)
x = np.random.randn(1, 3, 10, 10)
weights_tf = tf.convert_to_tensor(weights.numpy(), dtype=tf.float32)
# PyTorch adopts [outputC, inputC, kH, kW]
weights_torch = torch.Tensor(weights.permute((3, 2, 0, 1)))
# Tensorflow defaults to NHWC
x_tf = tf.convert_to_tensor(x.transpose((0, 2, 3, 1)), dtype=tf.float32)
x_torch = torch.Tensor(x)
# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
weights_tf,
strides=[1, 1, 1, 1],
padding="SAME")
# PyTorch Conv2D
torch_conv2d = F.conv2d(x_torch, weights_torch, padding=1, stride=1)
sess.run(tf.global_variables_initializer())
tf_result = sess.run(tf_conv2d)
diff = np.mean(np.abs(tf_result.transpose((0, 3, 1, 2)) - torch_conv2d.detach().numpy()))
print('Mean of Abs Diff: {0}'.format(diff))
The result of this execution is:
Mean of Abs Diff: 2.0443112092038973e-08
When I change stride to 2, the results start to vary.
# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
weights_tf,
strides=[1, 2, 2, 1],
padding="SAME")
# PyTorch Conv2D
torch_conv2d = F.conv2d(x_torch, weights_torch, padding=1, stride=2)
The result of this execution is:
Mean of Abs Diff: 0.2104552686214447
According to PyTorch documentation, conv2d uses zero-padding defined by the padding argument. Thus, zeros are added to the left, top, right, and bottom of the input in my example.
If PyTorch simply adds padding on both sides based on the input parameter, it should be easy to replicate in Tensorflow.
# Manually add padding - consistent with PyTorch
paddings = tf.constant([[0, 0], [1, 1], [1, 1], [0, 0]])
x_tf = tf.convert_to_tensor(x.transpose((0, 2, 3, 1)), dtype=tf.float32)
x_tf = tf.pad(x_tf, paddings, "CONSTANT")
# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
weights_tf,
strides=[1, 2, 2, 1],
padding="VALID")
The result of this comparison is:
Mean of Abs Diff: 1.6035047067930464e-08
What this tells me is that if I am somehow able to replicate the default padding behavior from Tensorflow into PyTorch, then my results will be similar.
This question inspected the behavior of padding in Tensorflow. TF documentation explains how padding is added for "SAME" convolutions. I discovered these links while writing this question.
Now that I know the padding strategy of Tensorflow, I can implement it in PyTorch.
To replicate the behavior, padding sizes are calculated as described in the Tensorflow documentation. Here, I test the padding behavior by setting stride=2 and padding the PyTorch input.
import tensorflow as tf
import numpy as np
import torch
import torch.nn.functional as F
np.random.seed(0)
sess = tf.Session()
# Create random weights and input
weights = torch.empty(3, 3, 3, 8)
torch.nn.init.constant_(weights, 5e-2)
x = np.random.randn(1, 3, 10, 10)
weights_tf = tf.convert_to_tensor(weights.numpy(), dtype=tf.float32)
weights_torch = torch.Tensor(weights.permute((3, 2, 0, 1)))
# Tensorflow padding behavior. Assuming that kH == kW to keep this simple.
stride = 2
if x.shape[2] % stride == 0:
pad = max(weights.shape[0] - stride, 0)
else:
pad = max(weights.shape[0] - (x.shape[2] % stride), 0)
if pad % 2 == 0:
pad_val = pad // 2
padding = (pad_val, pad_val, pad_val, pad_val)
else:
pad_val_start = pad // 2
pad_val_end = pad - pad_val_start
padding = (pad_val_start, pad_val_end, pad_val_start, pad_val_end)
x_tf = tf.convert_to_tensor(x.transpose((0, 2, 3, 1)), dtype=tf.float32)
x_torch = torch.Tensor(x)
x_torch = F.pad(x_torch, padding, "constant", 0)
# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
weights_tf,
strides=[1, stride, stride, 1],
padding="SAME")
# PyTorch Conv2D
torch_conv2d = F.conv2d(x_torch, weights_torch, padding=0, stride=stride)
sess.run(tf.global_variables_initializer())
tf_result = sess.run(tf_conv2d)
diff = np.mean(np.abs(tf_result.transpose((0, 3, 1, 2)) - torch_conv2d.detach().numpy()))
print('Mean of Abs Diff: {0}'.format(diff))
The output is:
Mean of Abs Diff: 2.2477470551507395e-08
I wasn't quite sure why this was happening when I started writing this question, but a bit of reading clarified this very quickly. I hope this example can help others.