Develop Reference

Getting the expected dimensions of the Jacobian with JAX?

I am trying to get the Jacobian for a simple parameterization function within JAX. The code is as follows:
# imports
import jax
import jax.numpy as jnp
from jax import random
# simple parameterization function
def reparameterize(v_params):
theta = v_params[0] + jnp.exp(v_params[1]) * eps
return theta
Suppose I initialize eps to be a vector of shape (3,) and v_params to be of shape (3, 2):
key = random.PRNGKey(2022)
eps = random.normal(key, shape=(3,))
key, _ = random.split(key)
v_params = random.normal(key, shape=(3, 2))
I want the Jacobian to be an array of shape (3, 2) but by using
jacobian(vmap(reparameterize))(v_params)
returns an array of shape (3, 3, 3, 2). If I re-initialize with only a single eps:
key, _ = random.split(key)
eps = random.normal(key, shape=(1, ))
key, _ = random.split(key)
v_params = random.normal(key, shape=(2, ))
and call jacobian(reparameterize)(v_params) I get what I want, e.g., an array of shape (2, ). Effectively looping over all eps and stacking the results of each Jacobian gives me the desired Jacobian (and shape). What am I missing here? Thanks for your help!

For a function f that maps an input of shape shape_in to an output of shape shape_out, the jacobian will have shape (*shape_out, *shape_in).
In your case, vmap(reparameterize) takes an array of shape (3, 2) and returns an array of shape (3, 3), so the output of the jacobian is an array of shape (3, 3, 3, 2).
It's hard to tell from your question what computation you were intending, but if you want a jacobian the same shape as the input, you need a function that maps the input to a scalar. Perhaps the sum is what you had in mind?
result = jacobian(lambda x: vmap(reparameterize)(x).sum())(v_params)
print(result.shape)
# (3, 2)

Find vertical and horizontal patterns in 2D feature matrix using Conv2D

I want to build a model using Conv2D which will find vertical and horizontal patterns which describes the relationship between a series of features in a 2D matrix with Height=N and Width=6. This is not an image, but a list of events which consists from some variables, represented as a 2D matrix, each event has length=6 (6 variables), and the matrix size is Nx6. N is not a constant value.
I want the model to find some patterns using vertically and horizontally kernels like this:
Here is my code:
x = tf.keras.Input(shape=(None, 6, 1), name='input')
h1 = tf.keras.layers.Conv2D(128, (1, 6), name="conv2d_1", strides=(1, 1), activation="relu")(x)
h2 = tf.keras.layers.Conv2D(128, (6, 1), name="conv2d_2", strides=(1, 1), activation="relu")(x)
h1 = tf.keras.layers.Reshape((1, 128))(h1)
h2 = tf.keras.layers.Reshape((1, 6*128))(h2)
h3 = tf.keras.layers.Concatenate()([h1, h2])
h4 = tf.keras.layers.Dense(100,name='dense-1', activation='relu')(h3)
y1 = tf.keras.layers.Dense(1, name='output1', activation='sigmoid')(h4)
y2 = tf.keras.layers.Dense(1, name='output2', activation='linear')(h4)
net = tf.keras.Model(inputs=[x], outputs=[y1, y2])
opt = tf.keras.optimizers.get('Adagrad')
opt.learning_rate = 0.003
net.compile(loss='mean_squared_error', optimizer=opt, metrics=['accuracy'])
net.summary()
for i in range(len(exp)):
x = np.array(exp[i]).reshape((1,len(exp[i]),6,1))
loss = net.train_on_batch(x, np.array([[exp[i]] + [np.sum(x, where=[False, False, False, False, False, True])]]))
clear_output(wait=True)
print(loss)
I tried different ways to build this model, but cannot find a correct way, and each time I get different errors, the last one is:
non-broadcastable operand with shape (1,151,6,1) doesn't match the broadcast shape (1,151,6,6)
Please advice how to solve this. I realy want to go this way because I want to inspect this model and results, if it is able to find the horizontal and vertical patterns for correlation distribution of event features that have variables in series with other events variables.

How to reproduce the Bottleneck Blocks in Mobilenet V3 with Keras API?

Using Keras API, I am trying to write the MobilenetV3 as explained in this article: https://arxiv.org/pdf/1905.02244.pdf with the architecture as described in this picture:
For that, I need to implement the bottloneck_blocks from the previous article https://arxiv.org/pdf/1801.04381.pdf. See image for architecture:
I managed to glue together the Initial and final Conv layers:
from tensorflow.keras.layers import Input, Conv2D, Add, AvgPool2D, UpSampling2D
first_input = Input(shape=(256, 256, 3))
firt_conv = Conv2D(16,3, strides=2, name="FirstConv2d", padding="same")(first_input)
bneck1 = add_bottleneck_block(firt_conv, 16, 16)
bneck2 = add_bottleneck_block(bneck1, 64, 24, strides=2)
#... Skiping all the other BottleNeck Blocks for simplicity
lastBneck = add_bottleneck_block(second2LastBneck, 960, 160, bneck_depth=5)
middleConv = Conv2D(160, 1 , strides=1, name="MiddleConv", )(bneck3)
pool7 = AvgPool2D(7, strides=1, padding='same', name="7x7Pool")(middleConv)
SecondLastConv = Conv2D(1280, 1, strides=1, name="SecondLastConv")(pool7)
lastConv = Conv2D(3,1, strides=1, name="lastConv1x1")(SecondLastConv)
upScale = UpSampling2D(2)(lastConv) # This layer is application specific for my training.
v3 = tf.keras.models.Model(inputs=[first_input], outputs=upScale)
v3.compile(optimizer='adam', loss=tf.keras.losses.BinaryCrossentropy(),)
v3.summary()
Where the bottleneck_block is given in the next snippet of code (modified from https://towardsdatascience.com/mobilenetv2-inverted-residuals-and-linear-bottlenecks-8a4362f4ffd5)
def bottleneck_block(x, expand=64, squeeze=16, strides=1, bneck_depth=3):
"""
Bottleneck block with Activation and batch normalization commented since
I don't believe this is the issue in my problem
"""
m = tf.keras.layers.Conv2D(expand, (1,1), strides=1)(x)
#m = tf.keras.layers.BatchNormalization()(m)
#m = tf.keras.layers.Activation('relu6')(m)
m = tf.keras.layers.DepthwiseConv2D(bneck_depth, padding='same', strides=strides)(m)
#m = tf.keras.layers.BatchNormalization()(m)
#m = Activation('relu6')(m)
m = tf.keras.layers.Conv2D(squeeze, (1,1), strides=1)(m)
#m = tf.keras.layers.BatchNormalization()(m)
return tf.keras.layers.Add()([m, x])
However, in bneck2 I get the following error:
ValueError: Operands could not be broadcast together with shapes (16, 16, 24) (128, 128, 16)
I know the error means the dimension of the inputs and outputs are off, but I don't know how to fix it to structure the network as the MobileNetV3.
What am I missing here?
For reference, here is source code in the tensorflow repo for the same network: https://github.com/tensorflow/models/blob/a174bf5b1db0e2c1e04697ff5aae5182bd1c60e7/research/slim/nets/mobilenet/mobilenet_v3.py#L130

The Solution is to modify the bottleneck_block as described in the V3 author's repo:
import tensorflow as tf
def bottleneck_block(x, expand=64, squeeze=16, strides=1, bneck_depth=3, se=False):
"""
se stands for squeeze_excite
"""
m = tf.keras.layers.Conv2D(expand, (1,1), strides=1)(x)
m = tf.keras.layers.BatchNormalization()(m)
#m = tf.keras.layers.Activation('relu6')(m)
m = tf.keras.layers.DepthwiseConv2D(bneck_depth, padding='same', strides=strides)(m)
m = tf.keras.layers.BatchNormalization()(m)
#m = Activation('relu6')(m)
if se:
m = squeeze_excite_block(m, ratio=4)
m = tf.keras.layers.Conv2D(squeeze, (1,1), strides=1, padding='same')(m)
m = tf.keras.layers.BatchNormalization()(m)
if (
# stride check enforces that we don't add residuals when spatial
# dimensions are None
strides == 1 and
# Depth matches
m.get_shape().as_list()[3] == x.get_shape().as_list()[3]
):
m = tf.keras.layers.Add()([m, x])
return m
The check in dimension and stride prevents the error I initially got when adding two nets that do not match the dimension

In your bottolneck layers, there are Add() ops.
Now, Add expects two tensors with the same shape. But, as you have skipped so many layers when this line is run, tf.keras.layers.Add()([m, x]) - m and x have different dimensions.
So, either design a smaller network with fewer layers or just implement all of the intermediate layers.

"Only one input size may be -1, not both 0 and 2" error in Keras

This is summary of my model.
My model is basically similar to a convolution network.
I want my model to work regardless of the width of the input. So the width size appears as None.
and I want to attach a decoder to my model.
However, when I attach the decoder, an error occurs. (If I don't attach the decoder, the program works fine)
This part is my decoder part (below)
if args.decoder==True:
decoder = ConvCapsuleLayer(kernel_size=args.kernel_size, num_capsule=4, num_atoms=1, strides=1, padding='same',
routings=1)(conv_cap)
_, H, W, C, A = decoder.get_shape()
y = layers.Input(shape=(n_class,))
masked_by_y = Mask()([decoder, y])
masked = Mask()(decoder)
def shared_decoder(mask_layer):
recon_1 = layers.Conv2DTranspose(4, (5,5), strides=(2, 2), padding='same', kernel_initializer='he_normal', name='decoder_1', activation='relu')(mask_layer)
recon_2 = layers.Conv2DTranspose(8, (5,5), strides=(2, 2), padding='same', kernel_initializer='he_normal', name='decoder_2', activation='relu')(recon_1)
recon_3 = layers.Conv2DTranspose(1, (1,1), strides=(1, 1), padding='same', kernel_initializer='he_normal', name='decoder_3', activation='linear')(recon_2)
return recon_3
if args.decoder==True:
train_model = models.Model(inputs=[x, y], outputs=[out_seg, shared_decoder(masked_by_y)]) # [x:image,y: mask] // [out_seg:length, reconstruction output]
eval_model = models.Model(x, [out_seg, shared_decoder(masked)])
else:
train_model = models.Model(inputs=x, outputs=out_seg)
eval_model = models.Model(inputs=x, outputs=out_seg)
return train_model, eval_model
mask_1 is my Mask layer.
If a label is given, only the channel of the label is returned. (masked_by_y)
If a label is not given, this layer only returns channel with the largest sum of the element values ​​in conv_capsule_layer_1. (masked)
The shape of conv_capsule_layer_1 is (batch_size = None, height = 50, width = None, num_channel = 4, 1)
That is, the mask layer returns a channel having the largest sum of the element values ​​among the four channels.
Then, use Conv2DTranspose to make it equal to the size of the original input using the returned value (output of mask layer).
However, the following error occurs
InvalidArgumentError (see above for traceback): Only one input size may be -1, not both 0 and 2
[[Node: mask_1/Reshape_1 = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mask_1/boolean_mask/Gather, mask_1/Reshape_1/shape)]]
How can I make the length variable not using -1? I already tried that non_zero_masked = K.reshape(non_zero,[-1, masked.shape[1], masked.shaped[2],1])
This is function name is call in my Mask layer
def call(self, inputs, **kwargs):
if type(inputs) is list: # true label is provided with shape = [None, n_classes], i.e. one-hot code.
assert len(inputs) == 2
inputs, mask = inputs
inputs = K.squeeze(inputs, axis=-1) # [batch, input_height, input_width, num_cap, num_atom] -> [batch, input_height, input_width, num_cap]
else: # if no true label, mask by the max length of capsules. Mainly used for prediction
inputs = K.squeeze(inputs, axis=-1) #[batch, input_height, input_width, num_cap]
x = K.softmax(K.sqrt(K.sum(K.square(inputs), axis=(1,2)) + K.epsilon())) # x: [batch, 4]
mask = K.one_hot(indices=K.argmax(x, 1), num_classes=x.get_shape().as_list()[1]) # mask: [batch,4]
expand_mask = K.reshape(mask,[-1,1,1,mask.shape[1]]) #[batch_size, 1, 1, num_class]
masked = inputs*expand_mask
non_zero = tf.boolean_mask(masked, tf.not_equal(masked,0))
non_zero_masked = K.reshape(non_zero,[-1, masked.shape[1], -1,1])
return non_zero_masked
Does anybody know why this error is happening? How can I solve it?

CNTK: cuDNN failure 7: CUDNN_STATUS_MAPPING_ERROR

i'm trying to create a simple convolution from an 224 * 244 * 3 image down to and 56 * 56 * 3 tensor which i want to compare to another image.
For that purpose i create a composite reader
scale = ImageDeserializer.scale(width=224,
height=224,
channels=3,
scale_mode="pad",
pad_value=114,
interpolations='linear')
scale2 = ImageDeserializer.scale(width=56,
height=56,
channels=3,
scale_mode="pad",
pad_value=114,
interpolations='linear')
image_source = ImageDeserializer(os.path.join(path, "images_map.txt"))
image_source.ignore_labels()
image_source.map_features('features', [scale])
mask_source = ImageDeserializer(os.path.join(path, "images_mask_map.txt"))
mask_source.ignore_labels()
mask_source.map_features('mask', [scale2])
return MinibatchSource([image_source, mask_source])
and with that reader a create an input map
input_map = {
input_var: reader_train["features"],
input_var_mask: reader_train["mask"]
}
the cnn looks like this
conv1 = cntk.layers.Convolution((5, 5), filterdims[0], pad=True, activation=cntk.ops.relu)(input_var)
maxpool1 = cntk.layers.MaxPooling((2, 2), (2, 2))(conv1)
conv2 = cntk.layers.Convolution((4, 4), filterdims[1], pad=True, activation=cntk.ops.relu)(maxpool1)
maxpool2 = cntk.layers.MaxPooling((2, 2), (2, 2))(conv2)
conv3 = cntk.layers.Convolution((4, 4), 3, pad=True, activation=cntk.ops.relu)(maxpool2)
return conv3 # shape is (3, 56, 56) conv3 = z in the error equation
with inputs
input_var = cntk.ops.input_variable((3, 224, 224), np.float32)
input_var_mask = cntk.ops.input_variable((3, 56, 56), np.float32)
and the error function
f2 = cntk.ops.element_times(cntk.ops.constant(0.00390625), input_var_mask, name="f2")
err = cntk.ops.reshape(cntk.ops.minus(z, f2), (56 * 56 * 3))
sq_err = cntk.ops.element_times(err, err)
mse = cntk.ops.reduce_mean(sq_err)
rmse_loss = cntk.ops.sqrt(mse)
rmse_eval = cntk.ops.sqrt(mse)
When i train to train it everything works fine until
data = reader_train.next_minibatch(min(minibatch_size, epoch_size - sample_count), input_map=input_map) # fetch minibatch.
trainer.train_minibatch(data) # Error as in title
where i get the
cuDNN failure 7: CUDNN_STATUS_MAPPING_ERROR ; GPU=0 ; hostname=STEPHENPC
train_minibatch_overload_for_minibatchdata
return _cntk_py.Trainer_train_minibatch_overload_for_minibatchdata(self, *args)
RuntimeError: cuDNN failure 7: CUDNN_STATUS_MAPPING_ERROR ; GPU=0 ; hostname=STEPHENPC ; expr=err
cudaStreamDestroy failed (PrefetchGPUDataTransferer dtor): an illegal memory access was encountered (cuda error 77)
Can some help me and tell me the cause of the error?
Thanks in advance

According to other projects that have run into this issue (see for example here) this may indicate a bug in the cudnn library. CNTK does not use texture memory so it is best to report this problem to NVidia.