Develop Reference

Dimension mismatch during Keras to ONNX conversion (2D output)

I am observing a dimension mismatch in Keras to ONNX conversion.
I saved my model as a .h5 file.
It can successfully be saved and loaded again.
However, when converting it to an ONNX model, I get different output dimensions.
I think I experience this due to 2D output, because one of my output dimension is simply disappeared.
Loading Keras model...
>>> keras_model = load_model('model_checkpoints/DGCNN_modelbest_with_noise.h5')
>>> keras_output = keras_model.output
>>> keras_output
<tf.Tensor 'dense_2/truediv_5:0' shape=(None, 432, 5) dtype=float32>
Converting Keras model to ONNX...
>>> input_keras_model = 'model_checkpoints/DGCNN_modelbest_with_noise.h5'
>>> output_onnx_model = 'model_checkpoints/DGCNN_modelbest_with_noise.onnx'
>>> keras_model = load_model(input_keras_model)
>>> onnx_model = onnxmltools.convert_keras(keras_model)
>>> onnxmltools.utils.save_model(onnx_model, output_onnx_model)
Loading ONNX model...
>>> model = onnx.load("model_checkpoints/DGCNN_modelbest_with_noise.onnx")
>>> for _output in model.graph.output:
... m_dict = MessageToDict(_output)
... dim_info = m_dict.get("type").get("tensorType").get("shape").get("dim")
... output_shape = [d.get("dimValue") for d in dim_info]
... print(m_dict["name"])
... print(output_shape)
...
dense_2
[None, None, '5']
Any suggestions?
What am I doing wrong?
I don't see many examples for multidimensional output layers. Is this the reason?
Thank you for your time.

I have no problem following the example I try by loading and run it still have the same results but I using the pdb format.
The pdb format is a molecular format that includes sutures and using from model.save( ... )
### ( 1 ) : Save and convert
import tensorflow as tf
import tf2onnx
import onnx
model = tf.keras.Sequential()
#model.add(tf.keras.layers.InputLayer(input_shape=(1, 100, 100, 3), name='DekDee Input'))
model.add(tf.compat.v1.layers.dense(4, activation="relu", name='output1'))
Name specific and types is significant
input_signature = [tf.TensorSpec([3, 3], tf.float32, name='input1')]
#Use from_function for tf functions
onnx_model, _ = tf2onnx.convert.from_keras(model, input_signature, opset=13)
onnx.save(onnx_model, "F:\\models\\onnx\\model.onnx")
OR
model.save("F:\\models\\onnx\\modelpb")
Command : python -m tf2onnx.convert --saved-model "F:\models\onnx\modelpb" --output "F:\\models\\onnx\\model_2.onnx" --opset 13
### ( 2 ) : Load and run
import onnxruntime as ort
import numpy as np
import tensorflow as tf
Change shapes and types to match model
input1 = np.zeros((3, 3), np.float32)
sess = ort.InferenceSession("F:\\models\\onnx\\model.onnx", providers=["CUDAExecutionProvider"])
results_ort = sess.run(["output1"], {"input1": input1})
F:\temp\Python\tf_onnx>python onnx_verification_test_2.py
[array([[0., 0., 0., 0.],
[0., 0., 0., 0.],
[0., 0., 0., 0.]], dtype=float32)]

I get the exception "input that isn't a symbolic tensor". Can I convert numpy.ndarray to a tensor?

I am currently trying to implement ArcFace as loss function for my Inception v4 model. I use a pretrained model from tensorflow.slim, which returns the pre-activation logits and endpoints as result as stated in Inception_v4.py:
Returns:
>>net: a Tensor with the logits (pre-softmax activations) if num_classes
is a non-zero integer, or the non-dropped input to the logits layer
if num_classes is 0 or None.
>>end_points: the set of end_points from the inception model.
I don't know, if I understand the paper correctly, but it seems like I could use those logits as input for my ArcFace layer. I can't post a picture, but here is the architecture of ArcFace.
My ArcFace layer looks like this:
from keras.layers import Layer
from keras import regularizers
class ArcFace(Layer):
def __init__(self, n_classes, s=64.0, m=0.50, regularizer=None, **kwargs):
super(ArcFace, self).__init__(**kwargs)
self.n_classes = n_classes
self.s = s
self.m = m
self.regularizer = regularizers.get(regularizer)
def build(self, input_shape):
super(ArcFace, self).build(input_shape[0])
self.W = self.add_weight(name='W',
shape=(input_shape[0][-1], self.n_classes),
initializer='glorot_uniform',
trainable=True,
regularizer=self.regularizer)
def call(self, inputs):
x, y = inputs
x = tf.nn.l2_normalize(x, axis=1)
W = tf.nn.l2_normalize(self.W, axis=0)
logits = tf.matmul(x, W)
if self.m>0:
logits = self.apply_arcface_margin(logits, y, self.m)
logits *= self.s
out = tf.nn.softmax(logits)
return out
def apply_arcface_margin(self, cosine_sim, one_hot_labels, arcface_margin):
theta = tf.acos(cosine_sim, name='acos')
selected_labels = tf.where(tf.greater(theta, math.pi - arcface_margin),
tf.zeros_like(one_hot_labels),
one_hot_labels,
name='selected_labels')
final_theta = tf.where(tf.cast(selected_labels, dtype=tf.bool),
theta + arcface_margin,
theta,
name='final_theta')
return tf.cos(final_theta, name='cosine_sim_with_margin')
def compute_output_shape(self, input_shape):
return (None, self.n_classes)
When I am running the program I get:
Layer arc_face_1 was called with an input that isn't a symbolic tensor. Received type: <class 'numpy.ndarray'>. Full input: [<tf.Tensor 'input_1:0' shape=(?, 1000) dtype=float32>, array([[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]])].
I don't understand the error, cause for me it looks like a tensor. Is there a way to convert the input? If you need further information, just let me know.
I use:
TF version 1.15
Keras version 2.3.1
Python 3.7
Thank you in advance!

I have tried this code on tf 2.8 and the issue does not exist please upgrade your tf to 2.x version to resolve this.

Getting Keras / Tensorflow to output OneHotCategorical, but operation has None for gradient

Problem description
I have inputs x that are indicator variables, and outputs y, where each row is a random one-hot vector that depends on the values of x (data sample shown below).
I want to train a model that essentially learns the probabilistic relationship between x and y in the form of per-column weights. The model must "choose" one, and only one, indicator to output. My current approach is to sample a categorical random variable and produce a one-hot vector as a prediction.
The issue is that I'm getting an error ValueError: An operation has `None` for gradient when I try to train my Keras model.
I find this error odd, because I've trained mixture networks using Keras and Tensorflow, which use tf.contrib.distributions.Categorical, and I did not run into any gradient-related issues.
Code
Experiment
import tensorflow as tf
import tensorflow.contrib.distributions as tfd
import numpy as np
from keras import backend as K
from keras.layers import Layer
from keras.models import Sequential
from keras.utils import to_categorical
def make_xy_prob(rng, size=10000):
rng = np.random.RandomState(rng) if isinstance(rng, int) else rng
cols = 3
weights = np.array([[1, 2, 3]])
# generate data and drop zeros for now
x = rng.choice(2, (size, cols))
is_zeros = x.sum(axis=1) == 0
x = x[~is_zeros]
# use weights to create probabilities for determining y
weighted_x = x * weights
prob_x = weighted_x / weighted_x.sum(axis=1, keepdims=True)
y = np.row_stack([to_categorical(rng.choice(cols, p=p), cols) for p in prob_x])
# add zeros back and shuffle
zeros = np.zeros(((size - len(x), cols)))
x = np.row_stack([x, zeros])
y = np.row_stack([y, zeros])
shuffle_idx = rng.permutation(size)
x = x[shuffle_idx]
y = y[shuffle_idx]
return x, y
class OneHotGate(Layer):
def build(self, input_shape):
self.kernel = self.add_weight(name='kernel', shape=(1, input_shape[1]), initializer='ones')
def call(self, x):
zero_cond = x < 1
x_shape = tf.shape(x)
# weight indicators so that more probability is assigned to more likely columns
weighted_x = x * self.kernel
# fill zeros with -inf so that zero probability is assigned to that column
ninf_fill = tf.fill(x_shape, -np.inf)
masked_x = tf.where(zero_cond, ninf_fill, weighted_x)
onehot_gate = tf.squeeze(tfd.OneHotCategorical(logits=masked_x, dtype=x.dtype).sample(1))
# fill gate with zeros where input was originally zero
zeros_fill = tf.fill(x_shape, 0.0)
masked_gate = tf.where(zero_cond, zeros_fill, onehot_gate)
return masked_gate
def experiment(epochs=10):
K.clear_session()
rng = np.random.RandomState(2)
X, y = make_xy_prob(rng)
input_shape = (X.shape[1], )
model = Sequential()
gate_layer = OneHotGate(input_shape=input_shape)
model.add(gate_layer)
model.compile('adam', 'categorical_crossentropy')
model.fit(X, y, 64, epochs, verbose=1)
Data sample
>>> x
array([[1., 1., 1.],
[0., 1., 0.],
[1., 0., 1.],
...,
[1., 1., 1.],
[1., 1., 1.],
[1., 1., 0.]])
>>> y
array([[0., 0., 1.],
[0., 1., 0.],
[1., 0., 0.],
...,
[0., 0., 1.],
[1., 0., 0.],
[1., 0., 0.]])
Error
ValueError: An operation has `None` for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.

The problem lies in the fact that in OneHotCategorical performs a discontinuous sampling - what causes gradient computation to fail. In order to replace this discontinuous sampling with a continuous (relaxed) version one may try to use RelaxedOneHotCategorical (which is based on interesting Gumbel Softmax technique).

Input an integer with placeholder in tensorflow?

I want to feed a batch_size integer as a placeholder in Tensorflow. But it does not act as an integer. Consider the following example:
import tensorflow as tf
max_length = 5
batch_size = 3
batch_size_placeholder = tf.placeholder(dtype=tf.int32)
mask_0 = tf.one_hot(indices=[0]*batch_size_placeholder, depth=max_length, on_value=0., off_value=1.)
mask_1 = tf.one_hot(indices=[0]*batch_size, depth=max_length, on_value=0., off_value=1.)
# new session
with tf.Session() as sess:
feed = {batch_size_placeholder : 3}
batch, mask0, mask1 = sess.run([
batch_size_placeholder, mask_0, mask_1
], feed_dict=feed)
When I print the values of batch, mask0 and mask1 I have the following:
print(batch)
>>> array(3, dtype=int32)
print(mask0)
>>> array([[0., 1., 1., 1., 1.]], dtype=float32)
print(mask1)
>>> array([[0., 1., 1., 1., 1.],
[0., 1., 1., 1., 1.],
[0., 1., 1., 1., 1.]], dtype=float32)
Indeed I thought mask0 and mask1 must be the same, but it seems that Tensorflow does not treat batch_size_placeholder as an integer. I believe it would be a tensor, but is there anyway that I can use it as an integer in my computations?
Is there anyway I can fix this problem? Just FYI, I used tf.one_hot as just an example, I want to run train/validation during training in my code where I will need a lot of other computations with different values for batch_size in training and in validation steps.
Any help would be appreciated.

In pure python usage, [0]*3 will be [0,0,0]. However, batch_size_placeholder is a placeholder, during the graph execution, it will be a tensor. [0]*tensor will be regarded as tensor multiplication. In your case, it will be a 1-d tensor which has 0 value. To correctly use batch_size_placeholder, you should create a tensor which has the same length as batch_size_placeholder.
mask_0 = tf.one_hot(tf.zeros(batch_size_placeholder, dtype=tf.int32), depth=max_length, on_value=0., off_value=1.)
It will have the same result as mask_1.
A simple example to show the difference.
batch_size_placeholder = tf.placeholder(dtype=tf.int32)
a = [0]*batch_size_placeholder
b = tf.zeros(batch_size_placeholder, dtype=tf.int32)
with tf.Session() as sess:
print(sess.run([a, b], feed_dict={batch_size_placeholder : 3}))
# [array([0], dtype=int32), array([0, 0, 0], dtype=int32)]

How to get integer labels from MNIST examples after specifying one_hot=True?

I have been attempting this tutorial on Youtube (explination of .cls and .labels at 1m31s) which is just a simple MNIST classifier model. But I was unable to complete it due to an apparently missing function in Tensorflow.
>>>from tensorflow.examples.tutorials.mnist import input_data
>>>data = input_data.read_data_sets("data/MNIST", one_hot=True)
>>>one_hot_labels = data.test.labels #mat shape=(num_images X num_classes)
>>>cls_labels = data.test.cls #mat shape=(num_images X 1)
Traceback (most recent call last):
File "/home/file.py", line 5, in <module>
cls_labels = data.test.cls
AttributeError: 'DataSet' object has no attribute 'cls'
After searching on Google for ".cls" reference in TF, I was unable to find any information pertaining to it.
A dirty example that made things work:
>>>data = input_data.read_data_sets("data/MNIST", one_hot=True)
>>>data2 = input_data.read_data_sets("data/MNIST")
>>>one_hot_labels = data.test.labels #mat shape=(num_images X num_classes)
>>>cls_labels = data2.test.labels #mat shape=(num_images X 1)
I am using Tensorflow 0.10.0 on Linux and am wondering if the .cls option has been removed?
If so, is there an alternative method for encoding an array of classifier names from an array of one_hot vectors?
Thanks

Your labels are in this type of array (one hot) for example :
array([[ 0., 0., 0., ..., 1., 0., 0.],
[ 0., 0., 1., ..., 0., 0., 0.],
[ 0., 1., 0., ..., 0., 0., 0.],
The number 1. is in the position of the array where tells you which label is.
To get a integer label from this data you have to get the index with:
data.test.cls = np.argmax(data.test.labels, axis=1)

Currently we use attribute images for image data and labels for classes(labels). For example,
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("data/MNIST", one_hot=True)
# data
images = mnist.test.images
# label
labels = mnist.test.labels
# without one-hot
mnist = input_data.read_data_sets("data/MNIST", one_hot=False)
# original data
images = mnist.test.images.reshape([-1, 28, 28])
print(images.shape)
# label
labels = mnist.test.labels
print(labels)