Develop Reference

I get the exception "input that isn't a symbolic tensor". Can I convert numpy.ndarray to a tensor?

I am currently trying to implement ArcFace as loss function for my Inception v4 model. I use a pretrained model from tensorflow.slim, which returns the pre-activation logits and endpoints as result as stated in Inception_v4.py:
Returns:
>>net: a Tensor with the logits (pre-softmax activations) if num_classes
is a non-zero integer, or the non-dropped input to the logits layer
if num_classes is 0 or None.
>>end_points: the set of end_points from the inception model.
I don't know, if I understand the paper correctly, but it seems like I could use those logits as input for my ArcFace layer. I can't post a picture, but here is the architecture of ArcFace.
My ArcFace layer looks like this:
from keras.layers import Layer
from keras import regularizers
class ArcFace(Layer):
def __init__(self, n_classes, s=64.0, m=0.50, regularizer=None, **kwargs):
super(ArcFace, self).__init__(**kwargs)
self.n_classes = n_classes
self.s = s
self.m = m
self.regularizer = regularizers.get(regularizer)
def build(self, input_shape):
super(ArcFace, self).build(input_shape[0])
self.W = self.add_weight(name='W',
shape=(input_shape[0][-1], self.n_classes),
initializer='glorot_uniform',
trainable=True,
regularizer=self.regularizer)
def call(self, inputs):
x, y = inputs
x = tf.nn.l2_normalize(x, axis=1)
W = tf.nn.l2_normalize(self.W, axis=0)
logits = tf.matmul(x, W)
if self.m>0:
logits = self.apply_arcface_margin(logits, y, self.m)
logits *= self.s
out = tf.nn.softmax(logits)
return out
def apply_arcface_margin(self, cosine_sim, one_hot_labels, arcface_margin):
theta = tf.acos(cosine_sim, name='acos')
selected_labels = tf.where(tf.greater(theta, math.pi - arcface_margin),
tf.zeros_like(one_hot_labels),
one_hot_labels,
name='selected_labels')
final_theta = tf.where(tf.cast(selected_labels, dtype=tf.bool),
theta + arcface_margin,
theta,
name='final_theta')
return tf.cos(final_theta, name='cosine_sim_with_margin')
def compute_output_shape(self, input_shape):
return (None, self.n_classes)
When I am running the program I get:
Layer arc_face_1 was called with an input that isn't a symbolic tensor. Received type: <class 'numpy.ndarray'>. Full input: [<tf.Tensor 'input_1:0' shape=(?, 1000) dtype=float32>, array([[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]])].
I don't understand the error, cause for me it looks like a tensor. Is there a way to convert the input? If you need further information, just let me know.
I use:
TF version 1.15
Keras version 2.3.1
Python 3.7
Thank you in advance!

I have tried this code on tf 2.8 and the issue does not exist please upgrade your tf to 2.x version to resolve this.

Tensor slicing looses the shape information in TensorFlow

I'm trying to dynamically slice a tensor to automatically adjust its shape for the next iteration. However, I realized that when sliced in Graph mode the shape information of the tensor is lost hence I can not further apply certain operations on it which requires knowing the shape of a given tensor. Below I attached an example code, in my specific example the opt_with_slicing function is within a vectorized_map which is defined in a larger function that takes care of auto differentiation. Since the original function is too large to include here I simplified it accordingly;
a = tf.constant(np.linspace(0.,10.,11,endpoint=True)[::-1])
b = tf.ones((2,10))
def opt_with_slicing(x, some_cutoff: float):
a, b = x
new_size = tf.math.count_nonzero(
tf.cast(a >= some_cutoff, dtype=tf.int32), dtype=tf.int32
)
tf.print(f"new size {new_size}, initial size {a.get_shape()}")
test1 = b[:, :new_size]
test2 = tf.slice(b, [0, 0], [b.get_shape()[0], new_size])
tf.print(f"test1 shape {test1.get_shape()}, test2 shape {test2.get_shape()}")
return test1, test2
tf.function(opt_with_slicing)([a,b], 5.)
# Output:
# new size Tensor("count_nonzero/Cast_1:0", shape=(), dtype=int32), initial size (11,)
# test1 shape (2, None), test2 shape (2, None)
# (<tf.Tensor: shape=(2, 6), dtype=float32, numpy=
# array([[1., 1., 1., 1., 1., 1.],
# [1., 1., 1., 1., 1., 1.]], dtype=float32)>,
# <tf.Tensor: shape=(2, 6), dtype=float32, numpy=
# array([[1., 1., 1., 1., 1., 1.],
# [1., 1., 1., 1., 1., 1.]], dtype=float32)>)
As you can see from the print out the shape information of test1 and test2 is lost and since this is a dynamic operation I have no way to know the new_size prior to the execution. Is there a way to reinstate the shape information of the function without breaking the graph mode?
PS: I tried the same with boolean_mask as well;
mask = tf.greater_equal(a, some_cutoff)
masked_shape = tf.boolean_mask(a, mask).get_shape()[0]
but masked_shape turns out to be None as well.
System info:
Tensorflow v2.5.0
Python v3.8.2

Understanding torch.nn.Flatten

I understand that Flatten removes all of the dimensions except for one. For example, I understand flatten():
> t = torch.ones(4, 3)
> t
tensor([[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.]])
> flatten(t)
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
However, I don't get Flatten, especially I don't get meaning of this snippet from the doc:
>>> input = torch.randn(32, 1, 5, 5)
>>> m = nn.Sequential(
>>> nn.Conv2d(1, 32, 5, 1, 1),
>>> nn.Flatten()
>>> )
>>> output = m(input)
>>> output.size()
torch.Size([32, 288])
I felt the output should have size [160], because 32*5=160.
Q1. So why it outputted size [32,288]?
Q2. I also don't get meaning of shape information given in the doc:
Q3. And also meaning of parameters:

It is a difference in the default behaviour. torch.flatten flattens all dimensions by default, while torch.nn.Flatten flattens all dimensions starting from the second dimension (index 1) by default.
You can see this behaviour in the default values of the start_dim and end_dim arguments. The start_dim argument denotes the first dimension to be flattened (zero-indexed), and the end_dim argument denotes the last dimension to be flattened. So, when start_dim=1, which is the default for torch.nn.Flatten, the first dimension (index 0) is not flattened, but it is included when start_dim=0, which is the default for torch.flatten.
The reason behind this difference is probably because torch.nn.Flatten is intended to be used with torch.nn.Sequential, where typically a series of operations are performed on a batch of inputs, where each input is treated independently of the others. For example, if you have a batch of images and you call torch.nn.Flatten, the typical use case would be to flatten each image separately, and not flatten the whole batch.
If you do want to flatten all dimensions using torch.nn.Flatten, you can simply create the object as torch.nn.Flatten(start_dim=0).
Finally, the shape information in the docs just covers how the shape of the tensor will be affected, illustrating that the first (index 0) dimension is left as it is. So, if you have an input tensor of shape (N, *dims), where *dims is an arbitrary sequence of dimensions, the output tensor will have the shape (N, product of *dims), since all dimensions except the batch dimension are flattened. For example, an input of shape (3,10,10) will have an output of shape (3, 10 x 10) = (3, 100).

Getting Keras / Tensorflow to output OneHotCategorical, but operation has None for gradient

Problem description
I have inputs x that are indicator variables, and outputs y, where each row is a random one-hot vector that depends on the values of x (data sample shown below).
I want to train a model that essentially learns the probabilistic relationship between x and y in the form of per-column weights. The model must "choose" one, and only one, indicator to output. My current approach is to sample a categorical random variable and produce a one-hot vector as a prediction.
The issue is that I'm getting an error ValueError: An operation has `None` for gradient when I try to train my Keras model.
I find this error odd, because I've trained mixture networks using Keras and Tensorflow, which use tf.contrib.distributions.Categorical, and I did not run into any gradient-related issues.
Code
Experiment
import tensorflow as tf
import tensorflow.contrib.distributions as tfd
import numpy as np
from keras import backend as K
from keras.layers import Layer
from keras.models import Sequential
from keras.utils import to_categorical
def make_xy_prob(rng, size=10000):
rng = np.random.RandomState(rng) if isinstance(rng, int) else rng
cols = 3
weights = np.array([[1, 2, 3]])
# generate data and drop zeros for now
x = rng.choice(2, (size, cols))
is_zeros = x.sum(axis=1) == 0
x = x[~is_zeros]
# use weights to create probabilities for determining y
weighted_x = x * weights
prob_x = weighted_x / weighted_x.sum(axis=1, keepdims=True)
y = np.row_stack([to_categorical(rng.choice(cols, p=p), cols) for p in prob_x])
# add zeros back and shuffle
zeros = np.zeros(((size - len(x), cols)))
x = np.row_stack([x, zeros])
y = np.row_stack([y, zeros])
shuffle_idx = rng.permutation(size)
x = x[shuffle_idx]
y = y[shuffle_idx]
return x, y
class OneHotGate(Layer):
def build(self, input_shape):
self.kernel = self.add_weight(name='kernel', shape=(1, input_shape[1]), initializer='ones')
def call(self, x):
zero_cond = x < 1
x_shape = tf.shape(x)
# weight indicators so that more probability is assigned to more likely columns
weighted_x = x * self.kernel
# fill zeros with -inf so that zero probability is assigned to that column
ninf_fill = tf.fill(x_shape, -np.inf)
masked_x = tf.where(zero_cond, ninf_fill, weighted_x)
onehot_gate = tf.squeeze(tfd.OneHotCategorical(logits=masked_x, dtype=x.dtype).sample(1))
# fill gate with zeros where input was originally zero
zeros_fill = tf.fill(x_shape, 0.0)
masked_gate = tf.where(zero_cond, zeros_fill, onehot_gate)
return masked_gate
def experiment(epochs=10):
K.clear_session()
rng = np.random.RandomState(2)
X, y = make_xy_prob(rng)
input_shape = (X.shape[1], )
model = Sequential()
gate_layer = OneHotGate(input_shape=input_shape)
model.add(gate_layer)
model.compile('adam', 'categorical_crossentropy')
model.fit(X, y, 64, epochs, verbose=1)
Data sample
>>> x
array([[1., 1., 1.],
[0., 1., 0.],
[1., 0., 1.],
...,
[1., 1., 1.],
[1., 1., 1.],
[1., 1., 0.]])
>>> y
array([[0., 0., 1.],
[0., 1., 0.],
[1., 0., 0.],
...,
[0., 0., 1.],
[1., 0., 0.],
[1., 0., 0.]])
Error
ValueError: An operation has `None` for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.

The problem lies in the fact that in OneHotCategorical performs a discontinuous sampling - what causes gradient computation to fail. In order to replace this discontinuous sampling with a continuous (relaxed) version one may try to use RelaxedOneHotCategorical (which is based on interesting Gumbel Softmax technique).

How to use reset_states(states) function in Keras?

I'm trying to set the LSTM internal state before training each batch.
I'm sharing my test code and findings, hoping to find an answer and help others that are addressing similar problems.
In particular, for each data I have a feature X (which doesn't change over time) and a sequence P = p1, p2, p3,... p30.
The goal is: given X and p1,p2,p3 predict p4, p5, .. p30.
To this aim, I want to initialize the hidden state of an LSTM with X, as done in several works (e.g., neuraltalk), then the LSTM has to be fit with p1,p2,p3 to predict p4,..,p30.
This initialization is needed before each batch (batch_size=1), therefore I need to have the control of the LSTM states initialization.
Considerint this question Initializing LSTM hidden state Tensorflow/Keras I've tested the following code:
First of all I've added some prints in the reset_states() function defined in recurrent.py, in order to understand what exactly happens.
def reset_states(self, states=None):
if not self.stateful:
raise AttributeError('Layer must be stateful.')
batch_size = self.input_spec[0].shape[0]
if not batch_size:
raise ValueError('If a RNN is stateful, it needs to know '
'its batch size. Specify the batch size '
'of your input tensors: \n'
'- If using a Sequential model, '
'specify the batch size by passing '
'a `batch_input_shape` '
'argument to your first layer.\n'
'- If using the functional API, specify '
'the time dimension by passing a '
'`batch_shape` argument to your Input layer.')
# initialize state if None
if self.states[0] is None:
self.states = [K.zeros((batch_size, self.units))
for _ in self.states]
print "reset states A (all zeros)"
elif states is None:
for state in self.states:
K.set_value(state, np.zeros((batch_size, self.units)))
print "reset states B (all zeros)"
else:
if not isinstance(states, (list, tuple)):
states = [states]
print "reset states C (list or tuple copying)"
if len(states) != len(self.states):
raise ValueError('Layer ' + self.name + ' expects ' +
str(len(self.states)) + ' states, '
'but it received ' + str(len(states)) +
' state values. Input received: ' +
str(states))
for index, (value, state) in enumerate(zip(states, self.states)):
if value.shape != (batch_size, self.units):
raise ValueError('State ' + str(index) +
' is incompatible with layer ' +
self.name + ': expected shape=' +
str((batch_size, self.units)) +
', found shape=' + str(value.shape))
K.set_value(state, value)
print "reset states D (set values)"
print value
print "\n"
Here is the test code:
import tensorflow as tf
from keras.layers import LSTM
from keras.layers import Input
from keras.models import Model
import numpy as np
import keras.backend as K
input = Input(batch_shape=(1,3,1))
lstm_layer = LSTM(10,stateful=True)(input)
>>> reset states A (all zeros)
As you can see, the first print is executed when the lstm layer is created
model = Model(input,lstm_layer)
model.compile(optimizer="adam", loss="mse")
with tf.Session() as sess:
tf.global_variables_initializer().run()
h = sess.run(model.layers[1].states[0])
c = sess.run(model.layers[1].states[1])
print h
>>> [[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]
print c
>>> [[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]
The internal states have been set to all zeros.
As an alternative the function reset_states() can be used
model.layers[1].reset_states()
>>> reset states B (all zeros)
The second message has been printed in this case. Everything seem to work correctly.
Now I want to set the states with arbitrary values.
new_h = K.variable(value=np.ones((1, 10)))
new_c = K.variable(value=np.ones((1, 10))+1)
model.layers[1].states[0] = new_h
model.layers[1].states[1] = new_c
with tf.Session() as sess:
tf.global_variables_initializer().run()
h = sess.run(model.layers[1].states[0])
c = sess.run(model.layers[1].states[1])
print h
>>> [[ 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]]
print c
>>> [[ 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.]]
Ok, I've successfully set both hidden states with my vectors of all one and all two.
However, it worth to exploit the class function reset_states() which takes as input the states.
This function exploits the function K.set_values(x,values) which expects 'values' to be a numpy array.
new_h_5 = np.zeros((1,10))+5
new_c_24 = np.zeros((1,10))+24
model.layers[1].reset_states([new_h_5,new_c_24])
It seems to work, indeed the output is:
>>> reset states D (set values)
>>> [[ 5. 5. 5. 5. 5. 5. 5. 5. 5. 5.]]
>>>
>>>
>>>
>>>
>>> reset states D (set values)
>>> [[ 24. 24. 24. 24. 24. 24. 24. 24. 24. 24.]]
However, if i want to check if the states have been initializated I find the previous initialization values (all one, all two).
with tf.Session() as sess:
tf.global_variables_initializer().run()
hh = sess.run(model.layers[1].states[0])
cc = sess.run(model.layers[1].states[1])
print hh
>>> [[ 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]]
print cc
>>> [[ 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.]]
What is exactly happening here? Why the function seems working according to the prints but doesn't change the values of the internal states?

As you may read here, value parameter sets a value by which a variable should be initialized. So when you call tf.global_variables_initializer().run() your states are initialized with values defined here:
new_h = K.variable(value=np.ones((1, 10)))
new_c = K.variable(value=np.ones((1, 10))+1)
Edit:
It seemed obvious for me but once again I will explain why reset_states doesn't work.
Variable definition: When you defined your inner states to be variables initialized by a certain value the n this certain vaklue will be set every time you call variable_initializer.
Reset states: it will update a current value of this variable but it will not change a default value of initializer. In order to do that you need to reassign this states by yet another variable with a given states set as default.