In the following code segment, even though the initial values and expected_shape do not match, the code runs and gives output as 1.0. Here is the code.
import tensorflow as tf
import numpy as np
X = tf.placeholder(dtype=tf.float32)
y = tf.placeholder(dtype=tf.float32)
W = tf.Variable(0.0, expected_shape=(3,1))
b = tf.Variable(1.0)
sess = tf.Session()
init = tf.global_variables_initializer()
sess.run(init)
print(sess.run(tf.add(W,b)))
Isn't it supposed to throw an error?
As of now, it is advised to make use of tf.get_variable and avoid tf.Variable as much as possible.
Now coming to your question on why expected_shape is not having any effect, if you look into source code, it is mentioned as it is Deprecated and ignored. If you further look into the function of _init_from_args, the parameter of expected_shape is totally ignored and that value is not used for further processing.
Related
I would like to use TensorFlow's eager execution functionality to optimize the components of a vector. In all the documented examples, each trainable variable is just a scalar, with collections represented by lists of these. However, the loss function I have in mind involves performing vector manipulations on those components, and so this is inconvenient.
For example, let us use the Adam optimizer to normalize a 3-component vector:
import tensorflow as tf
import tensorflow.contrib.eager as tfe
import numpy as np
tf.enable_eager_execution()
def normalize(din=[2.0,1.0,0.0], lr=0.001,
nsteps=100):
d = tfe.Variable(din)
def loss(dvec):
return tf.sqrt((1.0 - tf.tensordot(dvec, dvec, 1))**2)
def grad(dvec):
with tf.GradientTape() as tape:
loss_val = loss(dvec)
return tape.gradient(loss_val, dvec)
optimizer = tf.train.AdamOptimizer(learning_rate=lr)
for i in range(nsteps):
grads = grad(d)
optimizer.apply_gradients(zip(grads, d)) #Throws error
return d
This code correctly computes the required gradients. However, the "optimizer.apply_gradients" line throws some kind of error, seemingly no matter what I do, essentially because tfe.Variable is not an iterable.
In this specific example the error is "AttributeError: Tensor.name is meaningless when eager execution is enabled". We could also try, for example,
zip(grads, [d[i] for i in range(3)])
instead of d, but then the interpreter complains that d is not iterable.
What is the correct way to pair grads with d?
Optimizer.apply_gradients requires its first argument to be a list of (gradient, variable) pairs.
In the code above, neither grads nor d is a list (try print(type(grads)) for example), so the error is from the call to zip. I think what you want instead is:
optimizer.apply_gradients(zip([grads], [d]))
Or, more simply:
optimizer.apply_gradients([(grads, d)])
Also, FYI, as eager execution is stabilizing more things are moving out of the experimental "contrib" namespace, so you don't need the tfe module for your example (tf.Variable will work just fine) if you're using a recent version of TensorFlow (1.11, 1.12 etc.). Making your whole program look like:
import tensorflow as tf
import numpy as np
tf.enable_eager_execution()
def normalize(din=[2.0,1.0,0.0], lr=0.001,
nsteps=100):
d = tf.Variable(din)
def loss(dvec):
return tf.sqrt((1.0 - tf.tensordot(dvec, dvec, 1))**2)
def grad(dvec):
with tf.GradientTape() as tape:
loss_val = loss(dvec)
return tape.gradient(loss_val, dvec)
optimizer = tf.train.AdamOptimizer(learning_rate=lr)
for i in range(nsteps):
dd = grad(d)
optimizer.apply_gradients([(dd, d)])
return d
Hope that helps!
I have a complicated use case which I've distilled down to just incrementing a variable in Tensorflow.
a = tf.Variable(1, trainable=False)
b = tf.constant(2)
a = tf.assign_add(a, b)
In [32]: type(a)
Out[32]: tensorflow.python.framework.ops.Tensor
My actual use case is actually generating a new random tensor under certain conditions each time my custom Keras layer is called, but seems like it boils down to a variable turning into a tensor if I do anything to it. Is the correct use case to wrap each a = tf.Variable(tf.assign(a, b)) and have a change everytime my keras layer is called?
You are overthinking it. tf.assign_add returns an op that adds to a variable. The fact that it also return the resulting value is for convenience only — the variable is affected.
Example:
import tensorflow as tf
a = tf.Variable(1, trainable=False)
b = tf.constant(2)
c = tf.assign_add(a, b)
sess = tf.InteractiveSession()
tf.global_variables_initializer().run()
print(sess.run(a))
# 1: the original value
print(sess.run(c))
# 3: the result of the addition
print(sess.run(a))
# 3: OK, the variable has indeed been added to
I want to optimize a cost function. This cost function contains variables and other parameters that are not variables. This non-variable parameters are obtained from the variables.
Here is a toy example that illustrates the point:
import numpy as np
import tensorflow as tf
r_init = np.array([5.0,6.0])
x = tf.get_variable("x_var", initializer = r_init[0], trainable = True)
y = tf.get_variable("y_var", initializer = r_init[1], trainable = True)
def cost(x,y):
a = x
return a*((x-1.0)**2+(y-1.0)**2)
train_op = tf.train.AdamOptimizer(learning_rate=0.05).minimize(cost(x,y))
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for i in range(100):
print(sess.run([cost(x,y), train_op]))
print('x=', x.eval(session=sess))
print('y=', y.eval(session=sess))
As you can see, the parameter a is defined from the variable x, on the other hand a should not be a variable, I want the optimizer to see it as a constant. This constant should be updated as the variable x is updated in the optimization process.
How can I define a non-variable parameter a from the variable x? I am making this up, but intuitively, what comes to my mind is something like:
a = tf.to_constant(x)
Any ideas?
You are looking for tf.stop_gradient:
a = tf.stop_gradient(x)
Quoting the docs,
This is useful any time you want to compute a value with TensorFlow but need to pretend that the value was a constant.
How can one specify a custom initializer as the third argument for tf.get_variable()? Specifically, I have a variable y which I want to initialize using another (already initialized) variable x.
This is easy to do using tf.Variable(), just say, y = tf.Variable(x.initialized_value()). But I couldn't find an analog in the documentation for tf.get_variable().
You can use x.initialized_value() as well. For example:
import tensorflow as tf
x = tf.Variable(1.0)
y = tf.get_variable('y', initializer=x.initialized_value())
with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
print(sess.run(y)) # prints 1.0
Say in tensorflow, I created a variable by
C = tf.Variable(tf.random_uniform([n_sample, n_sample], -1, 1), name='C'),
now I want to get a pointer to the first column of the variable, is there anyway I could do that? Would tf.slice(C, [0,0], [n_sample,1]) give me what I want or it will just create another variable with value stored in C.
The reason that I want to do it is because my optimization function is dependent on both C and each columns of C.
As far as I know you can't really get access to the data itself (i.e. like a pointer). The reasoning being is that the code will be data agnostic so that it can pass around the data to different CPUs or GPUs without you worrying about that part (or you could specify device to use but that gets cumbersome).
So tf.slice would be the correct function to use.
you could do :
for i in range(n_sample):
curr_slice = tf.slice(C, [i,0], [n_sample,1])
do_something(curr_slice)
this isn't the most efficient version but it's what you asked for in the comments.
for i inVectorized range(n_sample):approach
curr_sliceloss = tf.slice(C, [i,0], [n_sample,1])
y.assign_add( tf.nn.l2_loss(tf.sub(curr_slice,X - tf.matmul(X,curr_slice)C)) + lambdalamb * tf.nn.l2_loss(curr_slice) C)
loss=tf.reduce_sum(y)
Vectorized approach much cleaner:
loss = tf.nn.l2_loss(X - tf.matmul(X,C)) + lamb * tf.nn.l2_loss(C)
train_step = tf.train.GradientDescentOptimizer(0.01).minimize(loss)
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
sess.run(train_step)
You might need to initialize the some of the values by creating placeholders.
Alternatively I couldn't find it in skflow yet but in scikit learn it's a simple 3 liner.
from sklearn.linear_model import Ridge
clf = Ridge(alpha=1.0)
clf.fit(X, W)