There is a function tf.get_variable('name') which allows to "implicitly" pass parameters into function like:
def function(sess, feed):
with tf.variable_scope('training', reuse=True):
cost = tf.get_variable('cost')
value = sess.run(cost, feed_dict=feed)
# other statements
But what if one want to pass a tf.placeholder into function? Is there same mechanism for placeholders, i.e. something like tf.get_placeholder():
def function(sess, cost, X_train, y_train):
# Note this is NOT a valid TF code
with tf.variable_scope('training', reuse=True):
features = tf.get_placeholder('features')
labels = tf.get_placeholder('labels')
feed = {features: X_train, labels: y_train}
value = sess.run(cost, feed_dict=feed)
print('Cost: %s' % value)
Or it doesn't make too much sense to do it and better to just construct placeholders inside of function?
Placeholders are just... placeholders. It's pointless "getting" a placeholder as if it has some sort of state (that's what get variable does, returns a variable in its current state).
Just use the same python variable everywhere.
Also, if you don't want to pass a python variable because your method signaturl becomes ugly, you can exploit the fact that you're building a graph and the graph itself contains the information about the declared placeholders.
You can do something like:
#define your placeholder
a = tf.placeholder(tf.float32, name="asd")
# then, when you need it, fetch if from the graph
graph = tf.get_default_graph()
placeholder = graph.get_tensor_by_name("asd:0")
Aside the fact that if you are working in the same script you should not need this, you can do that by getting the tensor by name, as in Tensorflow: How to get a tensor by name?
For instance
p = tf.placeholder(tf.float32)
p2 = tf.get_default_graph().get_tensor_by_name(p.name)
assert p == p2
Related
From the tensorflow tutorial section https://www.tensorflow.org/guide/low_level_intro#feeding
the following code creates the placeholders and assign it to variables 'x' and 'y' and is passed to the run method.
x = tf.placeholder(tf.float32)
y = tf.placeholder(tf.float32)
z = x + y
print(sess.run(z, feed_dict={x: 3, y: 4.5}))
How does the sess.run() method know the name of the variables 'x' and 'y'. ie. How does the run method know the keys of the feed_dict argument. Is there a mechanism in the python to figure out the name of the variables ?
Most of objects in tensorflow can be found with a string.
When you invoke tf.placeholder(tf.float32), tensorflow will do the following:
create a node with the Placeholder op
add this node to default graph
return the node output tensor
You can set a name for any node, say tf.placeholder(tf.float32, name='myplaceholder'), if you don't specify a node name, tensorflow will generate one, you can use print x.op to see the name of the op.
A tensor is named with the node name plus the output index, for example
x = tf.placeholder(tf.float32)
print x
you will see something like Placeholder:0, which is the tensor name.
So, in you code, tensorflow can first get tensor name from x, and iterate the default graph to find proper node.
You can also use string for feed_dict, {"Placeholder:0": 3}
Suppose there's a model defined as class like this:
class SimpleAutoencoder(object):
def __init__(self, x):
self.x = x
self.input_dim = 92
self.latent_dim = 10
self.build_model()
def build_model(self):
latent = tf.contrib.layers.fully_connected(self.x,
self.latent_dim,
scope='latent',
activation_fn=tf.nn.relu)
self.x_hat = tf.contrib.layers.fully_connected(latent,
self.input_dim,
scope='output',
activation_fn=tf.nn.sigmoid)
self.loss = tf.losses.mean_squared_error(self.x, self.x_hat)
self.train_op = tf.AdamOptimizer().minimize(self.loss)
You train it using an input pipeline for feeding your data:
...
x = iterator.get_next()
model = SimpleAutoencoder(x)
...
## train and save it to disk
Now, when using a placeholder for self.x while building the model, I can give it a name and access the input variable easily when I restore the model to do inference. But with the input pipeline, x is no variable, constant or placeholder and therefore I cannot give it a proper name. How can I inject new data into x and feed it through the graph?
Even though the training works, I have considered that I might do it wrong somehow, as the code looks really ugly to me (the part with giving the pipeline output to the init function).
Please help me with this one! Thank you!
You could get x's name using x.name,
or, you could rename x to a name to your taste using x = tf.identity(x, name='my_name'),
(With these two solutions, you can feed your values with the name of the tensor -- even if x is not a placeholder:
sess.run(my_ops, feed_dict{tensor_name: tensor_value})
)
or, your could replace the entire input pipeline with a placeholder (explained here for the opposite problem — replacing a placeholder with a Dataset input)
I want to optimize a cost function. This cost function contains variables and other parameters that are not variables. This non-variable parameters are obtained from the variables.
Here is a toy example that illustrates the point:
import numpy as np
import tensorflow as tf
r_init = np.array([5.0,6.0])
x = tf.get_variable("x_var", initializer = r_init[0], trainable = True)
y = tf.get_variable("y_var", initializer = r_init[1], trainable = True)
def cost(x,y):
a = x
return a*((x-1.0)**2+(y-1.0)**2)
train_op = tf.train.AdamOptimizer(learning_rate=0.05).minimize(cost(x,y))
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for i in range(100):
print(sess.run([cost(x,y), train_op]))
print('x=', x.eval(session=sess))
print('y=', y.eval(session=sess))
As you can see, the parameter a is defined from the variable x, on the other hand a should not be a variable, I want the optimizer to see it as a constant. This constant should be updated as the variable x is updated in the optimization process.
How can I define a non-variable parameter a from the variable x? I am making this up, but intuitively, what comes to my mind is something like:
a = tf.to_constant(x)
Any ideas?
You are looking for tf.stop_gradient:
a = tf.stop_gradient(x)
Quoting the docs,
This is useful any time you want to compute a value with TensorFlow but need to pretend that the value was a constant.
I am starting to use TensorFlow (with Python) and was wondering: when using a placeholder in a function, why not have an argument in my function which would feed a TensorFlow constant rather than the placeholder?
Here is an example (the difference is in x):
def sigmoid(z):
x = tf.constant(z, dtype=tf.float32, name = "x")
sigmoid = tf.sigmoid(x)
with tf.Session() as sess:
result = sess.run(sigmoid)
return result
instead of:
def sigmoid(z):
x = tf.placeholder(tf.float32, name = "...")
sigmoid = tf.sigmoid(x)
with tf.Session() as sess:
result = sess.run(sigmoid, feed_dict={x:z})
return result
The idea with Tensorflow is that you will repeat the same calculation on lots of data. when you write the code you are setting up a computational graph that later you will execute on the data. In your first example, you have hard-coded the data to a constant. This is not a typical tensorflow use case. The second example is better because it allows you to reuse the same computational graph with different data.
I've been trying to gather the gradient steps for each step of the GradientDescentOptimizer within TensorFlow, however I keep running into a TypeError when I try to pass the result of apply_gradients() to sess.run(). The code I'm trying to run is:
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
x = tf.placeholder(tf.float32,[None,784])
W = tf.Variable(tf.zeros([784,10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x,W)+b)
y_ = tf.placeholder(tf.float32,[None,10])
cross_entropy = -tf.reduce_sum(y_*log(y))
# note that up to this point, this example is identical to the tutorial on tensorflow.org
gradstep = tf.train.GradientDescentOptimizer(0.01).compute_gradients(cross_entropy)
sess = tf.Session()
sess.run(tf.initialize_all_variables())
batch_x,batch_y = mnist.train.next_batch(100)
print sess.run(gradstep, feed_dict={x:batch_x,y_:batch_y})
Note that if I replace the last line with print sess.run(train_step,feed_dict={x:batch_x,y_:batch_y}), where train_step = tf.GradientDescentOptimizer(0.01).minimize(cross_entropy), the error is not raised. My confusion arises from the fact that minimize calls compute_gradients with exactly the same arguments as its first step. Can someone explain why this behavior occurs?
The Optimizer.compute_gradients() method returns a list of (Tensor, Variable) pairs, where each tensor is the gradient with respect to the corresponding variable.
Session.run() expects a list of Tensor objects (or objects convertible to a Tensor) as its first argument. It does not understand how to handle a list of pairs, and hence you get a TypeError which you try to run sess.run(gradstep, ...)
The correct solution depends on what you are trying to do. If you want to fetch all of the gradient values, you can do the following:
grad_vals = sess.run([grad for grad, _ in gradstep], feed_dict={x: batch_x, y: batch_y})
# Then, e.g., nuild a variable name-to-gradient dictionary.
var_to_grad = {}
for grad_val, (_, var) in zip(grad_vals, gradstep):
var_to_grad[var.name] = grad_val
If you also want to fetch the variables, you can execute the following statement separately:
sess.run([var for _, var in gradstep])
...though note that—without further modification to your program—this will just return the initial values for each variable.
You will have to run the optimizer's training step (or otherwise call Optimizer.apply_gradients()) to update the variables.
minimize calls compute_gradients followed by apply_gradients: it's possible you're missing the second step.
compute_gradients just returns the grads / variables, but doesn't apply the update rule to them.
Here is an example: https://github.com/tensorflow/tensorflow/blob/f2bd0fc399606d14b55f3f7d732d013f32b33dd5/tensorflow/python/training/optimizer.py#L69