Currently, I have the following code:
x = tf.placeholder(tf.int32, name = "x")
y = tf.Variable(0, name="y")
y = 2*x**2 + 5
for i in range(1,10):
print("Value of y for x = ",i, " is: ",sess.run(y, feed_dict={x:i}))
However, when I try to display this on tensorboard, this gets messy.
Ideally I'd want to do y= tf.Variable(2*x**2 +5) but tensorflow throws an error telling me that x is uninitialized.
Or perhaps I shouldn't use tf.Variable and use something else?
If you really want to do that with a tf.Variable, you can do that in two ways. You can use the desired expression as the initialization value for the variable. Then, when you initialize the variable, you pass the x value in the feed_dict.
import tensorflow as tf
# Placeholder shape must be specified or use validate_shape=False in tf.Variable
x = tf.placeholder(tf.int32, (), name="x")
# Initialization value for variable is desired expression
y = tf.Variable(2 * x ** 2 + 5, name="y")
with tf.Session() as sess:
for i in range(1,10):
# Initialize variable on each iteration
sess.run(y.initializer, feed_dict={x: i})
# Show value
print("Value of y for x =", i , "is:", sess.run(y))
Alternatively, you can do the same thing with a tf.assign operation. In this case, you pass the x value when you run the assignment.
import tensorflow as tf
# Here placeholder shape is not stricly required as tf.Variable already gives the shape
x = tf.placeholder(tf.int32, name="x")
# Specify some initialization value for variable
y = tf.Variable(0, name="y")
# Assign expression value to variable
y_assigned = tf.assign(y, 2 * x** 2 + 5)
# Initialization can be skipped in this case since we always assign new value
with tf.Graph().as_default(), tf.Session() as sess:
for i in range(1,10):
# Assign vale to variable in each iteration (and get value after assignment)
print("Value of y for x =", i , "is:", sess.run(y_assigned, feed_dict={x: i}))
However, as pointed out by Nakor, you may not need a variable if y is simply supposed to be the result of that expression for whatever value x takes. The purpose of a variable is to hold a value that will be maintained in future calls to run. So you would need it only if you want to set y to some value depending on x, and then maintain the same value even if x changes (or even if x is not provided at all).
I think you misunderstood was a tf.Variable is. To quote the Tensorflow documentation:
A tf.Variable represents a tensor whose value can be changed by
running ops on it. Unlike tf.Tensor objects, a tf.Variable exists
outside the context of a single session.run call.
So variables will be your biases and weights in your neural network. These will vary when training your network. In Tensorflow, if you want to use your variables, you need to initialize them (using a constant or random value). That's what your error is about: you're defining y as a tf.Variable so it needs to be initialized.
However, your y is deterministic, it's not a tf.Variable. You can just remove the line where you define y and it works fine:
import tensorflow as tf
x = tf.placeholder(tf.int32, name = "x")
y = 2*x**2 + 5
with tf.Session() as sess:
for i in range(1,10):
print("Value of y for x = ",i, " is: ",sess.run(y, feed_dict={x:i}))
It returns:
Value of y for x = 1 is: 7
Value of y for x = 2 is: 13
Value of y for x = 3 is: 23
Value of y for x = 4 is: 37
Value of y for x = 5 is: 55
Value of y for x = 6 is: 77
Value of y for x = 7 is: 103
Value of y for x = 8 is: 133
Value of y for x = 9 is: 167
Related
See the code snippet:
import tensorflow as tf
x = tf.Variable(1)
op = tf.assign(x, x + 1)
with tf.Session() as sess:
tf.global_variables_initializer().run()
print(sess.run([x, op]))
There are two possible results:
x=1 and op=2
x=2 and op=2
They depend on the order of evaluation, for the first case, x is evaluated before op, and for the second case, x is evaluated after op.
I have run the code many times, but the result is always x=2 and op=2. So I guess that tensorflow can guarantee x is evaluated after op. Is it right? And how does tensorflow guarantee the dependence?
Update
For the case above, the result is determinate. But in the follow case, the result is not determinate.
import tensorflow as tf
x = tf.Variable(1)
op = tf.assign(x, x + 1)
x = x + 0 # add this line
with tf.Session() as sess:
tf.global_variables_initializer().run()
for i in range(5):
print(sess.run([x, op]))
In the first code, x is Variable and op depends on x, so x is always evaluated after op. But in the second case, x becomes Tensor, and op depend on Variable x(After x = x + 0, x is overrided). So the op doesn't depend on Tensor x.
The order in which tensors are evaluated is undefined. See the API docs (towards, the very bottom, in the "Returns" info on Session.run()). As such, you should not rely on them being executed in a particular order. If you need to guarantee an order you should probably use separate run calls for the different tensors/ops.
This works:
with tf.device('/cpu:0'):
# x = tf.get_variable('x', shape=(), initializer=tf.constant_initializer(1), dtype=tf.int32)
x = tf.Variable(1)
op = tf.assign(x, x+1)
with tf.control_dependencies([op]):
x = x + 0
# x = tf.multiply(x, 3)
# x = tf.add(x, 0)
but not always:
with tf.device('/cpu:0'):
# x = tf.get_variable('x', shape=(), initializer=tf.constant_initializer(1), dtype=tf.int32)
x = tf.Variable(1)
with tf.device('/gpu:0'): # add this line.
op = tf.assign(x, x+1)
with tf.control_dependencies([op]):
x = x + 0
# x = tf.multiply(x, 3)
# x = tf.add(x, 0)
I think the problem is from:
Some operation (like x = x + 0, read value from the Variable and then add 0) depends on the value of the Variable, while the Variable is changed by some assign operation (like op = tf.assign(x, x+1)). If there are no dependencies, these two operations are parallel. So when reading the value of Variable, it is not sure whether assign is already done.
transferring data from different devices. Even there are dependencies, it is still indeterminate.
In brief, if all variables and operations are in the same device, then with tf.control_dependencies should guarantee that op is before the add operation.
Note that:
(these notes below are not important but might help you)
tf.assign operations only update the Variable, not Tensor.
When you do x=x+0, the new x becomes a Tensor; but the op = tf.assign(x, x+1) returns the ref of Variable. So op should always be determinate because it depends on the current value of Variable which will not be changed by other operations.
GPU does not support int32 variable. When I run your code snippet on my machine (tf-gpu1.12), variable is created on CPU but ops are done on GPU. You can check the variables and ops by config = tf.ConfigProto(log_device_placement=True).
I am confusing about following code: y = x and y = tf.identity(x).
More precisely, I get confusing when I run following code snippets:
code 1:
import tensorflow as tf
x = tf.Variable(0.0, name="x")
with tf.control_dependencies([x_plus_1]):
y = x
init = tf.initialize_all_variables()
with tf.Session() as sess:
init.run()
for i in range(5):
print(y.eval())
This will give output: 0.0 0.0 0.0 0.0 0.0
Code 2:
import tensorflow as tf
x = tf.Variable(0.0, name="x")
with tf.control_dependencies([x_plus_1]):
y = tf.identity(x, name='id')
init = tf.initialize_all_variables()
with tf.Session() as sess:
init.run()
for i in range(5):
print(y.eval())
The only change is from y=x to y=tf.identity(x), but now the results is 1.0 2.0 3.0 4.0 5.0
Thank you very much.
x is a Tensor, let's call it tensor_1. When you say y = x you are saying that the variable y has the value of the Tensor tensor_1. When you do y = tf.identity(x) you are creating a new Tensor, tensor_2, which value will be the same as tensor_1. But it is a different node in your graph, so the value from tensor_1 to tensor_2 has to move. That is why the with tf.control_dependencies([x_plus_1]) does something in your second code but nothing in the first one. Because in the first code you are not creating any new Tensor which control_depdendencies can work with.
To sum up y = x makes the variable y to point to the same object in x, but y = tf.identity(x) creates a new object with the content of x.
I'm fairly new to tensorflow, tried to calculate argmin of a quadratic function. I want to see the value of x and y after each iteration. Code:
import tensorflow as tf
x = tf.Variable(1.0,name="x")
y = x**2 - 4*x + 3
alpha = 0.05
optimizer = tf.train.AdamOptimizer(learning_rate = alpha).minimize(y)
num_epochs = 20
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
for epoch in range(num_epochs):
print("Epoch: %d" %epoch)
opt,x,result = sess.run([optimizer,x,y])
print(result)
The error I get is argument has invalid type , must be a string or Tensor.
It works if I don't try to get the value of x, just y and opt.
In your line
opt,x,result = sess.run([optimizer,x,y])
you assign the evaluated result of the x operation to variable x - thus, in the next iteration, x is no longer tf.Variable(1.0,name="x") but the result from the previous iteration. Just use another name for the variable and it should work.
I am new to tensorflow , I am not able to understand the difference of variable and constant, I get the idea that we use variables for equations and constants for direct values , but why code #1 works only and why not code#2 and #3, and please explain in which cases we have to run our graph first(a) and then our variable(b) i.e
(a) session.run(model)
(b) print(session.run(y))
and in which case I can directly execute this command
i.e
print(session.run(y))
Code #1 :
x = tf.constant(35, name='x')
y = tf.Variable(x + 5, name='y')
model = tf.global_variables_initializer()
with tf.Session() as session:
session.run(model)
print(session.run(y))
Code #2 :
x = tf.Variable(35, name='x')
y = tf.Variable(x + 5, name='y')
model = tf.global_variables_initializer()
with tf.Session() as session:
session.run(model)
print(session.run(y))
Code #3 :
x = tf.constant(35, name='x')
y = tf.constant(x + 5, name='y')
model = tf.global_variables_initializer()
with tf.Session() as session:
session.run(model)
print(session.run(y))
In TensorFlow the differences between constants and variables are that when you declare some constant, its value can't be changed in the future (also the initialization should be with a value, not with operation).
Nevertheless, when you declare a Variable, you can change its value in the future with tf.assign() method (and the initialization can be achieved with a value or operation).
The function tf.global_variables_initializer() initialises all variables in your code with the value passed as parameter, but it works in async mode, so doesn't work properly when dependencies exists between variables.
Your first code (#1) works properly because there is no dependencies on variable initialization and the constant is constructed with a value.
The second code (#2) doesn't work because of the async behavior of tf.global_variables_initializer(). You can fix it using tf.variables_initializer() as follows:
x = tf.Variable(35, name='x')
model_x = tf.variables_initializer([x])
y = tf.Variable(x + 5, name='y')
model_y = tf.variables_initializer([y])
with tf.Session() as session:
session.run(model_x)
session.run(model_y)
print(session.run(y))
The third code (#3) doesn't work properly because you are trying to initialize a constant with an operation, that isn't possible. To solve it, an appropriate strategy is (#1).
Regarding to your last question. You need to run (a) session.run(model) when there are variables in your calculation graph (b) print(session.run(y)).
I will point the difference when using eager execution.
As of Tensorflow 2.0.b1, Variables and Constant trigger different behaviours when using tf.GradientTape. Strangely, the official document is not verbal about it enough.
Let's look at the example code in https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/GradientTape
x = tf.constant(3.0)
with tf.GradientTape(persistent=True) as g:
g.watch(x)
y = x * x
z = y * y
dz_dx = g.gradient(z, x) # 108.0 (4*x^3 at x = 3)
dy_dx = g.gradient(y, x) # 6.0
del g # Drop the reference to the tape
You had to watch x which is a Constant. GradientTape does NOT automatically watch constants in the context. Additionally, it can watch only one tensor per GradientTape. If you want to get gradients of multiple Constants, you need to nest GradientTapes. For example,
x = tf.constant(3.0)
x2 = tf.constant(3.0)
with tf.GradientTape(persistent=True) as g:
g.watch(x)
with tf.GradientTape(persistent=True) as g2:
g2.watch(x2)
y = x * x
y2 = y * x2
dy_dx = g.gradient(y, x) # 6
dy2_dx2 = g2.gradient(y2, x2) # 9
del g, g2 # Drop the reference to the tape
On the other hand, Variables are automatically watched by GradientTape.
By default GradientTape will automatically watch any trainable variables that are accessed inside the context. Source: https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/GradientTape
So the above will look like,
x = tf.Variable(3.0)
x2 = tf.Variable(3.0)
with tf.GradientTape(persistent=True) as g:
y = x * x
y2 = y * x2
dy_dx = g.gradient(y, x) # 6
dy2_dx2 = g.gradient(y2, x2) # 9
del g # Drop the reference to the tape
print(dy_dx)
print(dy2_dx2)
Of course, you can turn off the automatic watching by passing watch_accessed_variables=False. The examples may not be so practical but I hope this clears someone's confusion.
Another way to look to the differences is:
tf.constant : are fixed values, and hence not trainable.
tf.Variable: these are tensors (arrays) that were initialized in a session and are trainable (with trainable i mean this can be optimized and can changed over time)
When tensorflow's session is runned, i need to get the same value of y. How can i get y with same value, not rerun this graph?
import tensorflow as tf
import numpy as np
x = tf.Variable(0.0)
tf.set_random_seed(10)
x_plus1 = x+tf.random_normal([1], mean=0.0, stddev=0.01,dtype=tf.float32)
y = tf.Variable([1.0])
y += x_plus1
z = y + tf.random_normal([1], mean=0.0, stddev=0.01,dtype=tf.float32)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
print(z.eval())
for i in range(5):
print(y.eval())
Here, i want to get y that contributes to z.
You can evaluate y and z simultaneously with sess.run(), that runs the needed parts of the graph only once, hence the value for y will be the one used for z.
with tf.Session() as sess:
sess.run(init)
z_value, y_value = sess.run([z, y])
print(z_value)
print(y_value)
Modify the with block as following, this way you only evaluate the graph once, before the for loop, then you can print it as many times as you like:
with tf.Session() as sess:
sess.run(init)
print(z.eval())
yy = y.eval()
for i in range(5):
print(yy)