I'm fairly new to tensorflow, tried to calculate argmin of a quadratic function. I want to see the value of x and y after each iteration. Code:
import tensorflow as tf
x = tf.Variable(1.0,name="x")
y = x**2 - 4*x + 3
alpha = 0.05
optimizer = tf.train.AdamOptimizer(learning_rate = alpha).minimize(y)
num_epochs = 20
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
for epoch in range(num_epochs):
print("Epoch: %d" %epoch)
opt,x,result = sess.run([optimizer,x,y])
print(result)
The error I get is argument has invalid type , must be a string or Tensor.
It works if I don't try to get the value of x, just y and opt.
In your line
opt,x,result = sess.run([optimizer,x,y])
you assign the evaluated result of the x operation to variable x - thus, in the next iteration, x is no longer tf.Variable(1.0,name="x") but the result from the previous iteration. Just use another name for the variable and it should work.
Related
Currently, I have the following code:
x = tf.placeholder(tf.int32, name = "x")
y = tf.Variable(0, name="y")
y = 2*x**2 + 5
for i in range(1,10):
print("Value of y for x = ",i, " is: ",sess.run(y, feed_dict={x:i}))
However, when I try to display this on tensorboard, this gets messy.
Ideally I'd want to do y= tf.Variable(2*x**2 +5) but tensorflow throws an error telling me that x is uninitialized.
Or perhaps I shouldn't use tf.Variable and use something else?
If you really want to do that with a tf.Variable, you can do that in two ways. You can use the desired expression as the initialization value for the variable. Then, when you initialize the variable, you pass the x value in the feed_dict.
import tensorflow as tf
# Placeholder shape must be specified or use validate_shape=False in tf.Variable
x = tf.placeholder(tf.int32, (), name="x")
# Initialization value for variable is desired expression
y = tf.Variable(2 * x ** 2 + 5, name="y")
with tf.Session() as sess:
for i in range(1,10):
# Initialize variable on each iteration
sess.run(y.initializer, feed_dict={x: i})
# Show value
print("Value of y for x =", i , "is:", sess.run(y))
Alternatively, you can do the same thing with a tf.assign operation. In this case, you pass the x value when you run the assignment.
import tensorflow as tf
# Here placeholder shape is not stricly required as tf.Variable already gives the shape
x = tf.placeholder(tf.int32, name="x")
# Specify some initialization value for variable
y = tf.Variable(0, name="y")
# Assign expression value to variable
y_assigned = tf.assign(y, 2 * x** 2 + 5)
# Initialization can be skipped in this case since we always assign new value
with tf.Graph().as_default(), tf.Session() as sess:
for i in range(1,10):
# Assign vale to variable in each iteration (and get value after assignment)
print("Value of y for x =", i , "is:", sess.run(y_assigned, feed_dict={x: i}))
However, as pointed out by Nakor, you may not need a variable if y is simply supposed to be the result of that expression for whatever value x takes. The purpose of a variable is to hold a value that will be maintained in future calls to run. So you would need it only if you want to set y to some value depending on x, and then maintain the same value even if x changes (or even if x is not provided at all).
I think you misunderstood was a tf.Variable is. To quote the Tensorflow documentation:
A tf.Variable represents a tensor whose value can be changed by
running ops on it. Unlike tf.Tensor objects, a tf.Variable exists
outside the context of a single session.run call.
So variables will be your biases and weights in your neural network. These will vary when training your network. In Tensorflow, if you want to use your variables, you need to initialize them (using a constant or random value). That's what your error is about: you're defining y as a tf.Variable so it needs to be initialized.
However, your y is deterministic, it's not a tf.Variable. You can just remove the line where you define y and it works fine:
import tensorflow as tf
x = tf.placeholder(tf.int32, name = "x")
y = 2*x**2 + 5
with tf.Session() as sess:
for i in range(1,10):
print("Value of y for x = ",i, " is: ",sess.run(y, feed_dict={x:i}))
It returns:
Value of y for x = 1 is: 7
Value of y for x = 2 is: 13
Value of y for x = 3 is: 23
Value of y for x = 4 is: 37
Value of y for x = 5 is: 55
Value of y for x = 6 is: 77
Value of y for x = 7 is: 103
Value of y for x = 8 is: 133
Value of y for x = 9 is: 167
I've attempted converting a Python-side training loop to Tensorflow to (hypothetically) make the code run faster - not having to pass control over to cpu constantly. However, I can't manage using tf.while_loop.
Here's the code that works:
import numpy as np
import tensorflow as tf
from tqdm import tqdm
from sklearn.datasets import load_iris
from sklearn.preprocessing import RobustScaler
x, y = load_iris(True)
x = RobustScaler().fit_transform(x)
shape = (10, 10)
max_epochs = 1000
graph = tf.Graph()
sess = tf.Session(graph=graph)
x = x.astype(np.float64)
# Construct graph
with graph.as_default():
weights = tf.get_variable(
'weights', shape, initializer=tf.constant_initializer, dtype=tf.float64
)
curr_epoch = tf.placeholder(dtype=tf.int64, shape=())
with tf.name_scope('data'):
data = tf.data.Dataset.from_tensor_slices(x)
data = data.shuffle(buffer_size=10000)
data = data.repeat(max_epochs)
data = data.batch(1)
data = data.make_one_shot_iterator().get_next()
with tf.name_scope('update'):
update_op = make_update_op(weights)
init = tf.global_variables_initializer()
sess.run(init)
for i in tqdm(range(max_epochs)):
for _ in range(x.shape[0]):
sess.run(update_op, feed_dict={
curr_epoch: i
})
np_weights = sess.run(weights)
print(np_weights) # Correctly prints an array of 150's.
Now, if I create an update function to pass tf.while_loop, an error is thrown.
def make_update_op(w):
return w.assign(
w + 0.001
)
# In the code above:
update_op = tf.while_loop(lambda _: True, make_update_op, (weights,), maximum_iterations=x.shape[0])
# No inner loop:
for i in tqdm(range(max_epochs)):
sess.run(update_op, feed_dict={
curr_epoch: i
})
Line 22, in make_update_op
return w.assign(
AttributeError: 'Tensor' object has no attribute 'assign'
I don't quite understand what is happening even after reading the documentation. weights is a Variable after all. What could be done to correctly make the training loop?
The tensor that you're trying to assign a new value within a while loop is a result of a sequence of multiple operations-tensors (operation is node in the graph, while tensor is a directed edge). In particular, the while loop will produce:
Variable/Read-->while/Enter-->while/Merge-->while/Switch-->while/Identity
What you're trying to assign here is a tensor while/Identity.
tf.while_loop is usually used to iterate over the dimensions of a tensor (also over the None - the unknown dimension). You're trying to iterate over the variables that are fully defined. You don't need to create a tf.while_loop for that. Just create operations that update each variable and group these operations together:
update_ops = [w.assign(w + 0.001) for w in weights]
update_op = tf.group(update_ops)
Now, when you execute the update_op with tf.Session() interface it will update all variables.
Example:
import tensorflow as tf
v1 = tf.Variable(tf.ones((1, 2), dtype=tf.float32))
v2 = tf.Variable(2*tf.ones((1, 3), dtype=tf.float32))
update_ops = [w.assign(w + 0.001) for w in [v1, v2]]
update_op = tf.group(update_ops)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
print('before update:')
print(v1.eval(), v2.eval())
print('after update:')
sess.run(update_op) # <-- update your variables
print(v1.eval(), v2.eval())
# before update:
# [[1. 1.]] [[2. 2. 2.]]
# after update:
# [[1.001 1.001]] [[2.001 2.001 2.001]]
Turns out, all that was missing was the fact that one cannot assign to a variable inside a loop as Vlad pointed out. Instead, one can return the new value of a variable.
def make_update_op(w):
return w + 0.001
new_w = tf.while_loop(lambda _: True, make_update_op, (weights,), maximum_iterations=x.shape[0])
update_op = weights.assign(new_w)
To use more variables one would need to return the same amount from the function and unpack them in Python, but the principle is the same.
def make_update_op(w, d):
return w + 0.001, d
new_w, _ = tf.while_loop(lambda *_: True, make_update_op, (weights, data), maximum_iterations=x.shape[0])
update_op = weights.assign(new_w)
I have the following code trying to optimize for a linear model with two inputs and three parameters (m_1, m_2 and b). Initially, I had issues with importing the data in a way such that the feed_dict would accept them, which I solved by putting it in a numpy array instead.
Now the optimizer function will run smoothly (and the outputs look roughly like it is optimizing the parameters), but as soon as I try to return the cost with the line at the end:
cost_val = sess.run(cost)
It returns the following error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'Placeholder_2' with dtype float and shape [?,1]
[[Node: Placeholder_2 = Placeholder[dtype=DT_FLOAT, shape=[?,1], _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
If I comment out that line alone, everything runs smoothly.
I tried changing the cost function from the more complicated one I was using to something simpler, but the error persists. I know this is probably related to the data input shape(?), but can't figure how the data would work for the optimizer but not the cost function.
# reading in data
filename = tf.train.string_input_producer(["file.csv"])
reader = tf.TextLineReader(skip_header_lines=1)
key, value = reader.read(filename)
rec_def = [[1], [1], [1]]
input_1, input_2, col3 = tf.decode_csv(value, record_defaults=rec_def)
# parameters
learning_rate = 0.001
training_steps = 300
x = tf.placeholder(tf.float32, [None,1])
x2 = tf.placeholder(tf.float32, [None,1])
m = tf.Variable(tf.zeros([1,1]))
m2 = tf.Variable(tf.zeros([1,1]))
b = tf.Variable(tf.zeros([1]))
y_ = tf.placeholder(tf.float32, [None,1])
y = tf.matmul(x,m) + tf.matmul(x2,m2) + b
# cost function
# cost = tf.reduce_mean(tf.log(1+tf.exp(-y_*y)))
cost = tf.reduce_sum(tf.pow((y_-y),2))
# Gradient descent optimizer
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
# initializing variables
init = tf.global_variables_initializer()
with tf.Session() as sess:
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(coord=coord)
sess.run(init)
for i in range(training_steps):
xs = np.array([[sess.run(input_1)]])
ys = np.array([[sess.run(input_2)]])
label = np.array([[sess.run(col3)]])
feed = {x:xs, x2:ys, y_:label}
sess.run(optimizer, feed_dict=feed)
cost_val = sess.run(cost)
coord.request_stop()
coord.join(threads)
The cost tensor is a function of the placeholder tensors and this requires them to have a value. Since the call to sess.run(cost) isn't feeding those placeholders, you're seeing the error. (Putting it another way - what values of x and y_ do you want to compute the cost for?)
So you want to change the line:
cost_val = sess.run(cost)
to:
cost_val = sess.run(cost, feed_dict=feed)
Hope that helps.
I am new to tensorflow , I am not able to understand the difference of variable and constant, I get the idea that we use variables for equations and constants for direct values , but why code #1 works only and why not code#2 and #3, and please explain in which cases we have to run our graph first(a) and then our variable(b) i.e
(a) session.run(model)
(b) print(session.run(y))
and in which case I can directly execute this command
i.e
print(session.run(y))
Code #1 :
x = tf.constant(35, name='x')
y = tf.Variable(x + 5, name='y')
model = tf.global_variables_initializer()
with tf.Session() as session:
session.run(model)
print(session.run(y))
Code #2 :
x = tf.Variable(35, name='x')
y = tf.Variable(x + 5, name='y')
model = tf.global_variables_initializer()
with tf.Session() as session:
session.run(model)
print(session.run(y))
Code #3 :
x = tf.constant(35, name='x')
y = tf.constant(x + 5, name='y')
model = tf.global_variables_initializer()
with tf.Session() as session:
session.run(model)
print(session.run(y))
In TensorFlow the differences between constants and variables are that when you declare some constant, its value can't be changed in the future (also the initialization should be with a value, not with operation).
Nevertheless, when you declare a Variable, you can change its value in the future with tf.assign() method (and the initialization can be achieved with a value or operation).
The function tf.global_variables_initializer() initialises all variables in your code with the value passed as parameter, but it works in async mode, so doesn't work properly when dependencies exists between variables.
Your first code (#1) works properly because there is no dependencies on variable initialization and the constant is constructed with a value.
The second code (#2) doesn't work because of the async behavior of tf.global_variables_initializer(). You can fix it using tf.variables_initializer() as follows:
x = tf.Variable(35, name='x')
model_x = tf.variables_initializer([x])
y = tf.Variable(x + 5, name='y')
model_y = tf.variables_initializer([y])
with tf.Session() as session:
session.run(model_x)
session.run(model_y)
print(session.run(y))
The third code (#3) doesn't work properly because you are trying to initialize a constant with an operation, that isn't possible. To solve it, an appropriate strategy is (#1).
Regarding to your last question. You need to run (a) session.run(model) when there are variables in your calculation graph (b) print(session.run(y)).
I will point the difference when using eager execution.
As of Tensorflow 2.0.b1, Variables and Constant trigger different behaviours when using tf.GradientTape. Strangely, the official document is not verbal about it enough.
Let's look at the example code in https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/GradientTape
x = tf.constant(3.0)
with tf.GradientTape(persistent=True) as g:
g.watch(x)
y = x * x
z = y * y
dz_dx = g.gradient(z, x) # 108.0 (4*x^3 at x = 3)
dy_dx = g.gradient(y, x) # 6.0
del g # Drop the reference to the tape
You had to watch x which is a Constant. GradientTape does NOT automatically watch constants in the context. Additionally, it can watch only one tensor per GradientTape. If you want to get gradients of multiple Constants, you need to nest GradientTapes. For example,
x = tf.constant(3.0)
x2 = tf.constant(3.0)
with tf.GradientTape(persistent=True) as g:
g.watch(x)
with tf.GradientTape(persistent=True) as g2:
g2.watch(x2)
y = x * x
y2 = y * x2
dy_dx = g.gradient(y, x) # 6
dy2_dx2 = g2.gradient(y2, x2) # 9
del g, g2 # Drop the reference to the tape
On the other hand, Variables are automatically watched by GradientTape.
By default GradientTape will automatically watch any trainable variables that are accessed inside the context. Source: https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/GradientTape
So the above will look like,
x = tf.Variable(3.0)
x2 = tf.Variable(3.0)
with tf.GradientTape(persistent=True) as g:
y = x * x
y2 = y * x2
dy_dx = g.gradient(y, x) # 6
dy2_dx2 = g.gradient(y2, x2) # 9
del g # Drop the reference to the tape
print(dy_dx)
print(dy2_dx2)
Of course, you can turn off the automatic watching by passing watch_accessed_variables=False. The examples may not be so practical but I hope this clears someone's confusion.
Another way to look to the differences is:
tf.constant : are fixed values, and hence not trainable.
tf.Variable: these are tensors (arrays) that were initialized in a session and are trainable (with trainable i mean this can be optimized and can changed over time)
When tensorflow's session is runned, i need to get the same value of y. How can i get y with same value, not rerun this graph?
import tensorflow as tf
import numpy as np
x = tf.Variable(0.0)
tf.set_random_seed(10)
x_plus1 = x+tf.random_normal([1], mean=0.0, stddev=0.01,dtype=tf.float32)
y = tf.Variable([1.0])
y += x_plus1
z = y + tf.random_normal([1], mean=0.0, stddev=0.01,dtype=tf.float32)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
print(z.eval())
for i in range(5):
print(y.eval())
Here, i want to get y that contributes to z.
You can evaluate y and z simultaneously with sess.run(), that runs the needed parts of the graph only once, hence the value for y will be the one used for z.
with tf.Session() as sess:
sess.run(init)
z_value, y_value = sess.run([z, y])
print(z_value)
print(y_value)
Modify the with block as following, this way you only evaluate the graph once, before the for loop, then you can print it as many times as you like:
with tf.Session() as sess:
sess.run(init)
print(z.eval())
yy = y.eval()
for i in range(5):
print(yy)