I am learning to use tensorflow library. Every time I try to use simplest (I guess) possible example of gradient-based learning I get the same error, which makes me stuck.
Here is the code:
import tensorflow as tf
x = tf.constant(1, dtype=tf.float32, name='X')
a = tf.Variable(1, dtype=tf.float32, name= 'A')
y = tf.constant(10, dtype=tf.float32, name='Y')
ey = tf.multiply(x, a)
los = (y - ey)**2
optim = tf.train.GradientDescentOptimizer(learning_rate=0.2)
sess = tf.Session()
sess.run(tf.global_variables_initializer())
for counter in range(100):
grd = sess.run(optim.compute_gradients(loss=los))
sess.run(optim.apply_gradients(grads_and_vars=grd))
On the last line, I get following error:
AttributeError: 'numpy.float32' object has no attribute 'op'.
Thank you in advance for any tips.
The input of apply_gradients must be a tensor not a numpy array. Because, it is not meant to be called at each step of the learning but just once when building your graph. In fact, it "creates" the "training step" in which variables are updated using the computed gradient
You should use that kind of code instead:
import tensorflow as tf
x = tf.constant(1, dtype=tf.float32, name='X')
a = tf.Variable(1, dtype=tf.float32, name= 'A')
y = tf.constant(10, dtype=tf.float32, name='Y')
ey = tf.multiply(x, a)
los = (y - ey)**2
optim = tf.train.GradientDescentOptimizer(learning_rate=0.2)
grads, vars = zip(*optimizer.compute_gradients(loss))
train_step = optim.apply_gradients(grads_and_vars=grads)
sess = tf.Session()
sess.run(tf.global_variables_initializer())
for counter in range(100):
grd = sess.run(train_step, feed_dict = {...})
The tensorflow converts tensor into numpy array when stored and evaluated using sess.run(). The apply_gradients method of GradientDescentOptimizer raises TypeError since it requires tensor as input as described here. This causing problem in the original code.
for counter in range(100):
grd = sess.run(optim.compute_gradients(loss=los))
sess.run(optim.apply_gradients(grads_and_vars=grd))
The solution to this problem is to define all the operations before starting session.
There may be cases where gradients need to be post-processed before updating weights. All such post-processing logic need to be defined before session begins.
Here is a simple example:
import tensorflow as tf
x = tf.constant(1, dtype=tf.float32, name='X')
a = tf.Variable(1, dtype=tf.float32, name= 'A')
y = tf.constant(10, dtype=tf.float32, name='Y')
ey = tf.multiply(x, a)
loss = (y - ey)**2
optim = tf.train.GradientDescentOptimizer(learning_rate=0.2)
grads = optim.compute_gradients(loss)
train_step = optim.apply_gradients(grads_and_vars=grads)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for counter in range(100):
sess.run(train_step)
Related
This question is related to this question, which provides a solution that works in Tensorflow 1.15, but doesn't work anymore in TF2
I'm taking part of the code from that question and adapting it slightly (removed the frozen model's multiple inputs and, with it, the need for nest).
Note: I'm separating the code in blocks, but they're meant to be run as on file (i.e., I won't repeat the unnecessary imports in each block)
First, we generate a frozen graph to use as dummy test network:
import numpy as np
import tensorflow.compat.v1 as tf
def dump_model():
with tf.Graph().as_default() as gf:
x = tf.placeholder(tf.float32, shape=(None, 123), name='x')
c = tf.constant(100, dtype=tf.float32, name='C')
y = tf.multiply(x, c, name='y')
z = tf.add(y, x, name='z')
with tf.gfile.GFile("tmp_net.pb", "wb") as f:
raw = gf.as_graph_def().SerializeToString()
print(type(raw), len(raw))
f.write(raw)
dump_model()
Then, we load the frozen model and wrap it in a Keras Model:
persisted_sess = tf.Session()
with tf.Session().as_default() as session:
with tf.gfile.FastGFile("./tmp_net.pb",'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
persisted_sess.graph.as_default()
tf.import_graph_def(graph_def, name='')
print(persisted_sess.graph.get_name_scope())
for i, op in enumerate(persisted_sess.graph.get_operations()):
tensor = persisted_sess.graph.get_tensor_by_name(op.name + ':0')
print(i, '\t', op.name, op.type, tensor)
x_tensor = persisted_sess.graph.get_tensor_by_name('x:0')
y_tensor = persisted_sess.graph.get_tensor_by_name('y:0')
z_tensor = persisted_sess.graph.get_tensor_by_name('z:0')
from tensorflow.compat.v1.keras.layers import Lambda, InputLayer
from tensorflow.compat.v1.keras import Model
from tensorflow.python.keras.utils import layer_utils
input_x = InputLayer(name='x', input_tensor=x_tensor)
input_x.is_placeholder = True
output_y = Lambda(lambda x: y_tensor, name='output_y')(input_x.output)
output_z = Lambda(lambda x_b: z_tensor, name='output_z')(input_x.output)
base_model_inputs = layer_utils.get_source_inputs(input_x.output)
base_model = Model(base_model_inputs, [output_y, output_z])
Finally, we run the model on some random data and verify that it runs without errors:
y_out, z_out = base_model.predict(np.ones((3, 123), dtype=np.float32))
y_out.shape, z_out.shape
In Tensorflow 1.15.3, the output of the above is ((3, 123), (3, 123)), however, if I run the same code in Tensorflow 2.1.0, the first two blocks run without a problem, but then the third fails with:
TypeError: An op outside of the function building code is being passed
a "Graph" tensor. It is possible to have Graph tensors
leak out of the function building context by including a
tf.init_scope in your function building code.
For example, the following function will fail:
#tf.function
def has_init_scope():
my_constant = tf.constant(1.)
with tf.init_scope():
added = my_constant * 2
The graph tensor has name: y:0
The error seems related to Tensorflow's automatic "compilation" and optimization of functions, but I don't know how to interpret it, what the source of the error is, or how to resolve.
What is the correct way to wrap the frozen model in Tensorflow 2?
I can run your whole example fine in 2.2.0 like this.
import tensorflow as tf
from tensorflow.core.framework.graph_pb2 import GraphDef
import numpy as np
with tf.Graph().as_default() as gf:
x = tf.compat.v1.placeholder(tf.float32, shape=(None, 123), name='x')
c = tf.constant(100, dtype=tf.float32, name='c')
y = tf.multiply(x, c, name='y')
z = tf.add(y, x, name='z')
with open('tmp_net.pb', 'wb') as f:
f.write(gf.as_graph_def().SerializeToString())
with tf.Graph().as_default():
gd = GraphDef()
with open('tmp_net.pb', 'rb') as f:
gd.ParseFromString(f.read())
x, y, z = tf.graph_util.import_graph_def(
gd, name='', return_elements=['x:0', 'y:0', 'z:0'])
del gd
input_x = tf.keras.layers.InputLayer(name='x', input_tensor=x)
input_x.is_placeholder = True
output_y = tf.keras.layers.Lambda(lambda x: y, name='output_y')(input_x.output)
output_z = tf.keras.layers.Lambda(lambda x: z, name='output_z')(input_x.output)
base_model_inputs = tf.keras.utils.get_source_inputs(input_x.output)
base_model = tf.keras.Model(base_model_inputs, [output_y, output_z])
y_out, z_out = base_model.predict(np.ones((3, 123), dtype=np.float32))
print(y_out.shape, z_out.shape)
# (3, 123) (3, 123)
The "trick" is to wrap the model construction within a with tf.Graph().as_default(): block, which will ensure everything is created in graph mode within the same graph object.
However, it may be simpler to wrap the graph loading and computation within a #tf.function, which would avoid this kind of error and make the model construction more transparent:
import tensorflow as tf
from tensorflow.core.framework.graph_pb2 import GraphDef
import numpy as np
#tf.function
def my_model(x):
gd = GraphDef()
with open('tmp_net.pb', 'rb') as f:
gd.ParseFromString(f.read())
y, z = tf.graph_util.import_graph_def(
gd, name='', input_map={'x:0': x}, return_elements=['y:0', 'z:0'])
return [y, z]
x = tf.keras.Input(shape=123)
y, z = tf.keras.layers.Lambda(my_model)(x)
model = tf.keras.Model(x, [y, z])
y_out, z_out = model.predict(np.ones((3, 123), dtype=np.float32))
print(y_out.shape, z_out.shape)
# (3, 123) (3, 123)
Another possible way to do this would be
import tensorflow as tf
input_layer = tf.keras.Input(shape=[123])
keras_graph = input_layer.graph
with keras_graph.as_default():
with tf.io.gfile.GFile('tmp_net.pb', 'rb') as f:
graph_def = tf.compat.v1.GraphDef()
graph_def.ParseFromString(f.read())
tf.graph_util.import_graph_def(graph_def, name='', input_map={'x:0': input_layer})
y_tensor = keras_graph.get_tensor_by_name('y:0')
z_tensor = keras_graph.get_tensor_by_name('z:0')
base_model = tf.keras.Model(input_layer, [y_tensor, z_tensor])
And then
y_out, z_out = base_model.predict(tf.ones((3, 123), dtype=tf.float32))
print(y_out.shape, z_out.shape)
# (3, 123) (3, 123)
GAN Discriminator
I use this code below get the disriminator of GAN neural network:
import tensorflow as tf
import numpy as np
from IPython.display import display, Audio
tf.reset_default_graph()
saver = tf.train.import_meta_graph('./infer/infer.meta')
graph = tf.get_default_graph()
sess = tf.InteractiveSession()
saver.restore(sess, tf.train.latest_checkpoint('model/'))
# here is z with underline, it doesn't showing ceractly in stack.
# I use random data to test this function.
_z = np.random.uniform(-1., 1., size=[5, 257])
x = graph.get_tensor_by_name('x:0')
D_z = graph.get_tensor_by_name('D_z:0')
D_z = sess.run(D_z, {x: _z})
print(D_z)
Custom Keras loss function
I wan to create a function to customed the keras loss fuction:
# Load the graph
tf.reset_default_graph()
saver = tf.train.import_meta_graph('./infer/infer.meta')
graph = tf.get_default_graph()
sess = tf.InteractiveSession()
saver.restore(sess, tf.train.latest_checkpoint('model/'))
def gan_loss(y_true, y_pred):
_z = y_pred
x = graph.get_tensor_by_name('x:0')
D_z = graph.get_tensor_by_name('D_z:0')
D_z = sess.run(D_z, {x: _z})
return D_z
Problem I have met
I got the problem that show me: Can not feed tesor, you have to feed it with numpy or other type of data.
TypeError: The value of a feed cannot be a tf.Tensor object. Acceptable feed values include Python scalars, strings, lists, or numpy ndarrays.
I fond related problem in Stak:Converting Tensor to np.array using K.eval() in Keras returns InvalidArgumentError
Tensorflow: How to feed a placeholder variable with a tensor?
GAN neural network , How i get the discriminator
X = tf.placeholder(tf.float32, [None, 257], name='x')
D_z, h3 = discriminator(X)
D_z = tf.identity(D_z, name='D_z')
D_vars = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='GAN/Discriminator')
# global_step = tf.train.get_or_create_global_step()
saver = tf.train.Saver(D_vars)
infer_dir = './infer/'
tf.train.write_graph(tf.get_default_graph(), infer_dir, 'infer.pbtxt')
infer_metagraph_fp = os.path.join(infer_dir, 'infer.meta')
tf.train.export_meta_graph(
filename=infer_metagraph_fp,
clear_devices=True,
saver_def=saver.as_saver_def())
tf.reset_default_graph()
I was able to reproduce your error using below simple code where I am feeding a tensor to feed_dict -
Code to reproduce the error -
%tensorflow_version 1.x
import tensorflow as tf
print(tf.__version__)
import numpy as np
x = tf.placeholder(tf.float32)
y = x * 42
with tf.Session() as sess:
a = tf.constant(2)
train_accuracy = y.eval(session=sess,feed_dict={x: a})
print(train_accuracy)
Output -
1.15.2
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-18-44556cb4551b> in <module>()
10 with tf.Session() as sess:
11 a = tf.constant(2)
---> 12 train_accuracy = y.eval(session=sess,feed_dict={x: a})
13 print(train_accuracy)
3 frames
/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
1129 'For reference, the tensor object was ' +
1130 str(feed_val) + ' which was passed to the '
-> 1131 'feed with key ' + str(feed) + '.')
1132
1133 subfeed_dtype = subfeed_t.dtype.as_numpy_dtype
TypeError: The value of a feed cannot be a tf.Tensor object. Acceptable feed values include Python scalars, strings, lists, numpy ndarrays, or TensorHandles. For reference, the tensor object was Tensor("Const_6:0", shape=(), dtype=int32) which was passed to the feed with key Tensor("Placeholder_9:0", dtype=float32).
I was able to fix it when I converted the tensor to numpy type for feed_dict. So in your case, convert y_pred to numpy type.
Fixed Code -
%tensorflow_version 1.x
import tensorflow as tf
print(tf.__version__)
import numpy as np
x = tf.placeholder(tf.float32)
y = x * 42
with tf.Session() as sess:
a = tf.constant(2)
a = np.array(a.eval())
train_accuracy = y.eval(session=sess,feed_dict={x: b})
print(train_accuracy)
Output -
1.15.2
84.0
Hope this answers your question. Happy Learning.
I have the same question for tf.contrib.rnn.LSTMBlockCell and tf.contrib.cudnn_rnn.CudnnCompatibleLSTMCell:
How do I initialize the LSTM weights from numpy arrays correctly? The following code-snipped executes, but does not seem
to do what I am looking for:
train_data = np.load('mnist_train_data.npy').reshape(-1,28,28)
train_label = np.load('mnist_train_label.npy')
params = [np.random.randn(28+128, 4*128), np.zeros(4*128)]
X = tf.placeholder(tf.float32, shape=[54999, 28, 28])
y = tf.placeholder(tf.int64, None)
state = LSTMStateTuple(*(tf.zeros((54999, 128), dtype=tf.float32) for _ in range(2)))
cell = tf.contrib.rnn.LSTMBlockCell(128)
cell.build(tf.TensorShape((None, 28)))
cell.set_weights(params)
initial_weights = cell.get_weights()
print(np.array_equal(params[0], initial_weights[0]))
w1 = tf.Variable(np.random.randn(128, 10), dtype=tf.float32)
b1 = tf.Variable(np.zeros(10), dtype=tf.float32)
full_seq, current_state = tf.nn.dynamic_rnn(cell, X, initial_state=state, dtype=tf.float32)
output = tf.matmul(current_state[1], w1)
output += b1
loss = tf.losses.softmax_cross_entropy(y, output)
train_step = tf.train.AdamOptimizer(0.01).minimize(loss)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for i in range(1):
feed_dict = {X: train_data, y: train_label}
sess.run(train_step, feed_dict=feed_dict)
final_weights = cell.get_weights()
print(np.array_equal(initial_weights[0], final_weights[0]))
This prints out False in the first print-statement, so the numpy arrays do not actually seem to be used as weights.
Moreover, after the training session, this prints out True thus implying, that these weights are not actually updated during training.
Thanks in advance for any help on the subject.
I try to experiment with in_top_k function to see what exactly this function is doing. But I found some really confusing behavior.
First I coded as follows
import numpy as np
import tensorflow as tf
target = tf.constant(np.random.randint(2, size=30).reshape(30,-1), dtype=tf.int32, name="target")
pred = tf.constant(np.random.rand(30,1), dtype=tf.float32, name="pred")
result = tf.nn.in_top_k(pred, target, 1)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
targetVal = target.eval()
predVal = pred.eval()
resultVal = result.eval()
Then it generates the following error:
ValueError: Shape must be rank 1 but is rank 2 for 'in_top_k/InTopKV2' (op: 'InTopKV2') with input shapes: [30,1], [30,1], [].
Then I changed my code to
import numpy as np
import tensorflow as tf
target = tf.constant(np.random.randint(2, size=30), dtype=tf.int32, name="target")
pred = tf.constant(np.random.rand(30,1).reshape(-1), dtype=tf.float32, name="pred")
result = tf.nn.in_top_k(pred, target, 1)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
targetVal = target.eval()
predVal = pred.eval()
resultVal = result.eval()
But now the error becomes
ValueError: Shape must be rank 2 but is rank 1 for 'in_top_k/InTopKV2' (op: 'InTopKV2') with input shapes: [30], [30], [].
So should the input be rank 1 or rank 2?
For in_top_k, the targets need to be rank 1 (class indices) and the predictions rank 2 (scores for each class). This can be seen from the docs easily.
This means that the two error messages actually complain about different inputs each time (targets the first time and predictions the second time), which funnily enough isn't mentioned in the messages at all... Either way, the following snippet should be more like it:
import numpy as np
import tensorflow as tf
target = tf.constant(np.random.randint(2, size=30), dtype=tf.int32, name="target")
pred = tf.constant(np.random.rand(30,1), dtype=tf.float32, name="pred")
result = tf.nn.in_top_k(pred, target, 1)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
targetVal = target.eval()
predVal = pred.eval()
resultVal = result.eval()
Here, we basically combine the "best of both snippets": Predictions from the first one and targets from the second one. However, the way I understand the docs, even for binary classification we need two values for the predictions, one for each class. So something like
import numpy as np
import tensorflow as tf
target = tf.constant(np.random.randint(2, size=30), dtype=tf.int32, name="target")
pred = tf.constant(np.random.rand(30,1), dtype=tf.float32, name="pred")
pred = tf.concat((1-pred, pred), axis=1)
result = tf.nn.in_top_k(pred, target, 1)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
targetVal = target.eval()
predVal = pred.eval()
resultVal = result.eval()
I have the following code trying to optimize for a linear model with two inputs and three parameters (m_1, m_2 and b). Initially, I had issues with importing the data in a way such that the feed_dict would accept them, which I solved by putting it in a numpy array instead.
Now the optimizer function will run smoothly (and the outputs look roughly like it is optimizing the parameters), but as soon as I try to return the cost with the line at the end:
cost_val = sess.run(cost)
It returns the following error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'Placeholder_2' with dtype float and shape [?,1]
[[Node: Placeholder_2 = Placeholder[dtype=DT_FLOAT, shape=[?,1], _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
If I comment out that line alone, everything runs smoothly.
I tried changing the cost function from the more complicated one I was using to something simpler, but the error persists. I know this is probably related to the data input shape(?), but can't figure how the data would work for the optimizer but not the cost function.
# reading in data
filename = tf.train.string_input_producer(["file.csv"])
reader = tf.TextLineReader(skip_header_lines=1)
key, value = reader.read(filename)
rec_def = [[1], [1], [1]]
input_1, input_2, col3 = tf.decode_csv(value, record_defaults=rec_def)
# parameters
learning_rate = 0.001
training_steps = 300
x = tf.placeholder(tf.float32, [None,1])
x2 = tf.placeholder(tf.float32, [None,1])
m = tf.Variable(tf.zeros([1,1]))
m2 = tf.Variable(tf.zeros([1,1]))
b = tf.Variable(tf.zeros([1]))
y_ = tf.placeholder(tf.float32, [None,1])
y = tf.matmul(x,m) + tf.matmul(x2,m2) + b
# cost function
# cost = tf.reduce_mean(tf.log(1+tf.exp(-y_*y)))
cost = tf.reduce_sum(tf.pow((y_-y),2))
# Gradient descent optimizer
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
# initializing variables
init = tf.global_variables_initializer()
with tf.Session() as sess:
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(coord=coord)
sess.run(init)
for i in range(training_steps):
xs = np.array([[sess.run(input_1)]])
ys = np.array([[sess.run(input_2)]])
label = np.array([[sess.run(col3)]])
feed = {x:xs, x2:ys, y_:label}
sess.run(optimizer, feed_dict=feed)
cost_val = sess.run(cost)
coord.request_stop()
coord.join(threads)
The cost tensor is a function of the placeholder tensors and this requires them to have a value. Since the call to sess.run(cost) isn't feeding those placeholders, you're seeing the error. (Putting it another way - what values of x and y_ do you want to compute the cost for?)
So you want to change the line:
cost_val = sess.run(cost)
to:
cost_val = sess.run(cost, feed_dict=feed)
Hope that helps.