I try to simply save and restore a graph, but the simplest example does not work as expected (this is done using version 0.9.0 or 0.10.0 on Linux 64 without CUDA using python 2.7 or 3.5.2)
First I save the graph like this:
import tensorflow as tf
v1 = tf.placeholder('float32')
v2 = tf.placeholder('float32')
v3 = tf.mul(v1,v2)
c1 = tf.constant(22.0)
v4 = tf.add(v3,c1)
sess = tf.Session()
result = sess.run(v4,feed_dict={v1:12.0, v2:3.3})
g1 = tf.train.export_meta_graph("file")
## alternately I also tried:
## g1 = tf.train.export_meta_graph("file",collection_list=["v4"])
This creates a file "file" that is non-empty and also sets g1 to something that looks like a proper graph definition.
Then I try to restore this graph:
import tensorflow as tf
g=tf.train.import_meta_graph("file")
This works without an error, but does not return anything at all.
Can anyone provide the necessary code to simply just save the graph for "v4" and completely restore it so that running this in a new session will produce the same result?
To reuse a MetaGraphDef, you will need to record the names of interesting tensors in your original graph. For example, in the first program, set an explicit name argument in the definition of v1, v2 and v4:
v1 = tf.placeholder(tf.float32, name="v1")
v2 = tf.placeholder(tf.float32, name="v2")
# ...
v4 = tf.add(v3, c1, name="v4")
Then, you can use the string names of the tensors in the original graph in your call to sess.run(). For example, the following snippet should work:
import tensorflow as tf
_ = tf.train.import_meta_graph("./file")
sess = tf.Session()
result = sess.run("v4:0", feed_dict={"v1:0": 12.0, "v2:0": 3.3})
Alternatively, you can use tf.get_default_graph().get_tensor_by_name() to get tf.Tensor objects for the tensors of interest, which you can then pass to sess.run():
import tensorflow as tf
_ = tf.train.import_meta_graph("./file")
g = tf.get_default_graph()
v1 = g.get_tensor_by_name("v1:0")
v2 = g.get_tensor_by_name("v2:0")
v4 = g.get_tensor_by_name("v4:0")
sess = tf.Session()
result = sess.run(v4, feed_dict={v1: 12.0, v2: 3.3})
UPDATE: Based on discussion in the comments, here a the complete example for saving and loading, including saving the variable contents. This illustrates the saving of a variable by doubling the value of variable vx in a separate operation.
Saving:
import tensorflow as tf
v1 = tf.placeholder(tf.float32, name="v1")
v2 = tf.placeholder(tf.float32, name="v2")
v3 = tf.mul(v1, v2)
vx = tf.Variable(10.0, name="vx")
v4 = tf.add(v3, vx, name="v4")
saver = tf.train.Saver([vx])
sess = tf.Session()
sess.run(tf.initialize_all_variables())
sess.run(vx.assign(tf.add(vx, vx)))
result = sess.run(v4, feed_dict={v1:12.0, v2:3.3})
print(result)
saver.save(sess, "./model_ex1")
Restoring:
import tensorflow as tf
saver = tf.train.import_meta_graph("./model_ex1.meta")
sess = tf.Session()
saver.restore(sess, "./model_ex1")
result = sess.run("v4:0", feed_dict={"v1:0": 12.0, "v2:0": 3.3})
print(result)
The bottom line is that, in order to make use of a saved model, you must remember the names of at least some of the nodes (e.g. a training op, an input placeholder, an evaluation tensor, etc.). The MetaGraphDef stores the list of variables that are contained in the model, and helps to restore these from a checkpoint, but you are required to reconstruct the tensors/operations used in training/evaluating the model yourself.
Because tf.train.import_meta_graph is deprecated version now.
replace tf.train.import_meta_graph in your code with tf.compat.v1.train.import_meta_graph
It will solve your error.
Related
I have been following the methodology of saving/restoring GPflow models with success. But now I've run into a snag.
When I try to restore a model with a Linear mean function, the restore crashes with an error.
I think that the issue comes in the naming convention of the tensorflow Linear mean function object. The above "-44dbadbb-0" is random and changes every time the model is rebuilt, so if I check the tensor names when I saved the model with
from tensorflow.python.tools.inspect_checkpoint import print_tensors_in_checkpoint_file
print_tensors_in_checkpoint_file(file_name='./model.ckpt', tensor_name='', all_tensors=False)
I get the return:
Linear-eeb5f9f3-0/A/unconstrained (DT_DOUBLE) [1,1]
Linear-eeb5f9f3-0/b/unconstrained (DT_DOUBLE) [1]
model/X/dataholder (DT_DOUBLE) [15,1]
model/Y/dataholder (DT_DOUBLE) [15,1]
model/kern/kernels/0/lengthscales/unconstrained (DT_DOUBLE) []
model/kern/kernels/0/variance/unconstrained (DT_DOUBLE) []
model/kern/kernels/1/lengthscales/unconstrained (DT_DOUBLE) []
model/kern/kernels/1/variance/unconstrained (DT_DOUBLE) []
model/likelihood/variance/unconstrained (DT_DOUBLE) []
Where the Linear function clearly has a different name from the model which is trying to be restored.
I have tried to fix this by renaming the variables before the restore, but this doesn't work with tensorflow. I also tried different saving/restoring methods, but then I have problems with being able to sample from the model.
Saving the Model
import gpflow
import numpy as np
import random
import tensorflow as tf
# define data
rng = np.random.RandomState(4)
X = rng.uniform(0, 5.0, 15)[:, np.newaxis]
Y = np.sin((X[:, 0] - 2.5) ** 2).reshape(len(X),1)
# define the mean function
mf = gpflow.mean_functions.Linear(np.ones((1,1)),np.zeros((1,)))
# create the GP model
with gpflow.defer_build():
k = gpflow.kernels.Matern32(1)+gpflow.kernels.RBF(1)
m = gpflow.models.GPR(X, Y, kern=k,name='model',mean_function=mf)
m.likelihood.variance = 1e-03
m.likelihood.trainable = False
tf.global_variables_initializer()
tf_session = m.enquire_session()
m.compile( tf_session )
gpflow.train.ScipyOptimizer().minimize(m)
saver = tf.train.Saver()
save_path = saver.save(tf_session, "./model.ckpt")
print("Model saved in path: %s" % save_path)
Restoring the Model
import gpflow
import numpy as np
import random
import tensorflow as tf
# define data
rng = np.random.RandomState(4)
X = rng.uniform(0, 5.0, 15)[:, np.newaxis]
Y = np.sin((X[:, 0] - 2.5) ** 2).reshape(len(X),1)
# define the mean function
mf = gpflow.mean_functions.Linear(np.ones((1,1)),np.zeros((1,)))
with gpflow.defer_build():
k = gpflow.kernels.Matern32(1)+gpflow.kernels.RBF(1)
m = gpflow.models.GPR(X, Y, kern=k,name='model',mean_function=mf)
m.likelihood.variance = 1e-03
m.likelihood.trainable = False
# construct and compile the tensorflow session
tf.global_variables_initializer()
tf_session = m.enquire_session()
m.compile( tf_session )
saver = tf.train.Saver()
save_path = saver.restore(tf_session, "./model.ckpt")
print("Model loaded from path: %s" % save_path)
m.anchor(tf_session)
The code crashes at save_path = saver.restore(tf_session, "./model.ckpt") with the error:
NotFoundError (see above for traceback): Key Linear-44dbadbb-0/A/unconstrained not found in checkpoint...
The defer_build() does a bunch of things - but one part of constructing the entire model (i.e. tensorflow graph) in one go is that all the tensorflow variables & placeholders get consistent names, with all their names relating to the name of the model itself (which you set by passing the name='model' keyword argument to the model constructor).
In your code, however, the Linear mean function is constructed outside of the defer_build() scope. This means gpflow has to construct a graph for it right away - including setting up variables for the parameters (slope & offset in this case). All tensorflow variables live in a global name space, so the only way of allowing more than a single object to be created is to assign them randomized names. (E.g., imagine wanting to construct a sum of two kernels of the same type!)
Fortunately, the fix is easy: simply move the construction of the mean function into the defer_build block:
with gpflow.defer_build():
# define the mean function
mf = gpflow.mean_functions.Linear(np.ones((1,1)), np.zeros((1,)))
k = gpflow.kernels.Matern32(1) + gpflow.kernels.RBF(1)
m = gpflow.models.GPR(X, Y, kern=k, mean_function=mf, name='model')
m.likelihood.variance = 1e-03
m.likelihood.trainable = False
# construct and compile the tensorflow session
tf.global_variables_initializer()
tf_session = m.enquire_session()
m.compile(tf_session)
If you do this in both the "save" and "load" scripts, everything runs and hopefully as you expect it. Hope this helps!
Does tensorflow need to set the reuse ==True when finish training and inference?
I have a network like this:
def __build_net(self,placeholder,reuse=False):
with tf.variable_scope('siamse',reuse=reuse):
layer = tf.layers.dense(placeholder,3000,activation=tf.nn.leaky_relu)
layer = tf.layers.batch_normalization(layer)
embedding= tf.layers.dense(layer,300,activation = tf.nn.leaky_relu)
print('Siamse Net has built',flush=True)
return embedding
And I create two network share same parameter:
self.embedding1=self.__build_net(self.centers_placeholder)
self.embedding2=self.__build_net(self.neighbors_placeholder,reuse=True)
I used this network to generate embeddings of some kind of data.
My question is: Do I need to set the reuse to True when doing inference(generate embedding) like this:
with tf.Session() as sess:
self.saver.restore(sess,self.store_path+self.model_type+'_model_'+str(self.model_num)+'_'+str(self.center_size)+'_'+str(self.neighbor_size)+'.ckpt')
embedding = self.__build_net(self.centers_placeholder,reuse=True)
embeddings = sess.run(embedding,feed_dict = {self.centers_placeholder : data})
Or like this:
with tf.Session() as sess:
self.saver.restore(sess,self.store_path+self.model_type+'_model_'+str(self.model_num)+'_'+str(self.center_size)+'_'+str(self.neighbor_size)+'.ckpt')
embedding = self.__build_net(self.centers_placeholder,reuse=False)
embeddings = sess.run(embedding,feed_dict = {self.centers_placeholder : data})
And then, When set the variable scope, do I need to give a name to each layer?
Thanks!
No....reuse means whether you need to use a previously defined variable.
Say, you've created a variable called 'foo/v':
with tf.variable_scope("foo"):
v = tf.get_variable("v", [1])
print(v.name) ---> foo/v:0
Running the following will give:
with tf.variable_scope("foo"):
v1 = tf.get_variable("v", [1]) ---> gives error as name 'foo/v' exists
print(v1.name)
with tf.variable_scope("foo", reuse=False):
v1 = tf.get_variable("v", [1]) ---> gives error as name 'foo/v' exists
print(v1.name)
with tf.variable_scope("foo", reuse=True):
v1 = tf.get_variable("v", [1])
print(v1.name) ---> foo/v:0
with tf.variable_scope("foo", reuse=tf.AUTO_REUSE):
v1 = tf.get_variable("v", [1])
print(v1.name) ---> foo/v:0
But if you run the following from the very begining:
with tf.variable_scope("foo", reuse=True):
v1 = tf.get_variable("v", [1])
print(v1.name) ---> gives error as 'foo/v' does not exist (thus cannot be reused).
Thus I prefer setting reuse=tf.AUTO_REUSE all the time.
For a detailed explanation, please read How Does Variable Scope Work? from the TensorFlow official guide.
By the way, tf.layers.batch_normalization has a training option that needs to be set False during inference. See the explanations here.
I have read through previous strings. My data are in the form of an array fed to a placeholder. Trying to convert the data to a tensor before feeding produces a different (inverse) error message. Other solutions similarly do not seem to work in this situation. Here is minimal code.
from __future__ import print_function
import numpy as np
import tensorflow as tf
from tensorflow.contrib.factorization import KMeans
X = tf.placeholder(tf.float32, shape=[None, 10], name="X")
data = np.random.randn(2,10)
def lump(X):
# Build KMeans graph
kmeans = KMeans(inputs=X, num_clusters=k, distance_metric='cosine',
use_mini_batch=True)
(all_scores, cluster_idx, scores, cluster_centers_initialized, cluster_centers_var, init_op,
train_op) = kmeans.training_graph()
cluster_idx = cluster_idx[0] # fix for cluster_idx being a tuple
avg_distance = tf.reduce_mean(scores)
return cluster_idx, scores
# Initialize the variables (i.e. assign their default value)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
idx, d = sess.run(lump,feed_dict={X: data})
Correct, you can't evaluate just lump, because it's a function (returning tensors), not a tensor or an op. You probably meant to do something like this:
cluster_idx, scores = lump(X)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
idx, d = sess.run([cluster_idx, scores], feed_dict={X: data})
Note that lump() is invoked before tf.global_variables_initializer(), because it defines new variables in the graph, so they must be initialized.
The code still fails, because lump is clearly not finished and has issues with dimensions, but it is the right way to evaluate something in a session.
The network structure has already been loaded into the default global graph. I want to create another graph with the same structure and load checkpoints into this graph.
If the code is like this, it will throw error: ValueError: No variables to save in the last line. However, the second line works fine. Why? Does GraphDef returned by as_graph_def() contains variable definition/name?
inference_graph_def = tf.get_default_graph().as_graph_def()
saver = tf.train.Saver()
with tf.Graph().as_default():
tf.import_graph_def(inference_graph_def)
saver1 = tf.train.Saver()
If the code like this, it will throw error Cannot interpret feed_dict key as Tensor: The name 'save/Const:0' refers to a Tensor which does not exist in last line. Howerver, it works fine with the 3rd line removed.
inference_graph_def = tf.get_default_graph().as_graph_def()
saver = tf.train.Saver()
with tf.Graph().as_default():
tf.import_graph_def(inference_graph_def)
with session.Session() as sess:
saver.restore(sess, checkpoint_path)
So, does this mean Saver cannot work in different graphs even though they have the same structure?
Any help would be appreciated~
Here's an example of using a MetaGraphDef, which unlike GraphDef saves variable collections, to initialize a new graph using a previously saved graph.
import tensorflow as tf
CHECKPOINT_PATH = "/tmp/first_graph_checkpoint"
with tf.Graph().as_default():
some_variable = tf.get_variable(
name="some_variable",
shape=[2],
dtype=tf.float32)
init_op = tf.global_variables_initializer()
first_meta_graph = tf.train.export_meta_graph()
first_graph_saver = tf.train.Saver()
with tf.Session() as session:
init_op.run()
print("Initialized value in first graph", some_variable.eval())
first_graph_saver.save(
sess=session,
save_path=CHECKPOINT_PATH)
with tf.Graph().as_default():
tf.train.import_meta_graph(first_meta_graph)
second_graph_saver = tf.train.Saver()
with tf.Session() as session:
second_graph_saver.restore(
sess=session,
save_path=CHECKPOINT_PATH)
print("Variable value after restore", tf.global_variables()[0].eval())
Prints something like:
Initialized value in first graph [-0.98926258 -0.09709156]
Variable value after restore [-0.98926258 -0.09709156]
Note that the checkpoint is still important! Loading the MetaGraph does not restore the values of Variables (it doesn't contain those values), just the bookkeeping which tracks their existence (collections). SavedModel format addresses this, bundling MetaGraphs with checkpoints and other metadata for running them.
Edit: By popular demand, here's an example of doing the same thing with a GraphDef. I don't recommend it. Since none of the collections are restored when the GraphDef is loaded, we have to manually specify the Variables we want the Saver to restore; the "import/" default naming scheme is easy enough to fix with a name='' argument to import_graph_def, but removing it isn't super helpful since you'd need to manually fill in the variables collection if you wanted the Saver to work "automatically". Instead I've chosen to specify a mapping manually when creating the Saver.
import tensorflow as tf
CHECKPOINT_PATH = "/tmp/first_graph_checkpoint"
with tf.Graph().as_default():
some_variable = tf.get_variable(
name="some_variable",
shape=[2],
dtype=tf.float32)
init_op = tf.global_variables_initializer()
first_graph_def = tf.get_default_graph().as_graph_def()
first_graph_saver = tf.train.Saver()
with tf.Session() as session:
init_op.run()
print("Initialized value in first graph", some_variable.eval())
first_graph_saver.save(
sess=session,
save_path=CHECKPOINT_PATH)
with tf.Graph().as_default():
tf.import_graph_def(first_graph_def)
variable_to_restore = tf.get_default_graph().get_tensor_by_name(
"import/some_variable:0")
second_graph_saver = tf.train.Saver(var_list={
"some_variable": variable_to_restore
})
with tf.Session() as session:
second_graph_saver.restore(
sess=session,
save_path=CHECKPOINT_PATH)
print("Variable value after restore", variable_to_restore.eval())
How can I assign a value to a tf Variable inside a function?
Based on the link here, it say thats you have to run a sess on the tf tensor. I want to update the tf variable inside the function after few calculations.
Example:
def update(weights):
value_1 = 0
value_2 = 2
........... some code here ...........
weights['layer_1'] = tf.multiply(weights['layer_1'],value_1)
weights['layer_2'] = tf.multiply(weights['layer_2'],value_2)
............some code here.............
I can't do the above code. But how do I use assign to make this code work?
You have to use assign, which take a Tensor which has to be exactly the same shape as the original Variable. If you want to have different shape use the validate_shape=False. But you have to keep in mind that you'll get the actual changes on run time, thus you will code the behavior of your variable not assigning values.
Here an example that shows variable assignment with variable shapes:
import tensorflow as tf
var = tf.Variable(tf.zeros((1, 3)))
new_v = tf.assign(var, tf.ones((5, 7)), validate_shape=False)
init_op = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init_op)
print sess.run([var])
print sess.run([new_v])
For your particular example you could try:
def update(weights):
value_1 = tf.constant(0)
value_2 = tf.constant(2)
........... some code here ...........
weights['layer_1'] = tf.assign(weights['layer_1'], tf.multiply(weights['layer_1'],value_1))
weights['layer_2'] = tf.assign(weights['layer_2'], tf.multiply(weights['layer_2'],value_2))
............some code here.............
This works for me -
import tensorflow as tf
import numpy as np
# function to randomly initialize weights for a specific layer
def assign_var(layer_number):
weight_value = np.random.rand(5,3) # or any calculations you need
weight_var = tf.get_variable('weights_layer_'+str(layer_number))
return tf.assign(weight_var,weight_value)
with tf.Session() as sess:
sess.run(assign_var(1))
sess.run(assign_var(2))
EDIT The problem with the above code is - it keeps adding to the graph every time you call the function.
Alternatively, I think this should be better.
import tensorflow as tf
import numpy as np
var_name = tf.placeholder(tf.string)
weight_value = tf.placeholder(tf.float32)
weight_var = tf.get_variable(var_name)
assign_weights = tf.assign(weight_var,weight_value)
sess = tf.Session()
# function to randomly initialize weights for a specific layer
def assign_var(layer_number):
rand_weight_value = np.random.rand(5,3) # or any calculations you need
sess.run(assign_weights,{var_name:'weights_layer'+str(layer_number),weight_value:rand_weight_value})
assign_var(1) # assigns random weight values to layer 1