I want to set the gpu limitation as instructed in this topic.
But my code goes like this:
deep_grap = tf.Graph()
with deep_grap.as_default():
### graph definition here
### graph definition here
with tf.Session(graph=deep_grap) as sess:
tf.initialize_all_variables().run()
### more computations here
In this case, how do I set the configuration in my code?
I don't have a direct sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))line here. Thanks!
You can pass the session configuration, a tf.ConfigProto in the tf.Session() initializer in the with statement:
deep_graph = tf.Graph()
with deep_graph.as_default():
### graph definition here
### graph definition here
config = tf.ConfigProto(gpu_options=...)
with tf.Session(graph=deep_graph, config=config) as sess:
tf.initialize_all_variables().run()
### more computations here
deep_grap = tf.Graph()
with deep_grap.as_default():
### graph definition here
### graph definition here
init = tf.initialize_all_variables()
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.333)
cfg = tf.ConfigProto(gpu_options=gpu_options)
with tf.Session(graph=deep_grap, config=cfg) as sess:
sess.run(init)
### more computations here
Related
I am trying to run the MTCNN and FaceNet model on 2 cameras simultaneously. So, I am not getting any error while doing this but the code doesn't give me any results.
It just loads both the models and doesn't give me any predictions. Can anyone help me with this?
I have created 2 separate graphs and sessions using g=tf.Graph for MTCNN and FaceNet.
I think this error is coming due to multi-processing with TensorFlow as it might try to load MTCNN input to the Facenet graph. *this is my assumption.
Please let me know if you have any ideas about this. Thanks.
FaceNet:
with face_rec_graph.graph.as_default():
self.sess = tf.Session()
with self.sess.as_default():
self.__load_model(model_path)
self.x = tf.get_default_graph() \
.get_tensor_by_name("input:0")
self.embeddings = tf.get_default_graph() \
.get_tensor_by_name("embeddings:0")
self.phase_train_placeholder = tf.get_default_graph() \
.get_tensor_by_name("phase_train:0")
print("Model loaded")
face_rec_graph was created as follows:
class FaceRecGraph(object):
def __init__(self):
self.graph = tf.Graph();
MTCNN:
graph = tf.Graph()
with graph.as_default():
with open(model_path, 'rb') as f:
graph_def = tf.GraphDef.FromString(f.read())
tf.import_graph_def(graph_def, name='')
self.graph = graph
config = tf.ConfigProto(
allow_soft_placement=True,
intra_op_parallelism_threads=4,
inter_op_parallelism_threads=4)
config.gpu_options.allow_growth = True
self.sess = tf.Session(graph=graph, config=config)
There is no error coming just both the cameras stop giving any result.
I have very simple model which consists of one tf.Variable() and here is who code:
import tensorflow as tf
save_path="model1/model1.ckpt"
num_input = 2
n_nodes_hl1 = 2
with tf.variable_scope("model1"):
hidden_1_layer = {
'weights' : tf.Variable(tf.random_normal([num_input, n_nodes_hl1]), name='Weight1')
}
def train_model():
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
save_model(sess)
def save_model(sess):
saver = tf.train.Saver(tf.global_variables(), save_path)
saver.save(sess, save_path)
def load_model(sess):
saver = tf.train.Saver(tf.global_variables(), save_path)
saver.restore(sess, save_path)
def run_model():
print("model1 running...")
with tf.Session() as sess:
load_model(sess)
x = sess.run(hidden_1_layer)
print(x)
#train_model()
The second model is completely the same, but with changed names "model1" to "model2". Both models are trained, saved and work perfect separately. So now I want to test them using following script:
import model1 as m1
import model2 as m2
m1.run_model()
m2.run_model()
And here I got an error message:
NotFoundError (see above for traceback): Key model2/Weight2 not found in checkpoint
So it looks like running imports causes adding all variables to common graph (even though they are in separate variable scopes) and then it cannot find variable from model2 saved in checkpoint in model1.
Can anyone solve my problem?
Is it possible in Tensorflow to run a few different models in one script?
EDIT - PROBLEM SOLVED
The solution is very easy. What you have to do is to create separate graphs for each model like. It means that all tensors you declare or calculate must be within that graph. You also must put it as an argument in Session, like: tf.Session(graph=self.graph)
Whole example below:
import tensorflow as tf
save_path="model1/model1.ckpt"
class model1:
num_input = 2
n_nodes_hl1 = 2
def init(self):
self.graph = tf.Graph()
with self.graph.as_default():
with tf.variable_scope("model1"):
self.hidden_1_layer = {
'weights' : tf.Variable(tf.random_normal([self.num_input, self.n_nodes_hl1]), name='Weight1')
}
def train_model(self):
init = tf.global_variables_initializer()
with tf.Session(graph = self.graph) as sess:
sess.run(init)
self.save_model(sess)
def save_model(self, sess):
saver = tf.train.Saver(tf.global_variables(), save_path)
saver.save(sess, save_path)
def load_model(self, sess):
saver = tf.train.Saver(tf.global_variables(), save_path)
saver.restore(sess, save_path)
def run_model(self):
print("model1 running...")
with tf.Session(graph = self.graph) as sess:
self.load_model(sess)
x = sess.run(self.hidden_1_layer)
print(x)
Oh! the common "I want to use several models" question! just make sure that you reset the graph after each model:
tf.reset_default_graph()
Your code would look like:
import tensorflow as tf
import model1 as m1
m1.run_model()
tf.reset_default_graph()
import model2 as m2
m2.run_model()
Why? The moment you create a variable in tensorflow using tf.Variable, that variable is added to the default graph. If you import both models one after the other, you just created all the variables in the default graph! This is by far the easiest solution. Consider the default graph as a blackboard: you can draw your fancy ML model, but you need to wipe it clean before reuse!
NOTE: If you are wondering, the alternative is to create separate graphs for each of the models, but it is much more worrysome and I only recommend it for times when you must have both models at the same time.
EXTRA: Encapsulating your model in a Tensorflow class
A fancier way to do it while avoiding several graphs (seriously, it is horrible!) is to encapsulate the whole model in a class. Thus, your code would look like this:
import tensorflow as tf
class model():
self.num_input = 2
self.n_nodes_hl1 = 2
def init(self, new_save_path)
self.save_path=new_save_path
tf.reset_default_graph()
with tf.variable_scope("model1"):
self.hidden_1_layer = {
'weights' : tf.Variable(tf.random_normal([self.num_input,
self.n_nodes_hl1]), name='Weight1')
}
self.saver = tf.train.Saver(tf.global_variables(), self.save_path)
self.sess = tf.Session()
self.sess.run(tf.global_variables_initializer())
def save_model(self):
self.saver.save(self.sess, self.save_path)
def load_model(self):
self.saver.restore(self.sess, self.save_path)
def run_model(self):
print("model1 running...")
load_model()
x = sess.run(self.hidden_1_layer)
print(x)
#train_model(self)
This way you could simply do:
import model
m1 = model('model1/model1.ckpt') # These two lines could be put into one
m1.run_model() # m1 = model('model1/model1.ckpt').run_model()
m2 = model('model2/model2.ckpt')
m2.run_model()
You still want it in a for loop?
import model
model_file_list = ['model1/model1.ckpt', 'model2/model2.ckpt']
for model_file in model_list:
m = model(model_file ).run_model()
# Run tests, print stuff, save stuff here!
Say I have defined a function that loads one label/features pair from a TfRecords file as follows
def read_one_image(tfrecords_path):
queue = tf.train.string_input_producer([tfrecords_path])
reader = tf.TFRecordReader()
key, value = reader.read(queue)
features = tf.parse_single_example(value,
features={'label': tf.FixedLenFeature([], tf.int64),
'image': tf.FixedLenFeature([784], tf.int64)})
label = features['label']
image = features['image']
return label, image
Fetching the images in a session works fine if I keep the session open:
tf.reset_default_graph()
label, image = read_one_image("mnist_train.tfrecords")
sess = tf.Session()
init = tf.global_variables_initializer()
sess.run(init)
tf.train.start_queue_runners(sess=sess)
for i in range(10):
one_label, one_image = sess.run([label, image])
print(one_label, one_image.shape)
However, if I use a context manager like so
g = tf.Graph()
with g.as_default():
label, image = read_one_image("mnist_train.tfrecords")
with tf.Session(graph=g) as sess:
sess.run(tf.global_variables_initializer())
tf.train.start_queue_runners(sess=sess)
for i in range(10):
one_label, one_image = sess.run([label, image])
print(one_label, one_image.shape)
I get an error: 7 ERROR:tensorflow:Exception in QueueRunner: Attempted to use a closed Session.(784,)
Maybe I am misunderstanding how the queue runner works, but since I called the sess.run method, it should have fetched a data pair 10 times. Now, is there a way to quit/exit/close the session without exhausting the queue?
You need to the tf.train.Coordinator
sess.run(tf.global_variables_initializer())
coord = tf.train.Coordinator()
tf.train.start_queue_runners(sess=sess, coord=coord)
I am trying to restore graph from model which I train with TensorFlow tutorials, then I try to restore the model:
import tensorflow as tf
import reader
from ptb_word_lm import PTBInput, PTBModel, get_config, run_epoch
def main(_):
checkpoint_path = "/Users/roger/data/ptb_out"
checkpoint_path = tf.train.latest_checkpoint(checkpoint_path)
raw_data = reader.ptb_raw_data("/Users/roger/data/simple-examples/small_data")
train_data, valid_data, test_data, _ = raw_data
config = get_config()
eval_config = get_config()
eval_config.batch_size = 1
eval_config.num_steps = 1
with tf.Session() as session:
initializer = tf.random_uniform_initializer(-config.init_scale,
config.init_scale)
saver = tf.train.import_meta_graph(checkpoint_path + ".meta")
saver.restore(session, checkpoint_path)
with tf.name_scope("Test"):
test_input = PTBInput(config=eval_config, data=test_data, name="TestInput")
with tf.variable_scope("Model", reuse=True, initializer=initializer):
mtest = PTBModel(is_training=False, config=eval_config,
input_=test_input)
test_perplexity = run_epoch(session, mtest)
print("Test Perplexity: %.3f" % test_perplexity)
if __name__ == "__main__":
tf.app.run()
However, I find that Varible Model/embedding which created here is not restored from graph. So I get error like this:
ValueError: Variable Model/embedding does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope?
So how can I restore the model correctly?
I think, since you set reuse=True in your variable scope, it tries to find that variable instead of creating it when you call PTBModel(). If you use get_variable() with reuse=True in a scope, it will never create a variable.
I'm loading a pretrained network into Tensorflow using the methods below that are within a Network class (hence the calls to self.xyz). First, define_network() is called, then I do initialization of other variables and optimizers, then load_model() is called.
However, despite using tf.variable_scope(self.name) the variables from the graph are loaded into the generic space of variables. This is problematic as I have two instances of this class that each load the same network and I want to separate the out into different scopes.
How can I load the variables into a specific scope?
P.S. Feel free to correct me on any errors in my code!
def load_model(self):
with tf.variable_scope(self.name) as scope:
self.saver.restore(self.sess, self.model_path)
print("Loaded model from {}".format(self.model_path))
def define_model(self):
with tf.variable_scope(self.name) as scope:
self.saver = tf.train.import_meta_graph(self.model_path + '.meta')
print("Loaded model from {}".format(self.model_path + '.meta'))
graph = tf.get_default_graph()
self.inputs = []
inp_names = ['i_hand1:0', 'i_hand2:0', 'i_flop1:0', 'i_flop2:0', 'i_flop3:0',
'i_turn:0', 'i_river:0', 'i_other:0', 'i_allowed_mod:0', 'keras_learning_phase:0']
for inp in inp_names:
self.inputs.append(tf.get_default_graph().get_tensor_by_name(inp))
self.outputs = tf.get_default_graph().get_tensor_by_name("Tanh:0")
self.add_output_conversions()
all_vars = tf.trainable_variables()
for var in all_vars:
self.var[var.name] = var
I think your problem can be solved by adding an argument into
self.saver = tf.train.import_meta_graph(self.model_path + '.meta', 'import_scope'=self.name)
Here's the reference