I have created a model with 3 hidden layers and trained it with the specific data-set.
How can I visualize the Model, with the neuron connections and weights at each iteration.
Here is the snippet of the python code :
#<ALL IMPORT STATEMENTS>
MODEL_DIR = <model_name>
def make_estimator(model_dir):
config = run_config.RunConfig(model_dir=model_dir)
feat_cols = [tf.feature_column.numeric_column("x", shape=<number_of_feat_cols>)]
return estimator.DNNClassifier(config=config, hidden_units=[<>,<>,<>],feature_columns=feat_cols,n_classes=2,optimizer=tf.train.GradientDescentOptimizer(learning_rate=0.001))
data = pd.read_csv(<csv_file>)
feat_data = data.drop('Type',axis=1)
feat_data_matrix = feat_data.as_matrix()
labels = data['Type']
labels_matrix = labels.as_matrix()
deep_model = make_estimator(MODEL_DIR)
input_fn = estimator.inputs.numpy_input_fn(x={'x':feat_data_matrix}, y=labels_matrix, shuffle=True, batch_size=10, num_epochs=1000)
tr_steps = <step_size>
deep_model.train(input_fn=input_fn,steps=tr_steps)
print ("Training Done")
In the code above, I have not created any tensorflow session, without it where can I implement the TensorBoard APIs for visualizing the model ?
By using the Python API simply call the method tf.summary.FileWriter
Then if you load the file written by the SummaryWriter into TensorBoard, the graph is shown.
You have to load the graph like this:
# Launch the graph.
current_session = tf.Session()
current_session.run(init)
# Create a summary writer, add the 'graph' to the event file.
writer = tf.summary.FileWriter(<some-directory>, current_session.graph)
See here.
Related
I am using Tensorflow Object Detection API. Recently it was updated to Tensorflow2.
And with it authors put out a great Colab https://github.com/tensorflow/models/blob/master/research/object_detection/colab_tutorials/eager_few_shot_od_training_tf2_colab.ipynb. They fine-tune RetinaNet on new dataset, however I don't understand how can I use this to fine-tune CenterNet (and EfficientDet).
They have the following code for initialising RetinaNet model:
tf.keras.backend.clear_session()
print('Building model and restoring weights for fine-tuning...', flush=True)
num_classes = 1
pipeline_config = 'models/research/object_detection/configs/tf2/ssd_resnet50_v1_fpn_640x640_coco17_tpu-8.config'
checkpoint_path = 'models/research/object_detection/test_data/checkpoint/ckpt-0'
# Load pipeline config and build a detection model.
#
# Since we are working off of a COCO architecture which predicts 90
# class slots by default, we override the `num_classes` field here to be just
# one (for our new rubber ducky class).
configs = config_util.get_configs_from_pipeline_file(pipeline_config)
model_config = configs['model']
model_config.ssd.num_classes = num_classes
model_config.ssd.freeze_batchnorm = True
detection_model = model_builder.build(
model_config=model_config, is_training=True)
# Set up object-based checkpoint restore --- RetinaNet has two prediction
# `heads` --- one for classification, the other for box regression. We will
# restore the box regression head but initialize the classification head
# from scratch (we show the omission below by commenting out the line that
# we would add if we wanted to restore both heads)
fake_box_predictor = tf.compat.v2.train.Checkpoint(
_base_tower_layers_for_heads=detection_model._box_predictor._base_tower_layers_for_heads,
# _prediction_heads=detection_model._box_predictor._prediction_heads,
# (i.e., the classification head that we *will not* restore)
_box_prediction_head=detection_model._box_predictor._box_prediction_head,
)
fake_model = tf.compat.v2.train.Checkpoint(
_feature_extractor=detection_model._feature_extractor,
_box_predictor=fake_box_predictor)
ckpt = tf.compat.v2.train.Checkpoint(model=fake_model)
ckpt.restore(checkpoint_path).expect_partial()
# Run model through a dummy image so that variables are created
image, shapes = detection_model.preprocess(tf.zeros([1, 640, 640, 3]))
prediction_dict = detection_model.predict(image, shapes)
_ = detection_model.postprocess(prediction_dict, shapes)
print('Weights restored!')
I tried to do similar thing with CenterNet model (it is used for inferencing in this Colab tutorial https://github.com/tensorflow/models/blob/master/research/object_detection/colab_tutorials/inference_tf2_colab.ipynb):
pipeline_config = 'models/research/object_detection/configs/tf2/centernet_hourglass104_512x512_coco17_tpu-8.config'
model_dir = 'models/research/object_detection/test_data/checkpoint/'
num_classes = 1
# Load pipeline config and build a detection model
configs = config_util.get_configs_from_pipeline_file(pipeline_config)
model_config = configs['model']
model_config.center_net.num_classes = num_classes
detection_model = model_builder.build(
model_config=model_config, is_training=True)
# Restore checkpoint
ckpt = tf.compat.v2.train.Checkpoint(
model=detection_model)
ckpt.restore(os.path.join(model_dir, 'ckpt-0')).expect_partial()
However, an exception is thrown because shapes are not compatible (because I changed number of classes). In the example with RetinaNet this trick was used (as I understand) to make tensors of right shapes:
fake_box_predictor = tf.compat.v2.train.Checkpoint(
_base_tower_layers_for_heads=detection_model._box_predictor._base_tower_layers_for_heads,
# _prediction_heads=detection_model._box_predictor._prediction_heads,
# (i.e., the classification head that we *will not* restore)
_box_prediction_head=detection_model._box_predictor._box_prediction_head,
)
fake_model = tf.compat.v2.train.Checkpoint(
_feature_extractor=detection_model._feature_extractor,
_box_predictor=fake_box_predictor)
But how can I discover what should I write inside checkpoint function? (for example, _base_tower_layers_for_heads=detection_model._box_predictor._base_tower_layers_for_heads or _box_prediction_head=detection_model._box_predictor._box_prediction_head)
I have trained a model and saved all the files (meta, index, checkpoint, etc.) using the saver = tf.compat.v1.train.Saver() function, and now I want to re-load that model in order to test on new data. It works fine, but my question is, every time I run the restored model on the same dataset (i.e. run it once on a testing dataset, and then start it over and run it again on that same dataset) I get very different results. I'm hoping to be able to run it over and over again on the same dataset, but get the same results.
I have two separate .py files, one for training and one for testing/loading the model to test on the dataset. My training variables/placeholders look something like this in the training.py file (in case it's relevant):
# set some tensorflow variables and placeholders, etc.
self.X = tf.compat.v1.placeholder(tf.float32, (None, self.state_size))
self.REWARDS = tf.compat.v1.placeholder(tf.float32, (None))
self.ACTIONS = tf.compat.v1.placeholder(tf.int32, (None))
feed_forward = tf.layers.dense(self.X, self.LAYER_SIZE, activation = tf.nn.relu)
self.logits = tf.layers.dense(feed_forward, self.OUTPUT_SIZE, activation = tf.nn.softmax)
input_y = tf.one_hot(self.ACTIONS, self.OUTPUT_SIZE)
loglike = tf.math.log((input_y * (input_y - self.logits) + (1 - input_y) * (input_y + self.logits)) + 1) # tf.log
rewards = tf.tile(tf.reshape(self.REWARDS, (-1,1)), [1, self.OUTPUT_SIZE])
self.cost = -tf.reduce_mean(loglike * (rewards + 1)) # leave this as a negative, so that the minimize function of the Adam optimizer will keep improving
# Adam Optimizer
self.optimizer = tf.compat.v1.train.AdamOptimizer(learning_rate = self.LEARNING_RATE).minimize(self.cost) # minimize(self.cost)
# Start the Tensorflow session
self.sess = tf.compat.v1.InteractiveSession()
self.sess.run(tf.compat.v1.global_variables_initializer())
...
saver = tf.compat.v1.train.Saver()
save_path = saver.save(self.sess, "./agent_output/" + name + "_model")
And in the testing.py file, it looks something like this:
...
# Start the Tensorflow session
self.sess = tf.compat.v1.InteractiveSession()
new_saver = tf.train.import_meta_graph('./agent_output/' + name + '_model.meta')
new_saver.restore(self.sess, tf.train.latest_checkpoint('./agent_output/'))
print('Model loaded step 1')
#saver = tf.compat.v1.train.Saver()
#saver.restore(self.sess, "./agent_output/" + name + "_model")
#print('Model Restored!')
self.sess.run(tf.compat.v1.global_variables_initializer())
Just to give you an idea of what I'm working with. As you can see, I've tried the import_meta_graph and the commented out saver.restore method, but I think I'm missing something, or if it's even possible in my case?
I'm just hoping someone can point me in the right direction. What I've discovered on my own is that there should be a way to not only load the variables, but also the graph? Or maybe I need to implement that during the training? I'm running Python 3.6 and Tensorflow 1.14 (I believe? Not 2.0).
Your problem is probably running self.sess.run(tf.compat.v1.global_variables_initializer()) after restoring the model. You only need to run this for a fresh model not a restored one. Try it without this line.
These discussion talked (1,2) about adding new layers to Tensorflow graph and retrain the model.
And the following code shows to add in new layer to restored trainable model.
import tensorflow as tf
sess=tf.Session()
#First let's load meta graph and restore weights
saver = tf.train.import_meta_graph('my_test_model-1000.meta')
saver.restore(sess,tf.train.latest_checkpoint('./'))
# Now, let's access and create placeholders variables and
# create feed-dict to feed new data
graph = tf.get_default_graph()
w1 = graph.get_tensor_by_name("w1:0")
w2 = graph.get_tensor_by_name("w2:0")
feed_dict ={w1:13.0,w2:17.0}
#Now, access the op that you want to run.
op_to_restore = graph.get_tensor_by_name("op_to_restore:0")
#Add more to the current graph
add_on_op = tf.multiply(op_to_restore,2)
print sess.run(add_on_op,feed_dict)
#This will print 120.
But I like to add in layers to restored frozen graph.
I have frozen model only for an application. I like to add in layers to the model and freeze again.
Those layers are more for post processing and not necessary to train so not in the trained model.
The reason why is I am converting the freeze graph to TensorRT and I like to include those layers into Int8 engine.
I hope below will help you. I have a custom Op which was supposed to be added to my existing graph which i loaded from .pb file (freezed model file)
With this i was able to append new nodes to my existing graph.
Source code below:
import tensorflow as tf
from tensorflow.python.framework import load_library
from tensorflow.python.platform import resource_loader
from tensorflow.core.protobuf import saved_model_pb2
from tensorflow.python.util import compat
# Utility functions for Loading and Freezing graphs
def load_graph(frozen_graph_filename):
with tf.gfile.GFile(frozen_graph_filename, "rb") as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
with tf.Graph().as_default() as graph:
tf.import_graph_def(graph_def, name="")
return graph
def freeze_graph(sess, output_graph):
output_node_names = [
"custom_op_zero","custom_op_zero_1"
output_node_names = ",".join(output_node_names)
output_graph_def = tf.graph_util.convert_variables_to_constants(
sess,
tf.get_default_graph().as_graph_def(),
output_node_names.split(",")
)
with tf.gfile.GFile(output_graph, "wb") as f:
f.write(output_graph_def.SerializeToString())
print("{} ops written to {}.".format(len(output_graph_def.node), output_graph))
## load custom Ops shared object file
zero_out_ops = load_library.load_op_library(
resource_loader.get_path_to_datafile('my-op/tensorflow_zero_out/python/ops/_zero_out_ops.so'))
zero_out = zero_out_ops.zero_out
frozen_graph = load_graph("frozen_model.pb")
all_tensors = [tensor for op in frozen_graph.get_operations() for tensor in op.values()]
#print (all_tensors[29])
# Input to the new node is the output of last node
zero_out_custom = zero_out(all_tensors[-1],name="custom_op_zero")
zero_out_custom1 = zero_out(all_tensors[-1],name="custom_op_zero_1")
#print (new_op)
# save new freezed model file
with tf.Session(graph=frozen_graph) as persisted_sess:
for op in persisted_sess.graph.get_operations():
print(op)
freeze_graph(persisted_sess,"new_model.pb")
When following the tensorflow image classification tutorial, at first it caches the bottleneck of each image:
def: cache_bottlenecks())
I have rewritten the training using tensorflow's Estimator. This really simplified all the code. However I want to cache the bottleneck features here.
Here is my model_fn. I want to cache the results of the dense layer so I can make changes to the actual training without having to compute the bottlenecks each time.
How can I accomplish that?
def model_fn(features, labels, mode, params):
is_training = mode == tf.estimator.ModeKeys.TRAIN
num_classes = len(params['label_vocab'])
module = hub.Module(params['module_spec'], trainable=is_training and params['train_module'])
bottleneck_tensor = module(features['image'])
with tf.name_scope('final_retrain_ops'):
logits = tf.layers.dense(bottleneck_tensor, units=num_classes, trainable=is_training) # save this?
def train_op_fn(loss):
optimizer = tf.train.AdamOptimizer()
return optimizer.minimize(loss, global_step=tf.train.get_global_step())
head = tf.contrib.estimator.multi_class_head(n_classes=num_classes, label_vocabulary=params['label_vocab'])
return head.create_estimator_spec(
features, mode, logits, labels, train_op_fn=train_op_fn
)
TF cannot work as you code. You should:
Export bottleneck to file from the raw net.
Use bottleneck result as input, use another net to train your data.
To expand on what #Feng said:
see TFRecords and TFExamples and Load Images
Something like this should work (untested):
# Serialize the data into two tfrecord files
tf.enable_eager_execution()
feature_extractor = ...
features_file = tf.python_io.TFRecordWriter('features.tfrec')
label_file = tf.python_io.TFRecordWriter('labels.tfrec')
for images, labels in dataset:
features = feature_extractor(images)
features_file.write(tf.serialize_tensor(features))
label_file.write(tf.serialize_tensor(labels))
# Parse the files and zip them together
def parse(type, shape):
_def parse(x):
result = tf.parse_tensor(x, out_type=shape)
result = tf.reshape(result, FEATURE_SHAPE)
return result
return parse
features_ds = tf.data.TFRecordDataset('features.tfrec')
features_ds = features_ds.map(parse(tf.float32, FEATURE_SHAPE), num_parallel_calls=AUTOTUNE)
labels_ds = tf.data.TFRecordDataset('labels.tfrec')
labels_ds = labels_ds.map(parse(tf.float32, FEATURE_SHAPE), num_parallel_calls=AUTOTUNE)
ds = tf.data.Dataset.zip(features_ds, labels_ds)
ds = ds.unbatch().shuffle().repeat().batch().prefetch()...
You might also be able to do it using Dataset.cache, but I'm not 100% sure of the details.
I'm trying to fine tune inception models, and validate it with test data. But all the examples given at tensorflow slime web page only either fine-tuning or testing, there is not any example that doing both at same graph and session.
Basically I want to this.
with tf.Graph().as_default():
image, image_raw, label,image_name, label_name = dut.distorted_inputs(params,is_training=is_training)
test_image, test_image_raw, test_label,test_image_name, test_label_name = dut.distorted_inputs(params,is_training=False)
# I'm creating as it is suggested at github slim page:
logits, _ =inception.inception_v2(image, num_classes=N, is_training=True)
tf.get_variable_scope().reuse_variables()
logits_tes, _ =inception.inception_v2(test_image, num_classes=N, is_training=Test)
err=tf.sub(logits, label)
losses = tf.reduce_mean(tf.reduce_sum(tf.square(err)))
# total_loss = model_loss+losses
total_loss = losses+slim.losses.get_total_loss()
test_err=tf.sub(test_logits, test_label)
test_loss= tf.reduce_mean(tf.reduce_sum(tf.square(test_err)))
optimizer = tf.train.AdamOptimizer(learning_rate=0.001)
train_op = slim.learning.create_train_op(total_loss, optimizer)
final_loss = slim.learning.train(
train_op,
logdir=params["cp_file"],
init_fn=ut.get_init_fn(slim,params),
number_of_steps=2,
summary_writer=summary_writer
)
this code fails As it can be seen, I don't have loop separately to call my test models, I want to test my model on my test data at each 10th batch.
Does calling train with number_of_steps=10 and then using the evaluation code work?