I've started recently to play with tensorflow and, more specifically, with the new dataset API.
I've successfully used a dataset to feed training data to my simple model by plugging dataset's iterators to the nodes of my graph representing input and label. Something like:
input = input_dataset.make_one_shot_iterator().get_next()
label = label_dataset.make_one_shot_iterator().get_next()
Now I'm wondering what to do when I have to do inference on a user input, that is, the user gives me one single input value and I have to make my prediction. If I had a placeholder I would just put the user input in a feed_dict, but with the dataset api I have very little idea how to do something similar. Shall I have a separate graph only for inference in which my input variable is a placeholder?
I've tried already to make a feedable iterator as described here but that only works with a placeholder for strings, while my input are int32.
Thanks for any advice.
For that specific purpose, tensorflow provides tf.placeholder_with_default API
# Create a Dataset
dataset = tf.data.Dataset.zip((input_dataset, label_dataset)).batch(32).repeat(...)
# Create Iterator
input, label = dataset.make_one_shot_iterator()
# Create Placholders
x = tf.placeholder_with_default(input, shape=[...], name='input')
y = tf.placeholder_with_default(label, shape-[...], name='label')
def nn_model(features, labels):
logits = ...
loss = tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=labels, logits=logits))
optimizer = tf.train.AdamOptimizer(learning_rate=0.01).minimize(loss)
return optimizer, loss
# Create Model
train_op, loss_op = nn_model(x, y)
# Training
sess.run(train_op)
# Inference
sess.run(logits, feed_dict={x:..., y:...})
Related
I'm trying to train a pre-trained model in Python 3.8 with Keras 2.3.1 and Tensorflow 2.2.3. Since my dataset is very large, I have to use a data generator. I want to assign sample weights to each sample, to make certain samples more important than others. I've already defined the weights that I want to assign to each sample, but I'm looking for a way to implement them in my training. Here's my custom data generator with the code that I've tried so far:
def __iter__(self):
batch_token_ids, batch_segment_ids = [], []
sample_weights = np.ones(batch_size)
for file in dataset:
for sample in file:
# Read the input, output, and sample weight from the dataset.
inputs, outputs, weight = get_data(sample)
# Encode the input and output using a tokenizer.
token_ids, segment_ids = tokenizer.encode(inputs, outputs, maxlen=maxlen)
batch_token_ids.append(token_ids)
batch_segment_ids.append(segment_ids)
# Add the weight to a np array (sample_weights)
sample_weights[len(batch_token_ids)-1] = weight
# Yield after getting batch_size samples.
if len(batch_token_ids) == self.batch_size:
"""
The input format that my model supports is the token and segment ids in x,
with None as the value for y. Changing it raises an error.
"""
yield [batch_token_ids, batch_segment_ids], None, sample_weights
batch_token_ids, batch_segment_ids = [], []
sample_weights = np.ones(batch_size)
According to the Keras documentation, I'm supposed to input the sample weights as the third value that the generator returns. However, I noticed that nothing changed in the training after implementing this. After doing some debugging into what happens to the sample weights after yielding, I noticed that the weights never get used when I try this, because of this code here in training.py of the Keras engine: (line 655)
if y is not None:
# Long code to process the inputs and sample weights.
# Since I don't have a y input, this doesn't get run.
else:
y = []
sample_weights = []
Is there a way to implement sample weights for my code without changing the input format? Also, if I'm using a custom loss function, would I need to change that as well for the sample weights to take effect?
I have trained a deep learning model by Tensorflow. The model is saved as a saved_model.pb and also there are variables and checkpoints saved during training. The model's task is regression. The output is a single scalar.
I use the saved model on some new data to test the model accuracy. Now, I want to use the saved model to save the vector just before the last layer which do regression. I'm going to use that vector by different regressors to take and improve the best result. I have also visualized the graph by Tensorboard, and I realized the vector I'm looking for is the output of an operation node in Tensorflow. I attach the screenshot of the desired part of graph.
How it is possible to use a saved model to do that? I mean to save the output vector just before the regression layer. https://drive.google.com/file/d/1tn8kGO6KBYSaD-Yz5Xp5OYQmdAniY84d/view?usp=sharing
Furthermore, I have some codes to get the predicted single scalar.
my_predictor = predictor.from_saved_model(export_dir)
mae = []
for output in read_fn(file_references=file_names,
mode=tf.estimator.ModeKeys.EVAL,
params=READER_PARAMS):
img = output['features']['x']
lbl = output['labels']['y']
test_id = output['img_id']
num_crop_predictions = 4
crop_batch = extract_random_example_array(
image_list=img,
example_size=[32, 32, 32],
n_examples=num_crop_predictions)
y_ = my_predictor.session.run(
fetches=my_predictor._fetch_tensors['logits'],
feed_dict={my_predictor.feed_tensors['x']: crop_batch})
y_ = np.mean(y_)
mae.append(np.abs(y_ - lbl))
I have changed the part
y_ = my_predictor.session.run(
fetches=my_predictor._fetch_tensors['logits'],
feed_dict={my_predictor.feed_tensors['x']: crop_batch})
to
y_ = my_predictor.session.run(
fetches=my_predictor.graph,
feed_dict={my_predictor.feed_tensors['x']: crop_batch})
due to feed the 3D inputs, and fetch the desired node of graph. But, the output of an operation node (which is what I'm looking for (pool/global_avg_pool)) is not in the graph.
This is how I did it on a DNN learning transfer.
Save Trained Model
self._saver.save(self._session, self._get_save_path(name))
Restore Model
## restore the graph
imported_meta = tf.train.import_meta_graph("%s.meta"%self._get_save_path(name))
graph = tf.get_default_graph()
## get reference of output of operation DNN/hidden5_out and set it to variable
## here ":0" indicates the output
self._frozen_out = self._graph.get_tensor_by_name("DNN/hidden5_out:0)
## restore the variables in session
self._session = tf.Session(graph=graph)
imported_meta.restore(self._session, self._get_save_path(name))
Calculate Frozen Output Once
I need to do this because it needs for placeholders to calculate frozen output. I think it is not necessary if you don't need to recalculate tensor value by feeding feed_dict
tensor = sess.eval(self._frozen_out, feed_dict=feed_dict)
Reuse the Frozen Output
## here dnn is an operation which uses output of operation "DNN/hidden5_out"
## I am passing the value of it directly as feed_dict so it will not go down the
## graph to calculate it
sess.eval(dnn, feed_dict={self._frozen_out: tensor})
Hope this helps, comment if it is not clear.
I'm trying to train a TF estimator using the RNNEstimator() class, but I'm having trouble with defining the estimator. My goal is the following:
Create a tf.data.Dataset.
Feed it into the RNN estimator.
The first part seems to be working correctly. I define the
def _parse_func(record):
# takes tf record as input and returns the following tensors
# numeric_tensor.shape = (5,170) and y.shape=()
return {'numerical': numeric_tensor,}, y
def input_fn(filenames=['data.tfrecord']):
# Returns parsed tf record i.e. the tf.data.Dataset
dataset = tf.data.TFRecordDataset(filenames=filenames)
dataset = dataset.map(map_func=_parse_func)
dataset = dataset.repeat()
dataset = dataset.batch(batch_size=BATCH_SIZE)
return dataset
Now let's move onto the meaty part.
Estimators take care of creating the session and graph. So I simply create the estimator in the following format:
# create the column
column = tf.contrib.feature_column.sequence_numeric_column('numerical')
# create the estimator
estimator = RNNEstimator(
head=tf.contrib.estimator.regression_head(),
sequence_feature_columns=[column],
num_units=[32, 16], cell_type='lstm')
# train the estimator
estimator.train(input_fn=input_fn, steps=100)
However, this doesn't work. It gives me a variety of errors! In particularly, at the moment I get:
TypeError: Input must be a SparseTensor.
Additionally, I seem to be unable to change the loss to log-loss. I tried setting it by passing it to the head parameter using:
head = tf.contrib.estimator.regression_head(loss_fn=tf.losses.log_loss)
I'm working with the new tf.data.Dataset API and I can't seem to figure out how to perform inference. Ultimately, I want to convert my model to a TensorRT graph and run it on the TX2, and all of the examples I have found assume you have a tf.placeholder for the input. Here is pseudocode for how I am training. The [...] is just meant to be a placeholder since I didn't actually run the code. Let's not debate the model, as it is just suppose to give an example:
import tensorflow as tf
# Setup iterator
datain = tf.data.FixedLengthRecordDataset(datafiles, record_bytes1)
labels = tf.data.FixedLengthRecordDataset(labelfiles, record_bytes2)
dataset = tf.data.Dataset.zip((datain, labels))
dataset = dataset.prefetch(batch_size)
dataset = dataset.repeat(n_epoch)
iterator = dataset.make_initializable_iterator()
sess = tf.Session()
sess.run(iterator.initializer)
[batch_x, batch_y] = iterator.get_next()
# Define model function (let's not debate model except as relevant to question)
def model_fn(xin):
x0 = tf.transpose(tf.reshape(xin, [...], name='input'))
w = tf.Variable(tf.truncated_normal([...], stddev=0.1))
x1 = tf.nn.conv2d(x0, w, strides=[...], padding='VALID')
b = tf.Variable(tf.constant(0.0, shape=[...]))
x2 = tf.nn.bias_add(x1, b)
x3 = tf.nn.relu(x2, name='output')
return x3
# Setup training environment
model = model_fn(batch_x)
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=model, labels=batch_y))
optimizer = tf.train.AdamOptimizer(learning_rate=1e-3).minimize(loss)
# Train Model
while True:
try:
sess.run(optimizer)
except tf.errors.OutOfRangeError:
break
# Save model
saver = tf.train.Saver(name='saver')
saver.save(sess, 'temp/path')
My question is how do I get this into TensorRT without having the input be a tf.placeholder? All of the example I can find use a tf.placeholder as the input. This example suggests that I can replace the iterator with a placeholder using the SavedModel class, but I cannot seem to find any documentation on how to accomplish that.
Thanks!
EDIT: Here is my solution thanks to the help below
from tensorflow.python.tools import optimize_for_inference_lib
import uff
# You can feed data to the IteratorGetNext node using feed_dict
input_node_name = 'iterator_scope_name/IteratorGetNext'
output_node_name = 'model_scope_name/output'
# Run inference on the trained model:
graph = tf.get_default_graph()
batch_x = graph.get_tensor_by_name(input_node_name + ':0')
networkout = graph.get_tensor_by_name(output_node_name + ':0')
testdata, testlabel = custom_data_reader_fn(data_folder)
# This will evaluate the model
label = sess.run(networkout, feed_dict={batch_x: testdata})
# Freeze model and create a UFF file:
graph_def = graph.as_graph_def() # Convert the graph to a serialized pb
frozen_graph_def = tf.graph_util.convert_variables_to_constants(sess,
graph_def, [output_node_name])
opt_graph_def = optimize_for_inference_lib.optimize_for_inference(
frozen_graph_def, [input_node_name], [output_node_name],
tf.float32.as_datatype_enum)
uff.from_tensorflow(opt_graph_def, [output_node_name], quiet=False,
output_filename='opt_model.uff')
that will write out a UFF file that TensorRT can utilize. The biggest issues that I encountered was:
I didn't realize that the optimize_for_inference_lib.optimize_for_inference operation replaced the iterator with a tf.placeholder
I did not know what node to feed data to for evaluation: you can feed data to the IteratorGetNext node
Since you already have a trained graph saved in a checkpoint, in theory the simplest solution for you is to export the inference graph via optimize_for_inference.
This tool works both for already-frozen graphs and, as is your case, for graphs with variables still defined.
Assuming you go for the frozen graph way, the first step is to transform your graph's variables in constants via:
python freeze_graph.py \
--input_graph=temp/path/graph.pbtxt \
--input_checkpoint=temp/path/your_model_name.ckpt \
--output_graph=frozen_model.pb \
--output_node_names=name_of_the_output_tensor_you_want_to_use
This will generate a new binary file called frozen_model.pb that has the Variable operations replaced with Const ops with the values loaded from the checkpoint file.
Then, you need to generate the inference graph with:
python optimize_for_inference.py \
--input=frozen_model.pb \
--output=inference.pb \
--frozen_graph=True \
--input_names=IteratorGetNext
--output_names=name_of_the_output_tensor_you_want_to_use
This will replace the IteratorGetNext node with a float placeholder. You might want to choose another node, in which case just change the name. You can also change the type of the generated placeholder via the --placeholder_type_enum option. In that case, you need to provide an integer value matching the datatype you want from the DataType enum.
NOTE: I said "in theory" because actually inspecting the generated inception graph from a test I made it seems there are still some weird ops in there that are not really necessary for inference. You might have to further process your graph via nvidia's Graph Surgeon or TF's graph transform tool
I'm attempting to make predictions using a trained convolutional neural network, slightly modified from the example in the example expert tensorflow tutorial. I have followed the instructions at https://www.tensorflow.org/versions/master/how_tos/reading_data/index.html to read data from a CSV file.
I have trained the model and evaluated its accuracy. I then saved the model and loaded it into a new python script for making predictions. Can I still use the batching method detailed in the link above or should I use feed_dict instead? Most tutorials I've seen online use the latter.
My code is shown below, I have essentially duplicated the code for reading from my training data, which was stored as lines within a single .csv file. Conv_nn is simply a class that contains the convolutional neural network detailed in the expert MNIST tutorial. Most of the content is probably not very useful except for the part where I run the graph.
I suspect I have badly mixed up training and prediction - I'm not sure if the test images are being fed to the prediction operation correctly or if it is valid to use the same batch operations for both datasets.
filename_queue = tf.train.string_input_producer(["data/test.csv"],num_epochs=None)
reader = tf.TextLineReader()
key, value = reader.read(filename_queue)
# Defaults force key value and label to int, all others to float.
record_defaults = [[1]]+[[46]]+[[1.0] for i in range(436)]
# Reads in a single row from the CSV and outputs a list of scalars.
csv_list = tf.decode_csv(value, record_defaults=record_defaults)
# Packs the different columns into separate feature tensors.
location = tf.pack(csv_list[2:4])
bbox = tf.pack(csv_list[5:8])
pix_feats = tf.pack(csv_list[9:])
onehot = tf.one_hot(csv_list[1], depth=98)
keep_prob = 0.5
# Creates batches of images and labels.
image_batch, label_batch = tf.train.shuffle_batch(
[pix_feats, onehot],
batch_size=50,num_threads=4,capacity=50000,min_after_dequeue=10000)
# Creates a graph of variables and operation nodes.
nn = Conv_nn(x=image_batch,keep_prob=keep_prob,pixels=33*13,outputs=98)
# Launch the default graph.
with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
saver.restore(sess, 'model1.ckpt')
print("Model restored.")
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(sess=sess,coord=coord)
prediction=tf.argmax(nn.y_conv,1)
pred = sess.run([prediction])
coord.request_stop()
coord.join(threads)
This question is old, but I am going to answer anyway, as it has been viewed nearly 1000 times.
So if your model had Y labels and X inputs then
prediction=tf.argmax(Y,1)
result = prediction.eval(feed_dict={X: [data]}, session=sess)
This evaluates a single input, for example a single mnist image, but it can be a batch.