Could you provide an example of using the high-level API Estimators with placeholders and feeding batches like for a basic use:
for step in xrange(max_steps):
batch_of_inputs,batch_of_targets= get_batch_from_disk(step)# e.g.batches are stored as list where step is and index of the list
feed_dict = {x:batch_of_inputs,y:batch_of_targets}
_, loss_value = sess.run([train_op, loss],
feed_dict=feed_dict)
How to do the same with Estimator API?
Estimator takes batch_size, steps, input_fuc or feed_fun as an argument of the fit function (see doc https://www.tensorflow.org/versions/master/api_docs/python/contrib.learn/estimators) but it is not clear for me how to implement a function which will load data as a batch from e.g. disk?
I don't think whether estimators are really meant to be used with placeholders. They use the concept of input_fn which is properly described here.
If you realy need to use a placeholder you might use a FeedFnHook:
def input_fn(): # empty input_fn, returns features and labels
return {}, {}
feed_dict = {x:batch_of_inputs,y:batch_of_targets}
def feed_fn(): # feed_fn with hardcoded feed_dict
return feed_dict
hooks = [tf.train.FeedFnHook(feed_fn=feed_fn)]
estimator.train(input_fn=input_fn, hooks=hooks, steps=1)
Related
I'm trying to implement a boosting model using Tensorflow "BoostedTreesRegressor".
For that, I need to implement a custom loss function where during training, the loss will be calculated according to the logic defined in my custom function rather than using the usual mean_squared_error.
I read in articles that this can be implemented using the interface, "BoostedTreesEstimator" by specifying a head. So, I tried to implement my model as follows:
#define custom loss function to calculate smape
def custom_loss_fn(labels, logits):
return (np.abs(logits - labels) / (np.abs(logits) + np.abs(labels))) * 2
#create input functions
def make_input_fn(X, y, n_epochs=None, shuffle=True):
def input_fn():
dataset = tf.data.Dataset.from_tensor_slices((dict(X), y))
if shuffle:
dataset = dataset.shuffle(NUM_EXAMPLES)
dataset = dataset.repeat(n_epochs)
dataset = dataset.batch(NUM_EXAMPLES)
return dataset
return input_fn
train_input_fn = make_input_fn(dftrain, y_train)
eval_input_fn = make_input_fn(dfeval, y_eval, n_epochs=1, shuffle=False)
my_head = tf.estimator.RegressionHead(loss_fn=custom_loss_fn)
#Training a boosted trees model
est = tf.estimator.BoostedTreesEstimator(feature_columns,
head=my_head,
n_batches_per_layer=1,
n_trees=90,
max_depth=2)
est.train(train_input_fn, max_steps=100)
predictions = list(est.predict(eval_input_fn))
This code provided an error as follows:
'Subclasses of Head must implement create_estimator_spec() or 'NotImplementedError: Subclasses of Head must implement create_estimator_spec() or _create_tpu_estimator_spec().
As I read in articles, create_estimator_spec() is used when we define a model_fn() when creating a new Estimator. Here, I do not want to create any new models or Estimators, I only want to use a custom loss function (instead of default mean squared error) when training where the training model should be equal to BoostedTreesRegressor/BoostingTreesEstimator.
It is a great help if anybody can give me some hint to implement this model.
Make sure you aren't using numpy functions in your loss function--you cannot convert tensors to numpy arrays. Try replacing np.abs with tf.abs. You might be getting the NotImplementedError because your loss function is breaking.
I'm currently saving and restoring neural network models using Tensorflow's Saver class, as shown below:
saver.save(sess, checkpoint_prefix, global_step=step)
saver.restore(sess, checkpoint_file)
This saves .ckpt files of the model to a specified path. Because I am running multiple experiments, I have limited space to save these models.
I would like to know if there is a way to save these models without saving content in specified directories.
Ex. can I just pass some object at the last checkpoint to some evaluate() function and restore the model from that object?
So far as I see, the save_path parameter in tf.train.Saver.restore() is not optional.
Any insight would be much appreciated.
Thanks
You can use the loaded graph and weights to evaluate in the same way that you train. You just need to change the input to be from your evaluation set. Here is some pseudo code of a train loop with an evaluation loop every 1000 iterations (assumes you have created a tf.Session called sess):
x = tf.placeholder(...)
loss, train_step = model(x)
for i in range(num_step):
input_x = get_train_data(i)
sess.run(train_step, feed_dict={x: input_x})
if i % 1000 == 0 and i != 0:
eval_loss = 0
for j in range(num_eval):
input_x = get_eval_data(j)
eval_loss += sess.run(loss, feed_dict={x: input_x})
print(eval_loss/num_eval)
If you're using tf.data for your input then you can just create a tf.cond to select which input to use:
is_training = tf.placeholder(tf.bool)
next_element = tf.cond(is_training,
lambda: get_next_train(),
lambda: get_next_eval())
get_next_train and get_next_eval would have to create all ops that are used for reading the dataset, or else there will be side affects of running the above code.
This way you don't have to save anything to disc if you don't want to.
I am working on a Tensorflow estimator using RNN (GRUCell).
I use zero_state to initialize the first state, it requires a fixed size.
My problem is that I want to be able to use the estimator to predict with a single sample (batchsize=1).
When it load the serialized estimator, it complain that the size of the batch I use for prediction does not match the training batch size.
If I reconstruct the estimator with a different batch size, I cannot load what has been serialized.
Is there an elegant way to use zero_state in an estimator?
I saw some solutions using a variable to store batch size, but using feed_dict method. I don't find how to make it work in the context of an estimator.
Here is the core of my simple test RNN in the estimator:
cells = [ tf.nn.rnn_cell.GRUCell(self.getNSize()) for _ in range(self.getNLayers())]
multicell = tf.nn.rnn_cell.MultiRNNCell(cells, state_is_tuple=False)
H_init = tf.Variable( multicell.zero_state( batchsize, dtype=tf.float32 ), trainable=False)
H = tf.Variable( H_init )
Yr, state = tf.nn.dynamic_rnn(multicell, Xo, dtype=tf.float32, initial_state=H)
Would someone have a clue on that?
EDIT:
Ok, I try various things on this problem.
I now try to filter the variables I load from the checkpoint to remove 'H', which is used as internal state of the recurrent cells. For prediction, I can leave it with all 0 values.
So far, I did that:
First I define a hook:
class RestoreHook(tf.train.SessionRunHook):
def __init__(self, init_fn):
self.init_fn = init_fn
def after_create_session(self, session, coord=None):
print("--------------->After create session.")
self.init_fn(session)
Then in my model_fn:
if mode == tf.estimator.ModeKeys.PREDICT:
logits = tf.nn.softmax(logits)
# Do not restore H as it's batch size might be different.
vlist = tf.contrib.framework.get_variables_to_restore()
vlist = [ x for x in vlist if x.name.split(':')[0] != 'architecture/H']
init_fn = tf.contrib.framework.assign_from_checkpoint_fn(tf.train.latest_checkpoint(self.modelDir), vlist, ignore_missing_vars=True)
spec = tf.estimator.EstimatorSpec(mode=mode,
predictions = {
'logits': logits,
},
export_outputs={
'prediction': tf.estimator.export.PredictOutput( logits )
},
prediction_hooks=[RestoreHook(init_fn)])
I took this piece of code from https://github.com/tensorflow/tensorflow/issues/14713
But it does not work yet. It seems that it still trying to load H from the file... I checked that it is not in vlist.
I am still looking for a solution.
You can get batch size form other tensor example
decoder_initial_state = cell.zero_state(array_ops.shape(attention_states)[0],
dtypes.float32).clone(cell_state=encoder_state)
I found a solution:
I create the variables for the initial state for both batchsize=64 and batchsize=1.
At training I use the first one to initialize the RNN.
At Predict time, I use the second one.
It works as both those variables will be serialized and restored by the estimator code so it will not complain.
The drawback is that the query batch size (in my case, 1) bust be known at training time (when it create both variables).
I am writing a high-level tensorflow application exactly the same way this minst estimator is build except that I am building a simple RNN that predicts sequences. I am new to tensorflow so I am trying to get head around an issue that might be actually simple for people who have worked in tensorflow high level api before.
Here is a snippet of my code to give an idea:
def main(argv=None):
"""Run the training experiment."""
....
# Setup the Estimator
model_estimator = build_estimator(config, params)
# Setup and start training and validation
train_spec = tf.estimator.TrainSpec(
input_fn=lambda: get_train_inputs(128),
max_steps=2000)
...
tf.estimator.train_and_evaluate(model_estimator, train_spec, eval_spec)
def build_estimator(config, params):
return tf.estimator.Estimator(
model_fn=model_fn,
config=config,
params=params,
)
def model_fn(features, mode, params):
#Input data
_inputs = tf.placeholder(tf.int32, shape=[batch_size, times_steps])
_labels = tf.placeholder(tf.float32, shape=[batch_size, num_classes])
# Sequence lengths for dynamic allocation
_seqlens = tf.placeholder(tf.int32, shape=[batch_size])
...
return tf.estimator.EstimatorSpec(
mode=mode,
predictions=predictions,
loss=loss,
train_op=train_op,
eval_metric_ops=eval_metric_ops
)
Here is my input pipleine function:
# Get train inputs function
def get_train_inputs(batch_size):
def train_inputs(batch_size):
# Build dataset iterator
x_batch, y_batch, seqlen_batch = sequence_generator.get_sentence_batch(
batch_size, sequence_generator.train_x, sequence_generator.train_y, sequence_generator.train_seqlens)
features={'_inputs': x_batch, '_labels': y_batch, '_seqlens': seqlen_batch}
return features
return train_inputs(batch_size)
Due to the size of my code, I have only pasted relevant pieces of code here.
The problem here is that during:
train_spec = tf.estimator.TrainSpec(
input_fn=lambda: get_train_inputs(128),
max_steps=2000)
get_train_inputs(128) feeds the features dictionary into _inputs placeholder of the model_fn so the _labels and _seqlens remain blank and throw out error during execution that no values specified for these place holders. The model_fn only accepts two feature parameters : features and labels. How do I feed all the three parameters _inputs, _labels and _seqlens into the model?
Any suggestions will be highly appreciated.
NOTE: The reason for inputing a third parameter _seqlens is because I am using tf.nn.dynamic_rnn in my model_fn which requires sequence lengths where as labels are being used in tf.nn.softmax_cross_entropy_with_logits in my softmax function.
You're not supposed to use placeholders at all with tf.Estimator. You should look into the tf.data API (see here). Your input function should return the get_next op of a one shot iterator. Apologies if you are already doing this, but it is not clear from your code what exactly your input function is returning.
Assuming you set this up to return a dict as in your example, you will then be able to simply use _inputs = features["_inputs"] etc. in your model function.
In addition to #xdurch0 answer, use FeatureColumns
tf.feature_column to describe the features of the dataset that are passed as inputs into the Estimator model_fn for training and evaluation.
Within the model_fn, use the method
tf.feature_column.input_layer()` to return a dense Tensor as an
input layer based on a specified FeatureColumn.
You can see examples of working with FeatureColumns here.
I would like to manage my training with a tf.estimator.Estimator but have some trouble to use it alongside the tf.data API.
I have something like this:
def model_fn(features, labels, params, mode):
# Defines model's ops.
# Initializes with tf.train.Scaffold.
# Returns an tf.estimator.EstimatorSpec.
def input_fn():
dataset = tf.data.TextLineDataset("test.txt")
# map, shuffle, padded_batch, etc.
iterator = dataset.make_initializable_iterator()
return iterator.get_next()
estimator = tf.estimator.Estimator(model_fn)
estimator.train(input_fn)
As I can't use a make_one_shot_iterator for my use case, my issue is that input_fn contains an iterator that should be initialized within model_fn (here, I use tf.train.Scaffold to initialize local ops).
Also, I understood that we can't only use input_fn = iterator.get_next otherwise the other ops will not be added to the same graph.
What is the recommended way to initialize the iterator?
As of TensorFlow 1.5, it is possible to make input_fn return a tf.data.Dataset, e.g.:
def input_fn():
dataset = tf.data.TextLineDataset("test.txt")
# map, shuffle, padded_batch, etc.
return dataset
See c294fcfd.
For previous versions, you can add the iterator's initializer in the tf.GraphKeys.TABLE_INITIALIZERS collections and rely on the default initializer.
tf.add_to_collection(tf.GraphKeys.TABLE_INITIALIZERS, iterator.initializer)