I am trying to understand training loop. While calculating training and test accuracy we replace x and y_ by training and test sets but while printing the result for cross entropy why we feed x and y_ by batch_xs and batch_ys respectively? I know that we have to assign a value to the placeholders but while calculating test accuracy batch_xs and batch_ys remain same and both are using train set values.
x = tf.placeholder(tf.float32, [None, nPixels])
y_ = tf.placeholder(tf.float32, [None, nLabels])
cross_entropy = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(y, y_))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
with tf.Session() as sess:
tf.initialize_all_variables().run()
for j in range(nSteps):
i = nstep % nSteps
batch_xs = np.reshape(x_train,(nSteps,bSize,nPixels))
batch_ys = np.reshape(y_train,(nSteps,bSize,nLabels))
sess.run(train_step, feed_dict={x: batch_xs[i], y_: batch_ys[i]})
if j % 100 ==0:
train_acc = sess.run(accuracy, feed_dict={x: x_train,y_: y_train})
test_acc = sess.run(accuracy, feed_dict={x: x_test, y_: y_test}))
loss = sess.run(cross_entropy, feed_dict={x: batch_xs[i], y_: batch_ys[i]})
You can calculate your training loss at the same time you do a training step (so that you do not have to make a separate call, passing your batch again later) with something like:
for j in range(nSteps):
# ...
_, train_loss = sess.run([train_step, cross_entropy],
feed_dict={x: batch_xs[i], y_: batch_ys[i]})
As for why you do not need to resize your training/testing sets when calculating your accuracy, let's look at the dimensions of your data throughout your code. You start out with:
x = tf.placeholder(tf.float32, [None, nPixels])
y_ = tf.placeholder(tf.float32, [None, nLabels])
That means that the dimensions of x are [num_samples, nPixels] and the dimensions of y_ are [num_samples, nLabels]. I assume x_train and y_train are just some subsample of this, so [num_train, nPixels] and [num_train, nLabels] respectively. Next you reshape these for your batches:
batch_xs = np.reshape(x_train,(nSteps,bSize,nPixels))
batch_ys = np.reshape(y_train,(nSteps,bSize,nLabels))
Now batch_xs has dimensions [num_steps, batch_size, nPixels] and batch_ys has dimensions [num_steps, batch_size, nLabels]. Note that the last dimensions, the number of features or output dimensions, hasn't changed.
Now for each iteration of your training loop, you take one element from the first dimension of these lists with batch_xs[i] and batch_ys[i]. The dimensions of these guys are now [batch_size, nPixels], and [batch_size, nLables], respectively. Again, the last dimension hasn't changed.
Finally you call your accuracy or cross_entropy ops on a batch or on the whole training set and TF doesn't care which it is! This is because the number of features/labels you are giving it is the same either way, and all that's changing is the number of data elements you are passing. In one case you are passing batch_size elements, and in the other you are passing num_train or num_test elements, but the last dimensions that specify what each sample looks like is the same! TF is magic and figures out how to handle the difference.
Old answer
It looks like you are just trying to get training loss out of your loop, thus you are passing your training batch to your cross_entropy function. A lot of times you might evaluate this at every batch and store the results so that you can plot your training loss over time later, but sometimes this calculation takes long enough that you want to only calculate it every so often. This seems to be the case for your code.
In general, it is a good idea to look at both training and test (or validation) loss while your training loop runs. This is because you are better able to see the effects of overfitting, where training loss continues to decrease and testing loss levels off or begins to increase. Something like:
if j % 100 == 0:
train_loss = sess.run(cross_entropy, feed_dict={x: batch_xs[i], y_: batch_ys[i]})
test_loss = sess.run(cross_entropy, feed_dict={x: x_test, y_: y_test})
# print or store your training and test loss
One of the major reasons for monitoring your testing loss is to stop your training early in the event of overfitting. You can do this by storing your best testing loss, and then comparing each new testing loss to your stored best. Each time you get a new testing loss that does not beat your best, you increment a counter, and when the counter hits a certain number (say 10 or 20), you kill the training loop and call it good. This way you catch when your training has stalled out on the validation/testing set, which is a better indicator of when it will no longer get better at generalizing to new data.
If you encounter a new validation loss that is better than your best, you store it and reset your counter.
Finally, it is often best to hold out two sets of data, one called the validation set, and one called the testing set. When you split it up like this, you typically use the validation set while training to look for overfitting and perform early stopping, and then you run one final test for accuracy/loss against the testing set that the network has never seen before to give you an idea of its overall generalization performance. That looks something like:
for epoch:
for batch in batches:
x = batch.x
y = batch.y
sess.run(train_step, feed_dict={x: x, y_: y})
if j % 100 == 0:
train_loss = sess.run(cross_entropy, feed_dict={x: x, y_: y})
test_loss = sess.run(cross_entropy, feed_dict={x: x_val, y_: y_val}) # note validation data
# print or store your training and test loss
# check for early stopping
test_loss = sess.run(cross_entropy, feed_dict={x: x_test, y_: y_test})
# Or test accuracy or whatever you want to evaluate.
# Point is the network never saw this data until now.
Related
I want to train Googles VGGish network (Hershey et al 2017) from scratch to predict classes specific to my own audio files.
For this I am using the vggish_train_demo.py script available on their github repo which uses tensorflow. I've been able to modify the script to extract melspec features from my own audio by changing the _get_examples_batch() function, and, then train the model on the output of this function. This runs to completetion and prints the loss at each epoch.
However, I've been unable to figure out how to get this trained model to generate predictions from new data. Can this be done with changes to the vggish_train_demo.py script?
For anyone who stumbles across this in the future, I wrote this script which does the job. You must save logmel specs for train and test data in the arrays: X_train, y_train, X_test, y_test. The X_train/test are arrays of the (n, 96,64) features and the y_train/test are arrays of shape (n, _NUM_CLASSES) for two classes, where n = the number of 0.96s audio segments and _NUM_CLASSES = the number of classes used.
See the function definition statement for more info and the vggish github in my original post:
### Run the network and save the predictions and accuracy at each epoch
### Train NN, output results
r"""This uses the VGGish model definition within a larger model which adds two
layers on top, and then trains this larger model.
We input log-mel spectrograms (X_train) calculated above with associated labels
(y_train), and feed the batches into the model. Once the model is trained, it
is then executed on the test log-mel spectrograms (X_test), and the accuracy is
ouput, alongside a .csv file with the predictions for each 0.96s chunk and their
true class."""
def main(X):
with tf.Graph().as_default(), tf.Session() as sess:
# Define VGGish.
embeddings = vggish_slim.define_vggish_slim(training=FLAGS.train_vggish)
# Define a shallow classification model and associated training ops on top
# of VGGish.
with tf.variable_scope('mymodel'):
# Add a fully connected layer with 100 units. Add an activation function
# to the embeddings since they are pre-activation.
num_units = 100
fc = slim.fully_connected(tf.nn.relu(embeddings), num_units)
# Add a classifier layer at the end, consisting of parallel logistic
# classifiers, one per class. This allows for multi-class tasks.
logits = slim.fully_connected(
fc, _NUM_CLASSES, activation_fn=None, scope='logits')
tf.sigmoid(logits, name='prediction')
linear_out= slim.fully_connected(
fc, _NUM_CLASSES, activation_fn=None, scope='linear_out')
logits = tf.sigmoid(linear_out, name='logits')
# Add training ops.
with tf.variable_scope('train'):
global_step = tf.train.create_global_step()
# Labels are assumed to be fed as a batch multi-hot vectors, with
# a 1 in the position of each positive class label, and 0 elsewhere.
labels_input = tf.placeholder(
tf.float32, shape=(None, _NUM_CLASSES), name='labels')
# Cross-entropy label loss.
xent = tf.nn.sigmoid_cross_entropy_with_logits(
logits=logits, labels=labels_input, name='xent')
loss = tf.reduce_mean(xent, name='loss_op')
tf.summary.scalar('loss', loss)
# We use the same optimizer and hyperparameters as used to train VGGish.
optimizer = tf.train.AdamOptimizer(
learning_rate=vggish_params.LEARNING_RATE,
epsilon=vggish_params.ADAM_EPSILON)
train_op = optimizer.minimize(loss, global_step=global_step)
# Initialize all variables in the model, and then load the pre-trained
# VGGish checkpoint.
sess.run(tf.global_variables_initializer())
vggish_slim.load_vggish_slim_checkpoint(sess, FLAGS.checkpoint)
# The training loop.
features_input = sess.graph.get_tensor_by_name(
vggish_params.INPUT_TENSOR_NAME)
accuracy_scores = []
for epoch in range(num_epochs):#FLAGS.num_batches):
epoch_loss = 0
i=0
while i < len(X_train):
start = i
end = i+batch_size
batch_x = np.array(X_train[start:end])
batch_y = np.array(y_train[start:end])
_, c = sess.run([train_op, loss], feed_dict={features_input: batch_x, labels_input: batch_y})
epoch_loss += c
i+=batch_size
#print no. of epochs and loss
print('Epoch', epoch+1, 'completed out of', num_epochs,', loss:',epoch_loss) #FLAGS.num_batches,', loss:',epoch_loss)
#If these lines are left here, it will evaluate on the test data every iteration and print accuracy
#note this adds a small computational cost
correct = tf.equal(tf.argmax(logits, 1), tf.argmax(labels_input, 1)) #This line returns the max value of each array, which we want to be the same (think the prediction/logits is value given to each class with the highest value being the best match)
accuracy = tf.reduce_mean(tf.cast(correct, 'float')) #changes correct to type: float
accuracy1 = accuracy.eval({features_input:X_test, labels_input:y_test})
accuracy_scores.append(accuracy1)
print('Accuracy:', accuracy1)#TF is smart so just knows to feed it through the model without us seeming to tell it to.
#Save predictions for test data
predictions_sigm = logits.eval(feed_dict = {features_input:X_test}) #not really _sigm, change back later
#print(predictions_sigm) #shows table of predictions, meaningless if saving at each epoch
test_preds = pd.DataFrame(predictions_sigm, columns = col_names) #converts predictions to df
true_class = np.argmax(y_test, axis = 1) #This saves the true class
test_preds['True class'] = true_class #This adds true class to the df
#Saves csv file of table of predictions for test data. NB. header will not save when using np.text for some reason
np.savetxt("/content/drive/MyDrive/..."+"Epoch_"+str(epoch+1)+"_Accuracy_"+str(accuracy1), test_preds.values, delimiter=",")
if __name__ == '__main__':
tf.app.run()
#'An exception has occurred, use %tb to see the full traceback.' error will occur, fear not, this just means its finished (perhaps as its exited the tensorflow session?)
I am using an initializable iterator in my code. The iterator returns batches of size 100 from a csv dataset that has 20.000 entries. During training, however, I came across a problem. Consider this piece of code:
def get_dataset_iterator(batch_size):
# parametrized with batch_size
dataset = ...
return dataset.make_initializable_iterator()
## build a model and train it (x is the input of my model)
iterator = get_dataset_iterator(100)
x = iterator.get_next()
y = model(x)
## L1 norm as loss, this works because the model is an autoencoder
loss = tf.abs(x - y)
## training operator
train_op = tf.train.AdamOptimizer(0.01).minimize(loss)
with tf.Session() as sess:
for epoch in range(100):
sess.run(iterator.initializer)
# iterate through the whole dataset once during the epoch and
# do 200 mini batch updates
for _ in range(number_of_samples // batch_size):
sess.run(train_op)
print(f'Epoch {epoch} training done!')
# TODO: print loss after epoch here
I am interested in the training loss AFTER finishing the epoch. It makes most sense to me that I calculate the average loss over the whole training set (e.g. feeding all 20.000 samples through the network and averaging their loss). I could reuse the dataset iterator here with a batch size of 20.000, but I have declared x as the input.
So the questions are:
1.) Does the loss calculation over all 20.000 examples make sense? I have seen some people do the calculation with just a mini-batch (the last batch of the epoch).
2.) How can I calculate the loss over the whole training set with an input pipeline? I have to inject all of training data somehow, so that I can run sess.run(loss) without calculating it over only 100 samples (because x is declared as input).
EDIT FOR CLARIFICATION:
If I wrote my training loop the following way, there would be some things that bother me:
with tf.Session() as sess:
for epoch in range(100):
sess.run(iterator.initializer)
# iterate through the whole dataset once during the epoch and
# do 200 mini batch updates
for _ in range(number_of_samples // batch_size):
_, current_loss = sess.run([train_op, loss])
print(f'Epoch {epoch} training done!')
print(current_loss)
Firstly, loss would still be evaluated before doing the last weight update. That means whatever comes out is not the latest value. Secondly, I would not be able to access current_loss after exiting the for loop so I would not be able to print it.
1) Loss calculation over the whole training set (before updating weights) does make sense and is called batch gradient descent (despite using the whole training set and not a mini batch).
However, calculating a loss for your whole dataset before updating weights is slow (especially with large datasets) and training will take a long time to converge. As a result, using a mini batch of data to calculate loss and update weights is what is normally done instead. Although using a mini batch will produce a noisy estimate of the loss it is actually good enough estimate to train networks with enough training iterations.
EDIT:
I agree that the loss value you print will not be the latest loss with the latest updated weights. Probably for most cases it really doesn't make much different or change results so people just go with how you have wrote the code above. However, if you really want to obtain the true latest loss value after you have done training (to print out) then you will just have to run the loss op again after you have done a train op e.g.:
for _ in range(number_of_samples // batch_size):
sess.run([train_op])
current_loss = sess.run([loss])
This will get your true latest value. Of course this wont be on the whole dataset and will be just for a minibatch of 100. Again the value is likely a good enough estimate but if you wish to calculate exact loss for whole dataset you will have to run through your entire set e.g. another loop and then average the loss:
...
# Train loop
for _ in range(number_of_samples // batch_size):
_, current_loss = sess.run([train_op, loss])
print(f'Epoch {epoch} training done!')
# Calculate loss of whole train set after training an epoch.
sess.run(iterator.initializer)
current_loss_list = []
for _ in range(number_of_samples // batch_size):
_, current_loss = sess.run([loss])
current_loss_list.append(current_loss)
train_loss_whole_dataset = np.mean(current_loss_list)
print(train_loss_whole_dataset)
EDIT 2:
As pointed out doing the serial calls to train_op then loss will call the iterator twice and so things might not work out nicely (e.g. run out of data). Therefore my 2nd bit of code will be better to use.
I think the following code will answer your questions:
(A) how can you print the batch loss AFTER performing the train step? (B) how can you calculate the loss over the entire training set, even though the dataset iterator gives only a batch each time?
import tensorflow as tf
import numpy as np
dataset_size = 200
batch_size= 5
dimension = 4
# create some training dataset
dataset = tf.data.Dataset.\
from_tensor_slices(np.random.normal(2.0,size=(dataset_size,dimension)).
astype(np.float32))
dataset = dataset.batch(batch_size) # take batches
iterator = dataset.make_initializable_iterator()
x = tf.cast(iterator.get_next(),tf.float32)
w = tf.Variable(np.random.normal(size=(1,dimension)).astype(np.float32))
loss_func = lambda x,w: tf.reduce_mean(tf.square(x-w)) # notice that the loss function is a mean!
loss = loss_func(x,w) # this is the loss that will be minimized
train_op = tf.train.GradientDescentOptimizer(0.1).minimize(loss)
# we are going to use control_dependencies so that we know that we have a loss calculation AFTER the train step
with tf.control_dependencies([train_op]):
loss_after_train_op = loss_func(x,w) # this is an identical loss, but will only be calculated AFTER train_op has
# been performed
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
# train one epoch
sess.run(iterator.initializer)
for i in range(dataset_size//batch_size):
# the training step will update the weights based on ONE batch of examples each step
loss1,_,loss2 = sess.run([loss,train_op,loss_after_train_op])
print('train step {:d}. batch loss before step: {:f}. batch loss after step: {:f}'.format(i,loss1,loss2))
# evaluate loss on entire training set. Notice that this calculation assumes the the loss is of the form
# tf.reduce_mean(...)
sess.run(iterator.initializer)
epoch_loss = 0
for i in range(dataset_size // batch_size):
batch_loss = sess.run(loss)
epoch_loss += batch_loss*batch_size
epoch_loss = epoch_loss/dataset_size
print('loss over entire training dataset: {:f}'.format(epoch_loss))
As for your question whether it makes sense to calculate loss over the entire training set - yes, it makes sense, for evaluation purposes. It usually does not make sense to perform training steps which are based on all of the training set since this set is usually very large and you want to update your weights more often, without needing to go over the entire training set each time.
I started to learn TensorFlow by reading a book which started by classifying MNIST digits.
Link to the code
MINIBATCH_SIZE = 50
STEPS = 5000
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for i in range(STEPS):
batch = mnist.train.next_batch(MINIBATCH_SIZE)
if i % 100 == 0:
train_accuracy = sess.run(accuracy, feed_dict={x: batch[0], y_: batch[1],
keep_prob: 1.0})
print("step {}, training accuracy {}".format(i, train_accuracy))
sess.run(train_step, feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})
X = mnist.test.images.reshape(10, 1000, 784)
Y = mnist.test.labels.reshape(10, 1000, 10)
test_accuracy = np.mean(
[sess.run(accuracy, feed_dict={x: X[i], y_: Y[i], keep_prob: 1.0}) for i in range(10)])
print("test accuracy: {}".format(test_accuracy))
This is the block of code which executes the session. My question is - the for loop iterates STEPS times and batch is the mini batch of size 50.
Shouldn't we iterate STEPS times over the whole training set? This code only trains 50 images in an epoch.
What am I missing here? How does the next_batch() method work
Question
How many iterations should be do over a training set?
Answer
The answer usually is "as many as it takes". Okay, I admit that doesn't really help at first so let's get some jargon out of the way. There is a term Epoch that means one whole pass over the data. That's kinda that minimum IMHO. If you don't go over the whole data set at least once then what's the point? The MNIST data set has about 50,000 training images (60,000 if you don't split our validation). So your tensorflow graph in order to accomplish 1 Epoch would have to process 50,000 images. If your batch size is 50, then that's 1,000 batches. In your above code your batch size is 50 and you do 5,000 batches, in effect you are doing 5 Epochs worth of processing, or 5 passes over the whole data set.
Question
How does the next_batch() method work?
Answer
next_batch returns the specified number of images and labels from the training set. It wraps around so that when you are at the end of your data set it starts back at the beginning. It makes it easy to just get more data instead of having to code the looping and wrapping and slicing of the data yourself.
Batch size
Tensorflow is using gradient descent: at each step in your for loop, you evaluate the error between your predictions and the actual digits to find a gradient for adjusting the weights in your neural net.
You could indeed go through your entire test set each time, but you'd be processing the whole set just to nudge the weights, then going through it all again just to nudge the weights again, and so on. This would work, but be slow for large data sets.
At the other extreme, you could just pick one example in the loop. This is called stochastic gradient descent. Because you only process one example at each step it's very fast, but it's not guaranteed to converge and progress will be rather "jerky".
The code here is doing batch gradient descent which is a halfway house between these two approaches. By processing 50 examples each time the weights are adjusted, you get faster training than full gradient descent and more stability that stochastic gradient descent.
next_batch
The next_batch method just gets the next N records from the test set. By default, as here, the records are shuffled. You can call it as many times as you like; once the records are exhausted it will start again from another shuffled set. You can see the code here if you're interested.
Experimentation
There are 60,000 training images in the MNIST dataset. You could run this code three times, setting MINIBATCH_SIZE to 1, 50 and 60000 respectively to see for yourself how it performs in each case.
At the moment, my neural network only uses the training dataset to train but i want to validate as well but i cant figure out how to do that exactly. Should i run test on entire validation set after every epoch? if yes, then from the testing, i will get an accuracy but i cant figure out what to do with that either? how do i adjust the parameters based on it? do i just run the optimiser again? in this case, how is it affected by the results from the validation set?
optimiser = tf.train.RMSPropOptimizer(learning_rate=learning_rate).minimize(cost)
# finally setup the initialisation operator
init_op = tf.global_variables_initializer()
train = ["/Users/User/project/data/train.tfrecords"]
dataset = d.getTrainData(train, trainSize, batch_size)
iterator = dataset.make_initializable_iterator()
x_batch, y_batch = iterator.get_next()
validate = ["/Users/User/project/data/validate.tfrecords"]
datasetV = d.getValData(validate, valSize, batch_size)
iteratorV = datasetV.make_initializable_iterator()
x_batch_V, y_batch_V = iteratorV.get_next()
with tf.Session() as sess:
saver = tf.train.Saver()
sess.run(init_op)
e = 0
for _ in range(epochs):
dataset.shuffle(dataSize)
e = e + 1
sess.run(iterator.initializer)
sess.run(iteratorV.initializer)
i = 1
try:
while True:
xy, z = sess.run((x_batch, y_batch))
summary, _, c = sess.run([merged, optimiser, cost],
feed_dict={x: xy, y: z})
print ("cost of batch ", i, ": ", c)
train_writer.add_summary(summary, e)
i = i + 1
except tf.errors.OutOfRangeError:
# Raised when we reach the end of the file.
pass
print ("end of epoch ", e)
save_path = saver.save(sess, "/Users/User/project/model/model.ckpt")
print("Model saved in file: %s" % save_path)
return
(btw i am using tensorflow version 1.2)
thanks for the help!
You need a validation set, that's different from the training set. If you don't have one consider splitting the training set using some random process to pick random examples to pick for the validation set. The only difference in your code should be that you do not run the training operation (remove optimiser from sess.run).
At the bare minimum is to run it once at the end of training and checking that the loss (and other metrics that might be important to you) are sufficiently close to the ones you get from training. The point is to make sure that you are not over-fitting on the training dataset. If you do plots you might want to want to run it from time to time to plot it with the training data. Looking at the plot is useful sometimes if you want to stop early when you see the model starts overfitting.
If you want to do hyper-parameter tuning, you would run the training with various changes to the hyper-parameters and pick the combination that gives you the best results over the validation data set. You should have a third data-set (test data set) that you can use at the very end to make sure that you don't accidentally overfit during hyper-parameter tuning.
I have a TensorFlow model, and one part of this model evaluates the accuracy. The accuracy is just another node in the tensorflow graph, that takes in logits and labels.
When I want to plot the training accuracy, this is simple: I have something like:
tf.scalar_summary("Training Accuracy", accuracy)
tf.scalar_summary("SomethingElse", foo)
summary_op = tf.merge_all_summaries()
writer = tf.train.SummaryWriter('/me/mydir/', graph=sess.graph)
Then, during my training loop, I have something like:
for n in xrange(1000):
...
summary, ..., ... = sess.run([summary_op, ..., ...], feed_dict)
writer.add_summary(summary, n)
...
Also inside that for loop, every say, 100 iterations, I want to evaluate the validation accuracy. I have a separate feed_dict for this, and I am able to evaluate the validation accuracy very nicely in python.
However, here is my problem: I want to make another summary for the validation accuracy, by using the accuracy node. I am not clear on how to do this though. Since I have the accuracy node it makes sense that I should be able to re-use it, but I am unsure how to do this exactly, such that I can also get the validation accuracy written out as a separate scalar_summary...
How might this be possible?
You can reuse the the accuracy node but you need to use two different SummaryWriters, one for the training runs and one for the test data. Also you have to assign the scalar summary for accuracy to a variable.
accuracy_summary = tf.scalar_summary("Training Accuracy", accuracy)
tf.scalar_summary("SomethingElse", foo)
summary_op = tf.merge_all_summaries()
summaries_dir = '/me/mydir/'
train_writer = tf.train.SummaryWriter(summaries_dir + '/train', sess.graph)
test_writer = tf.train.SummaryWriter(summaries_dir + '/test')
Then in your training loop you have the normal training and record your summaries with the train_writer. In addition you run the graph on the test set each 100th iteration and record only the accuracy summary with the test_writer.
# Record train set summaries, and train
summary, _ = sess.run([summary_op, train_step], feed_dict=...)
train_writer.add_summary(summary, n)
if n % 100 == 0: # Record summaries and test-set accuracy
summary, acc = sess.run([accuracy_summary, accuracy], feed_dict=...)
test_writer.add_summary(summary, n)
print('Accuracy at step %s: %s' % (n, acc))
You can then point TensorBoard to the parent directory (summaries_dir) and it will load both data sets.
This can be also found in the TensorFlow HowTo's https://www.tensorflow.org/versions/r0.11/how_tos/summaries_and_tensorboard/index.html
To run the same operation but get summaries with different feed_dict data, simply attach two summary ops to that op. Say you want to run accuracy op on both validation and test data and want to get summaries for both:
validation_acc_summary = tf.summary.scalar('validation_accuracy', accuracy) # intended to run on validation set
test_acc_summary = tf.summary.scalar('test_accuracy', accuracy) # intended to run on test set
with tf.Session() as sess:
# do your thing
# ...
# accuracy op just needs labels y_ and input x to compute logits
validation_summary_str = sess.run(validation_acc_summary, feed_dict=feed_dict={x: mnist.validation.images,y_: mnist.validation.labels})
test_summary_str = sess.run(test_acc_summary, feed_dict={x: mnist.test.images,y_: mnist.test.labels})
# assuming you have a tf.summary.FileWriter setup
file_writer.add_summary(validation_summary_str)
file_writer.add_summary(test_summary_str)
Also remember you can always pull raw (scalar) data out of the protobuff summary_str like this and do your own logging.