Develop Reference

is tensor2tensor/avg_checkpoints script wrong

I am training two sub-models in parallel, and want to average them after both of them are finished.
The model is implemented out of tensor2tensor (but still use tensorflow). Part of the definition is like this:
def build_model(self):
layer_sizes = [n,n,n]
kernel_sizes = [n,n,n]
tf.reset_default_graph()
self.graph = tf.Graph()
with self.graph.as_default():
self.num_classes = num_classes
# placeholder for parameter
self.learning_rate = tf.placeholder(tf.float32, name="learning_rate")
self.dropout_keep_prob = tf.placeholder(tf.float32, name="dropout_keep_prob")
self.phase = tf.placeholder(tf.bool, name="phase")
# Placeholders for regular data
self.input_x = tf.placeholder(tf.float32, [None, None, input_feature_dim], name="input_x")
self.input_y = tf.placeholder(tf.float32, [None, num_classes], name="input_y")
h = self.input_x
......[remaining codes]
I save it following:
def save_model(sess, output):
saver = tf.train.Saver()
save_path = saver.save(sess, os.path.join(output, 'model'))
When I load it, I use:
def load_model(self, sess, input_dir, logger):
if logger is not None:
logger.info("Start loading graph ...")
saver = tf.train.import_meta_graph(os.path.join(input_dir, 'model.meta'))
saver.restore(sess, os.path.join(input_dir, 'model'))
self.graph = sess.graph
self.input_x = self.graph.get_tensor_by_name("input_x:0")
self.input_y = self.graph.get_tensor_by_name("input_y:0")
self.num_classes = self.input_y.shape[1]
self.learning_rate = self.graph.get_tensor_by_name("learning_rate:0")
self.dropout_keep_prob = self.graph.get_tensor_by_name("dropout_keep_prob:0")
self.phase = self.graph.get_tensor_by_name("phase:0")
self.loss = self.graph.get_tensor_by_name("loss:0")
self.optimizer = self.graph.get_operation_by_name("optimizer")
self.accuracy = self.graph.get_tensor_by_name("accuracy/accuracy:0")
I use the avg_checkpoint to average two sub-models:
python utils/avg_checkpoints.py
--checkpoints path/to/checkpoint1,path/to/checkpoint2
--num_last_checkpoints 2
--output_path where/to/save/the/output
But I find problems when I check the avg_checkpoints code further:
for checkpoint in checkpoints:
reader = tf.train.load_checkpoint(checkpoint)
for name in var_values:
tensor = reader.get_tensor(name)
var_dtypes[name] = tensor.dtype
var_values[name] += tensor
tf.logging.info("Read from checkpoint %s", checkpoint)
for name in var_values: # Average.
var_values[name] /= len(checkpoints)
with tf.variable_scope(tf.get_variable_scope(), reuse=tf.AUTO_REUSE):
tf_vars = [
tf.get_variable(v, shape=var_values[v].shape, dtype=var_dtypes[v])
for v in var_values
]
placeholders = [tf.placeholder(v.dtype, shape=v.shape) for v in tf_vars]
assign_ops = [tf.assign(v, p) for (v, p) in zip(tf_vars, placeholders)]
global_step = tf.Variable(
0, name="global_step", trainable=False, dtype=tf.int64)
saver = tf.train.Saver(tf.all_variables())
# Build a model consisting only of variables, set them to the average values.
with tf.Session() as sess:
for p, assign_op, (name, value) in zip(placeholders, assign_ops,
six.iteritems(var_values)):
sess.run(assign_op, {p: value})
# Use the built saver to save the averaged checkpoint.
saver.save(sess, FLAGS.output_path, global_step=global_step)
It only save the variables, not all tensors. E.G. when I load it using the above load_model function, it cannot have "input_x:0" tensor. Is this script wrong, or I should modify it based on my use?
I am using TF r1.13. Thanks!

Why doesn't this test function output an image? pokeGAN Tutorial

I am new to python coding and using tensorflow, that being said this may be a dumb question. I was following the pokeGAN tutorial done by Siraj, and he doesn't really comment on the test function. I have trained the model, but when I uncomment the test function, it just exits with a code of 0, and gives me no image it may have generated. I know the exit code of 0 means there was no errors, but I am curious as to why its not generating an image. Is the function just not telling it to generate an image? Is there something else that needs to be uncommented (or commented) out to make it work correctly? Any help would be great.
Here is a link to the github for the code in its entirety: pokeGAN
Here is the actual test function:
def test():
random_dim = 100
with tf.variable_scope('input'):
real_image = tf.placeholder(tf.float32, shape=[None, HEIGHT, WIDTH, CHANNEL], name='real_image')
random_input = tf.placeholder(tf.float32, shape=[None, random_dim], name='rand_input')
is_train = tf.placeholder(tf.bool, name='is_train')
# wgan
fake_image = generator(random_input, random_dim, is_train)
real_result = discriminator(real_image, is_train)
fake_result = discriminator(fake_image, is_train, reuse=True)
sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())
variables_to_restore = slim.get_variables_to_restore(include=['gen'])
print(variables_to_restore)
saver = tf.train.Saver(variables_to_restore)
ckpt = tf.train.latest_checkpoint('./model/' + version)
saver.restore(sess, ckpt)

You're code misses some information to create an image and save it.
def test():
random_dim = 100
with tf.variable_scope('input'):
real_image = tf.placeholder(tf.float32, shape = [None, HEIGHT, WIDTH, CHANNEL], name='real_image')
random_input = tf.placeholder(tf.float32, shape=[None, random_dim], name='rand_input')
is_train = tf.placeholder(tf.bool, name='is_train')
# wgan
fake_image = generator(random_input, random_dim, is_train)
real_result = discriminator(real_image, is_train)
fake_result = discriminator(fake_image, is_train, reuse=True)
sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())
variables_to_restore = slim.get_variables_to_restore(include=['gen'])
print(variables_to_restore)
saver = tf.train.Saver(variables_to_restore)
ckpt = tf.train.latest_checkpoint('./model/' + version)
saver.restore(sess, ckpt)
#image creation
sample_noise = np.random.uniform(-1.0, 1.0, size=[64, random_dim]).astype(np.float32)
imgtest = sess.run(fake_image, feed_dict={random_input: sample_noise, is_train: False})
save_images(imgtest, [8,8] ,newPoke_path + '/epoch' + 'image.jpg')

With Tensorflow, 2 class classification using Neural Network

I'm trying 2 class classification of images by neural network using Tensorflow.
I want to extract 1000 pixels randomly.
However, I am in trouble with error:
"logits = inference(images_placeholder, keep_prob)
File "train5.py", line 83, in inference
list = random.sample(x_image(IMAGE_PIXELS),SAMPLE_PIXELS)
TypeError: 'Tensor' object is not callable"
Please tell me what should I do.
I will attach a code below.
import sys
sys.path.append('/usr/local/opt/opencv3/lib/python3.5.4/site-packages')
import cv2
import numpy as np
import tensorflow as tf
import tensorflow.python.platform
import tensorboard as tb
import os
import math
import time
import random
start_time = time.time()
# TensorBoard output information directory
log_dir = '/tmp/data1' #tensorboard --logdir=/tmp/data1
#directory delete and reconstruction
if tf.gfile.Exists(log_dir):
tf.gfile.DeleteRecursively(log_dir)
tf.gfile.MakeDirs(log_dir)
# Reserve memory
config = tf.ConfigProto(
gpu_options=tf.GPUOptions(allow_growth=True))
sess = sess = tf.Session(config=config)
NUM_CLASSES = 2
IMAGE_SIZE_x = 1024
IMAGE_SIZE_y = 768
IMAGE_CHANNELS = 1
IMAGE_PIXELS = IMAGE_SIZE_x*IMAGE_SIZE_y*IMAGE_CHANNELS
SAMPLE_PIXELS = 1000
flags = tf.app.flags
FLAGS = flags.FLAGS
flags.DEFINE_string('train', 'train.txt', 'File name of train data')
flags.DEFINE_string('test', 'test.txt', 'File name of train data')
flags.DEFINE_string('image_dir', 'trdata', 'Directory of images')
flags.DEFINE_string('train_dir', '/tmp/data', 'Directory to put the training data.')
flags.DEFINE_integer('max_steps', 20000, 'Number of steps to run trainer.')
flags.DEFINE_integer('batch_size', 10, 'Batch size'
'Must divide evenly into the dataset sizes.')
flags.DEFINE_float('learning_rate', 1e-5, 'Initial learning rate.')
def inference(images_placeholder, keep_prob):
""" Function to create predictive model
argument:
images_placeholder: image placeholder
keep_prob: dropout rate placeholder
Return:
y_conv:
"""
# Initialie with normal distribution with weight of 0.1
def weight_variable(shape):
initial = tf.truncated_normal(shape, stddev=0.1)
return tf.Variable(initial)
# Initialized with normal distribution with bias of 0.1
def bias_variable(shape):
initial = tf.constant(0.1, shape=shape)
return tf.Variable(initial)
# Reshape input
x_image = images_placeholder
# ramdom sumpling pixels
list = random.sample(x_image(IMAGE_PIXELS),SAMPLE_PIXELS)
x_list = [samples[i] for i in list]
# Input
with tf.name_scope('fc1') as scope:
W_fc1 = weight_variable([x_list,10])
b_fc1 = bias_variable([10])
h_fc1 = tf.nn.relu(tf.matmul(x_image,W_fc1) + b_fc1)
# Affine1
with tf.name_scope('fc2') as scope:
W_fc2 = weight_variable([10,10])
b_fc2 = bias_variable([10])
h_fc2 = tf.nn.relu(tf.matmul(h_fc1,W_fc2) + b_fc2)
# Affine2
with tf.name_scope('fc3') as scope:
W_fc3 = weight_variable([10,10])
b_fc3 = bias_variable([10])
h_fc3 = tf.nn.relu(tf.matmul(h_fc2,W_fc3) + b_fc3)
# Affine3
with tf.name_scope('fc4') as scope:
W_fc4 = weight_variable([10,10])
b_fc4 = bias_variable([10])
h_fc4 = tf.nn.relu(tf.matmul(h_fc3,W_fc4) + b_fc4)
# Affine4
with tf.name_scope('fc5') as scope:
W_fc5 = weight_variable([10,2])
b_fc5 = bias_variable([2])
# softmax regression
with tf.name_scope('softmax') as scope:
y_out=tf.nn.softmax(tf.matmul(h_fc4, W_fc5) + b_fc5)
# return
return y_out
def loss(logits, labels):
""" loss function
引数:
logits: logit tensor, float - [batch_size, NUM_CLASSES]
labels: labrl tensor, int32 - [batch_size, NUM_CLASSES]
返り値:
cross_entropy:tensor, float
"""
# cross entropy
cross_entropy = -tf.reduce_sum(labels*tf.log(tf.clip_by_value(logits,1e-10,1.0)))
# TensorBoard
tf.summary.scalar("cross_entropy", cross_entropy)
return cross_entropy
def training(loss, learning_rate):
#Adam
train_step = tf.train.AdamOptimizer(learning_rate).minimize(loss)
return train_step
def accuracy(logits, labels):
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(labels, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
tf.summary.scalar("accuracy", accuracy)
return accuracy
if __name__ == '__main__':
f = open(FLAGS.train, 'r')
# array data
train_image = []
train_label = []
for line in f:
# Separate space and remove newlines
line = line.rstrip()
l = line.split()
# Load data and resize
img = cv2.imread(FLAGS.image_dir + '/' + l[0])
img = cv2.resize(img, (IMAGE_SIZE_x, IMAGE_SIZE_y))
#transrate grayscale
img_gry = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# transrate one row and 0-1 float
train_image.append(img_gry.flatten().astype(np.float32)/255.0)
# Prepare with label 1-of-k method
tmp = np.zeros(NUM_CLASSES)
tmp[int(l[1])] = 1
train_label.append(tmp)
# transrate numpy
train_image = np.asarray(train_image)
train_label = np.asarray(train_label)
f.close()
f = open(FLAGS.test, 'r')
test_image = []
test_label = []
for line in f:
line = line.rstrip()
l = line.split()
img = cv2.imread(FLAGS.image_dir + '/' + l[0])
img = cv2.resize(img, (IMAGE_SIZE_x, IMAGE_SIZE_y))
#transrate grayscale
img_gry = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# transrate one row and 0-1 float
test_image.append(img_gry.flatten().astype(np.float32)/255.0)
tmp = np.zeros(NUM_CLASSES)
tmp[int(l[1])] = 1
test_label.append(tmp)
test_image = np.asarray(test_image)
test_label = np.asarray(test_label)
f.close()
with tf.Graph().as_default():
# Put the image Tensor
images_placeholder = tf.placeholder("float", shape=(None, IMAGE_PIXELS))
# Put the label Tensor
labels_placeholder = tf.placeholder("float", shape=(None, NUM_CLASSES))
# Put dropout rate Tensor
keep_prob = tf.placeholder("float")
# Load inference() and make model
logits = inference(images_placeholder, keep_prob)
# Load loss() and calculate loss
loss_value = loss(logits, labels_placeholder)
# Load training() and train
train_op = training(loss_value, FLAGS.learning_rate)
# calculate accuracy
acc = accuracy(logits, labels_placeholder)
# save
saver = tf.train.Saver()
# Make Session
sess = tf.Session()
# Initialize variable
sess.run(tf.global_variables_initializer())
# TensorBoard
summary_op = tf.summary.merge_all()
summary_writer = tf.summary.FileWriter(FLAGS.train_dir, sess.graph)
# Start training
for step in range(FLAGS.max_steps):
for i in range(int(len(train_image)/FLAGS.batch_size)):
batch = FLAGS.batch_size*i
sess.run(train_op, feed_dict={
images_placeholder: train_image[batch:batch+FLAGS.batch_size],
labels_placeholder: train_label[batch:batch+FLAGS.batch_size],
keep_prob: 0.5})
# Accuracy calculation for every steps
train_accuracy = sess.run(acc, feed_dict={
images_placeholder: train_image,
labels_placeholder: train_label,
keep_prob: 1.0})
print("step %d, training accuracy %g" %(step, train_accuracy))
# Added value to be displayed in Tensorflow every 1step
summary_str = sess.run(summary_op, feed_dict={
images_placeholder: train_image,
labels_placeholder: train_label,
keep_prob: 1.0})
summary_writer.add_summary(summary_str, step)
# Display accuracy on test data after training
print(" test accuracy %g"%sess.run(acc, feed_dict={
images_placeholder: test_image,
labels_placeholder: test_label,
keep_prob: 1.0}))
duration = time.time() - start_time
print('%.3f sec' %duration)
# Save model
save_path = saver.save(sess, os.getcwd() + "\\model.ckpt")

The error is this:
images_placeholder = tf.placeholder("float", shape=(None, IMAGE_PIXELS))
...
x_image = images_placeholder
list = random.sample(x_image(IMAGE_PIXELS),SAMPLE_PIXELS)
x_image, just like images_placeholder is a variable node, so x_image(...) doesn't make sense and obviously leads to an error "TypeError: 'Tensor' object is not callable".
I assume you're trying to sample SAMPLE_PIXELS from each image in a batch. Note that random.sample won't work here, because x_image is a symbolic variable, its value is only known during the session. You have to use tf.boolean_mask with a random mask in order to select random pixels from the image.

How do I get the tensorflow model to persist load?

Now I want to write 2 functions:
1 for loading model that I already trained,
2nd is using the the model to classify.
But the two function all need same session, so I make the session as a parameter, so as to seed it to the next function. But I received an error.
Here is my code. The first method is for loading the model, the second one is for using the model to predict something, but I have a few problems while init the session
def callmodel():
with tf.Graph().as_default():
#saver = tf.train.Saver()
model_path = 'E:/MyProject/MachineLearning/callTFModel/model/'
ckpt = tf.train.get_checkpoint_state(model_path)
sess = tf.Session()
saver = tf.train.import_meta_graph(ckpt.model_checkpoint_path + '.meta')
sess.run(tf.global_variables_initializer())
ckpt = tf.train.get_checkpoint_state(model_path)
if ckpt and ckpt.model_checkpoint_path:
saver.restore(sess, ckpt.model_checkpoint_path)
print("load model successful!")
return sess
else:
print("failed to load model!")
def test_one_image(sess,test_dir):
global p, logits
image = Image.open(test_dir)
image = image.resize([32, 32])
image_array = np.array(image)
image = tf.cast(image_array, tf.float32)
image = tf.reshape(image, [1, 32, 32, 3]) # 调整image的形状
p = mmodel(image, 1)
logits = tf.nn.softmax(p)
x = tf.placeholder(tf.float32, shape=[32, 32, 3])
prediction = sess.run(logits, feed_dict={x: image_array})
max_index = np.argmax(prediction)
if max_index == 0:
print('probability of good： %.6f' % prediction[:, 0])
else:
print('probability of Lack of glue： %.6f' % prediction[:, 1])
#######//test
sess=callmodel
path="c:/test/1001.jpg"
test_one_image(sess,path)
it occurs error:
File "E:/MyProject/python/C+pythonModel/test.py", line 175, in <module>
test_one_image(sess,path)
File "E:/MyProject/python/C+pythonModel/test.py", line 164, in test_one_image
prediction = sess.run(logits, feed_dict={x: image_array})
File "D:\study\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 895, in run
run_metadata_ptr)
File "D:\study\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1071, in _run
+ e.args[0])
TypeError: Cannot interpret feed_dict key as Tensor: Tensor Tensor("Placeholder:0", shape=(32, 32, 3), dtype=float32) is not an element of this graph.

The problem is not with using the session as parameter, it's with how you recover the input and output nodes of your graph: when you write
p = mmodel(image, 1)
logits = tf.nn.softmax(p)
x = tf.placeholder(tf.float32, shape=[32, 32, 3])
you are not recovering the corresponding nodes in the session graph, but creating new ones. You should instead use:
x= sess.graph().get_tensor_by_name("your_x_placeholder_name")
logits= sess.graph().get_tensor_by_name("your_logits_placeholder_name")
and then prediction = sess.run(logits, feed_dict={x: image_array})
Additionnally, you probably need to check if you have not made any mistake between image and image_array (right now you're reshaping image, not the array, which is useless if you feed with image_array...)

How to restore trained model and calculate test accuracy in Tensorflow

I have trained my CNN model and stored it in directory named model which contains files as shown below
\model
|--- checkpoint
|--- model.data-00000-of-00001
|--- model.index
|--- model.meta
I want to restore the model and calculate the test accuracy for that I am using the following code
import tensorflow as tf
import numpy as np
import cv2
import os
import glob
images = []
labels = []
img_names = []
cls = []
test_path = 'data\\cifar-10\\test'
image_size = 32
num_channels = 3
# Prepare input data
with open('data\\cifar-10\\wnids.txt') as f:
classes = f.readlines()
classes = [x.strip() for x in classes]
num_classes = len(classes)
for fields in classes:
index = classes.index(fields)
print('Read {} files (Index: {})'.format(fields, index))
path = os.path.join(test_path, fields, '*g')
files = glob.glob(path)
for fl in files:
image = cv2.imread(fl)
image = cv2.resize(image, (image_size, image_size),0,0, cv2.INTER_LINEAR)
image = image.astype(np.float32)
image = np.multiply(image, 1.0 / 255.0)
images.append(image)
label = np.zeros(len(classes))
label[index] = 1.0
labels.append(label)
flbase = os.path.basename(fl)
img_names.append(flbase)
cls.append(fields)
images = np.array(images)
labels = np.array(labels)
img_names = np.array(img_names)
cls = np.array(cls)
session = tf.Session()
tf_saver = tf.train.import_meta_graph('model\\model.meta')
tf_saver.restore(session, tf.train.latest_checkpoint('model'))
x = tf.placeholder(tf.float32, shape=[None, image_size, image_size, num_channels], name='x')
y_true = tf.placeholder(tf.float32, shape=[None, num_classes], name='y_true')
y_true_cls = tf.argmax(y_true, axis=1)
y_pred = tf.nn.softmax(layer_fc2, name='y_pred')
y_pred_cls = tf.argmax(y_pred, axis=1)
correct_prediction = tf.equal(y_pred_cls, y_true_cls)
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
feed_dict_test = {x: images, y_true: labels}
test_acc = session.run(accuracy, feed_dict=feed_dict_test)
msg = "Test Accuracy: {1:>6.1%}"
print(msg.format(test_acc))
On running the above code I'm getting the error
NameError: name 'layer_fc2' is not defined
How can I properly restore the model and calculate the test accuracy?

layer_fc2 is a python variable defined in your training script (where you define the graph) and it's not present here. What you need to do is to find this layer. Unfortunately, you didn't name it in train time. Change your create_fc_layer code to
def create_fc_layer(input, num_inputs, num_outputs, name, use_relu=True):
weights = create_weights(shape=[num_inputs, num_outputs])
biases = create_biases(num_outputs)
layer = tf.matmul(input, weights) + biases
if use_relu:
layer = tf.nn.relu(layer)
return tf.identity(layer, name=name) # return a named layer
...
layer_fc2 = create_fc_layer(input=layer_fc1, num_inputs=fc_layer_size, num_outputs=num_classes, name='layer_fc2', use_relu=False)
After this in your new script:
layer_fc2 = session.graph.get_operation_by_name('layer_fc2')
By the way, you also don't need to redefine y_pred, y_pred_cls, etc. Give them names and simply get it from the restored graph.