I am trying to run the following tensorflow code and it's working fine the first time. If I try running it again, it keeps throwing an error saying
ValueError: Variable layer1/weights1 already exists, disallowed. Did you mean to set reuse=True in VarScope? Originally defined at:
File "C:\Users\owner\Anaconda3\envs\DeepLearning_NoGPU\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
self._traceback = _extract_stack()
File "C:\Users\owner\Anaconda3\envs\DeepLearning_NoGPU\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "C:\Users\owner\Anaconda3\envs\DeepLearning_NoGPU\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
op_def=op_def)
If I restart the console and then run it, once again it runs just fine.
Given below is my implementation of the neural network.
import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler
import tensorflow as tf
learning_rate = 0.001
training_epochs = 100
n_input = 9
n_output = 1
n_layer1_node = 100
n_layer2_node = 100
X_train = np.random.rand(100, 9)
y_train = np.random.rand(100, 1)
with tf.variable_scope('input'):
X = tf.placeholder(tf.float32, shape=(None, n_input))
with tf.variable_scope('output'):
y = tf.placeholder(tf.float32, shape=(None, 1))
#layer 1
with tf.variable_scope('layer1'):
weight_matrix1 = {'weights': tf.get_variable(name='weights1',
shape=[n_input, n_layer1_node],
initializer=tf.contrib.layers.xavier_initializer()),
'biases': tf.get_variable(name='biases1',
shape=[n_layer1_node],
initializer=tf.zeros_initializer())}
layer1_output = tf.nn.relu(tf.add(tf.matmul(X, weight_matrix1['weights']), weight_matrix1['biases']))
#Layer 2
with tf.variable_scope('layer2'):
weight_matrix2 = {'weights': tf.get_variable(name='weights2',
shape=[n_layer1_node, n_layer2_node],
initializer=tf.contrib.layers.xavier_initializer()),
'biases': tf.get_variable(name='biases2',
shape=[n_layer2_node],
initializer=tf.zeros_initializer())}
layer2_output = tf.nn.relu(tf.add(tf.matmul(layer1_output, weight_matrix2['weights']), weight_matrix2['biases']))
#Output layer
with tf.variable_scope('layer3'):
weight_matrix3 = {'weights': tf.get_variable(name='weights3',
shape=[n_layer2_node, n_output],
initializer=tf.contrib.layers.xavier_initializer()),
'biases': tf.get_variable(name='biases3',
shape=[n_output],
initializer=tf.zeros_initializer())}
prediction = tf.nn.relu(tf.add(tf.matmul(layer2_output, weight_matrix3['weights']), weight_matrix3['biases']))
cost = tf.reduce_mean(tf.squared_difference(prediction, y))
optimizer = tf.train.AdamOptimizer(learning_rate).minimize(cost)
with tf.Session() as session:
session.run(tf.global_variables_initializer())
for epoch in range(training_epochs):
session.run(optimizer, feed_dict={X: X_train, y: y_train})
train_cost = session.run(cost, feed_dict={X: X_train, y:y_train})
print(epoch, " epoch(s) done")
print("training complete")
As the error suggests I tried adding reuse=True as a parameter in with tf.variable_scope(): but that again is not working.
I am running this inside a conda environment. I am using Python 3.5 and CUDA 8(But it shouldn't matter because this is not configured to run in the GPU) in windows 10.
This is a matter of how TF works. One needs to understand that TF has a "hidden" state - a graph being built. Most of the tf functions create ops in this graph (like every tf.Variable call, every arithmetic operation and so on). On the other hand actual "execution" happens in the tf.Session(). Consequently your code will usually look like this:
build_graph()
with tf.Session() as sess:
process_something()
since all actual variables, results etc. leave in session only, if you want to "run it twice" you would do
build_graph()
with tf.Session() as sess:
process_something()
with tf.Session() as sess:
process_something()
Notice that I am building graph once. Graph is an abstract representation of how things look like, it does not hold any state of computations. When you try to do
build_graph()
with tf.Session() as sess:
process_something()
build_graph()
with tf.Session() as sess:
process_something()
you might get errors during second build_graph() due to trying to create variables with the same names (what happens in your case), graph being finalised etc. If you really need to run things this way you simply have to reset graph in between
build_graph()
with tf.Session() as sess:
process_something()
tf.reset_default_graph()
build_graph()
with tf.Session() as sess:
process_something()
will work fine.
Related
TensorFlow graph API separates graph construction and execution. Because of this, I can't understand in which line neural network is executed.
"""
- model_fn: function that performs the forward pass of the model
- init_fn: function that initializes the parameters of the model.
- learning_rate: the learning rate to use for SGD.
"""
tf.reset_default_graph()
is_training = tf.placeholder(tf.bool, name='is_training')
with tf.device(device):
x = tf.placeholder(tf.float32, [None, 32, 32, 3])
y = tf.placeholder(tf.int32, [None])
params = init_fn() # Initialize the model parameters
scores = model_fn(x, params) # Forward pass of the model
loss = training_step(scores, y, params, learning_rate) # SGD
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for t, (x_np, y_np) in enumerate(train_dset):
feed_dict = {x: x_np, y: y_np}
loss_np = sess.run(loss, feed_dict=feed_dict)
As described in the Tensorflow documentation: (https://www.tensorflow.org/api_docs/python/tf/Session#run)
This method runs one "step" of TensorFlow computation, by running the
necessary graph fragment to execute every Operation and evaluate every
Tensor
In you example, sess.run(tf.global_variables_initializer()) run the initilization operation that create all the weights and tensors, loss_np = sess.run(loss, feed_dict=feed_dict) executes all the operations up to loss.
I hope this answers your question
I've a simple MNIST which I've successfully saved, being the code the next:
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data', one_hot=True)
import tensorflow as tf
sess = tf.InteractiveSession()
tf_save_file = './mnist-to-save-saved'
x = tf.placeholder(tf.float32, shape=[None, 784])
y_ = tf.placeholder(tf.float32, shape=[None, 10])
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
saver = tf.train.Saver()
sess.run(tf.global_variables_initializer())
y = tf.matmul(x, W) + b
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels = y_, logits = y))
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
saver.save(sess, tf_save_file)
for _ in range(1000):
batch = mnist.train.next_batch(100)
train_step.run(feed_dict={x: batch[0], y_: batch[1]})
correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
saver.save(sess, tf_save_file, global_step=1000)
print(accuracy.eval(feed_dict={x: mnist.test.images, y_: mnist.test.labels}))
Then, the next files are generated:
checkpoint
mnist-to-save-saved-1000.data-00000-of-00001
mnist-to-save-saved-1000.index
mnist-to-save-saved-1000.meta
mnist-to-save-saved.data-00000-of-00001
mnist-to-save-saved.index
mnist-to-save-saved.meta
Now, in order to use it in production (and so, for example, pass it a number image), I want to be able to execute the trained model by passing it any number image to make the prediction (I mean, not deploying yet a server but making this prediction "locally", having in the same directory that "fixed" number image, so using the model would be like when you run an executable).
But, considering the (mid-low?) API level of my code, I'm confused about what would be the easiest correct next step (if restoring, using an Estimator, etc...), and how to do it.
Although I've read the official documentation, I insist that they seem to be many ways, but some are a bit complex and "noisy" for a simple model like this.
Edit:
I've edit and re-run the mnist file, whose code is the same as above except for those lines:
...
x = tf.placeholder(tf.float32, shape=[None, 784], name='input')
...
correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1), name='result')
...
Then, I try to run this another .py code (in the same directory as the above code) in order to pass a local handwritten number image ("mnist-input-image.png") located in the same directory:
import tensorflow as tf
from PIL import Image
import numpy as np
image_test = Image.open("mnist-input-image.png")
image = np.array(image_test)
with tf.Session() as sess:
saver = tf.train.import_meta_graph('/Users/username/.meta')
new = saver.restore(sess, tf.train.latest_checkpoint('/Users/username/'))
graph = tf.get_default_graph()
input_x = graph.get_tensor_by_name("input:0")
result = graph.get_tensor_by_name("result:0")
feed_dict = {input_x: image}
predictions = result.eval(feed_dict=feed_dict)
print(predictions)
Now, if I correctly understand, I've to pass the image as numpy array. Then, my questions are:
1) Which is the exact file reference of those lines (since I've no .meta folder in my User folder)?
saver = tf.train.import_meta_graph('/Users/username/.meta')
new = saver.restore(sess, tf.train.latest_checkpoint('/Users/username/'))
I mean, to which exact files refer those lines (from my generated files list above)?
2) Translasted to my case, is correct this line to pass my numpy array into the feed dict?
feed_dict = {input_x: image}
A simple solution is to use your session object. When you have generated the checkpoint file, you can restore it with a Saver object.
By the way, do you know why most tutorials have their graph creation inside of a function? One good reason is because you can deserialize the graph quickly with your inputs.
The correct method to start a session is with the following:
# Use your placeholders, variables, etc to create the entire graph.
# Usually you return the input placeholder,
# prediction and the loss/accuracy here.
# You don't need the accuracy.
x, y, _ = make_your_graph(test_X, test_y)
# This object is the interface for serialization in tf
saver = tf.train.Saver()
with tf.Session() as sess:
# Takes your current model's checkpoint. "./checkpoint" is your checkpoint file.
saver.restore(sess, tf.train.latest_checkpoint("./checkpoint"))
prediction = sess.run(y)
Want to run more than 1 data point for your already-booted up session?
Then replace the last line with a feed dict:
while waiting_for_new_y():
another_y = get_new_y()
feed_dict = {x: [another_y]}
another_prediction = sess.run(y, feed_dict)
First of all , give value to name parameter in each object which you want to use later , so that you can use it later by it's name:
change this :
x = tf.placeholder(tf.float32, shape=[None, 784])
to
x = tf.placeholder(tf.float32, shape=[None, 784],name='input')
and
correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
to
correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1),name='result')
Now run this small script to store model :
import tensorflow as tf
with tf.Session() as sess:
saver = tf.train.import_meta_graph('/Users/dummy/.meta')
new=saver.restore(sess, tf.train.latest_checkpoint('/Users/dummy/'))
graph = tf.get_default_graph()
input_x = graph.get_tensor_by_name("input:0")
result = graph.get_tensor_by_name("result:0")
feed_dict = {input_x: mnist.test.images,} #here you feed your new data for example i am feeding mnist
predictions = result.eval(feed_dict=feed_dict)
print(predictions)
And you will get output.
I have saved a model and now I am trying to restore it, after restoring it works for the first time correctly but when I press 'TEST' button on the same running program to test another image it gives error
ValueError: At least two variables have the same name: Variable_2/Adam
def train_neural_network(x):
prediction = neural_network_model(x)#logits
softMax=tf.nn.softmax_cross_entropy_with_logits(
logits=prediction, labels=y)#prediction and original comapriosn
cost = tf.reduce_mean(softMax)#total loss
optimizer = tf.train.AdamOptimizer(learning_rate=0.001).minimize(cost)#learning_rate=0.01
hm_epochs = 20
new_saver = tf.train.Saver()
with tf.Session() as sess:
global s
s=sess
sess.run(tf.global_variables_initializer())
new_saver = tf.train.import_meta_graph('../MY_MODELS/my_MNIST_Model_1.meta')
new_saver.restore(s, tf.train.latest_checkpoint('../MY_MODELS'))
correct = tf.equal(tf.argmax(prediction, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct, 'float'))
print('Accuracy:', accuracy.eval(
{x: mnist.test.images, y: mnist.test.labels}))
The graph which you have loaded already has all the variables needed for inference. You need to load the tensors like accuracy from the saved graph. In your case you have same variables declared outside, which conflicts with the variables in the graph.
During training, if you have named your tensor accuracy with a name='accuracy', then you can load it from the graph using: get_tensor_by_name('accuracy:0'). In your example, you need to load the input tensors x and y also from the graph. Your code should be like:
def inference():
loaded_graph = tf.Graph()
new_saver = tf.train.Saver()
with tf.Session(graph=loaded_graph) as sess:
new_saver = tf.train.import_meta_graph('../MY_MODELS/my_MNIST_Model_1.meta')
new_saver.restore(s, tf.train.latest_checkpoint('../MY_MODELS'))
#Get the tensors by their variable name
# Note: the names of the following tensors have to be declared in your train graph for this to work. So just name them appropriately.
_accuracy = loaded_graph.get_tensor_by_name('accuracy:0')
_x = loaded_graph.get_tensor_by_name('x:0')
_y = loaded_graph.get_tensor_by_name('y:0')
print('Accuracy:', _accuracy.eval(
{_x: mnist.test.images, _y: mnist.test.labels}))
First of all, I'm very new in Python and Tensorflow either.
I'm trying on demo of link: https://www.tensorflow.org/get_started/mnist/beginners
and it runs well.
However, I would like to debug (or log) the value of some placeholders, variables which are changed when I run Session.run(). I
Could you please show me the way to "debug" or log them when Session running in the loops?
Here is my code
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("mnist/", one_hot=True)
x = tf.placeholder(tf.float32, [None, 784])
W = tf.Variable(tf.zeros([784,10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b)
y1 = tf.add(tf.matmul(x,W),b)
y_ = tf.placeholder(tf.float32, [None, 10])
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))
cross_entropy1 = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y1, y_))
train_step = tf.train.GradientDescentOptimizer(0.05).minimize(cross_entropy1)
sess = tf.InteractiveSession()
tf.global_variables_initializer().run()
for _ in range(1000):
batch_xs, batch_ys = mnist.train.next_batch(100)
sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
sess.run(tf.argmax(y,1), feed_dict={x: mnist.test.images, y_: mnist.test.labels})
In this script, I would like to log the value of y and tf.argmax(y, 1) for each test image processed.
Mrry answered it best in this stackoverflow answer: https://stackoverflow.com/a/33633839/6487788
Exactly what you are asking (printing during sess.run) would be this part of his answer:
To print the value of a tensor without returning it to your Python program, you can use the tf.Print() op, as And suggests in another answer. Note that you still need to run part of the graph to see the output of this op, which is printed to standard output. If you're running distributed TensorFlow, the tf.Print() op will print its output to the standard output of the task where that op runs.
This would be this code for argmax:
argmaxy = tf.Print(tf.argmax(y,1))
correct_prediction = tf.equal(argmaxy, tf.argmax(y_,1))
Good luck!
While #rmeerten's answer is correct, you can consider also using TensorBoard which can be a useful tool for debugging your models and seeing what's happening. For background, you can also check out the TensorBoard session from the TensorFlow Dev Summit.
I'm learning how to use Tensorflow with the MNIST tutorial, but I'm blocking on a point of the tutorial.
Here is the code provided :
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
x = tf.placeholder(tf.float32, [None, 784])
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b)
y_ = tf.placeholder(tf.float32, [None, 10])
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
saver = tf.train.Saver()
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
for i in range(1000):
batch_xs, batch_ys = mnist.train.next_batch(100)
sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
But I actually don't understand at all how the variables "W" (The weight) and "b" (The bias) are changed while the computing ?
On each batch, they are initialized at zero, but after ?
I don't see at all where in the code they're going to change ?
Thanks you very much in advance!
TensorFlow variables maintain their state from one run() call to the next. In your program they will be initialized to zero, and then progressively updated in the training loop.
The code that changes the values of the variables is created, implicitly, by this line:
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
In TensorFlow, a tf.train.Optimizer is a class that creates operations for updating variables, typically based on the gradient of some tensor (e.g. a loss) with respect to those variables. By default, when you
call Optimizer.minimize(), TensorFlow creates operations to update all variables on which the given tensor (in this case cross_entropy) depends.
When you call sess.run(train_step), this runs a graph that includes those update operations, and therefore instructs TensorFlow to update the values of the variables.