In Tensorflow and Python, I am doing the following sort of thing to understand Tensorflow and debug code by observing multiple variables at the end of a computation.
with tf.Session():
print "var1 ="
print (var1.eval({x:myInputs, y:myOutputs}))
print "var2 ="
print (var2.eval({x:myInputs, y:myOutputs}))
Does Tensorflow rerun the entire graph computation for each eval() call? Seems inefficient to rerun the entire graph just to print out one variable (tensor). If that is what is happening, is there a way to run the graph/process/eval once and then print out the values of each variable, without rerunning the entire graph?
When you call Tensor.eval(), TensorFlow (i) works out what subgraph of the whole graph it needs to run to produce the value of that tensor, then (ii) runs that entire graph.
It is often more efficient to use Session.run() to fetch the values of multiple tensors at once. For example, you could rewrite your code as follows to run the graph once:
with tf.Session() as sess:
val1, val2 = sess.run([var1, var2], {x:myInputs, y:myOutputs})
print "var1 =", val1
print "var2 =", val2
Related
I'm finding a way to getting a tensor's value. In most case, the problem would be solved by calling "sess.run(target_op)". However, I want to know another way. I am editing the code downloaded from GitHub so there's already a session running code there. Without touching the session running part, is there any way to get some specific tensor value? In my case, the code is built for getting accuracy for image recognition. While session runs and doing the accuracy evaluation I also want to get "prediction" tensor value in the same session without creating another session. For example, an operation like tf.Print shows tensor value througha terminal window without running session directly(in the first figure we just have to do sess.run(e) to print out tensor from c)
example of tf.Print
a = tf.constant(5)
b = tf.constant(3)
c = tf.add(a,b)
#print tensor c (which is 8)
d = tf.Print(c,[c])
f = tf.constant(2)
e = tf.multiply(f,d)
sess = tf.Session()
#print operation can be executed without running the session directly
g = sess.run(e)`
Like the tf.Print is there any operation that gets tensor value without running session directly? (like the second figure)
example of operation I am looking for
More specifically, what I want is to get the value of tensor(with actual numbers and arrays, not just 'tensor' data structure)and pass it to the global variable to access the value freely even after the session closes. The session only executes the operator which is located at end of the graph while the tensor I want the value is located in the middle of the graph. With restriction that I cannot create more session than the original code has, is there any way to get the specific tensor value?( I can't use .eval() or .run() because either needs to access 'session'. the code I am editing runs the code by using slim.evaluate_once function and as session() is binded to the function, I cannot approach to session())
There is no reason why you can't just call any tensor from the graph, provided you feed in the appropriate feed_dict. For instance say you want a tensor called biasAdd:0 and your so called-end tensor is called prediction
Then you can just get this tensor and evaluate it:
tensor = graph.get_tensor_by_name("biasAdd:0")
tensor_value, prediction_value = ses.run([tensor, prediction],... )
In tensorflow you have to use run or eval to get a numerical value from the graph
In Keras, if you make a custom loss function in a Jupyter notebook, you can not print anything. For instance if you have:
def loss_func(true_label, NN_output):
true_cat = true_label[:,0]
pred_cat = NN_output[:,0]
indicator = NN_output[:,1]
print("Hi!")
custom_term = K.mean(K.abs(indicator))
return binary_crossentropy(true_cat, pred_cat) + custom_term
Nothing will print when the function is evaluated.
As a workaround, in case I am doing some debugging, I have found that I can write to a file in a cost function, which can be useful if I want to print something standard like an int or a string.
However, trying to write out a tensor like indicator to a file gives the unbelievably helpful output:
Tensor("loss_103/model_105_loss/Print:0", shape=(512,), dtype=float32)
I know TF provides a tf.Print() method to print the value of a tensor, but I don't understand how that plays with Jupyter. Other answers have said that tf.Print() writes to std. err, which means trying
sys.stderr = open('test.txt', 'w')
should theoretically allow me to get my output from a file, but unfortunately this doesn't work (at least in Jupyter).
Is there any general method to get a representation of my tensor as a string? How do people generally get around this barrier to seeing what your code does? If I come up with something more fancy than finding a mean, I want to see exactly what's going on in the steps of my calculation to verify it works as intended.
Thanks!
You can do something like the below code:
def loss_func(true_label, NN_output):
true_cat = true_label[:,0]
true_cat = tf.Print(true_cat, [true_cat], message="true_cat: ") # added line
pred_cat = NN_output[:,0]
pred_cat = tf.Print(pred_cat, [pred_cat], message="pred_cat: ") # added line
indicator = NN_output[:,1]
custom_term = K.mean(K.abs(indicator))
return binary_crossentropy(true_cat, pred_cat) + custom_term
Basically I have added two lines to print the values of true_cat, pred_cat.
To print something, you have to include the print statement in the tf graph by the above statements.
However, the trick is it's going to print on jupyter notebook console where you're running the notebook not on the ipython notebook itself.
References:
How to print the value of a Tensor object in TensorFlow?
Printing the loss during TensorFlow training
https://www.tensorflow.org/api_docs/python/tf/Print
import tensorflow as tf
def read_data(file_queue):
reader = tf.TextLineReader(skip_header_lines=1)
key, value = reader.read(file_queue)
defaults = [[0], [0]]
value1, value2 = tf.decode_csv(value, defaults)
return tf.stack(value1, value2)
But I would like to deal with some of the data, such as "3 wins 2 fails" converted to [3, 2]
Using tensorflow you're describing a computation. value1,value2 (and any other operation resulting from a tf.* call) are symbolic variables, that points to nodes in a graph.
That's why if you print "the data" you get Tensor("DecodeCSV:0", shape=(), dtype=int32), that's the python representation of "the data".
The actual data, instead, is present only once the graph is built and placed into a Session.
In short, if you want to extract the "real data" you have to exit from the tensorflow graph and fetch the values (thus forcing the execution of the operations described in the graph).
You have to do something like:
sess = tf.Session()
v1,v2 = sess.run([value1, value2])
return v2,v2
However, this is not the correct way of using tensorflow.
Instead, you have to describe as much as possible the computation, and then execute everything into the graph when needed.
(creating a session, allocate the memory, place the graph into it, execute the operations, data transfer, ... are heavy operations that should be done not so frequently)
Therefore, I suggest you of having a look at the control flow operation that tensorflow offers: https://www.tensorflow.org/api_guides/python/control_flow_ops#Comparison_Operators
You can control the flow of the values into the graph using them, avoiding to force useless exchanges of data between tensorflow and python
I'm trying to run open code of A3C reinforcement learning algorithm to learn A3C in A3C code
However,I got several errors and I could fix except one.
In the code, ref() which is a member function of tf.Variable is used (1,2), but in recent tensorflow version 0.12rc, that function seems to be deprecated.
So I don't know what is the best way to replace it (I don't understand exactly why the author used ref()). When I just changed it to the variable itself (for example v.ref() to v), there was no error, but reward is not changed. It seems it cannot learn and I guess it is because the variables are not properly updated.
Please advise me what is the proper way to modify the code to work.
The new method tf.Variable.read_value() is the replacement for tf.Variable.ref() in TensorFlow 0.12 and later.
The use case for this method is slightly tricky to explain, and is motivated by some caching behavior that causes multiple uses of a remote variable on a different device to use a cached value. Let's say you have the following code:
with tf.device("/cpu:0")
v = tf.Variable([[1.]])
with tf.device("/gpu:0")
# The value of `v` will be captured at this point and cached until `m2`
# is computed.
m1 = tf.matmul(v, ...)
with tf.control_dependencies([m1])
# The assign happens (on the GPU) after `m1`, but before `m2` is computed.
assign_op = v.assign([[2.]])
with tf.control_dependencies([assign_op]):
with tf.device("/gpu:0"):
# The initially read value of `v` (i.e. [[1.]]) will be used here,
# even though `m2` is computed after the assign.
m2 = tf.matmul(v, ...)
sess.run(m2)
You can use tf.Variable.read_value() to force TensorFlow to read the variable again later, and it will be subject to whatever control dependencies are in place. So if you wanted to see the result of the assign when computing m2, you'd modify the last block of the program as follows:
with tf.control_dependencies([assign_op]):
with tf.device("/gpu:0"):
# The `read_value()` call will cause TensorFlow to transfer the
# new value of `v` from the CPU to the GPU before computing `m2`.
m2 = tf.matmul(v.read_value(), ...)
(Note that, currently, if all of the ops were on the same device, you wouldn't need to use read_value(), because TensorFlow doesn't make a copy of the variable when it is used as the input to an op on the same device. This can cause a lot of confusion—for example when you enqueue a variable to a queue!—and it's one of the reasons that we're working on enhancing the memory model for variables.)
Sorry for not posting entire snippets -- the code is very big and spread out, so hopefully this can illustrate my issue. I have these:
train = theano.function([X], output, updates=update_G,
givens={train_mode=:np.cast['int32'](1)})
and
test = theano.function([X], output, updates=update_G,
givens={train_mode=:np.cast['int32'](0)})
to my understanding, givens would input the value of train_mode (i.e. 1/0) wherever it's needed to compute the output.
The output is computed in the lines of this:
...
network2 = Net2()
# This is sort of a dummy variable so I don't get a NameError when this
# is called before `theano.function()` is called. Not sure if this is the
# right way to do this.
train_mode = T.iscalar('train_mode')
output = loss(network1.get_outputs(network2.get_outputs(X, train_mode=train_mode)),something).mean()
....
class Net2():
def get_outputs(self, x, train_mode):
from theano.ifelse import ifelse
import theano.tensor as T
my_flag = ifelse(T.eq(train_mode, 1), 1, 0)
return something if my_flag else something_else
So train_mode is used as an argument in one of the nested functions, and I use it to tell between train and test as I'd like to handle them slightly differently.
However, when I try to run this, I get this error:
theano.compile.function_module.UnusedInputError: theano.function was
asked to create a function computing outputs given certain inputs, but
the provided input variable at index 1 is not part of the computational
graph needed to compute the outputs: <TensorType(int32, scalar)>.To make
this error into a warning, you can pass the parameter
on_unused_input='warn' to theano.function. To disable it completely, use
on_unused_input='ignore'.
If I delete the givens parameter, the error disappears, so to my understanding Theano believes that my train_mode is not necessary for compute the function(). I can use on_unusued_input='ignore' as per their suggestion, but that would just ignore my train_mode if they think it's unused. Am I going around this the wrong way? I basically just want to train a neural network with dropout, but not use dropout when evaluating.
why you use "=" sign? I think, it made train_mode not readable, my code works well by writing:
givens = {train_mode:1}