In Keras, if you make a custom loss function in a Jupyter notebook, you can not print anything. For instance if you have:
def loss_func(true_label, NN_output):
true_cat = true_label[:,0]
pred_cat = NN_output[:,0]
indicator = NN_output[:,1]
print("Hi!")
custom_term = K.mean(K.abs(indicator))
return binary_crossentropy(true_cat, pred_cat) + custom_term
Nothing will print when the function is evaluated.
As a workaround, in case I am doing some debugging, I have found that I can write to a file in a cost function, which can be useful if I want to print something standard like an int or a string.
However, trying to write out a tensor like indicator to a file gives the unbelievably helpful output:
Tensor("loss_103/model_105_loss/Print:0", shape=(512,), dtype=float32)
I know TF provides a tf.Print() method to print the value of a tensor, but I don't understand how that plays with Jupyter. Other answers have said that tf.Print() writes to std. err, which means trying
sys.stderr = open('test.txt', 'w')
should theoretically allow me to get my output from a file, but unfortunately this doesn't work (at least in Jupyter).
Is there any general method to get a representation of my tensor as a string? How do people generally get around this barrier to seeing what your code does? If I come up with something more fancy than finding a mean, I want to see exactly what's going on in the steps of my calculation to verify it works as intended.
Thanks!
You can do something like the below code:
def loss_func(true_label, NN_output):
true_cat = true_label[:,0]
true_cat = tf.Print(true_cat, [true_cat], message="true_cat: ") # added line
pred_cat = NN_output[:,0]
pred_cat = tf.Print(pred_cat, [pred_cat], message="pred_cat: ") # added line
indicator = NN_output[:,1]
custom_term = K.mean(K.abs(indicator))
return binary_crossentropy(true_cat, pred_cat) + custom_term
Basically I have added two lines to print the values of true_cat, pred_cat.
To print something, you have to include the print statement in the tf graph by the above statements.
However, the trick is it's going to print on jupyter notebook console where you're running the notebook not on the ipython notebook itself.
References:
How to print the value of a Tensor object in TensorFlow?
Printing the loss during TensorFlow training
https://www.tensorflow.org/api_docs/python/tf/Print
Related
I'm looking for a way to replace or delete the last message wrote by python's logging module. The goal is to log a change in variables once it occurs. If the variable changes again, the old log message should be deleted and the new one printed instead.
Hi,
I am using pythons's logging module for a deep learning project I'm currently working on. As some GPUs just don't have enough memory to support the default batch size during training and there is no apparent connection between batch size and actual memory usage that could be used for calculations beforehand, I'm catching the runtime error once it occurs and decrease the batch size by one.
This process can be repeated quite a bit and I'm always logging which batch size did not work and which will be the next one tried. Instead of having 10-30 of these messages (or more) I'd like to simply delete the last one and replace it with the newer one instead.
I've already checked out the python logging documentation, stumbled upon the LogRecord object, but upon trying to deal with it, it seems this object does not actually keep a record of all logs, but rather saves some more info on one specific log instead.
If there is simply no way to do this, I will look into some kind of bundling solution as described here: Python logging: bundle reoccurring messages
The code below shows the log message I'm looking to replace.
Any help is greatly apreciated.
training_not_successful = True
while training_not_successful:
try:
model.run_training(global_settings['epochs'],
train_loader,
test_loader,
global_settings['checkpoint_output_path'],
model_name,
global_settings['best_net_criterion'])
training_not_successful = False
except MemoryError:
logging.warning("Ran out of CUDA memory using batch size " + str(batch_size) +
". Trying again with batch size " + str(batch_size-1))
batch_size -= 1
train_loader, test_loader = get_train_test_loaders(
train_dataset_list,
test_dataset_list,
value_counts,
batch_size
)
I believe (correct me if I'm wrong) that the logging module does not allow to supress newlines, meaning that it's simply not possible to do something like that.
It is possible to do it with a print though:
import shutil
def display(variable, rewritable=False):
columns, lines = shutil.get_terminal_size(fallback=(80, 20))
text = str(variable)
filled = text + ((columns - len(text)) * ' ')
print(filled, end='\r' if rewritable else '\n')
if __name__ == "__main__":
from random import random
from time import sleep
for i in range(10):
display(f"x = {random()}", True)
sleep(1)
display(f"x = 0.0") # test if old value is overwritten completely
display("Done!")
Tested this on linux, but it should work everywhere. (the shutil.get_terminal_size function)
It's not mandatory, but very nice when entire line is overwritten as opposed to only the part that's changed.
The key is character \r - it returns the cursor to the end of the line, that's it. Now you can start writing from the front again, overwriting the line if it has anything else, which is exactly what you want.
Display function is simple, but I'll explain it anyway:
First line gets terminal size, what we need is the width of the line, so we can pad the text with spaces and fill entire line with spaces, to completely overwrite previous line no matter what it had.
Then we convert our variable to a string.
After that, it's just simple math, our string takes n characters, so the rest should be spaces, so we add width - n spaces to final string, and then we print it - entire line is overwritten.
rewritable flag allows to control when the variable should be rewritten next time you call display.
While this is not what you want, as it does not use logging module, since there's no way (that I know of) to make logging module to print \r instead of \n, I think this is a good enough substitute, that could be used if it turns out that you can indeed do this with logging module.
I'm finding a way to getting a tensor's value. In most case, the problem would be solved by calling "sess.run(target_op)". However, I want to know another way. I am editing the code downloaded from GitHub so there's already a session running code there. Without touching the session running part, is there any way to get some specific tensor value? In my case, the code is built for getting accuracy for image recognition. While session runs and doing the accuracy evaluation I also want to get "prediction" tensor value in the same session without creating another session. For example, an operation like tf.Print shows tensor value througha terminal window without running session directly(in the first figure we just have to do sess.run(e) to print out tensor from c)
example of tf.Print
a = tf.constant(5)
b = tf.constant(3)
c = tf.add(a,b)
#print tensor c (which is 8)
d = tf.Print(c,[c])
f = tf.constant(2)
e = tf.multiply(f,d)
sess = tf.Session()
#print operation can be executed without running the session directly
g = sess.run(e)`
Like the tf.Print is there any operation that gets tensor value without running session directly? (like the second figure)
example of operation I am looking for
More specifically, what I want is to get the value of tensor(with actual numbers and arrays, not just 'tensor' data structure)and pass it to the global variable to access the value freely even after the session closes. The session only executes the operator which is located at end of the graph while the tensor I want the value is located in the middle of the graph. With restriction that I cannot create more session than the original code has, is there any way to get the specific tensor value?( I can't use .eval() or .run() because either needs to access 'session'. the code I am editing runs the code by using slim.evaluate_once function and as session() is binded to the function, I cannot approach to session())
There is no reason why you can't just call any tensor from the graph, provided you feed in the appropriate feed_dict. For instance say you want a tensor called biasAdd:0 and your so called-end tensor is called prediction
Then you can just get this tensor and evaluate it:
tensor = graph.get_tensor_by_name("biasAdd:0")
tensor_value, prediction_value = ses.run([tensor, prediction],... )
In tensorflow you have to use run or eval to get a numerical value from the graph
I have set up a problem with Pyomo that optimises the control strategy of a CHP unit, which is laid out (very roughly) in the following way:
class Problem
def OptiControl
<Pyomo Concrete model formulation>
return (model.obj.value())
The problem class is there because there are several control strategies methods that I'm investigating, and so I call these by using (for example) b = problem1.OptiControl().
The problem is that if I try to return the value of the objective my script gets stuck and when I Ctrl+C out of it I get (' Signal', 2, 'recieved, but no process queued'). Also if I write model.write() the script ends normally but nothing is displayed in IPython. Trying to print model.obj.value() also doesn't work.
I assume this has something to do with the fact that I'm calling Pyomo in a function because the model worked succesfully before, but I don't know how to get round this.
EDIT: writing the values to a file also doesn't work. If it helps, this is the excerpt of my code where I solve the model:
opt = SolverFactory("glpk") # Choose solver
solution = opt.solve(model) # Solve model
model.write()
with open('ffs.txt','w') as f:
model.obj.expr()
for t in model.P:
f.write(model.f["CHP",t].value)
I ended up working around this problem by re-writing my model as an AbstractModel, writing my data to a .dat file and reading that (clunky, but I need results now so it had to be done). For some reason it works now and does everything I expect it would do.
It also turned out my problem was infeasible which might have added to my problems, but at least with this method I could use results.write() and I was able to find that out which wasn't the case before.
Sorry for not posting entire snippets -- the code is very big and spread out, so hopefully this can illustrate my issue. I have these:
train = theano.function([X], output, updates=update_G,
givens={train_mode=:np.cast['int32'](1)})
and
test = theano.function([X], output, updates=update_G,
givens={train_mode=:np.cast['int32'](0)})
to my understanding, givens would input the value of train_mode (i.e. 1/0) wherever it's needed to compute the output.
The output is computed in the lines of this:
...
network2 = Net2()
# This is sort of a dummy variable so I don't get a NameError when this
# is called before `theano.function()` is called. Not sure if this is the
# right way to do this.
train_mode = T.iscalar('train_mode')
output = loss(network1.get_outputs(network2.get_outputs(X, train_mode=train_mode)),something).mean()
....
class Net2():
def get_outputs(self, x, train_mode):
from theano.ifelse import ifelse
import theano.tensor as T
my_flag = ifelse(T.eq(train_mode, 1), 1, 0)
return something if my_flag else something_else
So train_mode is used as an argument in one of the nested functions, and I use it to tell between train and test as I'd like to handle them slightly differently.
However, when I try to run this, I get this error:
theano.compile.function_module.UnusedInputError: theano.function was
asked to create a function computing outputs given certain inputs, but
the provided input variable at index 1 is not part of the computational
graph needed to compute the outputs: <TensorType(int32, scalar)>.To make
this error into a warning, you can pass the parameter
on_unused_input='warn' to theano.function. To disable it completely, use
on_unused_input='ignore'.
If I delete the givens parameter, the error disappears, so to my understanding Theano believes that my train_mode is not necessary for compute the function(). I can use on_unusued_input='ignore' as per their suggestion, but that would just ignore my train_mode if they think it's unused. Am I going around this the wrong way? I basically just want to train a neural network with dropout, but not use dropout when evaluating.
why you use "=" sign? I think, it made train_mode not readable, my code works well by writing:
givens = {train_mode:1}
In Tensorflow and Python, I am doing the following sort of thing to understand Tensorflow and debug code by observing multiple variables at the end of a computation.
with tf.Session():
print "var1 ="
print (var1.eval({x:myInputs, y:myOutputs}))
print "var2 ="
print (var2.eval({x:myInputs, y:myOutputs}))
Does Tensorflow rerun the entire graph computation for each eval() call? Seems inefficient to rerun the entire graph just to print out one variable (tensor). If that is what is happening, is there a way to run the graph/process/eval once and then print out the values of each variable, without rerunning the entire graph?
When you call Tensor.eval(), TensorFlow (i) works out what subgraph of the whole graph it needs to run to produce the value of that tensor, then (ii) runs that entire graph.
It is often more efficient to use Session.run() to fetch the values of multiple tensors at once. For example, you could rewrite your code as follows to run the graph once:
with tf.Session() as sess:
val1, val2 = sess.run([var1, var2], {x:myInputs, y:myOutputs})
print "var1 =", val1
print "var2 =", val2