I have a working input pipeline that uses multiple threads to read data from tfrecords files, decodes and preprocesses the data, and serves them to the graph. I used this example as a starting point. My problem is that I need the input batch again in the loss layer, to compute a reconstruction error of the input. If I now just add the dequeuing operation multiple times to my graph, i.e. once at the input layer and once at the loss layer, I obviously get two times a different batch since tensorflow evaluates the operation twice. What is the easiest way to overcome this?
Probably you can try adding identity operation after dequeuing and connect other nodes to that identity operation.
Related
I am trying to save my trained model to a file to ship to my inference server, and then benchmark it. The only issue is that my model (using keras) has init layers (such as conv1/weights/RandomUniform, for example) still in it, which can not be run on the device I am benchmarking.
So, how do I go about removing all of these layers from my graph?
I've tried using tfgraph.finalize(), convert_variables_to_constants, and remove_training_nodes, and none seem to remove these nodes.
The exact layer it breaks on is:
'res_net50/conv1/kernel/Initializer/random_uniform/RandomUniform'
The strip_unused_nodes transformation allows you to provide a set of input and output nodes. The graph transformation tool will remove all branches of the graph that do not feed into the output nodes.
Having said that, my gut feeling is that removing the initialisation nodes will not have a significant impact on inference time because the operations will not be evaluated. They do however consume some memory if they are not pruned.
As stated in the question, I need to attach a tensor to a particular point in the computation graph in Pytorch.
What I'm trying to do is this:
while geting outputs from all mini-batches, accumulate them in a list and when one epoch finishes, calculate the mean. Then, I need to calculate loss according to the mean, therefore backpropagation must consider all these operations.
I am able to do that when the training data is not much (without detaching and storing). However, this is not possible when it gets bigger. If I don't detach output tensors each time, I'm running out of GPU memories and if I detach, I lose the track of output tensors from the computation graph. Looks like this is not possible no matter how many GPUs I have since PyTorch does only use first 4 for storing output tensors if I don't detach before saving them into a list even if I assign more than 4 GPUs.
Any help is really appreciated.
Thanks.
Suppose I have a fully-trained TensorFlow network that converts an input A into an output B, without any additional inputs. I want to iterate the network to generate a series of states A,B,C,D,... where each output is fed as an input to the network in order to generate the next state (like a Markov chain).
Is there a way to do this iteration entirely within TensorFlow (ie, include iteration within the graph) while recording a snapshot of the output at each timestep? This seems to me like it might involve RNN, but I am unsure how to build an RNN when I already have a working network for the forward pass.
I am trying to replicate a neural network for depth estimation. The original authors have taken a pre-trained network and added between the fully connected layer and the convolutional layer a 'Superpixel Pooling Layer'. In this layer, the convolutional feature maps are upsampled and the features per superpixel are averaged.
My problem is that in order to successfully achieve this, I need to calculate the superpixels per image. How can I access the data being used by keras/tensorflow during batch processing to perform SLIC oversegmentation?
I considered splitting the tasks and working by pieces i.e. feed the images into the convolutional network. Process the outputs separately and then feed them into a fully connected layer. However, this makes further training of the network impossible.
At the time it seems to be impossible to actually access the data within the symbolic tensor. It also seems unlikely that such functionality will be added in the future since in the Tensorflow page it says:
A Tensor object is a symbolic handle to the result of an operation, but
does not actually hold the values of the operation's output.
Keras allows for the creation of personalized layers. However, these are limited by the available backend operations. As such, it is simply not possible to access the batch data.
I am using tensorflow for a classification problem.
I have some utility for saving and loading my network. When I restore the network, I can specify a different batch size than for training.
My problem is that I am getting different test results when I restore with a different batch size. However, there is no difference when using the same batch size.
EDIT: Please note that I am not using dropout.
The difference is between 0% and 1% (0.5% on average).
My network is a fully connected layer that predicts two different outputs. I did not have the issue when I only had one task to predict.
My loss op is a sum of both losses.
What could be the issue? Does it have to do with Tensorflow's parallelization strategy?
This normally means that you did not set the phase_train parameter back to false after testing.