I'm trying to use tf.contrib.layers.fully_connected() in one of my projects, and it's been deprecated in tensorflow 2.0. Is there an equivalent function, or should I just keep tensorflow v1.x in my virtual environment for this projcet?
In TensorFlow 2.0 the package tf.contrib has been removed (and this was a good choice since the whole package was a huge mix of different projects all placed inside the same box), so you can't use it.
In TensorFlow 2.0 we need to use tf.keras.layers.Dense to create a fully connected layer, but more importantly, you have to migrate your codebase to Keras. In fact, you can't define a layer and use it, without creating a tf.keras.Model object that uses it.
tf-slim, as a standalone package, already included tf.contrib.layers.you can install by pip install tf-slim,call it by from tf_slim.layers import layers as _layers; _layers.fully_conntected(..).The same as the original, easy to replace
use: tf.compat.v1.layers.dense
for example, instead of
Z = tf.contrib.layers.fully_connected(F, num_outputs, activation_fn=None)
you can replace it with:
Z = tf.compat.v1.layers.dense(F, num_outputs, activation = None)
tf.contrib.layers.fully_connected() is a perfect mess. It is a very old historical mark(or a prehistory DNN legacy). Google has completely deprecated the function since Google hated it. There is no any direct function in TensoFlow 2.x to replace tf.contrib.layers.fully_connected(). Therefore, it is not worth inquiring and getting to know the function.
Related
I'm using a codebase that was written in 2017/18 and I found the following code:
audio_norm = audio_norm.unsqueeze(0)
audio_norm = torch.autograd.Variable(audio_norm, requires_grad=False)
I am aware that wrapping tensors in Variable formerly allowed their gradients to be incorporated into the computation graph Torch builds for previous versions of Torch (now no longer needed) but I'm confused what the utility of wrapping a tensor in torch.autograd.Variable(my_tensor, requires_grad=False) would be.
Could someone explain if this was an idiom and what the analogous modern Torch code would be? My guess would be calling detach on the tensor to stop its gradients being tracked.
For reference, the relevant line from the codebase is line 45 from the data_utils.py script of NVIDIA's Tacotron 2 implementation. Thanks.
In PyTorch 0.3.1 and earlier, any tensor involved in a computation that needed to be tracked by autograd had to be wrapped in a Variable. Semantically Variable.requires_grad in PyTorch 0.3.1 and earlier is equivalent to Tensor.requires_grad now. Basically, requires_grad=False simply tells autograd that you will never need the gradient w.r.t. that variable/tensor. Mathematical operations are only ever recorded (i.e. a computation graph is constructed) if at least one input variable/tensor has requires_grad=True.
Note that any code using PyTorch newer than 0.3.1 does not actually require the use of Variable, this includes the code in the repository you provided (which explicitly requires PyTorch >= 1.0). In 0.4 the functionality of Variable was merged into the Tensor class. In modern PyTorch, you simply have to set the requires_grad attribute of the tensor to achieve the same behavior. By default, a new user-defined tensor is already constructed with requires_grad=False, so the modern equivalent of the code you posted is usually to just delete the Variable line. If you aren't sure if the tensor already has requires_grad == False then you could explicitly set it.
audio_norm = audio_norm.unsqueeze(0)
audio_norm.requires_grad_(False)
You can read the legacy documentation here for more information.
you are looking for
audio_norm = audio_norm.unsqueeze(0)
audio_norm = torch.tensor(audio_norm)
if you need it to require grad then
audio_norm = torch.tensor(audio_norm, require_grad=True)
I am training my model using model.train_on_batch(). I dig into Keras code how to use train_ob_batch properly. I notice before calling this function, Keras calls several routines, including model._make_train_function().
But, I found many Github sources, they use model.train_on_batch without calling _make_train_function() first.
So, my question is whether calling _make_train_function() has to be called first before train_on_batch? I am using Keras 2.2.4 and Tensorflow 1.12.0
Thanks.
What is the difference between the different fully connected layers available in tensorflow. I understand that there could 2 versions: Object oriented and functional, but I was able to find 4 different layers in tensorflow:
tf.keras.layers.Dense
tf.layers.dense
tf.layers.Dense
tf.contrib.layers.fully_connected
The documentation contains examples using all of them. I'd also like to know when to use each layer.
Keras is a deep learning library which functions as a wrapper over 'lower level' languges such as Tensorflow and Theano. It has recently been integrated as a Tensorflow project and is part of the code-base. If you are using 'raw' Tensorflow, you should not use this layer.
Tensorflow defines a functional interface. Layers and operations that are lowercase are typically part of this. These functions are used as building blocks when defining a custom layer or a loss function.
This is the layer you should be using.
This comes from the contrib library - features that are typically more experimental and volatile. Once a feature is deemed stable, you should use its other implementation (3). (4) will still be present in the library to maintain backwards compatability.
Is a Keras wrapper function. Its functionality is same as 3. Checkout Keras.
Its a functional interface for tensorflow.
Commonly used.
Function under development.
Technically speaking first 3 have same functionality (same inputs and outputs).
Is there any documentation on whether these are the same or different modules and the differences between these?
keras.layers.Conv2D
keras.layers.Convolution2D
keras.layers.convolutional.Conv2D
keras.layers.convolutional.Convolution2D
They're aliases for the same functionality. The reason behind this is related to the new Keras 2 API, which tries to give users some time to migrate their code to the new one, and to use the shorter one with different parameters. The other aliases eventually are going to be deprecated, since when using old Keras API shows warnings.
See Keras code
I want to save a model that I have trained.
Since it is using shared variables (like Weights, bias and so on) and since it should be readable on machines without Theano installed, I wanted to use the theano.misc.pkl_utils.dump() function.
However, it seems as if that is only installed in bleeding edge installations (the current github file looks different than my local one).
Is that really the case? And why is the description in the docs then?
I am using theano 0.7.0 and I'm seriously confused about this.
If that feature is not yet available (I can't install bleeding edge right now), what are other ways? I'm sure that I am not the only one trying to save a trained model the easiest way possible ;-)
Thank you a lot,
Roman
If you train your model using Theano, the parameters of the model will be eventually shared variables (probably a list of shared variables if the network consists of several layers). It is possible to pickle the list of shared variables and unpickle it later. However you might have problems to unpickle such variables in another machine, e.g. with no Theano installation or if you train in a GPU-capable machine that generates CudaNdarrays and then you want to load back the model in a non-GPU-capable machine. What I recommend you is the following: convert every shared variable of the list of parameters into a numpy ndarray:
params_numpy = [numpy.asarray(p.get_value()) for p in params]
where params is a list of shared variables. Then you can safely pickle/unpickle params_numpy.