I am exploring the following tensorflow example: https://github.com/googledatalab/notebooks/blob/master/samples/TensorFlow/LSTM%20Punctuation%20Model%20With%20TensorFlow.ipynb which apparently is written in tf v1, so I upgraded with the v2 upgrade script and there were three main inconsistencies:
ERROR: Using member tf.contrib.rnn.DropoutWrapper in deprecated module tf.contrib. tf.contrib.rnn.DropoutWrapper cannot be converted automatically. tf.contrib will not be distributed with TensorFlow 2.0, please consider an alternative in non-contrib TensorFlow, a community-maintained repository such as tensorflow/addons, or fork the required code.
ERROR: Using member tf.contrib.legacy_seq2seq.sequence_loss_by_example in deprecated module tf.contrib. tf.contrib.legacy_seq2seq.sequence_loss_by_example cannot be converted automatically. tf.contrib will not be distributed with TensorFlow 2.0, please consider an alternative in non-contrib TensorFlow, a community-maintained repository such as tensorflow/addons, or fork the required code.
ERROR: Using member tf.contrib.framework.get_or_create_global_step in deprecated module tf.contrib. tf.contrib.framework.get_or_create_global_step cannot be converted automatically. tf.contrib will not be distributed with TensorFlow 2.0, please consider an alternative in non-contrib TensorFlow, a community-maintained repository such as tensorflow/addons, or fork the required code.
So for compatibility I manually replaced framework.get_or_create_global_step with tf.compat.v1.train.get_or_create_global_step, and also rnn.DropoutWrapper with tf.compat.v1.nn.rnn_cell.DropoutWrapper.
But I was unable to find a solution on how to handle the tf.contrib.legacy_seq2seq.sequence_loss_by_example method, since I cannot find a backwards compatible alternative. I tried installing Tensroflow Addons and use its seq2seq loss function, but wasn't able to figure out how to adapt it to work with the rest of the code.
Stumbled across some errors like Consider casting elements to a supported type. or Logits must be a [batch_size x sequence_length x logits] tensor, because probably i am not implementing something correctly.
So my question: Are you aware if legacy_seq2seq.sequence_loss_by_example is supported through some third party addon/library which I can use, and more importantly, could someone show me how to implement supported tensorflow v2 alternative of this loss function, so it acts similarly to the code below.
output = tf.reshape(tf.concat(axis=1, values=outputs), [-1, size])
softmax_w = tf.compat.v1.get_variable("softmax_w", [size, len(TARGETS)], dtype=tf.float32)
softmax_b = tf.compat.v1.get_variable("softmax_b", [len(TARGETS)], dtype=tf.float32)
logits = tf.matmul(output, softmax_w) + softmax_b
self._predictions = tf.argmax(input=logits, axis=1)
self._targets = tf.reshape(input_.targets, [-1])
loss = tfa.seq2seq.sequence_loss(
[logits],
[tf.reshape(input_.targets, [-1])],
[tf.ones([batch_size * num_steps], dtype=tf.float32)])
self._cost = cost = tf.reduce_sum(input_tensor=loss) / batch_size
self._final_state = state
Full code here.
First install tensorflow addons using:
pip install tensorflow-addons
Then import it in your program:
import tensorflow_addons as tfa
And use:
tfa.seq2seq.sequence_loss
Related
I am trying to implement CTC loss to audio files but I get the following error:
TensorFlow has no attribute 'to_int32'
I'm running tf.version 2.0.0.
I think it's with the version, I'm currently using, as we see the error is thrown in the package itself ' tensorflow_backend.py' code.
I have imported packages as "tensorflow.keras.class_name" with backend as K. Below is the screenshot.
You can cast the tensor in TensorFlow 2 as follows:
tf.cast(my_tensor, tf.int32)
You can read the documentation of the method in https://www.tensorflow.org/api_docs/python/tf/cast
You can also see that the to_int32 is deprecated and was used in TensorFlow 1
https://www.tensorflow.org/api_docs/python/tf/compat/v1/to_int32
After you make the import just write
tf.to_int=lambda x: tf.cast(x, tf.int32)
This is similar to writing the behavior of tf.to_int in everywhere in the code, so you don't have to manually edit a TF1.0 code
I want to use this TF Hub asset:
https://tfhub.dev/google/imagenet/resnet_v1_50/feature_vector/3
Versions:
Version: 1.15.0-dev20190726
Eager mode: False
Hub version: 0.5.0
GPU is available
Code
feature_extractor_url = "https://tfhub.dev/google/imagenet/resnet_v1_50/feature_vector/3"
feature_extractor_layer = hub.KerasLayer(module,
input_shape=(HEIGHT, WIDTH, CHANNELS))
I get:
ValueError: Importing a SavedModel with tf.saved_model.load requires a 'tags=' argument if there is more than one MetaGraph. Got 'tags=None', but there are 2 MetaGraphs in the SavedModel with tag sets [[], ['train']]. Pass a 'tags=' argument to load this SavedModel.
I tried:
module = hub.Module("https://tfhub.dev/google/imagenet/resnet_v1_50/feature_vector/3",
tags={"train"})
feature_extractor_layer = hub.KerasLayer(module,
input_shape=(HEIGHT, WIDTH, CHANNELS))
But when I try to save the model I get:
tf.keras.experimental.export_saved_model(model, tf_model_path)
# model.save(h5_model_path) # Same error
NotImplementedError: Can only generate a valid config for `hub.KerasLayer(handle, ...)`that uses a string `handle`.
Got `type(handle)`: <class 'tensorflow_hub.module.Module'>
Tutorial here
It's been a while, but assuming you have migrated to the TF2, this can easily be accomplished with the most recent model version as follows:
import tensorflow as tf
import tensorflow_hub as hub
num_classes=10 # For example
m = tf.keras.Sequential([
hub.KerasLayer("https://tfhub.dev/google/imagenet/resnet_v1_50/feature_vector/5", trainable=True)
tf.keras.layers.Dense(num_classes, activation='softmax')
])
m.build([None, 224, 224, 3]) # Batch input shape.
# train as needed
m.save("/some/output/path")
Please update this question if that doesn't work for you. I believe your issue arose from mixing hub.Module with hub.KerasLayer. The model version you were using was in TF1 Hub format, so within TF1 it is meant to be used exclusively with hub.Module, and not mixed with hub.KerasLayer. Within TF2, hub.KerasLayer can load TF1 Hub format models directly from their URL for composition in larger models, but they cannot be fine-tuned.
Please refer to this compatibility guide for more information
You should use tf.keras.models.save_model(model,'NeuralNetworkModel')
You will get saved model in a folder that can be used later in your sequential nework
I want to implement an RNN using Tensorflow1.13 on GPU. Following the official recommendation, I write the following code to get a stack of RNN cells
lstm = [tk.layers.CuDNNLSTM(128) for _ in range(2)]
cells = tk.layers.StackedRNNCells(lstm)
However, I receive an error message:
ValueError: ('All cells must have a state_size attribute. received cells:', [< tensorflow.python.keras.layers.cudnn_recurrent.CuDNNLSTM object at 0x13aa1c940>])
How can I correct it?
This may be a Tensorflow bug and I would suggest creating an issue on Github. However, if you want to by pass the bug, you can use:
import tensorflow as tf
import tensorflow.keras as tk
lstm = [tk.layers.CuDNNLSTM(128) for _ in range(2)]
stacked_cells = tf.nn.rnn_cell.MultiRNNCell(lstm)
This will work but it will give a deprecation warning that you can suppress.
Thanks #qlzh727. Here, I quote the response:
Either StackedRNNCells or StackedRNNCells only works with Cell, not layer. The difference between the cell and layer in RNN is that cell will only process one time step within the whole sequence, whereas the layer will process the whole sequence. You can treat RNN layer as:
for t in whole_time_steps:
output_t, state_t = cell(input_t, state_t-1)
If you want to stack 2 LSTM layers to together with cudnn in 1.x, you can do:
l1 = tf.layers.CuDNNLSTM(128, return_sequence=True)
l2 = tf.layers.CuDNNLSTM(128)
l1_output = l1(input)
l2_oupput = l2(l1_output)
In tf 2.x, we unify the cudnn and normal implementation together, you can just change the example above with tf.layers.LSTM(128, return_sequence=True), which will use the cudnn impl if available.
I am coding with tensorflow 1.5.0, python 3.5. I want to create a hashtable. Since I intend to assign values to it later, I create it in the init function like this.(the values and shape are randomly given)
enter image description here
but then I encounter a problem like this
enter image description here
Can anyone help me?
It seems that the implementation of HashTable in your version of TensorFlow does not provide kernels for every possible combination of key and value types. There are two things you can do:
According to your error message, there is a kernel implementation for 64-bit integer keys and 32-bit float values. So one possible fix is to simply change the data type of keys to tf.int64:
keys = tf.constant([1, 2, 3]), dtype=tf.int64)
Another possibility is to update TensorFlow to a version where this combination of key and value is implemented. It seems this was added in version v1.11.0-rc0 (see commmit), so upgrading to that or a later version (in general it is more recommendable to upgrade to a stable version instead of a release candidate) should also fix the problem.
The answers by #jdehesa is really great. It works for me!!! my tf version is 1.4, python=3.6
Here is my code which works:
import tensorflow as tf
from tensorflow.contrib.lookup import *
k = tf.range(1, 3, dtype=tf.int64)
v = tf.range(5, 7, dtype=tf.int64)
table = tf.contrib.lookup.HashTable(
tf.contrib.lookup.KeyValueTensorInitializer(k, v, key_dtype=tf.int64, value_dtype=tf.int64), -1)
out = table.lookup(tf.constant([2,1], dtype=tf.int64))
with tf.Session() as sess:
print(sess.run([k, v]))
table.init.run()
print(out.eval())
I’m trying to have a ConvLSTM as part of my functioning tensorflow network, because I had some issues with using the tensorflow ConvLSTM implementation, I settled for using the ConvLSTM2D Keras Layer instead.
To make Keras available in my Tensorflow session I used the blogposts suggestion (I’m using the Tensorflow backend):
https://blog.keras.io/keras-as-a-simplified-interface-to-tensorflow-tutorial.html
import tensorflow as tf
sess = tf.Session()
from keras import backend as K
K.set_session(sess)
A snippet of my code (that what causes the issues):
# state has a shape of [1, 75, 32, 32] with batchsize=1
state = tf.concat([screen, screen2, non_spatial], axis=1)
# Reshaping state to get time=1 to have the right shape for the ConvLSTM
state_reshaped = tf.reshape(state, [1, 1, 75, 32, 32])
# Keras ConvLSTM2D Layer
# I tried leaving out the batch_size for the input_shape but it didn't make a difference for the error and it seems to be fine
lstm_layer = ConvLSTM2D(filters=5, kernel_size=(3, 3), input_shape=(1, 1, 75, 32, 32), data_format='channels_first', stateful=True)(state_reshaped)
fc1 = layers.fully_connected(inputs=layers.flatten(lstm_layer), num_outputs=256, activation_fn=tf.nn.relu)
This gives me the following error:
AttributeError: 'ConvLSTM2D' object has no attribute 'outbound_nodes’”
I have no idea what this means. I thought it might has to do with mixing Keras ConvLSTM and tensorflows flatten. So I tried using Keras Flatten() instead like this:
# lstm_layer shape is (5, 5, 30, 30)
lstm_layer = Flatten(data_format='channels_first')(lstm_layer)
fc1 = layers.fully_connected(inputs=lstm_layer, num_outputs=256, activation_fn=tf.nn.relu)
and got the following error: ValueError: The last dimension of the inputs to 'Dense' should be defined. Found 'None'.
This error is caused by Flatten(), for whatever reason, having an output shape of (?, ?) and the fullyconnected layer needing to have a defined shape for the last dimension but I don't understand why it would be undefined. It was defined before.
Using Reshape((4500,))(lstm_layer) instead gives me the same no attribute 'outbound_nodes' error.
I googled the issue and I seem to not be the only one but I couldn't find a solution.
How can I solve this issue?
Is the unknown output shape of Flatten() a bug or wanted behavior, if so why?
I encountered the same problem and had a bit of a dig into the tensorflow code. The problem is that there was some refactoring done for Keras 2.2.0 and tf.keras hasn't yet been updated to this new API.
The 'outbound_nodes' attribute was renamed to '_outbound_nodes' in Keras 2.2.0. It's pretty easy to fix, there's two references in base.py you need to update:
/site-packages/tensorflow/python/layers/base.py
After updating it works fine for me.
In my case I was getting the error on a custom subclass, but the following solution can be applied nonetheless, if you subclass ConvLSTM2D and add this to your new class:
#property
def outbound_nodes(self):
if hasattr(self, '_outbound_nodes'):
print("outbound_nodes called but _outbound_nodes found")
return getattr(self, '_outbound_nodes', [])
I found the solution, even though I don't know why it works.
Currently I'm using Tensorflow 1.8 and Keras 2.2. If you downgrade Keras to ~2.1.1 it works without any problems and you can easily use Keras layers together with tensorflow. This fixed AttributeError: 'ConvLSTM2D' object has no attribute 'outbound_nodes’” and then I just used layers.flatten(lstm_layer) and everything worked.
As others have pointed out, this is because of a mismatch between your installed tensorflow and keras libraries.
Their solutions work, but in my opinion, the cleanest and easiest way to solve this is by using the keras layers contained within the tensorflow package itself rather than by using the keras library directly.
i.e, replace
from keras.layers import ConvLSTM2D
by
from tensorflow.python.keras.layers import ConvLSTM2D
This will ensure that your tensorflow and keras function calls / objects are always compatible, and solved this issue for me.