I am using WALS method in order to perform matrix factorization. Initially in tensorflow 1.13 I can import factorization_ops using
from tensorflow.contrib.factorization.python.ops import factorization_ops
As described in the documentation
Wals model can be called from factorization_ops by using
factorization_ops.WALSModel
Using same command in tensorflow 2.0 giving me following error
ModuleNotFoundError: No module named 'tensorflow.contrib.factorization
Going through the issue there appears to be no way out to use WALSModel in tensorflow 2.0+.
Also it has been mentioned here in tensorflow release updates that tf.contrib has been deprecated, and functionality has been either migrated to the core TensorFlow API, to an ecosystem project such as tensorflow/addons or tensorflow/io, or removed entirely.
How can I use WALS model in tensorflow 2.0 (Currently I am using 2.0.0-rc0 on windows machine) ? Is WALSModel has been removed or I am missing out some information ?
I believe WALS is not supported in TF 2.0 ...The official recommendation model is Neural Collaborative Filter (NCF)
I hope this helps.
M
I have the same issue, but I don't really have time to write a library myself unfortunately. There are several potential options that I am considering:
Stick with TF1.X until someone creates a library
Switch to using lightfm to continue using WALS
Switch to neural collaborative filtering using embedding layers with keras and a dot product layer. See this paper https://arxiv.org/abs/1708.05031, and this code implementation:
from tensorflow.keras.layers import Input, Embedding, Flatten, Dot, Dense
from tensorflow.keras.models import Model
#import tensorflow.distribute
def get_compiled_model(n_users, n_items, embedding_dims=20):
# Product embedding
prod_input = Input(shape=[1], name="Item-Input")
prod_embedding = Embedding(n_items+1, embedding_dims, name="Item-Embedding")(prod_input)
prod_vec = Flatten(name="Flatten-Product")(prod_embedding)
# User embedding
user_input = Input(shape=[1], name="User-Input")
user_embedding = Embedding(n_users+1, embedding_dims, name="User-Embedding")(user_input)
user_vec = Flatten(name="Flatten-Users")(user_embedding)
# The output is the dot product of the two, i.e. a one-hot vector
dot_product = Dot(name="Dot-Product", axes=1)([prod_vec, user_vec])
# compile - uncomment these two lines to make training distributed
# dist_strat = distribute.Strategy()
# with dist_strat.scope():
model = Model(inputs = [user_input, prod_input], outputs = dot_product)
model.compile(
optimizer='adam',
loss='mean_squared_error'
)
return model
I have compared the Tensorflow implementation of WALS to other implementations with respect to compute resources and accuracy (https://github.com/gtsoukas/cfzoo). The comparison suggests that the implicit Python package (https://github.com/benfred/implicit) is a good replacement that delivers superior performance.
I think that WALS has been removed. As part of tf.contrib it is not supported by TF2 and I do not think it fits in any of core or sub-projects.
Your best bet is probably to make it available as a third-party library.
I expect to use it for my project, but the need to re-write it (mainly copy what was in TF1 and make it work as a separate library compatible with TF2) reduce the priority of this task...
Let us know if you start to code something. Thanks.
Alexis.
Related
I am looking for a layer that randomly shears a batch of images, such as the preprocessing layers in tf.keras.layers.experimental.preprocessing. However, there doesn't seem to be any such layer.
There are a few similar questions to implementing shear layers in TF, but those use deprecated methods in tf.contrib (here and here). Can someone point me in a direction to implement random shears in tensorflow 2?
Few of the operations of tf.contrib in Tensorflow 2.x is moved to Tensorflow addons.
Equivalent for tf.contrib.image.transoform is tfa.image.transform
For more details on library find here
And also take a look at Random_shear using tf.keras.preprocessing.image.random_shear
I'm perplexed by the Tensorflow post-training quantization process. The official site refers to Tensorflow Lite Quantization. Unfortunately, this doesn't work in my case, that is, TFLiteConverter returns errors for my Mask RCNN model:
Some of the operators in the model are not supported by the standard TensorFlow Lite runtime and are not recognized by TensorFlow. If you have a custom implementation for them you can disable this error with --allow_custom_ops, or by setting allow_custom_ops=True when calling tf.lite.TFLiteConverter(). Here is a list of builtin operators you are using: <...>. Here is a list of operators for which you will need custom implementations: DecodeJpeg, StatelessWhile.
Basically, I've tried all available options offered by TFLiteConverter including experimental ones. I'm not quite surprised with those errors as it might make sense not to support decodejpeg for the mobile, however, I want my model to be served by Tensorflow Serving, thus I don't know why Tensorflow Lite is the official choice to go for.
I've also tried Graph Transform Tool, which seems to be deprecated, and discovered 2 issues. Firstly it's impossible to quantize with bfloat16 or float16, only int8. Secondly, the quantized model breaks with the error:
Broadcast between [1,20,1,20,1,256] and [1,1,2,1,2,1] is not supported yet
what isn't an issue in the regular model.
Furthermore, it's worth to mention my model was originally built with Tensorflow 1.x, and then ported to Tensorflow 2.1 via tensorflow.compat.v1.
This issue stole a significant amount of my time. I'd be grateful for any cue.
You can convert the model to Tensorflow Lite and use unsupported ops (Like DecodeJpeg) from TF, this is called SELECT TF OPS, see the guide here on how to enable it during conversion.
Is there (more or less) simple way to write a complicated NN model so it will be trainable in the eager mode? Are there examples of a such code?
For example, I want to use the InceptionResnetV2. I have the code created with tf.contrib.slim. According to this link, https://github.com/tensorflow/tensorflow/issues/16182 , slim is deprecated and I need to use Keras. And I really can't use a slim code for training with eager because I can't take the list of variables and apply gradients (ok, I can try to wrap model into the GradientTape but not sure what to do with regularization loss).
Ok, let's try Keras.
In [30]: tf.__version__
Out[30]: '1.13.1'
In [31]: tf.enable_eager_execution()
In [32]: from keras.applications.inception_resnet_v2 import InceptionResNetV2
In [33]: model = InceptionResNetV2(weights=None)
...
/usr/local/lib/python3.6/dist-packages/keras_applications/inception_resnet_v2.py in InceptionResNetV2(include_top, weights, input_tensor, input_shape, pooling, classes, **kwargs)
246
247 if input_tensor is None:
--> 248 img_input = layers.Input(shape=input_shape)
249 else:
250 if not backend.is_keras_tensor(input_tensor):
...
RuntimeError: tf.placeholder() is not compatible with eager execution.
Doesn't work by default.
In this tutorial they say that I need to make my own class of a model and maintain variables by myself https://www.tensorflow.org/tutorials/eager/custom_training#define_the_model. I'm not sure that I want to do it for Inception. Too many variables to create and maintain. It's like returning back to old versions of TF, in days when even slim didn't exist.
In this tutorial networks are created using Keras https://www.tensorflow.org/tutorials/eager/custom_training_walkthrough#create_a_model_using_keras but I doubt that I can easily maintain complicated structure in a such way, by only defining model without using it with Input. For example, in this article, if I understand correctly, author initialize keras Input and propagate it through the model (which causes RuntimeError when used with Eager, as you seen earlier). I can make my own model by subclassing the model class as here: https://www.tensorflow.org/api_docs/python/tf/keras/Model . Oops, in this way I need to maintain layers, not variables. It seems as almost the same problem to me.
There is an interesting mention of AutoGrad here https://www.tensorflow.org/beta/guide/autograph#keras_and_autograph . They only overwrite __call__, so it seems like I don't need to maintain variables in this case, but I didn't test it yet.
So, is there any simple solution?
Wrap slim model in GradientTape? How can I then apply reg loss to the weights?
Track every variable by myself? Sounds a little bit painful.
Use Keras? How to use it with eager when I have branches and complicated structure in the model?
Your first approach is probably the most common. This error:
RuntimeError: tf.placeholder() is not compatible with eager execution.
is because one cannot use a tf.placeholder in eager mode. There is no concept of such a thing when executing eagerly.
You could use the tf.data API to build a dataset for your training data and feed that to the model. Something like this with the datasets replaced with your real data:
import tensorflow as tf
tf.enable_eager_execution()
model = tf.keras.applications.inception_resnet_v2.InceptionResNetV2(weights=None)
model.compile(tf.keras.optimizers.Adam(), loss=tf.keras.losses.categorical_crossentropy)
### Replace with tf.data.Datasets for your actual training data!
train_x = tf.data.Dataset.from_tensor_slices(tf.random.normal((10,299,299,3)))
train_y = tf.data.Dataset.from_tensor_slices(tf.random.uniform((10,), maxval=10, dtype=tf.int32))
training_data = tf.data.Dataset.zip((train_x, train_y)).batch(BATCH_SIZE)
model.fit(training_data)
This approach works as is in TensorFlow 2.0 too as mentioned in your title.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I am using Keras (with Theano) to train my CNN model. Does anyone has idea how can I use it in my C++ application? Does anyone tried something similar? I have idea to write some python code that will generate a c++ code with network functions - any suggestion on it?
I found a similar question here how to use Tensorflow Keras model in C++ but without answer.
To answer my own question and have a solution - I wrote a plain c++ solution called keras2cpp (its code available on github).
In this solution you store network architecture (in json) and weights (in hdf5). Then you can dump a network to a plain text file with provided script. You can use obtained text file with network in pure c++ code. There are no dependencies on python libraries or hdf5. It should work for theano and tensorflow backend.
I found myself in a similar situation but needed to not only support forward passes of sequential Keras models in C++ but also of more complex models build with the functional API.
So I wrote a new library called frugally-deep. You can find it on GitHub and it is published under the MIT License: https://github.com/Dobiasd/frugally-deep
Additionally to supporting many common layer types it can keep up with (and sometimes even beat) the performance of TensorFlow on a single CPU. You can find up-to-date benchmark results for some common model in the repo.
By automatic testing frugally-deep guarantees that the output of a model used with it in C++ is exactly the same as if run with Keras in Python.
If your keras model is trained using tensorflow backend, you can save the keras model as a tensorflow model following this code:
https://github.com/amir-abdi/keras_to_tensorflow
Here is a shorter version of the code:
from keras import backend as K
from tensorflow.python.framework import graph_util
from tensorflow.python.framework import graph_io
weight_file_path = 'path to your keras model'
net_model = load_model(weight_file_path)
sess = K.get_session()
constant_graph = graph_util.convert_variables_to_constants(sess, sess.graph.as_graph_def(), 'name of the output tensor')
graph_io.write_graph(constant_graph, 'output_folder_path', 'output.pb', as_text=False)
print('saved the constant graph (ready for inference) at: ', osp.join('output_folder_path', 'output.pb'))
You Can try this one
https://github.com/gosha20777/keras2cpp
Keras2cpp is a small library for running trained Keras models from a C++ application without any dependencies.
Supported Keras layers:
- Dense
- Convolution1D
- Convolution2D
- Convolution3D
- Flatten
- ELU
- Activation
- MaxPooling2D
- Embedding
- LocallyConnected1D
- LocallyConnected2D
- LSTM
- GRU
- CNN
- BatchNormalization
Supported activation:
- linear
- relu
- softplus
- tanh
- sigmoid
- hard_sigmoid
- elu
- softsign
- softmax
Design goals:
Compatibility with networks generated by Keras using TensorFlow backend.
CPU only.
No external dependencies, standard library, C++17.
Model stored in memory.
The solutions found here are quite good, but if your model has some different types of layers not supported by these libraries, I would recommend doing the following:
Converting the Keras model to a tensorflow model.
Freeze the model and use Tranform graph tool provided by tensorflow (you'll have to build it from source with bazel)
Compile the C++ API tensorflow library to use it in your project.
Use the C++ API tensorflow library and link the libraries to your project.
If you want to use a something differentcompiler than bazel (like g++ for example) you can follow this great tuturial:
http://tuatini.me/building-tensorflow-as-a-standalone-project/
The easiest way is probably to make a system call to a Python script that writes the predictions to a binary or HDF5 file, which can be read in from C++. You can also directly integrate Python into C++.
If you need to deploy and distribute this easily, you can look into self-contained Python installations like Anaconda, but your best bet may be to avoid Keras and use the C++ interface to Caffe or Tensorflow. I wouldn't recommend Tensorflow since using it from C++ isn't standard; see this discussion. Caffe is arguably the second most-popular deep learning library so you can't really go wrong.
I had a similar need--I wanted to embed Keras models in a C++ application--and decided to write my own library: Kerasify
Design goals of Kerasify:
Compatibility with image processing Sequential networks generated by Keras using Theano backend. (Could work with Tensorflow if you switch around matrix col/row ordering).
No external dependencies, standard library, C++11 features OK.
Model stored on disk in binary format that can be quickly read.
Model stored in memory in contiguous block for better cache performance.
Doesn't throw exceptions, returns only bool on error.
CPU only, no GPU
Example code, unit tests, etc. at the github link. It's not fully complete, it only supports the narrow subset of Keras functions I'm using, but it should be extensible with a little effort.
I have been working with Keras for a week or so. I know that Keras can use either TensorFlow or Theano as a backend. In my case, I am using TensorFlow.
So I'm wondering: is there a way to write a NN in Keras, and then print out the equivalent version in TensorFlow?
MVE
For instance suppose I write
#create seq model
model = Sequential()
# add layers
model.add(Dense(100, input_dim = (10,), activation = 'relu'))
model.add(Dense(1, activation = 'linear'))
# compile model
model.compile(optimizer = 'adam', loss = 'mse')
# fit
model.fit(Xtrain, ytrain, epochs = 100, batch_size = 32)
# predict
ypred = model.predict(Xtest, batch_size = 32)
# evaluate
result = model.evaluate(Xtest)
This code might be wrong, since I just started, but I think you get the idea.
What I want to do is write down this code, run it (or not even, maybe!) and then have a function or something that will produce the TensorFlow code that Keras has written to do all these calculations.
First, let's clarify some of the language in the question. TensorFlow (and Theano) use computational graphs to perform tensor computations. So, when you ask if there is a way to "print out the equivalent version" in Tensorflow, or "produce TensorFlow code," what you're really asking is, how do you export a TensorFlow graph from a Keras model?
As the Keras author states in this thread,
When you are using the TensorFlow backend, your Keras code is actually building a TF graph. You can just grab this graph.
Keras only uses one graph and one session.
However, he links to a tutorial whose details are now outdated. But the basic concept has not changed.
We just need to:
Get the TensorFlow session
Export the computation graph from the TensorFlow session
Do it with Keras
The keras_to_tensorflow repository contains a short example of how to export a model from Keras for use in TensorFlow in an iPython notebook. This is basically using TensorFlow. It isn't a clearly-written example, but throwing it out there as a resource.
Do it with TensorFlow
It turns out we can actually get the TensorFlow session that Keras is using from TensorFlow itself, using the tf.contrib.keras.backend.get_session() function. It's pretty simple to do - just import and call. This returns the TensorFlow session.
Once you have the TensorFlow session variable, you can use the SavedModelBuilder to save your computational graph (guide + example to using SavedModelBuilder in the TensorFlow docs). If you're wondering how the SavedModelBuilder works and what it actually gives you, the SavedModelBuilder Readme in the Github repo is a good guide.
P.S. - If you are planning on heavy usage of TensorFlow + Keras in combination, have a look at the other modules available in tf.contrib.keras
So you want to use instead of WX+b a different function for your neurons. Well in tensorflow you explicitly calculate this product, so for example you do
y_ = tf.matmul(X, W)
you simply have to write your formula and let the network learn. It should not be difficult to implement.
In addition what you are trying to do (according to the paper you link) is called batch normalization and is relatively standard. The idea being you normalize your intermediate steps (in the different layers). Check for example https://www.google.ch/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0ahUKEwikh-HM7PnWAhXDXRQKHZJhD9EQFggyMAE&url=https%3A%2F%2Farxiv.org%2Fabs%2F1502.03167&usg=AOvVaw1nGzrGnhPhNGEczNwcn6WK or https://www.google.ch/url?sa=t&rct=j&q=&esrc=s&source=web&cd=4&ved=0ahUKEwikh-HM7PnWAhXDXRQKHZJhD9EQFghCMAM&url=https%3A%2F%2Fbcourses.berkeley.edu%2Ffiles%2F66022277%2Fdownload%3Fdownload_frd%3D1%26verifier%3DoaU8pqXDDwZ1zidoDBTgLzR8CPSkWe6MCBKUYan7&usg=AOvVaw0AHLwD_0pUr1BSsiiRoIFc
Hope that helps,
Umberto