I have a TensorFlow Keras model (TensorFlow 2.6.0); here's a basic example:
import tensorflow as tf
x = inp = tf.keras.Input((5,))
x = tf.keras.layers.Dense(7, activation="relu")(x)
x = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(inp, x)
I would like to get all the tf.Operation objects in the graph for the model, select specific operations, then create a new tf.function or tf.keras.Model to output the values of those tensors on arbitrary inputs.
For example, in my simple model above, I might want to get the outputs of all relu operators. I know in that case, I could redefine the model to include the output of that layer as another output of the model, but the point here is that I already have the model (it's much more complicated than above), and there are specific operators that I want to find to get the outputs of.
Have you tried this:
all_ops = tf.get_default_graph().get_operations()
If you got an empty list and you use Tensorflow 2.x , you may try this:
import tensorflow as tf
print(tf.__version__)
tf.compat.v1.disable_eager_execution() # disable eager execution
a = tf.constant([1],name='aa')
print(tf.compat.v1.get_default_graph().get_operations())
print(tf.compat.v1.get_default_graph().get_tensor_by_name('aa:0'))
Related
I have this line of code (for Tensorflow 1.0):
tf.placeholder(tf.float32, [None, n, p])
n and p are just random numbers.
How to translate this line of code into tf.keras.input for Tensorflow 2.0?
Thanks a lot in advance!
From comments
Issue was resolved after creating two different virtual environments
for TF 2.x and 1.x.
For more information you can refer detail answer mentioned by Denver
here.
(paraphrased from lindo)
From Tensorflow documentation
In TF 2.x, you can just pass tensors directly into ops and layers. If you want to explicitly set up your inputs, you can see Keras functional API on how to use tf.keras.Input to replace tf.compat.v1.placeholder. tf.function arguments also do the job of tf.compat.v1.placeholder. For more details please read Better performance with tf.function.
Input produces a symbolic tensor-like object (i.e. a placeholder). This can be used with lower-level TensorFlow ops that take tensors as inputs as shown below
x = Input(shape=(32,))
y = tf.square(x) # This op will be treated like a layer
model = Model(x, y)
For more details you can refer Migrate your TensorFlow 1 code to TensorFlow 2
I have tried tons of methods like onnx2keras, pytorch2keras and so on. But there would always be something wrong...
Since my model is not really complicated: just a ResNet18-encoder + Decoder with some skip-connections. I'm considering to simply transfer them one layer by another, from pytorch to Keras.
Before I try I'd like to ask if you have similar experience? I know there's set_weights method, but that's for keras-to-keras so nothing special. However, Keras is object-based model, so how can I assign name-based weights, e.g. 'encoder.bn1.bias', 'encoder.bn1.running_mean', 'encoder.bn1.running_var' to a BN? I don't want TF1.x solutions because all of my work is on TF2.x.
So In my opinion, it would be something like:
# 1. Save weights and names from pytorch model
weights_dict = torch_mode.static_dict()
# 2. Construct Keras model
keras_model = tf.keras.models.Model(...)
# 3. Now load weights for each layer in Keras model
for var_name, weight in weights_dict.items():
# Assign conv with weight with'encoder.conv1.weight'
# Assign BN with 'encoder.bn1.weight', 'encoder.bn1.bias', 'encoder.bn1.running_mean', 'encoder.bn1.running_var', 'encoder.bn1.num_batches_tracked'
But I don't know how... Look forward to your opinions!
Could you try pt2keras and see if it works?
link: https://github.com/JWLee89/pt2keras/
To install pt2keras, type the following in the terminal:
pip install -U pt2keras
Below is a simple example for converting resnet18.
import tensorflow as tf
from torchvision.models.resnet import resnet18
from pt2keras import Pt2Keras
if __name__ == '__main__':
input_shape = (1, 3, 224, 224)
# Grab model
model = resnet18(pretrained=False).eval()
# Create pt2keras object
converter = Pt2Keras()
# convert model
keras_model: tf.keras.Model = converter.convert(model, input_shape, strict=True)
# Save the model
keras_model.save('output_model.h5')
# Do whatever else that you want afterwards ...
I have attached the converted keras model visualized using netron:
Before I try I'd like to ask if you have similar experience? I know there's set_weights method, but that's for keras-to-keras so nothing special. However, Keras is object-based model, so how can I assign name-based weights, e.g. 'encoder.bn1.bias', 'encoder.bn1.running_mean', 'encoder.bn1.running_var' to a BN? I don't want TF1.x solutions because all of my work is on TF2.x.
Unfortunately, as far as I know, you cannot attach name-based weights to individual parameters in Keras like you can in PyTorch, since keras is layer-based. However, you can name the batch-norm layer, which I am guessing is not very useful to you.
I would like to create a model consisting of 2 convolutional, one flatten, and one dense layer in Keras. This would be a model with shared weights, so without any predefined input layer.
It is possible to do using the sequential way:
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv2D(10,3,2,'valid',activation=tf.nn.relu))
model.add(tf.keras.layers.Conv2D(20,3,2,'valid',activation=tf.nn.relu))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(200,activation=tf.nn.relu))
However, using the Functional API, produces a TypeError:
model2 = tf.keras.layers.Conv2D(10,3,2,'valid',activation=tf.nn.relu)
model2 = tf.keras.layers.Conv2D(20,3,2,'valid',activation=tf.nn.relu)(model2)
model2 = tf.keras.layers.Flatten()(model2)
model2 = tf.keras.layers.Dense(200,activation=tf.nn.relu)(model2)
Error :
TypeError: Inputs to a layer should be tensors. Got: <tensorflow.python.keras.layers.convolutional.Conv2D object at 0x7fb060598100>
Is it impossible to do this way, or am I missing something?
The keras sequential api is designed to be easier to use, and as a result is less flexible than the functional api. The benefit of this is that an input 'layer' shape can be inferred automatically by whatever shape of the data you pass to it. The downside is that this easier to use model is simplified, and so you can't do things like using multiple inputs.
From the keras docs:
A Sequential model is not appropriate when:
Your model has multiple inputs or multiple outputs
Any of your layers has multiple inputs or multiple outputs
You need to do layer sharing
You want non-linear topology (e.g. a residual connection, a
multi-branch model)
The functional api is designed to be more flexible i.e. multiple inputs, and so it doesn't make any sort of automatic inference for you, hence the error. You must explicitly pass an input layer in this case. For your use case, it might seem odd that it doesn't automatically infer the shape, however when you consider the wider use-case scenario it makes sense.
So the second scenario should be :
model2 = tf.keras.layers.Input((10,3,2)) # specified input layer
model2 = tf.keras.layers.Conv2D(10,3,2,'valid',activation=tf.nn.relu)(model2)
model2 = tf.keras.layers.Conv2D(20,3,2,'valid',activation=tf.nn.relu)(model2)
model2 = tf.keras.layers.Flatten()(model2)
model2 = tf.keras.layers.Dense(200,activation=tf.nn.relu)(model2)
Update
If you want to create two separate models and join them together, you should use the functional API, and then due to it's constraints you must therefore use input layers. So you could do something like:
import tensorflow as tf
from tensorflow.keras.layers import Input, Flatten, Dense, concatenate, Conv2D
from tensorflow.keras.models import Model
input1 = Input((10,3,2))
model1 = Dense(200,activation=tf.nn.relu)(input1)
input2 = Input((10,3,2))
model2 = Dense(200,activation=tf.nn.relu)(input2)
merged = concatenate([model1, model2])
merged = Conv2D(10,3,2,'valid',activation=tf.nn.relu)(merged)
merged = Flatten()(merged)
merged = Dense(200,activation=tf.nn.relu)(merged)
model = Model(inputs=[input1, input2], outputs=merged)
Above we have two separate inputs and then two Dense layers - you can build these separate lines however you want, and then to merge them together to pass them through a convolutional layer you need to use a tf.keras.layers.concatenate layer, and then you can continue the joint model from there. Wrapping the whole thing inside a Model object then allows you access training and inference methods like fit/predict etc.
The linking in keras works by propagating tensors through the layers. So in your second example, at the beginning model2 is an instance of a keras.layers.Layer and not a tf.Tensor that why you get the error.
Input creates a tensor which can then be used to link the layers. So if there is not a specific reason, you just add one:
model2 = tf.keras.layers.Input((10,3,2))
model2 = tf.keras.layers.Conv2D(10,3,2,'valid',activation=tf.nn.relu)(model2)
I'm looking to do SVD for a custom optimizer in Keras (specifically, I want to port the the Shampoo optimizer to Keras.
In Tensorflow, I would use tensorflow.python.ops.linalg_ops.svd(), however, there is no function like this in keras.backend.
Can SVD be carried out in a purely Keras setting, or can I somehow use the Tensorflow function directly (and if so, how)?
EDIT: Just for future reference, there actually exists a wrapper function allowing the direct use of a native tf optimizer in Keras:
import keras as ks
from tensorflow.contrib.opt import AdamWOptimizer
tfopt = AdamWOptimizer()
ksopt = ks.optimizers.TFOptimizer(tfopt)
Unfortunately though, it does not seem to work with the Shampoo optimizer specifically.
If you are using keras with a tensorflow backend, than keras backend is tensorflow.
This means that when you call a method from keras backend, it actually calls a method of tensorflow.
Therefore you could use both keras backend operations and tensorflow together and interchangeably.
For example, in the given code:
tensor = ...
m = K.mean(tensor)
...
I could change the line K.mean(tensor) to tf.mean(tensor)
tensor = ...
m = tf.mean(tensor)
...
So you can just use the tensorflow SVD operation as you would use it if it was a function of keras backend :)
For example if you would like to have
tensor = ...
res = K.some_submodule.svd(tensor)
...
Than you can instead just do
tensor = ...
res = tensorflow.python.ops.linalg_ops.svd(tensor)
...
I'm implementing WGAN and need to clip weight variables.
I'm currently using Tensorflow with Keras as high-level API. Thus building layers with Keras to avoid manually creation and initialization of variables.
The problem is WGAN need to clip weight varibales, This can be done using tf.clip_by_value(x, v0, v1) once I got those weight variable tensors, but I don't know to how to get them safely.
One possible solution maybe using tf.get_collection() to get all trainable variables. But I don't know how to get only weight variable without bias variables.
Another solution is layer.get_weights(), but it get numpy arrays, although I can clip them with numpy APIs and set them using layer.set_weights(), but this may need CPU-GPU corporation, and may not be a good choice since clip operation needs to be performed on each train step.
The only way I know is access them directly using exact variable names which I can get from TF lower level APIs or TensorBoard, but this is may not be safe since naming rule of Keras is not guaranteed to be stable.
Is there any clean way to perform clip_by_value only on those Ws with Tensorflow and Keras?
You can use constraints(here) class to implement new constraints on parameters.
Here is how you can easily implement clip on weights and use it in your model.
from keras.constraints import Constraint
from keras import backend as K
class WeightClip(Constraint):
'''Clips the weights incident to each hidden unit to be inside a range
'''
def __init__(self, c=2):
self.c = c
def __call__(self, p):
return K.clip(p, -self.c, self.c)
def get_config(self):
return {'name': self.__class__.__name__,
'c': self.c}
import numpy as np
from keras.models import Sequential
from keras.layers import Dense
model = Sequential()
model.add(Dense(30, input_dim=100, W_constraint = WeightClip(2)))
model.add(Dense(1))
model.compile(loss='mse', optimizer='rmsprop')
X = np.random.random((1000,100))
Y = np.random.random((1000,1))
model.fit(X,Y)
I have tested the running of the above code, but not the validity of the constraints. You can do so by getting the model weights after training using model.get_weights() or model.layers[idx].get_weights() and checking whether its abiding the constraints.
Note: The constrain is not added to all the model weights .. but just to the weights of the specific layer its used and also W_constraint adds constrain to W param and b_constraint to b (bias) param