i'm learning keras these days, and i met an error when using scikit-learn API.Here are something maybe useful:
ENVIRONMENT:
python:3.5.2
keras:1.0.5
scikit-learn:0.17.1
CODE
import pandas as pd
from keras.layers import Input, Dense
from keras.models import Model
from keras.models import Sequential
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.cross_validation import train_test_split
from sklearn.cross_validation import cross_val_score
from sqlalchemy import create_engine
from sklearn.cross_validation import KFold
def read_db():
"get prepared data from mysql."
con_str = "mysql+mysqldb://root:0000#localhost/nbse?charset=utf8"
engine = create_engine(con_str)
data = pd.read_sql_table('data_ml', engine)
return data
def nn_model():
"create a model."
model = Sequential()
model.add(Dense(output_dim=100, input_dim=105, activation='softplus'))
model.add(Dense(output_dim=1, input_dim=100, activation='softplus'))
model.compile(loss='mean_squared_error', optimizer='adam')
return model
data = read_db()
y = data.pop('PRICE').as_matrix()
x = data.as_matrix()
model = nn_model()
model = KerasRegressor(build_fn=model, nb_epoch=2)
model.fit(x,y) #something wrong here!
ERROR
Traceback (most recent call last):
File "C:/Users/Administrator/PycharmProjects/forecast/gridsearch.py", line 43, in <module>
model.fit(x,y)
File "D:\Program Files\Python35\lib\site-packages\keras\wrappers\scikit_learn.py", line 135, in fit
**self.filter_sk_params(self.build_fn.__call__))
TypeError: __call__() missing 1 required positional argument: 'x'
Process finished with exit code 1
the model works well without packaging with kerasRegressor, but i wanna using sk_learn's gridSearch after this, so i'm here for help. I tried but still have no idea.
something maybe helpful:
keras.warappers.scikit_learn.py
class BaseWrapper(object):
def __init__(self, build_fn=None, **sk_params):
self.build_fn = build_fn
self.sk_params = sk_params
self.check_params(sk_params)
def fit(self, X, y, **kwargs):
'''Construct a new model with build_fn and fit the model according
to the given training data.
# Arguments
X : array-like, shape `(n_samples, n_features)`
Training samples where n_samples in the number of samples
and n_features is the number of features.
y : array-like, shape `(n_samples,)` or `(n_samples, n_outputs)`
True labels for X.
kwargs: dictionary arguments
Legal arguments are the arguments of `Sequential.fit`
# Returns
history : object
details about the training history at each epoch.
'''
if self.build_fn is None:
self.model = self.__call__(**self.filter_sk_params(self.__call__))
elif not isinstance(self.build_fn, types.FunctionType):
self.model = self.build_fn(
**self.filter_sk_params(self.build_fn.__call__))
else:
self.model = self.build_fn(**self.filter_sk_params(self.build_fn))
loss_name = self.model.loss
if hasattr(loss_name, '__name__'):
loss_name = loss_name.__name__
if loss_name == 'categorical_crossentropy' and len(y.shape) != 2:
y = to_categorical(y)
fit_args = copy.deepcopy(self.filter_sk_params(Sequential.fit))
fit_args.update(kwargs)
history = self.model.fit(X, y, **fit_args)
return history
error occored in this line:
self.model = self.build_fn(
**self.filter_sk_params(self.build_fn.__call__))
self.build_fn here is keras.models.Sequential
models.py
class Sequential(Model):
def call(self, x, mask=None):
if not self.built:
self.build()
return self.model.call(x, mask)
So, what's that x mean and how to fix this error?
Thanks!
xiao, I ran into the same issue! Hopefully this helps:
Background and The Issue
The documentation for Keras states that, when implementing Wrappers for scikit-learn, there are two arguments. The first is the build function, which is a "callable function or class instance". Specifically, it states that:
build_fn should construct, compile and return a Keras model, which will then be used to fit/predict. One of the following three values could be passed to build_fn:
A function
An instance of a class that implements the call method
None. This means you implement a class that inherits from either KerasClassifier or KerasRegressor. The call method of the present class will then be treated as the default build_fn.
In your code, you create the model, and then pass the model as the value for the argument build_fn when creating the KerasRegressor wrapper:
model = nn_model()
model = KerasRegressor(build_fn=model, nb_epoch=2)
Herein lies the issue. Rather than passing your nn_model function as the build_fn, you pass an actual instance of the Keras Sequential model. For this reason, when fit() is called, it cannot find the call method, because it is not implemented in the class you returned.
Proposed Solution
What I did to make things work is pass the function as build_fn, rather than an actual model:
data = read_db()
y = data.pop('PRICE').as_matrix()
x = data.as_matrix()
# model = nn_model() # Don't do this!
# set build_fn equal to the nn_model function
model = KerasRegressor(build_fn=nn_model, nb_epoch=2) # note that you do not call the function!
model.fit(x,y) # fixed!
This is not the only solution (you could set build_fn to a class that implements the call method appropriately), but the one that worked for me. I hope it helps you!
User-defined keyword arguments passed to __init__() that is to say, all keyword arguments that were given to __init__() will be passed to model_build_fn directly. For example, calling KerasClassifier(myparam=10) will result in a model_build_fn(my_param=10)
here's an example:
class MyMultiOutputKerasRegressor(KerasRegressor):
# initializing
def __init__(self, **kwargs):
KerasRegressor.__init__(self, **kwargs)
# simpler fit method
def fit(self, X, y, **kwargs):
KerasRegressor.fit(self, X, [y]*3, **kwargs)
(...)
def get_quantile_reg_rpf_nn(layers_shape=[50,100,200,100,50], inDim= 4, outDim=1, act='relu'):
# do model stuff...
(...)
initialize the Keras regressor:
base_model = MyMultiOutputKerasRegressor(build_fn=get_quantile_reg_rpf_nn,
layers_shape=[50,100,200,100,50], inDim= 4,
outDim=1, act='relu', epochs=numEpochs,
batch_size=batch_size, verbose=0)
Related
I am trying to dump a model of MIL package and its metrics to the disk using pickle. I have no issues with the load function but when I try to extract the loaded contents (model, the metrics), I am getting an attribute error for the same.
I have added the code snippet for the metrics class here. You can also refer the trainer class from the author here
Is there any fix for this?
import numpy as np
import tensorflow as tf
from tensorflow.keras.backend import epsilon
from mil import metrics
class Metric(tf.keras.metrics.Metric):
""" Custom base class for implementing a metric,
each Metric subclass has to implement this methods. """
def __init__(self, name, **kwargs):
super(Metric, self).__init__(name=name, **kwargs)
def update_state_fn(self, y_true, y_pred, sample_weight=None):
""" Update the state of the metric
Parameters
----------
y_true : array-like containing the ground_truth
y_pred : array-like containing the network predictions
sample_weight : optional. weight more some predictions.
"""
raise NotImplementedError
def result(self):
""" Get the result of a metric """
raise NotImplementedError
def reset_states(self):
""" Reset the state of the metric """
raise NotImplementedError
class Specificity(Metric):
def __init__(self, name='specificity', **kwargs):
super(Specificity, self).__init__(name=name, **kwargs)
self.specificity = self.add_weight(name='specificity', initializer='zeros')
self.specificity.assign(np.nan)
self.tn = metrics.TrueNegatives()
self.fp = metrics.FalsePositives()
def update_state_fn(self, y_true, y_pred, sample_weight=None):
self.tn.update_state(y_true, y_pred, sample_weight)
self.fp.update_state(y_true, y_pred, sample_weight)
tn = self.tn.result()
fp = self.fp.result()
value = tf.where(tn + fp == 0, np.nan, tn / (tn + fp + epsilon()))
self.specificity.assign(value)
def result(self):
return self.specificity
def reset_states(self):
self.specificity.assign(np.nan)
self.tn.reset_states()
self.fp.reset_states()
I have added the entire code for calling this function: (from Github)
# importing dataset
from mil.data.datasets import musk1
# importing bag_representation
from mil.bag_representation import MILESMapping
# importing validation strategy
from mil.validators import LeaveOneOut
# importing final model, which in this case is the SVC classifier from sklearn
from mil.models import SVC
# importing trainer
from mil.trainer import Trainer
# importing preprocessing
from mil.preprocessing import StandarizerBagsList
# importing metrics, which in this case are from tf keras metrics
from mil.metrics import AUC
# loading dataset
(bags_train, y_train), (bags_test, y_test) = musk1.load()
# instantiate trainer
trainer = Trainer()
# preparing trainer
metrics = ['spec']
model = SVC(kernel='linear', C=1, class_weight='balanced')
pipeline = [('scale', StandarizerBagsList()), ('disc_mapping', MILESMapping())]
trainer.prepare(model, preprocess_pipeline=pipeline ,metrics=metrics)
# fitting trainer
valid = LeaveOneOut()
history = trainer.fit(bags_train, y_train, sample_weights='balanced', validation_strategy=valid, verbose=1)
# printing validation results for each fold
print(history['metrics_val'])
# predicting metrics for the test set
trainer.predict_metrics(bags_test, y_test)
import pickle
with open('model_pkl.pkl', 'wb') as files:
pickle.dump(trainer, files)
with open('/home/mylaptop/Res/model_pkl.pkl','rb') as sample_test:
abc = pickle.load(sample_test)
Stack trace:
AttributeError Traceback (most recent call last)
<ipython-input-84-9214b6d16ff4> in <module>
1 with open('/home/mylaptop/Res/model_pkl.pkl','rb') as sample_test:
----> 2 abc = pickle.load(sample_test)
AttributeError: 'Specificity' object has no attribute 'update_state_fn'
I'm using forward hooks to extract layer values from a pre-trained CNN and use them as features for my model. I also want to use torchscript for inference. The problem is that when I try to export any other method than 'forward' I get an error that 'forward' is missing for the registered forward hooks.
I have a minimal example:
from typing import Iterable, Callable, Tuple
from torch import Tensor, nn, ones, jit, empty
from torchvision.models import resnet50
class FeatureExtractor(nn.Module):
def __init__(self, model: nn.Module, layers: Iterable[str]):
super().__init__()
self.model = model
self.layers = layers
self.hooks = []
for layer_id in layers:
layer = dict([*self.model.named_modules()])[layer_id]
hook = layer.register_forward_hook(self.save_outputs_hook(layer_id))
self.hooks.append(hook)
def save_outputs_hook(self, layer_id: str) -> Callable:
def fn(_, input: Tuple[Tensor], output):
print('Hi')
return fn
def forward(self, x: Tensor):
return self.model(x)
#jit.export
def predict(self, x: Tensor):
return self.model(x)
if __name__ == '__main__':
dummy_input = ones(10, 3, 224, 224)
resnet_features = FeatureExtractor(resnet50(), layers=["layer4", "avgpool"])
features = resnet_features(dummy_input)
script = jit.trace(resnet_features, dummy_input)
This fails with:
RuntimeError: Couldn't find method: 'forward' on class: '__torch__.torch.nn.modules.container.___torch_mangle_141.Sequential (of Python compilation unit at: 0x7fdc5a676da8)'
If I deregister the hooks or export forward instead of predict this of course runs without problem. Ist there any way to make 'forward' mandatory to integrate for jit so it will be seen by the hooks?
Use jit.script(resnet_features) instead of jit.trace(resnet_features, dummy_input) and it should work.
I need to copy a keras model and there is no way that I know of which can be done unless the model is not a tf.keras.models.Model() subclass.
Note: The use copy.deepcopy() will work without giving any errors however it will result in another error whenever the copy is used.
Example:
import tensorflow as tf
class MyModel(tf.keras.Model):
def __init__(self):
super(MyModel, self).__init__()
self.dense1 = tf.keras.layers.Dense(4, activation=tf.nn.relu)
self.dense2 = tf.keras.layers.Dense(5, activation=tf.nn.softmax)
self.dropout = tf.keras.layers.Dropout(0.5)
def call(self, inputs, training=False):
x = self.dense1(inputs)
if training:
x = self.dropout(x, training=training)
return self.dense2(x)
if __name__ == '__main__':
model1 = MyModel()
model2 = tf.keras.models.clone_model(model1)
Results in:
Traceback (most recent call last):
File "/Users/emadboctor/Library/Application Support/JetBrains/PyCharm2020.3/scratches/scratch.py", line 600, in <module>
model2 = tf.keras.models.clone_model(model1)
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/models.py", line 430, in clone_model
return _clone_functional_model(
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/models.py", line 171, in _clone_functional_model
raise ValueError('Expected `model` argument '
ValueError: Expected `model` argument to be a functional `Model` instance, but got a subclass model instead.
Currently, we can't use tf.keras.models.clone_model for subclassed model API whereas we can for sequential and functional API. From doc,
model Instance of Model (could be a functional model or a Sequential model).
Here is a workaround for your need. It makes sense if we need to copy a trained model, where we can get some optimized parameters. So, the main task is we need to create a new model by copying an existing model. The most convenient way for now of this scenario is to get trained weight and set to the newly created model instances. Let first build a model, train it and then get and set weight matrices to the new model.
import tensorflow as tf
import numpy as np
class ModelSubClassing(tf.keras.Model):
def __init__(self, num_classes):
super(ModelSubClassing, self).__init__()
self.conv1 = tf.keras.layers.Conv2D(32, 3, strides=2, activation="relu")
self.gap = tf.keras.layers.GlobalAveragePooling2D()
self.dense = tf.keras.layers.Dense(num_classes)
def call(self, input_tensor, training=False):
# forward pass: block 1
x = self.conv1(input_tensor)
x = self.gap(x)
return self.dense(x)
def build_graph(self, raw_shape):
x = tf.keras.layers.Input(shape=raw_shape)
return tf.keras.Model(inputs=[x], outputs=self.call(x))
# compile
sub_classing_model = ModelSubClassing(10)
sub_classing_model.compile(
loss = tf.keras.losses.CategoricalCrossentropy(),
metrics = tf.keras.metrics.CategoricalAccuracy(),
optimizer = tf.keras.optimizers.Adam())
# plot for debug
tf.keras.utils.plot_model(
sub_classing_model.build_graph(x_train.shape[1:]),
show_shapes=False,
show_dtype=False,
show_layer_names=True,
expand_nested=False,
dpi=96,
)
DataSet
(x_train, y_train), (_, _) = tf.keras.datasets.mnist.load_data()
# train set / data
x_train = np.expand_dims(x_train, axis=-1)
x_train = x_train.astype('float32') / 255
# train set / target
y_train = tf.keras.utils.to_categorical(y_train, num_classes=10)
# fit
sub_classing_model.fit(x_train, y_train, batch_size=128, epochs=1)
# 469/469 [==============================] - 2s 2ms/step - loss: 8.2821
New Model / Copy
For the subclasses model, we have to initiate the class object.
sub_classing_model_copy = ModelSubClassing(10)
sub_classing_model_copy.build((x_train.shape))
sub_classing_model_copy.set_weights(sub_classing_model.get_weights()) # <- get and set wg
# plot for debug ; same as original plot
# but know, layer name is no longer same
# i.e. if, old: conv2d_40 , new/copy: conv2d_41
tf.keras.utils.plot_model(
sub_classing_model_copy.build_graph(x_train.shape[1:]),
show_shapes=False,
show_dtype=False,
show_layer_names=True,
expand_nested=False,
dpi=96,
)
def clones(module, N):
Creation of N identical layers.
:param module: module to clone
:param N: number of copies
:return: keras model of module copies
seqm=KM.Sequential()
for i in range(N):
m = copy.deepcopy(module)
m.name=m.name+str(i)
seqm.add(m)
return seqm
I'm trying to remove multiple layers form a pre-trained Keras model (ResNet50V2), but no matter what I do it's not working. I've read countless other questions on stack overflow, github issues, and forum posts related to this topic in the past month, and I still can't make it work... So I'll ask directly. What might I be doing wrong?
from ray.rllib.models.tf.tf_modelv2 import TFModelV2
from ray.rllib.utils.framework import try_import_tf
from ray.rllib.models import ModelCatalog
tf = try_import_tf()
def resnet_core(x):
x = tf.keras.applications.resnet_v2.preprocess_input(x)
resnet = tf.keras.applications.ResNet50V2(
include_top=False,
weights="imagenet",
)
remove_n = 130
for i in range(remove_n):
resnet._layers.pop()
print(len(resnet._layers))
s = tf.keras.models.Model(resnet.input, resnet._layers[-1].output, name='resnet-core')
for layer in s.layers:
print('adding layer',layer.name)
for layer in s.layers[:]:
layer.trainable = False
s.build(None)
return s(x)
class ImpalaCNN(TFModelV2):
def __init__(self, obs_space, action_space, num_outputs, model_config, name):
super().__init__(obs_space, action_space, num_outputs, model_config, name)
inputs = tf.keras.layers.Input(shape=obs_space.shape, name="observations")
x = inputs
x = resnet_core(x)
x = tf.keras.layers.Flatten()(x)
x = tf.keras.layers.ReLU()(x)
x = tf.keras.layers.Dense(units=256, activation="relu", name="hidden")(x)
logits = tf.keras.layers.Dense(units=num_outputs, name="pi")(x)
value = tf.keras.layers.Dense(units=1, name="vf")(x)
self.base_model = tf.keras.Model(inputs, [logits, value])
self.register_variables(self.base_model.variables)
def forward(self, input_dict, state, seq_lens):
obs = tf.cast(input_dict["obs"], tf.float32)
logits, self._value = self.base_model(obs)
return logits, state
def value_function(self):
return tf.reshape(self._value, [-1])
# Register model in ModelCatalog
ModelCatalog.register_custom_model("impala_cnn_tf", ImpalaCNN)
The error I'm getting is:
...
File "/Users/manu/anaconda3/envs/procgen/lib/python3.7/site-packages/ray/rllib/evaluation/rollout_worker.py", line 376, in __init__
self._build_policy_map(policy_dict, policy_config)
File "/Users/manu/anaconda3/envs/procgen/lib/python3.7/site-packages/ray/rllib/evaluation/rollout_worker.py", line 859, in _build_policy_map
policy_map[name] = cls(obs_space, act_space, merged_conf)
File "/Users/manu/anaconda3/envs/procgen/lib/python3.7/site-packages/ray/rllib/policy/tf_policy_template.py", line 143, in __init__
obs_include_prev_action_reward=obs_include_prev_action_reward)
File "/Users/manu/anaconda3/envs/procgen/lib/python3.7/site-packages/ray/rllib/policy/dynamic_tf_policy.py", line 163, in __init__
framework="tf")
File "/Users/manu/anaconda3/envs/procgen/lib/python3.7/site-packages/ray/rllib/models/catalog.py", line 317, in get_model_v2
registered))
ValueError: It looks like variables {<tf.Variable 'default_policy/
conv4_block4_1_conv/kernel:0' ... }
were created as part of <impala_cnn_tf.ImpalaCNN object at
0x19a8ccc90> but does not appear in model.variables()
({<tf.Variable 'default_policy/pi/
kernel:0' shape=(256, 15) dtype=float32> ...}). Did you forget to call
model.register_variables() on the variables in question?
The error seems to indicate some variables from the layers I'm trying to skip were not registered, but that's because I don't want to use them! Any ideas?
More context in case it helps:
If I set remove_n = 0 I don't see the error (but of course the whole ResNet50V2 is being used)
I'm a newbie with Keras and ML. This might be a very dumb question.
The reason I'm trying to remove many layers, not just the last one, is that I want the model to fit on a small GPU.
I'm trying to train the model using rllib for the aicrowd procgen competition (using the competition to learn Keras/RL)
Full code here: https://github.com/maraoz/neurips2020-procgen-starter-kit/blob/master/models/impala_cnn_tf.py#L55
Full error log here: https://pastebin.com/X0Dk7wdd
Thanks in advance!
Rather than popping off layers, you could try accessing the 130th layer from the last layer. Then, you can build a new model using the input of your original model and the output of this layer.
model = tf.keras.models.Model(resnet.input, resnet.layers[-130].output)
This will do essentially the same thing as what you tried but its much easier and safer since you aren't accessing any private properties of the model itself.
I'm using TensorFlow (1.1) high-level API Estimators to create my neural net. But I'm using it into a class and I have to call an instance of my class to generate the model of the neural network. (Here self.a)
class NeuralNetwork(object):
def __init__(self):
""" Create neural net """
regressor = tf.estimator.Estimator(model_fn=self.my_model_fn,
model_dir="/tmp/data")
// ...
def my_model_fn(self, features, labels, mode):
""" Generate neural net model """
self.a = a
predictions = ...
loss = ...
train_op = ...
return tf.estimator.EstimatorSpec(
mode=mode,
predictions=predictions,
loss=loss,
train_op=train_op)
But I get the error :
ValueError: model_fn [...] has following not expected args: ['self'].
I tried to remove the self for the args of my model but got another error TypeError: … got multiple values for keyword argument.
Is there any way to use these EstimatorSpec into a class ?
It looks like the Estimator's argument checking is a bit overzealous. As a workaround, you can wrap the member-function model_fn in a lambda like so:
import tensorflow as tf
class ModelClass(object):
def __init__(self):
self._constant = 2.
self.regressor = tf.estimator.Estimator(
model_fn=lambda features, labels, mode: self._model_fn(
features, labels, mode))
def _model_fn(self, features, labels, mode):
loss = tf.constant(self._constant)
train_op = tf.no_op()
return tf.estimator.EstimatorSpec(
mode=mode,
loss=loss,
train_op=train_op)
ModelClass()
However, this is rather annoying. Would you mind filing a feature request on Github to relax this argument checking for member functions?
Update: Should be fixed in TensorFlow 1.3+. Thanks, Yuan!