I am training multiple models (on Google Colab) and saving them using joblib. I am then downloading these files and loading them using joblib, but I am getting the following error;
Unable to restore custom object of type _tf_keras_metric
I am compiling the models using metrics=['accuracy', specificity, tf.keras.metrics.Precision()], where specificity is a custom metric function from https://www.sabinasz.net/unbalanced-classes-machine-learning/.
mobilenet_model = MobileNetV3Small(include_top=True, weights=None, classes=2, input_shape=(100, 100, 3))
mobilenet_model.compile(loss='binary_crossentropy', metrics=['accuracy', specificity, tf.keras.metrics.Precision()])
trained_model = mobilenet_model.fit(X_train, y_train)
joblib.dump(trained_model, 'mobilenet_model.joblib')
I tried to load them using keras.models.load_model but I get a Error opening file (File signature not found) which according to Error opening file in H5PY (File signature not found) means the file is corrupt (I imagine that the way joblib saves models is not compatible with keras's save_model function).
I tried using keras.models.save_model to save a file and passing in custom_objects={'specificity': specificity, 'precision': tf.keras.metrics.Precision()}, but the model seems to be saved without the pretrained weights because when I evaluate it, it only has 0.5 accuracy, when it was getting 0.9 when I trained it.
So what can I do?
Related
I'm having a problem loading a Tensorflow model I downloaded. Based on the error, it sounds like incompatibility issue with model from older Tensorflow? I tried to load it on Tensorflow 2.4, 2.3 with no success. It throws a completely different error.
Inside model folder, I see checkpoint, saved_model, pipeline.config.
Inside model/saved_model folder, I see variables and saved_model.pb.
If I try to load using tf.keras.models.load_model("model/saved_model"), I get this.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
ValueError: Unable to create a Keras model from SavedModel at design-system-detector/rico/models/mobilenetv2-50k/saved-model/saved_model. This SavedModel was exported with `tf.saved_model.save`, and lacks the Keras metadata file. Please save your Keras model by calling `model.save` or `tf.keras.models.save_model`.
Note that you can still load this SavedModel with `tf.saved_model.load`.
If I load with tf.saved_model.load, it just returns _UserObject, and I can't run model.predict.
AttributeError: '_UserObject' object has no attribute 'predict'
What's the best way to use this model to predict?
I am Trying to deploy a tensorflow keras model using amazon sagemaker. Process finishes successfully, yet i get different prediction results when predicted directly using keras and when calling sagemaker endpoint to make predictions.
I used these steps in deploying the model to sagemaker.
Check the following example.
data = np.random.randn(1, 150, 150, 3)
# predict using amazon sagemaker
sagemaker_predict = uncompiled_predictor.predict(data)
print(sagemaker_predict)
#predict same using keras
val = model.predict(data)
print(val)
>>{'predictions': [[0.491645753]]}
[[0.]]
Is this something supposed to happen? For my knowledge it should be the same. For some reason data gets corrupted or sagemaker weights get reinitialized. Any ideas?
Not suppose to happen.
See what you get if you deloy the model directly to TensorFlow serving (which is what the SageMaker inference container wraps).
To experiment faster you can work with SageMaker inference container in local mode, so you can start/stop an endopint in seconds.
Finally found a solution. It seems to be a problem with .h5 (HDF5) weights file, for some reason sagemaker seems not to extract weight from .h5. Therefore changed the weights file to TensorFlow SavedModel format
As for tensorflow keras save and serialize documentation
There are two formats you can use to save an entire model to disk: the TensorFlow SavedModel format, and the older Keras H5 format. The recommended format is SavedModel. It is the default when you use model.save().
You can switch to the H5 format by:
Passing save_format='h5' to save().
Passing a filename that ends in .h5 or .keras to save()
So instead of saving weights as
model.save("my_model.h5")
save as
model.save("my_model")
And load the same weights as
keras.models.load_model("my_model")
This will save your file in TensorFlow SavedModel format which you can follow in the above documentation to load and deploy to sagemaker.
I've got some keras models I saved to disk using model.save('path/to/location').
When I go to load these models in a Colab notebook with model = keras.models.load_model('path/to/location'), they load fine and work entirely as expected.
When I'm VSCode however, and try to load the model with the same methodology in the iPython/Jupyter interactive window, I get the following error message:
OSError: Unable to open file (file read failed: time = Sat Jul 24 20:19:55 2021
, filename = 'path/to/location', file descriptor = 58, errno = 21, error message = 'Is a directory'
The file is meant to be a directory according to the Keras documentation, and I confirm that it contains what the documentation says it should contain:
assets saved_model.pb variables
So why isn't it working in my interactive window in VSCode, but does work in Colab?
I've also tried taking the Colab notebook, opening it with Anaconda/Jupyter notebook and I am able to replicate the error.
Is there a fundamental problem with loading a Keras model outside of Colab?
I've checked and rechecked the following:
All packages up to date in Conda
Directory typed correctly
Tried both relative and absolute paths, same error message
Haven't done anything to the files or folders since saving the models
Update 26 Jul # 11:30
By way of a minimal implementation, please see below. Hopefully this sheds some more light on the problem.
Very simple example:
import keras
# setting keras model to sequential mode
model = Sequential()
# layer 1
model.add(Dense(10, activation='tanh', input_shape=(30, 1)))
# layer 2
model.add(Flatten())
# layer 3
model.add(Dense(1))
optimizer = Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, clipnorm=1.0)
# compile the model
model.compile(optimizer, loss='mean_squared_error')
# training the model
model.fit(X_train, y_train, epochs=50, batch_size=32)
# making some predictions
y_pred = model.predict(X_test)
# evaluate the output
evaluate(y_test, y_pred)
# saving the model
keras.models.save_model(model, save_path)
The above code works in both Colab and Jupyter, the model compiles, predicts and is evaluated successfully.
Then to load the model:
loaded_model = keras.models.load_model(save_path)
In Colab, this works fine, the model works as expected. In Jupyter when the above line is run, the OSError is generated.
Also worth noting, keras.models.save_model(model, save_path) and model.save(save_path) were both tried. Again, in Colab both methods save and load successfully, in Jupyter both methods saved successfully, generating the same save files as in Colab, but using either method still fails to load in Jupyter.
If the model is saved with either method in Jupyter, they can both be loaded successfully in Colab as well.
Is it perhaps something to do with out of date packages in my Conda environment? I've updated both Keras and Tensorflow in Conda to the same versions running in Colab and the error persists.
If I try to use model = tf.saved_model.load(path_to_dir) in both Colab or Jupyter, the model loads but the Keras calls to methods like .predict() and others fails, I suspect this is due to loading in tf rather than Keras.
I was trying to implement Facial Recognition using the pretrained models from Keras-Openface Project and amazingly explained and implimented by Martin Krasser here
The OpenFace project provides pre-trained models that were trained with the public face recognition datasets FaceScrub and CASIA-WebFace. The Keras-OpenFace project converted the weights of the pre-trained nn4.small2.v1 model to CSV files which were then converted here to a binary format that can be loaded by Keras with load_weights
The Code is simply :
nn4_small2_pretrained = create_model()
nn4_small2_pretrained.load_weights('weights/nn4.small2.v1.h5')
Before going to the error to give more insight on what's happening in create_model(), you can go through the code here
Now I need nn4_small2_pretrained in my predict.py file (It is initially a part of train.py to train my custom images), but if I do
from train import nn4_small2_pretrained
Or write the code
nn4_small2_pretrained = create_model()
nn4_small2_pretrained.load_weights('weights/nn4.small2.v1.h5')
all over again, then the prediction file takes a lot of time to compile as it goes through the whole process again. So to resolve this, I tried to dump the model in a pickle file like so
# Save the nn4 pretrained model to pickle file
f = open('pretrained_model.pickle', 'wb')
pickle.dump(nn4_small2_pretrained, f)
When I run the code it gives me this error
File "train.py", line 24, in <module>
pickle.dump(nn4_small2_pretrained, f)
TypeError: can't pickle _thread.lock objects
I have started with Deel Learning Models and Pickle recently and I cannot figure out what's wrong.
Any help would be much appreciated.
Thanks.
I see that the create_model() creates an instance of Keras Model. If it is a Keras Model then you can save the model using model.save(filepath).
Refer this link for other options.
I'm looking to run a basic fully-connected neural network for the MNIST dataset with the C++ API v1.2 from Tensorflow. I have trained the model and exported it using tf.train.Saver() in Python. This gave me a checkpoint file, a data file, an index file and a meta file.
I know that the data file contains the saved variables while the meta file contains the graph from using Tensorboard on a previous project.
However, I am not sure what is the recommended way to load those files
and run the trained model in a C++ environment in v1.2, since all the
tutorials and questions I've found are for older versions which differ
substantially.
I've found that tensorflow::ops::Restore should be the method to do such a thing, but I know that inference in Tensorflow isn't well supported, as such I am not certain what parameters should I give it in order to receive the trained model that I can just put into a session->Run() and receive an accuracy statement when fed test data.