Below is how I trained an XGBClassifier and saved it:
import pickle
from xgboost import XGBClassifier
# train
model = XGBClassifier()
model.fit(X, y)
# export
pickle.dump(model, open('model.pickle', 'wb'))
This is how I loaded the model and made predictions
loaded_model = pickle.load(open('model.pickle', 'rb'))
y_pred = loaded_model.predict(X)
The model predictions are OK if the model was loaded from within the same python process where the training was performed, but the predictions are not OK (random) if the model was loaded from a different python process than the one used for training.
Note, I've the same problem if model.save_model and model.load_model were used instead of pickle.
The simple checks I did shows the model was saved and loaded properly; the dumps of model._Booster (acquired via model._Booster.dump_model(some_file)) and loaded_model._Booster are identical.
Python version: 3.7.5
xgboost version: tried both 0.80 and 0.90
Any suggestion is appreciated.
In my case, i had changed column order while predicting which led to different performance. The column order for training data and prediction data Must be same
Related
I have trained a DistilBERT classification model using huggingface and the the model seems to be working well, with a loss of around 0.3 after testing the best model after training with the code:
trainer.evaluate()
However, upon a new run of trying to load the model and attempting to use the trainer class to evaluate the loss again on the same dataset, the reported loss is horrible and seemingly random, often 7+ but not always the same. This is done largely the same as done for training and uses the same seed:
dataset = DatasetModule(
datasets=config.datasets,
tokenizer=tokenizer_name,
seed=42,)
model = DistilBertForTokenClassification.from_pretrained(
model_path,
num_labels=dataset.label_number,
id2label=dataset.id2tag,
label2id=dataset.tag2id,)
trainer: Trainer = Trainer(
model=model,
train_dataset=dataset.train_dataset,
eval_dataset=dataset.valid_dataset,)
metrics = trainer.evaluate(dataset.valid_dataset)
print(metrics)
At the same time, the performance of model in other metrics, such as f1 after loading the model in the same way are about what I would expect. I suspect and may test using a pickle to save the trainer and see if that gives the expected results, though not ideal.
So my question is why is this happening? Is it a bug? Am I going about the process wrong, and if so how can I go about easily loading a model and testing a dataset on it to get a loss?
Thanks!
I trained GradientBoostingClassifier and saved like this
pickle.dump(clf,open("C:/Users/R2D2/Documents/Python/Supervised/TrainedModel.dat",'wb'))
and loaded the saved model like this
loaded_model = pickle.load(open("C:/Users/R2D2/Documents/Python/Supervised/TrainedModel.dat",'rb'))
I get %94 accuracy on test data and only %50 on training data but after training these results were both %94 for training and test data. How can it change after saved?
This look similar but doesn't help
I have trained a model and saved it in .h5 format. when I used the same file, train the model, saved it and load it it gives good accuracy like 84%. But when I used code to only load the model and use it to evaluate on test data the accuracy is very low like 1.004%. I am unable to understand the reason behind it.
history=model.fit([x1, x2], y,
epochs=1,
batch_size=256,
shuffle = False,
verbose = 1,
validation_split=0.2,
class_weight=custom_weight_dict,
callbacks=[early_stopping_cb]
)
model.save('my_models/model_context1.h5')
Then I used following code to load the saved model and evaluate it on test data.
model=load_model("my_models/model_context1.h5")
print(model.summary())
score = model.evaluate([x1_test, x2_test], y_test)
print("Accuracy after loading Model:", score[1]*100)
What's the wrong with the way I am trying to evaluate a trained model accuracy on test data? and using the saved .h5 file.
I suspect it is due to the fact that if you used model.save by default it saves it in what they call the SavedModel format. This is different than the .h5 format.To save it in the .h5 format you pass save_format='h5 into model.save or pass it a filename that ends in .h5.
Documentation is here..See the example. The SavedModel format is the preferred format so I would suggest you use that since eventually they will depreciate the .h5 format.
I am using tensorflow version '2.0.0' and keras version '2.3.0' to develop the model. Here's how I saved the model:
seed = 1234
random.seed(seed)
np.random.seed(seed)
tf.compat.v1.random.set_random_seed(seed)
I then save the entire model as instructed here:
model.save('some_model_name.h5')
I am getting an accuracy of about 95% during training. When I load the model from a different python session, like:
# Recreate the exact same model
new_model = load_model('some_model_name.h5', custom_objects={'SeqSelfAttention': SeqSelfAttention})
score = new_model.evaluate([x_img_train, x_txt_train], y_train, verbose=2)
print("%s: %.2f%%" % (new_model.metrics_names[1], score[1]*100))
The accuracy now is about 4%. Please note that I have batch norm and dropout layers. How can I make the predictions of my model consistent across different sessions?
Firstly, I have downgraded the TensorFlow version to 1.13.1, owing to stability issues of 2.0.0.
Secondly, I had to ensure a few things before I could achieve some level of reproducibility:
Use Adagrad optimizer instead of Adam gave me performance comparable to the train session. When every time I loaded the session, it was giving me a high variance in the predictions (for Adam)
Loading architecture from json and loading model weights subsequently gave me different results as compared to saving and loading weights only. The former approach seemed to produce comparable performance (to training)
Using tf.session to train and saving it and reloading the tf.session in a new python session did the trick.
There is no variation in the results with or without dropouts or Batch norm.
Please note that following these steps gave me some level of consistency although it's not 100% reproducible. If you're facing a similar issue, perhaps these insights could help.
After loading the model in a new kernel instance, make sure to config losses and metrics again with .compile() in the same way you did before saving.
For example:
old_model = tf.keras.Sequential([ ... ])
old_model.compile(loss = 'mean_squared_error', optimizer = 'sgd', metrics = ['accuracy'])
old_model.fit(train_ds, validation_data=valid_ds, epochs=3)
old_model.evaluate(test_ds)
old_model.save('some_model_name.h5')
Then in the new kernel:
from tensorflow.keras.models import load_model
new_model = load_model("some_model_name.h5")
new_model.compile(loss = 'mean_squared_error', optimizer = 'sgd', metrics = ['accuracy'])
new_model.evaluate(test_ds) # should be the same now
I have trained and tested a Feed Forward Neural Network using Keras in Python with a dataset. But each time, in order to recognize a new test set with external data (external since the data are not included within the dataset), I have to re-train the Feed Forward Neural Network to compute the test set. For instance each time I have to do:
model.fit (data, output_data)
prediction=model.predict_classes(new_test)
print "Prediction : " prediction
Obtaining correct output:
Prediction: [1 2 3 4 5 1 2 3 1 2 3]
Acc: 100%
Now I would test a new test set, namely "new_test2.csv" without re-training again, just using what the network has learned. I am also thinking about a sort of real time recognition.
How I should do that?
Thanks in advance
With a well trained model you can make predictions on any new data. You don´t have to retrain anything because (hopefully) your model can generalize it´s learning to unseen data and will achieve comparable accuracy.
Just feed in the data from "new_test2.csv" to your predict function:
prediction=model.predict_classes(content_of_new_test2)
Obviously you need data of the same type and classes. In addition to that you need to apply any transformations to the new data in the same way you may have transformed the data you trained your model on.
If you want realtime predictions you could setup an API with Flask:
http://flask.pocoo.org/
Regarding terminology and correct method of training:
You train on a training set (e.g. 70% of all the data you have).
You validate your training with a validation set (e.g. 15% of your data). You use the accuracy and loss values from your training to tune your hyperparameters.
You then evaluate your models final performance by predicting data from your test set (again 15% of your data). That has to be data, your network hasn´t seen before at all and hasn´t been used by you to optimize training parameters.
After that you can predict on production data.
If you want to save your trained model use this (taken from Keras documentation):
from keras.models import load_model
model.save('my_model.h5') # creates a HDF5 file 'my_model.h5'
del model # deletes the existing model
# returns a compiled model
# identical to the previous one
model = load_model('my_model.h5')
https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model
In your training file, you can save the model using
model.save('my_model.h5')
Later, whenever you want to test, you can load it with
from keras.models import load_model
model = load_model('my_model.h5')
Then you can call model.predict and whatnot.