Develop Reference

Large reported loss after loading a fine-tuned HuggingFace model and using trainer.evaluate()

I have trained a DistilBERT classification model using huggingface and the the model seems to be working well, with a loss of around 0.3 after testing the best model after training with the code:
trainer.evaluate()
However, upon a new run of trying to load the model and attempting to use the trainer class to evaluate the loss again on the same dataset, the reported loss is horrible and seemingly random, often 7+ but not always the same. This is done largely the same as done for training and uses the same seed:
dataset = DatasetModule(
datasets=config.datasets,
tokenizer=tokenizer_name,
seed=42,)
model = DistilBertForTokenClassification.from_pretrained(
model_path,
num_labels=dataset.label_number,
id2label=dataset.id2tag,
label2id=dataset.tag2id,)
trainer: Trainer = Trainer(
model=model,
train_dataset=dataset.train_dataset,
eval_dataset=dataset.valid_dataset,)
metrics = trainer.evaluate(dataset.valid_dataset)
print(metrics)
At the same time, the performance of model in other metrics, such as f1 after loading the model in the same way are about what I would expect. I suspect and may test using a pickle to save the trainer and see if that gives the expected results, though not ideal.
So my question is why is this happening? Is it a bug? Am I going about the process wrong, and if so how can I go about easily loading a model and testing a dataset on it to get a loss?
Thanks!

load_model to evaulate a test data on already trained model gives very low accuracy

I have trained a model and saved it in .h5 format. when I used the same file, train the model, saved it and load it it gives good accuracy like 84%. But when I used code to only load the model and use it to evaluate on test data the accuracy is very low like 1.004%. I am unable to understand the reason behind it.
history=model.fit([x1, x2], y,
epochs=1,
batch_size=256,
shuffle = False,
verbose = 1,
validation_split=0.2,
class_weight=custom_weight_dict,
callbacks=[early_stopping_cb]
)
model.save('my_models/model_context1.h5')
Then I used following code to load the saved model and evaluate it on test data.
model=load_model("my_models/model_context1.h5")
print(model.summary())
score = model.evaluate([x1_test, x2_test], y_test)
print("Accuracy after loading Model:", score[1]*100)
What's the wrong with the way I am trying to evaluate a trained model accuracy on test data? and using the saved .h5 file.

I suspect it is due to the fact that if you used model.save by default it saves it in what they call the SavedModel format. This is different than the .h5 format.To save it in the .h5 format you pass save_format='h5 into model.save or pass it a filename that ends in .h5.
Documentation is here..See the example. The SavedModel format is the preferred format so I would suggest you use that since eventually they will depreciate the .h5 format.

XGBClassifier: Bad predictions after training, saving and loading a model

Below is how I trained an XGBClassifier and saved it:
import pickle
from xgboost import XGBClassifier
# train
model = XGBClassifier()
model.fit(X, y)
# export
pickle.dump(model, open('model.pickle', 'wb'))
This is how I loaded the model and made predictions
loaded_model = pickle.load(open('model.pickle', 'rb'))
y_pred = loaded_model.predict(X)
The model predictions are OK if the model was loaded from within the same python process where the training was performed, but the predictions are not OK (random) if the model was loaded from a different python process than the one used for training.
Note, I've the same problem if model.save_model and model.load_model were used instead of pickle.
The simple checks I did shows the model was saved and loaded properly; the dumps of model._Booster (acquired via model._Booster.dump_model(some_file)) and loaded_model._Booster are identical.
Python version: 3.7.5
xgboost version: tried both 0.80 and 0.90
Any suggestion is appreciated.

In my case, i had changed column order while predicting which led to different performance. The column order for training data and prediction data Must be same

How can I compute categorical accuracy after I predict the results in Keras model?

I have built a Keras model and while training, the categorical accuracy metric reaches 0.78.
However after training the model, when I predict the output of same training data when I run the following code:
predicted_labels = model.predict(input_data)
acc = sklearn.metrics.accuracy_score(true_labels, predicted_labels)
the accuracy is 0.39.
To summarize, I don't get same accuracy result for Keras and Sklearn.

There are many ways of measuring accuracy, and sklearn might not be using the same as Keras.
You may take your compiled model and lossAndMetrics = model.evaluate(input_data, true_labels) to see loss and metrics that are surely the same you used for training.
PS: it's not rare to have a bad result for test/validation data if your model is overfitting.

Test neural network using Keras Python

I have trained and tested a Feed Forward Neural Network using Keras in Python with a dataset. But each time, in order to recognize a new test set with external data (external since the data are not included within the dataset), I have to re-train the Feed Forward Neural Network to compute the test set. For instance each time I have to do:
model.fit (data, output_data)
prediction=model.predict_classes(new_test)
print "Prediction : " prediction
Obtaining correct output:
Prediction: [1 2 3 4 5 1 2 3 1 2 3]
Acc: 100%
Now I would test a new test set, namely "new_test2.csv" without re-training again, just using what the network has learned. I am also thinking about a sort of real time recognition.
How I should do that?
Thanks in advance

With a well trained model you can make predictions on any new data. You don´t have to retrain anything because (hopefully) your model can generalize it´s learning to unseen data and will achieve comparable accuracy.
Just feed in the data from "new_test2.csv" to your predict function:
prediction=model.predict_classes(content_of_new_test2)
Obviously you need data of the same type and classes. In addition to that you need to apply any transformations to the new data in the same way you may have transformed the data you trained your model on.
If you want realtime predictions you could setup an API with Flask:
http://flask.pocoo.org/
Regarding terminology and correct method of training:
You train on a training set (e.g. 70% of all the data you have).
You validate your training with a validation set (e.g. 15% of your data). You use the accuracy and loss values from your training to tune your hyperparameters.
You then evaluate your models final performance by predicting data from your test set (again 15% of your data). That has to be data, your network hasn´t seen before at all and hasn´t been used by you to optimize training parameters.
After that you can predict on production data.
If you want to save your trained model use this (taken from Keras documentation):
from keras.models import load_model
model.save('my_model.h5') # creates a HDF5 file 'my_model.h5'
del model # deletes the existing model
# returns a compiled model
# identical to the previous one
model = load_model('my_model.h5')
https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model

In your training file, you can save the model using
model.save('my_model.h5')
Later, whenever you want to test, you can load it with
from keras.models import load_model
model = load_model('my_model.h5')
Then you can call model.predict and whatnot.