Develop Reference

Print results more meaningfully in Python code?

I'm performing SVM on a dataset with 10 folds for different values of C and gamma
from sklearn.datasets import load_digits, load_iris, load_breast_cancer, load_wine
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.utils import shuffle
from sklearn import preprocessing
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import accuracy_score, zero_one_loss, confusion_matrix
import pandas as pd
import numpy as np
z = pd.read_csv('/home/user/iris.csv', header=0)
X = z.iloc[:, :-1]
y = z.iloc[:, -1:]
X = np.array(X)
y = np.array(y)
# Performing standard scaling
scaler = preprocessing.MinMaxScaler()
X_scaled = scaler.fit_transform(X)
c = [0.1, 0.5]
gamma_values = [1e-1, 1]
for z in c:
for v in gamma_values:
# Defining the SVM with 'rbf' kernel
svc = SVC(kernel='rbf',C=z, gamma=v, random_state=50)
skf = StratifiedKFold(n_splits=10, shuffle=True)
acc_score = []
#skf.get_n_splits(X, y)
for train_index, test_index in skf.split(X, y):
X_train, X_test = X_scaled[train_index], X_scaled[test_index]
y_train, y_test = y[train_index], y[test_index]
# Training the model
svc.fit(X_train, np.ravel(y_train))
# Prediction on test dataste
y_pred = svc.predict(X_test)
# Obtaining the accuracy scores of the model
score = accuracy_score(y_test, y_pred)
acc_score.append(score)
print(np.array(acc_score))
#print the accuracy score for each of the C values
print('Mean accuracy score: %0.3f' % np.array(acc_score).mean())
which results in output like below
[0.52 0.6 0.49 0.6 0.55 0.6 0.5 0.51 0.63 0.54]
Mean accuracy score: 0.554
[0.51 0.45 0.54 0.42 0.53 0.45 0.52 0.48 0.5 0.39]
Mean accuracy score: 0.479
[0.73 0.76 0.7 0.64 0.61 0.68 0.71 0.61 0.71 0.71]
Mean accuracy score: 0.686
[0.76 0.6 0.66 0.61 0.67 0.66 0.69 0.74 0.63 0.65]
Mean accuracy score: 0.667
However, I want to print the results more meaningfully like below :
[0.52 0.6 0.49 0.6 0.55 0.6 0.5 0.51 0.63 0.54]
Mean accuracy score for (C=0.1,gamma=0.1): 0.554
[0.51 0.45 0.54 0.42 0.53 0.45 0.52 0.48 0.5 0.39]
Mean accuracy score (C=0.1, gamma = 1): 0.479
[0.73 0.76 0.7 0.64 0.61 0.68 0.71 0.61 0.71 0.71]
Mean accuracy score (C=0.5, gamma = 0.1): 0.686
[0.76 0.6 0.66 0.61 0.67 0.66 0.69 0.74 0.63 0.65]
Mean accuracy score (C=0.5, gamma = 1): 0.667
How can I print the results more meaningfully in the existing code ?

try this:
# (1)
print('Mean accuracy score (C=%0.1f, gamma=%0.1f): %0.3f' % (z, v, np.array(acc_score).mean()))
# (2)
print("Mean accuracy score (C={}, gamma={}): {}".format(z, v, np.array(acc_score).mean()))
# (3)
print("Mean accuracy score (C="+str(z)+", gamma="+str(v)+"): "+str(np.array(acc_score).mean()))
output:
Mean accuracy score (C=0.1, gamma=0.1): 0.554

You can use fancy output formatting:
print(f'Mean accuracy score (C={z:.1f}, gamma={v:.1f}): {np.array(acc_score).mean()}')

Metrics F1 warning zero division

I want to calculate the F1 score of my models. But I receive a warning and get a 0.0 F1-score and I don't know what to do.
here is the source code:
def model_evaluation(dict):
for key,value in dict.items():
classifier = Pipeline([('tfidf', TfidfVectorizer()),
('clf', value),
])
classifier.fit(X_train, y_train)
predictions = classifier.predict(X_test)
print("Accuracy Score of" , key , ": ", metrics.accuracy_score(y_test,predictions))
print(metrics.classification_report(y_test,predictions))
print(metrics.f1_score(y_test, predictions, average="weighted", labels=np.unique(predictions), zero_division=0))
print("---------------","\n")
dlist = { "KNeighborsClassifier": KNeighborsClassifier(3),"LinearSVC":
LinearSVC(), "MultinomialNB": MultinomialNB(), "RandomForest": RandomForestClassifier(max_depth=5, n_estimators=100)}
model_evaluation(dlist)
And here is the result:
Accuracy Score of KNeighborsClassifier : 0.75
precision recall f1-score support
not positive 0.71 0.77 0.74 13
positive 0.79 0.73 0.76 15
accuracy 0.75 28
macro avg 0.75 0.75 0.75 28
weighted avg 0.75 0.75 0.75 28
0.7503192848020434
---------------
Accuracy Score of LinearSVC : 0.8928571428571429
precision recall f1-score support
not positive 1.00 0.77 0.87 13
positive 0.83 1.00 0.91 15
accuracy 0.89 28
macro avg 0.92 0.88 0.89 28
weighted avg 0.91 0.89 0.89 28
0.8907396950875212
---------------
Accuracy Score of MultinomialNB : 0.5357142857142857
precision recall f1-score support
not positive 0.00 0.00 0.00 13
positive 0.54 1.00 0.70 15
accuracy 0.54 28
macro avg 0.27 0.50 0.35 28
weighted avg 0.29 0.54 0.37 28
0.6976744186046512
---------------
C:\Users\Cey\anaconda3\lib\site-packages\sklearn\metrics\_classification.py:1272: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
_warn_prf(average, modifier, msg_start, len(result))
Accuracy Score of RandomForest : 0.5714285714285714
precision recall f1-score support
not positive 1.00 0.08 0.14 13
positive 0.56 1.00 0.71 15
accuracy 0.57 28
macro avg 0.78 0.54 0.43 28
weighted avg 0.76 0.57 0.45 28
0.44897959183673475
---------------
Can someone tell me what to do? I only receive this message when using the "MultinomialNB()" classifier
Second:
When extending the dictionary by using the Gausian classifier (GaussianNB()) I receive this error message:
TypeError: A sparse matrix was passed, but dense data is required. Use X.toarray() to convert to a dense numpy array.
What should I do here ?

Together with UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples (main credits go there) and #yatu's answer, I could at least find a workaround for the warning:
UndefinedMetricWarning: Precision is ill-defined and being set to 0.0
due to no predicted samples. Use zero_division parameter to control
this behavior. _warn_prf(average, modifier, msg_start, len(result))
Quote from sklearn.metrics.f1_score in the Notes at the bottom:
When true positive + false positive == 0, precision is undefined. When
true positive + false negative == 0, recall is undefined. In such
cases, by default the metric will be set to 0, as will f-score, and
UndefinedMetricWarning will be raised. This behavior can be modified
with zero_division.
Thus, you cannot avoid this error if your data does not output a difference between true positives and false positives.
That being said, you can only suppress the warning at least, adding zero_division=0 to the functions mentioned in the quote. In either case, set to 0 or 1, you will get a 0 value as the return anyway.
precision = precision_score(y_test, y_pred, zero_division=0)
print('Precision score: {0:0.2f}'.format(precision))
recall = recall_score(y_test, y_pred, zero_division=0)
print('Recall score: {0:0.2f}'.format(recall))
f1 = f1_score(y_test, y_pred, zero_division=0)
print('f1 score: {0:0.2f}'.format(recall))

Can someone tell me what to do? I only receive this message when using the "MultinomialNB()" classifier
The first error seems to be indicating that a specific label is not predicted when using the MultinomialNB, which results in an undefined f-score, or ill-defined, since the missing values are set to 0. This is explained here
When extending the dictionary by using the Gausian classifier (GaussianNB()) I receive this error message:
TypeError: A sparse matrix was passed, but dense data is required. Use X.toarray() to convert to a dense numpy array.
As per this question, the error is quite explicit, the issue is that TfidfVectorizer is returning a sparse matrix, which cannot be used as input for the GaussianNB. So the way I see it, you either avoid using the GaussianNB, or you add an intermediate transformer to turn the sparse array to dense, which I wouldn't advise being the result of a tf-idf vectorization.

Sklearn - plotting classification report gives a different output than basic avg?

I wanted to leverage this answer How to plot scikit learn classification report? turning an sklearn classification report into a heatmap.
It's all working with their sample report, however my classification report looks slightly different and is thus screwing up the functions.
Their report (notice the avg / total):
sampleClassificationReport =
precision recall f1-score support
Acacia 0.62 1.00 0.76 66
Blossom 0.93 0.93 0.93 40
Camellia 0.59 0.97 0.73 67
Daisy 0.47 0.92 0.62 272
Echium 1.00 0.16 0.28 413
avg / total 0.77 0.57 0.49 858
My report with metrics.classification_report(valid_y, y_pred) :
precision recall f1-score support
0 1.00 0.18 0.31 11
1 0.00 0.00 0.00 14
2 0.00 0.00 0.00 19
3 0.50 0.77 0.61 66
4 0.39 0.64 0.49 47
5 0.00 0.00 0.00 23
accuracy 0.46 180
macro avg 0.32 0.27 0.23 180
weighted avg 0.35 0.46 0.37 180
The issue, from the selected answer in the heatmap link, is here:
for line in lines[2 : (len(lines) - 2)]:
t = line.strip().split()
if len(t) < 2: continue
classes.append(t[0])
v = [float(x) for x in t[1: len(t) - 1]]
support.append(int(t[-1]))
class_names.append(t[0])
print(v)
plotMat.append(v)
Because I get the error:
ValueError: could not convert string to float: 'avg'
So the problem truly is how my classification report is being outputted. What can I change here to match the sample?
EDIT: what Ive tried:
df = pd.DataFrame(metrics.classification_report(valid_y, y_pred)).T
df['support'] = df.support.apply(int)
df.style.background_gradient(cmap='viridis',
subset=pd.IndexSlice['0':'9', :'f1-score'])
Error:
ValueError: DataFrame constructor not properly called!

With the advent of output_dict param in classification_report, there is no hassle for parsing the report. You can directly use the output of classification report to be read as pd.DataFrame. Then, you could use the pd.Style option to render the heat map.
Example:
from sklearn.metrics import classification_report
import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, GridSearchCV
X, y = make_classification(n_samples=1000, n_features=30,
n_informative=12,
n_clusters_per_class=1, n_classes=10,
class_sep=2.0, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, stratify=y)
clf = LogisticRegression(max_iter=1000, random_state=42).fit(X_train, y_train)
df = pd.DataFrame(classification_report(clf.predict(X_test),
y_test, digits=2,
output_dict=True)).T
df['support'] = df.support.apply(int)
df.style.background_gradient(cmap='viridis',
subset=pd.IndexSlice['0':'9', :'f1-score'])

How can I improve massively classification report of one class using ensemble model?

I have a dataset including
{0: 6624, 1: 75} 0 for nonobservational sentences and 1 for observational sentences. (basically, I annotate my sentences using Named Entity Recognition, If there is a specific entity like DATA, TIME, LONG (coordinate) I put label 1)
Now I want to make a model to classify them, the best model (CV =3 FOR ALL) that I made is the ensembling model of
clf= SGDClassifier()
trial_05=Pipeline([("vect",vec),("clf",clf)])
which has:
precision recall f1-score support
0 1.00 1.00 1.00 6624
1 0.73 0.57 0.64 75
micro avg 0.99 0.99 0.99 6699
macro avg 0.86 0.79 0.82 6699
weighted avg 0.99 0.99 0.99 669
[[6611 37]
[ 13 38]]
and this model which used resampled sgd for classifcation
precision recall f1-score support
0 1.00 0.92 0.96 6624
1 0.13 1.00 0.22 75
micro avg 0.92 0.92 0.92 6699
macro avg 0.56 0.96 0.59 6699
weighted avg 0.99 0.92 0.95 6699
[[6104 0]
[ 520 75]]
As you see the problem in both cases is class 1, but in forst one we have fairly good precision and f1 score versus in the second one we have a very good recall
So I decided to use ensemble model using both in this way:
from sklearn.ensemble import VotingClassifier#create a dictionary of our models
estimators=[("trail_05",trial_05), ("resampled", SGD_RESAMPLED_Model)]#create our voting classifier, inputting our models
ensemble = VotingClassifier(estimators, voting='hard')
now I have this result:
precision recall f1-score support
0 0.99 1.00 1.00 6624
1 0.75 0.48 0.59 75
micro avg 0.99 0.99 0.99 6699
macro avg 0.87 0.74 0.79 6699
weighted avg 0.99 0.99 0.99 6699
[[6612 39]
[ 12 36]]
As you the ensembe model has better precision regarding to class 1,but worse recall and f1 socre which caused to worse confusion matrix regarding classed 1 (36 TP vs 38 TP for class 1)
MY aim is to improve TP for class one (f1 score, recall for class 1)
what do you recommend to improve TP for class one (f1score, recall for class 1?
generaly do you have any idea regarding my workflow?
I have tried parameter tuning, it i does not improve sgd model.

GridSearchCV returns a different value

I split my dataset in two parts: training set and test set. For now just forget the test set and use the training set with the function GridSearchCV of the package sklearn.model_selection to search the best parameters for an SVM:
Cs = [0.001, 0.01, 0.1, 1, 10, 100, 1000]
gammas = [0.001, 0.01, 0.1, 1]
# Set the parameters by cross-validation
param_grid = [{'kernel': ['rbf'], 'gamma': gammas, 'C': Cs}]
clf = GridSearchCV(svm.SVC(), param_grid = param_grid, cv=nfolds, verbose=1.)
clf.fit(x_train, labels)
after found my best C and gamma parameters, I create an SVM and I fit it with the training set (used before to search the best C and gamma):
model = svm.SVC(kernel='rbf', C = clf.best_params_['C'], gamma = clf.best_params_['gamma'])
model.fit(x_train, y_train)
At this point I tried one thing, I used the predict() function of the GridSearchCV object and the one of the svm.SVC object:
predicted_label1 = model.predict(x_test)
predicted_label2 = clf.predict(x_test)
and then I used the classification_report(y_test, predicted_label) to valuate my two predicted_label vectors. In my mind I should obtain the same values but this not happens...Here my output:
precision recall f1-score support
0.0 0.24 0.97 0.39 357
1.0 0.00 0.00 0.00 358
2.0 0.00 0.00 0.00 357
3.0 0.00 0.00 0.00 357
avg / total 0.06 0.24 0.10 1429
fine parametri
training set and test set saved
Create SVM classifier
precision recall f1-score support
0.0 0.70 0.63 0.66 357
1.0 0.89 0.90 0.90 358
2.0 0.89 0.94 0.91 357
3.0 0.85 0.88 0.86 357
avg / total 0.83 0.84 0.83 1429
The first is from the GridSearchCV and the second from the SVM...
Is this normal?
What does GridSearchCV returns? Does it fit with the passed training set?