I'm trying to use plot_confusion_matrix,
from sklearn.metrics import confusion_matrix
y_true = [1, 1, 0, 1]
y_pred = [1, 1, 0, 0]
confusion_matrix(y_true, y_pred)
Output:
array([[1, 0],
[1, 2]])
Now, while using the followings; using 'classes' or without 'classes'
from sklearn.metrics import plot_confusion_matrix
plot_confusion_matrix(y_true, y_pred, classes=[0,1], title='Confusion matrix, without normalization')
or
plot_confusion_matrix(y_true, y_pred, title='Confusion matrix, without normalization')
I expect to get similar output like this except the numbers inside,
Plotting simple diagram, it should not require the estimator.
Using mlxtend.plotting,
from mlxtend.plotting import plot_confusion_matrix
import matplotlib.pyplot as plt
import numpy as np
binary1 = np.array([[4, 1],
[1, 2]])
fig, ax = plot_confusion_matrix(conf_mat=binary1)
plt.show()
It provides same output.
Based on this
it requires a classifier,
disp = plot_confusion_matrix(classifier, X_test, y_test,
display_labels=class_names,
cmap=plt.cm.Blues,
normalize=normalize)
Can I plot it without a classifier?
plot_confusion_matrix expects a trained classifier. If you look at the source code, what it does is perform the prediction to generate y_pred for you:
y_pred = estimator.predict(X)
cm = confusion_matrix(y_true, y_pred, sample_weight=sample_weight,
labels=labels, normalize=normalize)
So in order to plot the confusion matrix without specifying a classifier, you'll have to go with some other tool, or do it yourself.
A simple option is to use seaborn:
import seaborn as sns
cm = confusion_matrix(y_true, y_pred)
f = sns.heatmap(cm, annot=True)
I am a bit late here, but I thought other people might benefit from my answer.
As others have mentioned using plot_confusion_matrix is not an option without the classifier but it is still possible to use sklearn to obtain a similar-looking confusion matrix without the classifier. The function below does exactly this.
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
def confusion_ma(y_true, y_pred, class_names):
cm = confusion_matrix(y_true, y_pred, normalize='true')
disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=class_names)
disp.plot(cmap=plt.cm.Blues)
return plt.show()
The confusion_matrix function returns a simple ndarry matrix. By passing this together with labels of the predictions to the ConfusionMatrixDisplay function a similar looking matrix is obtained. In the definition I've added the class_names to be displayed instead of 0 and 1, chosen to normalize the output and specified a colormap - change accordingly to your needs.
Since plot_confusion_matrix require the argument 'estimator' not to be None, the answer is: no, you can't. But you can plot your confusion matrix in other ways, for example see this answer: How can I plot a confusion matrix?
I tested the following "identity classifier" in a Jupyter notebook running the conda_python3 kernel in Amazon SageMaker. The reason is that SageMaker's transformation job is async and so does not allow the classifier to be used in the parameters of plot_confusion_matrix, y_pred has to be calculated before calling the function.
IC = type('IdentityClassifier', (), {"predict": lambda i : i, "_estimator_type": "classifier"})
plot_confusion_matrix(IC, y_pred, y_test, normalize='true', values_format='.2%');
So while plot_confusion_matrix indeed expects an estimator, you'll not necessarily have to use another tool IMO, if this solution fits your use case.
simplified POC from the notebook
I solved the problem of using a customized classifier; you can build any custom classifier and pass it to the plot_confusion matrix as a class:
class MyModelPredict(object):
def __init__(self, model):
self._estimator_type = 'classifier'
def predict(self, X):
return your_custom_prediction
model = MyModelPredict()
plot_confusion_matrix(model, X, y_true)
Related
How can I save a confusion matrix as png? I've saw this answer:
How to save Confusion Matrix plot so that I can call it for future reference?
from sklearn.metrics import plot_confusion_matrix
y_true = [0,1,1,1,0]
y_pred = [1,1,1,1,0]
IC = type('IdentityClassifier', (), {"predict": lambda i : i, "_estimator_type": "classifier"})
cm = plot_confusion_matrix(IC, y_pred, y_true, normalize='true', values_format='.2%')
cm.figure_.savefig('confusion_matrix.png')
The result that I'm getting is just a black png image.
I think, you should update sklearn to the latest version and then use:
from sklearn.metrics import ConfusionMatrixDisplay
y_true = [0,1,1,1,0]
y_pred = [1,1,1,1,0]
IC = type('IdentityClassifier', (), {"predict": lambda i : i, "_estimator_type": "classifier"})
cm=ConfusionMatrixDisplay.from_estimator(IC, y_pred, y_true, normalize='true', values_format='.2%')
cm.figure_.savefig('confusion_matrix.png')
im really new to machine learning and was tasked with creating a ROC curve. I get the error ValueError: Found input variables with inconsistent numbers of samples: [200, 400].
Here is a picture of my code for the ROC curve.
import sklearn.metrics as metrics
from sklearn.metrics import roc_curve, auc
y_pred = basic_model.predict(X_test).ravel()
nn_fpr_keras, nn_tpr_keras, nn_thresholds_keras = roc_curve(y_test, y_pred)
auc_keras = auc(nn_fpr_keras, nn_tpr_keras)
plt.plot(nn_fpr_keras, nn_tpr_keras, marker='.', label='Neural Network (auc = %0.3f)' % auc_keras)
I actually know what the problem is I am using 'sparse_categorical_crossentropy' for loss.(My advisor said I had to use it). Basically They also told me that now all I have to do is to change my X_test go from a 2D array to a 1D array. I have tried to use .flatten() but that didn't help? Any ideas on what I should do or any resources that I could study?
I just needed to set X_train=X_train[;,0]
Ravel() seems to not work.
i am new to python and deep learning, i trained a multi classifier model and want to plot a confusion matrix but i am facing an error
here is my code
from sklearn.metrics import plot_confusion_matrix
import matplotlib.pyplot as plt
from sklearn.metrics import ConfusionMatrixDisplay
Y_pred = model.predict_generator(test_generator)
y_pred = np.argmax(Y_pred, axis=1)
category_names = sorted(os.listdir('D:/DiabaticRetinopathy/mq_dataset/DR_Normal/train'))
print(category_names)
cm = confusion_matrix(test_generator.classes, y_pred)
plot_confusion_matrix(cm, classes = category_names, title='Confusion Matrix', normalize=False, figname = 'Confusion_matrix_concrete.jpg')
i upddated my sklearn to 0.24 version. i restarted my kernel after updating but still its giving an error:
TypeError: plot_confusion_matrix() got an unexpected keyword argument 'classes'
use labels instead of classes, then Remove title, figname
plot_confusion_matrix(X = test_generator.classes, y_true = y_pred,labels= category_names, normalize=False)
Documentation: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.plot_confusion_matrix.html
There is a keyword labels, but not classes, so you can change it to that.
The error states that the keyword classes you provided is not a keyword that this function recognizes. This happens in your last line.
The documentation gives a list of the keywords that you can use:
doc
I'm attempting to do a toy linear regression in Python with TensorFlow, using the pre-built estimator tf.contrib.learn.LinearRegressor instead of building my own estimator.
The inputs I'm using are real-valued numbers between 0 and 1, and the outputs are just 3*inputs. TensorFlow seems to fit the data (no errors raised), but the outputs have no correlation to what they should be.
I'm not sure I'm getting the predictions done correctly- the documentation for the predict() function is pretty sparse.
Any ideas for how to improve the fitting?
import numpy as np
import pandas as pd
import tensorflow as tf
import itertools
import matplotlib.pyplot as plt
#Defining data set
x = np.random.rand(200)
y = 3.0*x
data = pd.DataFrame({'X':x, 'Y':y})
training_data = data[50:]
test_data= data[:50]
COLUMNS = ['Y','X']
FEATURES = ['X']
LABELS = 'Y'
#Wrapper function for the inputs of LinearRegressor
def get_input_fn(data_set, num_epochs=None, shuffle=True):
return tf.estimator.inputs.pandas_input_fn(
x=pd.DataFrame(data_set[FEATURES]),
y=pd.Series(data_set[LABELS]),
num_epochs=num_epochs,
shuffle=shuffle)
feature_cols = [tf.feature_column.numeric_column(k) for k in FEATURES]
regressor = tf.contrib.learn.LinearRegressor(feature_columns=feature_cols)
regressor.fit(input_fn=get_input_fn(test_data), steps=100)
results = regressor.predict(input_fn=get_input_fn(test_data,
num_epochs=1))
predictions = list(itertools.islice(results, 50))
#Visualizing the results
fig = plt.figure(figsize=[8,8])
ax = fig.add_subplot(111)
ax.scatter(test_data[LABELS], predictions)
ax.set_xlabel('Actual')
ax.set_ylabel('Predicted')
plt.show()
Scatter plot of results
Figured out the answer, answering here for posterity-
my input function to LinearRegressor had shuffle=True set as an argument, and my predict() call did not set shuffle=False. So the outputs were shuffled around, making them look like they didn't converge!
Can someone please tell how to use ensembles in sklearn using partial fit.
I don't want to retrain my model.
Alternatively, can we pass pre-trained models for ensembling ?
I have seen that voting classifier for example does not support training using partial fit.
The Mlxtend library has an implementation of VotingEnsemble which allows you to pass in pre-fitted models. For example if you have three pre-trained models clf1, clf2, clf3. The following code would work.
from mlxtend.classifier import EnsembleVoteClassifier
import copy
eclf = EnsembleVoteClassifier(clfs=[clf1, clf2, clf3], weights=[1,1,1], fit_base_estimators=False)
When set to false the fit_base_estimators argument in EnsembleVoteClassifier ensures that the classifiers are not refit.
In general, when looking for more advanced technical features that sci-kit learn does not provide, look to mlxtend as a first point of reference.
Workaround:
VotingClassifier checks that estimators_ is set in order to understand whether it is fitted, and is using the estimators in estimators_ list for prediction.
If you have pre trained classifiers, you can put them in estimators_ directly like the code below.
However, it is also using LabelEnconder, so it assumes labels are like 0,1,2,... and you also need to set le_ and classes_ (see below).
from sklearn.ensemble import VotingClassifier
from sklearn.preprocessing import LabelEncoder
clf_list = [clf1, clf2, clf3]
eclf = VotingClassifier(estimators = [('1' ,clf1), ('2', clf2), ('3', clf3)], voting='soft')
eclf.estimators_ = clf_list
eclf.le_ = LabelEncoder().fit(y)
eclf.classes_ = seclf.le_.classes_
# Now it will work without calling fit
eclf.predict(X,y)
Unfortunately, currently this is not possible in scikit VotingClassifier.
But you can use http://sebastianraschka.com/Articles/2014_ensemble_classifier.html (from which VotingClassifer is implemented) to try and implement your own voting classifier which can take pre-fitted models.
Also we can look at the source code here and modify it to our use:
from sklearn.preprocessing import LabelEncoder
import numpy as np
le_ = LabelEncoder()
# When you do partial_fit, the first fit of any classifier requires
all available labels (output classes),
you should supply all same labels here in y.
le_.fit(y)
# Fill below list with fitted or partial fitted estimators
clf_list = [clf1, clf2, clf3, ... ]
# Fill weights -> array-like, shape = [n_classifiers] or None
weights = [clf1_wgt, clf2_wgt, ... ]
weights = None
#For hard voting:
pred = np.asarray([clf.predict(X) for clf in clf_list]).T
pred = np.apply_along_axis(lambda x:
np.argmax(np.bincount(x, weights=weights)),
axis=1,
arr=pred.astype('int'))
#For soft voting:
pred = np.asarray([clf.predict_proba(X) for clf in clf_list])
pred = np.average(pred, axis=0, weights=weights)
pred = np.argmax(pred, axis=1)
#Finally, reverse transform the labels for correct output:
pred = le_.inverse_transform(np.argmax(pred, axis=1))
It's not too hard to implement the voting. Here's my implementation:
import numpy as np
class VotingClassifier(object):
""" Implements a voting classifier for pre-trained classifiers"""
def __init__(self, estimators):
self.estimators = estimators
def predict(self, X):
# get values
Y = np.zeros([X.shape[0], len(self.estimators)], dtype=int)
for i, clf in enumerate(self.estimators):
Y[:, i] = clf.predict(X)
# apply voting
y = np.zeros(X.shape[0])
for i in range(X.shape[0]):
y[i] = np.argmax(np.bincount(Y[i,:]))
return y
The Mlxtend library has an implementation works, you still need to call the fit function for the EnsembleVoteClassifier. Seems the fit function doesn't really modify any parameters rather checking the possible label values. In the example below, you have to give an array contains all the possible values appear in original y(in this case 1,2) to eclf2.fit It doesn't matter for X.
import numpy as np
from mlxtend.classifier import EnsembleVoteClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
import copy
clf1 = LogisticRegression(random_state=1)
clf2 = RandomForestClassifier(random_state=1)
clf3 = GaussianNB()
X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
y = np.array([1, 1, 1, 2, 2, 2])
for clf in (clf1, clf2, clf3):
clf.fit(X, y)
eclf2 = EnsembleVoteClassifier(clfs=[clf1, clf2, clf3],voting="soft",refit=False)
eclf2.fit(None,np.array([1,2]))
print(eclf2.predict(X))