Develop Reference

FutureWarning in scikit-learn Logistic Regression solver

I have been using a course on Udemy for learning Machine-Learning. I have found a lot of deprecated code and now I have this issue:
The code:
from sklearn.linear_model import LogisticRegression
classifier = LogisticRegression(random_state = 0)
classifier.fit(X_train, y_train)
The warning:
C:\Users\admin\Anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
FutureWarning)
How can I get rid of this deprecation warning?

In scikit-learn v0.20, which you probably use, the default value for the solver used in LogisticRegression was liblinear; from the docs:
solver : str, {‘newton-cg’, ‘lbfgs’, ‘liblinear’, ‘sag’, ‘saga’}, default: ‘liblinear’.
This changed in v0.22 (current latest) to lbfgs.
So, in order to avoid surprizes from this change, scikit-learn warns you for this change in the default that will come in future versions, in order to keep your code future-proof.
To get rid of it, just define explicitly a solver in your LogisticRegression definition, i.e.
classifier = LogisticRegression(random_state = 0, solver='lbfgs') # default in v0.22
or
classifier = LogisticRegression(random_state = 0, solver='liblinear') # default until v0.21
The first documentation link provided above shows all the available options, along with some short comment/advice on each one.

Well, the warning message is telling you. All you need to do is to explicitly specify which solver to use:
classifier = LogisticRegression(random_state = 0, solver='lbfgs')
(or any other solver you want to use)
For available options, see the sklearn docs.

Try using
classifier = LogisticRegression(random_state=0, solver="liblinear")
And checkout solver parameter in the documentation: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html

What is the canonical way to obtain parameters of a trained classifier in scikit-learn?

Once a scikit-learn classifier is trained:
import sklearn.cluster
clf = sklearn.cluster.KMeans()
clf.fit(X)
there are (at least) two options to obtain values of its parameters. Specifically,
By referring to a parameter name with a traling underscore:
clf.n_clusters_
From a dictionary obtained with get_params():
ps = clf.get_params()
ps['n_clusters']
Which of these approaches is the preferred one?

I would say clf.get_params() because you don't always know what parameters might be available for a given estimator and this method will return everything, unless you know exactly what you are looking for. It also has a deep argument which when set to true, "...will return the parameters for this estimator and contained subobjects that are estimators"

sklearn mixture.GMM in python using univariate GMM

In R, mclust has an argument 'modelNames' where you can define which model to implement. I wish to do a univariate modeling which is also modelNames <- 'V' in mclust under mixture.GMM in python. However, the only thing I find that I can tweak with is the covariance_type. Nonetheless, when I run the same data using R and mixture.GMM under sklearn, I get different fitting despite the same number of fitted components. What could I change in mixture.GMM to indicate I am using a univariate variable variance?
mclust code:
function(x){Mclust(ma78[x,],G=2,modelNames="V",verbose=FALSE)}
GMM code:
gmm = GMM(n_components = 2).fit(data)

With univariate data, the covariance can either be equal or unique (variable). With Mclust these options are modelNames = "E" or "V", respectively.
With sklearn, they appear to be covariance_type = "tied" or "full". Possibly, something like this for variable Gaussian mixture model
gmm = mixture.GaussianMixture(n_components = 2, covariance_type='full').fit(data)
Even using Mclust or sklearn alone there can be instanced that you may not get same parameter values for different runs - this is because the estimates can depend on the initial values. One way to avoid this is using a larger number of starts if such option is available.

found the answer on stats.stackexchange. The only thing you have to do is to reshape your data data.reshape(-1, 1) before you pass it into sklearn.mixture.GaussianMixture
Andreas

how to enforce Monotonic Constraints in XGBoost with ScikitLearn?

I build up a XGBoost model using scikit-learn and I am pretty happy with it. As fine tuning to avoid overfitting, I'd like to ensure monotonicity of some features but there I start facing some difficulties...
As far as I understood, there is no documentation in scikit-learn about xgboost (which I confess I am really surprised about - knowing that this situation is lasting for several months). The only documentation I found is directly on http://xgboost.readthedocs.io
On this website, I found out that monotonicity can be enforced using "monotone_constraints" option.
I tried to use it in Scikit-Learn but I got an error message "TypeError: init() got an unexpected keyword argument 'monotone_constraints'"
Do you know a way to do it ?
Here is the code I wrote in python (using spyder):
grid = {'learning_rate' : 0.01, 'subsample' : 0.5, 'colsample_bytree' : 0.5,
'max_depth' : 6, 'min_child_weight' : 10, 'gamma' : 1,
'monotone_constraints' : monotonic_indexes}
#'monotone_constraints' ~ = "(1,-1)"
m07_xgm06 = xgb.XGBClassifier(n_estimators=2000, **grid)
m07_xgm06.fit(X_train_v01_oe, Label_train, early_stopping_rounds=10, eval_metric="logloss",
eval_set=[(X_test1_v01_oe, Label_test1)])

In order to do this using the xgboost sklearn API, you need to upgrade to xgboost 0.81. They fixed the ability to set parameters controlled via kwargs as part of this PR:
https://github.com/dmlc/xgboost/pull/3791

XGBoost Scikit-Learn API currently (0.6a2) doesn't support monotone_constraints. You can use Python API instead. Take a look into example.
This code in the example can be removed:
params_constr['updater'] = "grow_monotone_colmaker,prune"

How would you expect monotone constraints to work for a general classification problem where the response might have more than 2 levels? All the examples I've seen relating to this functionality are for regression problems. If your classification response only has 2 levels, try switching to regression on an indicator variable and then choose an appropriate score threshold for classification.
This feature appears to work as of the latest xgboost / scikit-learn, provided that you use an XGBregressor rather than an XGBclassifier and set monotone_constraints via kwargs.
The syntax is like this:
params = {
'monotone_constraints':'(-1,0,1)'
}
normalised_weighted_poisson_model = XGBRegressor(**params)
In this example, there is a negative constraint on column 1 in the training data, no constraint on column 2, and a positive constraint on column 3. It is up to you to keep track of which is which - you cannot refer to columns by name, only by position, and you must specify an entry in the constraint tuple for every column in your training data.

Semi-supervised learning for regression by scikit-learn

Can Label Propagation be used for semi-supervised regression tasks in scikit-learn?
According to its API, the answer is YES.
http://scikit-learn.org/stable/modules/label_propagation.html
However, I got the error message when I tried to run the following code.
from sklearn import datasets
from sklearn.semi_supervised import label_propagation
import numpy as np
rng=np.random.RandomState(0)
boston = datasets.load_boston()
X=boston.data
y=boston.target
y_30=np.copy(y)
y_30[rng.rand(len(y))<0.3]=-999
label_propagation.LabelSpreading().fit(X,y_30)
It shows that "ValueError: Unknown label type: 'continuous'" in the label_propagation.LabelSpreading().fit(X,y_30) line.
How should I solve the problem? Thanks a lot.

It looks like the error in the documentation, code itself clearly is classification only (beggining of the .fit call of the BasePropagation class):
check_classification_targets(y)
# actual graph construction (implementations should override this)
graph_matrix = self._build_graph()
# label construction
# construct a categorical distribution for classification only
classes = np.unique(y)
classes = (classes[classes != -1])
In theory you could remove the "check_classification_targets" call and use "regression like method", but it will not be the true regression since you will never "propagate" any value which is not encountered in the training set, you will simply treat the regression value as the class identifier. And you will be unable to use value "-1" since it is a codename for "unlabeled"...