I am trying to apply a univariate feature selection method using the Python module scikit-learn to a regression (i.e. continuous valued response values) dataset in svmlight format.
I am working with scikit-learn version 0.11.
I have tried two approaches - the first of which failed and the second of which worked for my toy dataset but I believe would give meaningless results for a real dataset.
I would like advice regarding an appropriate univariate feature selection approach I could apply to select the top N features for a regression dataset. I would either like (a) to work out how to make the f_regression function work or (b) to hear alternative suggestions.
The two approaches mentioned above:
I tried using sklearn.feature_selection.f_regression(X,Y).
This failed with the following error message:
"TypeError: copy() takes exactly 1 argument (2 given)"
I tried using chi2(X,Y). This "worked" but I suspect this is because the two response values 0.1 and 1.8 in my toy dataset were being treated as class labels? Presumably, this would not yield a meaningful chi-squared statistic for a real dataset for which there would be a large number of possible response values and the number in each cell [with a particular response value and value for the attribute being tested] would be low?
Please find my toy dataset pasted into the end of this message.
The following code snippet should give the results I describe above.
from sklearn.datasets import load_svmlight_file
X_train_data, Y_train_data = load_svmlight_file(svmlight_format_train_file) #i.e. change this to the name of my toy dataset file
from sklearn.feature_selection import SelectKBest
featureSelector = SelectKBest(score_func="one of the two functions I refer to above",k=2) #sorry, I hope this message is clear
featureSelector.fit(X_train_data,Y_train_data)
print [1+zero_based_index for zero_based_index in list(featureSelector.get_support(indices=True))] #This should print the indices of the top 2 features
Thanks in advance.
Richard
Contents of my contrived svmlight file - with additional blank lines inserted for clarity:
1.8 1:1.000000 2:1.000000 4:1.000000 6:1.000000#mA
1.8 1:1.000000 2:1.000000#mB
0.1 5:1.000000#mC
1.8 1:1.000000 2:1.000000#mD
0.1 3:1.000000 4:1.000000#mE
0.1 3:1.000000#mF
1.8 2:1.000000 4:1.000000 5:1.000000 6:1.000000#mG
1.8 2:1.000000#mH
As larsmans noted, chi2 cannot be used for feature selection with regression data.
Upon updating to scikit-learn version 0.13, the following code selected the top two features (according to the f_regression test) for the toy dataset described above.
def f_regression(X,Y):
import sklearn
return sklearn.feature_selection.f_regression(X,Y,center=False) #center=True (the default) would not work ("ValueError: center=True only allowed for dense data") but should presumably work in general
from sklearn.datasets import load_svmlight_file
X_train_data, Y_train_data = load_svmlight_file(svmlight_format_train_file) #i.e. change this to the name of my toy dataset file
from sklearn.feature_selection import SelectKBest
featureSelector = SelectKBest(score_func=f_regression,k=2)
featureSelector.fit(X_train_data,Y_train_data)
print [1+zero_based_index for zero_based_index in list(featureSelector.get_support(indices=True))]
You could also try to do feature selection by L1/Lasso regularization. The class specifically designed for this is RandomizedLasso which will train LassoRegression on multiple subsamples of your data and select features that are selected most frequently by these models. You can also just use Lasso, LassoLars or SGDClassifier to do same thing without the benefit of resampling but faster.
Related
I have a dataframe with multiple x columns and one y column. I'd like to predict the linear relationship between y and multiple x variables.
so I am using smf.ols() model to predict the formula. I am wondering if I need to scale the data before fit the data using ols().
I checked ols website and it seems that they never talk about data scaling , for example, below website
https://www.statsmodels.org/devel/example_formulas.html
at the mean time, I used to take a course from datacamp and they don't mention about data scaling either, for example, below screenshot from datacamp course. You can see the regressed coefficient for each variable is not in the same order, like 3655 vs 83.
Here is what I did for my regression. I am wondering for my below example if we need to add scaling like
from sklearn.preprocessing import StandardScaler
scaler=StandardScaler()
scaler.fit(df_crossplot)
df_scaled=scaler.transform(df_crossplot)
then after that, I input df_scaled into the below function? do I have to do this above step? My hesitation is, if I scale it, then how to convert regressed formula back to a new formula based on original scale? Thanks for your help.
import statsmodels.formula.api as smf
def linear_regression_statsmodel(df_crossplot,crossplot_y,crossplot_x_list):
formula_crossplot=crossplot_y+'~'
for x in crossplot_x_list:
formula_crossplot=formula_crossplot+'+'+x
model_crossplot=smf.ols(formula=formula_crossplot,data=df_crossplot).fit()
df_crossplot['regressed']=model_crossplot.params[0]
regressed_x_string=f'{model_crossplot.params[0]:,.2f}'
for ix,x in enumerate(crossplot_x_list):
df_crossplot['regressed']=df_crossplot['regressed']+df_crossplot[x]*model_crossplot.params[ix+1]
if model_crossplot.params[ix+1]>0:
regressed_x_string=regressed_x_string+f'+{model_crossplot.params[ix+1]:,.2f}*{x}'
else: # no need + sign since we have already negative sign
regressed_x_string=regressed_x_string+f'{model_crossplot.params[ix+1]:,.2f}*{x}'
return df_crossplot,model_crossplot,regressed_x_string
This question already has an answer here:
interpreting Graphviz output for decision tree regression
(1 answer)
Closed 1 year ago.
from sklearn import tree
import graphviz
import shap
X,y = shap.datasets.boston()
clf = tree.DecisionTreeRegressor(max_depth=2).fit(X, y)
gives us the following tree:
The values are confusing to me, I understand that the values at leaves are the predictions once that leaf is reached. However what do the values at nodes represent?
I found a few SO posts/documentation for Classification but not for regression.
EDIT: Thinking about if further I see that they're most likely just the values of those bins if the tree was cut short. Not sure why exactly they're used in SHAP though.
Let's focus in one node, for example:
X12<= 14.4 refers to the next split you will apply to your data. In
this case you will use the feature X 12.
Samples= 430, refers to all training samples that are in this node.
Check that the root node is 506 (which is the sum of his son nodes
(430+76))
If we make a prediction at this internal node, we will predict that
the values is value=19.934 and we will be committing an mse= 40.273, which refers to the error.
Obviously, when we are splitting data with more nodes, we are reducing the number of samples and of course the mse, since we are narrowing down. The value vary since we are being more precis.
About shap, you are only using this library to import the dataset, nothing more. You could have imported the data without using the shap library. There are different ways to import Boston data, for example using Sklearn:
from sklearn.datasets import load_boston
X, y = load_boston(return_X_y=True)
Anyway, you should check if it's the exact dataset (e.g., one data set have more samples).
I am trying to create an AI that reads my dataset and states whether an input outside the data is 1 or 0
My dataset has column for qualitative data and column for a boolean. Here is a sample from it:
Imports:
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.metrics import classification_report
import re
import string
Open and cleaning dataset:
saisei_data = saisei_data.dropna(how='any',axis=0)
saisei_data = saisei_data.sample(frac=1)
X = saisei_data['Data']
y = saisei_data['Conscious']
saisei_data
Vectorisation:
from sklearn.feature_extraction.text import TfidfVectorizer
vectorization = TfidfVectorizer()
xv_train = vectorization.fit_transform(X_train)
xv_test = vectorization.fit_transform(X_test)
Example Algorithm - Logistic Regression:
LR = LogisticRegression()
LR.fit(xv_train,y_train)
pred_lr=LR.predict(xv_test) # Here is where I get an error
Everything works fine until I predict using the logistic regression algorithm.
The Error:
ValueError: X has 112 features per sample; expecting 23
This seems to change to similar errors such as:
ValueError: X has 92 features per sample; expecting 45
I am new to machine learning so I don't really know what I'm doing when it comes to using the algorithms, however I tried printing the xv_test variable and here is a sample of the output (also changes often):
Any ideas?
That is because you erroneously apply .fit_transform() to your test data; and, in this case, you are lucky enough that the process produces a programming error, thus alerting you that you are doing something methodologically wrong (which is not always the case).
We never apply either .fit() or .fit_transform() to unseen (test) data. The fitting should be done only once with the training data, like you have done here:
xv_train = vectorization.fit_transform(X_train)
For subsequent transformations of unseen (test) data, we use only .transform(). So, your next line should be
xv_test = vectorization.transform(X_test)
That way, the features in the test set will be the same with the ones in the training set, as it should be in the first place.
Notice the difference between the two methods in the docs (emphasis mine):
fit_transform:
Learn vocabulary and idf, return document-term matrix.
transform:
Transform documents to document-term matrix.
Uses the vocabulary and document frequencies (df) learned by fit (or fit_transform).
and recall that we don't ever use the test set to learn anything.
So, simple general mnemonic rule, applicable practically everywhere:
The terms "fit" and "test data" are always (always...) incompatible; mixing them will create havoc.
I'm relatively new to Python and am trying to get some data prepped to train a RandomForest. For various reasons, we want the data to be discrete, so there are a few continuous variables that need to be discretized. I found qcut in pandas, which seems to do what I want - I can set a number of bins, and it will discretize the variable into that many bins, trying to keep the counts in each bin even.
However, the output of pandas.qcut is a list of Intervals, and the RandomForest classifier in scikit-learn needs a string. I found that I can convert an interval into a string by using .astype(str). Here's a quick example of what I'm doing:
import pandas as pd
from random import sample
vals = sample(range(0,100), 100)
cuts = pd.qcut(vals, q=5)
str_cuts = pd.qcut(vals, q=5).astype(str)
and then str_cuts is one of the variables passed into a random forest.
However, the intent of this system is to train a RandomForest, save it to a file, and then allow someone to load it at a later date and get a classification for a new test instance, that is not available at training time. And because the classifier was trained on discretized data, the new test instance will need to be discretized before it can be used. So what I want to be able to do is read in a new instance, apply the already-established discretization scheme to it, convert it to a string, and run it through the random forest. However, I'm getting hung up on the best way to 'apply the discretization scheme'.
Is there an easy way to handle this? I assume there's no straight-forward way to convert a string back into an Interval. I can get the list of all Interval values from the discretization (ex: cuts.unique()) and apply that at test-time, but that would require saving/loading a discretization dictionary alongside the random forest, which seems clunky, and I worry about running into issues trying to recreate a categorical variable (coming mostly from R, which is extremely particular about the format of categorical variables). Or is there another way around this that I'm not seeing?
Use the labelsargument in qcut and use pandas Categorical.
Either of those can help you create categories instead of interval for your variable. Then, you can use a form of encoding, for example Label Encoding or Ordinal Encoding to convert the categories (the factors if you're used to R) to numerical values which the Forest will be able to use.
Then the process goes :
cutting => categoricals => encoding
and you don't need to do it by hand anymore.
Lastly, some gradient boosted trees libraries have support for categorical variables though it's not a silver bullet and will depend on your goal. See catboost and lightgbm.
For future searchers, there are benefits to using transformers from scikit-learn instead of pandas. In this case, KBinsDiscretizer is the scikit equivalent of qcut.
It can be used in a pipeline, which will handle applying the previously-learned discretization to unseen data without the need for storing the discretization dictionary separately or round trip string conversion. Here's an example:
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import KBinsDiscretizer
pipeline = make_pipeline(KBinsDiscretizer(n_bins=3, encode='ordinal', strategy='quantile'),
RandomForestClassifier())
X, y = make_classification()
X_train, X_test, y_train, y_test = train_test_split(X, y)
pipeline.fit(X_train, y_train)
predictions = pipeline.predict(X_test)
If you really need to convert back and forth between pandas IntervalIndex and string, you'll probably need to do some parsing as described in this answer: https://stackoverflow.com/a/65296110/3945991 and either use FunctionTransformer or write your own Transformer for pipeline integration.
While it may not be the cleanest-looking method, converting a string back into an interval is indeed possible:
import pandas as pd
str_intervals = [i.replace("(","").replace("]", "").split(", ") for i in str_cuts]
original_cuts = [pd.Interval(float(i), float(j)) for i, j in str_intervals]
The python package Fancyimpute provides several methods for the imputation of missing values in Python. The documentation provides examples such as:
# X is the complete data matrix
# X_incomplete has the same values as X except a subset have been replace with NaN
# Model each feature with missing values as a function of other features, and
# use that estimate for imputation.
X_filled_ii = IterativeImputer().fit_transform(X_incomplete)
This works fine when applying the imputation method to a dataset X. But what if a training/test split is necessary? Once
X_train_filled = IterativeImputer().fit_transform(X_train_incomplete)
is called, how do I impute the test set and create X_test_filled? The test set needs to be imputed using the information from the training set. I guess that IterativeImputer() should returns and object that can fit X_test_incomplete. Is that possible?
Please note that imputing on the whole dataset and then split into training and test set is not correct.
The package looks like it mimic's scikit-learn's API. And after looking in the source code, it looks like it does have a transform method.
my_imputer = IterativeImputer()
X_trained_filled = my_imputer.fit_transform(X_train_incomplete)
# now transform test
X_test_filled = my_imputer.transform(X_test)
The imputer will apply the same imputations that it learned from the training set.