Develop Reference

UserWarning: X does not have valid feature names, but Linear Regression was fitted with feature names [duplicate]

I'm getting the following warning after upgrading to version 1.0 of scikit-learn:
UserWarning: X does not have valid feature names, but IsolationForest was
fitted with feature name
I cannot find in the docs on what is a "valid feature name". How do I deal with this warning?

I got the same warning message with another sklearn model. I realized that it was showing up because I fitted the model with a data in a dataframe, and then used only the values to predict. From the moment I fixed that, the warning disappeared.
Here is an example:
model_reg.fit(scaled_x_train, y_train[vp].values)
data_pred = model_reg.predict(scaled_x_test.values)
This first code had the warning, because scaled_x_train is a DataFrame with feature names, while scaled_x_test.values is only values, without feature names. Then, I changed to this:
model_reg.fit(scaled_x_train.values, y_train[vp].values)
data_pred = model_reg.predict(scaled_x_test.values)
And now there are no more warnings on my code.

I was getting very similar error but on module DecisionTreeClassifier for Fit and Predict.
Initially I was sending dataframe as input to fit with headers and I got the error.
When I trimmed to remove the headers and sent only values then the error got disappeared.
Sample code before and after changes.
Code with Warning:
model = DecisionTreeClassifier()
model.fit(x,y) #Here x includes the dataframe with headers
predictions = model.predict([
[20,1], [20,0]
])
print(predictions)
Code without Warning:
model = DecisionTreeClassifier()
model.fit(x.values,y) #Here x.values will have only values without headers
predictions = model.predict([
[20,1], [20,0]
])
print(predictions)

I had also the same problem .The problem was due to fact that I fitted the model with X train data as dataframe (model.fit(X,Y)) and I make a prediction with with X test as an array ( model.predict([ [20,0] ]) ) . To solve that I have converted the X train dataframe into an array as illustrated bellow .
BEFORE
model = DecisionTreeClassifier()
model.fit(X,Y) # X train here is a dataFrame
predictions = model.predict([20,0]) ## generates warning
AFTER
model = DecisionTreeClassifier()
X = X.values # conversion of X into array
model.fit(X,Y)
model.predict([ [20,0] ]) #now ok , no warning

The other answers so far recommend (re)training using a numpy array instead of a dataframe for the training data. The warning is a sort of safety feature, to ensure you're passing the data you meant to, so I would suggest to pass a dataframe (with correct column labels!) to the predict function instead.
Also, note that it's just a warning, not an error. You can ignore the warning and proceed with the rest of your code without problem; just be sure that the data is in the same order as it was trained with!

I got the same error while using dataframes but by passing only values it is no more there
use
reg = reg.predict( x[['data']].values , y)
It is showing error because our dataframe has feature names but we should fit the data as 2d array(or matrix) with values for training or testing the dataset.
Here is the image of the same thing mentioned above image of jupytr notebook code

Using sklearn's roc_auc_score for OneVsOne Multi-Classification?

So I am working on a model that attempts to use RandomForest to classify samples into 1 of 7 classes. I'm able to build and train the model, but when it comes to evaluating it using roc_auc function, I'm able to perform 'ovr' (oneVsrest) but 'ovo' is giving me some trouble.
roc_auc_score(y_test, rf_probs, multi_class = 'ovr', average = 'weighted')
The above works wonderfully, I get my output, however, when I switch multi_class to 'ovo' which I understand might be better with class imbalances, I get the following error:
roc_auc_score(y_test, rf_probs, multi_class = 'ovo')
IndexError: too many indices for array
(I pasted the whole traceback below!)
Currently my data is set up as follow:
y_test (61,1)
y_probs (61, 7)
Do I need to reshape my data in a special way to use 'ovo'?
In the documentation, https://thomasjpfan.github.io/scikit-learn-website/modules/generated/sklearn.metrics.roc_auc_score.html, it says "binary y_true, y_score is supposed to be the score of the class with greater label. The multiclass case expects shape = [n_samples, n_classes] where the scores correspond to probability estimates."
Additionally, the whole traceback seems to hint as using maybe using a more binary array (hopefully that's the right term! I'm new to this!)
Very, very thankful for any ideas/thoughts!

#Tirth Patel provided the right answer, I needed to reshape my test set using one hot encoding. Thank you!

Feature Names Mismatch when Passing X_test to .predict() Function (Again, Still)

Ok I'm still having this issue and I'm at a loss as to where I'm going wrong. I thought I had a working solution, but I was wrong.
After finding a regression pipeline through TPOT, I go to use the .predict(X_test) function and I get the following error message:
ValueError: Number of features of the model must match the input. Model n_features is 117 and input n_features is 118
I read somewhere on Github that XGBoost likes to have the X features passed to it in the form of a Numpy Array, and not a Pandas Dataframe. So I did that and now I receive this error message whenever a RandomForestRegressor ends up in my pipeline.
So I investigate:
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=test_size, random_state=seed, shuffle=False)
# Here is where I convert the features to numpy arrays
X_train=X_train.values
X_test=X_test.values
print('[INFO] Printing the shapes of the training/testing feature/label sets...')
print(X_train.shape)
print(X_test.shape)
print(Y_train.shape)
print(Y_test.shape)
[INFO] Printing the shapes of the training/testing feature/label sets...
(1366, 117)
(456, 117)
(1366,)
(456,)
# Notice 117 rows for X columns...
# Now print the X_test shape just before the predict function...
print(X_test.shape)
(456, 117)
# Still 117 columns, so call predict:
predictions = best_model.predict(X_test)
ValueError: Number of features of the model must match the input. Model n_features is 117 and input n_features is 118
WHY!!!!!!?????
Now the tricky thing is, I'm using a custom tpot_config to only use the regressors XGBRegressor, ExtraTreesRegressor, GradientBoostingRegressor, AdaBoostRegressor, DecisionTreeRegressor, and RandomForestRegressor, so I need to come up with a way to train and predict the features whereby all of them will work with the data in the same way, so that no matter what pipeline it comes up with, I won't have this issue each time I go to run my code!
There have been similar questions asked at these links on SO:
Here
Here
Here
Here
... but I don't understand why my model is not predicting, when I AM passing it the same number of (X) features as was used in training the model!? Where am I going wrong here???
EDIT
I should also mention, that leaving the features as dataframes and not converting them to numpy arrays sometimes gives me a "feature names mismatch" error when XGBRegressor is in the pipeline as well. So I'm at a loss as to how to handle both the list of tree regressors (which like Dataframes) and XGBoost (which likes Numpy arrays). I have also tried “re-arranging” the columns(?) to make sure that the X_train and X_test Dataframes are in the same order like some have suggested but that didn’t do anything.
I have posted my full code in a Google Colab notebook here where you can make comments on it. How can I pass the testing data to the .predict() function no matter what pipeline TPOT comes up with??????

Thanks to weixuanfu at GitHub, I may have found a solution by moving the feature_importance code section down to the bottom of the my code, and yes using numpy arrays for the features. If I run into this issue again, I will be posting it below:
https://github.com/EpistasisLab/tpot/issues/738

Prediction values in Python

Can someone help me with the code for getting the predicted probability values? My model is working fine and is giving me predictions as 1 and 0 however I need the probability values also. The code is in two python files. The first file uses the training data set to create the map file. The second python file (scoring file) uses the map file on the test data to predict. Can someone let me know the code I should insert to get the probability values.
The below code is from the scoring file and here I need the code the get the probability values
pred = model.predict(X.values)
data["Predicted"] = pred
# I NEED THE CODE HERE TO GET THE PROBABILITY VALUES.
data.to_excel(r'result.xlsx', index=False)
Thanks a lot

Check if your model has predict_proba method.
The usage is same as the same predict method.
prob = model.predict_proba(X.values)
Edit:
Some of the learning model implementations from sklearn provide the predict_proba method. It is not a metric but as I said, a method of the class of the learning model.
For example:
from sklearn.tree import DecisionTreeClassifier
# after split you have X_train,y_train,X_testy_test
model = DecisionTreeClassifier()
model.fit(X_train,y_train)
proba = model.predict_proba(X_test)

I am no longer able to edit my question so putting it here.
Thanks for all the help. I am using random forest model. This is my code and the line 4 below is giving an error. If I remove line 4, the code runs but in the final excel file I do not get the probabilities but only the predictions as 1 and 0. Can someone please let me know how to resolve this error. The last line of the error says
ValueError: Wrong number of items passed 2, placement implies 1
pred = model.predict(X.values)
data["Predicted"] = pred
prob = model.predict_proba(X.values)
data["Pred Value"]= prob - this line causes error
data.to_excel(r'result.xlsx', index=False)'
Thanks

Scikit-learn feature selection for regression data

I am trying to apply a univariate feature selection method using the Python module scikit-learn to a regression (i.e. continuous valued response values) dataset in svmlight format.
I am working with scikit-learn version 0.11.
I have tried two approaches - the first of which failed and the second of which worked for my toy dataset but I believe would give meaningless results for a real dataset.
I would like advice regarding an appropriate univariate feature selection approach I could apply to select the top N features for a regression dataset. I would either like (a) to work out how to make the f_regression function work or (b) to hear alternative suggestions.
The two approaches mentioned above:
I tried using sklearn.feature_selection.f_regression(X,Y).
This failed with the following error message:
"TypeError: copy() takes exactly 1 argument (2 given)"
I tried using chi2(X,Y). This "worked" but I suspect this is because the two response values 0.1 and 1.8 in my toy dataset were being treated as class labels? Presumably, this would not yield a meaningful chi-squared statistic for a real dataset for which there would be a large number of possible response values and the number in each cell [with a particular response value and value for the attribute being tested] would be low?
Please find my toy dataset pasted into the end of this message.
The following code snippet should give the results I describe above.
from sklearn.datasets import load_svmlight_file
X_train_data, Y_train_data = load_svmlight_file(svmlight_format_train_file) #i.e. change this to the name of my toy dataset file
from sklearn.feature_selection import SelectKBest
featureSelector = SelectKBest(score_func="one of the two functions I refer to above",k=2) #sorry, I hope this message is clear
featureSelector.fit(X_train_data,Y_train_data)
print [1+zero_based_index for zero_based_index in list(featureSelector.get_support(indices=True))] #This should print the indices of the top 2 features
Thanks in advance.
Richard
Contents of my contrived svmlight file - with additional blank lines inserted for clarity:
1.8 1:1.000000 2:1.000000 4:1.000000 6:1.000000#mA
1.8 1:1.000000 2:1.000000#mB
0.1 5:1.000000#mC
1.8 1:1.000000 2:1.000000#mD
0.1 3:1.000000 4:1.000000#mE
0.1 3:1.000000#mF
1.8 2:1.000000 4:1.000000 5:1.000000 6:1.000000#mG
1.8 2:1.000000#mH

As larsmans noted, chi2 cannot be used for feature selection with regression data.
Upon updating to scikit-learn version 0.13, the following code selected the top two features (according to the f_regression test) for the toy dataset described above.
def f_regression(X,Y):
import sklearn
return sklearn.feature_selection.f_regression(X,Y,center=False) #center=True (the default) would not work ("ValueError: center=True only allowed for dense data") but should presumably work in general
from sklearn.datasets import load_svmlight_file
X_train_data, Y_train_data = load_svmlight_file(svmlight_format_train_file) #i.e. change this to the name of my toy dataset file
from sklearn.feature_selection import SelectKBest
featureSelector = SelectKBest(score_func=f_regression,k=2)
featureSelector.fit(X_train_data,Y_train_data)
print [1+zero_based_index for zero_based_index in list(featureSelector.get_support(indices=True))]

You could also try to do feature selection by L1/Lasso regularization. The class specifically designed for this is RandomizedLasso which will train LassoRegression on multiple subsamples of your data and select features that are selected most frequently by these models. You can also just use Lasso, LassoLars or SGDClassifier to do same thing without the benefit of resampling but faster.