Develop Reference

Error when imputing minimum values using SimpleImputer

I'm trying to use the minimum values of each column to replace missing values but keep getting an error. Below is my code:
from sklearn.impute import SimpleImputer
numeric_cols = [X_test.select_dtypes(exclude=['object']).columns]
numeric_df = X_test.select_dtypes(exclude=['object'])
for col in numeric_cols:
my_imputer = SimpleImputer(strategy='constant', fill_value=X_test[col].min())
imputed_numeric_X_test = pd.DataFrame(my_imputer.fit_transform(numeric_df))
imputed_numeric_X_test.columns = numeric_df.columns
This is the error I get when I run it:
ValueError: 'fill_value'=MSSubClass 20.0
LotFrontage 21.0
LotArea 1470.0
OverallQual 1.0
OverallCond 1.0
YearBuilt 1879.0
YearRemodAdd 1950.0
MasVnrArea 0.0
BsmtFinSF1 0.0
BsmtFinSF2 0.0
BsmtUnfSF 0.0
TotalBsmtSF 0.0
1stFlrSF 407.0
2ndFlrSF 0.0
LowQualFinSF 0.0
GrLivArea 407.0
BsmtFullBath 0.0
BsmtHalfBath 0.0
FullBath 0.0
HalfBath 0.0
BedroomAbvGr 0.0
KitchenAbvGr 0.0
TotRmsAbvGrd 3.0
Fireplaces 0.0
GarageYrBlt 1895.0
GarageCars 0.0
GarageArea 0.0
WoodDeckSF 0.0
OpenPorchSF 0.0
EnclosedPorch 0.0
3SsnPorch 0.0
ScreenPorch 0.0
PoolArea 0.0
MiscVal 0.0
MoSold 1.0
YrSold 2006.0
dtype: float64 is invalid. Expected a numerical value when imputing numerical data
What is wrong and how can I fix it?

SimpleImputer only supports a single value for fill_value, not a per-column specification. Adding that was discussed in Issue19783, but passed on, and wouldn't support taking the columnwise minimum anyway. I can't find any discussion to add a custom callable option for strategy, which would seem to be the clearest solution. So I think you're stuck doing it manually or with a custom transformer. To do it somewhat manually, you could use the ColumnTransformer approach specified in the linked Issue.

TypeError: Unable to build `Dense` layer with non-floating point dtype <dtype: 'string'> : My data has no String type data in it

I'm trying to build a Neural Network for classification. I preprocessed all the data and it looks like this:
0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.0 0.0 0.0 0.0 2 0.12436986167881312 -0.426405420419126 1
although everything looks okay and datatype is int or float but I'm still getting the following error:
File "C:\Users\spark\anaconda3\lib\site-packages\tensorflow\python\keras\layers\core.py", line 1002, in build
'dtype %s' % (dtype,))
TypeError: Unable to build `Dense` layer with non-floating point dtype <dtype: 'string'>
Most features are dummies or scaled by standardscaler they are floats. and just to be sure I checked data type of last and 4th last column(which are integers in Bold) they are also integers. So..
why am I getting this error? how do I resolve this.
Below is the code I'm using:
X = dataset.iloc[:, 1:].values
y = dataset.iloc[:, 0].values
pred_set = prediction_set.values
temp_dataset = np.concatenate([X, pred_set], axis=0)
'''Encoding Features'''
index_list = [1,2,4,6,7]
reverse_index = []
for i in range(len(index_list)):
reverse_index.append(index_list[i]-temp_dataset.shape[1])
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
for i in range(len(reverse_index)):
index = reverse_index[i]
ct = ColumnTransformer(transformers = [('encoder', OneHotEncoder(), [index])], remainder = 'passthrough')
temp_dataset = ct.fit_transform(temp_dataset)
X = temp_dataset[:891]
pred_set = temp_dataset[891:]
'''Train - Test split'''
from sklearn.model_selection import train_test_split
Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size = 0.2, random_state = 0)
'''Feature scaling'''
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
Xtrain[:, (-2,-3)] = sc.fit_transform(Xtrain[:, (-2,-3)])
Xtest[:, (-2,-3)] = sc.transform(Xtest[:, (-2,-3)])
pred_set[:, (-2,-3)] = sc.transform(pred_set[:, (-2,-3)])
Xtrain.astype(float)
Xtest.astype(float)
pred_set.astype(float)
import tensorflow as tf
classifier = tf.keras.models.Sequential()
classifier.add(tf.keras.layers.Dense(units= 20, activation='relu'))
classifier.add(tf.keras.layers.Dense(units= 20, activation='relu'))
classifier.add(tf.keras.layers.Dense(units= 1, activation='sigmoid'))
classifier.compile(optimizer='adam', loss = 'binary_crossentropy', metrics=['accuracy'])
classifier.fit(Xtrain, ytrain)

Try to cast your data to float, works both with pandas or numpy:
df.astype(float)
According to comment, do not try to convert just a column, simply convert whole dataset.
In case there are some characters that are not digits, the following code should help:
df[-1] = pd.to_numeric(df[-1], errors='coerce')

During handling of the above exception, another exception occurred when using SHAP to interpret keras neural network model

The x_train looks like this (22 features):
total_amount reward difficulty duration discount bogo mobile social web income ... male other_gender age_under25 age_25_to_35 age_35_to_45 age_45_to_55 age_55_to_65 age_65_to_75 age_75_to_85 age_85_to_105
0 0.006311 0.2 0.50 1.000000 1.0 0.0 1.0 1.0 1.0 0.355556 ... 1.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0
1 0.015595 0.2 0.50 1.000000 1.0 0.0 1.0 1.0 1.0 0.977778 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0
The label is 0 and 1, it's a binary classification problem, here's the code for building the model, and I was following this page to implement SHAP:
#use SHAG
deep_explainer = shap.DeepExplainer(nn_model_2, x_train[:100])
# explain the first 10 predictions
# explaining each prediction requires 2 * background dataset size runs
shap_values = deep_explainer.shap_values(x_train)
This gave me error:
KeyError: 0
During handling of the above exception, another exception occurred
I have no idea what this message is complaining, I tried to use SHAP with a XGBoost and Logistic Regression model and they both work fine, I'm new to keras and SHAP, can someone have a look for me and how I can solved it? Many thanks.

I think SHAP (whatever it is) is expecting a Numpy array and so indexing x_train like a Numpy array, it yields an error. Try:
shap_values = deep_explainer.shap_values(x_train.values)

converting to numeric data for decision tree from a matrix?

I have some data in a form of numpy array as follows:
array([['vhigh', '2', '2', 'small', 'low', 'unacc'],
['vhigh', '2', '2', 'small', 'med', 'unacc'],
['vhigh', '2', '2', 'small', 'high', 'good']], dtype=object)
that is extracted from the car dataset available at:
https://archive.ics.uci.edu/ml/datasets/Car+Evaluation
I want to use this data to apply a classification decision tree by using scikit and I managed to convert the first column, or category, into a set of numbers with:
y=data[:,0]
y=le.fit_transform(y)
print y
because I was getting an error that said:
could not convert string to float
the problem that I have is when I want to convert the array into one hot encoding. I have done the following:
X=data[:,1:]
enc=preprocessing.LabelEncoder()
enc.fit(X)
Xn=enc.transform(X)
Xn=Xn.reshape(-1,1)
ohe=preprocessing.OneHotEncoder(sparse=False)
and the error I get is:
bad input shape (1728L, 6L)
What am I doing wrong? or is there another way around to convert from categorical to numeric an array?
Thanks

For the last sklearn version (>20.0) you can just use OneHotEncoder:
df = pd.read_csv(
'https://archive.ics.uci.edu/ml/machine-learning-databases/car/car.data', header=None)
X, y = df.iloc[:,1:] , df.iloc[:,0]
encoded_y = preprocessing.LabelEncoder().fit_transform(y)
sklearn >= 20.0:
ohe = preprocessing.OneHotEncoder(sparse=False)
encoded_x = ohe.fit_transform(X)
>>> pd.DataFrame(encoded_x, columns=ohe.get_feature_names())
x0_high x0_low x0_med x0_vhigh x1_high x1_low ...
0 0.0 0.0 0.0 1.0 0.0 0.0 ...
1 0.0 0.0 0.0 1.0 0.0 0.0 ...
2 0.0 0.0 0.0 1.0 0.0 0.0 ...
3 0.0 0.0 0.0 1.0 0.0 0.0 ...
4 0.0 0.0 0.0 1.0 0.0 0.0 ...
5 0.0 0.0 0.0 1.0 0.0 0.0 ...
...
sklearn < 20.0:
from sklearn.feature_extraction import DictVectorizer
dv = DictVectorizer(sparse=False)
encoded_x = dv.fit_transform(X.to_dict(orient='records'))
pd.DataFrame(encoded_x, columns=dv.get_feature_names())
Fitting a classifier:
from sklearn.tree import DecisionTreeClassifier
DecisionTreeClassifier().fit(encoded_x, encoded_y)

Understanding Text feature extraction TfidfVectorizer in python scikit-learn

Reading the documentation for text feature extraction in scikit-learn, I am not sure how the different arguments available for TfidfVectorizer (and may be other vectorizers) affect the outcome.
Here are the arguments I am not sure how they work:
TfidfVectorizer(stop_words='english', ngram_range=(1, 2), max_df=0.5, min_df=20, use_idf=True)
The documentation is clear on the use of stop_words/ max_df (both have similar effect and may be one can be used instead of the other). However, I am not sure if these options should be used together with ngrams. Which one occurs/handled first, ngrams or stop_words? why? Based on my experiment, stop words are removed first, but the purpose of ngrams is to extract phrases, etc. I am not sure about the effect of this sequence (Stops removed and then ngramed).
Second, does it make sense to use max_df/min_df arguments together with use_idf argument? aren't the purpose of these similar?

I see several questions in this post.
How do the different arguments in TfidfVectorizer interact with one another?
You really have to use it quite a bit to develop a sense of intuition (has been my experience anyway).
TfidfVectorizer is a bag of words approach. In NLP, sequences of words and their window is important; this kind of destroys some of that context.
How do I control what tokens get outputted?
Set ngram_range to (1,1) for outputting only one-word tokens, (1,2) for one-word and two-word tokens, (2, 3) for two-word and three-word tokens, etc.
ngram_range works hand-in-hand with analyzer. Set analyzer to "word" for outputting words and phrases, or set it to "char" to output character ngrams.
If you want your output to have both "word" and "char" features, use sklearn's FeatureUnion. Example here.
How do I remove unwanted stuff?
Use stop_words to remove less-meaningful english words.
The list of stop words that sklearn uses can be found at:
from sklearn.feature_extraction.stop_words import ENGLISH_STOP_WORDS
The logic of removing stop words has to do with the fact that these words don't carry a lot of meaning, and they appear a lot in most text:
[('the', 79808),
('of', 40024),
('and', 38311),
('to', 28765),
('in', 22020),
('a', 21124),
('that', 12512),
('he', 12401),
('was', 11410),
('it', 10681),
('his', 10034),
('is', 9773),
('with', 9739),
('as', 8064),
('i', 7679),
('had', 7383),
('for', 6938),
('at', 6789),
('by', 6735),
('on', 6639)]
Since stop words generally have a high frequency, it might make sense to use max_df as a float of say 0.95 to remove the top 5% but then you're assuming that the top 5% is all stop words which might not be the case. It really depends on your text data. In my line of work, it's very common that the top words or phrases are NOT stop words because I work with dense text (search query data) in very specific topics.
Use min_df as an integer to remove rare-occurring words. If they only occur once or twice, they won't add much value and are usually really obscure. Furthermore, there's generally a lot of them so ignoring them with say min_df=5 can greatly reduce your memory consumption and data size.
How do I Include stuff that's being stripped out?
token_pattern uses a regex pattern \b\w\w+\b which means that tokens have to be at least 2 characters long so words like "I", "a" are removed and also numbers like 0 - 9 are removed. You'll also notice it removes apostrophes
What happens first, ngram generation or stop word removal?
Let's do a little test.
import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.feature_extraction.stop_words import ENGLISH_STOP_WORDS
docs = np.array(['what is tfidf',
'what does tfidf stand for',
'what is tfidf and what does it stand for',
'tfidf is what',
"why don't I use tfidf",
'1 in 10 people use tfidf'])
tfidf = TfidfVectorizer(use_idf=False, norm=None, ngram_range=(1, 1))
matrix = tfidf.fit_transform(docs).toarray()
df = pd.DataFrame(matrix, index=docs, columns=tfidf.get_feature_names())
for doc in docs:
print(' '.join(word for word in doc.split() if word not in ENGLISH_STOP_WORDS))
This prints out:
tfidf
does tfidf stand
tfidf does stand
tfidf
don't I use tfidf
1 10 people use tfidf
Now let's print df:
10 and does don for in is \
what is tfidf 0.0 0.0 0.0 0.0 0.0 0.0 1.0
what does tfidf stand for 0.0 0.0 1.0 0.0 1.0 0.0 0.0
what is tfidf and what does it stand for 0.0 1.0 1.0 0.0 1.0 0.0 1.0
tfidf is what 0.0 0.0 0.0 0.0 0.0 0.0 1.0
why don't I use tfidf 0.0 0.0 0.0 1.0 0.0 0.0 0.0
1 in 10 people use tfidf 1.0 0.0 0.0 0.0 0.0 1.0 0.0
it people stand tfidf use \
what is tfidf 0.0 0.0 0.0 1.0 0.0
what does tfidf stand for 0.0 0.0 1.0 1.0 0.0
what is tfidf and what does it stand for 1.0 0.0 1.0 1.0 0.0
tfidf is what 0.0 0.0 0.0 1.0 0.0
why don't I use tfidf 0.0 0.0 0.0 1.0 1.0
1 in 10 people use tfidf 0.0 1.0 0.0 1.0 1.0
what why
what is tfidf 1.0 0.0
what does tfidf stand for 1.0 0.0
what is tfidf and what does it stand for 2.0 0.0
tfidf is what 1.0 0.0
why don't I use tfidf 0.0 1.0
1 in 10 people use tfidf 0.0 0.0
Notes:
use_idf=False, norm=None when these are set, it's equivalent to using sklearn's CountVectorizer. It will just return counts.
Notice the word "don't" was converted to "don". This is where you'd change token_pattern to something like token_pattern=r"\b\w[\w']+\b" to include apostrophes.
we see a lot of stop words
Let's remove stopwords and look at df again:
tfidf = TfidfVectorizer(use_idf=False, norm=None, stop_words='english', ngram_range=(1, 2))
Outputs:
10 10 people does does stand \
what is tfidf 0.0 0.0 0.0 0.0
what does tfidf stand for 0.0 0.0 1.0 0.0
what is tfidf and what does it stand for 0.0 0.0 1.0 1.0
tfidf is what 0.0 0.0 0.0 0.0
why don't I use tfidf 0.0 0.0 0.0 0.0
1 in 10 people use tfidf 1.0 1.0 0.0 0.0
does tfidf don don use people \
what is tfidf 0.0 0.0 0.0 0.0
what does tfidf stand for 1.0 0.0 0.0 0.0
what is tfidf and what does it stand for 0.0 0.0 0.0 0.0
tfidf is what 0.0 0.0 0.0 0.0
why don't I use tfidf 0.0 1.0 1.0 0.0
1 in 10 people use tfidf 0.0 0.0 0.0 1.0
people use stand tfidf \
what is tfidf 0.0 0.0 1.0
what does tfidf stand for 0.0 1.0 1.0
what is tfidf and what does it stand for 0.0 1.0 1.0
tfidf is what 0.0 0.0 1.0
why don't I use tfidf 0.0 0.0 1.0
1 in 10 people use tfidf 1.0 0.0 1.0
tfidf does tfidf stand use \
what is tfidf 0.0 0.0 0.0
what does tfidf stand for 0.0 1.0 0.0
what is tfidf and what does it stand for 1.0 0.0 0.0
tfidf is what 0.0 0.0 0.0
why don't I use tfidf 0.0 0.0 1.0
1 in 10 people use tfidf 0.0 0.0 1.0
use tfidf
what is tfidf 0.0
what does tfidf stand for 0.0
what is tfidf and what does it stand for 0.0
tfidf is what 0.0
why don't I use tfidf 1.0
1 in 10 people use tfidf 1.0
Take-aways:
the token "don use" happened because don't I use had the 't stripped off and because I was less than two characters, it was removed so the words were joined to don use... which actually wasn't the structure and could potentially change the structure a bit!
Answer: stop words are removed, short characters are removed, then ngrams are generated which can return unexpected results.
does it make sense to use max_df/min_df arguments together with use_idf argument?
My opinion, the whole point of term-frequency inverse document frequency is to allow re-weighting of the highly frequent words (words that would appear a the top of a sorted frequency list). This re-weighting will take the highest frequency ngrams and move them down the list to a lower position. Therefore, it's supposed to handle max_df scenarios.
Maybe it's more of a personal choice whether you want to move them down the list ("re-weight" / de-prioritize them) or remove them completely.
I use min_df a lot and it makes sense to use min_df if you're working with a huge dataset because rare words won't add value and will just cause a lot of processing issues. I don't use max_df much but I'm sure there are scenarios when working with data like all of Wikipedia that this might make sense to remove the top x%.

The stop word removal will not affect your ngrams. A vocabulary (tokens) list is first created according to your tokenizer and ngram range, then stop words are removed from this list (so only unigrams will be affected as the stop word list contains ungrams only). Note that it is not the same if you remove the stop words in the tokenization step (what people often do), then they won't be included in the bigrams either.
Using min_df may in fact counter the effect of tf idf as a word that appeared maybe twice in only one document will have a high score (remember scores are for a document). It depends on the application of your system (information retrieval/ text categorization). If the threshold is low, it shouldn't affect a lot text classification, but retrieval might be biased (what if I want to find documents with "Spain" and it only appears once, in one document, in the entire collection?). Max_df is affected thanks to use_idf as you said, but if you remove the word from vocabulary it might have an stronger impact than only weighting it low. It depends again on what you plan to do with the weights.
Hope this helps.