Develop Reference

Area under the histogram is not 1 when using density in plt.hist

Consider the following dataset with random data:
test_dataset = np.array([ -2.09601881, -4.26602684, 1.09105452, -4.59559669,
1.05865251, -0.93076762, -14.70398945, -18.01937129,
4.64126152, -10.34178822, -9.46058493, -5.66864965,
-3.17562022, 15.7030379 , 10.59675205, -5.80882413,
-24.00604149, -4.81518663, -1.94333927, 1.18142171,
12.72030312, 3.84917581, -0.4468796 , 11.91828567,
-17.99171774, 9.35108712, -5.57233376, 5.77547128,
5.49296099, -10.96132844, -18.75174336, 5.27843303,
25.73548956, -21.58043021, -14.24734733, 12.57886018,
-22.10002076, 1.72207555, -6.0411867 , -3.63568527,
7.26542117, -0.21449529, -6.64974714, -0.94574606,
-4.23339431, 16.76199734, -12.42195793, 18.965854 ,
-23.85336123, -15.55104466, 6.17215868, 7.34993316,
8.62461351, -16.30482638, -16.35601099, 1.96857833,
18.74440399, -22.48374434, -10.895831 , -10.14393648,
-17.62768751, 4.83388855, 20.1578181 , 6.04299626,
0.97198296, -3.40889754, -10.62734293, 1.70240472,
20.4203839 , 10.26751364, 15.47859675, -10.97940064,
1.82728251, 4.22894717, 8.31502887, -5.48502811,
-1.09244874, -11.32072796, -24.88520436, -7.42108403,
19.4200716 , 4.82704045, -12.46290135, -15.18466755,
6.37714692, -11.06825059, 5.10898588, -9.07485484,
1.63946084, -12.2270078 , 12.63776832, -25.03916909,
2.42972082, -14.22890171, 18.2199446 , 6.9819771 ,
-12.07795089, 2.59948596, -16.90206575, 6.35192719,
7.33823106, -23.69653447, -11.66091871, -19.40251179,
-12.64863792, 11.04004231, 13.7247356 , -16.36107329,
20.43227515, 17.97334692, 16.92675175, -5.62051239,
-8.66304184, -8.40848514, -23.20919855, 0.96808137,
-5.03287253, -3.13212582, 18.81155666, -8.27988284,
3.85708447, 12.43039322, 17.98003878, 18.11009997,
-3.74294421, -16.62276121, 9.4446743 , 2.2060981 ,
8.34853736, 14.79144713, -1.91113975, -5.17061419,
4.53451746, 8.19090358, 7.98343201, 11.44592322,
-16.9132677 , -25.92554857, 10.10638432, -8.09236786,
20.8878207 , 19.52368296, 0.85858125, 2.61760415,
9.21360649, -8.1192651 , -6.94829273, 2.73562447,
13.40981323, -9.05018331, -17.77563166, -21.03927199,
4.10415845, -1.31550732, 5.68284828, 15.08670773,
-19.78675315, 12.94697869, -11.51797637, 1.91485992,
16.69417993, -16.04271622, -1.14028558, 9.79830109,
-18.58386093, -7.52963269, -10.10059878, -25.2194216 ,
-0.10598426, -15.77641532, -14.15999125, 14.35011271,
11.15178588, -14.43856266, 15.84015226, -3.41221883,
11.90724469, 0.57782081, 18.82127466, -6.01068727,
-19.83684476, 2.20091942, -1.38707755, -8.62821053,
-11.89000913, -11.69539815, 5.70242019, -3.83781841,
5.35894135, -0.30995954, 21.76661212, 8.52974329,
-9.13065082, -11.06209 , -12.00654618, 2.769838 ,
-12.21579496, -27.2686534 , -4.58538197, -6.94388425])
I'd like to plot normalized histogram of it, so in the plt.hist options I choose density=True:
import numpy as np
import matplotlib.pyplot as plt
data1, bins, _ = plt.hist(test_dataset, density=True);
print(np.trapz(data1))
print(sum(data1))
which outputs the following histogram:
0.18206124014272715
0.18866449755723017
From matplotlib documentation:
The density parameter, which normalizes bin heights so that the integral of the histogram is 1. The resulting histogram is an approximation of the probability density function.
But from my example it is clearly seen that the integral of the histogram is NOT 1 and strongly depends on the number of bins: if I specify it for example to be 40 the sum will increase:
data1, bins, _ = plt.hist(test_dataset, density=True);
print(np.trapz(data1))
print(sum(data1))
0.7508847002777762
0.7546579902289207
Is it incorrect description in documentation or I misunderstand some issues here?

you do not calculate the area, area you should calculate as follow (in your example):
sum(data1 * np.diff(bins)) == 1

Python OptBinning package's OptimalBinning and BinningProcess giving different results sometimes

I'm using the OptBinning package to bin some numeric data. I'm following this example to do this. And from this tutorial I read that "... the best way to view BinningProcess is as a wrapper for OptimalBinning", which implies that they should both give the same outputs. However, I'm seeing they give different outputs for some features and the same for others. Why is this the case? Below is an example showing how the two methods lead to the same output for 'mean radius' but not 'worst radius' using the breast cancer data in sklearn.
import pandas as pd
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from optbinning import BinningProcess
from optbinning import OptimalBinning
# Load data
data = load_breast_cancer()
df = pd.DataFrame(data.data, columns=data.feature_names)
# Bin 'mean radius' data using OptimalBinning method
var = 'mean radius'
x = df[var]
y = data.target
optb = OptimalBinning(name=var, dtype="numerical")
optb.fit(x, y)
binning_table = optb.binning_table
binning_table.build()['WoE']
0 -3.12517
1 -2.71097
2 -1.64381
3 -0.839827
4 -0.153979
5 2.00275
6 5.28332
7 0
8 0
Totals
Name: WoE, dtype: object
# Bin 'mean radius' using BinningProcess method
var = ['mean radius']
bc_pipe = Pipeline([('WOE Binning', BinningProcess(variable_names=var))])
preprocessor = ColumnTransformer([('Numeric Pipeline', bc_pipe, var)], remainder='passthrough')
preprocessor.fit(df, y)
df_processed = preprocessor.transform(df)
df_processed = pd.DataFrame(df_processed, columns=df.columns)
df_processed[var[0]].unique()
array([ 5.28332344, -3.12517033, -1.64381421, -0.15397917, 2.00275405,
-0.83982705, -2.71097154])
## We see that the Weight of Evidence (WoE) values are the same for 'mean radius' using both methods (except for the 0's, which we can ignore for now)
# Bin 'worst radius' using OptimalBinning process
var = 'worst radius'
x = df[var]
y = data.target
optb = OptimalBinning(name=var, dtype="numerical")
optb.fit(x, y)
binning_table = optb.binning_table
binning_table.build()['WoE']
0 -4.56645
1 -2.6569
2 -0.800606
3 -0.060772
4 1.61976
5 5.5251
6 0
7 0
Totals
Name: WoE, dtype: object
# Bin 'worst radius' using BinningProcess method
var = ['worst radius']
bc_pipe = Pipeline([('WOE Binning', BinningProcess(variable_names=var))])
preprocessor = ColumnTransformer([('Numeric Pipeline', bc_pipe, var)], remainder='passthrough')
preprocessor.fit(df, y)
df_processed = preprocessor.transform(df)
df_processed = pd.DataFrame(df_processed, columns=df.columns)
df_processed[var[0]].unique()
array([0.006193 , 0.003532 , 0.004571 , 0.009208 , 0.005115 , 0.005082 ,
0.002179 , 0.005412 , 0.003749 , 0.01008 , 0.003042 , 0.004144 ,
0.01284 , 0.003002 , 0.008093 , 0.005466 , 0.002085 , 0.004142 ,
0.001997 , 0.0023 , 0.002425 , 0.002968 , 0.004394 , 0.001987 ,
0.002801 , 0.007444 , 0.003711 , 0.004217 , 0.002967 , 0.003742 ,
0.00456 , 0.005667 , 0.003854 , 0.003896 , 0.003817 , ... ])
## We now see that for 'worst radius' the two WoE's are not the same. Why?

I think the problem is due to the default behaviour of ColumnTransformer option remainder="passthrough". The remaining columns are concatenated, and that's why the position of the transformed variables changes. If you look at the dataframe, the first column contains the WoE values of the feature "worst radius". As an example, please try the following:
binning_process = BinningProcess(variable_names=var)
binning_process.fit(df[var], y)
np.unique(binning_process.transform(df[var]).values)
The binning process, as expected, will return the same WoE values. See also: https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html
By default, only the specified columns in transformers are transformed and combined in the output, and the non-specified columns are dropped. (default of 'drop'). By specifying remainder='passthrough', all remaining columns that were not specified in transformers will be automatically passed through. This subset of columns is concatenated with the output of the transformers.

ValueError: x and y must be the same size In Python while creating KMeans Model

I'm building a Kmeans clustering model with a churn dataset and am getting an error that says ValueError: x and y must be the same size when trying to create cluster graph.
I'll post both my function and the graph code here in a sec, but in trying to narrow it down, I think it may have something to do with this line of code in the function:
x=kmeans.cluster_centers_[:,0]
, y=kmeans.cluster_centers_[:,1]
Here's the full code
def Create_kmeans_cluster_graph(df_final, data, n_clusters, x_title, y_title, chart_title):
""" Display K-means cluster based on data """
kmeans = KMeans(n_clusters=n_clusters # No of cluster in data
, random_state = random_state # Selecting same training data
)
kmeans.fit(data)
kmean_colors = [plotColor[c] for c in kmeans.labels_]
fig = plt.figure(figsize=(12,8))
plt.scatter(x= x_title + '_norm'
, y= y_title + '_norm'
, data=data
, color=kmean_colors # color of data points
, alpha=0.25 # transparancy of data points
)
plt.xlabel(x_title)
plt.ylabel(y_title)
plt.scatter(x=kmeans.cluster_centers_[:,0]
, y=kmeans.cluster_centers_[:,1]
, color='black'
, marker='X' # Marker sign for data points
, s=100 # marker size
)
plt.title(chart_title,fontsize=15)
plt.show()
return kmeans.fit_predict(df_final[df_final.Churn==1][[x_title+'_norm', y_title +'_norm']])
//Graph
df_final['Cluster'] = -1 # by default set Cluster to -1
df_final.iloc[(df_final.Churn==1),'Cluster'] = Create_kmeans_cluster_graph(df_final
,df_final[df_final.Churn==1][['Tenure_norm','MonthlyCharge_norm']]
,3
,'Tenure'
,'MonthlyCharges'
,"Tenure vs Monthlycharges : Churn customer cluster")
df_final['Cluster'].unique()

You get that error because of this line:
plt.scatter(x= x_title + '_norm'
, y= y_title + '_norm'
, data=data
, color=kmean_colors # color of data points
, alpha=0.25 # transparancy of data points
)
If you use plt.scatter, it does not take in data= as an argument, you can read the help page. You can either do:
plt.scatter(data[x_title + '_norm'],data[y_title + '_norm'],...)
Or you use the plot.scatter method on a pandas dataframe, which I did in a edited version of your function:
def Create_kmeans_cluster_graph(df_final, data, n_clusters, x_title, y_title, chart_title):
plotColor = ['k','g','b']
kmeans = KMeans(n_clusters=n_clusters , random_state = random_state)
kmeans.fit(data)
kmean_colors = [plotColor[c] for c in kmeans.labels_]
data.plot.scatter(x= x_title + '_norm', y= y_title + '_norm',
color=kmean_colors,alpha=0.25)
plt.xlabel(x_title)
plt.ylabel(y_title)
plt.scatter(x=kmeans.cluster_centers_[:,0],y=kmeans.cluster_centers_[:,1],
color='black',marker='X',s=100)
return kmeans.labels_
On an example dataset, it works:
import pandas as pd
import numpy as np
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
random_state = 42
np.random.seed(42)
df_final = pd.DataFrame({'Tenure_norm':np.random.uniform(0,1,50),
'MonthlyCharge_norm':np.random.uniform(0,1,50),
'Churn':np.random.randint(0,3,50)})
Create_kmeans_cluster_graph(df_final
,df_final[df_final.Churn==1][['Tenure_norm','MonthlyCharge_norm']]
,3
,'Tenure'
,'MonthlyCharge'
,"Tenure vs Monthlycharges : Churn customer cluster")

Can't get the fit with lmfit

I want to do a fit using lmfit but I am having some issues. Here is my code:
from lmfit import Model
import numpy as np
def fit_func(x,a,b,c):
return a*(b-x)**(5/8)+c
x = np.array([ 131.871 , 218.825 , 305.046 , 390.533 ,
475.128 , 558.959 , 642.001 , 724.307 ,
805.794 , 886.422 , 966.20900001, 1045.19300001,
1123.39300001, 1200.75800001, 1277.23700001, 1352.83300001,
1427.57800001, 1501.49800001, 1574.55300001, 1646.69500001,
1717.90800001, 1788.22100001, 1857.65100001, 1926.18300001,
1993.76400001, 2060.37000001, 2126.00900001, 2190.70600001,
2254.44800001, 2317.20000001, 2378.92000001, 2439.60300001,
2499.25800001, 2557.89000001, 2615.46600001, 2671.95000001,
2727.30900001, 2781.54300001, 2834.64700001, 2886.60600001,
2937.38000001, 2986.92900001])
y = np.array([ 0. , 3.14159265, 6.28318531, 9.42477796,
12.56637061, 15.70796327, 18.84955592, 21.99114858,
25.13274123, 28.27433388, 31.41592654, 34.55751919,
37.69911184, 40.8407045 , 43.98229715, 47.1238898 ,
50.26548246, 53.40707511, 56.54866776, 59.69026042,
62.83185307, 65.97344573, 69.11503838, 72.25663103,
75.39822369, 78.53981634, 81.68140899, 84.82300165,
87.9645943 , 91.10618695, 94.24777961, 97.38937226,
100.53096491, 103.67255757, 106.81415022, 109.95574288,
113.09733553, 116.23892818, 119.38052084, 122.52211349,
125.66370614, 128.8052988 ])
fit_model = Model(fit_func)
params = fit_model.make_params()
params['b'].set(5000, min=3500)
result = fit_model.fit(y, x=x)
But I am getting this error:
ValueError: The model function generated NaN values and the fit aborted! Please check your model function and/or set boundaries on parameters where applicable. In cases like this, using "nan_policy='omit'" will probably not work.
What am I doing wrong? I tried to adjust the a, b, c parameters by hand and a=-1.2, b=3600, c=196 give a pretty good fit, so the program should be able to find something similar to that.

Two things are missing:
a) you need to pass params to fit_model.fit() as with
result = fit_model.fit(y, params, x=x)
b) you need to give initial values for all parameters. Un-initialized parameters will have a value of -np.inf, which is deliberately chosen because it will throw such errors.
You say you know reasonable values for a, b, and c. Use that knowledge! Something like
fit_model = Model(fit_func)
params = fit_model.make_params(a=-1, b=4000, c=200)
params['b'].min = x.max() * (1.000001) # prevent (negative number)**fraction
result = fit_model.fit(y, params, x=x)
print(result.fit_report())
should work.

Tensorflow Grab Predictions and Indices for values above thresholds

What is the easiest way to grab the corresponding prediction values and indices based on those above a certain threshold?
Consider this problem:
sess = tf.InteractiveSession()
predictions = tf.constant([[ 0.32957435, 0.82079124, 0.54503286, 0.51966476, 0.63359714,
0.92034972, 0.13774526, 0.45154464, 0.18284607, 0.14604568],
[ 0.78612137, 0.98291659, 0.4841609 , 0.63260579, 0.21568334,
0.82978213, 0.05054879, 0.09517837, 0.28309393, 0.01788473],
[ 0.05706763, 0.24366784, 0.04608512, 0.32987678, 0.2342416 ,
0.91725373, 0.60084391, 0.51787591, 0.74161232, 0.30830121],
[ 0.67310858, 0.6250236 , 0.42477703, 0.37107778, 0.65123832,
0.97282803, 0.59533679, 0.49564457, 0.54935825, 0.63008392],
[ 0.70233917, 0.48129809, 0.59114349, 0.63535333, 0.71188867,
0.4799161 , 0.90896237, 0.86089945, 0.47896886, 0.83451629],
[ 0.82923532, 0.8950938 , 0.99231505, 0.05526769, 0.98151541,
0.18153167, 0.63851702, 0.07426929, 0.91846335, 0.81246626],
[ 0.12850153, 0.23018432, 0.29871917, 0.71228445, 0.13235569,
0.41061044, 0.98215759, 0.90024149, 0.53385031, 0.92247963],
[ 0.87011361, 0.44218826, 0.01772344, 0.87317121, 0.52231467,
0.86476815, 0.25352192, 0.31709731, 0.38249743, 0.74694788],
[ 0.15262914, 0.49544573, 0.49644637, 0.07461977, 0.13706958,
0.18619633, 0.86163998, 0.03700352, 0.51173556, 0.40018845]])
score_idx = tf.where(predictions > 0.8)
scores = tf.SparseTensor(score_idx, tf.gather_nd(predictions, score_idx), dense_shape=tf.shape(predictions, out_type=tf.int64))
dense_scores = tf.sparse_tensor_to_dense(scores)
print(sess.run([scores, dense_scores]))
I can easily get a sparse tensor that has all of the predictions above 0.8, but ultimately I am looking to return two separate 1D tensors:
Predicted Indices = list of indexes above threshold (0.8 in example)
Scores = the scores for the corresponding examples
So for the first row which is:
[ 0.32957435, 0.82079124, 0.54503286, 0.51966476, 0.63359714,
0.92034972, 0.13774526, 0.45154464, 0.18284607, 0.14604568]
I am looking to return:
predicted_indices = [1,5]
scores = [0.821, 0.920]
Is there a simple solution that I am missing?