Develop Reference

How do I remove skewness from a distribution?

I am working with the most famous Credit Card Fraud Detection dataset which includes 28 PCA transformed columns. I'm dealing with the most skewed feature of all which after running the following snippet of code turns out to be V28:
abs_skew_values = pca.skew().abs().sort_values(ascending=False)
selected_feature = abs_skew_values.index[0] # index[0]: most skewed feature
selected_feature # 'V28'
pca is the Pandas DataFrame containing the entire dataset with the PCA columns (V1, V2, V3, etc.).
Now, I wanted to test two things:
How much does the original distribution resemble a normal distribution?
How much skeweness (left or right) is there in the original distribution?
The first thing I have done is plot the histogram of the feature V28:
There are a lot of data points far from 0, these are right skewing the distribution with a score of 11.192. Also, tons of outliers outside of the boxplot fences.
I fixed this by applying a log transformation sign(x) * log(|x|) rather than plain log(x) because there are negative values in the distribution.
It significantly reduced the skew score to 0.184 and you can see less outliers in the distribution.
Running some normality tests also give an insight into how this is clearly not coming from a normal distribution.
Anderson-Darling test
---------------------
15.000: 0.576, data does not look normal (reject H0)
10.000: 0.656, data does not look normal (reject H0)
5.000: 0.787, data does not look normal (reject H0)
2.500: 0.918, data does not look normal (reject H0)
1.000: 1.092, data does not look normal (reject H0)
D'Agostino K^2 test
-------------------
statistic=96189.836, pvalue=0.000
It turns out that, after the log transformation, there are only 26 outliers that may (or may not) be outliers in other features, therefore I don't think I can outright remove them from the original dataset.
So, my question is, am I right in assuming that the transformation I applied is enough to correct the skewness that originally came from the given distribution?
Bonus points: Why is the pvalue in D'Agostino's test exactly 0, shouldn't it be a small number?

Removing Outliers in a Multi-feature Regression Problem

I have a regression problem having 1 target and 10 features. When I look at the outliers for each feature by box-plot, they have different number of outliers. While removing outliers, do I need to also remove the associated target values with those outliers?
I mean, let's say: for #1 feature I have 12 outliers and I removed them with 12 target values. Then, for #2 feature I have 23 outliers and I removed them with 23 target values, as well, and so on. The procedure would be like this, or how should I proceed?

I imagine each row of your data contains an ID, the value of the target and 10 feature values, one of each feature. To answer our question: if you want to remove the outliers, you have to remove the whole observation/row - the value that you classify as an outlier, the corresponding target value, as well as all other 9 corresponding feature values. So you would have to filter each row for the entry of feature_i being smaller than the threshold_i that you defined as an outlier.
The reason is that a multilinear regression calculates the influence of an incremental change in one feature on the target, assuming all other 9 features being constant. Removing a single feature value without removing the target and the other features of this observation simply does not work in such a model (assuming you are using an OLS).
However, I would be cautious with removing outliers. I don't know about your sample size and what you consider an outlier and it would help to know more about your research question, data and methodology.

Correct way of normalizing and scaling the MNIST dataset

I've looked everywhere but couldn't quite find what I want. Basically the MNIST dataset has images with pixel values in the range [0, 255]. People say that in general, it is good to do the following:
Scale the data to the [0,1] range.
Normalize the data to have zero mean and unit standard deviation (data - mean) / std.
Unfortunately, no one ever shows how to do both of these things. They all subtract a mean of 0.1307 and divide by a standard deviation of 0.3081. These values are basically the mean and the standard deviation of the dataset divided by 255:
from torchvision.datasets import MNIST
import torchvision.transforms as transforms
trainset = torchvision.datasets.MNIST(root='./data', train=True, download=True)
print('Min Pixel Value: {} \nMax Pixel Value: {}'.format(trainset.data.min(), trainset.data.max()))
print('Mean Pixel Value {} \nPixel Values Std: {}'.format(trainset.data.float().mean(), trainset.data.float().std()))
print('Scaled Mean Pixel Value {} \nScaled Pixel Values Std: {}'.format(trainset.data.float().mean() / 255, trainset.data.float().std() / 255))
This outputs the following
Min Pixel Value: 0
Max Pixel Value: 255
Mean Pixel Value 33.31002426147461
Pixel Values Std: 78.56748962402344
Scaled Mean: 0.13062754273414612
Scaled Std: 0.30810779333114624
However clearly this does none of the above! The resulting data 1) will not be between [0, 1] and will not have mean 0 or std 1. In fact this is what we are doing:
[data - (mean / 255)] / (std / 255)
which is very different from this
[(scaled_data) - (mean/255)] / (std/255)
where scaled_data is just data / 255.

Euler_Salter
I may have stumbled upon this a little too late, but hopefully I can help a little bit.
Assuming that you are using torchvision.Transform, the following code can be used to normalize the MNIST dataset.
train_loader = torch.utils.data.DataLoader(
datasets.MNIST('./data', train=True
transform=transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))
])),
Usually, 'transforms.ToTensor()' is used to turn the input data in the range of [0,255] to a 3-dimensional Tensor. This function automatically scales the input data to the range of [0,1]. (This is equivalent to scaling the data down to 0,1)
Therefore, it makes sense that the mean and std used in the 'transforms.Normalize(...)' will be 0.1307 and 0.3081, respectively. (This is equivalent to normalizing zero mean and unit standard deviation.)
Please refer to the link below for better explanation.
https://pytorch.org/vision/stable/transforms.html

I think you misunderstand one critical concept: these are two different, and inconsistent, scaling operations. You can have only one of the two:
mean = 0, stdev = 1
data range [0,1]
Think about it, considering the [0,1] range: if the data are all small positive values, with min=0 and max=1, then the sum of the data must be positive, giving a positive, non-zero mean. Similarly, the stdev cannot be 1 when none of the data can possibly be as much as 1.0 different from the mean.
Conversely, if you have mean=0, then some of the data must be negative.
You use only one of the two transformations. Which one you use depends on the characteristics of your data set, and -- ultimately -- which one works better for your model.
For the [0,1] scaling, you simply divide by 255.
For the mean=0, stdev=1 scaling, you perform the simple linear transformation you already know:
new_val = (old_val - old_mean) / old_stdev
Does that clarify it for you, or have I entirely missed your point of confusion?

Purpose
Two of the most important reasons for features scaling are:
You scale features to make them all of the same magnitude (i.e. importance or weight).
Example:
Dataset with two features: Age and Weight. The ages in years and the weights in grams! Now a fella in the 20th of his age and weights only 60Kg would translate to a vector = [20 yrs, 60000g], and so on for the whole dataset. The Weight Attribute will dominate during the training process. How is that, depends on the type of the algorithm you are using - Some are more sensitive than others: E.g. Neural Network where the Learning Rate for Gradient Descent get affected by the magnitude of the Neural Network Thetas (i.e. Weights), and the latter varies in correlation to the input (i.e. features) during the training process; also Feature Scaling improves Convergence. Another example is the K-Mean Clustering Algorithm requires Features of the same magnitude since it is isotropic in all directions of space. INTERESTING LIST.
You scale features to speed up execution time.
This is straightforward: All these matrices multiplications and parameters summation would be faster with small numbers compared to very large number (or very large number produced from multiplying features by some other parameters..etc)
Types
The most popular types of Feature Scalers can be summarized as follows:
StandardScaler: usually your first option, it's very commonly used. It works via standardizing the data (i.e. centering them), that's to bring them to a STD=1 and Mean=0. It gets affected by outliers, and should only be used if your data have Gaussian-Like Distribution.
MinMaxScaler: usually used when you want to bring all your data point into a specific range (e.g. [0-1]). It heavily gets affected by outliers simply because it uses the Range.
RobustScaler: It's "robust" against outliers because it scales the data according to the quantile range. However, you should know that outliers will still exist in the scaled data.
MaxAbsScaler: Mainly used for sparse data.
Unit Normalization: It basically scales the vector for each sample to have unit norm, independently of the distribution of the samples.
Which One & How Many
You need to get to know your dataset first. As per mentioned above, there are things you need to look at before, such as: the Distribution of the Data, the Existence of Outliers, and the Algorithm being utilized.
Anyhow, you need one scaler per dataset, unless there is a specific requirement, such that if there exist an algorithm that works only if data are within certain range and has mean of zero and standard deviation of 1 - all together. Nevertheless, I have never come across such case.
Key Takeaways
There are different types of Feature Scalers that are used based on some rules of thumb mentioned above.
You pick one Scaler based on the requirements, not randomly.
You scale data for a purpose, for example, in the Random Forest Algorithm you do NOT usually need to scale.

Well the data gets scaled to [0,1] using torchvision.transforms.ToTensor() and then the normalization (0.1306,0.3081) is applied.
You can look about it in the Pytorch documentation : https://pytorch.org/vision/stable/transforms.html.
Hope that answers your question.

Scipy Detrend in python

I detrended my data in python using the following code from scipy.signal.detrend
detrended =signal.detrend(feature, axis=-1, type='constant', bp=0, overwrite_data=True)
np.savetxt('constant detrend.csv', detrended, delimiter=',', fmt='%s')
The last line saves the data into a csv file then i reload this data to run some models. I found that the my RandomForest model performs really well with the detrended dataset.
So next will be to make forecasts using this model. However i am a bit unsure of how i can move from the detrended dataset to a more meaningful dataset that i can understand. From my understanding the detrend removed the mean and normalized the data. But if i do my predictions i need to be able to see the actual numbers of my forecasts not the detrended numbers.
Is there a way i can readd the mean and renormalize to get a 'meanful dataset' that i can interpret. For example my dataset has a rainfall variable. So for each month i can see how much it rained. But when i detrended, the rainfall value is no longer the actual rainfall value. When i make forecasts i want to be able to say in this month it rained 200mm but my forecasts don't tell me this since the data has been detrended.
Any help would be appraciated.

According to the docs, detrend simply removes the least squares line fit from the data. When you use type='constant', it's even simpler, since it just removes the mean:
If type == 'constant', only the mean of data is subtracted.
The source code bears this out. After checking the inputs, the entire computation is done in one line (scipy/signal/signaltools.py, line 3261):
ret = data - np.expand_dims(np.mean(data, axis), axis)
The easiest way to get the subtracted mean is to implement the calculation by hand, given how simple it is.
mean = np.mean(feature, axis=-1, keepdims=True)
detrended = feature - mean
You can save the mean to a file, or do whatever else you want with it. To "retrend", just add the mean back:
point = prediction + mean
If you had some other manipulation you were concerned with, like normalizing to the maximum, you could handle it the same way.
max = np.amax(detrended, axis=-1, keepdims=True)
detrended /= max
In this case you'd have to multiply before offsetting to retrend:
point = prediction * max + mean
Simple manipulations like this are easy to reproduce by hand. A more complicated function might be hard to reproduce reliably, but would also be more likely to return the parameters it uses, at least optionally.

Normalizations in sklearn and their differences

I have read many articles suggested this formula
N = (x - min(x))/(max(x)-min(x))
for normalization
but when i dig into the normalizor of sklearn somewhere i found they are using this formula
x / np.linalg.norm(x)
As the later use l2-norm by default. Which one should I use? Why is there a difference in between both?

There are different normalization techniques and sklearn provides for many of them. Please note that we are looking at 1d arrays here. For a matrix these operations are applied to each column (have a look at this post for an in depth example Scaling features for machine learning) Let's go through some of them:
Scikit-learn's MinMaxScaler performs (x - min(x))/(max(x)-min(x)) This scales your array in such a way that you only have values between 0 and 1. Can be useful if you want to apṕly some transformation afterwards where no negative values are allowed (e.g. a log-transform or in scaling RGB pixels like done in some MNIST examples)
scikit-learns StandardScaler performs (x-x.mean())/x.std() which centers the array around zero and scales by the variance of the features. This is a standard transformation and is appicable in many situations but keep in mind that you will get negative values. This is especially useful when you have gaussian sampled data which is not centered around 0 and/or does not have a unit variance.
Scikit-learn's Normalizer performs x / np.linalg.norm(x). This sets the length of your array/vector to 1. Might come in handy if you want to do some linear algebra stuff like if you want to implement the Gram-Schmidt Algorithm.
Scikit-learn's RobustScaler can be used to scale data with outliers. Mean and standard deviation are not robust to outliers therefore this scaler uses the median and scales the data to quantile ranges.
There are other non-linear transformations like QuantileTransformer that scales be quantile ranges and PowerTransformer that maps any distribution to a distribution similar to a Gaussian distribution.
And there are many other normalizations used in machine learning and there vast amount can be confusing. The idea behind normalizing data in ML is usually that you want dont want your model to treat one feature differently than others simply because it has a higher mean or a larger variance. For most standard cases I use MinMaxScaler or StandardScaler depending on whether scaling according to the variance seems important to me.

np.ling.norm is given by:
np.linalg.norm(x) = sqrt((sum_i_j(abs(x_i_j)))^2)
so lets assume you have:
X= (1 2
0 -1)
then with this you would have:
np.linalg.norm(x)= sqr((1+2+0+1)^2)= sqr(16)=4
X = (0.25 0.5
0 -0.25)
with the other approach you would have:
min(x)= -1
max(x)= 2
max(x)-min(x)=3
X = (0.66 1
0.33 0)
So the min(x)/max(x) is also called MinMaxScaler, there all the values are always between 0-1, the other approaches normalizes your values , but you can still have negativ values. Depending on your next steps you need to decide which one to use.

Based on the API description
Scikit-learn normalizer scales input vectors individually to a unit norm (vector length).
That is why it uses the L2 regularizer (you can also use L1 as well, as explained in the API)
I think you are looking for a scaler instead of a normalizer by your description. Please find the Min-Max scaler in this link.
Also, you can consider a standard scaler that normalizes value by removing its mean and scales to its standard deviation.