GaussianNormalizerperforms a statistical transformation on numerical data. It approximates the shape of the overall column. Then, it converts the data to a different shape: a standard normal distribution (aka a bell curve with mean = 0 and standard deviation = 1).
from rdt.transformers.numerical import GaussianNormalizer
gn = GaussianNormalizer()
missing_value_replacement: Add this argument to replace missing values during the transform phase
model_missing_values: Add this argument to create another column describing whether the values are missing
distribution: In the first step of the normalization, the transformer approximates the shape (aka distribution) of the overall column after searching through multiple options. Use this parameter to limit the options it searches through.
enforce_min_max_values: Add this argument to allow the transformer to learn the min and max allowed values from the data.
learn_rounding_scheme: Add this argument to allow the transformer to learn about rounded values in your dataset.
Your decision to use this transformer is based on how you plan to use the transformed data. For example, algorithms such as the Gaussian Copula require normalized data. If you're planning to use such an algorithm, this transformer might be a good pre-processing step.
Below are some definitions for the mathematical terms we've used in this doc.
- A distribution is mathematical formula that describes the overall shape of data. A distribution has parameters that precisely describe it. For example a bell curve is a distribution with parameters for mean and standard deviation.
- A parametric distribution is a distribution that has a preset number of parameters with specific meanings. For example, a bell curve is a parametric distribution because we know it has 2 parameters (mean and standard deviation)
- A gaussian or standard normal distribution is a bell curve with mean = 0 and standard deviation = 1. Other distribution names such as gamma, beta and student t have precise meanings. Refer to this list of probability distributions for more info.
GaussianNormalizerapproximates the column's shape (aka distribution) by searching through multiple options. The more accurate the approximation, the better the accuracy. However, there is a tradeoff between accuracy and the transformation time.
- Searching through more distributions takes a longer time but leads to greater accuracy. To save time, you can input a specific distribution if you already know the specific shape of the column.
- Searching through parametric distributions is faster that non-parametric distributions but can have lower accuracy. For the highest accuracy (that takes the longest amount of time) use the non-parametric
When setting the
model_missing_valuesparameter, consider whether the "missingness" of the data is something important. For example, maybe the user opted out of supplying the info on purpose, or maybe a missing value is highly correlated with another column your dataset. If "missingness" is something you want to account for, you should model missing values.