Data Preprocessing and Feature Engineering Reference

Targets scikit-learn 1.9.1. Snippets with caller-supplied data/columns are illustrative; fit all learned preprocessing inside the training folds when estimating predictive performance.

Overview

Data preprocessing transforms raw data into a format suitable for machine learning models. This includes scaling, encoding, handling missing values, and feature engineering.

Feature Scaling and Normalization

StandardScaler

**StandardScaler (sklearn.preprocessing.StandardScaler)**

• Standardizes features to zero mean and unit variance
• Formula: z = (x - mean) / std
• Use when: Different feature scales distort a distance, kernel, or regularization penalty; standardization does not make data Gaussian
• Often useful for SVM, KNN, neural networks, PCA, and regularized linear models; choose scales using scientific units and noise. For sparse data use with_mean=False
• Example:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)  # Use same parameters as training

# Access learned parameters
print(f"Mean: {scaler.mean_}")
print(f"Std: {scaler.scale_}")

MinMaxScaler

**MinMaxScaler (sklearn.preprocessing.MinMaxScaler)**

• Scales features to a given range (default [0, 1])
• Formula: X_scaled = (X - X.min) / (X.max - X.min)
• Training values map to the requested range; unseen values can exceed it unless clip=True. Clipping can hide distribution shift
• Sensitive to outliers
• Example:

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler(feature_range=(0, 1))
X_scaled = scaler.fit_transform(X_train)

# Custom range
scaler = MinMaxScaler(feature_range=(-1, 1))
X_scaled = scaler.fit_transform(X_train)

RobustScaler

**RobustScaler (sklearn.preprocessing.RobustScaler)**

• Scales using median and interquartile range (IQR)
• Formula: X_scaled = (X - median) / IQR
• Use when: Data contains outliers
• Robust to outliers
• Example:

from sklearn.preprocessing import RobustScaler

scaler = RobustScaler()
X_scaled = scaler.fit_transform(X_train)

Normalizer

**Normalizer (sklearn.preprocessing.Normalizer)**

• Normalizes samples individually to unit norm
• Common norms: 'l1', 'l2', 'max'
• Use when: Need to normalize each sample independently (e.g., text features)
• Example:

from sklearn.preprocessing import Normalizer

normalizer = Normalizer(norm='l2')  # Euclidean norm
X_normalized = normalizer.fit_transform(X)

MaxAbsScaler

**MaxAbsScaler (sklearn.preprocessing.MaxAbsScaler)**

• Scales by maximum absolute value
• Training range lies within [-1, 1]; new extrema can exceed it
• Doesn't shift/center data (preserves sparsity)
• Use when: Data is already centered or sparse
• Example:

from sklearn.preprocessing import MaxAbsScaler

scaler = MaxAbsScaler()
X_scaled = scaler.fit_transform(X_sparse)

Encoding Categorical Variables

OneHotEncoder

**OneHotEncoder (sklearn.preprocessing.OneHotEncoder)**

• Creates binary columns for each category
• Use when: Nominal categories (no order), tree-based models or linear models
• Example:

from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder(sparse_output=False, handle_unknown='ignore')
X_encoded = encoder.fit_transform(X_categorical)

# Get feature names
feature_names = encoder.get_feature_names_out(['color', 'size'])

# Handle unknown categories during transform
X_test_encoded = encoder.transform(X_test_categorical)

OrdinalEncoder

**OrdinalEncoder (sklearn.preprocessing.OrdinalEncoder)**

• Encodes categories as integers
• Use when: Ordinal categories (ordered), or tree-based models
• Example:

from sklearn.preprocessing import OrdinalEncoder

# Default sorted category order is not necessarily a scientific ordering
encoder = OrdinalEncoder()
X_encoded = encoder.fit_transform(X_categorical)

# Custom ordering
encoder = OrdinalEncoder(categories=[['small', 'medium', 'large']])
X_encoded = encoder.fit_transform(X_categorical)

LabelEncoder

**LabelEncoder (sklearn.preprocessing.LabelEncoder)**

• Encodes target labels (y) as integers
• Use for: Target variable encoding
• Example:

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()
y_encoded = le.fit_transform(y)

# Decode back
y_decoded = le.inverse_transform(y_encoded)
print(f"Classes: {le.classes_}")

Target Encoding (with internal cross-fitting)

from sklearn.preprocessing import TargetEncoder
from sklearn.model_selection import StratifiedKFold

encoder = TargetEncoder(cv=StratifiedKFold(5, shuffle=True, random_state=42))
X_train_encoded = encoder.fit_transform(X_train_categorical, y_train)
X_test_encoded = encoder.transform(X_test_categorical)

fit_transform uses out-of-fold encodings for training rows, while fit(...).transform(...) does not; they intentionally differ. Put the encoder inside the model pipeline for outer CV. In 1.9, custom internal splitters are supported if every row occurs in exactly one validation fold; use metadata routing for groups. Ordinary TimeSeriesSplit does not satisfy that partition requirement. Use a study-specific causal encoding design for forecasting. Plain shuffled folds assume independent rows. The old encoder shuffle and random_state parameters are deprecated; configure the splitter instead. Numeric integer regression targets may be inferred as multiclass, so set target_type='continuous' when appropriate.

Non-linear Transformations

Power Transforms

PowerTransformer

• Makes data more Gaussian-like
• Methods: 'yeo-johnson' (works with negative values), 'box-cox' (positive only)
• Use when: Data is skewed, algorithm assumes normality
• Example:

from sklearn.preprocessing import PowerTransformer

# Yeo-Johnson (handles negative values)
pt = PowerTransformer(method='yeo-johnson', standardize=True)
X_transformed = pt.fit_transform(X)

# Box-Cox (positive values only)
pt = PowerTransformer(method='box-cox', standardize=True)
X_transformed = pt.fit_transform(X)

Quantile Transformation

QuantileTransformer

• Transforms features to follow uniform or normal distribution
• Robust to outliers
• Use when: Want to reduce outlier impact
• Example:

from sklearn.preprocessing import QuantileTransformer

# Transform to uniform distribution
qt = QuantileTransformer(output_distribution='uniform', random_state=42)
X_transformed = qt.fit_transform(X)

# Transform to normal distribution
qt = QuantileTransformer(output_distribution='normal', random_state=42)
X_transformed = qt.fit_transform(X)

Log Transform

import numpy as np

# Log1p (log(1 + x)) - handles zeros
X_log = np.log1p(X)

# Or use FunctionTransformer
from sklearn.preprocessing import FunctionTransformer

log_transformer = FunctionTransformer(np.log1p, inverse_func=np.expm1)
X_log = log_transformer.fit_transform(X)

Missing Value Imputation

SimpleImputer

Default missing_values=np.nan does not normalize every sentinel: convert categorical None/pd.NA consistently. In 1.9 even constant-strategy imputers drop entirely missing columns unless keep_empty_features=True. Decide explicitly whether the schema should preserve those columns.

**SimpleImputer (sklearn.impute.SimpleImputer)**

• Basic imputation strategies
• Strategies: 'mean', 'median', 'most_frequent', 'constant'
• Example:

from sklearn.impute import SimpleImputer

# For numerical features
imputer = SimpleImputer(strategy='mean')
X_imputed = imputer.fit_transform(X)

# For categorical features
imputer = SimpleImputer(strategy='most_frequent')
X_imputed = imputer.fit_transform(X_categorical)

# Fill with constant
imputer = SimpleImputer(strategy='constant', fill_value=0)
X_imputed = imputer.fit_transform(X)

Iterative Imputer

IterativeImputer

• Models each feature with missing values as function of other features
• Experimental: enabling import remains required. Iterative imputation does not itself propagate missing-data uncertainty; inspect convergence and assumptions
• Example:

from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer

imputer = IterativeImputer(max_iter=10, random_state=42)
X_imputed = imputer.fit_transform(X)

KNN Imputer

KNNImputer

• Imputes using k-nearest neighbors
• Use when: Features are correlated
• Example:

from sklearn.impute import KNNImputer

imputer = KNNImputer(n_neighbors=5)
X_imputed = imputer.fit_transform(X)

Feature Engineering

Polynomial Features

PolynomialFeatures

• Creates polynomial and interaction features
• Use when: Need non-linear features for linear models
• Example:

from sklearn.preprocessing import PolynomialFeatures

# Degree 2: includes x1, x2, x1^2, x2^2, x1*x2
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X)

# Get feature names
feature_names = poly.get_feature_names_out(['x1', 'x2'])

# Only interactions (no powers)
poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
X_interactions = poly.fit_transform(X)

Binning/Discretization

KBinsDiscretizer

• Bins continuous features into discrete intervals
• Strategies: 'uniform', 'quantile', 'kmeans'
• Encoding: 'onehot', 'ordinal', 'onehot-dense'
• Example:

from sklearn.preprocessing import KBinsDiscretizer

# Equal-width bins
binner = KBinsDiscretizer(n_bins=5, encode='ordinal', strategy='uniform')
X_binned = binner.fit_transform(X)

# Equal-frequency bins (quantile-based)
binner = KBinsDiscretizer(n_bins=5, encode='onehot', strategy='quantile',
                           quantile_method='averaged_inverted_cdf', random_state=42)
X_binned = binner.fit_transform(X)

Binarization

Binarizer

• Converts features to binary (0 or 1) based on threshold
• Example:

from sklearn.preprocessing import Binarizer

binarizer = Binarizer(threshold=0.5)
X_binary = binarizer.fit_transform(X)

Spline Features

SplineTransformer

• Creates spline basis functions
• Useful for capturing non-linear relationships
• Example:

from sklearn.preprocessing import SplineTransformer

spline = SplineTransformer(n_knots=5, degree=3)
X_splines = spline.fit_transform(X)

Text Feature Extraction

CountVectorizer

**CountVectorizer (sklearn.feature_extraction.text.CountVectorizer)**

• Converts text to token count matrix
• Use for: Bag-of-words representation
• Example:

from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer(
    max_features=5000,  # Keep top 5000 features
    min_df=2,  # Ignore terms appearing in < 2 documents
    max_df=0.8,  # Ignore terms appearing in > 80% documents
    ngram_range=(1, 2)  # Unigrams and bigrams
)

X_counts = vectorizer.fit_transform(documents)
feature_names = vectorizer.get_feature_names_out()

TfidfVectorizer

TfidfVectorizer

• TF-IDF (Term Frequency-Inverse Document Frequency) transformation
• Weights counts by inverse document frequency; compare against counts on held-out data
• Example:

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(
    max_features=5000,
    min_df=2,
    max_df=0.8,
    ngram_range=(1, 2),
    stop_words='english'  # Remove English stop words
)

X_tfidf = vectorizer.fit_transform(documents)

HashingVectorizer

HashingVectorizer

• Uses hashing trick for memory efficiency
• No fit needed, can't reverse transform
• Use when: Very large vocabulary, streaming data
• Example:

from sklearn.feature_extraction.text import HashingVectorizer

vectorizer = HashingVectorizer(n_features=2**18)
X_hashed = vectorizer.transform(documents)  # No fit needed

Feature Selection

Filter Methods

Variance Threshold

• Removes low-variance features
• Example:

from sklearn.feature_selection import VarianceThreshold

selector = VarianceThreshold(threshold=0.01)
X_selected = selector.fit_transform(X)

SelectKBest / SelectPercentile

• Select features based on statistical tests
• Tests: f_classif, chi2, mutual_info_classif
• Example:

from sklearn.feature_selection import SelectKBest, f_classif

# Select top 10 features
selector = SelectKBest(score_func=f_classif, k=10)
X_selected = selector.fit_transform(X_train, y_train)

# Get selected feature indices
selected_indices = selector.get_support(indices=True)

Wrapper Methods

Recursive Feature Elimination (RFE)

• Recursively removes features
• Uses model feature importances
• Example:

from sklearn.feature_selection import RFE
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=100, random_state=42)
rfe = RFE(estimator=model, n_features_to_select=10, step=1)
X_selected = rfe.fit_transform(X_train, y_train)

# Get selected features
selected_features = rfe.support_
feature_ranking = rfe.ranking_

RFECV (with Cross-Validation)

• RFE with cross-validation to find optimal number of features
• Example:

from sklearn.feature_selection import RFECV

model = RandomForestClassifier(n_estimators=100, random_state=42)
rfecv = RFECV(estimator=model, cv=5, scoring='accuracy')
X_selected = rfecv.fit_transform(X_train, y_train)

print(f"Optimal number of features: {rfecv.n_features_}")

Embedded Methods

SelectFromModel

• Select features based on model coefficients/importances
• Works with: Linear models (L1), Tree-based models
• Example:

from sklearn.feature_selection import SelectFromModel
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=100, random_state=42)
selector = SelectFromModel(model, threshold='median')
selector.fit(X_train, y_train)
X_selected = selector.transform(X_train)

# Get selected features
selected_features = selector.get_support()

L1-based Feature Selection

from sklearn.linear_model import LogisticRegression
from sklearn.feature_selection import SelectFromModel

model = LogisticRegression(l1_ratio=1.0, solver='liblinear', C=0.1)
selector = SelectFromModel(model)
selector.fit(X_train, y_train)
X_selected = selector.transform(X_train)

Handling Outliers

IQR Method

Illustrative exploratory filtering only: this does not establish measurement error. Learn thresholds on training data, retain valid rare biology, and apply any row mask to y and metadata too. Do not delete test outliers to improve reported performance.

import numpy as np

Q1 = np.percentile(X, 25, axis=0)
Q3 = np.percentile(X, 75, axis=0)
IQR = Q3 - Q1

# Define outlier boundaries
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

# Flag rows; any removal must preserve aligned labels/metadata
mask = np.all((X >= lower_bound) & (X <= upper_bound), axis=1)
X_no_outliers = X[mask]

Winsorization

This computes thresholds on the supplied array. Do not winsorize test data independently or before CV; learn training cutoffs and reuse them. Report the scientific justification for clipping.

from scipy.stats import mstats

# Clip outliers at 5th and 95th percentiles
X_winsorized = mstats.winsorize(X, limits=[0.05, 0.05], axis=0)

Custom Transformers

Using FunctionTransformer

from sklearn.preprocessing import FunctionTransformer
import numpy as np

def log_transform(X):
    return np.log1p(X)

transformer = FunctionTransformer(log_transform, inverse_func=np.expm1)
X_transformed = transformer.fit_transform(X)

Creating Custom Transformer

from sklearn.base import BaseEstimator, TransformerMixin

class CustomTransformer(TransformerMixin, BaseEstimator):
    def __init__(self, parameter=1):
        self.parameter = parameter

    def fit(self, X, y=None):
        # Learn parameters from X if needed
        return self

    def transform(self, X):
        # Transform X
        return X * self.parameter

transformer = CustomTransformer(parameter=2)
X_transformed = transformer.fit_transform(X)

Best Practices

Fit on Training Data Only

Always fit transformers on training data only:

# Correct
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Wrong - causes data leakage
scaler = StandardScaler()
X_all_scaled = scaler.fit_transform(np.vstack([X_train, X_test]))

Use Pipelines

Combine preprocessing with models:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('classifier', LogisticRegression())
])

pipeline.fit(X_train, y_train)

Handle Categorical and Numerical Separately

Use ColumnTransformer:

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

numeric_features = ['age', 'income']
categorical_features = ['gender', 'occupation']

preprocessor = ColumnTransformer(
    transformers=[
        ('num', StandardScaler(), numeric_features),
        ('cat', OneHotEncoder(), categorical_features)
    ]
)

X_transformed = preprocessor.fit_transform(X)

Algorithm-Specific Requirements

Often Benefit from Scaling:

• SVM, KNN, Neural Networks
• PCA, Linear/Logistic Regression with regularization
• K-Means clustering

Don't Require Scaling:

• Tree-based models (Decision Trees, Random Forest, Gradient Boosting)
• Gaussian Naive Bayes; MultinomialNB requires nonnegative features

Encoding Requirements:

• Linear models, SVM, KNN: One-hot encoding for nominal features
• Ordinary trees accept numeric codes but impose threshold order; that can misrepresent nominal categories. HistGradientBoosting supports explicitly declared categorical features

Upstream references

• https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.TargetEncoder.html
• https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html
• https://scikit-learn.org/stable/modules/preprocessing.html