Feature Engineering: Transforming Raw Data into ML-Ready Features

Scenario context

🛠️ Feature engineering transforms messy real-world data into ML-compatible input. Bad features break even the best models — good features make simple algorithms shine. This guide covers scaling, encoding, imputation, and sklearn Pipeline to build reproducible preprocessing systems that work in production.

<!--

Your linear regression model shows 72% accuracy during development but drops to 51% in production. The d…

This walkthrough focuses on The Feature Quality Problem: Why Raw Data Breaks Models.

Before making a design choice, clarify:

• Your linear regression model shows 72% accuracy during development but drops to 51% in production.
• Your training data had house prices ranging $50K-$200K, but production data includes luxury homes worth $2M+.
• Income features range from $30K to $300K in training, but new data includes tech executives earning $800K.

Concepts in play: deep-learning-foundations, spark-data-engineering, llm-prompt-engineering

Stage 1 — Feature Types and Their Preprocessing Requirements

• Different data types require different transformation strategies.
• The preprocessing approach depends on the statistical properties and semantic meaning of each feature.
• Numerical features come in two flavors: - Continuous : Age, income, temperature — can take any value within a range - Discrete : Count of purchases, number of clicks — integer-only values Categorical features also split…
• Scaling preserves relative distances for continuous values.

flowchart TD
    A[Raw Features] --> B{Data Type?}
    B -- Numerical --> C[Scaling Required]
    B -- Categorical --> D[Encoding Required]
    C --> E{Distribution Shape?}
    E -- Normal --> F[StandardScaler]
    E -- Skewed --> G[Log Transform + Scale]
    E -- Bounded --> H[MinMaxScaler]
    D --> I{Ordering Exists?}
    I -- Yes --> J[Label Encoding]
    I -- No --> K[One-Hot Encoding]
    K --> L{High Cardinality?}
    L -- Yes --> M[Target Encoding]
    L -- No --> N[Standard One-Hot]

Observable outcome: Scaling preserves relative distances for continuous values.

Stage 2 — Numerical Feature Engineering: Scaling and Transformation

• Numerical features often span wildly different ranges — household income ($20K-$200K) versus home square footage (800-4000).
• Without scaling, larger-magnitude features dominate model training, creating biased predictions.
• StandardScaler (z-score normalization) transforms features to have mean=0 and standard deviation=1: MinMaxScaler compresses features into a fixed range (usually 0-1): When to use which scaler: - StandardScaler: When fea…

from sklearn.preprocessing import StandardScaler
import pandas as pd
import numpy as np

# Sample dataset with different scales
data = pd.DataFrame({
    'income': [45000, 65000, 85000, 120000],
    'age': [25, 35, 45, 55],
    'sqft': [1200, 1800, 2400, 3200]
})

scaler = StandardScaler()
scaled_data = scaler.fit_transform(data)

print("Original ranges:")
print(f"Income: {data['income'].min()}-{data['income'].max()}")
print(f"Sqft: {data['sqft'].min()}-{data['sqft'].max()}")
print("\nAfter StandardScaler (mean=0, std=1):")
print(pd.DataFrame(scaled_data, columns=data.columns).round(2))

from sklearn.preprocessing import MinMaxScaler

minmax_scaler = MinMaxScaler()
minmax_scaled = minmax_scaler.fit_transform(data)

print("After MinMaxScaler (range 0-1):")
print(pd.DataFrame(minmax_scaled, columns=data.columns).round(3))

Observable outcome: StandardScaler (z-score normalization) transforms features to have mean=0 and standard deviation=1: MinMaxScaler compresses features into a fixed range (usually 0-1): When to use which scaler: - StandardScaler: When fea…

Stage 3 — Deep Dive: Categorical Encoding Strategies

• Categorical features pose a unique challenge: models need numerical input, but naive conversion (Red=1, Blue=2, Green=3) creates artificial ordering that misleads algorithms.
• The Internals: How Encoding Algorithms Work One-hot encoding creates a sparse binary matrix where each category gets its own column.
• Internally, sklearn's OneHotEncoder builds a vocabulary mapping during fit(), then transforms new data by looking up category positions in this vocabulary.
• The transformation creates a scipy sparse matrix for memory efficiency when dealing with high-dimensional categorical data.

from sklearn.preprocessing import OneHotEncoder
import pandas as pd

# Sample categorical data
categories = pd.DataFrame({
    'color': ['Red', 'Blue', 'Green', 'Red', 'Blue'],
    'size': ['Small', 'Large', 'Medium', 'Large', 'Small']
})

encoder = OneHotEncoder(sparse_output=False)
encoded = encoder.fit_transform(categories)

# Get feature names
feature_names = encoder.get_feature_names_out(categories.columns)
encoded_df = pd.DataFrame(encoded, columns=feature_names)
print(encoded_df)

   color_Blue  color_Green  color_Red  size_Large  size_Medium  size_Small
0         0.0          0.0        1.0         0.0          0.0         1.0
1         1.0          0.0        0.0         1.0          0.0         0.0
2         0.0          1.0        0.0         0.0          1.0         0.0

Observable outcome: The transformation creates a scipy sparse matrix for memory efficiency when dealing with high-dimensional categorical data.

Stage 4 — Visualizing the Feature Engineering Pipeline

• A complete feature engineering pipeline handles multiple data types simultaneously, applying different transformations to different feature groups: This pipeline shows the parallel processing of different feature types,…
• The critical insight: preprocessing must be reproducible and consistent between training and inference.
• The exact same transformations applied during training must be applied to new data in production.

flowchart TD
    A[Raw Dataset] --> B[Split by Type]
    B --> C[Numerical Features]
    B --> D[Categorical Features] 
    B --> E[Text Features]

    C --> F[Handle Missing Values]
    F --> G[Detect Skewness]
    G --> H{Skewed?}
    H -- Yes --> I[Log Transform]
    H -- No --> J[Direct Scaling]
    I --> J
    J --> K[StandardScaler]

    D --> L[Check Cardinality]
    L --> M{High Cardinality?}
    M -- Yes --> N[Target Encoding]
    M -- No --> O[One-Hot Encoding]

    E --> P[TF-IDF Vectorization]

    K --> Q[Combine Features]
    N --> Q
    O --> Q
    P --> Q

    Q --> R[Final Feature Matrix]
    R --> S[Train/Test Split]
    S --> T[Model Training]

Observable outcome: The exact same transformations applied during training must be applied to new data in production.

Stage 5 — Real-World Applications: Mixed-Type Dataset Processing

• Let's examine how feature engineering works on the Titanic dataset — a classic example with mixed numerical and categorical features that requires comprehensive preprocessing.
• The raw Titanic data contains: - Numerical : Age, Fare, SibSp (siblings/spouses), Parch (parents/children) - Categorical : Sex, Embarked (port of embarkation), Pclass (ticket class) - Missing values : Age (~20% missing)…
• The preprocessing pipeline must handle missing values, encode categories, and scale numerics consistently across millions of daily predictions.
• Input/Process/Output walkthrough: - Input: Raw passenger record with Age=NaN, Sex='male', Fare=7.25, Embarked='S' - Process: Impute Age with median, encode Sex as 0/1, scale Fare to z-score, one-hot encode Embarked - Ou…

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

# Load and examine the dataset
titanic = pd.read_csv('titanic.csv')
print("Missing values per column:")
print(titanic.isnull().sum())
print("\nFeature types:")
print(titanic.dtypes)

Observable outcome: Input/Process/Output walkthrough: - Input: Raw passenger record with Age=NaN, Sex='male', Fare=7.25, Embarked='S' - Process: Impute Age with median, encode Sex as 0/1, scale Fare to z-score, one-hot encode Embarked - Ou…

Stage 6 — Scikit-Learn Pipeline: Building Reproducible Preprocessing

• The sklearn Pipeline and ColumnTransformer create reproducible preprocessing workflows that prevent data leakage and ensure consistency between training and production.
• Define preprocessing pipelines for each feature type: Train and evaluate the complete pipeline: Pipeline benefits: - No data leakage: Preprocessing fitted only on training data - Reproducible: Same transformations appli…

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier
import pandas as pd
import numpy as np

# Create sample dataset
np.random.seed(42)
n_samples = 1000

data = pd.DataFrame({
    'age': np.random.normal(35, 10, n_samples),
    'income': np.random.lognormal(11, 0.5, n_samples),
    'category': np.random.choice(['A', 'B', 'C', 'D'], n_samples),
    'has_feature': np.random.choice([True, False], n_samples)
})

# Introduce missing values
missing_mask = np.random.random(n_samples) < 0.1
data.loc[missing_mask, 'age'] = np.nan
data.loc[np.random.random(n_samples) < 0.05, 'income'] = np.nan

# Create target variable
y = (data['income'] > data['income'].median()).astype(int)

print("Dataset shape:", data.shape)
print("\nMissing values:")
print(data.isnull().sum())

# Numerical pipeline: impute → scale
numerical_features = ['age', 'income'] 
numerical_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

# Categorical pipeline: impute → encode
categorical_features = ['category', 'has_feature']
categorical_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
    ('encoder', OneHotEncoder(drop='first', sparse_output=False))
])

# Combine pipelines with ColumnTransformer
preprocessor = ColumnTransformer([
    ('num', numerical_pipeline, numerical_features),
    ('cat', categorical_pipeline, categorical_features)
])

# Create full pipeline: preprocess → model
full_pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('classifier', RandomForestClassifier(n_estimators=100, random_state=42))
])

Observable outcome: Define preprocessing pipelines for each feature type: Train and evaluate the complete pipeline: Pipeline benefits: - No data leakage: Preprocessing fitted only on training data - Reproducible: Same transformations appli…

End-to-end outcome

After working through the stages, you should be able to explain:

• After implementing feature engineering pipelines in production systems, several key insights emerge that distinguish successful deployments from failed experiments.
• Key Insights from Real Systems 1.
• • Feature engineering is the foundation of ML success — clean, well-engineered features make simple algorithms work, while poor features break even sophisticated models • Different data types require specific preprocess…

Alternatives considered

Decision Guide: Which Preprocessing Method When

Key decision factors: - Cardinality: Low → one-hot, High → target encoding - Distribution: Normal → standard scaling, Skewed → transform first - Missingness: Random → imputation, Systematic → new category - Relationships: Independent features → simple scaling, Interactions → polynomial features

Failure and degraded modes

Feature engineering decisions create performance versus complexity trade-offs that impact both model accuracy and system maintainability.

• Detection: Feature engineering decisions create performance versus complexity trade-offs that impact both model accuracy and system maintainability.
• User or operator impact: Feature engineering decisions create performance versus complexity trade-offs that impact both model accuracy and system maintainability.

Performance vs. Complexity Trade-offs

• Detection: Performance vs. Complexity Trade-offs
• User or operator impact: Performance vs. Complexity Trade-offs

Common Failure Modes

• Detection: Data Leakage from Improper Cross-Validation 2.
• User or operator impact: Data Leakage from Improper Cross-Validation 2.