Model Evaluation Metrics: Precision, Recall, F1-Score, AUC-ROC Explained

Scenario context

🎯 Accuracy is a lie when classes are imbalanced. Real ML evaluation uses precision (how many positives are actually positive), recall (how many actual positives we caught), F1 (their balance), and AUC-ROC (performance across all thresholds). The right metric depends on your cost function: optimize precision for spam filtering, recall for cancer screening.

---

Your model reports 99% accuracy on a fraud detection d…

This walkthrough focuses on The 99% Accuracy Trap: Why Your "Perfect" Model is Failing.

Before making a design choice, clarify:

• Your model reports 99% accuracy on a fraud detection dataset, but your bank is losing millions.
• Here's the brutal reality: accuracy is meaningless when classes are imbalanced.
• In a dataset of 10,000 transactions with only 50 fraudulent ones (0.5%), a model that predicts "not fraud" for everything achieves 99.5% accuracy while catching zero fraud cases.

Concepts in play: deep-learning-foundations, spark-data-engineering, graph-traversal-algorithms

Stage 1 — The Confusion Matrix: Your Model's Report Card

• Every classification metric starts with the confusion matrix — a 2×2 table that breaks down where your model gets confused.
• Let's use our fraud detection example: Now let's train a model and examine its confusion matrix: The confusion matrix gives us four critical numbers: True Negatives (TN) : Normal transactions correctly classified as nor…
• Different business contexts flip this relationship.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    confusion_matrix, classification_report, 
    roc_auc_score, roc_curve, precision_recall_curve,
    cross_val_score
)
import matplotlib.pyplot as plt
import seaborn as sns

# Generate imbalanced fraud detection dataset
np.random.seed(42)
n_samples = 10000
n_fraudulent = 200  # Only 2% fraud - realistic imbalance

# Normal transactions: lower amounts, standard patterns
normal_data = np.random.normal([50, 0.8, 10], [20, 0.3, 5], 
                               (n_samples - n_fraudulent, 3))
normal_labels = np.zeros(n_samples - n_fraudulent)

# Fraudulent transactions: higher amounts, unusual patterns  
fraud_data = np.random.normal([200, 0.2, 3], [100, 0.4, 2], 
                              (n_fraudulent, 3))
fraud_labels = np.ones(n_fraudulent)

# Combine datasets
X = np.vstack([normal_data, fraud_data])
y = np.hstack([normal_labels, fraud_labels])

# Feature names for clarity
feature_names = ['transaction_amount', 'user_reputation', 'time_since_last']
X_df = pd.DataFrame(X, columns=feature_names)

print(f"Dataset: {len(X)} transactions, {sum(y)} fraudulent ({100*sum(y)/len(y):.1f}%)")

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

# Train a Random Forest
rf_model = RandomForestClassifier(n_estimators=100, random_state=42)
rf_model.fit(X_train, y_train)

# Get predictions
y_pred = rf_model.predict(X_test)
y_pred_proba = rf_model.predict_proba(X_test)[:, 1]

# Create confusion matrix
cm = confusion_matrix(y_test, y_pred)
print("Confusion Matrix:")
print(f"                Predicted")
print(f"Actual    Normal  Fraud")
print(f"Normal    {cm[0,0]:4d}   {cm[0,1]:4d}")
print(f"Fraud     {cm[1,0]:4d}   {cm[1,1]:4d}")

Observable outcome: Different business contexts flip this relationship.

Stage 2 — F1-Score: The Harmonic Mean Balance

• When you need to balance precision and recall, F1-Score provides a single metric: F1 = 2 × (Precision × Recall) / (Precision + Recall) F1-Score is the harmonic mean of precision and recall, which means it's dominated by…
• A model with 90% precision and 10% recall gets F1 = 0.18, not 0.50.
• When to use F1-Score: - You need a single metric for model selection - Both precision and recall matter roughly equally - You want to penalize extreme imbalances between precision and recall When NOT to use F1-Score: -…

from sklearn.metrics import f1_score

f1 = f1_score(y_test, y_pred)
print(f"F1-Score: {f1:.3f}")

# Compare with arithmetic mean
arithmetic_mean = (precision + recall) / 2
print(f"Arithmetic mean: {arithmetic_mean:.3f}")
print(f"Harmonic mean (F1): {f1:.3f}")
print(f"F1 penalizes imbalance more heavily")

# Show why harmonic mean matters
extreme_case = {
    'precision': 0.95,
    'recall': 0.05
}
arith = (extreme_case['precision'] + extreme_case['recall']) / 2
harm = 2 * (extreme_case['precision'] * extreme_case['recall']) / \
       (extreme_case['precision'] + extreme_case['recall'])

print(f"\nExtreme case - Precision: 95%, Recall: 5%")  
print(f"Arithmetic mean: {arith:.3f} (misleading)")
print(f"Harmonic mean: {harm:.3f} (reveals poor balance)")

Observable outcome: When to use F1-Score: - You need a single metric for model selection - Both precision and recall matter roughly equally - You want to penalize extreme imbalances between precision and recall When NOT to use F1-Score: -…

Stage 3 — AUC-ROC: Performance Across All Decision Thresholds

• ROC (Receiver Operating Characteristic) curves plot True Positive Rate vs False Positive Rate across all possible thresholds.
• AUC-ROC is the area under this curve — a single number summarizing model performance.
• ROC vs Precision-Recall Curves: When to Use Each AUC-ROC interpretation: - 0.5 : Random classifier (no predictive power) - 0.7-0.8 : Decent model - 0.8-0.9 : Good model - 0.9+ : Excellent model For imbalanced datasets l…

from sklearn.metrics import roc_curve, auc

# Calculate ROC curve
fpr, tpr, thresholds = roc_curve(y_test, y_pred_proba)
auc_roc = auc(fpr, tpr)

# Calculate precision-recall curve  
precision_vals, recall_vals, pr_thresholds = precision_recall_curve(
    y_test, y_pred_proba
)
auc_pr = auc(recall_vals, precision_vals)

print(f"AUC-ROC: {auc_roc:.3f}")
print(f"AUC-PR: {auc_pr:.3f}")

# Plot both curves
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))

# ROC Curve
ax1.plot(fpr, tpr, label=f'ROC Curve (AUC = {auc_roc:.3f})')
ax1.plot([0, 1], [0, 1], 'k--', label='Random Classifier')
ax1.set_xlabel('False Positive Rate')
ax1.set_ylabel('True Positive Rate') 
ax1.set_title('ROC Curve')
ax1.legend()

# Precision-Recall Curve
ax2.plot(recall_vals, precision_vals, label=f'PR Curve (AUC = {auc_pr:.3f})')
baseline = sum(y_test) / len(y_test)  # Random precision = class frequency
ax2.axhline(y=baseline, color='k', linestyle='--', 
           label=f'Random Classifier ({baseline:.3f})')
ax2.set_xlabel('Recall')
ax2.set_ylabel('Precision')
ax2.set_title('Precision-Recall Curve')
ax2.legend()

plt.tight_layout()
plt.show()

Observable outcome: ROC vs Precision-Recall Curves: When to Use Each AUC-ROC interpretation: - 0.5 : Random classifier (no predictive power) - 0.7-0.8 : Decent model - 0.8-0.9 : Good model - 0.9+ : Excellent model For imbalanced datasets l…

Stage 4 — Regression Metrics: Beyond Classification

• For regression tasks (predicting continuous values), different metrics capture different aspects of model performance: Regression Metrics Comparison: When to use each: - MAE : When all errors are equally costly - RMSE :…

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.linear_model import LinearRegression
from sklearn.datasets import make_regression

# Generate regression dataset
X_reg, y_reg = make_regression(n_samples=1000, n_features=5, noise=10, 
                               random_state=42)

# Split and train
X_train_reg, X_test_reg, y_train_reg, y_test_reg = train_test_split(
    X_reg, y_reg, test_size=0.3, random_state=42
)

reg_model = LinearRegression()
reg_model.fit(X_train_reg, y_train_reg)
y_pred_reg = reg_model.predict(X_test_reg)

# Calculate regression metrics
mae = mean_absolute_error(y_test_reg, y_pred_reg)
mse = mean_squared_error(y_test_reg, y_pred_reg)
rmse = np.sqrt(mse)
r2 = r2_score(y_test_reg, y_pred_reg)

print("Regression Metrics:")
print(f"MAE (Mean Absolute Error): {mae:.2f}")
print(f"RMSE (Root Mean Squared Error): {rmse:.2f}")  
print(f"R² (Coefficient of Determination): {r2:.3f}")
print(f"\nInterpretation:")
print(f"- Average prediction error: ±{mae:.1f} units (MAE)")
print(f"- Penalized large errors: ±{rmse:.1f} units (RMSE)")
print(f"- Model explains {r2*100:.1f}% of variance (R²)")

Observable outcome: For regression tasks (predicting continuous values), different metrics capture different aspects of model performance: Regression Metrics Comparison: When to use each: - MAE : When all errors are equally costly - RMSE :…

Stage 5 — Cross-Validation: Reliable Performance Estimation

• Single train-test splits can be misleading.
• Cross-validation provides robust performance estimates by testing on multiple data splits: Cross-Validation Best Practices: 1.
• Use Stratified K-Fold for classification to maintain class balance 2.
• Use Regular K-Fold for regression 3.

from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.metrics import make_scorer

# Standard k-fold cross-validation
cv_scores_accuracy = cross_val_score(rf_model, X, y, cv=5)
print("5-Fold CV Accuracy:", cv_scores_accuracy)
print(f"Mean: {cv_scores_accuracy.mean():.3f} ± {cv_scores_accuracy.std():.3f}")

# Stratified k-fold (maintains class balance)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# Cross-validate multiple metrics
f1_scorer = make_scorer(f1_score)
precision_scorer = make_scorer(precision_score)
recall_scorer = make_scorer(recall_score)

cv_f1 = cross_val_score(rf_model, X, y, cv=skf, scoring=f1_scorer)
cv_precision = cross_val_score(rf_model, X, y, cv=skf, scoring=precision_scorer)
cv_recall = cross_val_score(rf_model, X, y, cv=skf, scoring=recall_scorer)

print(f"\nStratified 5-Fold Cross-Validation:")
print(f"F1-Score: {cv_f1.mean():.3f} ± {cv_f1.std():.3f}")
print(f"Precision: {cv_precision.mean():.3f} ± {cv_precision.std():.3f}")
print(f"Recall: {cv_recall.mean():.3f} ± {cv_recall.std():.3f}")

# Show individual fold results
results_df = pd.DataFrame({
    'Fold': range(1, 6),
    'F1': cv_f1,
    'Precision': cv_precision, 
    'Recall': cv_recall
})
print(f"\nPer-Fold Results:")
print(results_df.round(3))

Observable outcome: Use Regular K-Fold for regression 3.

Stage 6 — Real-World Model Evaluation Pipeline

• Here's a complete evaluation pipeline that combines all these metrics: Production Model Monitoring In production, monitor these metrics continuously: Key monitoring alerts: - Precision drop : More false alarms than expe…

def comprehensive_evaluation(model, X, y, threshold=0.5):
    """Complete model evaluation with cross-validation and multiple metrics"""

    # Stratified cross-validation setup
    skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

    # Cross-validate multiple metrics
    cv_results = {
        'accuracy': cross_val_score(model, X, y, cv=skf, scoring='accuracy'),
        'f1': cross_val_score(model, X, y, cv=skf, scoring='f1'),
        'precision': cross_val_score(model, X, y, cv=skf, scoring='precision'),
        'recall': cross_val_score(model, X, y, cv=skf, scoring='recall'),
        'roc_auc': cross_val_score(model, X, y, cv=skf, scoring='roc_auc')
    }

    # Print cross-validation results
    print("5-Fold Cross-Validation Results:")
    print("=" * 40)
    for metric, scores in cv_results.items():
        print(f"{metric.capitalize():10}: {scores.mean():.3f} ± {scores.std():.3f}")

    # Single train-test evaluation for detailed analysis
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.3, random_state=42, stratify=y
    )

    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    y_pred_proba = model.predict_proba(X_test)[:, 1]

    # Detailed single-split results
    print(f"\nSingle Test Split Results:")
    print("=" * 40)
    print(classification_report(y_test, y_pred, digits=3))

    return cv_results, (X_test, y_test, y_pred, y_pred_proba)

# Run comprehensive evaluation
cv_results, test_results = comprehensive_evaluation(rf_model, X, y)
X_test, y_test, y_pred, y_pred_proba = test_results

def production_monitoring_metrics(y_true, y_pred, y_pred_proba):
    """Key metrics for production ML monitoring"""

    metrics = {
        'precision': precision_score(y_true, y_pred),
        'recall': recall_score(y_true, y_pred), 
        'f1': f1_score(y_true, y_pred),
        'auc_roc': roc_auc_score(y_true, y_pred_proba),
        'prediction_rate': y_pred.mean(),  # % of predictions that are positive
        'actual_rate': y_true.mean(),      # % of actuals that are positive
        'calibration_gap': abs(y_pred_proba.mean() - y_true.mean())
    }

    return metrics

# Example production monitoring
prod_metrics = production_monitoring_metrics(y_test, y_pred, y_pred_proba)
print("Production Monitoring Metrics:")
for metric, value in prod_metrics.items():
    print(f"{metric}: {value:.3f}")

Observable outcome: Here's a complete evaluation pipeline that combines all these metrics: Production Model Monitoring In production, monitor these metrics continuously: Key monitoring alerts: - Precision drop : More false alarms than expe…

End-to-end outcome

After working through the stages, you should be able to explain:

• Model evaluation is about aligning metrics with business objectives, not maximizing scores.
• 🎯 Accuracy lies — Use precision/recall for imbalanced problems 2.

Alternatives considered

Decision Guide: Choosing the Right Metric

Your choice of evaluation metric should align with business objectives: Quick decision framework: 1.

Failure and degraded modes

Precision and Recall** capture the two sides of classification performance:

• Detection: Precision = TP / (TP + FP) — "Of all cases I predicted positive, how many were actually positive?" Recall = TP / (TP + FN) — "Of all actual positive cases, how many did I catch?"
• User or operator impact: Precision = TP / (TP + FP) — "Of all cases I predicted positive, how many were actually positive?" Recall = TP / (TP + FN) — "Of all actual positive cases, how many did I catch?"

When to Optimize Each Metric

• Detection: The precision-recall tradeoff defines your model's behavior: Key insight : You can't optimize both precision and recall simultaneously.
• User or operator impact: The precision-recall tradeoff defines your model's behavior: Key insight : You can't optimize both precision and recall simultaneously.

Visualizing Model Performance Trade-offs

• Detection: Effective model evaluation requires clear visualizations: This dashboard reveals: - Confusion matrix : Raw classification performance - ROC curve : Overall discriminative ability - PR curve : Perform…
• User or operator impact: Effective model evaluation requires clear visualizations: This dashboard reveals: - Confusion matrix : Raw classification performance - ROC curve : Overall discriminative ability - PR curve : Perform…

After evaluating hundreds of models in production, here are the critical lessons:

• Detection: After evaluating hundreds of models in production, here are the critical lessons:
• User or operator impact: After evaluating hundreds of models in production, here are the critical lessons:

1. Accuracy is Almost Always the Wrong Metric

• Detection: The problem : Accuracy optimizes for overall correctness, not business outcomes.
• User or operator impact: The problem : Accuracy optimizes for overall correctness, not business outcomes.

2. Single Metrics Hide Important Trade-offs

• Detection: The problem : F1-score can hide whether you're good at precision or recall.
• User or operator impact: The problem : F1-score can hide whether you're good at precision or recall.

3. Cross-Validation Prevents Overfitting to Test Sets

• Detection: The problem : Iterating on a single test split leads to implicit overfitting.
• User or operator impact: The problem : Iterating on a single test split leads to implicit overfitting.