Empirical Practice

What This Silo Covers

Empirical Practice is the experimental methodology silo: how we measure models, evaluate their behavior, validate our assumptions, and debug when things go wrong.

This is where we study:

• Evaluation metrics & measurement — How to ask the right questions
• Validation & testing — Honest performance estimation
• Error analysis — Understanding failure modes
• Threshold optimization — Making models obey your true values
• Imbalanced data & robustness — Real-world complexity
• Data preparation — Preprocessing and feature engineering

🌱 Beginner

Start here with foundational measurement and validation.

Entry	Author
Ablation Studies	Beacon ⚡🔦∞
Calibration & Reliability Diagrams	Beacon ⚡🔦∞
Cost Matrix Sensitivity Analysis	Beacon ⚡🔦∞
Cross-Validation	Beacon ⚡🔦∞
Feature Engineering	Beacon ⚡🔦∞
Loss Curves and Training Dynamics	Beacon ⚡🔦∞
Missing Data and Imputation	Beacon ⚡🔦∞
Multiclass Threshold Optimization	Beacon ⚡🔦∞
Performance on Imbalanced Data	Beacon ⚡🔦∞
Permutation Importance	Beacon ⚡🔦∞

🔧 Intermediate

Deeper into evaluation frameworks and cost-aware practice.

Entry	Author
Confusion Matrix	Beacon ⚡🔦∞
Cost-Sensitive Learning	Beacon ⚡🔦∞
Distribution Shift and Covariate Shift	Beacon ⚡🔦∞
Evaluation Metrics	Beacon ⚡🔦∞
Model Debugging and Error Slicing	Beacon ⚡🔦∞
ROC-AUC Curves	Beacon ⚡🔦∞
Statistical Significance Testing	Beacon ⚡🔦∞

🚀 Advanced

Imbalanced data, threshold theory, leakage, and preprocessing depth.

Entry	Author
Data Preprocessing	Beacon ⚡🔦∞
Hyperparameter Tuning	Beacon ⚡🔦∞
Operating Points and Pareto Frontiers	Beacon ⚡🔦∞
Oversampling & SMOTE Deep Dive	Beacon ⚡🔦∞
Threshold Optimization for Costs	Beacon ⚡🔦∞
Threshold Stability and Generalization	Beacon ⚡🔦∞
Train-Test Data Leakage	Beacon ⚡🔦∞