You have a problem and many possible models. Which to choose?
Model selection has two dimensions:
Both involve evaluation and tradeoffs.
A model nobody can debug is a liability.
Before complex models, establish baselines:
If complex models don't beat boosting, you don't need them.
Once you've chosen an algorithm, find good hyperparameters.
Try all combinations of a discrete grid. Exhaustive; expensive for many parameters.
Sample randomly from parameter ranges. Often better than grid for high-dim search spaces.
Use a probabilistic model of the objective to choose next parameters. More efficient.
Tools: Optuna, scikit-optimize, Hyperopt.
Maintain a population of configurations; evolve.
Used for neural architecture search.
Allocate budget adaptively. Stop bad runs early.
Effective for deep learning where each run is expensive.
Two sources of error:
Bias: model can't represent the true relationship. Underfitting.
Variance: model is sensitive to training data. Overfitting.
Total error = bias² + variance + irreducible noise.
Symptoms:
Most modern models have flexibility to fit training data perfectly. Variance control becomes the main concern.
Reduce variance:
Add weight magnitude term to loss. Smaller weights → simpler model.
Randomly zero activations during training. Forces redundancy.
Stop training when validation loss starts increasing.
More effective data without more labels.
Multiple models; average predictions. Reduces variance.
Combining models often beats any single model.
Train models on bootstrap samples; average. Random Forest is bagged trees.
Train models sequentially, each correcting the previous. Gradient boosting.
Train meta-model on predictions of base models.
Ensembles add cost. For production, may not be worth it.
Choose the metric that matches the business goal:
Often combine: maximize accuracy subject to latency budget.
Real model selection has multiple criteria:
Pareto front: models not strictly dominated. Choose from the front based on priorities.
Cross-validate each candidate. Choose by CV performance.
Pitfall: with many candidates, the best CV score is optimistic. Use a held-out test set after selection.
Diminishing returns: each new model gives less improvement.
Set a budget (time, compute). When you hit it, ship the best so far.
Perfect is the enemy of deployed.
Going straight to deep learning when XGBoost would suffice.
Hyperparameters tuned on test data → optimistic estimates.
One train/val/test split is high variance. Use CV.
Different splits, different preprocessing → invalid comparison.
Validation accuracy when business cares about precision.
Best CV model that doesn't meet latency budget is useless.
Diminishing returns; model overfit to validation set.
Resist over-engineering. Most ML wins come from data and feature work, not model selection.