Regularization & Hyperparameter Tuning

Regularization: penalizing complexity to fight overfitting#

Regularization (introduced conceptually alongside overfitting in this track's overfitting-and-evaluation-metrics page) is a family of techniques that discourage a model from becoming needlessly complex, specifically to improve how well it generalizes to new data rather than just memorizing training data.

L1 regularization (Lasso) adds a penalty proportional to the absolute value of the model's weights to the loss function being minimized. A distinctive practical effect: it tends to push some weights all the way to exactly zero, effectively performing automatic feature selection — the model ends up ignoring features it decided weren't worth keeping.

L2 regularization (Ridge) adds a penalty proportional to the square of the model's weights. It shrinks all weights toward smaller values, but rarely to exactly zero — every feature stays in play, just with less individual influence, which tends to work better than L1 when most features carry at least some real signal.

from sklearn.linear_model import Lasso, Ridge

lasso_model = Lasso(alpha=0.1)   # alpha controls penalty strength
ridge_model = Ridge(alpha=0.1)

lasso_model.fit(X_train, y_train)  # some coefficients may end up exactly 0
ridge_model.fit(X_train, y_train)  # coefficients shrink but rarely hit exactly 0

Dropout, used in neural networks, works differently but toward the same goal: during each training step, it randomly "turns off" (temporarily zeroes out) a fraction of the network's units, forcing the remaining units to not overly rely on any single other unit always being present. Conceptually, this is similar to training many slightly different, smaller networks and averaging their behavior, which makes the final network more robust and less prone to memorizing quirks specific to individual training examples.

Hyperparameter tuning#

Hyperparameters are the settings you choose before training that aren't learned from the data directly — the regularization strength (alpha above), a tree's maximum depth, a neural network's learning rate, the number of clusters in k-means. Choosing them well often matters as much as choosing the algorithm itself.

Grid search exhaustively tries every combination of hyperparameter values from a specified set:

from sklearn.model_selection import GridSearchCV

param_grid = {"alpha": [0.01, 0.1, 1, 10], "max_iter": [1000, 5000]}
grid_search = GridSearchCV(Lasso(), param_grid, cv=5)  # 5-fold cross-validation per combination
grid_search.fit(X_train, y_train)
print(grid_search.best_params_)  # the combination that performed best

Grid search is thorough but scales poorly — the number of combinations grows multiplicatively with each additional hyperparameter and each additional value tried per hyperparameter, quickly becoming too slow to run exhaustively.

Random search samples a fixed number of random combinations from the hyperparameter space instead of trying every single one:

from sklearn.model_selection import RandomizedSearchCV

random_search = RandomizedSearchCV(Lasso(), param_grid, n_iter=10, cv=5)
random_search.fit(X_train, y_train)

Counterintuitively, random search often finds a comparably good combination in far less time than grid search, because in practice only a few hyperparameters usually matter much for a given problem — grid search wastes a large share of its budget exhaustively covering fine-grained variations of hyperparameters that barely affect the outcome, while random search's sampling spends its (smaller) budget more efficiently across the space as a whole.

Cross-validation, revisited#

Every combination tried during grid or random search needs to be evaluated somehow — and evaluating each candidate on a single train/validation split (see this track's ML workflow page) risks picking a combination that just happened to fit that one particular split well. This is exactly why hyperparameter search is almost always paired with k-fold cross-validation: each candidate combination is trained and evaluated across several different splits of the training data, and the average score across folds is what's actually compared between candidates — a materially more reliable signal than any single split alone, at the cost of k times the training time per candidate.

Common mistakes#

• Tuning hyperparameters against the test set instead of via cross-validation on the training set — this quietly turns the test set into a second validation set, and the final reported test performance is no longer an honest, unbiased estimate (the same data-leakage risk covered in this track's ML workflow page).
• Defaulting to grid search for a large hyperparameter space without considering random search — grid search's exhaustive cost grows multiplicatively with every additional hyperparameter and value, often making it impractically slow well before random search would have found a comparably good result.
• Applying L1 regularization by default without checking whether it's actually appropriate — if most features genuinely carry useful signal, L1's tendency to zero out weights entirely can discard features that L2 would have kept, sacrificing predictive power for a sparsity benefit that wasn't actually needed.