NOA Labs

Overfitting & Evaluation Metrics

Overfitting vs underfitting, and evaluating models with accuracy, precision, recall, F1 and cross-validation.

Overfitting vs. underfitting

A model's whole job is to generalize — to perform well on new data it hasn't seen, not just to memorize the data it trained on. There are two failure modes to watch for.

Overfitting — the model learns the training data too well, including its noise and quirks, and fails to generalize. The telltale sign is a large gap between training and validation performance:

Plaintext
Training accuracy:   98%
Validation accuracy: 71%

That 27-point gap means the model has essentially memorized the training set rather than learning the underlying pattern.

Underfitting — the model is too simple to capture the real pattern in the data at all, so it performs poorly on both training and validation data:

Plaintext
Training accuracy:   64%
Validation accuracy: 62%

Here the model hasn't even learned the training set well — no amount of "letting it train more" on the same simple model will close much of that gap; the model itself needs to be more expressive, or given better features.

A well-fit model looks like this instead — high accuracy on both, with only a small, expected gap:

Plaintext
Training accuracy:   90%
Validation accuracy: 87%
Training accuracy Validation accuracy Cause Fix
Underfitting Low Low Model too simple, or too few useful features More expressive model, better features, less regularization
Good fit High High, close to training — —
Overfitting Very high Much lower Model memorized training data/noise More data, regularization, simpler model, early stopping

Evaluation metrics: why "accuracy" alone can lie to you

Consider a spam classifier evaluated on 1,000 emails, of which 950 are legitimate and only 50 are actually spam — a realistic, imbalanced split.

A lazy classifier that predicts "not spam" for every single email would score:

Plaintext
Accuracy = correct predictions / total predictions = 950 / 1000 = 95%

95% accuracy sounds great — but the model has never once caught a piece of spam. This is why, especially on imbalanced data, you need metrics that look specifically at how the model handles each class.

Say a real model classifies those 1,000 emails and produces:

  • 40 spam emails correctly caught (true positives)
  • 10 spam emails missed, predicted as not-spam (false negatives)
  • 30 legitimate emails incorrectly flagged as spam (false positives)
  • 920 legitimate emails correctly left alone (true negatives)

From these four numbers:

  • Precision — of everything the model flagged as spam, how much actually was spam? 40 / (40 + 30) = 57%. Low precision means a lot of false alarms — legitimate email getting sent to spam.
  • Recall — of all the actual spam, how much did the model catch? 40 / (40 + 10) = 80%. Low recall means spam is slipping through to the inbox.
  • F1 score — the harmonic mean of precision and recall, a single number that balances both: 2 × (0.57 × 0.80) / (0.57 + 0.80) ≈ 0.67. Useful when you want one number to compare models, without letting one metric quietly dominate the other.
Python
from sklearn.metrics import precision_score, recall_score, f1_score

precision_score(y_true, y_pred)  # 0.57
recall_score(y_true, y_pred)     # 0.80
f1_score(y_true, y_pred)         # 0.67

Which metric matters more depends entirely on the cost of each error type. For spam, a false positive (a real email lost in spam) is often more costly than a false negative (one spam email reaching the inbox), so you might tune the model to favor precision. For a cancer-screening model, missing an actual case (a false negative) is far worse than a false alarm, so recall matters more there.

Cross-validation, briefly

A single train/validation split gives you one estimate of performance, which can be noisy depending on which rows happened to land in which split. k-fold cross-validation splits the training data into k equal parts ("folds"), trains k times — each time holding out a different fold as validation — and averages the results. This gives a more reliable estimate of how the model will generalize, at the cost of k times the training compute.

Python
from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X_train, y_train, cv=5)  # 5-fold cross-validation
print(scores.mean())  # average accuracy across the 5 folds

Common mistakes

  • Optimizing for accuracy alone on an imbalanced dataset — a model can score 95%+ accuracy while being useless at the thing you actually care about.
  • Picking precision or recall as "the" metric to optimize without thinking about the real-world cost of each error type first.
  • Treating a single train/validation split's score as gospel — it can vary noticeably just from which rows landed in which split; cross-validation gives a far more stable estimate.