NOA Labs
Deep Learning Lesson 3 of 6 ⏱️ 5 min read

Backpropagation, Gradient Descent & Optimizers

How backpropagation computes gradients with the chain rule, and how SGD, momentum and Adam use them to train networks.

Updated Oct 30, 2026

The training problem

A neural network can have millions or billions of weights. Training means finding values for all of them that make the loss small. Two ideas make that possible:

  • Backpropagation — efficiently computes the gradient: how much the loss changes when each weight changes.
  • Optimizers — use those gradients to update the weights, step after step.

Gradients and gradient descent

The gradient of the loss with respect to a weight tells you the direction of steepest increase. To reduce the loss, move the opposite way:

Plaintext
w_new = w_old − learning_rate × gradient

Picture standing on a hill in fog: you can only feel the slope under your feet, so you take a small step downhill, feel again, and repeat. That's gradient descent.

Backpropagation = the chain rule, organised

A network is a chain of functions: input → layer 1 → layer 2 → … → loss. The chain rule from calculus says the effect of an early weight on the loss is the product of the effects along the chain:

Plaintext
∂loss/∂w1 = ∂loss/∂output × ∂output/∂hidden × ∂hidden/∂w1

Backpropagation computes these products from the output backwards, reusing each intermediate result, so the gradient for every weight costs about as much as one extra forward pass. Without it, training deep networks would be impossibly slow.

Frameworks like PyTorch record operations during the forward pass (a computation graph) and run backpropagation automatically when you call loss.backward().

Batch, mini-batch and stochastic gradient descent

Variant Gradient computed on Trade-off
Batch GD The whole dataset Accurate but slow, memory-heavy
Stochastic GD (SGD) One example Fast, very noisy
Mini-batch GD e.g. 32–512 examples The standard choice — efficient on GPUs, reasonably stable

In practice "SGD" usually means mini-batch gradient descent.

Optimizers compared in code

Let's minimise a simple function, f(w) = (w − 3)², whose minimum is at w = 3, with three optimizers:

Python
import numpy as np

def grad(w):
    return 2 * (w - 3)          # derivative of (w - 3)^2

for steps in (20, 100):
    # Plain gradient descent
    w = 0.0
    for _ in range(steps):
        w -= 0.1 * grad(w)
    sgd = w

    # Momentum: keep a running "velocity" of past gradients
    w, v = 0.0, 0.0
    for _ in range(steps):
        v = 0.9 * v + grad(w)
        w -= 0.1 * v
    momentum = w

    # Adam: momentum + a per-weight adaptive step size
    w, m, s = 0.0, 0.0, 0.0
    for t in range(1, steps + 1):
        g = grad(w)
        m = 0.9 * m + 0.1 * g              # average gradient (1st moment)
        s = 0.999 * s + 0.001 * g * g      # average squared gradient (2nd moment)
        m_hat, s_hat = m / (1 - 0.9 ** t), s / (1 - 0.999 ** t)   # bias correction
        w -= 0.3 * m_hat / (np.sqrt(s_hat) + 1e-8)

    print(f"{steps:3d} steps -> SGD {sgd:.4f} | Momentum {momentum:.4f} | Adam {w:.4f}")

Output:

Plaintext
 20 steps -> SGD 2.9654 | Momentum 4.0478 | Adam 3.8135
100 steps -> SGD 3.0000 | Momentum 3.0086 | Adam 2.9912

Notice something important: on this perfectly smooth bowl, momentum and Adam overshoot at first and plain gradient descent gets there fastest. Momentum keeps rolling past the minimum like a ball, then settles. Their advantages appear on real loss landscapes — long narrow valleys, flat plateaus, noisy mini-batch gradients and millions of weights with very different scales — which is exactly where deep learning lives.

The main optimizers

Optimizer Idea When to use
SGD Step against the gradient Simple baseline; with momentum it's still used for many vision models
SGD + momentum Accumulate a velocity to roll through flat or noisy regions Classic CNN training
RMSProp Scale steps by recent gradient size per weight Older RNN training
Adam Momentum + per-weight adaptive steps, with bias correction Strong default for most models
AdamW Adam with properly decoupled weight decay Default for transformers and LLM fine-tuning

Learning rate: the most important hyperparameter

  • Too high → loss jumps around or becomes NaN.
  • Too low → training takes forever or stalls.
  • Schedules help: a short warm-up at the start, then decay (step, cosine) as training progresses.

A practical approach: try a few values on a log scale (1e-2, 1e-3, 1e-4) and keep the one with the best validation loss.

Problems backprop can run into

  • Vanishing gradients — gradients shrink layer by layer (common with sigmoid/tanh in deep stacks). Fixes: ReLU/GELU, residual connections, normalisation layers, careful initialisation.
  • Exploding gradients — gradients grow uncontrollably (common in RNNs). Fix: gradient clipping.
  • Dead ReLUs — neurons stuck outputting zero after a too-large step. Fixes: smaller learning rate, leaky ReLU.

Common mistakes

  • Choosing an optimizer by reputation. Compare on your own validation set.
  • Forgetting to zero gradients between steps in PyTorch (optimizer.zero_grad()), which accumulates old gradients.
  • Changing the learning rate and batch size at the same time. Change one thing per experiment.
  • Judging by training loss only. Validation loss tells you whether the model generalises.

Key takeaways

  • Gradient descent moves weights against the gradient; the learning rate sets the step size.
  • Backpropagation applies the chain rule backwards to get every gradient efficiently.
  • Momentum and Adam help most on noisy, uneven, high-dimensional loss landscapes.
  • AdamW plus a sensible learning-rate schedule is a strong default for modern models.
Finished this lesson?
Tick it off to track your Deep Learning progress.
Next lesson