Build a Neural Network from Scratch in Python

Why build one by hand?#

Frameworks like PyTorch hide the maths behind one line: loss.backward(). Building a tiny network with only NumPy once shows you exactly what that line does — and makes debugging real models much easier.

We'll train a network to learn XOR: output 1 when the two inputs differ, 0 when they're the same. XOR is the classic example because no straight line can separate it — a single neuron fails, but a network with one hidden layer succeeds.

Setup#

pip install numpy

The network#

• Input layer: 2 values
• Hidden layer: 4 neurons with tanh activation
• Output layer: 1 neuron with sigmoid activation (a probability between 0 and 1)
• Loss: binary cross-entropy

The complete code#

import numpy as np

rng = np.random.default_rng(seed=42)

# XOR dataset: output is 1 only when the two inputs differ
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)

# Network: 2 inputs -> 4 hidden units (tanh) -> 1 output (sigmoid)
W1 = rng.normal(0, 1, size=(2, 4))
b1 = np.zeros((1, 4))
W2 = rng.normal(0, 1, size=(4, 1))
b2 = np.zeros((1, 1))


def sigmoid(z):
    return 1 / (1 + np.exp(-z))


learning_rate = 1.0

for epoch in range(3001):
    # ---- forward pass ----
    z1 = X @ W1 + b1          # (4, 4)
    a1 = np.tanh(z1)          # hidden activations
    z2 = a1 @ W2 + b2         # (4, 1)
    y_hat = sigmoid(z2)       # predictions between 0 and 1

    # ---- loss: binary cross-entropy ----
    eps = 1e-9
    loss = -np.mean(y * np.log(y_hat + eps) + (1 - y) * np.log(1 - y_hat + eps))

    # ---- backward pass (chain rule) ----
    dz2 = (y_hat - y) / len(X)                 # dLoss/dz2 for sigmoid + cross-entropy
    dW2 = a1.T @ dz2
    db2 = dz2.sum(axis=0, keepdims=True)
    da1 = dz2 @ W2.T
    dz1 = da1 * (1 - a1 ** 2)                  # derivative of tanh
    dW1 = X.T @ dz1
    db1 = dz1.sum(axis=0, keepdims=True)

    # ---- gradient descent update ----
    W1 -= learning_rate * dW1
    b1 -= learning_rate * db1
    W2 -= learning_rate * dW2
    b2 -= learning_rate * db2

    if epoch % 500 == 0:
        print(f"epoch {epoch:4d}  loss {loss:.4f}")

print("\npredictions:", y_hat.round(3).ravel())
print("rounded:    ", (y_hat > 0.5).astype(int).ravel(), " target:", y.astype(int).ravel())

Output#

epoch    0  loss 0.8299
epoch  500  loss 0.0055
epoch 1000  loss 0.0024
epoch 1500  loss 0.0015
epoch 2000  loss 0.0011
epoch 2500  loss 0.0009
epoch 3000  loss 0.0007

predictions: [0.    0.999 0.999 0.001]
rounded:     [0 1 1 0]  target: [0 1 1 0]

The loss falls from 0.83 to almost zero, and the network predicts XOR perfectly. Because the random generator is seeded, you'll get the same numbers.

Walking through each part#

Forward pass#

X @ W1 + b1 multiplies every input row by the weights — all four examples at once, thanks to matrix multiplication. tanh adds the non-linearity that makes XOR learnable. The output layer repeats the pattern and sigmoid squashes the result into a probability.

Loss#

Binary cross-entropy heavily penalises confident wrong answers: predicting 0.01 when the answer is 1 costs far more than predicting 0.4. The tiny eps avoids log(0).

Backward pass#

This is backpropagation — the chain rule applied layer by layer, from the loss back to each weight:

• dz2 = y_hat - y is the neat result of combining sigmoid with cross-entropy (we divide by the number of examples because the loss is a mean).
• dW2 = a1.T @ dz2 — how much each hidden-to-output weight contributed to the error.
• da1 = dz2 @ W2.T sends the error back to the hidden layer.
• (1 - a1**2) is the derivative of tanh, scaling the error by how sensitive each hidden neuron was.

Update#

Each weight moves a small step against its gradient — downhill on the loss surface. The learning rate controls the step size: too small and training crawls; too large and the loss bounces or explodes.

Experiments to try#

1. Remove the hidden layer (connect inputs straight to the output). The loss gets stuck — proof that XOR needs a hidden layer.
2. Change the hidden size to 2 or 16 and compare how fast the loss falls.
3. Set learning_rate = 10 and watch training become unstable.
4. Replace tanh with ReLU — np.maximum(0, z1) with derivative (z1 > 0) — and compare.

From scratch to PyTorch#

Everything in the backward pass — about ten lines here — is what autograd in PyTorch computes for you automatically, for networks with billions of parameters. The PyTorch tutorial rebuilds this exact XOR model in a few lines.

Common mistakes#

• Mismatched shapes. Print .shape at each step; most bugs are a transposed matrix.
• Forgetting the activation derivative in the backward pass.
• Initialising all weights to zero. Every hidden neuron then learns the same thing — random initialisation breaks the symmetry.
• No seed. Without one, results change every run and debugging gets harder.

Key takeaways#

• A neural network is matrix multiplications, activations, a loss and gradient updates.
• Backpropagation is the chain rule applied from the loss back to every weight.
• A hidden layer with a non-linear activation lets networks learn non-linear patterns like XOR.
• Frameworks automate the backward pass — but it's the same maths you just wrote.