Neural Networks Basics

The perceptron: the simplest possible "neuron"#

A perceptron is the simplest building block of a neural network: it takes several numeric inputs, multiplies each by a learned weight, sums the results (plus a learned "bias" term), and passes that sum through a simple function to produce a single output.

output = activation(w1*x1 + w2*x2 + w3*x3 + bias)

A single perceptron can only learn a linearly separable decision boundary — it can draw one straight line (or, in higher dimensions, one flat plane) to separate two classes. That's a hard ceiling: plenty of real patterns simply aren't separable by any single straight line, no matter how the weights are tuned — the classic textbook example is XOR (output true if exactly one of two binary inputs is true), which no single perceptron can learn regardless of training.

Stacking perceptrons: the multi-layer network#

A multi-layer neural network (also called a multi-layer perceptron) stacks many of these simple units into layers: an input layer (the raw features), one or more hidden layers (each made of many perceptron-like units, each connected to every unit in the layer before it), and an output layer (the final prediction).

Input layer        Hidden layer         Output layer
  x1  --.               .--o--.
        |--> o  o  o  o--|      |--> prediction
  x2  --'   (many units,  '--o--'
             each summing weighted
             inputs + activation)

The key insight: each hidden layer's units can each learn a different, simple pattern in the data, and stacking layers lets the network combine those simple patterns into progressively more complex ones — which is what lets a multi-layer network learn decision boundaries far more complex than any single straight line, including patterns like XOR that stump a single perceptron.

Activation functions#

Without a non-linear activation function between layers, stacking any number of layers mathematically collapses back down to a single linear transformation — no more expressive than one perceptron, no matter how many layers you add. The activation function is what actually gives depth its power:

• Sigmoid — squashes any input into a range between 0 and 1; historically common, still used for a final binary-classification output, but prone to a problem called "vanishing gradients" in deep networks (more on this below).
• ReLU (Rectified Linear Unit) — outputs the input directly if positive, and 0 otherwise (max(0, x)). Simple, computationally cheap, and the most common default choice for hidden layers in modern networks, largely because it suffers far less from vanishing gradients than sigmoid does.
• Softmax — used on the output layer for multi-class classification, converting a set of raw scores into a probability distribution across classes that sums to 1.

Backpropagation, at a high level#

Training a neural network means finding the weights that minimize the difference between its predictions and the true labels (measured by a loss function). Backpropagation is the algorithm that computes exactly how much each individual weight in the network contributed to the final error, working backward from the output layer to the input layer, using the chain rule from calculus:

1. Forward pass — input data flows through the network, layer by layer, producing a prediction.
2. Compute the loss — compare the prediction to the true label using a loss function (for example, mean squared error for regression, cross-entropy for classification).
3. Backward pass — starting from the loss, compute how much each weight (working backward, layer by layer) contributed to that error.
4. Update the weights — nudge every weight slightly in the direction that would have reduced the error, scaled by a learning rate (how large a step to take).

This four-step cycle repeats over many passes through the training data ("epochs") until the loss stops meaningfully improving. The name "backpropagation" refers specifically to step 3 — propagating the error signal backward through the network so every weight, no matter how deep, gets a specific, computed share of "blame" for the final error, which is what makes training a network with many layers computationally tractable at all.

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(4, 8),   # input layer -> hidden layer (4 features -> 8 units)
    nn.ReLU(),
    nn.Linear(8, 1),   # hidden layer -> output
    nn.Sigmoid(),      # squash to a 0-1 probability for binary classification
)

loss_fn = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

prediction = model(X_train)
loss = loss_fn(prediction, y_train)
loss.backward()       # backpropagation: computes each weight's contribution to the loss
optimizer.step()      # updates every weight accordingly

Common mistakes#

• Assuming a single perceptron (no hidden layers) can learn any pattern — it's fundamentally limited to linearly separable problems, and needs at least one hidden layer with a non-linear activation to learn anything more complex.
• Using no activation function (or a linear one) between layers — this collapses the entire network back to a single linear transformation, no matter how many layers were stacked, defeating the purpose of depth entirely.
• Setting the learning rate too high or too low without checking training behavior — too high causes the loss to bounce around or diverge instead of steadily decreasing; too low makes training so slow it looks like the model isn't learning at all, when it's just progressing extremely gradually.