Deep Learning Introduction
What deep learning is, how it relates to machine learning and AI, why it took off, and where it is used today.
What is deep learning?
Deep learning is a branch of machine learning that uses neural networks with many layers to learn patterns directly from raw data — pixels, audio waveforms, text. "Deep" refers to the number of layers stacked between input and output.
It's the technology behind image recognition, speech recognition, machine translation, self-driving perception, and every modern large language model.
AI vs machine learning vs deep learning
Artificial Intelligence → machines doing tasks that need "intelligence"
└── Machine Learning → systems that learn patterns from data
└── Deep Learning → ML with multi-layer neural networks
└── Transformers / LLMs → today's most powerful deep learning models
| Classic machine learning | Deep learning | |
|---|---|---|
| Features | Engineered by humans (e.g. "word counts", "average pixel brightness") | Learned automatically from raw data |
| Data needed | Works with small–medium datasets | Shines with large datasets |
| Hardware | CPU is usually fine | GPUs/TPUs for training |
| Interpretability | Often easier to explain | Harder to explain |
| Best for | Tabular data (spreadsheets, databases) | Images, audio, text, video |
A useful rule: for tables of numbers, start with classic ML (like gradient-boosted trees). For images, sound and language, deep learning usually wins.
The building block: an artificial neuron
A neuron takes inputs, multiplies each by a weight, adds a bias, and passes the result through an activation function:
output = activation(w1·x1 + w2·x2 + … + wn·xn + b)
Stack many neurons into a layer, and stack layers into a network. Each layer learns a slightly more abstract representation than the one before it — in an image model, early layers detect edges, middle layers detect shapes, and later layers detect objects.
Activation functions
Without non-linear activations, a deep network collapses into one linear function. Common choices:
| Activation | Formula (idea) | Typical use |
|---|---|---|
| ReLU | max(0, x) | Default for hidden layers |
| GELU | Smooth ReLU-like curve | Transformers |
| Sigmoid | Squashes to 0–1 | Binary output probability |
| Tanh | Squashes to −1–1 | Older RNNs, small networks |
| Softmax | Turns scores into probabilities that sum to 1 | Multi-class output |
How a network learns
Training repeats four steps over the data many times (epochs):
- Forward pass — compute predictions from inputs.
- Loss — measure how wrong the predictions are (e.g. cross-entropy for classification, mean squared error for regression).
- Backward pass (backpropagation) — compute how each weight affected the loss.
- Update — nudge every weight to reduce the loss using an optimizer such as SGD or Adam.
You'll code all four by hand in the next lesson.
Why deep learning took off after 2012
- Data — the internet produced massive labelled datasets (e.g. ImageNet).
- GPUs — graphics cards made the huge matrix multiplications fast.
- Better techniques — ReLU, dropout, batch normalisation, better initialisation and optimizers like Adam.
- Frameworks — PyTorch and TensorFlow made building networks accessible.
- Transformers (2017) — an architecture that scales extremely well, leading to modern LLMs.
Main types of neural networks
| Architecture | Designed for | Example uses |
|---|---|---|
| MLP (fully connected) | Fixed-size vectors | Tabular data, small tasks |
| CNN (convolutional) | Grid data like images | Image classification, medical scans |
| RNN / LSTM | Sequences | Older speech and text models, time series |
| Transformer | Sequences with attention | LLMs, translation, vision transformers |
| Diffusion model | Generation by denoising | Image and audio generation |
Where you'll use deep learning
Computer vision (face unlock, defect detection), NLP (chatbots, translation, summarisation), speech (voice assistants, captions), recommendation systems, healthcare imaging and generative AI.
Common mistakes
- Using deep learning for small tabular datasets. Gradient-boosted trees are often more accurate and far simpler.
- Skipping the basics. Understanding loss, gradients and overfitting matters more than memorising architectures.
- Training without a validation set. You can't see overfitting without held-out data.
- Expecting a GPU to fix bad data. Data quality limits every model.
Key takeaways
- Deep learning = multi-layer neural networks that learn features from raw data.
- Neurons compute weighted sums plus a bias, then a non-linear activation.
- Training loops through forward pass → loss → backpropagation → update.
- CNNs suit images, transformers suit language — and power today's LLMs.