Deep Learning Interview Questions

Deep learning interview questions#

These questions appear in ML engineer, data scientist and AI engineer interviews. Aim to explain the intuition first, then the maths if asked.

1. What is the difference between machine learning and deep learning?#

Deep learning is a subset of ML that uses multi-layer neural networks to learn features automatically from raw data. Classic ML usually relies on human-engineered features and works well on smaller, tabular datasets.

2. Why do neural networks need activation functions?#

Without non-linear activations, any stack of layers collapses into a single linear transformation, so the network could only learn linear relationships. Activations like ReLU let networks model complex, non-linear patterns.

3. Explain backpropagation.#

After the forward pass computes the loss, backpropagation applies the chain rule from the output layer backwards to compute the gradient of the loss with respect to every weight, reusing intermediate results so it's efficient. An optimizer then uses those gradients to update the weights.

4. What is the vanishing gradient problem?#

Gradients become tiny as they're multiplied back through many layers (especially with sigmoid or tanh), so early layers learn very slowly. Fixes: ReLU-family activations, residual connections, normalisation layers, careful initialisation, and LSTMs/transformers for sequences.

5. SGD vs Adam?#

SGD updates weights with a single global learning rate. Adam adds momentum and per-weight adaptive step sizes, usually converging faster with less tuning. Well-tuned SGD with momentum can generalise as well or better on some vision tasks; AdamW is the default for transformers.

6. What is overfitting and how do you prevent it?#

The model memorises training data and performs poorly on new data — training loss falls while validation loss rises. Prevention: more or augmented data, dropout, weight decay, early stopping, smaller models, and transfer learning.

7. What does dropout do?#

During training it randomly sets a fraction of activations to zero, forcing the network not to rely on any single neuron — a form of regularisation. It's turned off at inference time.

8. What is batch normalisation?#

It normalises a layer's activations using the mean and variance of the mini-batch, then applies a learned scale and shift. It stabilises and speeds up training. Transformers usually use layer normalisation instead, which normalises across features per example.

9. Why are CNNs good for images?#

Convolutions use small filters shared across the whole image, so they detect local patterns anywhere with far fewer parameters than fully connected layers. Stacked layers build a hierarchy from edges to objects.

10. Calculate the output size of a convolution.#

(input − kernel + 2 × padding) / stride + 1. For a 32 × 32 input, 5 × 5 kernel, padding 0 and stride 1: (32 − 5 + 0) / 1 + 1 = 28.

11. Why did LSTMs replace vanilla RNNs?#

LSTMs add a cell state controlled by forget, input and output gates, so information and gradients can flow across many time steps. This greatly reduces the vanishing gradient problem.

12. How does self-attention work?#

Each token produces a query, key and value. Attention scores are the scaled dot products of a token's query with every key, passed through softmax; the output is the weighted sum of values. Every token can directly use information from every other token.

13. Why are transformers faster to train than RNNs?#

They process all positions in parallel instead of step by step, which maps well to GPUs. The trade-off is attention cost that grows quadratically with sequence length.

14. What is transfer learning?#

Reusing a model pre-trained on a large dataset as the starting point for a new task, then fine-tuning it on a smaller dataset. It saves data, compute and time — standard practice for vision and NLP.

15. Training loss is falling but validation loss is rising. What do you do?#

That's overfitting. Stop at the best validation epoch (early stopping), add regularisation (dropout, weight decay), augment or collect more data, or reduce model size. Also check for data leakage or distribution differences between the splits.

16. What is the role of the learning rate, and what happens if it's wrong?#

It sets the update step size. Too high causes divergence or NaN losses; too low makes training extremely slow or stuck. Warm-up and decay schedules help.

Practice tip#

Re-implement the NumPy XOR network from memory, then rebuild it in PyTorch. If you can explain every line of both versions, you can answer most deep learning fundamentals questions confidently.