PyTorch Interview Questions

PyTorch interview questions#

These are common PyTorch questions in ML engineer, deep learning and AI engineer interviews. Be ready to write short code snippets for several of them.

1. What is a tensor?#

An n-dimensional array — like a NumPy array — that can run on CPUs or GPUs and track gradients. Every tensor has a shape, dtype and device.

2. What does requires_grad=True do?#

It tells autograd to record operations on that tensor so gradients can be computed for it with .backward(). Model parameters have it enabled automatically.

3. Explain autograd.#

PyTorch builds a computation graph during the forward pass. Calling loss.backward() applies the chain rule backwards through the graph and stores each parameter's gradient in its .grad attribute.

4. Why do we call optimizer.zero_grad()?#

Gradients accumulate by default. Without zeroing them, each step would add the new gradients to the old ones, producing wrong updates. (Accumulation is occasionally used on purpose to simulate larger batches.)

5. Write the core training loop.#

for X, y in loader:
    optimizer.zero_grad()
    loss = loss_fn(model(X), y)
    loss.backward()
    optimizer.step()

6. What's the difference between model.train() and model.eval()?#

They switch the behaviour of layers such as Dropout (active only in training) and BatchNorm (uses batch statistics in training, running statistics in eval). They do not turn off gradients — use torch.no_grad() or torch.inference_mode() for that.

7. torch.no_grad() vs torch.inference_mode()?#

Both disable gradient tracking. inference_mode is stricter and slightly faster; tensors created inside it can't later be used in autograd. Use it for pure inference.

8. What is a Dataset and a DataLoader?#

A Dataset defines how to get one example (__len__, __getitem__). A DataLoader wraps it to provide batching, shuffling and parallel loading.

9. nn.Module vs nn.Sequential?#

nn.Sequential chains layers in a straight line. Subclass nn.Module when you need custom logic in forward — branches, skip connections or multiple inputs and outputs.

10. Why use BCEWithLogitsLoss instead of Sigmoid + BCELoss?#

It combines both in a numerically stable way, avoiding overflow and log(0) issues. Likewise, CrossEntropyLoss expects raw logits and applies log-softmax internally.

11. How do you save and load a model?#

torch.save(model.state_dict(), "model.pt")
model = MyModel()
model.load_state_dict(torch.load("model.pt", weights_only=True))
model.eval()

Save the state_dict, not the pickled model object.

12. How do you move training to a GPU?#

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
X, y = X.to(device), y.to(device)

Every tensor in an operation must be on the same device.

13. How do you freeze layers?#

Set requires_grad_(False) on their parameters and pass only trainable parameters (or all of them — frozen ones get no gradients) to the optimizer. This is the basis of transfer learning.

14. What is view vs reshape?#

Both change shape. view requires the tensor to be contiguous in memory and always shares data; reshape works on any tensor and copies if necessary. When in doubt, use reshape.

15. Your loss becomes nan. What do you check?#

Learning rate too high, invalid inputs (nan/inf in data), log of zero in a custom loss, exploding gradients (add gradient clipping), or mixed-precision overflow.

16. What is gradient clipping?#

Capping the gradient norm before the optimizer step, e.g. torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0), to prevent exploding gradients — common for RNNs and transformers.

Practice tip#

Rebuild the circle classifier from this tutorial without looking: custom Dataset, DataLoader, model, training loop, validation and saving. If you can write it from memory, you're ready for most PyTorch coding questions.