LLM Interview Questions

A curated set of LLM-fundamentals interview questions — the vocabulary and mental models that come up before any discussion of prompting or fine-tuning specific models.

Fundamentals#

Q: What is a token, and why don't LLMs just process whole words? A token is a sub-word unit produced by a fixed tokenization algorithm — it can be a whole common word, a punctuation mark, or a fragment of a longer or rarer word (for example, "unbelievable" might split into "un", "believ", "able"). Using sub-word tokens rather than whole words lets a model handle a fixed, manageable vocabulary size while still being able to represent rare words, made-up words, and misspellings by combining smaller known pieces, rather than needing a separate vocabulary entry for every possible word.

Q: Why do LLMs have a limited context window instead of unlimited memory? The self-attention mechanism at the core of the transformer compares every token against every other token, which costs roughly the square of the sequence length in compute — doubling the input length roughly quadruples the attention computation. That quadratic scaling makes arbitrarily long context prohibitively expensive and slow, so every model ships with a fixed maximum number of tokens (input plus output combined) it can consider at once.

Q: Explain self-attention conceptually, without the matrix math. For every token in a sequence, self-attention computes how relevant every other token is to understanding it, and blends information from those other tokens weighted by relevance. In the sentence "The trophy didn't fit in the suitcase because it was too big," resolving what "it" refers to requires attending heavily to "trophy" and "fit" — self-attention gives the model a mechanism to do exactly that, for every token, simultaneously.

Q: Why did transformers replace RNNs for language modeling? RNNs process a sequence strictly one token at a time, so training and inference can't be parallelized across the sequence, and information from early tokens can degrade by the time the model reaches much later ones. Transformers replace that sequential dependency with self-attention, where every token can attend to every other token in a single parallel computation — this trains dramatically faster on modern hardware and gives every token direct access to every other token's information regardless of distance.

Q: What does it mean that an LLM is trained on "next-token prediction"? The model's entire training objective is: given some text, predict what token comes next, across an enormous amount of text. It sounds narrow, but doing this well at massive scale forces the model to implicitly learn grammar, facts, reasoning patterns, and code structure — which is why a model trained on this single simple objective ends up able to converse, summarize, and answer questions, not just autocomplete individual sentences.

Choosing the right adaptation approach#

Q: When would you choose RAG over fine-tuning for a factual knowledge gap? When the missing knowledge is specific facts that change over time or are private to your organization, RAG is almost always the better fit — updating a RAG system means updating a document store, with no retraining required, and answers stay checkable against a real source. Fine-tuning is a poor tool for teaching new, precise facts; it blends new information into the model's weights fuzzily rather than storing it as a reliable, retrievable fact, and updating a fact learned this way means retraining again.

Q: Why do LLMs hallucinate, and is it something a better model eliminates? LLMs are trained to produce a statistically plausible continuation of text, with no built-in mechanism distinguishing "I actually know this" from "this is a good guess" — both get generated with the same fluent, confident tone. It's a structural consequence of how these models are trained and generate text, not a simple bug; better models and better mitigation (grounding via RAG, explicit permission to say "I don't know," lower temperature for factual tasks) meaningfully reduce it, but no current approach eliminates it entirely.

Evaluation and production#

Q: What's the difference between a benchmark score and human evaluation, and why use both? A benchmark score measures performance on a fixed, standardized set of tasks with known answers — objective and comparable across models, but not necessarily predictive of how a model performs on your specific real-world application. Human evaluation judges actual output quality directly, which is the closest available proxy to real-world usefulness, but is slow, expensive, and subject to disagreement between raters. Production teams typically use benchmarks for quick, broad comparisons and human evaluation (often supplemented by LLM-as-judge for speed) to validate quality on their actual use case.

Q: What are the main biases to watch for when using LLM-as-judge evaluation? Judge models have documented tendencies to favor longer answers regardless of whether the extra length adds real value, to favor a particular writing style over substance, and in some setups to favor whichever answer is presented first or last in the comparison prompt. Because of this, LLM-as-judge scores are best treated as a fast, cheap, directional signal to combine with periodic human spot-checks, not as an unquestionable ground truth on their own.

Q: Why does caching often deliver a bigger cost win than switching to a cheaper model? Switching models still pays for every single call, just at a lower per-token rate. Caching (exact-match or semantic) can avoid the call entirely for repeated or near-identical requests, which is a larger saving whenever real traffic contains meaningful repetition — a common FAQ, similar classification inputs, or repeated sub-questions in a workflow. The two aren't mutually exclusive; a well-optimized production system typically does both.