How Transformers Work

The attention mechanism, conceptually#

The core idea behind the transformer is self-attention: for every token in a sequence, the model computes how much it should "pay attention to" every other token, in order to build a representation of that token's meaning in context — not just in isolation.

Consider this sentence, a classic example of pronoun ambiguity:

"The trophy didn't fit in the suitcase because it was too big."

What does "it" refer to — the trophy, or the suitcase? Humans resolve this instantly from context ("too big" implies the trophy didn't fit because it — the trophy — was too big, not the suitcase). Self-attention gives the model a mechanism to do the same thing: when processing the token "it", the model computes an attention score against every other token in the sentence, and learns (from patterns in its training data) to weight "trophy" much more heavily than "suitcase" for resolving what "it" refers to here. Change the sentence to "...because it was too small," and the correct referent flips to "suitcase" — a good model's attention pattern shifts accordingly, because that's the pattern needed to predict a sensible continuation.

Mechanically, each token produces three vectors — a query (what am I looking for?), a key (what do I contain?), and a value (what information do I actually offer if attended to). A token's query is compared against every other token's key to produce attention scores (how relevant each other token is), those scores are turned into weights, and the token's new representation becomes a weighted blend of every other token's value, according to those weights. This happens for every token, in every layer, simultaneously — which is what lets meaning propagate across an entire sentence (or document) in a handful of layers.

Encoder vs. decoder, briefly#

The original transformer paper described two halves:

• Encoder — reads the entire input at once and builds a rich contextual representation of it. Every token can attend to every other token, including ones that come after it. Good for understanding tasks (classification, extracting meaning from a fixed input).
• Decoder — generates output one token at a time, and each token can only attend to itself and the tokens before it (a restriction called "causal" or "masked" attention) — it can't peek at tokens it hasn't generated yet. Good for generation tasks.

Most modern general-purpose LLMs (the kind you interact with in a chat interface or via a completions API) are decoder-only: a single stack of decoder blocks that takes in a prompt and generates a continuation, one token at a time, using only causal attention. Encoder-only models (like BERT) are still widely used for tasks that just need to understand fixed text, such as classification or generating embeddings, without generating new text at all.

Why this parallelizes better than RNNs#

Before transformers, the standard architecture for sequence data was the recurrent neural network (RNN), which processes a sequence strictly one token at a time, feeding each step's output forward as input to the next step. That sequential dependency means token 500 can't be processed until token 499 has finished — training and inference are inherently serial, which is slow and makes it hard to fully exploit modern parallel hardware (GPUs/TPUs), and it also makes it harder for information from far-earlier tokens to survive intact by the time the model reaches token 500 (the "vanishing gradient" / long-range dependency problem).

Self-attention has no such sequential dependency — every token's attention computation against every other token can happen simultaneously, across the whole sequence, in one matrix multiplication. That means transformers parallelize dramatically better on modern hardware, which is a large part of why it became practical to train them on far more data than RNN-based models ever were, and why every token has direct, single-step access to every other token's information regardless of how far apart they are in the sequence.

Common mistakes#

• Thinking "attention" means the model literally re-reads the input like a human would — it's a learned, weighted combination over vector representations, computed in parallel across all tokens, not a sequential re-reading step.
• Assuming every modern LLM uses both an encoder and a decoder — most large general-purpose chat/completion models are decoder-only; encoder-decoder setups are more common for specific tasks like translation.
• Attributing the parallelization advantage to "more compute" alone — RNNs and transformers can use the same hardware, but only the transformer's architecture removes the sequential dependency that prevented RNNs from using that hardware efficiently in the first place.