Tokens & Context Windows

What tokens actually are#

LLMs don't process text as words or characters — they process it as tokens, which are sub-word pieces produced by a fixed tokenization algorithm (commonly byte-pair encoding or a close variant). A token might be a whole common word, a punctuation mark, a single character, or a fragment of a longer word.

Take the word "unbelievable." A typical modern tokenizer might split it into three tokens:

"unbelievable" → ["un", "believ", "able"]

Common short words ("the", "is", "cat") are usually a single token each, because they appear so often in training data that the tokenizer learned to represent them whole. Rarer or longer words, especially technical jargon, made-up words, or non-English text, tend to split into multiple sub-word tokens. This is also why a rough rule of thumb like "1 token ≈ 4 characters of English text" only holds on average — it varies a lot by language and vocabulary.

# Conceptual illustration — actual tokenizers differ in their exact vocabulary
text = "unbelievable"
tokens = tokenizer.encode(text)
# tokens -> [1917, 6842, 494]   (three integer token IDs)
# tokenizer.decode(tokens) -> "unbelievable"

Every LLM API bills and measures limits in tokens, not words or characters — which is why the same sentence in English and in a language with less training data representation can cost noticeably different numbers of tokens.

Context window: a hard limit#

A model's context window is the maximum number of tokens it can take into account at once — spanning the input prompt (system instructions, conversation history, any retrieved documents) and the output it generates, combined. If a model has a 32,000-token context window and your conversation history plus retrieved documents already total 31,000 tokens, there's only 1,000 tokens of room left for the model's own response, and none left over for further conversation without dropping earlier content.

This is a hard architectural limit, not a soft guideline — content beyond the context window is simply not visible to the model at all. Applications that need to work with content larger than the context window (a long document, a long-running conversation) have to actively manage this: summarizing or dropping older conversation turns, or retrieving only the most relevant chunks of a large document rather than including all of it (the approach covered in this site's RAG tutorials).

Why longer context costs more compute#

The core mechanism inside a transformer — self-attention — has every token attend to every other token in the sequence, to decide how relevant each one is. For a sequence of length n, that means roughly n × n (quadratic) attention computations, not n (linear). Doubling the input length roughly quadruples the attention computation, not just doubles it.

Sequence length:  1,000 tokens  →  ~1,000,000 attention computations
Sequence length:  2,000 tokens  →  ~4,000,000 attention computations (4×, not 2×)

This quadratic growth is the fundamental reason context windows historically couldn't just be made arbitrarily large — cost and latency scale faster than the context length itself. It's also why a lot of active research and engineering effort (more efficient attention variants, smarter caching of previously-computed attention) has gone specifically into making longer context windows practical, and why providers price long-context requests noticeably higher than short ones.

Common mistakes#

• Assuming "tokens" roughly equal "words" — code, non-English text, and rare words routinely use far more tokens per word than plain English prose does.
• Forgetting that the context window covers input and output together — a long prompt leaves proportionally less room for a long response.
• Assuming you can just keep appending an entire growing conversation history forever — eventually it exceeds the context window (and gets expensive well before that), which is why real applications truncate, summarize, or selectively retrieve older context instead.