Hallucination & Mitigation

Why LLMs hallucinate#

An LLM is trained to do one thing: predict a plausible continuation of text, based on patterns learned from its training data. It has no separate "do I actually know this, or am I guessing" check built into its generation process — it produces the statistically likely next token whether or not that token corresponds to a true fact, and it produces both with the same fluent, confident tone. Hallucination is the name for the resulting failure mode: a model generating text that's fluent, plausible, and simply false — a fabricated citation, a nonexistent API method, a wrong date, a confidently invented statistic.

This isn't a bug that a future model version simply removes — it's a direct consequence of how these models are trained and how they generate text:

• No grounding in a source of truth. Unless a mechanism like RAG (see this site's RAG tutorials) explicitly gives the model real documents to work from, everything it "knows" is compressed into its trained weights as statistical patterns, not stored facts it can look up and verify.
• Training rewards plausible-sounding text, not "I don't know." Training data overwhelmingly contains confident, complete answers; a model that frequently hedges or refuses is actively working against patterns it learned to imitate, so it under-produces genuine uncertainty by default.
• Confabulation under pressure to be specific. Asked for a specific detail it doesn't actually have reliable information about — an exact citation, a precise statistic, a real API's exact method name — the model tends to generate something that has the shape of a correct answer, because that shape is what its training data taught it a correct answer looks like.
• Compounding errors in longer generations. In a long, multi-step answer, an early small inaccuracy can become the premise for later reasoning, so one hallucinated fact can snowball into an increasingly confident, increasingly wrong chain of downstream claims.

Concrete mitigation techniques#

• Ground answers in retrieved context (RAG). Explicitly instructing the model to answer using only provided context, and to say "I don't know" when the context doesn't cover the question, gives it a real source to check against rather than relying purely on its trained-in memory — the single most effective mitigation for factual questions with a known-good source.
• Ask for citations, and verify them. Prompting a model to cite exactly which part of the provided context supports each claim makes fabrication easier to catch — a claim with no matching source text is a clear signal to flag or discard, whereas an unsourced claim in free-form prose has no such check.
• Lower the temperature for factual tasks. Temperature controls how much randomness is injected into token selection; a lower temperature makes the model more consistently pick its highest-confidence continuation, which reduces (but doesn't eliminate) the odds of it wandering into a fabricated specific.
• Explicitly permit — and reward — "I don't know." A system prompt that explicitly tells the model it's acceptable and preferred to say it doesn't know, rather than guessing, measurably reduces confident fabrication, precisely because it counteracts the default training bias toward always producing a complete-sounding answer.
• Use self-consistency or a second verification pass. Generating multiple independent answers and checking for agreement (self-consistency, covered in this site's Prompt Engineering tutorials), or running a separate pass specifically asking "does the following answer's claims hold up against the provided source," catches a meaningful share of hallucinations a single generation wouldn't.
• Constrain the output format for verifiable tasks. Where possible, ask for structured output that can be checked against a real system (a database, an API schema) programmatically, rather than free-form prose that requires manual judgment to verify.

A worked example#

Ungrounded prompt (prone to hallucination):

What is Acme Corp's current enterprise refund policy?

Absent any real information about "Acme Corp" in its training data, a model may still confidently produce a plausible-sounding but entirely fabricated policy, because "answer the question" pressure outweighs "admit I don't actually know this."

Grounded, mitigated prompt:

Using ONLY the context below, answer the question. If the answer is not
present in the context, respond exactly with "I don't have that information."

Context:
"Enterprise customers may request a refund within 45 days of purchase,
provided the account is in good standing."

Question: What is Acme Corp's current enterprise refund policy?

The second version gives the model real material to ground its answer in, and an explicit, low-cost way out ("I don't have that information") that doesn't require it to guess — both measurably reduce fabrication compared to the ungrounded version.

Common mistakes#

• Treating hallucination as something a "smarter" or newer model eliminates entirely — it's a structural consequence of how these models generate text, meaningfully reducible through grounding and mitigation techniques, but not something any model version fully eliminates on its own.
• Assuming RAG alone is a complete fix — a model can still misread, misquote, or blend retrieved context incorrectly even when the right source document was retrieved and provided.
• Not testing for hallucination on the specific task and domain in production — a model's hallucination rate varies a lot by topic (it hallucinates far more confidently on obscure or highly specific facts than on well-known ones), so a generic "it seems reliable" impression from casual testing doesn't reliably predict behavior on your actual use case.