RAG Introduction

What Retrieval-Augmented Generation is#

An LLM's knowledge comes entirely from what it saw during training — a fixed snapshot of text, frozen at whatever point its training data was collected. That creates two concrete problems for real applications: the model has no idea about your private data (your company's internal documents, a customer's account history, last week's support tickets), and it can't know about anything that happened after its training cutoff. Worse, when an LLM doesn't actually know something, it doesn't reliably say so — it can generate a fluent, confident-sounding answer that's simply wrong, a failure mode usually called hallucination.

Retrieval-Augmented Generation (RAG) addresses both problems with one idea: instead of relying purely on what the model memorized during training, retrieve relevant, real documents at the moment of the question, and give the model those documents as context to base its answer on. The model's job shifts from "recall this from memory" to "read these specific retrieved passages and answer using them" — much closer to how a person would answer a question by first looking something up, rather than answering from memory alone.

The high-level flow#

User question
     │
     ▼
Retrieve relevant chunks of your own documents (search a knowledge base)
     │
     ▼
Construct a prompt that includes those retrieved chunks as context
     │
     ▼
The LLM generates an answer grounded in that retrieved context

Concretely: a user asks "What's our refund policy for enterprise customers?" Instead of hoping the model somehow already knows your company's specific policy (it doesn't — that document was never part of any public training data), a RAG system searches your own internal documentation, finds the actual refund policy section, inserts that text into the prompt alongside the question, and asks the model to answer using it. The answer is now grounded in a real, current, checkable source document, not the model's frozen and possibly outdated training data.

This also makes answers auditable in a way pure model recall isn't: because the system knows exactly which documents were retrieved and handed to the model, an application can show the user "here's the source this answer came from," which both builds trust and makes it possible to catch cases where the model still got something wrong despite having the right source in front of it.

Common mistakes#

• Assuming RAG makes hallucination impossible — it substantially reduces it by grounding answers in real text, but a model can still misread, misquote, or mix up the retrieved context, especially if the retrieved passages are irrelevant, contradictory, or too fragmented to make sense on their own.
• Treating RAG as a replacement for good retrieval quality — if the retrieval step returns irrelevant or missing chunks, the generation step has nothing good to work with; RAG's answer quality is bottlenecked by retrieval quality, not just the model's generation ability.
• Reaching for RAG when the real need is up-to-date, structured data lookup (for example, "what's this customer's current account balance") — that's often better served by a direct database/API call than by embedding and retrieving documents at all.