Fine-Tuning vs RAG vs Prompting

Three different ways to adapt a model to your problem#

A pretrained LLM is a general-purpose base — genuinely useful out of the box, but rarely a perfect fit for a specific application's needs on day one. There are three fundamentally different levers for closing that gap: prompting (change what you ask, not the model), RAG (give the model access to specific documents at answer time), and fine-tuning (change the model's own weights through additional training). They solve different problems, and the single most common mistake in production LLM work is reaching for the wrong one — usually reaching for fine-tuning when the actual problem was a missing fact, which RAG solves far more cheaply.

Prompting: changing the instructions, not the model#

Prompting (covered in depth in this site's Prompt Engineering tutorials) means adjusting the input — instructions, examples, format requirements — while leaving the model itself completely unchanged. It's the cheapest, fastest lever by far: no training run, no infrastructure, changes take effect on the very next request.

Use prompting when: the model already has the knowledge or capability it needs, and the problem is really about getting consistent behavior, format, or tone out of it — not new facts and not a fundamentally new skill.

RAG: giving the model new facts at answer time#

RAG (covered in depth in this site's RAG tutorials) retrieves relevant documents at the moment of the question and inserts them into the prompt as context, so the model can answer using real, current, checkable material instead of relying purely on what it memorized during training.

Use RAG when: the problem is a knowledge gap — the model needs facts it wasn't trained on (your company's internal documents, something that happened after its training cutoff, a customer's specific account history) — and those facts change often enough that retraining for every update would be impractical. Updating a RAG system's knowledge just means updating the document store; there's no retraining step at all.

Fine-tuning: changing the model's own weights#

Fine-tuning takes a pretrained model and continues training it on a smaller, task-specific dataset, adjusting its internal weights so its default behavior shifts — its tone, its output format, how it handles a specialized task or domain-specific vocabulary — without needing to spell all of that out in every single prompt.

Use fine-tuning when: the problem is about behavior, not facts — you need the model to reliably produce a very specific output format on every response, adopt a particular voice or style consistently, or perform a specialized task (like structured extraction from a domain-specific document type) more accurately and more cheaply per call than a long, detailed prompt could achieve. Fine-tuning is a poor tool for teaching a model new, precise facts — it tends to blend new information in as a fuzzy statistical adjustment rather than a reliable, retrievable fact, and updating any of it means retraining again.

The decision framework#

Question	If yes, lean toward
Does the model need facts it wasn't trained on, or facts that change often?	RAG
Is the real problem inconsistent formatting, tone, or instruction-following?	Prompting (better prompt structure, few-shot examples)
Does the task need a consistently different behavior baked in, on every call, without repeating instructions?	Fine-tuning
Do you need the answer to cite a specific, checkable source document?	RAG
Is the desired behavior small and hard to describe precisely in words, but easy to demonstrate?	Fine-tuning (or few-shot prompting, at smaller scale)
Do you need to move fast, cheaply, and iterate quickly?	Prompting first, almost always

	Prompting	RAG	Fine-tuning
Cost to set up	Very low	Moderate (retrieval infrastructure)	High (data curation + training runs)
Cost to update	Free — edit the prompt	Low — update the document store	High — retrain
Good for new facts	No (bounded by context window)	Yes	Poor — facts blend in fuzzily
Good for behavior/format/style	Somewhat (with enough prompt detail)	Not directly	Yes — most reliable option
Answers are source-checkable	No	Yes	No
Iteration speed	Immediate	Fast	Slow (each change needs a training run)

They compose — this isn't an either/or choice#

In practice, many production systems use two or all three together: a fine-tuned model (for consistent output format and domain tone) that also uses RAG (for current, specific facts), driven through carefully engineered prompts (for the specific task framing on each call). The decision framework above is about which lever to reach for first to solve a specific, diagnosed problem — not a claim that a system can only ever use one.

Common mistakes#

• Reaching for fine-tuning to fix a factual gap — a model that doesn't know your return policy needs that policy handed to it as context (RAG), not a training run that will blend the fact in unreliably and go stale the moment the policy changes.
• Reaching for RAG to fix an inconsistent output format — if the model already has the right information but keeps formatting its answer differently every time, that's usually a prompting problem (a clearer format instruction, or few-shot examples), not a missing-document problem.
• Jumping straight to fine-tuning before trying a well-engineered prompt or a RAG setup — fine-tuning is the most expensive and slowest-to-iterate lever of the three, and it's common for a much simpler prompting or retrieval fix to solve the actual problem first.