Production RAG Architecture

Beyond the toy pipeline#

The pipeline covered in this track's building-a-rag-pipeline page — chunk, embed, store, retrieve, generate — is the right mental model, but a production system wraps real infrastructure around every one of those steps: an ingestion pipeline that runs continuously rather than once, caching to control cost and latency, and monitoring to catch quality regressions before users do. This page walks through a full production architecture, stage by stage, the way you'd actually explain it on a whiteboard.

Stage 1: the ingestion pipeline#

Production document sources aren't a folder you chunk once — they're wikis, ticketing systems, CMS content, and internal databases that change continuously. A production ingestion pipeline typically runs as a scheduled or event-triggered job, not a one-off script:

Source systems (wiki, CMS, tickets, docs)
        |
        v
Extraction (pull raw content + metadata: source, author, last-updated date)
        |
        v
Cleaning (strip boilerplate/navigation, normalize formatting, deduplicate)
        |
        v
Chunking (split into passages, with overlap -- see below)
        |
        v
Embedding (batch-embed chunks through the embedding model)
        |
        v
Upsert into the vector store (update existing chunks, don't just append)

Note the word "upsert," not "insert" — a production pipeline has to handle a document being edited, not just created, which means finding and replacing that document's previous chunks rather than accumulating stale duplicates alongside the new version indefinitely.

Stage 2: chunking strategy, revisited#

Chunk size and overlap (covered in this track's building-a-rag-pipeline page) are only part of a production chunking strategy — the other part is respecting document structure rather than splitting purely by character or token count. Splitting a table in half, or cutting a section header off from the paragraph beneath it, produces chunks that are individually less coherent and less retrievable. Production chunking typically splits along natural document boundaries first (headings, paragraphs, list items) and only falls back to a fixed size limit within a boundary that's still too large on its own — preserving structure wherever the structure itself is a natural, sensibly-sized unit.

Stage 3: retrieval and reranking#

A production retrieval step is rarely single-stage vector search alone — it typically layers hybrid search (keyword + vector) and reranking (both covered in this track's advanced-retrieval-techniques page) to get a materially better shortlist of chunks before they're ever handed to the LLM.

Stage 4: caching#

Two distinct caching layers pay off in a production RAG system:

• Embedding cache — avoids re-embedding identical or previously-seen queries, since embedding is itself a model call with its own latency and cost.
• Answer cache — for genuinely repeated or near-duplicate questions (a common FAQ), skips the entire retrieve-and-generate pipeline and returns a previously-computed answer directly, the single biggest latency and cost win available when traffic has real repetition (see this site's LLM tutorials on production deployment and cost).

Stage 5: generation and guardrails#

The generation step applies the grounding instructions covered throughout this track (answer only from context, say "I don't know" when the context doesn't cover it), typically alongside a lightweight validation pass — checking the response isn't empty, doesn't exceed a length bound, and doesn't obviously contradict the retrieved context — before it's returned to the caller.

Stage 6: monitoring#

A RAG system's quality can degrade silently in ways a simple uptime check will never catch — the service keeps responding with HTTP 200s while quietly answering worse. Production monitoring for RAG typically tracks:

• Retrieval quality signals — such as the average similarity score of returned chunks trending down over time, which often signals the corpus has drifted away from what the embedding model or queries expect.
• "I don't know" rate — a rising rate of the model declining to answer can mean the corpus has a growing coverage gap for what users are actually asking.
• Latency and cost per stage — tracked separately for retrieval versus generation, so a regression can be traced to the specific stage that caused it rather than "the whole thing got slower."
• User feedback signals — thumbs up/down, follow-up "that's not right" messages, or support escalations that started from a RAG-generated answer, as a real-world quality signal that automated metrics alone can miss.

The full picture#

Ingestion (continuous)        Query time
------------------------      ------------------------------------
Extract -> Clean -> Chunk      User query
     |                              |
     v                              v
  Embed                    Embedding cache check
     |                              |
     v                              v
Upsert into vector store    Hybrid search (keyword + vector)
                                     |
                                     v
                              Reranking (shortlist -> top-k)
                                     |
                                     v
                              Answer cache check
                                     |
                                     v
                           Generation (grounded, with guardrails)
                                     |
                                     v
                              Response + monitoring signals

Common mistakes#

• Treating ingestion as a one-time script instead of a running pipeline — production sources change continuously, and a stale index quietly serving outdated answers is a common, easy-to-miss failure mode (see this track's common-rag-failure-modes page).
• Chunking purely by a fixed character/token count with no regard for document structure — this routinely splits tables, code blocks, and headers away from their content in ways that make the resulting chunks individually harder to retrieve or understand.
• Monitoring only uptime and latency, with no visibility into retrieval or answer quality — a RAG system can be "up" and fast while quietly getting worse at actually answering correctly, and uptime monitoring alone will never surface that.