Building a RAG Pipeline

The pipeline, step by step#

A working RAG system is a small pipeline with distinct stages, each of which has real design decisions that affect final answer quality.

1. Chunking documents#

LLMs (and embedding models) can't sensibly process an entire large document as one unit — both because of context window limits and because embedding a whole long document into one vector tends to blur together many different topics, making it a poor match for any specific question. So documents get split into chunks first: smaller passages, typically a few hundred tokens each.

Chunk size is a real trade-off: chunks too small lose surrounding context (a chunk containing only "...and that's why it failed." with no antecedent is useless on its own); chunks too large dilute relevance (a 5,000-token chunk that's mostly irrelevant to the actual question still gets embedded as one blurred vector, and wastes context window space if retrieved). A common practice is also to use overlap between consecutive chunks (for example, the last 50 tokens of one chunk repeated as the start of the next), so a fact that happens to sit right at a chunk boundary isn't split in a way that makes it unretrievable from either side.

2. Embedding chunks and storing them#

Each chunk is passed through an embedding model to produce a vector, and that vector — plus the original chunk text and metadata like source document/page — is stored in a vector database. This is typically done once, ahead of time, as an indexing/ingestion job, not on every user query.

3. Embedding the user's query#

When a user asks a question, that same embedding model converts the query into a vector, in the same vector space as the stored document chunks.

4. Retrieving the top-k similar chunks#

The system searches the vector database for the k stored chunk vectors closest to the query vector (commonly k is somewhere between 3 and 10, tuned per application) — these are the chunks judged most semantically relevant to the question.

5. Constructing the final prompt#

The retrieved chunks get inserted into a prompt template alongside the original question, explicitly instructing the model to answer using that context:

You are a helpful assistant. Answer the question using ONLY the context
provided below. If the answer isn't in the context, say you don't know —
do not guess.

Context:
[chunk 1 text]
[chunk 2 text]
[chunk 3 text]

Question: What's our refund policy for enterprise customers?

6. Generating the answer#

The model generates a response, now grounded in the actual retrieved text rather than relying solely on whatever it happened to memorize during training.

Tying it together#

def answer_question(question, vector_db, embed_model, llm):
    # Steps 1 & 2 happened earlier, offline, during ingestion:
    #   chunks = chunk_documents(all_documents, chunk_size=300, overlap=50)
    #   vector_db.store([(embed_model.embed(c), c) for c in chunks])

    # 3. Embed the incoming question
    query_vector = embed_model.embed(question)

    # 4. Retrieve the most relevant chunks
    top_chunks = vector_db.search(query_vector, k=5)

    # 5. Build a prompt that includes the retrieved context
    context = "\n\n".join(chunk.text for chunk in top_chunks)
    prompt = f"""Answer the question using ONLY the context below.
If the answer isn't in the context, say you don't know.

Context:
{context}

Question: {question}"""

    # 6. Generate the grounded answer
    return llm.generate(prompt)

Common mistakes#

• Picking a chunk size without testing it against real questions — too small loses context, too large dilutes relevance and wastes retrieval slots on mostly-irrelevant text.
• Skipping the "if the answer isn't in the context, say you don't know" instruction — without it, the model will often fall back to its own training data (or invent an answer) rather than admitting the retrieved context didn't actually cover the question.
• Retrieving too few or too many chunks without tuning k — too few risks missing the one chunk that actually had the answer; too many crowds the context window with marginally relevant text and can dilute the model's attention on the genuinely useful chunks.