Retrieval Chains & RAG Integration

Why retrieval belongs inside a chain#

RAG (covered in depth in this site's RAG tutorials) has the same shape regardless of framework: embed a query, search a vector store for relevant chunks, hand those chunks to the model as context, generate an answer. LangChain doesn't change any of that — what it adds is a standard, composable way to wire "search a vector store" into the same |-pipe chain style used for prompts and models, so retrieval becomes just another swappable step rather than a separate, hand-wired subsystem sitting outside the rest of the application's chain logic.

Retrievers: a standard interface over any vector store#

A retriever in LangChain is a small, standardized interface: give it a query string, get back a list of relevant documents. Every supported vector store (Pinecone, Chroma, pgvector, Qdrant, and others) exposes this same retriever interface, so an application's chain logic doesn't need to change if the underlying vector store is swapped out later.

from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

vector_store = Chroma(
    collection_name="support-docs",
    embedding_function=embeddings,
    persist_directory="./chroma_db",
)

# Ingestion, done once ahead of time -- see this site's RAG tutorials for chunking strategy
vector_store.add_texts([
    "Enterprise customers can request a refund within 45 days of purchase.",
    "Standard customers can request a refund within 30 days of purchase.",
])

retriever = vector_store.as_retriever(search_kwargs={"k": 3})
retriever.invoke("What's the refund policy for enterprise customers?")
# -> [Document(page_content="Enterprise customers can request a refund within 45 days..."), ...]

A full RAG chain#

Composing a retriever into a chain follows the same |-pipe style as any other LangChain chain — the only new idea is a step that formats retrieved documents into a text block the prompt template can consume:

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import ChatOpenAI

prompt = ChatPromptTemplate.from_template("""Answer the question using ONLY the
context below. If the answer isn't in the context, say you don't know.

Context:
{context}

Question: {question}""")

model = ChatOpenAI(model="gpt-4o-mini")

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | model
    | StrOutputParser()
)

rag_chain.invoke("What's the refund policy for enterprise customers?")
# -> "Enterprise customers can request a refund within 45 days of purchase."

The dictionary at the start of the chain runs two branches in parallel against the same input: the question key passes the raw input straight through (RunnablePassthrough), while the context key sends that same input through the retriever, then formats the returned documents into one text block. Both results land as variables in the prompt template, which then flows into the model and parser exactly like any non-retrieval chain.

Common mistakes#

• Wiring the retriever directly into the prompt template without a formatting step — a prompt template expects a string variable, not a list of Document objects, so skipping format_docs (or an equivalent) produces a garbled or broken prompt.
• Forgetting that ingestion (embedding and storing documents) and querying use the same embedding model — mismatched embedding models between what was stored and what's used to embed the query silently produces poor retrieval, not an error.
• Hardcoding a fixed k in the retriever and never revisiting it — too small risks missing the one chunk that had the answer; too large crowds the prompt with marginal content (see this site's RAG tutorials for tuning k properly).