Advanced Retrieval Techniques

Why pure vector search isn't always enough#

Vector search (covered in this track's embeddings-and-vector-databases page) is excellent at semantic similarity — finding text that means the same thing even with different wording — but it has real, well-documented blind spots: it can miss an exact rare term, a specific product SKU, a code symbol, or an acronym that doesn't carry strong "meaning" signal the way a full sentence does, because a short exact identifier often doesn't embed distinctively from similar-looking ones. Production RAG systems generally don't rely on vector search alone once retrieval quality actually matters — they combine it with older, more literal techniques that vector search doesn't replace, only complement.

Hybrid search: keyword + vector#

Hybrid search runs both a traditional keyword search (commonly BM25, a well-established statistical ranking function that scores documents by term overlap and rarity) and a vector similarity search against the same query, then combines both result sets into a single ranking.

def hybrid_search(query, vector_store, keyword_index, k=10, alpha=0.5):
    vector_results = vector_store.search(embed(query), k=k)
    keyword_results = keyword_index.search(query, k=k)

    # Combine scores from both result sets (e.g. via reciprocal rank fusion),
    # weighting how much each method contributes via alpha
    combined = merge_and_rerank(vector_results, keyword_results, alpha=alpha)
    return combined[:k]

The BM25 side reliably catches the exact product code, error message, or rare technical term that vector search alone might rank low, while the vector side still catches semantically related content that shares no exact wording with the query. Neither approach dominates the other across all query types, which is exactly why combining them tends to outperform either one alone in practice.

Reranking#

Both keyword and vector search are optimized to be fast across a large corpus, which means they use relatively cheap scoring to narrow millions of documents down to a top-k shortlist quickly. Reranking adds a second, more expensive but more accurate pass: a specialized reranking model scores the shortlist of candidates (typically the top 20-50 from the first pass) against the query directly and re-orders them, since it's only being asked to compare a small, already-narrowed set rather than search the entire corpus.

candidates = hybrid_search(query, vector_store, keyword_index, k=30)
reranked = reranker_model.rerank(query, candidates)
top_chunks = reranked[:5]  # the final, most relevant chunks handed to the LLM

This two-stage pattern — cheap, fast retrieval to get a shortlist, then an expensive, accurate reranker to pick the true best few from that shortlist — is standard in production search systems generally, not unique to RAG, because running the expensive, accurate scoring against the entire corpus for every query would be far too slow.

Query expansion#

A user's literal query is sometimes a poor match for how the answer is actually phrased in the source documents — the classic vocabulary mismatch problem. Query expansion uses an LLM to generate additional related phrasings, sub-questions, or synonyms for the original query, then retrieves using all of them and merges the results:

Original query: "How do I get my money back?"

Expanded queries (generated by an LLM):
- "What is the refund policy?"
- "How to request a refund"
- "Return and reimbursement process"

Retrieving with all four phrasings and merging the results substantially increases the chance of hitting a document that used different wording than the user's original question, at the cost of extra retrieval calls (and an extra LLM call to generate the expansions in the first place).

Common mistakes#

• Relying on vector search alone for a corpus that includes a lot of exact identifiers, codes, or rare technical terms — this is precisely the case where pure semantic search underperforms and hybrid search earns its complexity.
• Reranking the entire corpus instead of a pre-narrowed shortlist — reranking models are accurate but too slow to run against millions of documents directly; they're meant to refine an already-cheap first pass, not replace it.
• Using query expansion without deduplicating the merged results — the same chunk can easily surface from multiple expanded queries, and retrieving it redundantly wastes context window space that could hold a genuinely different relevant chunk instead.