Building Production-Grade RAG Pipelines with Hybrid Search and Reranking

Retrieval-Augmented Generation (RAG) has transformed how large language models interact with proprietary and domain-specific knowledge bases. However, moving from an early prototype to a reliable production system reveals acute weaknesses in standard vector-only retrieval: loss of exact keyword context, embedding drift, and irrelevant chunk noise.

In this article, we will examine how to engineer a resilient, low-latency hybrid retrieval architecture incorporating BM25 sparse retrieval, dense embeddings, and cross-encoder rerankers.

The Limitations of Pure Vector Search

Dense vector embeddings (such as text-embedding-3-small or open-weight models like bge-large-en) represent semantic meaning in continuous vector spaces. While excellent for thematic matching, they often fail when queries contain:

1. Unique alphanumeric identifiers (e.g., ticket IDs, error codes, SKU numbers).
2. Rare industry jargon or acronyms not prevalent in the training corpus.
3. Exact entity negation or exact phrase matches.

# A common issue with pure cosine similarity retrieval
from langchain_community.vectorstores import Qdrant
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
# Pure dense retrieval often retrieves conceptually similar text 
# instead of the exact specification rule requested

The Hybrid Architecture: Dense + Sparse Fusion

To overcome this, production systems pair sparse lexical search (BM25) with dense semantic search, followed by Reciprocal Rank Fusion (RRF):

def reciprocal_rank_fusion(dense_results: list[str], sparse_results: list[str], k: int = 60) -> dict[str, float]:
    """Combines rankings from dense and sparse search using RRF."""
    rrf_scores = {}
    
    for rank, doc_id in enumerate(dense_results):
        rrf_scores[doc_id] = rrf_scores.get(doc_id, 0.0) + (1.0 / (k + rank + 1))
        
    for rank, doc_id in enumerate(sparse_results):
        rrf_scores[doc_id] = rrf_scores.get(doc_id, 0.0) + (1.0 / (k + rank + 1))
        
    return dict(sorted(rrf_scores.items(), key=lambda item: item[1], reverse=True))

Why Reciprocal Rank Fusion Works

RRF avoids the problem of score normalization across different vector spaces and scoring mechanisms. Instead of attempting to normalize cosine similarities (ranging from -1 to 1) with arbitrary BM25 scores (ranging from 0 to ∞), RRF operates strictly on the ordinal rank position of each candidate.

Cross-Encoder Reranking

Retrieving the top 30 candidates with hybrid fusion and feeding all of them directly to your LLM generator bloats the prompt context window, degrades latency, and leads to the "Lost in the Middle" phenomenon.

A Cross-Encoder model (e.g., bge-reranker-large or Cohere Rerank) scores the query and candidate chunk jointly, computing full token-to-token cross-attention:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-large")

def rerank_documents(query: str, candidates: list[dict], top_k: int = 5) -> list[dict]:
    pairs = [[query, doc["content"]] for doc in candidates]
    scores = reranker.predict(pairs)
    
    for doc, score in zip(candidates, scores):
        doc["rerank_score"] = float(score)
        
    sorted_docs = sorted(candidates, key=lambda x: x["rerank_score"], reverse=True)
    return sorted_docs[:top_k]

Measuring Production Retrieval Quality

Before deploying any pipeline modifications, you must benchmark using retrieval evaluation metrics:

• Hit Rate@K: Measures whether the ground-truth reference chunk appears in the top K retrieved candidates.
• Mean Reciprocal Rank (MRR): Evaluates the specific rank position of the first relevant chunk.
• Context Relevance & Faithfulness: Using Ragas or TruLens to evaluate hallucinations and generation grounding.

By combining hybrid search with cross-encoder reranking, our benchmark showed a +23% increase in Hit Rate@5 and cut hallucinations by nearly 35% on complex queries.