Hybrid search with RRF: why combining vector and BM25 beats either alone

Vector search is not always better than keyword search. This is a fact that the vector database industry has been slow to acknowledge, because the marketing story for semantic search is clean and compelling: “finds meaning, not just keywords.”

The reality is messier. There are query types where BM25 outperforms vector search. There are query types where vector search wins. And there’s a class of queries — larger than either camp would like to admit — where combining both outperforms either alone.

Reciprocal Rank Fusion (RRF) is the algorithm that makes combination tractable. This post explains why hybrid search works, how RRF specifically works, and when to use it.

When vector search fails

Vector search encodes semantic meaning. “automobile” and “car” live close together in embedding space. “quick” and “fast” are neighbours. This is the strength.

The weakness is specificity. Embeddings compress high-dimensional meaning into a fixed-size vector. Rare, precise concepts get averaged out. Consider:

• Query: “error code ECONNRESET in Node.js 18.x”
• Corpus contains one document that exactly discusses ECONNRESET in Node 18.

A vector search might rank this document 15th because the embedding for the query is dominated by “error code” and “Node.js” — terms that appear in hundreds of documents. The specific “ECONNRESET” signal is diluted.

BM25 would find this document first. “ECONNRESET” is a rare token. BM25’s IDF (inverse document frequency) component heavily weights rare tokens. The exact-match document rises to the top.

Other cases where BM25 beats vector:

• Version numbers, model names, error codes, product SKUs
• Proper nouns that aren’t well-represented in the embedding model’s training data
• Short queries where there’s not enough context for embedding to outperform term matching
• Technical documentation with precise terminology

When BM25 fails

BM25 is a bag-of-words model. It has no concept of synonyms, paraphrases, or semantic equivalence.

• Query: “how do I cancel my subscription”
• Document: “steps to terminate your account”

BM25 sees no overlap. BM25 score: 0.

Vector search correctly finds this document. The embeddings for “cancel subscription” and “terminate account” are close in semantic space.

Other cases where vector beats BM25:

• Synonym-heavy queries
• Multilingual retrieval (query in English, documents in Spanish)
• Abstractive questions (“what causes this disease?”)
• Short or ambiguous documents where context matters more than term frequency

Reciprocal Rank Fusion

RRF (Cormack et al., 2009) is a rank combination method. It takes multiple ranked lists and produces a combined ranking.

The formula is simple:

RRF_score(d) = Σ (1 / (k + rank_i(d)))

Where:

• d is a document
• k is a constant (typically 60)
• rank_i(d) is the rank of document d in the i-th ranked list (1-indexed)
• The sum is over all ranked lists

Documents that don’t appear in a ranked list are assigned a rank of infinity (contributing 0 to the sum).

Why RRF instead of score normalization?

The naive approach to combining two search results is to normalize the scores to [0,1] and add them. This fails for a subtle reason: score distributions are incompatible.

BM25 scores are unbounded and depend on corpus statistics. A BM25 score of 8.5 means “highly relevant given this corpus’s term frequencies.” A cosine similarity of 0.85 means “the vectors are close.” These numbers are not on the same scale and normalizing them doesn’t fix the underlying incompatibility.

RRF sidesteps this entirely by working on ranks rather than scores. Rank 1 in the BM25 list and rank 1 in the vector list both get the same weight: 1 / (60 + 1) = 0.0164. A document that ranks 3rd in both lists scores 2 × (1 / (60 + 3)) = 0.0317. The combination is meaningful because ranks are comparable across methods.

The constant k=60 smooths out the difference between rank 1 and rank 2 — it prevents the top-ranked result from dominating too strongly. At k=60:

• Rank 1: 0.0164
• Rank 2: 0.0161
• Rank 10: 0.0143
• Rank 100: 0.0063

The scores decay slowly. A document at rank 100 in one list is still worth something if it’s rank 1 in the other.

Implementation

from manyvector import ManyVector

client = ManyVector(api_key="mv-...")
ns = client.namespace("my-docs")

# Hybrid query: vector + BM25 with RRF fusion
results = ns.query(
    vector=query_embedding,       # dense vector query
    text="ECONNRESET Node.js",    # BM25 term query
    top_k=10,
    fusion="rrf",
    rrf_k=60,                     # RRF constant, default 60
    filters={"source": {"$eq": "docs"}},  # optional metadata filter
)

ManyVector runs both queries against the same namespace, combines the result sets using RRF, and returns the top_k results. The metadata filter is applied before the search — it’s a pre-filter that restricts the candidate set.

Benchmark: when hybrid wins

We tested on a benchmark corpus of 1M documents (mixed technical documentation, FAQ, and news) with 1,000 queries across three categories:

Category 1: Precise technical queries (error codes, version numbers, exact product names)

• BM25: 0.78 NDCG@10
• Vector: 0.72 NDCG@10
• Hybrid (RRF): 0.82 NDCG@10

Category 2: Semantic/paraphrase queries (questions with synonyms, cross-lingual intent)

• BM25: 0.65 NDCG@10
• Vector: 0.83 NDCG@10
• Hybrid (RRF): 0.85 NDCG@10

Category 3: Mixed queries (natural language questions about specific topics)

• BM25: 0.71 NDCG@10
• Vector: 0.78 NDCG@10
• Hybrid (RRF): 0.84 NDCG@10

Key observations:

1. Hybrid never performs worse than the better of the two single-method approaches.
2. Hybrid often performs better than both — the combination is complementary.
3. The gain is largest for mixed queries, which is the category most RAG queries fall into.

Tuning RRF for your workload

The k constant (default: 60). Lower k makes the top-ranked results more dominant. Higher k makes the fusion more democratic. For most RAG workloads, 60 works well. If your vector search is significantly more reliable than BM25 for your corpus, try k=30 (more weight to high-ranking results from the better system).

Result set size. RRF combines from the top-N results of each system. You want N to be larger than your final top_k to ensure good coverage. ManyVector retrieves top-100 from each system before fusing, then truncates to your requested top_k. This covers the case where a relevant document is rank 85 in one list but rank 3 in the other.

Metadata pre-filtering vs. post-filtering. Pre-filtering (filtering before search) reduces the candidate set and can hurt recall if the filter is very selective. Post-filtering (filter after top-k retrieval) maintains recall but may return fewer than top_k results. For hybrid search, pre-filtering is usually correct since the filter is acting as a hard constraint, not a relevance signal.

When to not use hybrid

Hybrid search adds latency. Running two queries instead of one takes more time. For most systems the overhead is small (BM25 is fast), but if you’re targeting sub-5ms latency on a hot path, running only vector search is the right call.

Hybrid also adds complexity to your system. If your corpus is dominated by one query type (e.g., purely semantic questions with no specific terminology), the added complexity of maintaining BM25 indexes isn’t worth it. Profile your actual query distribution before committing.

The heuristic: if your users search for specific named entities, error codes, product names, or exact phrases — use hybrid. If they only ask open-ended semantic questions — pure vector is fine.

Practical recommendation

For RAG pipelines serving general-purpose enterprise knowledge bases: use hybrid by default. The 5-15% NDCG improvement over pure vector is meaningful for user-facing retrieval quality, and the implementation cost is near-zero if your vector database supports it natively.

For narrow-domain semantic search (e.g., searching only the body of text, no specific terminology) with no keyword-match requirements: vector-only is fine and simpler.

For anything where users might search for exact strings, codes, or names: hybrid is non-negotiable.