Back to Blog
Abstract layered patterns representing lexical and vector retrieval fused in a hybrid search pipeline

RAG Hybrid Search: Reranking for Reliable Retrieval

9 min read

RAG hybrid search fixes a common retrieval failure: vector search finds passages that sound relevant but misses the exact policy number, error code, product name, or clause the user typed. Keyword search has the reverse problem. It catches literal terms but misses a useful paraphrase. Running both searches helps, but only if the pipeline combines their rankings correctly, limits reranking cost, preserves access filters, and proves that the added machinery improves retrieval.

This guide builds that pipeline from diagnosis through verification. You can implement the retrieval contract on any search stack that supports lexical and vector queries.

Diagnose retrieval before changing it

Do not add hybrid search because it appears on an architecture diagram. Start with failed queries and inspect the retrieved evidence before looking at the final model answer. A weak answer can come from retrieval, prompt construction, generation, or missing source data. Hybrid retrieval only addresses the first of those.

Label each failed query with the smallest useful category:

  • Exact identifier queries contain ticket IDs, product codes, names, dates, quoted phrases, or error strings.
  • Paraphrase queries use different words from the source but ask for the same concept.
  • Mixed-constraint queries combine a concept with an exact entity, date, location, or document type.
  • Ambiguous queries could match several documents and lack enough context to choose one.
  • Missing-corpus failures occur when the correct passage is not indexed or was parsed incorrectly.
  • Authorization exclusions occur when the passage exists but the caller cannot retrieve it.

The first three categories are the main case for hybrid search. The last three need a different fix. No fusion algorithm can recover a document that is absent, prohibited, or impossible to identify from the query.

Microsoft's hybrid search overview explains the underlying complement: full-text search is strong at exact terms while vector search finds conceptual similarity. Its implementation runs the two searches in parallel and merges their result lists. That is the basic shape to copy, even when your search engine is different.

Create a small failure set before implementation. Fifty representative queries are more useful than thousands of unlabeled production prompts. Include ordinary questions, identifiers, paraphrases, mixed constraints, and a few queries that should return nothing. For each query, record the expected source document and one or more relevant passages. This set becomes the retrieval fixture used throughout the rollout.

Build two independent candidate generators

Treat lexical and vector retrieval as independent candidate generators. Each should return stable document or chunk identifiers, a rank, its native score, and the metadata needed for authorization and debugging.

A practical result envelope looks like this: 

Candidate {
  chunk_id
  document_id
  source_version
  retrieval_channel
  native_rank
  native_score
  authorization_scope
}

The lexical branch should use the search engine's normal text ranking, often BM25, with field boosts that reflect the corpus. Titles, identifiers, and section headings may deserve more weight than body text. Do not hide aggressive fuzzy matching inside this branch at first. Exact-term behavior is the reason the branch exists, and broad fuzziness can flood the candidate set with near matches.

The vector branch should use the same text representation and embedding version used at indexing time. Apply the same document filters to both branches before results enter model context. A lexical result that bypasses a tenant, department, date, or document-state filter is not a useful recall improvement. It is an authorization or consistency defect.

Start with equal candidate depths, such as 30 results from each branch. Treat that as an inspectable starting point, not a universal optimum. Log branch latency, candidate count, filter count, and whether each expected passage appeared. The trace will show whether a later reranker made a bad decision or never received the right evidence.

Weaviate's hybrid search documentation provides a concrete implementation reference for combining BM25 and vector search and exposing component scores. Use that observability idea even if you do not use its blending method. Engineers need to see which branch contributed each result.

Fuse ranks instead of raw scores

Raw lexical and vector scores are not probabilities, and their scales are not comparable. A BM25 score of 12 is not inherently stronger than a cosine similarity of 0.82. The ranges can also shift with query length, corpus changes, embedding models, and search-engine settings. Normalizing scores can work, but it creates calibration work that many teams do not notice until production results drift.

Reciprocal Rank Fusion, or RRF, avoids that comparison. It assigns each candidate a contribution based on its position in each ranked list: 

rrf_score(document) = sum(1 / (rank_constant + rank_in_list))

A document that ranks well in both lists receives contributions from both. A document that appears in only one list can still survive. The rank constant controls how sharply the contribution falls as rank increases. Keep it fixed during the first evaluation so changes in candidate depth and reranking are easy to attribute.

Elastic's RRF reference documents this rank-based approach over independent retrievers. The design matters more than its API syntax: fuse positions rather than unrelated native scores.

Deduplicate by the identity used in the answer pipeline. If two chunks from the same document occupy most of the fused top results, decide whether that is useful evidence or crowding. One option is to keep chunks independent during fusion, then apply a per-document cap before reranking. Another is to fuse at document level and restore the best passages afterward. Pick one rule and test it against multi-section documents rather than adding diversity logic blindly.

Preserve a trace for every fused item:

  • lexical rank, if present
  • vector rank, if present
  • each RRF contribution
  • final fused rank
  • filters applied to each branch
  • source document and version

This trace turns a complaint such as "search got worse" into a specific finding. You can see whether the correct passage missed both branches, entered too low, lost during fusion, or lost during reranking.

Rerank a bounded candidate set

Fusion improves candidate ordering using ranks alone. A reranker performs a more expensive query-to-passage comparison over the fused shortlist. It can move a passage with the right answer above passages that merely share vocabulary or topic.

Use a bounded retrieve-then-rerank sequence:

  1. Retrieve from lexical and vector branches in parallel.
  2. Fuse and deduplicate the lists.
  3. Select the first N fused candidates.
  4. Send only those candidates to the reranker.
  5. Return the top K reranked passages to the model.

Pinecone's reranking guide describes this two-stage pattern and the tradeoff: increasing the rerank depth can improve relevance, but it also adds latency and token or model cost. That makes N an operating parameter, not a number to copy from a tutorial.

Start with a rerank depth between 20 and 50 and an answer context of perhaps 5 to 10 passages. Measure with your fixture set. If the expected evidence often appears below the rerank cutoff, increase branch depth or change fusion before spending more on the reranker. If the evidence enters the shortlist but finishes low, inspect reranker suitability, passage length, query formulation, and duplicated context.

Set an explicit rerank deadline. If it expires, return the fused ranking rather than failing the whole request. Record the fallback in the response trace. This keeps the retrieval path useful during a reranker outage and lets operations compare fallback quality with normal quality.

Do not let reranking change authorization. The reranker should only receive candidates already permitted for the caller, and its output must be a permutation or subset of those candidate identities. Reject unknown identifiers in its response. A model-based ranking service is not an access-control boundary.

Test the stages separately

A single answer score can hide the reason a change worked or failed. Compare four retrieval configurations against the same fixtures:

  1. lexical only
  2. vector only
  3. lexical plus vector with fusion
  4. fused candidates plus reranking

Measure recall at the candidate boundary first. For each query, ask whether at least one expected passage appears in the top 10, top 20, and rerank set. Then measure ranking quality with a metric such as reciprocal rank or normalized discounted cumulative gain. Finally, run answer-level evaluation using only the context each configuration produced.

Anthropic's contextual retrieval report is useful here because it evaluates retrieval changes rather than describing them only as architecture. Its implementation combines contextual embeddings, contextual BM25, and reranking. The reported results belong to Anthropic's tested corpora and configuration, not to yours, but the comparison structure is worth adopting.

Break results down by the query labels created during diagnosis. A blended average can conceal a regression in exact identifiers while showing an overall gain on paraphrases. The release gate should require no material regression for critical query classes, not just a higher global mean.

Also test negative behavior:

  • A forbidden document never appears in either branch, the fused list, reranker input, logs, or model context.
  • A deleted or superseded document does not reappear from a stale index.
  • A query with no supporting evidence produces an empty or low-confidence retrieval outcome.
  • A reranker timeout returns the fused fallback within the request deadline.
  • Duplicate chunks do not crowd all other evidence out of the final context.
  • Every returned passage maps to a current source version and a visible citation.

Tune one layer at a time

Hybrid pipelines have many knobs: lexical boosts, vector model, branch depth, RRF constant, duplicate policy, rerank depth, passage length, final context size, and deadlines. Changing several together creates a result you cannot explain or safely roll back.

Tune in this order:

  1. Fix indexing, parsing, and authorization defects.
  2. Establish lexical-only and vector-only baselines.
  3. Add fixed-parameter fusion and measure candidate recall.
  4. Adjust branch depth only if expected evidence misses the fused shortlist.
  5. Add reranking and measure ordering plus latency.
  6. Adjust final context size using answer quality and citation coverage.
  7. Canary the new path and keep the old retriever as a rollback option.

Later stages cannot restore candidates discarded earlier. A larger reranker is wasted if candidate generation misses the answer. More model context is wasted if the top results repeat the wrong document.

Production telemetry should include query class when it can be inferred safely, branch and reranker latency, fallback use, retrieved document IDs, source versions, ranking contributions, and user feedback tied to a trace. Avoid logging raw confidential queries or passages by default. Store enough structured evidence to reproduce the ranking without turning the search log into another sensitive corpus.

Decide whether the added complexity earned its place

Keep RAG hybrid search when the fixture set shows a meaningful recall gain for exact and mixed queries, reranking improves top-result ordering, and the request remains inside its latency and cost budgets. Keep vector-only or lexical-only retrieval for a route where the second branch adds no measurable value. Different corpus and query classes can justify different retrieval policies.

Your next action is concrete: take 50 failed or representative production queries, label them by retrieval failure, and run the four-way comparison. Do not deploy fusion or reranking until you can point to the queries it repairs, the cases it leaves unchanged, the latency it adds, and the fallback that runs when the reranker is unavailable.

References

About Fire In Belly: Independent senior engineering from Tallinn, Estonia. We design and build AI workflow automation with the lexical-and-vector fusion, bounded reranking, and stage-by-stage retrieval testing described above, at published fixed prices. Schedule a call to discuss your next project.