Agentsop Hybrid Retrieval

Hybrid Retrieval · Dense + Sparse SOP

Third-person operating model for a coder agent that owns retrieval recall on a corpus where both meaning and exact tokens matter. The audience is the LLM agent writing or reviewing retrieval code — not an end user.

One sentence: Dense captures meaning, sparse captures exact tokens; hybrid wins when both matter — but only fuse them when traffic actually carries exact-match queries, and tune the blend per query type or hybrid loses to dense.

1. 何时激活 (Activation Rules)

Activate this skill when any of the following holds:

1. The corpus contains exact-match tokens that a query may reference verbatim: error codes (ERR_SSL_PROTOCOL), SKUs / part numbers (A1-2293-X), API or function names (as_query_engine), proper nouns, legal/medical citations (42 U.S.C. § 1983), version strings, rare jargon, ticket IDs.
2. A bug report says "I searched the exact code/name/string and got nothing", or "the right document exists but dense retrieval ranks it below fuzzy near-misses".
3. PR review surfaces a retriever serving lexical-identity traffic but wired dense-only (index.as_retriever(...) / similarity_search(...) with no sparse leg).
4. You are tuning recall and have already exhausted the cheap dense knobs (prompt, embedding model, chunk size) per the [[llamaindex]] optimization ladder — hybrid is the next rung.
5. The user mentions hybrid search, BM25, sparse retrieval, RRF, QueryFusionRetriever, EnsembleRetriever, or vector-store-native hybrid (Qdrant/Weaviate/Pinecone).

Do not activate when:

• Traffic is purely semantic ("what does X mean?", "summarize the policy") with a lexical-identity share <5% — adding BM25 doubles index footprint for no gain.
• The corpus has no stable identifiers and no query ever quotes an exact string.
• No dense baseline + eval loop exists yet. Hybrid is a Stage-3 optimization ([[llamaindex]] Stage 3 step 4): baseline and measure before fusing.

2. 核心心智模型 (Core Mental Model)

Three principles. Violating any of them is why teams "try hybrid and conclude it didn't help".

Principle 1 — Dense and sparse fail in opposite directions

Dense embedding models destroy lexical identity by pooling token representations: querying a specific error string yields a vector that captures "document about SSL errors" rather than "document containing this exact string". BM25 does the inverse — it scores against an inverted index of exact tokens and is blind to synonyms and paraphrase. (TianPan, Hybrid search in production, 2026; cited in [[llamaindex]] Dilemma 2.)

Operational corollary: the symptom "I pasted the exact code and got nothing" is not a bug in the embedding model — it is the embedding model working as designed. The fix is a second retriever that indexes tokens, not a better embedding.

Principle 2 — Hybrid is traffic-driven, not theoretical

Whether to add sparse is decided by the query-type distribution of real traffic, not by a belief that "more retrievers = better". The decision threshold is the lexical share: the fraction of queries whose correct answer hinges on an exact token. Below ~5% → dense-only. 5–50% → hybrid. Above ~50% (code, logs, legal) → invert to sparse-first with dense as a rerank signal. (Cited in [[llamaindex]] OP-04 / Dilemma 2.)

Principle 3 — One global alpha underperforms; tune per query type

alpha is the dense↔sparse blend (alpha=1 → pure dense, alpha=0 → pure sparse). A semantic query wants high alpha; a lexical-identity query wants low alpha. A single global alpha picked to help lexical queries hurts the semantic slice — which is exactly why a flat alpha "loses to pure dense" and teams wrongly conclude hybrid failed. Tune alpha per query type, or route per type and pick alpha per route. (LlamaIndex alpha-tuning blog; [[llamaindex]] Dilemma 2 结果.)

The fusion method (RRF vs weighted) and the fusion parameter (alpha) are two separate decisions. RRF is score-scale-robust and parameter-light; weighted fusion is tunable but requires score normalization. Choosing neither deliberately is the third silent failure.

3. SOP 工作流 (Agentic Protocol)

Four stages. Each gates the next. This overlay assumes a working dense baseline + eval loop already exists (see [[llamaindex]] Stages 1–2); do not start here.

Stage 0 — Decide if the corpus needs sparse at all

1. Sample ≥50 real queries (or representative synthetic ones if pre-launch).
2. Label each: semantic ("what does X mean?") / lexical ("find error code ABC-123", "the function named foo_bar") / mixed.
3. Compute the lexical share:
   • < 5% → dense-only. Stop; hybrid is over-engineering. Record the decision.
   • 5–50% → add hybrid (Stages 1–3).
   • > 50% (code search, log search, legal citation lookup) → invert: BM25-first, dense as a fallback / rerank signal.
4. Confirm the corpus actually carries the tokens queries reference (an identifier that never appears verbatim in any chunk can't be recovered by BM25 either).

Artifact: a 3-line decision note co-located with the retriever recording the lexical share and the chosen shape.

Stage 1 — Wire BM25 + dense as two retrievers

from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.retrievers.bm25 import BM25Retriever

dense  = index.as_retriever(similarity_top_k=10)            # embedding leg
sparse = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=10)

Hard rules:

• Both legs index the same node set. A BM25 leg built over a different/stale node set silently drops recall (it can only return tokens it has).
• Retrieve wider per leg (top_k 10–20 each) than the final cut — fusion needs candidates to combine. Narrow after fusion (and after any reranker).
• BM25 needs the raw nodes/corpus, not just the vector store — persist or rebuild it alongside the index, or use a vector store with native hybrid (Stage 1-alt).

Stage 1-alt — vector-store-native hybrid. If the vector store does sparse internally (Qdrant sparse_vectors, Weaviate hybrid(alpha=...), Pinecone sparse-dense, pgvector + ts_rank), prefer it: one round-trip, one consistency domain, no separate BM25 index to keep in sync. Use the framework fusion path only when the store has no native hybrid.

Stage 2 — Fuse (RRF or alpha-weighted)

Pick the fusion method deliberately (OP-02):

# Reciprocal Rank Fusion — score-scale-robust, parameter-light. Good default.
fused = QueryFusionRetriever(
    [dense, sparse],
    mode="reciprocal_rerank",   # RRF: combine by rank, ignore raw score scales
    similarity_top_k=8,
    num_queries=1,              # set >1 to also fan out query rewrites
    use_async=True,
)

# Relative-score / weighted fusion — tunable blend; requires score normalization.
fused = QueryFusionRetriever(
    [dense, sparse],
    mode="relative_score",
    retriever_weights=[0.6, 0.4],   # ~ alpha=0.6 toward dense
    similarity_top_k=8,
)

Default to RRF unless you have a labeled set to tune weights on — RRF sidesteps the dense-cosine vs BM25-score scale mismatch that breaks naive weighted sums.

Stage 3 — Tune alpha by query type (the gate)

1. Build a small labeled set per query type (semantic / lexical / mixed) — reuse the Stage-0 sample.
2. Evaluate alpha (or retriever_weights) at {0.0, 0.25, 0.5, 0.75, 1.0} on each type separately, scoring recall / MRR / hit-rate per type.
3. Expect: lexical queries peak near low alpha (sparse-leaning), semantic near high alpha (dense-leaning), mixed in between.
4. If a single global alpha cannot satisfy both types without regressing one → route by query type and apply per-route alpha (hand off classification to [[agentsop-query-routing]] if present), or split into two retrievers selected per query.
5. Gate: hybrid must lift the lexical slice with no regression on the semantic slice vs the dense baseline. If it regresses semantic, the alpha is wrong, not hybrid.

The most common false negative: tuning one global alpha, watching semantic recall drop, and reverting to dense. The correct read is "alpha was global; tune per type".

4. 操作模型 (Operation Models)

Format: Trigger / Action / Output / Evidence.

OP-01 WhenHybridChecklist

• Trigger: Deciding whether to add sparse to a dense pipeline.
• Action: Run the Stage-0 lexical-share checklist. Confirm: (a) corpus has exact-match tokens, (b) traffic references them, (c) lexical share ≥5%, (d) those tokens appear verbatim in chunks. All four must hold.
• Output: A go/no-go with the lexical share recorded; "no" is a valid, common outcome.
• Evidence: [[llamaindex]] Dilemma 2 决策步骤; OP-04 ("trigger is traffic-driven, not theoretical").

OP-02 ChooseFusionMethod

• Trigger: Two legs wired; need to combine their result lists.
• Action: Default RRF (mode="reciprocal_rerank") — rank-based, robust to the dense-cosine vs BM25-score scale gap, no weight to tune. Use weighted / relative-score only when you have a labeled set to fit weights and have normalized scores. Never naive-sum un-normalized scores.
• Output: One deliberate fusion method + the reason it was chosen.
• Evidence: RRF (Cormack et al., 2009); LlamaIndex QueryFusionRetriever modes; LangChain EnsembleRetriever (RRF default).

OP-03 WireBM25Leg

• Trigger: Hybrid approved; sparse leg not yet built.
• Action: Build BM25Retriever.from_defaults(nodes=...) over the same node set as the dense index; persist/rebuild it as a deployment artifact alongside the vector index. Set per-leg similarity_top_k wide (10–20).
• Output: A sparse retriever consistent with the dense index, returning enough candidates to fuse.
• Evidence: LlamaIndex BM25Retriever docs; [[llamaindex]] OP-04 AddHybridBM25.

OP-04 PreferNativeHybrid

• Trigger: The vector store supports sparse internally.
• Action: Use the store's native hybrid (Qdrant sparse vectors / Prefetch + fusion, Weaviate hybrid(query, alpha=...), Pinecone sparse-dense vectors, pgvector full-text + vector) instead of a separate BM25 index. One round-trip, one consistency domain.
• Output: Hybrid with no second index to keep in sync; lower operational surface.
• Evidence: Qdrant hybrid queries docs; Weaviate hybrid search docs; Pinecone sparse-dense docs; [[llamaindex]] OP-04 ("or vendor hybrid (Qdrant/Milvus alpha)").

OP-05 AlphaTunePerType

• Trigger: Fusion wired; recall not yet optimized; or hybrid "loses to dense".
• Action: Evaluate alpha / weights at {0, 0.25, 0.5, 0.75, 1.0} on labeled subsets per query type, not globally. Lexical → low alpha, semantic → high.
• Output: Per-type alpha curve; the alpha that lifts lexical without regressing semantic.
• Evidence: LlamaIndex alpha-tuning in hybrid search blog; [[llamaindex]] Dilemma 2 ("tune per type, not globally — otherwise hybrid underperforms dense").

OP-06 RouteThenAlpha

• Trigger: No single global alpha satisfies both query types without regression.
• Action: Classify the query type first, then dispatch to a retriever configured with the per-type alpha (or pure dense / pure sparse). Hand classification to a router ([[agentsop-query-routing]] if available).
• Output: Each query gets its optimal blend; semantic and lexical slices both peak.
• Evidence: [[llamaindex]] Dilemma 2 结果 (per-type tuning); routing as the mechanism to apply it.

OP-07 InvertForLexicalCorpus

• Trigger: Lexical share >50% (code search, log search, legal/medical citation lookup).
• Action: Make BM25 the primary retriever; use dense as a fallback / rerank signal to catch paraphrase, not as the lead leg. Equivalent to alpha pinned low.
• Output: Recall dominated by exact-token matching where that is what users want; dense recovers the semantic minority.
• Evidence: [[llamaindex]] Dilemma 2 step 4 ("BM25-first, dense as fallback rerank signal"); TianPan production write-up.

OP-08 HybridEvalGate

• Trigger: Before merging any hybrid change.
• Action: Run the eval loop on both slices: assert lexical-slice recall/MRR rises vs dense baseline AND semantic-slice metrics do not regress. Gate merge on both. A lexical lift bought with a semantic regression is not a win.
• Output: Quantitative proof hybrid helped where intended and harmed nothing else.
• Evidence: [[llamaindex]] Stage 2 eval loop ("gate every change"); OP-10 EvalLoop.

5. 困境决策案例 (Dilemma Cases)