RAG

rag — own the retrieve → rerank → ground → cite → refuse pipeline

You own the pipeline that turns a corpus plus a question into a grounded, cited answer: chunk, optionally contextualize, index, retrieve hybrid, rerank, assemble a grounded prompt, cite the sources, and refuse when the context does not contain the answer.

You are judged by retrieval quality and answer faithfulness, not by raw vector math. If you find yourself tuning HNSW parameters, you wandered into the store underneath you (../vector-db/SKILL.md). If you are comparing embedding models or chunk sizes, that is the science beside you (embeddings-search).

The pipeline, and where each stage hands off

Each stage is a real branch — most failures live in one specific stage, and several stages delegate to a sibling skill rather than living here.

Three neighbors are not stages at all. Surfacing this answer inside a chat product (sessions, channels, UI) is ../chatbot/SKILL.md — it calls you, not the reverse. A multi-step tool loop with state, where retrieval is one tool among many, is ../building-agents/SKILL.md. Pulling schema-constrained fields out of text instead of a grounded prose answer is structured-extraction. rag is the retrieval brain those products call.

Retrieval is the bottleneck — measure recall before touching the prompt

Naive RAG pipelines fail at the retrieval step in up to ~40% of cases even when the correct document is in the corpus (StackAI / Lushbinary, 2026-06-02). Why this matters: if the right passage never reaches the model, no prompt wording can save the answer. So your first move on a broken pipeline is never the prompt — it is measuring whether retrieval delivered the goods.

Bad   → "answers are wrong, let me lower temperature and reword the system prompt."
Good  → measure context recall on a golden set; if the right chunk isn't in the top-K,
         fix chunking + hybrid + rerank first. Only then touch grounding.

Order of attack when answers are wrong: context recall → context precision → grounding prompt → generation params. The last item almost never moves the needle.

Chunking — the highest-leverage single fix

Chunk on structure, not on a blind character count. Why: too-small loses the context a passage needs to be interpretable; too-large dilutes the embedding so the relevant sentence gets averaged away (StackAI; EdenAI 2025, accessed 2026-06-02).

• Split on headings/sections first, then sub-split long sections to a target window.
• Keep overlap (~10–20% of the window) so a fact spanning a boundary survives in one chunk.
• Attach a stable chunk_id and source at creation — you will need them end to end for citations (see below). Never embed text and discard where it came from.

# Heading-aware split sketch; real size tuning belongs in embeddings-search.
def chunk_markdown(doc_id, text, target=800, overlap=120):
    sections, buf, head = [], [], None
    for line in text.splitlines():
        if line.startswith("#"):
            if buf: sections.append((head, "\n".join(buf))); buf = []
            head = line.lstrip("# ").strip()
        else:
            buf.append(line)
    if buf: sections.append((head, "\n".join(buf)))
    out, i = [], 0
    for head, body in sections:
        for start in range(0, max(1, len(body)), target - overlap):
            piece = body[start:start + target]
            out.append({"chunk_id": f"{doc_id}#{i}", "source": doc_id,
                        "heading": head, "text": piece})
            i += 1
    return out

The "what window/overlap maximizes recall for this corpus" study is embeddings-search; here you just need structurally sane chunks that keep their ids.

Contextual retrieval — prepend context before you embed

Anthropic's Contextual Retrieval (Sept 2024) prepends a short LLM-generated blurb to each chunk before embedding and before BM25 indexing, so an isolated chunk knows what document and section it belongs to. Why it matters: it cuts failed retrievals by ~35% (contextual embeddings alone), ~49% (contextual embeddings + contextual BM25), and ~67% once reranking is added (anthropic.com/news/contextual-retrieval, 2024-09; accessed 2026-06-02).

<document>{{WHOLE_DOC}}</document>
Here is the chunk we want to situate within the whole document:
<chunk>{{CHUNK}}</chunk>
Give a short, standalone context (1–2 sentences) that situates this chunk within the
document for search retrieval. Answer only with the context, nothing else.

Embed context + "\n" + chunk_text (not the bare chunk). The full implementation — caching the document prompt, batching, the BM25 side, and a runnable retrieve → rerank → answer skeleton — lives in references/pipeline.md.

Hybrid retrieval + RRF + rerank

Dense vectors miss exact terms (codes, names, error strings); BM25 keyword search catches them but misses paraphrase. Combine both, fuse with Reciprocal Rank Fusion (RRF), then rerank with a cross-encoder. The default-best quality/cost funnel (Microsoft Cloud Blog 2025-02-04; StackAI, accessed 2026-06-02):

retrieve ~150 candidates (dense + BM25, fused with RRF)
   → rerank all 150 with a cross-encoder
   → keep top ~20 for the prompt

The actual hybrid query (named vectors, sparse-dense, server-side fusion) is vector-db. Here you own the funnel and the reranker choice:

• Cohere Rerank 3.5 — managed cross-encoder, context length 4096, SOTA on BEIR and multilingual, available via Cohere API, Bedrock, Pinecone, Azure (docs.cohere.com/changelog/ rerank-v3.5, accessed 2026-06-02). Use when you want quality without hosting a model.
• A local cross-encoder (e.g. a bge-reranker) — use when data cannot leave your network or you need zero per-call cost; you pay in GPU/latency instead.

# Rerank the fused candidates down to the prompt set; keep ids intact.
import cohere
co = cohere.ClientV2()
ranked = co.rerank(model="rerank-v3.5", query=q,
                   documents=[c["text"] for c in candidates], top_n=20)
top = [candidates[r.index] for r in ranked.results]  # each still carries chunk_id + source