Context Ranking

Context Ranking

Context ranking is the process of ordering retrieved text chunks so the most relevant, diverse, and useful information rises to the top. In any retrieval pipeline, the initial search returns a broad set of candidates -- many of which are only tangentially related to the query. Ranking transforms this unordered candidate set into a prioritized list, enabling downstream steps (context assembly, prompt construction) to select the best material and discard the rest. Effective ranking is the difference between a grounded, precise answer and a vague, off-topic one.

Workflow

1. Collect Candidate Chunks: Gather the initial set of retrieved chunks from the search layer. This is typically the top-k results (k = 15-30) from a vector search, keyword search, or hybrid search. Each chunk arrives with a preliminary score (e.g., cosine similarity or BM25 score) and source metadata.

2. Apply First-Stage Scoring: Score each candidate with a fast, lightweight algorithm. BM25 is the standard choice for keyword relevance; cosine similarity between the query embedding and chunk embedding is the standard for semantic relevance. In hybrid pipelines, compute both scores and combine them using Reciprocal Rank Fusion (RRF) or a weighted linear combination. This stage is meant to be fast and run over all candidates.

3. Rerank with a Cross-Encoder: Pass the top candidates (typically 15-25) from the first stage through a cross-encoder reranker. Unlike bi-encoder embeddings that score query and document independently, a cross-encoder processes the query and chunk together with full attention, producing much more accurate relevance scores. Models like Cohere Rerank, bge-reranker-v2-m3, or ColBERTv2 are commonly used. This step is slower but dramatically improves precision.

4. Apply Diversity Selection: After reranking, the top results may cluster around a single subtopic, leaving other aspects of the query uncovered. Apply Maximal Marginal Relevance (MMR) or a similar diversity algorithm to penalize chunks that are too similar to already-selected chunks. This ensures the final ranked list covers the breadth of the query, not just its most obvious interpretation.

5. Assign Final Scores and Rank: Combine the reranker relevance score with the diversity penalty and any domain-specific boosting signals (e.g., recency boost, source authority weight) into a final composite score. Sort chunks by this composite score in descending order. The top-n chunks (n = 3-7) form the final ranked context to be injected into the prompt.

6. Attach Metadata and Confidence: Annotate each ranked chunk with its final score, source path, and a confidence tier (high / medium / low). This metadata helps the downstream prompt assembly step decide how to present the context and allows the model to calibrate its confidence when citing sources.

Key Concepts

• BM25: A probabilistic keyword-matching algorithm based on term frequency, inverse document frequency, and document length normalization. Excels at matching exact terms and rare keywords. Fast and interpretable, but blind to synonyms and paraphrases. The standard first-stage ranker for keyword search.
• Cosine Similarity: Measures the angle between two embedding vectors. Values range from -1 to 1, with higher values indicating greater semantic similarity. The standard first-stage ranker for semantic search. Quality depends heavily on the embedding model used.
• Cross-Encoder Reranking: A transformer model that takes the concatenation of query and document as input and outputs a relevance score. Because it applies full cross-attention between query and document tokens, it captures fine-grained relevance that bi-encoders miss. Typically 5-20x slower than cosine similarity but produces significantly better ranking.
• Maximal Marginal Relevance (MMR): An algorithm that iteratively selects chunks by balancing relevance to the query against redundancy with already-selected chunks. Controlled by a lambda parameter: lambda = 1.0 selects purely by relevance, lambda = 0.0 selects purely by diversity, and values around 0.5-0.7 balance both. Essential for multi-faceted queries.
• Reciprocal Rank Fusion (RRF): A score-combining method used in hybrid search. For each chunk, compute 1/(k + rank) for each ranking source, then sum. This produces a fused ranking that is robust to score scale differences between BM25 and cosine similarity.

Usage

Provide a query and a list of candidate text chunks (with optional preliminary scores and metadata). The skill scores, reranks, and diversifies the chunks, returning a ranked list with final scores and confidence tiers. Specify the desired number of output chunks (top-n) and an optional diversity parameter (MMR lambda).

Examples

Example 1: Ranking Code Search Results for a Debugging Query

Query: "Why does the WebSocket connection drop after 60 seconds of inactivity?"

Candidate Chunks (from hybrid search, top-8):

After Cross-Encoder Reranking:

After MMR Diversity Selection (top-5, lambda=0.6):

1. src/ws/server.ts:40-65 (score 0.96) -- Server-side 60s timeout config
2. nginx.conf:22-28 (score 0.93) -- Nginx proxy 60s read timeout (different source of the problem)
3. src/ws/client.ts:10-35 (score 0.89) -- Client missing pong handler (client-side root cause)
4. src/ws/server.ts:80-95 (score 0.85) -- Cleanup handler confirms the timeout behavior
5. docs/websocket.md:15-30 (score 0.78) -- Documentation confirming 60s default

The ranked list covers three distinct causes (server config, nginx proxy, client pong handler) plus confirmation from the cleanup handler and docs.