Context Optimization

Context Optimization

Context optimization is the process of refining the raw context assembled for an AI model so that every token contributes meaningfully to the task. In a typical RAG or agent pipeline, the retrieved context often contains redundant passages, marginally relevant chunks, and poorly ordered information. Optimization transforms this raw material into a lean, high-signal context block that improves answer quality, reduces inference cost, and makes the most of the model's attention budget.

Workflow

1. Audit the Raw Context: Inventory every piece of context that has been gathered -- retrieved documents, conversation history, tool outputs, and metadata. Measure the total token count and compare it against the available context budget. Identify the compression ratio needed if the raw context exceeds the budget.

2. Deduplicate Overlapping Content: Scan the context for near-duplicate passages that convey the same information. This is common in RAG pipelines where chunking with overlap produces multiple chunks covering the same paragraph, or when multiple source documents repeat the same facts. Use semantic similarity (cosine distance > 0.92) or exact n-gram overlap detection to identify duplicates, then keep only the most complete version of each piece of information.

3. Score Relevance and Information Density: Assign each context chunk two scores: a relevance score (how closely it relates to the current query) and an information density score (how many useful facts it conveys per token). Relevance can be measured via the retrieval score or a lightweight cross-encoder pass. Density can be estimated by counting named entities, code identifiers, numerical data, and key terms relative to chunk length. Multiply the two scores to produce a composite utility score.

4. Filter Low-Value Content: Remove chunks whose composite utility score falls below a threshold. A good starting point is to keep the top 60-70% of chunks by utility score. Also remove boilerplate text (copyright notices, navigation menus, repeated headers) that contributes zero information. Be conservative -- it is better to include a marginally relevant chunk than to lose a critical fact.

5. Reorder by Priority: Arrange the remaining chunks to maximize the model's attention. Place the highest-utility chunks first (models attend most to the beginning of the context) and the second-highest near the end (models also attend to recency). Avoid burying critical information in the middle of a long context block -- this is the "lost in the middle" zone where model attention is weakest.

6. Validate Coverage: After filtering and reordering, verify that the optimized context still covers all aspects of the query. If the query has multiple sub-questions, ensure at least one chunk addresses each. If coverage gaps appear, selectively re-add previously filtered chunks that fill the gap, even if their utility score was below the threshold.

Techniques

• Deduplication: Identifies and removes redundant passages using semantic similarity thresholds or n-gram overlap detection. Critical in RAG pipelines where overlapping chunks often repeat the same sentences. Keeps the most complete or highest-scoring version.
• Relevance Filtering: Removes chunks that fall below a relevance threshold. Uses the original retrieval score, a reranker score, or keyword overlap with the query as the signal. Aggressiveness should be tuned -- filtering too hard causes coverage gaps.
• Information Density Scoring: Estimates how much useful information a chunk contains per token. Dense chunks (packed with facts, code, or data) are preferred over verbose, low-density prose. Useful for deciding which chunks to keep when two have similar relevance scores.
• Priority-Based Ordering: Arranges chunks so the model sees the most important information first and last, avoiding the "lost in the middle" effect. This is especially impactful for long context windows (32K+ tokens) where attention degradation is more pronounced.
• Context Window Strategies: Different model context windows require different optimization approaches. For small windows (4K tokens), aggressive filtering and compression are essential. For medium windows (32K), focus on deduplication and ordering. For large windows (128K+), ordering and density scoring matter most, since there is room for more material but the lost-in-the-middle effect is amplified.

Usage

Provide the raw context (a list of text chunks with optional metadata and scores), the user query, and the target token budget. The skill returns an optimized context block -- deduplicated, filtered, scored, and reordered -- ready for prompt assembly. Optionally provide a coverage checklist (key topics the context must address) to prevent important information from being filtered out.

Examples

Example 1: Optimizing Context for a Multi-File Code Edit Task

Task: "Refactor the authentication module to use async/await instead of callbacks."

Raw Context (7 chunks, ~4,200 tokens):

Optimization Steps:

1. Deduplicate: Chunks 1 and 2 overlap on lines 20-45. Merge into a single chunk covering lines 1-55 of login.js.
2. Filter: Remove chunk 4 (README setup instructions, relevance 0.45) and chunk 6 (package.json, relevance 0.35) -- below the 0.50 threshold.
3. Reorder: Place merged login.js chunk first (highest relevance), then the token generation chunk, then middleware.js, then register.js.

Optimized Context (~2,800 tokens, 33% reduction):

1. src/auth/login.js:1-55 (merged) -- Login function with callback-based DB lookup and token refresh
2. src/auth/login.js:40-70 -- Token generation and session creation
3. src/auth/middleware.js:1-30 -- Auth middleware with callback token verification
4. src/auth/register.js:1-40 -- Registration function using callbacks