RAG Accuracy Optimizer

RAG Accuracy Optimizer

A skill for optimizing end-to-end accuracy in RAG systems.

Workflow Overview

Data Design → Chunking → Indexing → Retrieval → Generation → Testing → Monitoring

Each step impacts accuracy. Optimize each step in order.

1. Structured Data Design

SQL vs Vector DB — When to Use What?

Principle: Clearly structured data → SQL. Unstructured data requiring semantic understanding → Vector DB. Most production systems need both.

Schema Design Patterns by Domain

Insurance:

policies(policy_id, product_type, effective_date)
clauses(clause_id, policy_id, clause_number, title, content)
exclusions(exclusion_id, clause_id, description)
-- Vector: embedding for clause.content + exclusion.description

Finance:

securities(ticker, name, sector, exchange)
reports(report_id, ticker, period, report_type)
sections(section_id, report_id, heading, content)
-- Vector: embedding for section.content, metadata: ticker + period

Healthcare:

drugs(drug_id, generic_name, brand_name, category)
guidelines(guideline_id, condition, recommendation, evidence_level)
interactions(drug_a_id, drug_b_id, severity, description)
-- Vector: embedding for guidelines.recommendation

E-commerce:

products(product_id, name, category, brand, price)
reviews(review_id, product_id, rating, content)
specs(product_id, attribute, value)
-- Vector: embedding for review.content + product description

Metadata Tagging Strategy

Each chunk/document needs at minimum:

metadata = {
    "source": "policy_doc_v2.pdf",       # Origin
    "source_type": "pdf",                 # File type
    "domain": "insurance",                # Domain
    "category": "life_insurance",          # Classification
    "entity_id": "POL-2024-001",          # Related entity ID
    "section": "exclusions",              # Section in doc
    "chunk_index": 3,                      # Chunk position
    "total_chunks": 12,                    # Total chunks in doc
    "created_at": "2024-01-15",           # Creation date
    "version": "2.0",                      # Version
    "language": "en"                       # Language
}

Metadata principles:

• Always include source for traceability and citation
• entity_id enables pre-filtering before search → reduces noise
• chunk_index + total_chunks enables fetching surrounding context
• Domain-specific fields (clause_number, ticker, drug_id) vary by use case

Normalization vs Denormalization

Recommendation: Normalized for SQL source → Denormalized when creating chunks for Vector DB. Each chunk should contain sufficient context, no JOINs needed at retrieval time.

2. Chunking Strategies

Detailed code examples: read references/chunking-patterns.md

Choosing the Right Strategy

Data has clear structure (clauses, sections)?
  → Semantic chunking (by heading/section)

Long, continuous data (articles, transcripts)?
  → Fixed size + overlap (512 tokens, 10-20% overlap)

Need both overview + detail?
  → Hierarchical chunking (parent-child)

Domain-specific with its own logical units?
  → Domain-specific chunking

Chunk Size Guidelines

Semantic Chunking

Split by meaning (section, topic) instead of fixed size:

# Split by markdown headings
# Split by paragraph breaks (\n\n)
# Split by topic change (using NLP or LLM detection)

Overlap Strategy

• 10-20% overlap between adjacent chunks
• Ensures information at boundaries is not lost
• Chunk N ends with 1-2 opening sentences of chunk N+1

Hierarchical Chunking (Parent-Child)

Document (summary)
  └── Section (heading + key points)
        └── Paragraph (details)

• Search at paragraph level (most detailed)
• When matched, pull parent section for additional context
• Keep parent_id in metadata

Domain-Specific Chunking

• Insurance: 1 chunk = 1 clause
• Finance: 1 chunk = 1 report section, metadata = ticker + period
• Healthcare: 1 chunk = 1 guideline/recommendation
• E-commerce: 1 chunk = 1 review or 1 product description
• Legal: 1 chunk = 1 article/clause/section

Metadata Enrichment Per Chunk

Each chunk should be enriched with:

• Summary: 1-2 sentence content summary (LLM-generated)
• Keywords: Key terms (supports BM25)
• Questions: 2-3 questions this chunk can answer (hypothetical questions)
• Entities: Named entities (product names, codes, dates)

3. Retrieval Optimization

Detailed code examples: read references/retrieval-patterns.md

Recommended Retrieval Pipeline

User Query
  → Query Rewriting (expand/reformulate)
  → Multi-Query Generation (3-5 variants)
  → Metadata Filtering (narrow scope)
  → Hybrid Search (Vector + BM25)
  → Merge & Deduplicate
  → Reranking (top 20 → top 5)
  → Contextual Compression
  → LLM Generation (with citations)

Hybrid Search (Vector + BM25)

• Vector search: Find by meaning (semantic similarity)
• BM25 (keyword): Find by exact keywords (product names, codes)
• Combined: Weighted fusion or Reciprocal Rank Fusion (RRF)

final_score = α × vector_score + (1-α) × bm25_score
# α = 0.7 is a good starting point, tune per domain

Query Rewriting

Use LLM to reformulate the user question for clarity:

User: "does insurance pay?"
→ Rewritten: "Under what circumstances does life insurance pay out benefits?"

Multi-Query

From 1 question, generate 3-5 variants → search each variant → merge results:

Original: "Which bank has the highest savings rate?"
Query 1: "Compare savings interest rates across banks 2024"
Query 2: "Bank with highest deposit rate currently"
Query 3: "Top banks with best deposit interest rates"

Reranking

After retrieval, use a reranking model to re-sort by relevance:

• Cohere Rerank: Simple API, highly effective
• Cross-encoder: More accurate than bi-encoder, but slower
• GPT Rerank: Use LLM to evaluate relevance (expensive but flexible)

Retrieve top 20 → rerank → take top 3-5 for generation.

Contextual Compression

After reranking, compress each chunk: keep only the part relevant to the question.

Original chunk (500 tokens) → Compressed (150 tokens, relevant part only)

Reduces noise, saves context window, improves accuracy.

Metadata Filtering

Narrow the search space BEFORE vector search:

# Instead of searching all 1M chunks:
filter = {"domain": "insurance", "product_type": "life"}
# Only search within ~50K relevant chunks
results = vector_db.search(query, filter=filter, top_k=20)

4. Accuracy Testing & Monitoring

Test Suite Design

Create ground truth Q&A pairs:

{
    "test_cases": [
        {
            "question": "Does life insurance pay out for suicide?",
            "expected_answer": "No payout within the first 2 years",
            "expected_source": "clause_15_exclusions.pdf",
            "category": "exclusions",
            "difficulty": "medium"
        }
    ]
}

Recommendation: Minimum 50-100 test cases, evenly distributed across categories and difficulty levels.

Metrics

A/B Testing

Compare strategies by running the same test suite:

Config A: chunk_size=256, overlap=10%, no_rerank
Config B: chunk_size=512, overlap=20%, cohere_rerank
→ Compare MRR, NDCG, Answer Accuracy
→ Choose the config with better metrics

Error Analysis Framework

Classify errors to know where to optimize:

Production Monitoring

Log each query:

log_entry = {
    "timestamp": "2024-01-15T10:30:00",
    "query": "...",
    "retrieved_chunks": [...],
    "reranked_chunks": [...],
    "answer": "...",
    "confidence": 0.85,
    "latency_ms": 450,
    "user_feedback": None  # thumbs up/down
}

Alerts:

• Continuous confidence < 0.5 → review chunking/retrieval
• Latency > 2s → optimize index or reduce top_k
• Negative feedback > 20% → audit error patterns

5. Safeguards

Hallucination Prevention

Mandatory system prompt:

Answer ONLY based on the information provided in the context.
If you cannot find the information, respond: "I could not find this
information in the available data."
NEVER fabricate information.

Citation Enforcement

Require source citations:

Every answer must include [Source: file_name, section/clause].
If a specific source cannot be cited, mark it as "unverified".

Confidence Thresholds

if max_relevance_score < 0.3:
    return "No relevant information found."
elif max_relevance_score < 0.6:
    return answer + "\n⚠️ Low confidence. Please verify."
else:
    return answer + f"\n📎 Source: {sources}"

Answer Verification

Cross-check the answer with the DB:

1. Extract claims from the answer (using LLM)
2. Verify each claim against retrieved chunks
3. Flag claims without supporting evidence
4. Return only verified claims

6. Embedding Model Selection

Detailed comparison: read references/embedding-models.md

Quick Decision

Key Principles

• Dimension reduction: OpenAI embed-3 supports Matryoshka — reduce 3072→512 with only ~3% quality loss
• Normalize embeddings: Always normalize_embeddings=True when encoding for cosine similarity
• Batch processing: Encode in batches (256-2000 items) instead of one at a time
• Consistency: Use the SAME model for indexing and querying

7. Vector DB Comparison

Detailed comparison + HNSW tuning: read references/vector-db-comparison.md

Quick Decision

Already have PostgreSQL and <5M vectors? → pgvector
Just prototype/POC? → ChromaDB
Production, want zero-ops? → Pinecone
Need performance + HNSW control? → Qdrant
Need hybrid BM25+vector built-in? → Weaviate

HNSW Tuning Quick Reference

Trade-off: Higher M and ef → better recall but more RAM and slower. Tune per SLA.

8. Advanced Techniques

Detailed code examples: read references/advanced-rag.md