RAG Engineer

RAG Engineer

Overview

Use this skill when the user needs a Retrieval-Augmented Generation workflow that is measurable, debuggable, and grounded in evidence.

This skill is for designing or improving:

• corpus preparation and ingestion
• chunking and metadata strategy
• embedding and indexing choices
• semantic, keyword, or hybrid retrieval
• reranking and context assembly
• citation and provenance behavior
• retrieval evaluation and troubleshooting

The operating principle is simple: fix retrieval before tuning generation. If the right evidence is not found, ranked, filtered, and assembled correctly, prompt changes will mostly mask the problem.

Use the companion files when needed:

• references/domain-notes.md for chunking decisions, hybrid retrieval rules, metrics, and failure lookup
• examples/worked-example.md for a concrete end-to-end RAG tuning example

When to Use This Skill

Use this skill when:

• the user is building or repairing a knowledge-grounded assistant, search-backed chat system, or internal question-answering workflow
• the user needs help with embeddings, vector search, chunking, indexing, hybrid retrieval, reranking, or citations
• the user needs a retrieval eval plan instead of prompt-only iteration
• the corpus contains documents where provenance, freshness, filtering, or permissions matter
• the system must explain which retrieved evidence supports an answer

Do not use this skill by itself when:

• the task is mainly model fine-tuning without a retrieval component
• the task is generic web search UX rather than document-grounded retrieval engineering
• document permissions, tenant boundaries, or provenance cannot be enforced
• the user only wants a one-off prompt and there is no retrieval pipeline to design or debug

Operating Table

Workflow

1. Define the retrieval job before selecting tools

Document:

• what corpus or corpora are in scope
• who is allowed to retrieve which data
• what query classes matter most
• what a good retrieval result looks like
• what downstream answer behavior is required

At minimum, identify query classes such as:

• factual lookup
• semantic paraphrase
• identifier lookup
• policy or compliance lookup
• recent or freshness-sensitive lookup
• multi-hop or comparison queries

Do not start with embedding model or vector database debates. Start with expected retrieval behavior.

2. Build a small retrieval eval set first

Before tuning chunk size, prompts, or ranking:

• collect representative queries
• record the expected supporting document or passage for each
• separate retrieval success from answer quality
• keep the set small but realistic so it can be rerun often

Useful eval artifacts:

• query text
• query class
• expected document IDs or passage IDs
• any required metadata filters
• notes on ambiguity or acceptable alternatives

If the team cannot agree on expected evidence for a query, the requirement is probably underspecified.

3. Define the ingestion contract

Specify how documents become retrievable records:

• normalization rules
• deduplication rules
• document identifiers
• section extraction rules
• freshness fields
• ACL or tenant metadata
• source URL, file path, title, timestamp, version, and provenance fields

Good ingestion contracts make debugging possible later. Every chunk should be traceable back to a source document and section.

4. Choose chunking based on content structure and queries

Chunk by document-aware boundaries where possible, not by arbitrary length alone.

Preserve metadata that supports retrieval and citations:

• document ID
• section heading or path
• timestamp or effective date
• source type
• ACL or tenant tags
• version or freshness markers

Use references/domain-notes.md for a content-type chunking matrix and common failure patterns.

Avoid assuming one universal chunk size, overlap, or top-k value. Treat these as testable starting points, not truths.

5. Choose retrieval strategy for the corpus

Select retrieval behavior that matches the corpus:

• semantic retrieval for concept-heavy natural language content
• keyword or lexical retrieval for identifiers, exact phrases, version strings, and error codes
• metadata filtering for access control, tenant isolation, time ranges, product families, or content type
• hybrid retrieval when both semantic similarity and exact matching matter
• reranking when first-pass retrieval has adequate recall but poor ordering

A practical default is to test:

1. semantic-only
2. keyword-only or lexical fallback for exact terms
3. hybrid retrieval
4. hybrid plus reranking if latency and cost allow

6. Instrument the pipeline

Make the system observable enough to answer:

• which chunks were retrieved
• with what scores or rank positions
• under which filters
• from which corpus version
• which chunks were passed to the model
• which citations appeared in the answer

Log safely. Do not leak restricted content in debug traces. If needed, log chunk IDs and metadata instead of full text.

7. Evaluate retrieval separately from answer generation

Run retrieval checks before changing prompts.

For each query, ask:

• Was the relevant document retrieved at all?
• Was it retrieved high enough to survive truncation or reranking?
• Did filters wrongly exclude it?
• Did duplicate or near-duplicate chunks crowd out diversity?
• Did context assembly omit the best evidence?

Then evaluate answer behavior separately:

• correct use of retrieved evidence
• citation correctness
• abstention when evidence is weak
• handling of ambiguity or missing context

Do not blur retrieval failure with answer synthesis failure.