Agentsop Llamaindex

LlamaIndex · SOP

Third-person analytical view of how LlamaIndex thinks about turning private documents into a grounded answering system. The skill is for an LLM agent that writes / reviews / debugs RAG code — not for an end user reading docs.

何时激活 (Activation Rules)

Activate this skill when any of the following holds:

1. The user's request involves building, modifying, or debugging a RAG pipeline (retrieval over private/unstructured data + LLM synthesis).
2. The user mentions LlamaIndex (from llama_index...), LlamaParse, LlamaCloud, or a LlamaIndex-style primitive (VectorStoreIndex, SummaryIndex, IngestionPipeline, QueryEngine, SubQuestionQueryEngine, RouterQueryEngine, Settings, Workflows).
3. The user is comparing RAG frameworks (LlamaIndex vs LangChain vs Haystack vs raw vector store).
4. The user is choosing between stuffing context, RAG, or an agent for a knowledge task.
5. The user is debugging retrieval quality (hallucinations, wrong chunks, stale data, embedding drift) — even if the codebase predates LlamaIndex, the failure-mode taxonomy applies.
6. The user is evaluating a RAG system (faithfulness, relevancy, MRR, hit-rate).

Do not activate when:

• The task is pure agent orchestration with no retrieval (use LangGraph/CrewAI skill instead).
• The corpus is tiny (<100k tokens, static) and prompt-stuffing is the correct answer.
• The data is pure SQL/tabular with no unstructured component.

核心心智模型 (Core Mental Model)

LlamaIndex's design rests on three principles that distinguish it from "vector DB SDK + custom glue":

Principle 1 — The Index is a noun, not a verb

In LangChain, "indexing" is something you do to a vector store. In LlamaIndex, an Index is a first-class typed object with its own retrieval semantics. Picking the right Index is half the architecture decision.

The 5-layer pipeline:

Documents → Nodes → Index → Retriever → Query Engine → Response
   ↓         ↓        ↓         ↓             ↓
parsing   chunking  storage   filters    synthesis
metadata  graph     primitive  rerank    (refine/tree_sum/compact)

Each layer has a distinct failure mode and a distinct optimization knob. See references/R1-architecture.md for the layer-failure-knob mapping.

Principle 2 — A Node is a graph node, not a chunk

A Node carries: text, metadata, embedding, relationships (PREV/NEXT/PARENT/CHILD links), and lifecycle ids. The relationships field is what enables Hierarchical, Auto-Merging, and Sentence-Window retrieval. The mental flip: don't think "split into chunks", think "build a chunk-graph".

Principle 3 — Indices are not interchangeable

A RouterQueryEngine over multiple per-task indices is often the correct top-level shape, not a single monolithic VectorStoreIndex.

The 2025 shift

LlamaIndex now positions as "the leading document agent and OCR platform" (README). LlamaParse v2 + Workflows 1.0 (June 2025) + LlamaCloud mark a strategic move from "RAG framework" to "platform between messy documents and document-grounded agents". For a coder agent: assume Workflows for any new agentic code (QueryPipeline is deprecated).

SOP 工作流 (Agentic Protocol)

The protocol every RAG implementation must walk through. Each stage gates on the next.

Stage 0 — Frame the problem

Before code, answer:

1. Is the corpus unstructured + non-trivial size (>100k tokens) + growing? If not → see R4 boundaries; LlamaIndex may be the wrong tool.
2. Is retrieval quality the bottleneck (not orchestration)? If orchestration dominates → LangGraph leads, LlamaIndex becomes a retrieval tool inside it.
3. What is the query distribution? (lookup-only / summary / compare-contrast / mixed). This decides whether a single Index or a Router is needed.

Stage 1 — Baseline (cheap, fast, observable)

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.core.node_parser import SentenceSplitter

Settings.llm        = OpenAI(model="gpt-4o-mini")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.node_parser = SentenceSplitter(chunk_size=1024, chunk_overlap=20)

docs  = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(docs)
qe    = index.as_query_engine(similarity_top_k=4)

Pin Settings once at app boot, never inline. This eliminates the entire embedding-mismatch failure class (failure #4).

Stage 2 — Build the eval loop before optimizing anything

from llama_index.core.evaluation import (
    DatasetGenerator, FaithfulnessEvaluator,
    RelevancyEvaluator, RetrieverEvaluator,
)
qa = DatasetGenerator.from_documents(docs).generate_dataset_from_nodes(num=50)

Track {MRR, hit-rate, faithfulness, relevancy, p95 latency}. Every subsequent change must be gated on these numbers.

Most RAG failures in production trace to weak retrieval or sloppy ingestion — not the LLM. The eval loop is what surfaces them.

Stage 3 — Optimize in LlamaIndex's recommended order

From the official basic_strategies guide:

1. Prompt engineering (cheapest)
2. Embedding model (pick from MTEB; full re-embed if you change it)
3. Chunk size sweep ({256, 512, 1024, 2048}; default 1024 for prose, 80-160 for code)
4. Hybrid search (BM25 + dense) — only if traffic contains lexical-identity queries
5. Metadata filters — for multi-tenant / multi-collection corpora
6. Document/chunk decoupling — HierarchicalNodeParser+AutoMergingRetriever or SentenceWindowNodeParser
7. Reranking (Cohere / SentenceTransformer / ColBERT) — widen top_k to 20-50, rerank to 3-5

Note the order: prompts first, reranking last. Reranking is high-impact but expensive — exhaust cheap knobs first.

Stage 4 — Compose for query heterogeneity

Stage 5 — Production hardening

Apply the failure-mode checklist (R4). Top 5 non-negotiables:

• IngestionPipeline with docstore + UPSERTS_AND_DELETE for any live corpus.
• Settings.embed_model pinned at boot; embedding model name in index metadata.
• tree_summarize synthesizer when packing many chunks (mitigates lost-in-the-middle).
• Tracing/observability captures query + retrieved_nodes + scores + index_id + LLM prompt for every failure.
• Indices versioned as deployment artifacts; ingestion completes before traffic routing.

Stage 6 — Escalate to Workflows / Agents (only when justified)

Escalate when at least one of:

• A retrieval loop is needed ("retrieve → check → re-query").
• Tool calls beyond retrieval (calculator, web, code-exec).
• State surviving across query turns.
• Multiple specialized retrievers chosen at runtime.

Use Workflows 1.0 (event-driven), not deprecated QueryPipeline. Wrap query engines as QueryEngineTools and tune the description= carefully — it is the only signal the router/agent reads.

操作模型 (Operation Models)

Each operation: Trigger / Action / Output / Evidence.

OP-01 BaselineVectorIndex

• Trigger: First-pass RAG over a new corpus; retrieval-quality baseline unknown.
• Action: VectorStoreIndex.from_documents() with SentenceSplitter(1024, 20), top_k=4, default synthesizer. Ship to eval bench before tuning.
• Output: Working RAG endpoint + baseline {MRR, hit-rate, faithfulness, relevancy, p95}.
• Evidence: developers.llamaindex.ai/python/framework/optimizing/basic_strategies/basic_strategies/

OP-02 TuneChunkSize

• Trigger: Faithfulness below target OR retrieved chunks visibly truncated/incomplete.
• Action: Sweep chunk_size ∈ {256, 512, 1024, 2048} with overlap at ~10-20%; re-evaluate faithfulness + relevancy + latency. Default land: 1024 for prose, 80-160 for code.
• Output: Optimal chunk_size pinned + embedding model version locked in index metadata.
• Evidence: llamaindex.ai/blog/evaluating-the-ideal-chunk-size-for-a-rag-system-using-llamaindex-6207e5d3fec5 (faithfulness peaked at 1024 in LlamaIndex's own eval on Uber 10-K).

OP-03 AddReranker

• Trigger: Top-1 wrong but relevant docs appear in top-k (failure #1 / #10).
• Action: Add CohereRerank or SentenceTransformerRerank as a NodePostprocessor; widen retrieval top_k to 20-50, narrow to top_n=3-5 after rerank.
• Output: Faithfulness lift typically 5-15pp on noisy corpora; lower context-window pressure.
• Evidence: developers.llamaindex.ai/python/framework/optimizing/rag_failure_mode_checklist/ (#1, #10).

OP-04 AddHybridBM25

• Trigger: Traffic contains exact identifiers, error codes, SKUs, code symbols, rare jargon — pure dense silently misses them.
• Action: QueryFusionRetriever([vector_retriever, BM25Retriever]) or vendor hybrid (Qdrant/Milvus alpha). Tune alpha per query type, not globally.
• Output: Recall lift on lexical-identity queries with no degradation on semantic queries.
• Evidence: llamaindex.ai/blog/llamaindex-enhancing-retrieval-performance-with-alpha-tuning-in-hybrid-search-in-rag-135d0c9b8a00; BM25Retriever docs.

OP-05 DecoupleChunkScope

• Trigger: Chunk-size sweep produces no single winner (small wins precision, large wins context).
• Action: HierarchicalNodeParser + AutoMergingRetriever (for structured docs) or SentenceWindowNodeParser + MetadataReplacementPostProcessor (for flat prose). Embed small, return large.
• Output: Precision-recall pareto improvement; LLM gets surrounding context that small chunks alone lost.
• Evidence: AutoMergingRetriever / Hierarchical / SentenceWindow docs on developers.llamaindex.ai.

OP-06 RouteByQueryType

• Trigger: Corpus serves heterogeneous tasks (summary / lookup / compare) from one entry point.
• Action: Build per-task QueryEngines (SummaryIndex for digest, VectorStoreIndex for lookup, SubQuestionQueryEngine for compare) + a RouterQueryEngine with LLM or Pydantic selector. Carefully author each QueryEngineTool.description.
• Output: Each query lands on the structurally-correct retrieval primitive; latency stays bounded.
• Evidence: DeepLearning.AI Building Agentic RAG with LlamaIndex; router docs.

OP-07 DecomposeMultiHop

• Trigger: Compare/contrast queries; queries needing facts from >1 document; "what changed between X and Y?".
• Action: SubQuestionQueryEngine decomposes query → dispatches sub-questions to sub-engines → synthesizes.
• Output: Multi-hop answers a single retrieval cannot assemble.
• Evidence: developers.llamaindex.ai sub-question query engine docs.

OP-08 IngestionWithDocstore

• Trigger: Documents will update/delete over time (any production system).
• Action: IngestionPipeline(transformations=..., docstore=..., vector_store=..., docstore_strategy=UPSERTS_AND_DELETE). Run on a schedule, not manually.
• Output: Idempotent re-ingestion; no duplicate vectors; deletes propagate.
• Evidence: developers.llamaindex.ai/python/framework/module_guides/loading/ingestion_pipeline/; failure #3.

OP-09 MetadataFilters

• Trigger: Multi-tenant corpus; cross-contamination between sub-collections; access control needed.
• Action: Inject structured metadata at ingestion (tenant, doc_type, date); apply MetadataFilters at query time OR enable auto-retrieval to let an LLM emit filters.
• Output: Hard isolation between tenants; targeted retrieval without expensive rerank.
• Evidence: Failure #7; basic_strategies metadata filters section.

OP-10 EvalLoop

• Trigger: Any non-trivial RAG, pre-deploy AND continuously in production.
• Action: DatasetGenerator → labeled QA pairs; run FaithfulnessEvaluator + RelevancyEvaluator + RetrieverEvaluator(["mrr","hit_rate"]). Gate every change.
• Output: Quantitative regression test for every chunking / embedding / retriever / prompt change.
• Evidence: developers.llamaindex.ai/python/framework-api-reference/evaluation/; cookbook.openai.com/examples/evaluation/evaluate_rag_with_llamaindex.

OP-11 LockGlobalSettings

• Trigger: Multiple modules each instantiate LLM/embed independently — drift risk.
• Action: Set Settings.llm and Settings.embed_model once in app bootstrap. Forbid inline overrides in PR review.
• Output: Eliminates failure #4 (config drift) and #5 (embedding mismatch).
• Evidence: docs.llamaindex.ai/en/stable/module_guides/supporting_modules/service_context_migration/.

OP-12 AgenticWorkflow

• Trigger: Need loops, tool calls beyond retrieval, multi-step reasoning, or state across turns.
• Action: Build a Workflows 1.0 event-driven workflow OR a FunctionAgent/ReActAgent with QueryEngineTools. Do NOT use the deprecated QueryPipeline.
• Output: Cycle-capable agentic system with retrieval as one tool among many.
• Evidence: llamaindex.ai/blog/announcing-workflows-1-0-a-lightweight-framework-for-agentic-systems.

困境决策案例 (Dilemma Cases)

(Full text in references/R3-dilemma-cases.md. Summarized here.)

Dilemma 1 — Chunk size: precision vs context

困境: Small chunks → precise embeddings, fragmented context for the LLM. Large chunks → rich context, embeddings become "topic averages", recall on specific queries drops. Failure modes #2 and #6 are the two poles.

约束: Embedding model has a fixed input window; metadata is propagated into payload (so very small chunks become all-metadata — GitHub #12200, #13792); token budget caps how many chunks fit downstream.

决策步骤:

1. Generate ~20 eval QA pairs.
2. Sweep chunk_size ∈ {128, 256, 512, 1024, 2048} with overlap = 10-20%.
3. Build a VectorStoreIndex per config; record faithfulness, relevancy, latency.
4. If a single winner emerges → pin it.
5. If the frontier is non-flat → do not compromise; switch to small-embed/large-return via Hierarchical+AutoMerging or SentenceWindow.

结果: LlamaIndex's own published study (Uber 10-K) peaked at 1024 on both faithfulness and relevancy → 1024 became the framework default for prose. For code: 80-160 tokens. When the eval doesn't converge, decoupling wins; never average two bad chunk_sizes.

可提取的操作: OP-02 TuneChunkSize, OP-05 DecoupleChunkScope. Anti-pattern A1.