Agentsop Framework Selection

Framework-Fit Decision Tree at Project Kickoff · SOP (ENHANCE overlay)

Overlay posture: this is the capstone Phase-D skill — the most-cited entry at any project kickoff. It decides which layer(s) you need and which framework owns each layer. It does not teach any framework's API; for that, descend to the per-framework SOPs (langgraph-sop, llamaindex-sop, dspy-sop, crewai-sop, vllm-sop, aider-sop, dify-sop). Every load-bearing claim carries an inline source tag resolving in references/R1-decision-tree.md.

Neutrality note: vendor pages each claim the center of the universe (LangChain: "use LangGraph for production"; LlamaIndex: "the document agent platform"; Dify: "scaffolding is the bottleneck"). This skill quotes those claims but does not adopt any of them. The 7 SOPs disagree on the crossover points; we surface the disagreements rather than papering over them.

1. 何时激活 (When to Activate)

Activate when any of the following fire:

• A new LLM / agent / RAG project is starting and no framework has been chosen yet.
• Someone asks "which framework should I use?" / "LangChain or LlamaIndex?" / "LangGraph vs CrewAI?" / "do we need a framework at all?"
• A coder is about to pip install an orchestration / RAG / agent framework before having articulated what layers the project needs.
• A project already picked one framework "for everything" and is now fighting it in a layer it was never good at (e.g., doing deep RAG inside CrewAI, or hand-rolling retrieval inside LangGraph).
• A no-code / visual builder (Dify, Flowise, LangFlow) has hit a complexity ceiling and the team is asking "do we rewrite in code?"

Do not re-run this skill mid-implementation for a layer already chosen — that is churn. Run it once at kickoff, and again only when a new layer appears (e.g., "we now need to self-host the model" → triggers [[agentsop-llm-engine-selection]]).

Mental check: the wrong framework is the single highest-cost decision in the project — it is a one-week-to-reverse mistake, sometimes a one-month one. crewai-sop · OP-1, vllm-sop · OP-7. Spend 20 minutes on this tree before opening any tutorial.

2. 核心心智模型 (Core Mental Model)

Frameworks are layers, not competitors. The single most common kickoff error is treating "LangChain vs LlamaIndex vs DSPy vs CrewAI vs vLLM" as a horse race with one winner. They are not on the same axis. A mature LLM system is a stack:

┌─────────────────────────────────────────────────────────────┐
│  L7 App platform / UI / Auth   │ Dify, Flowise, LangFlow      │  ship-fast scaffolding
├─────────────────────────────────────────────────────────────┤
│  L6 Serving / inference        │ vLLM, SGLang, llama.cpp …    │ → see [[agentsop-llm-engine-selection]]
├─────────────────────────────────────────────────────────────┤
│  L3 Orchestration / control    │ LangGraph, CrewAI, Workflows │ → see [[agentsop-agent-topology-selection]]
├─────────────────────────────────────────────────────────────┤
│  L2 Retrieval / context        │ LlamaIndex, Haystack         │  ingestion, index, query
├─────────────────────────────────────────────────────────────┤
│  L1 Modeling / prompt-compile  │ DSPy, Outlines, Guidance     │  the LM call itself
├─────────────────────────────────────────────────────────────┤
│  Coding-agent surface (cross)  │ Aider, Cline, Cursor …       │  end-user product, not a layer
└─────────────────────────────────────────────────────────────┘

DSPy's own ecosystem doc draws this layering explicitly — DSPy "sits underneath LangChain, LlamaIndex, LangGraph as a compiler for individual LM calls" dspy-sop · R5. LlamaIndex's doc says "many production systems use both: LlamaIndex as the retrieval layer, LangGraph as the orchestration layer" llamaindex-sop · R5. Dify's doc describes the hybrid "Dify for frontend/RAG/auth

• LangGraph for core agent logic behind HTTP" dify-sop · R5. The convergence is unanimous: choose per layer, then check interop.

Three corollaries:

1. You may not need every layer. A static-corpus Q&A bot needs L1 only (stuff the context window). A RAG chatbot needs L1+L2. A durable multi-step agent needs L1+L2+L3. Only self-hosting adds L6. Only mixed-role teams add L7.
2. The cleanest combinations are additive; the awkward ones are same-layer. DSPy+LangGraph (compile-inside-node) and LlamaIndex+LangGraph (retrieve-then- orchestrate) are textbook dspy-sop · R5. DSPy+LangChain or DSPy+CrewAI are a "smell" — both are L1-ish prompt strategies fighting for the same slot dspy-sop · R5.
3. "No framework" is a legitimate answer for ≥1 layer. Frameworks earn their dependency surface only past a complexity threshold (§4 gate G0).

3. SOP (The Procedure)

The kickoff procedure runs as numbered steps (Pass A = Steps 1–6, Pass B = Step 7, Pass C = Step 8):

Pass A — Identify the layers you actually need. Walk the stack top to bottom and mark each layer needed / not-needed for this project:

1. L1 Modeling — Is there ≥1 non-trivial LM call whose prompt will be iterated, swapped across models, or whose output is parsed by code? (Almost always yes.)

2. L2 Retrieval — Does the system answer over private / large / changing data it cannot fit in context? (Yes → need retrieval. No → skip.)

3. L3 Orchestration — Is there cycle / branch / memory-across-turns / human approval / >1 coordinating agent? (Yes → need orchestration. A single straight- line call → skip; raw SDK suffices.)

4. L6 Serving — Are you hosting open-weights models yourself (vs calling a hosted API)? (Yes → [[agentsop-llm-engine-selection]]. No → skip.)

5. L7 App platform — Do non-engineers (PM/ops) need to co-author or operate the app, or do you need UI+API+auth+logging out of the box? (Yes → consider a platform. No → code framework.)

6. Coding-agent surface — Is the deliverable code edits in a repo? (Yes → [[agentsop-repo-state-gating]] then Aider/Cline/Cursor. This is orthogonal to L1–L7.)

7. Pass B — For each needed layer, apply the per-layer fit rubric (§4). Each layer has its own short decision tree; do not let one framework's gravity pull you into using it for a layer it is weak at.

8. Pass C — Check interop and the "do you even need a framework?" gate. Confirm the chosen pieces compose (additive, not same-layer collisions — §7 interop map), and run gate G0 on each layer to confirm a framework beats raw SDK there.

Output of the procedure: a one-line-per-layer decision, e.g. L1: raw prompts (will revisit DSPy at 30 labeled examples) · L2: LlamaIndex · L3: LangGraph · L6: hosted API (no self-host yet) · L7: none (code-first).

4. 操作模型 (Operations)

G0 — The "do you even need a framework?" gate (run per layer)

Before adopting any framework on a layer, confirm raw SDK is insufficient. Frameworks trade dependency-surface and a learning curve for batteries. Raw wins when the layer is trivial:

"If the corpus is small (<100k tokens) and static → no framework needed. Stuff the context window with prompt caching." llamaindex-sop · R5. "Plain LangChain [or raw SDK]: no cycles, no state-between-turns, no HITL — a single LLM call." langgraph-sop · R5.

OP-1 — Layer identification (Pass A)

Mark each of L1/L2/L3/L6/L7 + coding-surface as needed or not, per §3 Pass A. This is the highest-leverage step: most "wrong framework" pain is actually "picked an L3 tool when the project was L2," or vice versa.

OP-2 — L1 Modeling pick rubric

• Raw prompts — default for a single, rapidly-iterating call (G0).
• DSPy — when you have ≥2 LM calls, ≥30 labeled examples, and a measurable metric, and prompts are brittle on model swap. DSPy compiles prompts into a versioned artifact dspy-sop · R5. Combines additively into L3 (compile inside a LangGraph node) and L2 (compile the synthesizer downstream of a retriever).
• Outlines / Guidance / LMQL — when you need grammar-level guarantees (valid JSON / regex / BNF) on a single call. Orthogonal to DSPy: Outlines forces the shape, DSPy makes the prompt good dspy-sop · R5.
• Do not pick two L1 strategies that fight for the slot (DSPy + LangChain- prompt-templates is a "smell" dspy-sop · R5).

OP-3 — L2 Retrieval pick rubric

• LlamaIndex — the default retrieval layer; index is a first-class noun, ~5-line baseline RAG, built-in evaluators, LlamaParse for messy documents llamaindex-sop · R5. Reach for it whenever private/large/changing data must be retrieved.
• Haystack — when the system is "classical IR with an LLM bolted on" and YAML-configurable pipelines are valued by ops llamaindex-sop · R5.
• RAGFlow — when retrieval quality on hard documents (tables, layout, KG-RAG) is the literal bottleneck, not app assembly dify-sop · R5.
• Raw vector store (Pinecone/Qdrant/…) — only if you will reimplement ≤2 RAG primitives (G0); otherwise you rebuild LlamaIndex badly llamaindex-sop · R5.
• Repo-map (Aider tree-sitter), NOT embeddings — when the "corpus" is a code repo. Aider's symbol-map hits 70.3% file-selection on SWE-Bench Lite without an index aider-sop · R5. The SOPs disagree here (LlamaIndex assumes embeddings work for code; Aider's evidence says repo-map beats them) — for code, prefer the repo-map; for prose, prefer embeddings (§6).

OP-4 — L3 Orchestration pick rubric

First gate through [[agentsop-agent-topology-selection]] (single-agent + tools handles ~80% of "multi-agent" asks crewai-sop · DC-1). Then, if orchestration is needed:

• Single agent + tools — the baseline. create_react_agent (LangGraph), dspy.ReAct, a CrewAI single Agent, or a plain SDK tool loop. Start here.
• LangGraph — when the workflow needs cycles, durable state across crashes, human-in-the-loop (interrupt()), time-travel debugging, or supervisor/swarm/hierarchical parallelism langgraph-sop · R5. The 2026 production-reliability leader.
• CrewAI — when the domain maps cleanly to roles on a whiteboard (researcher → writer → reviewer), non-engineers must read/edit agent definitions (YAML), and you want idea→demo fastest crewai-sop · R5. Trade-off: shallow state, thin eval, no built-in persistence.
• LlamaIndex Workflows — when the project is retrieval-heavy and needs only some agency; staying in-ecosystem keeps retrieval primitives as first-class neighbors llamaindex-sop · R5.
• The graph-vs-role crossover is the central disagreement (§5, DC-1). Common pattern: prototype on CrewAI, port to LangGraph for production crewai-sop · R5.
• AutoGen — only if conversation/debate is the literal product; note it is in maintenance mode as of 2026 — adopt with eyes open crewai-sop · R5, langgraph-sop · R5. OpenAI Swarm — study/reference only, not production.

OP-5 — L6 Serving pick → delegate

If self-hosting open-weights models: hand off entirely to [[agentsop-llm-engine-selection]]. One-line summary of that skill's rubric: production+GPU+concurrent → vLLM (the 2026 default); prefix-heavy agent/RAG → A/B SGLang; NVIDIA-locked + 1–2wk budget → TensorRT-LLM; CPU/edge/≤1 user → llama.cpp; local dev → Ollama vllm-sop · R5, vllm-sop · OP-7. If calling a hosted API (OpenAI/Anthropic/…), skip L6 entirely.

OP-6 — L7 App-platform pick rubric

• Dify — when the bottleneck is scaffolding (UI+API+auth+vector store+logs +versioning in one box) and mixed-role teams (PM+ops+eng) co-author dify-sop · R5. Visual DAG with Code-node escape hatch.
• Flowise — fastest path for a single "chatbot + doc retrieval" demo; thin beyond that dify-sop · R5.
• LangFlow — when you want a visual builder you can later export to Python and evolve (LangChain-bound) dify-sop · R5.
• n8n + AI nodes — when the project is automation-first (400+ non-AI integrations) with AI sprinkled in, not AI-first dify-sop · R5.
• Mind the ceiling (§5 DC-3, §6): visual builders break down past ~40 nodes / HITL needs / >10 QPS/pod / extreme RAG / sub-second latency dify-sop · R5.

OP-7 — Coding-agent surface pick → gate first

If the deliverable is code edits in a repo, gate through [[agentsop-repo-state-gating]] (Aider et al. are for existing repos; greenfield → plain LLM chat aider-sop · R5). Then: terminal + git-clean history + scriptable → Aider; per-tool-call approval in VS Code → Cline; visual/autocomplete + closed product → Cursor; cross-IDE → Continue; autonomous ticket→PR → OpenHands aider-sop · R5. This surface is orthogonal to L1–L7 — a coding agent uses L1–L6 internally.

OP-8 — Interop check (Pass C)

Confirm chosen pieces compose. Additive (good): DSPy-in-node, LlamaIndex-retriever- as-tool, Dify-frontend + LangGraph-behind-HTTP, vLLM serving any of the above via its OpenAI-compatible API dspy-sop · R5, dify-sop · R5. Collisions (re-pick): two L1 prompt strategies; two L3 orchestrators owning the same control flow; a visual platform asked to do what its code-escape-hatch should.