Agentsop LLM Engine Selection

LLM Engine Selection SOP

State-of-the-art warning. This skill is dated May 2026. The inference-engine landscape moves in 3–6 month cycles (TGI exited maintenance into deprecation in late 2025; SGLang's RadixAttention regressed vLLM's lead in 2024; TensorRT-LLM dropped its proprietary-engine requirement in 2025; Ollama added concurrent-request support mid-2025). Re-verify before betting a quarter of eng budget on any choice below.

1. 何时激活 (When to activate)

Activate this skill any time a coder-agent must:

• Pick a serving stack for a new project (production hosting / batch / edge / dev laptop / multi-tenant SaaS / structured-output service).
• Defend an existing stack against a "let's switch to X" pressure.
• Migrate: justify or block a swap (e.g. TGI → vLLM, Ollama → vLLM, vLLM → TensorRT-LLM).
• Mix: design a multi-tier deployment (e.g. premium tier on TensorRT-LLM, free tier on vLLM-AWQ, dev on Ollama).
• Audit a recommendation that smells like benchmark-cherry-picking ("X is 5× faster").

Do not activate for:

• Tuning a single chosen engine — defer to the dedicated skill (vllm, sglang, tensorrt-llm, llama-cpp).
• Training/fine-tuning runtime selection — different problem class (accelerate, deepspeed, axolotl).
• Hosted-API procurement (OpenAI / Anthropic / Bedrock) — engine choice doesn't apply.

2. 核心心智模型 (Core Mental Model)

Engine choice is a function of (hardware × workload × constraint), not "which is fastest".

There is no global ranking. Every "X beats Y by N%" headline holds only inside an unstated (hardware, batch size, ISL/OSL, model, quantization, concurrency) tuple. Change any axis and the ranking flips.

2.1 The four-axis decision space

1. Hardware axis — NVIDIA H100/A100 (NVLink) ≠ NVIDIA L40S/RTX (PCIe-only) ≠ AMD MI300 ≠ Apple Silicon ≠ CPU-only. The interconnect topology matters as much as raw FLOPS [spheron.network 2026]. PCIe-only tensor parallelism collapses; NVLink rescues it.
2. Workload axis — production multi-user (throughput) vs latency-bound single-stream vs offline batch vs edge single-user vs structured-output service vs multi-LoRA SaaS. Each has a different winner.
3. Constraint axis — license (Apache vs proprietary), vendor lock-in tolerance, engineering budget (1 day vs 2 weeks setup), commercial-use clauses, on-prem vs cloud, P50 vs P99 SLA.
4. Maturity axis — the engine's coverage of YOUR model family. A 2025-launched MoE may run on vLLM day-1 but need a 3-month wait for TensorRT-LLM, and may never get a stable GGUF.

2.2 The default in 2026

For GPU-backed, multi-user, open-weights serving, the default is vLLM. It owns the production slot because it is vendor-neutral (NVIDIA/AMD/Intel/TPU/Apple-experimental), supports 200+ architectures including MoE/multimodal, ships an OpenAI-compatible API, and HuggingFace themselves recommend it over their own (now-maintenance-mode) TGI [yottalabs.ai 2026; vllm-project README].

You only reach past vLLM when one of three conditions binds:

• Workload-binding (heavy prefix sharing or structured generation → SGLang).
• Hardware-binding (NVIDIA-only + max throughput goal → TensorRT-LLM; CPU/edge/Apple → llama.cpp/MLX/Ollama).
• Operator-binding (dev laptop, want 5-min setup → Ollama; ≤1 concurrent user → llama.cpp).

2.3 "Fastest" is a category error

Throughput-per-GPU, throughput-per-dollar, P50 TTFT, P99 ITL, and developer-time-to-first-request are five different goals, and the engines optimize for different combinations:

Pick by which axis your project is binding on, not by which engine has the most stars.

3. SOP 工作流 (SOP Workflow)

[Step 0] Define the four-axis constraint vector
   ├─ Hardware: GPU vendor, count, interconnect (NVLink? PCIe?), VRAM/GPU
   ├─ Workload: # concurrent users, ISL/OSL distribution, shared-prefix %, structured-out %
   ├─ Constraint: license, vendor-lock tolerance, eng-days budget, P50/P99 SLA
   └─ Model: family (Llama/Qwen/Mixtral/DeepSeek/Mamba/...), size, quantization preference

[Step 1] Eliminate incompatible engines (hard filters)
   ├─ No NVIDIA GPU?            → drop TensorRT-LLM
   ├─ CPU/Apple/edge only?      → drop vLLM (production), TGI, TensorRT-LLM
   ├─ Need OSS-permissive only? → drop TensorRT-LLM (NVIDIA license)
   ├─ Mamba / brand-new arch?   → check vLLM+SGLang coverage; likely drop others
   └─ Multi-LoRA hot-swap?      → vLLM (best), TensorRT-LLM (good), SGLang (good); others drop

[Step 2] Map workload to engine strength
   ├─ Throughput + mixed traffic         → vLLM
   ├─ Shared prefixes (chatbot, agent)   → SGLang (≈29% over vLLM on shared-context [n1n.ai 2026])
   ├─ Structured JSON/regex at scale     → SGLang (RadixAttention + state machines)
   ├─ Peak throughput, NVIDIA-only, big budget → TensorRT-LLM (+30–50% over vLLM [n1n.ai 2026])
   ├─ Single user, dev laptop            → Ollama
   ├─ CPU/Apple/edge/embedded            → llama.cpp / MLX
   └─ Multi-tenant SaaS with LoRA fleet  → vLLM (multi-LoRA mature) or SGLang

[Step 3] Sanity-check hardware topology
   ├─ TP requires NVLink (≈900 GB/s) — not PCIe Gen4 (≈32 GB/s, ~28× slower)
   ├─ PCIe-only box → smaller TP + replicas, or PP, or switch engines
   ├─ Cross-NUMA across the same node → as bad as PCIe; pin to NUMA-local GPUs
   └─ Multi-node → require IB / RoCE, not Ethernet

[Step 4] Benchmark top 2 on YOUR workload
   ├─ Replay representative ISL/OSL distribution (NOT MMLU prompts)
   ├─ Measure P50 + P99 TTFT, P50 + P99 ITL, tokens/s/GPU, $/M-tokens
   ├─ Watch saturation: queue depth, KV occupancy, preemption count
   └─ Decide; document the constraint vector that justified the pick

[Step 5] Plan the escape hatch
   ├─ Note the workload threshold that would force a switch
   ├─ Keep the OpenAI-compatible API layer so swaps are mechanical
   └─ Re-evaluate every 6 months — engines evolve in quarters, not years

4. 操作模型 (Operation Model)

OP-1: Decision matrix lookup by (hardware, workload)

• Trigger: User asks "what should we serve X on?" and gives hardware + workload.
• Action: Look up the row in §7 table; eliminate by Step 1 hard filters; pick by Step 2 strength match.
• Output: A primary engine + a fallback, with the constraint vector that justified the pick.
• Evidence: §7 cross-framework matrix; [yottalabs.ai 2026]; [aimadetools.com 2026].

OP-2: Detect "wrong-tool-for-the-job" symptoms

• Trigger: A team is using engine E but reporting one of: low throughput despite tuning, unsupported model architecture, structured-output workarounds, single-user pain on a multi-user engine.
• Action: Match symptom → engine mismatch:
  • vLLM stuck below expected throughput on PCIe-only L40S → switch to PP or 2× independent TP=2 replicas (engine isn't the problem; topology is).
  • vLLM with heavy guided_decoding overhead on agent traffic → A/B SGLang.
  • TGI deployment, HF themselves now recommend vLLM/SGLang → plan migration.
  • Ollama in "production" with ≥10 concurrent users → graduate to vLLM.
  • llama.cpp on a serving-many-users box → graduate to vLLM.
• Output: Migration recommendation with cited reason.
• Evidence: [yottalabs.ai 2026]; [contracollective.com 2026].

OP-3: Quantization-driven engine pick

• Trigger: Quantization format dictates engine; team has GGUF / AWQ / GPTQ / FP8 weights already.
• Action:
  • GGUF only → llama.cpp / Ollama / LM Studio. vLLM has experimental GGUF; not production-grade.
  • AWQ / GPTQ → vLLM (first-class), SGLang (good), TensorRT-LLM (supported, needs engine rebuild).
  • FP8 (Hopper/Ada+) → vLLM, SGLang, TensorRT-LLM all support; near-lossless [arxiv 2411.02355].
  • MLX format (Apple) → MLX / mlx-lm; not portable to other engines.
• Output: Engine shortlist gated by available weights format.
• Evidence: [docs.vllm.ai/quantization]; [arxiv.org/abs/2411.02355].

OP-4: Structured-output workload routing

• Trigger: ≥30% of requests need JSON / regex / grammar-constrained output, OR the project is an agent with tool-call state machines.
• Action: Default to SGLang. RadixAttention + first-class state machines beat vLLM's guided_decoding at scale. If staying on vLLM, expect throughput hit on constrained requests; budget for it.
• Output: SGLang shortlisted as primary; vLLM as fallback if SGLang doesn't support the model.
• Evidence: [sglang.ai blog]; [n1n.ai 2026 SGLang +29% on shared-context].

OP-5: Apple Silicon / edge routing

• Trigger: Target platform is M-series Mac, iPhone/iPad, Jetson, Raspberry Pi, or other ARM/edge.
• Action:
  • M-series Mac, dev/local: Ollama (best UX) or MLX (best perf, Apple-native).
  • M-series Mac, production server: llama.cpp Metal backend or MLX. vLLM Metal/MPS is experimental as of 2026; do not ship on it [aimadetools.com 2026; contracollective.com 2026].
  • Jetson / ARM Linux: llama.cpp (broadest hardware support) or TensorRT-LLM if Jetson Orin.
  • CPU-only x86 server: llama.cpp with AVX-512; expect single-digit users.
• Output: Apple/edge-specific engine pick with format requirement (GGUF / MLX).
• Evidence: [contracollective.com 2026]; [aimadetools.com 2026].

OP-6: Multi-tier deployment design

• Trigger: Project has both interactive (premium, latency-bound) and batch (free, throughput-bound) tiers.
• Action: Run heterogeneous engines:
  • Premium / interactive on TensorRT-LLM (if NVIDIA-locked + budget) or vLLM FP8 with prefix caching.
  • Free / batch on vLLM AWQ-4 (high concurrency, lower cost) or SGLang if prefix-sharing.
  • Dev / staging on Ollama for fast model swaps.
  • Edge / on-device client on llama.cpp (GGUF).
• Output: 2–3 engine deployment plan with a single OpenAI-compatible gateway in front.
• Evidence: Common production pattern documented in [yottalabs.ai 2026; contracollective.com 2026].

OP-7: Bench-then-pick when the choice is close

• Trigger: Two engines both pass Step 1 + 2; team disagrees.
• Action: Run a 1-day benchmark with replayed traffic. Required metrics: P50 + P99 TTFT, P50 + P99 ITL, tokens/s/GPU, $/M-tokens, peak memory, model-load time. Do not benchmark on MMLU-style prompts; use real production ISL/OSL.
• Output: A decision memo with the constraint vector + measured numbers + the threshold that would flip the decision.
• Evidence: Standard practice; all cited comparison reports note that workload-specific benchmarks override published numbers.

5. 困境决策案例 (Dilemma Cases)

Dilemma 1: Team wants vLLM, but only has a PCIe-only 4-GPU box

Situation: A team standardized on vLLM but their cluster is 4× L40S on PCIe Gen4 (no NVLink). They configure --tensor-parallel-size 4 for Llama-3-70B-FP8 and see throughput collapse — single-replica TPS is ~30% of what the H100×4 benchmark advertises.

Tension:

• vLLM is the right engine, but TP=4 over PCIe is the wrong topology. The all-reduce after every layer chokes on the ~32 GB/s PCIe link vs the ~900 GB/s NVLink the benchmarks assumed [spheron.network 2026].
• Switching engines doesn't help — the same all-reduce penalty hits TensorRT-LLM and SGLang.

Resolution:

• First: drop to TP=2 × 2 replicas (each replica uses an NVLink-paired or NUMA-local pair, if any), or use pipeline parallelism PP=4, TP=1. Replicas eliminate the cross-GPU all-reduce on the hot path [developers.redhat.com 2026 step-5].
• Second: if 70B doesn't fit at TP=2 with FP8 on L40S (48 GB/GPU), step down to AWQ-4 or use Llama-3-8B until hardware upgrades.
• Third: if neither works, swap to TGI (similar PCIe penalty but lower TP-coupling overhead in some configs) or accept a switch to llama.cpp for low-concurrency workloads.
• Key lesson: the bottleneck was hardware topology, not engine choice. Always check interconnect before blaming the engine.

Evidence: [docs.vllm.ai parallelism_scaling]; [developers.redhat.com 2026]; [spheron.network 2026].

Dilemma 2: Structured JSON output at scale — SGLang vs vLLM

Situation: An agent platform serves Qwen-2.5-32B-Instruct with ~70% of requests demanding strict JSON schema output (tool-calling, function arguments). On vLLM with guided_decoding, throughput drops ~40% vs unconstrained baseline; P99 TTFT regresses.

Tension:

• vLLM has the broader model + production track record; team already operates it.
• SGLang's RadixAttention + native state-machine compilation gives ≈29% higher throughput on shared-context workloads and meaningful gains on constrained decoding [n1n.ai 2026]; agent traffic has both properties.
• Migration cost is real: deployment scripts, observability, ops runbooks.

Resolution:

• A/B: run SGLang as a sidecar on a fraction of traffic. Measure constrained-output TPS + correctness rate on real JSON schemas (not toy schemas).
• If SGLang wins ≥25% on YOUR traffic → migrate the structured-output path; keep vLLM for unconstrained traffic, OR consolidate on SGLang if it covers all your models.
• If SGLang model coverage misses a key model in your fleet → stay on vLLM, optimize guided_decoding (xgrammar backend), and re-evaluate next quarter.
• Heuristic: structured-output workload share above ~30% justifies SGLang shortlisting; above ~50% it usually wins.

Evidence: [sglang.ai/blog]; [n1n.ai 2026]; [yottalabs.ai 2026].