Agentsop Selfhost Decision

Self-host vs Managed-cloud Decision — A Project-Kickoff Rubric

Overlay, not a deep dive. This skill answers where to run (self-host vs managed), not which engine ([[agentsop-llm-engine-selection]]) or how to build the app ([[agentsop-dify]]). It fires first, at kickoff, and hands off to those once the side is chosen.

1. 何时激活 (When to Activate)

1.1 直接信号 (Direct triggers)

• At kickoff you must decide where an LLM or LLM platform runs: a managed API/cloud (OpenAI / Anthropic / Bedrock / Dify Cloud) vs your own GPUs / your own Docker (vLLM, self-hosted Dify).
• Cost pressure: monthly managed spend is climbing; someone says "should we just run our own and stop paying per token?"
• Compliance pressure: a data-residency / air-gap / regulated-data requirement (finance, medical, gov, GDPR region-lock) appears and the managed path is suddenly in question.
• You're comparing a self-hostable platform's tiers — e.g. Dify Cloud Pro ($59) vs self-deployed Docker [architjn.com/blog/dify-cloud-pricing-plans], or managed-vLLM-as-a-service vs your own H100s.

1.2 反向信号 (Skip this rubric when)

• The decision is already self-host, and the open question is which engine → go to [[agentsop-llm-engine-selection]] (vLLM vs TGI vs SGLang vs TensorRT-LLM vs llama.cpp).
• The decision is already self-host Dify, and the open question is how to build/operate it → go to [[agentsop-dify]].
• Single user / hobby / one stream — the answer is "just call the managed API"; no rubric needed.
• Training / fine-tuning siting — different cost structure (burst GPU, spot, not steady-state serving).

1.3 心智门槛 (Mental check)

This rubric exists because the loud reflex — "running our own is cheaper / more serious" — is true only above a volume crossover, and only if you have the ops capacity, and only if compliance hasn't already forced your hand. The job is to evaluate the gate before the slider, and to cost the ops burden, not just the GPU.

判断公式:

• 如果你的瓶颈是 per-token spend at high, predictable volume → self-host may win (run OP-1).
• 如果你的瓶颈是 data can't leave our boundary → compliance gate decides (run OP-2), cost is secondary.
• 如果你的瓶颈是 we don't have anyone to run a GPU box at 3am → managed wins regardless of the GPU math (run OP-3).

2. 核心心智模型 (Core Mental Model)

2.1 The trade, in one line

Self-host trades ops burden for control + unit-cost-at-scale. Managed trades $/token for zero ops. The crossover is a function of two axes: volume and compliance.

• Managed: you pay $/token, marginal, no fixed cost, no ops. Cost scales linearly with usage and never sleeps.
• Self-host: you pay a fixed floor (GPU + ops labor + infra) plus a tiny marginal cost. Cost is flat-then-cheap-per-unit — but only if utilization stays high.

2.2 Two axes, not one — and they are not symmetric

                COMPLIANCE  (a GATE — binary, evaluated FIRST)
                     │
   managed FORBIDDEN │  self-host (or in-region managed) MANDATORY
   ──────────────────┼──────────────────────────────────────────►  VOLUME
                     │                                    (a SLIDER —
   managed allowed   │   below V*: managed cheaper          continuous
                     │   above V*: self-host cheaper         crossover)
                     │   (IF utilization high + ops capacity exists)

• Compliance is a GATE: binary, evaluated before cost. A hard data-residency / air-gap NO fails the managed path regardless of volume [dify Case 3]. (Caveat: a managed in-region / VPC / BAA tier can re-open the gate.)
• Volume is a SLIDER: a continuous cost crossover V*. Below it managed wins; above it self-host wins — conditionally.

2.3 The crossover formula (OP-1)

managed_cost(V)   = V × $/token_managed
selfhost_cost(V)  = (GPU + ops_labor + infra_fixed) + V × $/token_marginal

V*  =  (GPU + ops_labor + infra_fixed) / ($/token_managed − $/token_marginal)

• Below V*: managed total cost is lower → use managed.
• Above V*: self-host total cost is lower → self-host if the two side-conditions hold.
• The term people forget is ops_labor — a fraction-of-FTE DevOps cost (backups, monitoring, upgrades, on-call). Cost it explicitly; it is rarely zero.

2.4 The two side-conditions that invalidate "above V* → self-host"

1. Utilization — idle GPUs invert the math. A GPU billed 24/7 but used 20% of the time has 5× the effective $/token. Self-host only wins at high, steady utilization.
2. Ops capacity — the GPU math assumes someone can actually run it. No DevOps owner → outage + opportunity cost erases the savings (OP-3).

2.5 The throughput feedback loop (OP-4)

Self-host volume isn't free of limits. A platform has a per-replica ceiling — Dify's is ~10 QPS/pod, gated by per-node DB queries [dify §6.2]; vLLM's is gated by KV-cache occupancy + preemption [vllm OP-5]. To hit volume V you need replicas = peak_QPS / ceiling, and that replica count feeds back into the GPU term of V*. High volume can need so many replicas that the crossover moves against you.

2.6 Reversibility is a design choice (OP-7)

The decision is not permanent. Keep a thin OpenAI-compatible API surface on both sides so flipping managed↔self-host is a config change, not a migration. Define the flip trigger up front (spend > V* for N months; new compliance rule; lost ops owner). Re-evaluate quarterly — both your volume and the price/quality frontier move (May 2026 stamp; re-measure).

3. SOP 工作流 (Standard Operating Procedure)

[Step 0] Confirm this is a "where to run" question
   ├─ "which engine?"      → [[agentsop-llm-engine-selection]], stop
   ├─ "how to build Dify?" → [[agentsop-dify]], stop
   └─ "managed vs our own?"→ continue

[Step 1] COMPLIANCE GATE first  (OP-2)  ── binary, overrides cost
   ├─ Regulated / residency / air-gap / contract boundary?
   │     ├─ YES, and no managed in-region/BAA/VPC tier  → self-host MANDATORY → Step 3
   │     └─ YES, but a compliant managed tier exists     → managed re-opened → Step 2
   └─ NO  → Step 2

[Step 2] VOLUME SLIDER  (OP-1)  ── compute the crossover
   ├─ Estimate monthly volume (reqs × tokens)
   ├─ managed_cost  = V × $/token
   ├─ selfhost_cost = (GPU + OPS_LABOR + infra) + V × marginal      ← include ops!
   ├─ V* = fixed / (managed_$tok − marginal_$tok)
   ├─ V below V*  → managed wins → Step 5 (plan fallback)
   └─ V above V*  → self-host candidate → Step 3

[Step 3] HONEST OPS-CAPACITY CHECK  (OP-3)  ── the trap door
   ├─ DevOps / SRE / on-call owner exists?           ─ no → managed wins anyway → Step 5
   ├─ Can run external Postgres/Redis/VectorDB + K8s? ─ no → managed or hire first
   └─ yes → Step 4

[Step 4] THROUGHPUT-HEADROOM CHECK  (OP-4)  ── re-feed into cost
   ├─ replicas = peak_QPS / per-replica_ceiling (~10 QPS/pod Dify; KV-bound vLLM)
   ├─ Re-run V* with the TRUE replica count (GPU term grows)
   ├─ still favorable → self-host → Step 5
   └─ flipped unfavorable → reconsider managed or HYBRID (Step 4b)

[Step 4b] HYBRID option  (OP-5)  ── if volume is spiky or tiered
   └─ self-host the steady floor + burst/overflow to managed; or
      compliance/premium → self-host, bulk → managed

[Step 5] PLAN THE FALLBACK  (OP-6, OP-7)
   ├─ Lock-in / license audit (same-image? egress? contract minimums?)
   ├─ Thin OpenAI-compatible abstraction so the flip is config, not migration
   ├─ Define the bidirectional flip trigger
   └─ Schedule quarterly re-evaluation

The ordering is load-bearing: gate (Step 1) before slider (Step 2) before capacity (Step 3) before throughput (Step 4). A compliance NO short-circuits everything; a missing ops owner short-circuits a favorable GPU cost.

4. 操作模型 (Operation Model: Trigger / Action / Output / Evidence)

OP-1: Volume-crossover calculation

• Trigger: cost is a stated pressure; you can estimate monthly volume.
• Action: estimate V = reqs/mo × (in+out tokens); managed = V × $/tok; selfhost = (GPU + ops_labor + infra) + V × marginal; solve V* = fixed / (managed_$tok − marginal_$tok). Place projected V relative to V*. Include the ops-labor term — do not assume it's zero.
• Output: a crossover V* and which side projected volume sits on, with ops labor explicit.
• Evidence: small team / small usage → Cloud $59–159/mo beats 养 DevOps; millions of calls → self-host [dify Case 3 决策矩阵; architjn.com/blog/dify-cloud-pricing-plans].

OP-2: Compliance / data-residency gate

• Trigger: data is regulated, customer-confidential, or contractually region/boundary-bound.
• Action: ask whether any law/contract/cert (HIPAA, finance, GDPR residency, air-gap, gov) forbids data leaving your boundary. YES → managed path fails regardless of volume → self-host (or a managed in-region/BAA/VPC tier, if one satisfies the boundary — check before defaulting to self-host). Evaluate before the volume slider.
• Output: a pass/fail on the managed path from compliance alone, decided first.
• Evidence: 合规/数据驻留 (金融、医疗) → self-host; air-gapped/内网 → self-host [dify Case 3]. Multi-tenant prefix privacy → cache salting [vllm OP-4 caveat].

OP-3: Honest ops-capacity check

• Trigger: self-host looks attractive on cost; validate you can actually run it.
• Action: self-host is not "docker compose up done" — it's reverse proxy + HTTPS + backups + monitoring + upgrade strategy + on-call. Score the team honestly: is there a DevOps/SRE owner? Production self-host means external PostgreSQL/Redis/Vector DB (not the bundled containers), K8s + Helm or equivalent, a tested upgrade path. No ops capacity → the "cheaper at scale" GPU math is a trap (outage + opportunity cost).
• Output: a go/no-go on self-host based on real ops capacity, with the unbundling requirement flagged.
• Evidence: 自部署 != docker compose up; 生产 K8s+Helm + 外部 PostgreSQL/Redis/Vector DB [dify Case 3 实操要点]. vLLM needs 2+N CPU cores per N GPUs or the API server bottlenecks before the GPU [vllm Anti-pattern 6].

OP-4: Throughput-headroom reality check (self-host)

• Trigger: self-host chosen on cost grounds; confirm the stack sustains the volume that justified it.
• Action: find the per-replica ceiling (Dify ~10 QPS/pod, node-level DB-query bottleneck; vLLM KV-cache occupancy + preemption). Compute replicas = peak_QPS / ceiling, then re-feed that replica count into OP-1's GPU+ops term. If the replica count is large, re-run the crossover; the "cheaper at scale" assumption may not survive.
• Output: a replica count required for the volume, fed back into V*.
• Evidence: ~10 QPS/pod hard limit, 每节点单独 DB query [dify §6.2]; triage num_requests_waiting / KV occupancy / preemption [vllm OP-5].

OP-5: Hybrid — self-host baseline + managed burst

• Trigger: volume justifies self-host for the baseline but traffic is spiky, or a premium/compliance tier coexists with a bulk tier.
• Action: provision self-host for the steady-state floor (high utilization keeps GPU $/token low); route burst/overflow to a managed API so you don't over-provision idle GPUs for peaks. Optionally split by tier: compliance/premium → self-host; bulk/non-sensitive → managed. Keep one gateway so the split is config, not a rewrite.
• Output: a baseline-self-host + burst-managed routing plan that keeps utilization high and caps over-provisioning.
• Evidence: two-replica tiering (FP8 premium / AWQ free) [vllm Dilemma 2]; add a replica / data-parallel pod when SLA violated under load [vllm Step 7].

OP-6: Same-image / lock-in audit

• Trigger: evaluating a self-hostable platform's paid tiers, or worried about managed-vendor lock-in.
• Action: for self-hostable platforms, check whether Community / Premium / Enterprise are the same Docker image gated by env vars / license keys (then the open tier is often technically sufficient — the paywall buys support, not capability). For managed APIs, audit lock-in surface: proprietary model, proprietary fine-tune format, egress cost, contract minimums. Prefer vendor-neutral self-host engines (vLLM, open weights) so the door stays open both ways. Confirm the license permits your commercial use.
• Output: a lock-in / license assessment per candidate, plus whether paid self-host tiers differ in substance.
• Evidence: Community/Premium/Enterprise 用同一 Docker image, 差异在环境变量 [dify Case 3]; license 是 Apache-like "not really" — 确认条款 [dify §6.4]; TensorRT-LLM beats vLLM 30–50% but locks you in [vllm §7].

OP-7: Fallback / exit plan

• Trigger: a decision is made (either way); de-risk before scaling on it.
• Action: if managed, define the flip-to-self-host trigger (spend > V* for N months; new compliance rule) and keep an open-weights equivalent identified. If self-host, define the flip-to-managed trigger (ops/outage cost exceeds savings; lost DevOps owner) and wire a managed fallback into the gateway. Keep the abstraction thin (OpenAI-compatible) so the flip is config, not migration. Re-evaluate quarterly.
• Output: a documented bidirectional flip trigger + a thin abstraction so reversing is cheap.
• Evidence: staged, reversible engine choices [vllm §7 common pattern]; bottleneck-driven re-evaluable decision [dify §1 判断公式].

5. 困境决策案例 (Dilemma Cases)

Dilemma 1: Low volume, but data-residency requires self-host

Situation: A regional healthcare startup runs a doc-grounded copilot. Volume is ~50k requests/mo — well below the cost crossover V*, where a managed API ($/token) would clearly be cheaper than buying and running a GPU. But patient data is bound by data-residency + air-gap rules.

Tension:

• The volume slider (OP-1) says managed — at this volume the GPU + ops floor is far more expensive than per-token spend.
• The compliance gate (OP-2) says managed is forbidden — data cannot leave the boundary.

Resolution heuristic:

• The gate overrides the slider. Evaluate compliance first; a hard residency/air-gap NO fails the managed path regardless of how cheap it is [dify Case 3: 合规/数据驻留, air-gapped → self-host].
• But right-size it. Compliance forcing self-host does not license self-host scale. Provision the smallest adequate stack — single replica, modest GPU — and accept the ops burden as the cost of compliance, not as an excuse to over-build (OP-1's V* still tells you how small you can go).
• Check the escape hatch first: if the managed provider offers an in-region / VPC / BAA tier that satisfies the boundary, that re-opens the (cheaper, zero-ops) managed path. Don't default to self-host before checking [OP-2].

Evidence: [dify Case 3 决策矩阵 (合规/数据驻留, air-gapped → self-host)]; [vllm OP-4 caveat (multi-tenant privacy / cache salting)].