LLM Cost Optimizer

Overview

This skill walks through analyzing and reducing LLM spend on a Vellum assistant. There are three layers:

1. Provider connections — named auth configs (e.g. anthropic-managed, my-personal-key)
2. Model profiles — named presets (provider + model + effort + thinking + contextWindow). Four managed defaults, with UI labels. Note that the keys do not track the labels: read the key, not the name, when pinning a call site.
   • balanced → Balanced (the general agent-loop profile)
   • quality-optimized → Quality (the expensive escalation profile)
   • cost-optimized → Budget (the cheap utility/background profile, and the one to pin for spend reduction)
   • latency-optimized → Fast (the low time-to-first-token profile, used by live voice; faster but not cheaper than Budget)
3. Call-site profile pins (llm.callSites.<id>.profile) — optional per-task overrides of the shipped defaults.

The concrete model behind each managed profile depends on the install: platform-managed installs and BYOK installs resolve different providers/models, and the catalog changes over time. Never assume which model a profile maps to — read assistant config get llm.profiles and the usage breakdown by model to see what actually ran.

How model selection works — read this before diagnosing

Every LLM call resolves exactly one winning profile through a strict first-usable-wins chain. Profiles never merge with each other:

1. Per-conversation / per-run override — the user's /model pick, an open assistant inference session, or a schedule's pinned profile
2. llm.activeProfile — applies to mainAgent (the chat loop) only; it IS the user's chat-model selection and outranks any llm.callSites.mainAgent pin
3. llm.callSites.<site>.profile — explicit per-site pin
4. The call site's shipped default intent, resolved through llm.defaultProvider
5. balanced intent (final anchor)

A rung only wins if its profile exists, is enabled, and carries its own provider + model; otherwise resolution silently falls to the next rung.

Consequences that change how you diagnose cost:

• A missing or empty llm.callSites block is healthy, not a red flag. Every call site ships with a sensible default intent: the agent loop and quality-sensitive sites (mainAgent, subagentSpawn, compactionAgent, callAgent, patternScan, narrativeRefinement, memoryConsolidation, memoryV2Consolidation, memoryV3SelectL2, recall, conversationStarters, identityIntro, emptyStateGreeting) default to balanced; everything else (classifiers, summarization, titles, copy generation, memory extraction/retrieval/sweeps, heartbeat, home-screen content, etc.) defaults to cost-optimized. Nothing "falls back" to an expensive model.
• Do not write a full llm.callSites blob that mirrors the shipped defaults. That freezes today's defaults into user config and silently opts the user out of future default improvements (and of tuning shipped alongside them, like cache and context-window settings). Pin only deliberate deviations.

Step 1 — Measure current spend

# Monthly totals
assistant usage totals --range month

# Break down by conversation (what the user actually did — use this for presentation)
assistant usage breakdown --group-by conversation --range month

# Break down by call site (what kind of work is expensive — use for diagnosis)
assistant usage breakdown --group-by call_site --range month

# Break down by model (what actually ran)
assistant usage breakdown --group-by model --range month

# Break down by profile (which selection produced it)
assistant usage breakdown --group-by inference_profile --range month

Cross-reference the call_site and inference_profile breakdowns: a background call site showing spend under an expensive profile means an override or pin routed it there — that is the interesting finding, not the config defaults.

Add --json when you need token-level detail (input vs output vs cache_creation vs cache_read) — high input volume on a cheap model can outweigh low volume on an expensive one.

Step 1b — Present costs in user-friendly terms

After gathering the data, present findings in a format the user can act on. Users think in terms of conversations they had and automations they set up — not call sites, inference profiles, or cache economics. Use the call_site, model, and inference_profile breakdowns for diagnosis, but lead the presentation with what the user recognizes.

Presentation structure:

1. One-line headline with total monthly cost.
2. Conversations table — the user's own activity, sorted by cost descending. Columns: conversation name, turns, cost, and a brief note on why it was expensive (e.g. "One heavy session drove 65% of your total spend"). Roll up small conversations into an "All other conversations" row to keep the table to 4-6 rows.
3. "What I'd change to cut costs" — 2-3 bullets in plain English, biggest lever first. No jargon. Instead of "drop the balanced profile effort from high to medium," say "your chat model is set to high effort — dropping to medium would save ~$X/week." Each bullet states what to change, why it helps, and the estimated savings.
4. A clear ask — "Want me to make either of those changes?"

What to omit from the user-facing presentation:

• Background work (memory processing, heartbeats, health checks) — users can't control these individually and they're already on cheap models by default. Mentioning them adds noise without actionable signal.
• Call site names, inference profile names, token counts, cache ratios — these are diagnostic internals. Use them to figure out what's expensive, then translate to plain English.
• Recurring automation costs unless one is a significant contributor (>$4/month). A monthly $0.17 digest doesn't need its own line in the summary.

Step 2 — Read the effective configuration

assistant inference callsites list      # per call site: winning profile, default vs pinned
assistant inference profiles list      # effective profiles: managed + user, with availability
assistant inference profiles active    # the chat-model selection
assistant inference providers default  # default provider + availability
assistant inference session list
assistant inference providers list
assistant schedules list

For each recurring schedule, check which profile its runs use — assistant schedules get <id> shows an "Inference profile" line. A schedule with no pinned profile runs under the mainAgent model selection (the active profile), not a cheap background profile.

Step 3 — What typically drives cost (check in this order)

1. The chat loop and everything that inherits its profile. llm.activeProfile (and per-conversation /model sessions) is the #1 lever. Note the inheritance paths: subagents spawned from a conversation with a profile override run under that profile, and memory retrospectives run under the source conversation's profile when memory.retrospective.matchConversationProfile is enabled (they show under memoryRetrospective in the breakdown but are priced at the chat profile — this is deliberate, for prompt-cache reuse).
2. Recurring schedules without a pinned profile. Schedule runs default to the mainAgent model selection, and a pinned schedule profile overrides the entire run (every call site in it). A frequent schedule left on an expensive chat profile is a classic silent cost driver — check it with assistant usage breakdown --group-by call_site --schedule <id>, and per-run cost with assistant schedules runs <id>.
3. Pins to quality-optimized. No call site should be statically pinned to it; it is an on-demand escalation profile.
4. High-volume background sites. memoryRouter runs with a very large input window by design; heartbeat, memory sweeps, and summarization run often. These are already on cost-optimized by default — check whether a pin or override moved them off it.
5. Cache economics. Repeated-prefix call sites benefit from caching; one-shot sites ship with caching disabled. If cache_creation dwarfs cache_read on a site, flag it.