LLM Provider Setup

Overview

This skill is the canonical procedure for adding a new LLM provider, model, or inference profile to a Vellum assistant. Follow the steps in order — each step's output feeds the next, and the final live-call verification is mandatory. Never skip ahead by writing raw config JSON.

Run the steps strictly sequentially — one command per step, and read its output before running the next. Never batch create → verify → activate into a single turn: creation can fail validation, verification can fail on a wrong model id or missing credential, and activation must not happen until verification passed. Each step's output is the gate for the next one.

Step 0 — Check what's already available (avoid collecting keys unnecessarily)

Managed (platform-credentialed) routing may already cover the user's need — no API key required:

assistant inference providers list          # provider entries; `vellum` is the platform-managed route
assistant inference providers default       # default provider + availability status
assistant inference profiles list           # effective profiles: managed + user, with availability

Managed first. If the user is signed in to Vellum and the model they asked for is served by the managed route, build the profile on it — --provider vellum --model <model-id>, no --connection, no credential, nothing to prompt for. Skip Steps 1 and 2 entirely and go to Step 3; there is no key to collect. If the model turns out not to be managed-routable, profile creation says so explicitly (Step 4) — only then fall back to key collection.

Collect an API key only when there is genuinely no managed option: the user is not signed in to Vellum, the model is not served by the managed route, or the user explicitly wants to use their own key.

Step 1 — Reuse an existing key, or securely collect a new one

Before prompting the user for anything, check whether a suitable key is already stored:

assistant credentials list

If a credential for the target provider exists, reuse it — reference it by vault path in Step 2 and skip the prompt. Only collect a new key when none exists (or the user explicitly wants to replace it).

Never ask for secrets in chat, and never send the user to the Settings page for this. The key must not enter the conversation, and the collection happens inline in the current conversation — the secure prompt renders a masked input right where the user already is:

assistant credentials prompt --service <provider> --field api_key \
  --label "<Provider> API Key" --placeholder "sk-..."

Exit code 0 = stored; exit code 130 = the user dismissed the prompt (a valid choice, not an error — ask whether they want to try again or stop). Any other non-zero exit is a real error.

Step 2 — Create the provider connection

Reference the stored credential by vault path — the assistant only ever handles the reference string:

assistant inference providers create <connection-name> \
  --provider <provider> \
  --auth api_key \
  --credential credential/<provider>/api_key

For self-hosted or OpenAI-compatible endpoints, use --provider openai-compatible and supply the endpoint's base URL plus at least one model id (both are required for this provider type — the endpoint advertises no fixed catalog). Pass --model once per model the endpoint serves:

assistant inference providers create <connection-name> \
  --provider openai-compatible \
  --auth api_key \
  --credential credential/<provider>/api_key \
  --base-url https://<host>/v1 \
  --model <model-id> \
  --model <another-model-id>

For a local, keyless endpoint (e.g. LM Studio, vLLM) use --auth none and drop --credential. The managed Vellum connection is not editable — create a new named connection instead of modifying it.

Step 3 — Discover a valid model id (do not guess)

Model ids are the most common failure point — never write one from memory:

assistant inference models list --provider <provider>

Pick from the catalog output. If the user wants a model not in the catalog (e.g. brand new or self-hosted), probe it with a live call before configuring anything:

assistant inference send --model <candidate-id> --max-tokens 32 "Reply with OK"