Fastllm Routing

FastLLM routing

A frontend model is a client-facing name backed by an ordered list of rules. The first rule whose conditions all hold wins and commits to its own targets; if none match, the defaults are used. Everything is pre-resolved into the snapshot, so the request path does no I/O to route.

Auth

Admin endpoints need a session cookie, not a bearer token. The gateway master key is not an admin credential.

curl -sk -c /tmp/ck -X POST https://192.168.10.129:4001/login \
  -H 'content-type: application/json' -d '{"name":"<user>","password":"<pw>"}'
curl -sk -b /tmp/ck https://192.168.10.129:4001/admin/frontend-models

Test before you apply

POST /admin/routing/dry-run answers "which model would this request hit, and which rule decided" without changing anything. Use it before and after any change here — it is how you check a rule does what you meant.

It cannot evaluate max_inflight_per_backend. The dry-run runs on the control plane, whose registry has no in-flight counters and no engine scrape, so every backend looks idle and a spill rule always reports as still matching. Only real traffic exercises that condition.

<!-- BEGIN GENERATED: endpoints -->

* optional field

<!-- END GENERATED: endpoints -->

Actions

Every rule has one, and every one is terminal — no rule contributes and passes on, so dry-run names a single deciding rule.

• route (default) — the rule's targets, ordered by its policy.
• deny — refuse. Needs deny_status, 4xx only (a 5xx would have every client library retrying something that is never going to be allowed).
• jump — continue in another frontend model's chain (jump_to), so shared policy is written once. A loop is refused at write time; a destination that was deleted is treated as no match and falls through.

tag is a field, not an action: it labels the usage rows the rule produced, so spend is answerable per decision. A rule that tags still routes.

A deny shows up in dry-run as a denied object with the status and message, and an empty candidates — which is a different thing from "nothing is serving", and the distinction is the point.

Cost as a condition (min/max_request_cost_micros) is priced at the cheapest model the frontend model can reach — one number per request, the same for every rule, so it does not depend on which rule is asking. Pairs with deny ("refuse anything over $0.50 however I route it"). Sending the expensive ones somewhere cheaper is not this; order the chain with the cheaper target first. All targets unpriced means no cost, and the rule declines.

Three levels, and they answer different questions

A rule's targets are an ordered failover chain. No policy, no weighted split — tried in the order written. POST /admin/rules/{id}/targets takes either provider_model_id or model_pool_id, never both.

A pool (/admin/model-pools) is a named group of provider models with one policy: cache-affinity, least-loaded, lowest-latency, round-robin, or unset for the weighted split by member weight. Cost is not one of them — a price is fixed, so it never balances; use a rule's min/max_request_cost_micros. Two pools may hold the same members and differ only in policy — that is what naming it buys. A pool expands in place: its chosen member, then its others, then the chain's next target.

A provider model has no policy. It is a model running on endpoints. Where several endpoints serve it they all go in its pool and the deployment's --policy spreads across them; there is nothing to set per model.

routing_rules.policy and frontend_models.policy no longer exist; a rule that balanced across its targets is a rule pointing at a pool.