Managed Model Endpoints

Managed model endpoints

A managed model endpoint is a model service the daemon owns: you register it once, then every compute_provider cell against it just works — the daemon swaps the resident model off the device (one model at a time, via the resident's own approved stop), runs your approved start script, waits for the readiness route, then runs your cell, streaming its lifecycle progress into the cell as it goes. You never run the container runtime yourself, never poll readiness in cells, and never see the credential value. Two verbs: register() (asks the user once) and ordinary inference cells.

Container specifics — image, registry login, internal port, cache mount target, readiness route — come from the model's own runbook skill; this skill is the translation contract.

Calling a registered endpoint — inference cells

Calling a registered endpoint — use the using-model-endpoint skill (this skill is the REGISTRATION contract; that one documents the call side in full).

The ONLY dispatch form is the compute_provider tool with the endpoint's registered name (list_compute shows them):

compute_provider(provider="boltz2-service", code="""
import requests
r = requests.post(BASE_URL + "/v1/infer", json=payload)
""")

The daemon brings the model up on demand (a first cold start downloads image + weights — minutes; let it run) and preloads BASE_URL into the cell — both as a Python variable (use it directly, as above) and as os.environ["BASE_URL"] (plus INFER_API_KEY for remote endpoints). Endpoints are not kernel environments: environment="boltz2-service" on a plain python cell fails — plain cells get no BASE_URL.

Enablement — once per machine

The user connects the family under Customize → Compute → Model endpoints (the setup flow saves the family credential first — connect-without-key is not a state) and picks ONE mode: Local (container registrations) or URL/remote (https against the configured host). Until connected, free_port()/register() raise a precise error — relay it; in the wrong mode they refuse with a teaching error naming the setting (existing endpoints of the unarmed leg keep dispatching — only NEW registrations refuse). Disconnecting is a full teardown: every active local service is stopped via its approved stop script and every registration (local AND hosted) is removed; caches stay on disk; a failing stop keeps that one row, FAILED. The "Local machine GPU" toggle never gates registration — it governs cell GPU access only; the approval card is the per-registration gate.

Credential contract (platform rule): every registration passes credential="NVIDIA_API_KEY" — the daemon rejects any other name. Locally the value feeds the start script's registry login and never enters your kernel env; for remote endpoints it authenticates the upstream and is delivered only into the inference cell's env (as INFER_API_KEY), never the repl kernel.

Register (repl kernel)

port = host.model_endpoints.free_port()        # local only; random 20000-29999

host.model_endpoints.register(
    name="boltz2-service",            # <model>-service -- descriptive, never the
                                      # bare model name (collides/ambiguous)
    url=f"http://127.0.0.1:{port}",   # LITERAL 127.0.0.1 -- `localhost` rejected
    credential="NVIDIA_API_KEY",      # the family credential NAME, never a value
    skill="<model-runbook-skill>",
    start=START_SCRIPT,               # composed below
    stop="docker stop boltz2-service",# exit 0 ONLY once actually stopped
    live="/v1/health/ready",          # readiness ROUTE (200 = model answers;
                                      # "up but loading" must read not-ready)
)

Name endpoints <model>-service (e.g. diffdock-service) — unambiguous in provider lists; never just the bare model name. Name the CONTAINER after the endpoint too (the template above does): the UI then follows the service's own logs live while it starts.

register() always cards the user (scripts verbatim, port, service dir, credential name). One exception: a byte-identical re-registration is silent — same bytes are approved forever; any byte change re-cards. The registration stays inspectable under Customize → Compute. Re-registering to fix scripts: reuse the existing url — never call free_port() again (the port is the endpoint's stable mutex).

Remote endpoints — upstream APIs (no lifecycle)

Pass url="https://<upstream>" and omit start/stop/live — no port, no scripts, no readiness. Requires URL/remote mode (the setup radio; in Local mode https registrations refuse). The url's HOST must equal the configured upstream host exactly — you pick the path leaf, never the authority. After approval, cells are plain HTTP clients of BASE_URL authenticating with $INFER_API_KEY. list_compute labels every row location: "local" | "remote".