Tao Launch Workflow

TAO Workflow Launch Intake

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

Use this skill before launching any TAO workflow or model action.

Quick Start

Run the platform helper, ask for platform and monitoring preferences, then run the selected platform detail helper before asking for credentials.

Non-Negotiable Launch Gate

This gate is model-agnostic. Apply it to every TAO model, data action, and application workflow before launching side-effecting work.

Do not create runner scripts, launch scripts, compatibility shims, workspace folders, state files, logs, or dependency-install side effects until the launch preflight passes.

Preflight passes only after all of these are true:

1. The execution platform is selected from the packaged platform helper.
2. Platform credentials and required credential groups are satisfied.
3. Model-specific credentials are satisfied.
4. The default container image is resolved from packaged model/action metadata, shown to the user, and either confirmed or replaced by an explicit image=<override>.
5. The platform access check succeeds from the launch host.
6. Dataset inputs are mapped to concrete spec keys and verified from the selected platform's point of view.
7. Required compute shape fields from the model/workflow skill are known.
8. Required local tools for the selected data/platform path are present, or the user approved installing the smallest missing dependency and preflight was rerun.
9. A launch review with image, platform, datasets, compute shape, expected runtime, and any generated/default configuration changes has been shown and confirmed by the user. For AutoML, the launch review must explicitly state recommendation count/budget, max concurrency, algorithm, metric, direction, and searched parameters/ranges even when defaults are used.

If any item is missing, ask for the missing input and stop before generating artifacts. This applies to AutoML, normal train/eval/infer/export/TRT, and DEFT/application workflows.

When preflight work clears a blocker, keep track of the original user request. After the fix, rerun the relevant preflight and continue toward that request; do not stop at "blocker fixed" unless the user explicitly asked only for the repair.

The Four-Verb Execution Contract

Once the launch gate passes and the producing model/data skill has authored the spec-bundle (schema: tao-artifacts), execution is exactly four verbs. Every platform skill implements them over its native CLI — the bank ships five (tao-run-on-docker, -slurm, -kubernetes, -brev, -virtualenv), and any externally installed platform skill joins the same contract (§ External platform skills); nothing else is platform-specific. $BANK = ${TAO_SKILL_BANK_PATH}.

• submit(spec-bundle) — resolve the data question first: if the inputs are already readable from the compute frame (a local path, an existing mount — tier A in place, the common local and the only air-gapped case), there is nothing to stage — record tier A and move on. Invoke tao-data-io only on a frame mismatch (remote URIs, cross-host paths, PTM fetches, tier-C result uploads). Then lint the assembled command with redact_secrets.py lint and open the record and launch, in that order:

  JOB_ID=$("$BANK/scripts/tao_job_record.py" open --platform <p> --image <img> \
    --network-arch <arch> --action <action> --storage-tier <A|B|C> --results-root <root>)
  # <native launch, naming the backend object after $JOB_ID>
  "$BANK/scripts/tao_job_record.py" mark "$JOB_ID" --state RUNNING --backend-ref <ref>

• status(id) — poll the native backend, map to the fixed vocabulary PENDING RUNNING COMPLETE ERROR CANCELED UNKNOWN; the native sub-state (ImagePullBackOff, PENDING-resources, slurm COMPLETING) rides in the transition message. Never read "what's running" from records — poll the backend.
• logs(id, tail) — native log fetch.
• cancel(id) — native cancel + orphan teardown, then mark <id> --state CANCELED.

Record-then-launch is the ordering invariant. open mints the id and binds results_dir before any launch, and the id it returns is the only handle the launch can use — a submit that skipped the gate or the open has no id, so it cannot launch. This is what keeps a run recoverable across a context break: results_dir is recorded before the backend object (which K8s TTL or docker --rm may later delete) ever exists.

When the producing spec-bundle declares execution, preserve it as model-owned action semantics across every application that reuses that model skill. The selected platform consumes the lifecycle; an application must not copy its commands into a private launcher. Platform-independent pre/post commands, runtime attestations, helper dependencies, distributed intent, and completion evidence belong in the producer's spec-bundle. Scheduler syntax, mounts, secrets, timeouts, ranks, and child-exit preservation remain platform-owned.

External platform skills

No registry, no interface file: a platform skill declares the contract by documenting the four verbs, and you verify by reading before first use. A skill with only native primitives may be used by inferring the mapping (bank invariants still bind; the mapping goes in the launch review; persist what worked). Rules and the no-equivalent hard floor: references/external-platforms.md.

Failure analysis & retry

When status reaches ERROR, read the log tail and classify before any retry — infrastructure faults are retriable (new record, --retry-of, up to 10), program faults never are. Full criteria, the two judgment calls (device-side asserts, downstream tracebacks), and the post-turn poller rules: references/failure-analysis-retry.md.

Initial Questions

After the user confirms what they want to do, ask which execution platform should run it. Discover the choices from the platform skills installed in this session — you already see them by name and description (tao-run-on-docker, -slurm, -kubernetes, -brev, -virtualenv, plus any externally installed one such as the official brev-cli skill). There is no central platform registry to read. If your runtime surfaces only the core router skills (e.g. Codex), list the bank's platform skills by reading skills/platform/tao-run-on-*/SKILL.md frontmatter (name + one-line description) under ${TAO_SKILL_BANK_PATH}.

Then ask:

• Which supported platform should run this workflow?
• Should I monitor the run in this chat? Monitoring means I keep polling the backend/job logs after launch and report progress until the job finishes, fails, or you ask me to stop, even if the job stays queued for hours or days. If disabled, I launch the job, give you the job id/log path, and stop polling. Default: monitor in chat.
• How often should I post status? Default: every 5 minutes. Use 1-2 minutes for smoke tests, 5 minutes for normal training, or 10-15 minutes for long runs.

Use long_running_enabled=true and status_interval_minutes=5 when the user accepts the defaults.

When monitoring is enabled, do not send a final summary just because several polls have elapsed or the job is still PENDING. Keep the turn attached and emit status every status_interval_minutes until a terminal state or explicit user stop/detach request. If the runtime environment cannot keep the chat turn open, say that clearly and leave a durable watcher/log path; do not imply that chat updates will continue after the turn ends.

Final-answer rule: a final response ends chat-side monitoring. While long_running_enabled=true and any launched job is non-terminal, status messages must be sent as in-progress updates and the agent must continue polling. Only send a final response when the workflow reaches terminal state, the user explicitly asks to detach/stop monitoring, or the runtime genuinely cannot keep the turn open; in that last case, say it is a runtime limitation and provide the exact durable status command/log path.

Missing-Input Prompt Shape

When intake inputs are missing, ask with the exact prompt shape in references/intake-prompts.md (one consolidated ask, concrete examples, no invented defaults).

Implementation Backend Resolution

After model ownership resolution, inspect the selected model's references/skill_info.yaml. If it declares backend_contracts, resolve the implementation before selecting an image or authoring a spec. An explicit backend wins when it supports the model/action; otherwise apply the packaged backend_selection policy and show its rationale. The selected backend metadata in skill_info.yaml owns its image. The referenced backend contract owns the entrypoint, configuration schema, data mappings, topology, checkpoint format, output layout, and status behavior. Never use a legacy top-level image fallback for a multi-backend frontend, and never treat one backend as a version of another.

Pass action, backend, and workload hints to the model resolver. When metadata declares a backend planner, use it. The shared Cosmos frontend, for example, uses scripts/cosmos_workflow.py plan to generate backend-native TOML and a launch sequence.

Container Image Confirmation

Before creating specs, runner scripts, workspaces, logs, state files, or submitting a job, resolve the image for the selected model/action:

${TAO_SKILL_BANK_PATH:-~/tao-skill-bank}/scripts/resolve_tao_image.py \
  --skill-bank ${TAO_SKILL_BANK_PATH:-~/tao-skill-bank} \
  --model <network> --action <action> --backend <auto-or-explicit> \
  --workload <workload-hint> --format text

If the helper is unavailable, read skills/models/<network>/config.json directly. Resolve image fields in this order:

1. backend_contracts.<selected-backend>.container_image, when present
2. actions.<action>.container_image
3. actions.<action>.image
4. top-level container_image
5. top-level image

Show the exact image and ask:

Container image for <network>/<action>:
default=<resolved image>

Use this image, or provide image=<override>?

If the user accepts, pass the resolved image as the job image. If the user overrides, require a non-empty image reference and pass that value instead. Do not silently launch on the default image. This confirmation applies to training, AutoML recommendations, evaluation, inference, export, TensorRT engine generation, and application workflows that submit TAO containers.