Inference Autopilot

Inference Autopilot

Run an evidence-driven optimization loop. Treat this skill as the control plane and the bundled scripts as deterministic utilities. Default to local, private, dry-run behavior.

Start Here

For a new private single-host deployment, start with the one-shot interface. The user supplies only local paths, workload, SLO, objective, budget, and optional GPU visibility. The script discovers NVIDIA or AMD GPU memory and topology, reads the local model config and weight sizes, regenerates the current SGLang parameter contract from server_args.py and sglang.launch_server --help, captures a bounded Nsight Systems baseline, routes trace and workload evidence to parameter families, runs bounded screening, applies mode-specific confirmation, and emits a deployable command only after the confirmation gates pass:

inferopt init --output /absolute/private/task.json
inferopt doctor --task /absolute/private/task.json --output /absolute/private/doctor.json
inferopt plan --task /absolute/private/task.json --output /absolute/private/plan.json
inferopt run --task /absolute/private/task.json --yes --output /absolute/private/final.json
inferopt report --result /absolute/private/final.json --output /absolute/private/report.md

inferopt run only detects and reports missing fused MoE tuning configs. Never run the high-cost kernel search as part of the normal workflow. If the user explicitly requests it after reviewing the report, use the standalone command emitted by the report:

inferopt tune-moe --task RUN_DIR/task.json --profile RUN_DIR/profile/nsys-diagnosis.json --result RUN_DIR/final.json --output-dir RUN_DIR/optional-fused-moe-tuning --yes --output RUN_DIR/optional-fused-moe-tuning.json

Treat generated configs as non-deployable until a separate end-to-end A/B benchmark passes the original workload, SLO, noise, and confirmation gates. Generated files must remain under configs/triton_<version>/ and use the installed SGLang naming contract (E, per-TP N, runtime GPU name, dtype, optional block shape/per-channel marker) with a JSON mapping from token batch size M to kernel tile parameters. If the log requests an _down.json file, the normal tuner is only partial: use SGLang's separate top-k capture/tuner workflow and pass --topk-ids-dir. Inference Autopilot refuses to authorize an up-only result in that case; both matching up and _down files are required.

For direct generation of paired files without end-to-end validation, use:

inferopt generate-moe-config \
  --repository /path/to/sglang \
  --model-path /path/to/model \
  --tp-size 4 --dtype fp8_w8a8 \
  --topk-ids-dir /path/to/topk_ids \
  --output-dir /path/to/moe-config-output --yes

This invokes the official separate tuner and refuses to create a fake _down.json without routing captures. Use --mode standard only when SGLang did not request a down-kernel config.

The standalone inferopt command does not require Codex. doctor never starts a server: it validates the local model, GPU/runtime, current SGLang CLI, profiler availability, and a transparent single-GPU memory estimate. It may recommend a quantized checkpoint class when the current model cannot fit. It also reports legal TP sizes across the complete selected single-host GPU set and treats a missing nsys installation as a blocking preflight error. It never downloads, switches, or approves a model variant without an explicit quality evaluation.

The underlying scripts remain available for controlled integration:

cp assets/task.autopilot.example.json /absolute/private/task.json
python3 scripts/autopilot.py validate --task /absolute/private/task.json
python3 scripts/autopilot.py plan --task /absolute/private/task.json --output /absolute/private/plan.json
python3 scripts/autopilot.py run --task /absolute/private/task.json --yes --output /absolute/private/final-pointer.json

Read references/hardware-profiles.json for the official-source GPU capability catalog. Runtime discovery overrides catalog values. Preserve the checked-out SGLang version's automatic defaults as the baseline; never transplant a winning parameter from one GPU or workload to another without measurement.

1. Locate the inference repository and read its local instructions.
2. Create a task specification using references/input-schema.md.
3. Run scripts/inferopt.py validate --spec <task.json>.
4. Run scripts/inferopt.py inventory --output <run-dir>/inventory.json on every target host when access is available.
5. Inspect the generated inventory and task constraints before launching anything.
6. Follow the optimization loop below.

For authorized single-host execution, copy assets/task.execute.example.json, read references/execution-schema.md, validate it, render the exact command plan, obtain explicit approval, and only then run:

python3 scripts/autotune.py validate --spec task.json
python3 scripts/autotune.py plan --spec task.json --output plan.json
python3 scripts/autotune.py run --spec task.json --yes --output final-pointer.json
python3 scripts/autotune.py report --run-dir /absolute/completed-run --output decision.json

If the framework is SGLang, read references/sglang-adapter.md. For another engine, discover its actual launch, benchmark, metrics, and profiling interfaces instead of assuming SGLang flags.

Safety Contract

• Default to dry_run. Require the user or task specification to opt into execution.
• Run only on hosts, endpoints, repositories, models, datasets, and output directories in scope.
• Never modify a production deployment, autoscaler, traffic router, cloud resource, driver, system service, or shared cluster without explicit authorization.
• Never download a model or dependency, reserve paid hardware, publish results, or send traces externally without explicit authorization.
• Never kill processes not started by the current experiment. Record owned PIDs and terminate only those PIDs.
• Never use broad process-kill commands as cleanup.
• Redact prompts, API keys, tokens, user identifiers, and proprietary model paths from reports.
• Enforce trial, wall-time, GPU-hour, and failure budgets. Stop immediately on correctness failure, repeated OOM, thermal/power anomaly, or error-rate violation.
• Keep baseline artifacts immutable. Store every candidate in a distinct trial directory.
• Do not accept a faster candidate unless correctness, stability, and SLO gates pass.
• Treat --yes as confirmation that the generated command plan was reviewed; never add it before approval.

Read references/safety-policy.md before executing benchmarks, profilers, or code changes.

Optimization Loop

Deployment Modes

Set deployment_mode in the autopilot task. Use online_latency for an interactive service: maximize the selected objective while every declared E2E, TTFT, TPOT, or ITL gate passes. When no latency SLO is declared, the configured secondary-regression limit still protects observed latency. Use offline_throughput for batch inference: maximize sustained aggregate throughput at calibrated batch pressure; latency remains recorded but is only a gate when the task explicitly declares it.

For offline workloads without SLOs, skip capacity calibration, remove the client concurrency cap, and let the resolved SGLang admission policy determine sustained pressure. Start the baseline service exactly twice: one bounded nsys capture and one unprofiled benchmark. Preserve that benchmark as an immutable reference for all candidates; never search or override max_running_requests. For an SLO-constrained workload, control load with the benchmark client's max_concurrency; treat the server's resolved max_running_requests as diagnostic evidence rather than a tuning parameter. Persist every benchmark command and resolved server value. For a model with a verified official cookbook, first compare complete, locally valid capability bundles: include the relevant model feature (for example MTP), prefix/KV cache policy, scheduler/admission, memory pool, CUDA Graph, and MoE backend variants when the workload and hardware make them applicable. Profile the fastest SLO-valid initial configuration, even if its single screening sample has not yet cleared the final improvement threshold. Then use Nsight evidence to select and refine parameter families, and screen and confirm every candidate against the original target workload. Never label a result as global best: report it as the best configuration within the checked-out SGLang parameter contract, tested search space, hardware, workload, and SLO gates. Use search_depth: thorough (the default) to add a one-factor sensitivity screen for every high-impact compatible family even when a short trace lacks a single hotspot. Use evidence_guided only when experiment budget is tight. Every run emits search-plan.json.parameter_audit, which accounts for each CLI-visible, non-deprecated ServerArgs as selected, excluded, or inapplicable with a concrete reason. It prevents a short candidate list from being mistaken for the whole startup-parameter surface. Render per-stage trial progress as an elapsed-time ASCII bar with completed and planned counts. Count completed, failed, and capability-skipped trials as processed. Do not invent a percentage for Nsight capture or model startup when their duration is not knowable in advance, and keep redirected logs free of terminal control characters. After the one-factor screen, the optimizer uses the remaining trial budget to test explicit combinations of independent screened winners. Every candidate that clears the configured improvement threshold is considered: pairs with the strongest candidate run first, followed by other pairs and larger combinations while budget remains. Parameter/environment conflicts and exact duplicates are excluded. Positive sub-threshold candidates may use leftover slots but never displace an above-threshold combination. Offline no-SLO runs reuse the preserved baseline and rerun only the selected candidate once; SLO-constrained runs retain repeated A/B confirmation. It does not perform an unbounded Cartesian product.