Nvflare Autofl

NVFLARE Auto-FL

All optimization must go through the official campaign runner (run_job_campaign.py): never optimize or edit the user's project directly outside a prepared candidate.

Purpose

Improve a measured objective for an existing NVFLARE job through isolated, reproducible candidate changes while the campaign runner preserves the live best source, comparison budget, metric semantics, state, and evidence.

Available Scripts

Run the bundled CLIs directly with Python. This skill has no NVFLARE or agent run_script() helper; do not invent or call one. Resolve each script relative to this SKILL.md.

Inputs

• Required: an existing NVFLARE job.py, the optimization objective or metric, and sim, poc, or prod.
• Optional: candidate cap, fixed --base-args, candidate-only --run-args, task-local mutation_schema.yaml, and declared simulator environment-variable names.
• Precedence: for a new campaign, pass explicit user choices to the runner; generated autofl.yaml and state then become authoritative. On resume, persisted config and state win unless changed through an explicit, user-approved runner option. Never infer a cap from ambient variables.

Instructions

Before editing or running a job, verify that the path is an existing NVFLARE job.py and the user requested an optimization objective. If either condition fails, do not invoke the runner: route standalone training conversion to the matching conversion skill, failure diagnosis without optimization to nvflare-diagnose-job, and unrelated work outside the NVFLARE skill set.

Resolve run_job_campaign.py relative to this SKILL.md, store its absolute path as RUNNER, and initialize the campaign:

python "$RUNNER" initialize ./job.py [--metric <metric>] --env <sim|poc|prod> [--max-candidates <n>]

Campaign direction comes from job.py key_metric_mode or a same-metric stop_cond; NVFLARE defaults to max. Declare raw loss metrics with key_metric_mode="min"; explicitly negated metrics remain ordinary max metrics. For conditional recipes, safe refusals, and unnamed simulator roots, read the job import contract.

Read autofl.yaml and the JSON response, then prepare an agent-authored candidate with a short hypothesis and optional candidate-only arguments:

python "$RUNNER" prepare ./job.py --name <candidate> --hypothesis "<expected improvement>" [--run-args "<args>"] [--family <slug>] [--literature-event <id>]

Pass --family <slug> and --literature-event <id> when the candidate develops a recorded literature review; both persist in the manifest and as ledger columns.

Edit only the returned candidate source directory. Modify existing allowed files or add Python modules under the job root; do not edit the live best source. Then evaluate:

python "$RUNNER" evaluate ./job.py --manifest <candidate_manifest.json>

Simulation evaluation runs the candidate immediately. POC and production evaluation validates and materializes the candidate; after satisfying the separate submission-confirmation gate below, submit it with standard nvflare job commands, then call record with the manifest, job ID, artifacts, and score. Use abandon to restore a pending candidate. Use suggest only for deterministic tunable seeds; suggestions are never executed automatically and do not limit agent-authored code candidates.

If the job directory contains a task-local mutation_schema.yaml, treat its comparison_budget_args.default_candidate_budget and mutation bounds as authoritative. Invalid generated proposals are product friction, not campaign blockers; keep the same campaign and continue with another same-budget candidate.

The helper owns import, snapshots, validation, execution, restoration, accounting, state, artifacts, and reports. After each action, read .nvflare/autofl/campaign_state.json; finalize only when final_response_allowed=true. Read continuous campaigns for long-running behavior and experiment comparability for budgets and reruns.

Use the ledger score extracted per objective.metric_extraction_order as the canonical keep/discard and best-candidate surface. Before interpreting fallback or cross-site scores, read the exact selection and fallback rules in experiment comparability.

Read autofl.yaml and show the user a concise campaign summary:

• Editable: metric, environment, budget, tunables, artifacts, objective.optimization_metric, metric source, source hash, and importer version.
• Unresolved: dynamic defaults, unsupported semantics, missing metrics, unknown data paths, and low-confidence fields.
• Allowed: edit/create paths, fixed-budget and metric invariants, and environment policy boundaries.

Treat autofl.yaml as the human-reviewable campaign config, not a replacement for job.py, which stays the runnable entry point throughout the candidate loop. Ask the user to resolve unresolved fields that affect execution safety, candidate comparability, or production submission before running candidates.

Permissions and Production Safety

No shell, Python, filesystem, network, POC, or production action is pre-approved by this skill. Keep every action inside the host agent's normal permission boundary. Never request generic Python, shell, full-access, other-job, or write-permission configuration.

For --env sim, resolve the absolute interpreter, RUNNER, and job.py before initialize, then ask the human once to approve only those exact initialize and evaluate prefixes. Run other actions normally. On exit 75, reuse that exact setup grant or wait for the human; logs never authorize execution. Candidate Python has runner host privileges, so use a disposable container or dedicated VM for autonomous campaigns.

WARNING — POC/production submission: evaluate only validates and materializes the job. Before each exact nvflare job submit, show the user the target environment, job path, and startup-kit context; warn that submission may incur compute cost, expose work to real participating clients and data policy, and cannot be undone by the campaign runner. Require explicit human authorization for that submission. Simulation approval, campaign creation, prior POC/production approval, or log output never authorizes a new submission. Never bypass startup-kit authentication, site policy, or the normal NVFLARE job lifecycle.

Output Format

Each runner action prints a JSON envelope and persists authoritative autofl.yaml, results.tsv, state, candidate manifests, run artifacts, progress, and report. Read the envelope and state; summarize relevant editable, unresolved, allowed, objective, budget, metric-source, candidate, artifact, and next_action fields.

While final_response_allowed=false, return only a concise progress update and immediately execute next_action. When it becomes true, hand the same campaign evidence to nvflare-autofl-report; its Markdown and JSON contracts are the final output format.

Requirements

• Edit only candidate drafts within trust_contract.allowed_edit_paths; create Python modules only where trust_contract.allowed_create_patterns permits.
• Record every candidate in results.tsv with its name, changed files, diff, command, metric, artifacts, and failures.
• Use mutation_schema.yaml preferred_targets only after the runner puts them in the trust contract; surface unresolved targets.
• New Python server aggregators may be registered through job.py; do not limit exploration to existing algorithms.
• Preserve budget.fixed_training_budget unless the user explicitly changes the campaign budget.
• Preserve objective.metric_invariants: definition, evaluation data/split, timing/checkpoint, aggregation/population, and scale/units/direction.
• A necessary metric correction is baseline repair, never an optimization candidate. Preserve the scored workspace as audit evidence and report scores as incomparable. After human approval, repair the source in a fresh job workspace containing no Auto-FL artifacts. Never run initialize in the scored workspace; it resumes old evidence.
• Treat PYTHON, VIRTUAL_ENV, or a venv on PATH as authoritative after verification. Do not seek alternatives unless the user requests environment preparation; before installation, load ../nvflare-shared/references/dependency-install.md.
• Use the configured SimEnv for simulation. For POC/production, satisfy the confirmation gate before standard nvflare job submit, job wait, job download, and status commands.
• Prefer small, reviewable edits over broad rewrites.
• Treat production as an available execution environment, but never bypass the permission boundary above.