Llm4ad Task Builder

Building an LLM4AD_Next Task Package

What LLM4AD_Next is

LLM4AD_Next is an automated algorithm-design platform: it combines an LLM with evolutionary optimization to discover and improve algorithms automatically. The user describes a problem (e.g. "evolve a solver for the Traveling Salesman Problem"), marks the code region to evolve with # EVOLVE_START / # EVOLVE_END, and the platform repeatedly asks an LLM to propose better code inside that region, scores each candidate with a custom evaluator, and keeps the best across generations. Better algorithms emerge through guided search rather than manual trial and error.

Your job with this skill: turn a user's problem into a complete, runnable task package, then verify it actually runs before handing it over.

AutoDiscovery runtime mode

When this Skill is installed together with algorithm-discovery, first help that Skill resolve the source-grounded function boundary, I/O contract, objective, constraints, evaluator data, and reproducibility requirements. Then call the runtime's build_algorithm_task tool with one complete description. Do not hand-write package files and do not leave package construction for project management: the tool invokes this project's official builder and validation pipeline and returns the exact runnable package to publish.

Only a package whose returned validation status is passed may be published. Project management imports that same package and starts no second requirements or build conversation. Evolution itself still starts only when the user requests it.

What a task package is

A task package is a self-contained directory with everything the platform needs to evolve algorithms for one problem:

The contracts each file must satisfy

Algorithm file (<algo_dir>/<algo>.py):

• The algorithm directory is the one named in version_control.local_path; the platform copies it into a git worktree for each candidate and edits only the code between the markers.
• The function to evolve is wrapped exactly in # EVOLVE_START and # EVOLVE_END comment markers (the platform only edits code between them).
• It reads its input as a single JSON string from sys.argv[1], and prints a JSON result to stdout — so the evaluator can run it as a subprocess.
• The file name must match the name the evaluator looks for (see below). In the TSP example both sides use solve.py.

Evaluator (<name>_evaluator.py):

• Before writing evaluator code, read reference/api-contract.md for the complete BaseEvaluator contract and common pitfalls.
• Subclasses BaseEvaluator, decorated @BaseEvaluator.register("<name>"), with a no-argument __init__(self) (the base __init__ takes no config).
• Implements metrics (list of Metric(name, type=MetricType.MINIMIZE, weight, description) — note MetricType.MINIMIZE, not bare MINIMIZE), name, and async def evaluate(self, cfg: EvalContext) -> EvaluationResult.
• cfg: EvalContext carries cfg.project_root (the candidate's worktree dir), cfg.data_path (the current data file), and cfg.timeout (seconds per instance).
• evaluate locates the algorithm file by hardcoded name under cfg.project_root (e.g. Path(cfg.project_root) / "solve.py"), runs it as a subprocess on cfg.data_path, parses its JSON output, validates it, and returns EvaluationResult(score, metrics, success, ...) (negative cost for a minimization objective). That hardcoded name must equal the algorithm file's actual name — this is the most common wiring mistake.

config.yaml (10 sections — generated programmatically, not hand-written):

• providers use ${LLM_BASE_URL} / ${LLM_API_KEY} / ${LLM_MODEL} env placeholders (the platform fills them in).
• The evaluator module reference (<file>.py:<ClassName>), the metrics list, the dataset path (data/sample), and the coder prompt_template's EVOLVE block must all be consistent with the other files.

debug_run.py / test_evaluator.py: standard boilerplate that runs config.yaml end-to-end and loads the evaluator, respectively.

See reference/api-contract.md for the BaseEvaluator contract and reference/templates/ for two complete, copyable template packages demonstrating different problem patterns (see reference/templates/README.md).

The information to gather before building

A correct package needs these decided (ask the user; let the agent fill sensible defaults where the user has no preference):

1. Problem description — what is being optimized. (Required — ask.)
2. Function/algorithm to evolve + input/output format — ask if the user has any constraints; if not, the agent designs it.
3. Evaluation metric(s) — what to minimize/maximize. (Required — ask clearly.)
4. Reuse existing code/evaluator? — user provides existing code, or generate from scratch. (Ask.)
5. Data source — user provides data, or generate sample instances. (Ask.)
6. Programming language — default Python. (Offer as a choice.)
7. Project name — the agent proposes one from the description; confirm with user.

Build workflow

1. Understand the problem and the 7 items above (inspect any user-provided code/data).
2. Write the algorithm file first; verify it runs standalone and prints valid JSON.
3. Write 2-3 sample data files under data/sample/.
4. Write the evaluator; keep its metrics consistent with config.yaml.
5. Write config.yaml, debug_run.py, test_evaluator.py.
6. Self-test (required): run test_evaluator.py (evaluator loads — no LLM, no network, cheap and always runnable). Then run debug_run.py, which executes the full pipeline and does call the LLM: it needs the LLM_BASE_URL / LLM_API_KEY / LLM_MODEL env vars set (the config uses these placeholders) and consumes tokens over the network. Read every error, fix the relevant file, re-run. Not done until both run cleanly. If no provider credentials are available, say so — a missing-credential failure of debug_run.py is not a package defect.