Coral New Task

Creating a new CORAL task

A CORAL task is three things that must line up:

examples/<task>/
├── task.yaml      # config: name, description, grader entrypoint, agent count
├── seed/          # starter code agents see when they begin (the repo_path)
│   └── solution.py
└── grader/        # standalone Python package
    ├── pyproject.toml
    └── src/<task>_grader/
        ├── __init__.py
        └── grader.py     # class Grader(TaskGrader): ...

The packaged form is the only supported form. The package gives the grader its own venv and ships everything the eval needs — grader code, helper modules, and hidden data (see "Hidden data" below).

Reference implementations

Look at these before writing anything new — copy the closest one and edit:

1. The seed

Whatever lives in seed/ is what the agent sees on first checkout — it's the working directory the grader will later score. The contract between seed/ and the grader is the program file: a Python file with a function the grader imports and calls.

The convention across examples is:

• solution.py (or initial_program.py) defining a top-level run() function.
• The grader passes program_file: "solution.py" via grader.args.
• run()'s signature is whatever the grader expects — usually () -> result or (input_path) -> result.

Put a real, runnable baseline here. Agents should be able to coral eval immediately and get a non-zero score, so they have a starting point to improve. A no-op skeleton that crashes is not a good baseline.

If the task needs data files at runtime (training data, fixtures), put them under seed/data/ and reference them by relative path from solution.py. The grader will see them at <codebase_path>/data/....

2. The grader

Packaged grader — the recommended path

grader/
├── pyproject.toml
└── src/<task>_grader/
    ├── __init__.py
    └── grader.py

pyproject.toml is a thin Hatchling package. Crib from examples/erdos/grader/pyproject.toml:

[project]
name = "<task>-grader"
version = "0.1.0"
description = "CORAL grader for the <task> task."
requires-python = ">=3.11"
dependencies = ["coral", "numpy"]   # Whatever the grader actually imports.

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["src/<task>_grader"]

Subclass TaskGrader and implement evaluate():

# grader/src/<task>_grader/grader.py
from coral.grader import TaskGrader
from coral.types import ScoreBundle


class Grader(TaskGrader):
    def evaluate(self) -> float | ScoreBundle:
        program_file = self.args.get("program_file", "solution.py")
        # self.codebase_path  — the agent's commit checked out detached
        # self.private_dir    — .coral/private/ (your hidden answer keys live here)
        # self.args           — dict from task.yaml grader.args
        # self.timeout        — grader.timeout in seconds (or None)
        # self.eval_logs_dir  — write subprocess logs / artifacts the agent should see post-grade

        try:
            result = run_program_and_score(...)
        except TimeoutError:
            return self.fail(f"Evaluation timed out after {self.timeout}s")
        except Exception as e:
            return self.fail(f"Evaluation failed: {e}")

        return self.score(result, explanation=f"score={result:.4f}")

What you have available on self:

Bundling data files with the grader

If the grader needs reference files (model weights, ground-truth answers, scoring fixtures), ship them inside the package and load via importlib.resources:

import importlib.resources

scorer_dir = str(importlib.resources.files("<task>_grader.scorers"))

examples/dna_design/grader/src/dna_design_grader/grader.py is the canonical pattern — note the scorers/ subpackage. Add the directory to [tool.hatch.build.targets.wheel] if it has non-Python files.

Heavy / optional dependencies

If the grader wants torch, grelu, etc., put them in optional-dependencies and have the grader fall back gracefully when missing — see examples/dna_design/grader/pyproject.toml. Then grader.setup becomes ["uv pip install -e ./grader[ml]"] for the full version.

Hidden data: put secrets under grader.private

Answer keys, hidden test fixtures, and anything the agent must not see go under grader.private in task.yaml — CORAL copies those paths into .coral/private/<name> (which every agent runtime is denied read access to) and the grader reads them via self.private_dir:

grader:
  private:
    - "taskdata"   # a sibling of grader/ → .coral/private/taskdata, read via Path(self.private_dir) / "taskdata"

The single rule: everything inside the grader/ package is visible to agents, and secrets live in grader.private outside grader/. The whole grader source is surfaced read-only to agents at <shared_dir>/grader/ (a symlink to the real package) so they can read how they're scored — so anything under grader/ is readable, including a grader.private path that points inside it (it gets copied to .coral/private/ and leaked through the surfaced source). Keep taskdata/ as a sibling of grader/ (declare it taskdata, resolving to <task_dir>/taskdata); coral validate errors if a private path is inside the package.

Non-secret bundled data (lookup tables, reference configs, helper modules) may live inside grader/ and be read via Path(__file__).parent / ... — just remember it's visible, so never put a secret there. E.g. examples/ADRS/txn_scheduling/ puts helper modules on sys.path that way.