Agent skill

Creating A Coral Task

by Human-Agent-Society in Human-Agent-Society/CORAL

Author a new CORAL task — the three pieces that must line up (task.yaml, seed/, a packaged grader/), the coral init → coral validate → smoke-test loop, and how to pick a grader pattern (stdout…

Apache-2.0Auto-check passedTesting & QA

Install Creating A Coral Task

skills CLI
$ npx skills add Human-Agent-Society/CORAL --skill creating-a-coral-task -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install Human-Agent-Society/CORAL creating-a-coral-task --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/Human-Agent-Society/CORAL.git skills-src && mkdir -p .claude/skills && cp -r skills-src/plugin/skills/creating-a-coral-task .claude/skills/creating-a-coral-task && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
creating-a-coral-task
GitHub stars
1.1k
Token cost
~2.2k tokens
SKILL.md length
895 words
Files
5 (incl. references)
Skills in repo
7
Repo updated
First seen
Licence
Apache-2.0

At a glance

Author a new CORAL task — the three pieces that must line up (task.yaml, seed/, a packaged grader/), the coral init → coral validate → smoke-test loop, and how to pick a grader pattern (stdout…

  • The user wants to create a CORAL task
  • SKILL.md covers The loop, The three pieces, Hidden data and Smoke-test, then scale, plus 1 more section
  • Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md
  • Port a benchmark into CORAL

What it does

Creating A Coral Task is an agent skill from Human-Agent-Society/CORAL. Author a new CORAL task — the three pieces that must line up (task.yaml, seed/, a packaged grader/), the coral init → coral validate → smoke-test loop, and how to pick a grader pattern (stdout float, test pass-rate, ratio-vs-baseline, multi-metric, or an LLM rubric judge). Use whenever the user wants to create a CORAL task, write or wire a grader, port a benchmark into CORAL, score open-ended outputs (reports/memos) with a judge, or debug a grader that crashes on the seed / ranks the leaderboard backwards / leaks…

Its SKILL.md is about 2.2k tokens, which your agent loads only when the skill is triggered. The skill folder holds 5 other files, including reference files (for example `references/cookbook.md`, `references/grader-api.md` and `references/rubric-judges.md`).

It sits in Testing & QA, covering QA and bug reports and Quizzes and assessments. The repository describes itself as: Open-source autoresearch powered by autonomous coding agents. Run Claude Code, OpenCode, and Codex with grading, shared knowledge, and multi-agent evolution. Accepted at COLM 2026. The licence is Apache-2.0.

When your agent uses it

  • The user wants to create a CORAL task
  • Port a benchmark into CORAL
  • Score open-ended outputs (reports/memos) with a judge
  • Debug a grader that crashes on the seed / ranks the leaderboard backwards / leaks the answer key

Example prompts

  • “/creating-a-coral-task”

Requirements

  • Python 3

What it can do on your machine

Read from SKILL.md and the folder at commit 0123dfb. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    No scripts in the folder and no shell commands in SKILL.md (its code samples are bash and python).

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Links to these hosts (documentation or services it may open):

    • coral.compounding-intelligence.ai

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Creating A Coral Task loads about 2.2k tokens when it runs, and up to ~8.1k if it reads all its reference files. Until then it costs about 174 tokens; SKILL.md has 895 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~174
When it runs · the whole SKILL.md, loaded when a task matches
~2.2k
With references · SKILL.md plus every file in references/, read only if the agent opens them
~8.1k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from Human-Agent-Society/CORAL at commit 0123dfb, republished under its Apache-2.0 licence (© Human-Agent-Society). 895 words, ~2,204 tokens.

Download SKILL.mdSave it as .claude/skills/creating-a-coral-task/SKILL.md (or your agent's skills folder). This skill also uses 4 other files; get the full folder from GitHub.
name
creating-a-coral-task
description
Author a new CORAL task — the three pieces that must line up (`task.yaml`, `seed/`, a packaged `grader/`), the `coral init` → `coral validate` → smoke-test loop, and how to pick a grader pattern (stdout float, test pass-rate, ratio-vs-baseline, multi-metric, or an LLM rubric judge). Use whenever the user wants to create a CORAL task, write or wire a grader, port a benchmark into CORAL, score open-ended outputs (reports/memos) with a judge, or debug a grader that crashes on the seed / ranks the leaderboard backwards / leaks the answer key. Deep references for the TaskGrader API, grader patterns, rubric judges, and the full task.yaml schema live alongside this skill.

Creating a CORAL task

A CORAL task is three things that must line up. Scaffold them with coral init, then iterate edit → coral validate until the grader scores the seed.

my-task/
├── task.yaml      # config: name, description, grader entrypoint, agent count
├── seed/          # starter code agents see at t=0 (this is workspace.repo_path)
│   └── solution.py
└── grader/        # standalone Python package — gets its own isolated venv
    ├── pyproject.toml
    └── src/my_task_grader/
        ├── __init__.py
        └── grader.py     # class Grader(TaskGrader): ...

The packaged grader is the only supported form — it gives the grader an isolated venv and bundles everything the eval needs (grader code, helpers, hidden answer keys). There is no eval/grader.py auto-discovery anymore.

Optimizing code the user already has? Scaffold inside a .coral_workspace/ at the root of their project (gitignored), and copy the code to optimize into seed/ — keeps CORAL's task/results out of their source tree. The coral-quickstart skill has the end-to-end .coral_workspace/ flow; this skill covers the grader you'll write once the code is in seed/.

"Optimize this" is a build instruction, not a question — never answer it with a process menu. A 1/2/3 like "point me to a task / create one / optimize outside coral" is the failure mode; do not produce it. The absence of a task.yaml is not ambiguity — it just means you build one from the current repo. Concretely: (1) dig for what's already measurable — a research/framework repo almost always ships an eval/benchmark script, a test suite, or a metric in its README/paper; that's your target and metric. (2) If no single number is obvious, construct one by wrapping the repo's existing evaluation — don't conclude "no measurable objective" just because there's no CORAL scaffold. (3) Scaffold the most plausible target and start building (a .coral_workspace/ + draft grader is cheap and reversible); state your assumption in one line and proceed. (4) Only as a last resort, if you've actually read the repo and it exposes nothing scorable, propose 2-3 concrete optimization targets you found (each with its metric), pick the most likely, and scaffold that — still not a process menu.

The loop

bash
coral init my-task        # scaffold all three pieces (a runnable end-to-end example)
cd my-task
# ... edit the three pieces for your problem ...
coral validate .          # bootstraps the grader venv, runs the grader on seed/, prints a score
# repeat edit → validate until the seed scores as you expect

coral validate succeeding is the one checkpoint that matters — it proves the grader can score the seed. Most "agents are stuck, every eval fails" reports trace to a grader that crashes on the seed, which validate would have caught. Always start from coral init rather than hand-writing the layout; the generated files are the canonical minimal example.

The three pieces

1. The seed (seed/) — what the agent checks out at t=0 and what the grader later scores. The contract between seed and grader is the program file: a file (e.g. solution.py) with a function or stdout convention the grader invokes, named in grader.args.program_file. Put a real, runnable baseline here — agents should coral eval immediately and get a non-zero score to beat. A skeleton that crashes is a bad baseline. Runtime data goes under seed/data/ and is read by relative path.

2. The grader (grader/) — subclass TaskGrader, implement evaluate(), return a number (or ScoreBundle). The minimum:

python
from coral.grader import TaskGrader


class Grader(TaskGrader):
    def evaluate(self) -> float:
        result = self.run_program(self.args.get("program_file", "solution.py"))
        if result.returncode != 0:
            return self.fail(f"crashed: {result.stderr[:200]}")
        try:
            return float(result.stdout.strip())
        except ValueError:
            return self.fail(f"expected a float, got {result.stdout[:80]!r}")

This stdout-float shape is one of several. Pick the pattern that matches how your task scores → references/cookbook.md:

Score by...Pattern
A number the program printsstdout float
Fraction of hidden tests passingtest pass-rate
Improvement over a baselineratio vs baseline
Several weighted criteriamulti-metric ScoreBundle
An LLM judging a report/memo/docrubric judge → references/rubric-judges.md

Full TaskGrader surface — every attribute (self.codebase_path, self.private_dir, self.args, self.eval_logs_dir, self.tune) and method (run_program, run_script, run_script_json, score, fail, bundle) — is in references/grader-api.md.

3. The task.yaml — wiring. The fields that must be right are grader.entrypoint, grader.direction, and workspace.repo_path: ./seed. Full annotated schema (agents, islands, sharing, gateway, all defaults) → references/task-yaml.md.

Show full SKILL.md (344 more words)Show less

Hidden data

The single rule: everything inside the grader/ package is visible to agents — the whole grader source is surfaced read-only at <shared_dir>/grader/ so they can read how they're scored — so secrets go in grader.private, in a dir outside grader/. CORAL copies those paths into .coral/private/ (which every runtime is denied read access to) and the grader reads them via self.private_dir. Declare a sibling, conventionally taskdata (resolving to <task_dir>/taskdata); a grader.private path inside grader/ would be copied to .coral/private/ and leaked via the surfaced source, so coral validate errors on it. Non-secret bundled data (lookup tables, helper modules) may sit inside grader/ and be read via Path(__file__).parent / ... — it's visible, so never put a secret there. Never put an answer key under seed/ either — agents read seed/ and will game the score.

Smoke-test, then scale

bash
coral start -c task.yaml agents.count=1 run.session=local   # one agent, foreground
# watch for one real eval, confirm the score moves, then:
coral stop

Once one agent evals cleanly, raise agents.count. Driving the run from here is the running-coral-experiments skill.

Common mistakes

MistakeSymptomFix
repo_path points at the task root, not ./seedGrader sees task.yaml/grader/ in codebase_pathPoint repo_path at ./seed.
direction backwardsLeaderboard ordered upside down"ratio, higher better" → maximize; "raw error/latency" → minimize.
Answer key under seed/ or anywhere inside grader/Agents read it and game the score — seed/ is their repo and the whole grader/ source is surfaced at <shared_dir>/grader/Put it under grader.private outside grader/ (sibling taskdata/), read via self.private_dir; coral validate errors on a private path inside grader/.
Grader writes under self.codebase_path and re-reads itFiles vanish — daemon force-removes the worktree after each evalWrite under self.eval_logs_dir.
Grader uses sys.executableMisses task deps from workspace.setupUse self.get_python_command() / self.run_program / self.run_script.
Runtime deps in grader.setupValidate passes, the run fails every evalRuntime deps → workspace.setup; grader-only deps → grader.setup.
Scoring speed without a correctness gateAgents "optimize" by returning garbage fastGate on correctness first, then score the metric.
parallel.max_workers > 1 with an unsafe graderSporadic port/GPU/scratch collisionsLeave at 1 unless provably concurrency-safe.
Skipping coral validateAgents start, fail every eval identicallyAlways validate first.

When in doubt, run coral init throwaway and read the generated files. Full config schema: https://coral.compounding-intelligence.ai/docs/api/config

© Human-Agent-Society, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 4 other files (references) in plugin/skills/creating-a-coral-task of Human-Agent-Society/CORAL.

  • SKILL.md
  • references/cookbook.md
  • references/grader-api.md
  • references/rubric-judges.md
  • references/task-yaml.md

Open the folder on GitHubat commit 0123dfb

Compare with similar skills

Creating A Coral Task next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Creating A Coral Task compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Creating A Coral Task this skillHuman-Agent-Society/CORAL1.1k—~2.2kAutomated safety check: PassApache-2.0
Evaluation Frameworkathola/claude-night-market341—~1.3kAutomated safety check: PassMIT
Reproduce Chat Statesdifferent-ai/openwork24k—~673Automated safety check: PassCustom licence
Dynamo Jira TicketDynamoDS/Dynamo2k—~1.1kAutomated safety check: PassApache-2.0
Moav E2EMotherofallVPNs/MoaV449—~1.9kAutomated safety check: NotesMIT
Launch Rlmarin-community/marin3.9k—~894Automated safety check: PassApache-2.0

Similar skills

  • Evaluation Framework

    athola/claude-night-market

    Provides weighted scoring, rubrics, and decision-threshold patterns.

    341 GitHub stars~1.3k tokensUpdated yesterday
    Testing & QAAuto-check passed
  • Reproduce Chat States

    different-ai/openwork

    Fires known chat states in the running OpenWork desktop app, such as provider errors, retries and tool steps, so you can check how each renders.

    24k GitHub stars~673 tokensUpdated yesterday
    Testing & QAAuto-check passed
  • Dynamo Jira Ticket

    DynamoDS/Dynamo

    Create structured Jira tickets for Dynamo from bug reports, failing tests, or feature requests.

    2k GitHub stars~1.1k tokensUpdated yesterday
    Testing & QAAuto-check passed
  • Moav E2E

    MotherofallVPNs/MoaV

    Run and debug MoaV's end-to-end tests — real protocol connectivity (client-test.sh) and the moav CLI smoke test — against a LIVE server, via the self-hosted e2e workflow or a local test VPS.

    449 GitHub stars~1.9k tokensUpdated 3 days ago
    Testing & QAAuto-check: notes
  • Launch Rl

    marin-community/marin

    Define, validate, submit, or restart a Marin SkyRL experiment through its artifact main.

    3.9k GitHub stars~894 tokensUpdated yesterday
    Testing & QAAuto-check passed
  • sfdx-hardis Training End-to-End Test

    hardisgroupcom/sfdx-hardis

    Walks the sfdx-hardis training course end to end as a learner would, against a real Developer Edition org and fork, fixing broken steps and screenshots that no longer match.

    403 GitHub stars~2.6k tokensUpdated yesterday
    Testing & QAAuto-check: notes

More from Human-Agent-Society/CORAL

  • Running Coral Experiments

    Human-Agent-Society/CORAL

    Run and manage CORAL experiments from the operator side — launch agents with coral start (dotlist overrides, model/count, tmux vs local), monitor with coral status / coral log / coral show / the web…

    1.1k GitHub stars~1.3k tokensUpdated 1 mo ago
    Auto-check passed
  • Coral Debug

    Human-Agent-Society/CORAL

    Verify and debug changes to CORAL itself — smallest reproduce loop per area (grader / daemon / CLI / hooks / manager / workspace / hub / template / config / web), where to look when something breaks…

    1.1k GitHub stars~1.8k tokensUpdated 1 mo ago
    Auto-check passed
  • Coral New Task

    Human-Agent-Society/CORAL

    End-to-end recipe for adding a new task under examples/ — the three pieces that have to line up (task.yaml, seed/, and grader/), what to put in each, the TaskGrader API surface, the coral validate →…

    1.1k GitHub stars~3.4k tokensUpdated 1 mo ago
    Auto-check passed
  • Promoting Dev To Main

    Human-Agent-Society/CORAL

    A skill your agent uses when preparing, reviewing, resolving conflicts for, or merging a CORAL release pull request from the long-lived dev branch into main.

    1.1k GitHub stars~588 tokensUpdated 1 mo ago
    Auto-check passed
  • Setting Up Coral

    Human-Agent-Society/CORAL

    One-time machine setup after installing the coral CLI — register local agent runtimes as named bindings with coral setup / coral setup agent, validate them with coral agents doctor (incl.

    1.1k GitHub stars~1.7k tokensUpdated 1 mo ago
    Auto-check passed
  • Coral Extend

    Human-Agent-Society/CORAL

    Add a new component to the CORAL framework itself — a new agent runtime under coral/agent/builtin/ (claudecode/codex/cursoragent style), a new CLI command in coral/cli/, a new bundled skill or…

    1.1k GitHub stars~1.6k tokensUpdated 1 mo ago
    Auto-check passed

Questions about Creating A Coral Task

What does Creating A Coral Task do?

Author a new CORAL task — the three pieces that must line up (task.yaml, seed/, a packaged grader/), the coral init → coral validate → smoke-test loop, and how to pick a grader pattern (stdout…. Creating A Coral Task is an agent skill from Human-Agent-Society/CORAL.yaml, seed/, a packaged grader/), the coral init → coral validate → smoke-test loop, and how to pick a grader pattern (stdout float, test pass-rate, ratio-vs-baseline, multi-metric, or an LLM rubric judge).

When should I use Creating A Coral Task?

Creating A Coral Task fits situations like: the user wants to create a CORAL task; port a benchmark into CORAL; score open-ended outputs (reports/memos) with a judge; debug a grader that crashes on the seed / ranks the leaderboard backwards / leaks the answer key.

How do I install Creating A Coral Task in Claude Code?

Run `npx skills add Human-Agent-Society/CORAL --skill creating-a-coral-task -a claude-code`. Or copy the skill folder (plugin/skills/creating-a-coral-task in Human-Agent-Society/CORAL) into .claude/skills/creating-a-coral-task in your project. Claude Code loads it when a task matches its description.

How do I install Creating A Coral Task in Codex?

Run `npx skills add Human-Agent-Society/CORAL --skill creating-a-coral-task -a codex`. Or copy the skill folder (plugin/skills/creating-a-coral-task in Human-Agent-Society/CORAL) into .agents/skills/creating-a-coral-task in your project. Codex loads it when a task matches its description.

Can I use Creating A Coral Task in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add Human-Agent-Society/CORAL --skill creating-a-coral-task -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/creating-a-coral-task, .gemini/skills/creating-a-coral-task, .github/skills/creating-a-coral-task and .opencode/skills/creating-a-coral-task in your project.

What does Creating A Coral Task need to run?

SKILL.md names no scripts, command-line tools or credentials: Creating A Coral Task is instructions for the agent only. Our summary lists: Python 3.

Does Creating A Coral Task access the network?

SKILL.md names 1 domain. As links in the text: coral.compounding-intelligence.ai. This is read from the text; nothing was executed.

Is Creating A Coral Task safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Creating A Coral Task use?

Creating A Coral Task is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Creating A Coral Task use?

About 2.2k tokens (SKILL.md is roughly 8.8k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 5.8k tokens, read only when the agent opens those files.

What are the alternatives to Creating A Coral Task?

Skills that share tags, products or a category with Creating A Coral Task: Evaluation Framework (athola/claude-night-market, 341 stars), Reproduce Chat States (different-ai/openwork, 24k stars), Dynamo Jira Ticket (DynamoDS/Dynamo, 2k stars) and Moav E2E (MotherofallVPNs/MoaV, 449 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Creating A Coral Task?

Human-Agent-Society (a GitHub organization) maintains it in Human-Agent-Society/CORAL, which has 1,060 GitHub stars. The repository holds 7 skills in this directory. The repository was last updated on September 8, 2026.

Source: Human-Agent-Society/CORAL on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.