Task Creator

SkillsBench Task Authoring

Repo context. This skill lives in the benchflow repo but authors tasks for benchflow-ai/skillsbench. Unqualified references below — CONTRIBUTING.md, MAINTAINER.md, docs/*.md, tasks/<task-id>/ — mean files in skillsbench, not in this repo. Work from a skillsbench checkout.

Build a task that scores well on the task principles. Two artifacts when you're done: a directory under tasks/<task-id>/ that bench tasks check accepts, and a PR description that maps cleanly to the PR template.

Workflow

1. propose      → one-paragraph proposal, gut-check against the proposal rubric
2. scaffold     → bench tasks init, plus the native task.md layout below
3. task.md      → frontmatter + human-written, outcome-focused prompt body
4. environment  → Dockerfile + bundled inputs; do NOT bake skills
5. tests        → 4–10 test functions, parametrize for bulk; check formulas AND values
6. oracle       → human-written reference solution that derives answers
7. skills       → 2–3 generalizable skills (or reuse existing ones from /tasks/*/environment/skills/)
8. validate     → bench tasks check + oracle eval (must reach 1.0)
9. self-review  → invoke task-review skill on the local path
10. agent runs  → Opus 4.8 / latest Codex with and without skills
11. submit      → PR with the table the template asks for

Each step is described below. Skip a step only if the rubric says it's optional for your track (research / multimodal). Skipping verification will get the PR rejected.

Step 1 — Propose

Before writing files, write a four-bullet proposal answering the proposal-stage rubric:

• What's the task? One paragraph.
• Who does this in real life? Job, domain, why someone pays for it.
• Why do skills help? What domain knowledge is non-obvious without a skill?
• How would you verify it? Specific output files, deterministic tests.

Sanity-check against the seven proposal criteria (motivated · skill-dependent · verifiable · well-specified · solvable · realistic · outcome-verified). If any is shaky, fix the idea before scaffolding. Posting the proposal in #task-ideas is optional but cheap insurance.

Step 2 — Scaffold

bench tasks init <task-id>          # generates the skeleton

Final layout (matches CONTRIBUTING.md):

tasks/<task-id>/
├── task.md                         # YAML frontmatter + agent-facing prompt
├── environment/
│   ├── Dockerfile
│   ├── <bundled inputs>            # CSV, xlsx, etc. — frozen test inputs
│   └── skills/                     # 2–3 skill dirs (optional, see Step 7)
├── oracle/
│   ├── solve.sh                    # oracle (human-written, derives answers)
│   └── <helpers>                   # e.g. recalc.py copied from xlsx skill
└── verifier/
    ├── test.sh                     # see assets/test.sh.template
    ├── test_outputs.py             # ≤10 functions, parametrize for bulk
    └── expected.<ext>              # ground-truth artifact when comparing outputs

Step 3 — task.md

The single biggest review failure is an AI-generated task prompt. Write the prompt body by hand. The point is communication, not preservation of source artifacts:

• Imperative tone — "Download…", "Compute…", "Save the result to /root/output.json".
• Explicit absolute paths for every input and output.
• Numbered or bulleted requirements if there are more than two steps.
• Equivalent, not verbatim. When the source is a docx, paper, or notebook, distill it. Drop author musings ("I would normally…"), fix typos, replace mixed straight/curly quotes, expand contractions. Match the intent perfectly; the wording is yours.
• Constraints listed. Excel-formula tasks: explicitly say "use Excel formulas, not Python-computed values" if the original implies it.
• No skill names. Never mention skills the agent will be given — it should discover them via the standard skill paths.
• Output paths must match what your tests check. The most common bug is the agent saving to /root/result.xlsx while the tests expect a different /root/... artifact path.
• Anchor a cutoff date for time-sensitive answers. If the agent fetches live data (regulations, market prices, scores, leaderboards, dataset versions) and the ground truth is a fixed snapshot, name the snapshot date in the instruction: "as of August 1, 2025", "based on the 2024-Q4 release", "using the 2026 ICD-10 code set". Without this anchor, an agent reading newer information would correctly diverge from the frozen ground truth and fail the test for the "right" reason. See references/time-invariance.md.

Length: 1–2 paragraphs ideally. If you need a page, the task is overspecified — split it. See references/instruction-anatomy.md for examples.

Step 4 — Environment

Write environment/Dockerfile from assets/Dockerfile.template. Three rules:

1. python:3.12-slim base. Avoid Ubuntu < 24.04 and Python < 3.12.
2. Pre-create /app and the agent's home dirs. bench's verifier hardening forces --rootdir=/app, and the skill-injection symlinks land in /home/agent/.codex/. Both must exist and be writable by the sandbox user. The template has the lines.
3. Do not bake skills. Drop any COPY skills /root/.codex/skills lines you might have copied from older tasks — bench eval run --skill-mode with-skill --skills-dir <skills-dir> injects skills at deploy time. Baking them in makes the without-skills run a no-op.

Bundle frozen inputs (CSVs, xlsx) in environment/. For tasks that need internet (research-track), declare the required network and environment policy in task.md frontmatter and prefer Playwright over urllib for any external API — government / ArcGIS / portal endpoints have caching and pagination quirks that bite raw HTTP clients but pass through a real browser context cleanly. See references/oracle-patterns.md §"External APIs".

Pin every Python package to an exact version. Don't pin apt packages.

Step 5 — Tests

Target 4–10 functions, ~10–20 parametrized cases total — see docs/unit-test-guidelines.md. Patterns:

• One workbook-structure test for "everything I need exists" (sheets present, file is valid, no ??? left, etc.). Combine "exists + parses + has the right shape" rather than three separate functions.
• One parametrized test per function family the author used. For an Excel task: one test_uses_xlookup, one test_uses_averageifs, etc. — each parametrized over a few sample cells. Reading expected.xlsx cell-by-cell to enumerate every formula is too dense (~100 cases) and rejected at review.
• One parametrized test for cached values matching expected.xlsx. This is what proves the agent ran recalc.
• One chart / artifact test.

Always use direct RC=$? capture in test.sh, never $? after a pipe — pytest … | tee always reports tee's exit code (zero), which silently makes every reward 1.0. The template gets this right.

Always copy the agent's output artifact into /logs/verifier/<filename> so reviewers can inspect it later. Multimodal tasks should attach the artifact to the PR.

See references/test-design.md for the complete rubric and worked examples.

Step 6 — Solution

oracle/solve.sh is human-written. Three points:

• Derive vs. copy. The rubric says "derive through computation" but also "skeptical of over-engineered solutions." For procedural tasks (compute a number, run a query, transform a file, patch code) the oracle reproduces the workflow in Python — that's derive. For tasks where the answer is a hand-crafted artifact the toolchain can't fully reproduce (Excel arrays, PowerPoint, audio/video, hand-laid PDF), cp /oracle/<artifact> /root/<output> is the lesser of two evils — that's copy. Flag the trade-off in the PR description so reviewers can weigh in. Details + decision examples in oracle-patterns.md §1 "Derive vs. copy".
• Self-contained. The oracle runs without skills, so anything a skill provides (e.g. recalc.py for Excel, a known-good .patch file for code tasks) must be copied into oracle/ and called as /oracle/<file>. For the copy-oracle pattern, ship oracle/<artifact> as a byte copy of verifier/<artifact>.
• Test thresholds match the saved artifact, not the instruction. Before committing tests, profile the expected artifact for actual counts (formulas per column, JSON keys, modified lines, etc.). Authors routinely diverge from their own instructions — what's saved is what the test must accept. See test-design.md §"Profile the expected artifact before setting thresholds".

oracle-patterns.md catalogs format-specific quirks: Excel + LibreOffice's array-formula <v/> empty bug, PowerPoint embedded-chart preservation, code-patch oracles, external API access via Playwright with retry, and benchflow's idle-600s pitfall on long-running subprocesses.

Step 7 — Skills

2–3 skills, generalizable, not task-specific. The repo already has reusable skills under tasks/*/environment/skills/ — xlsx (formulas + recalc), data-reconciliation (sum-constraint recovery), mesh-analysis (3D STL), etc. Copy an existing one rather than write a new one when it covers the domain knowledge you need.

A new skill should:

• Have a YAML frontmatter name and description that names both what it does and when to use it (the description is the trigger; the body only loads after).
• Provide non-obvious domain knowledge — endpoints, schemas, country groupings, gotchas. Don't write a Wikipedia summary.
• Be reusable beyond your task. Reviewers reject skills that only make sense for one set of inputs.
• Stay under ~500 lines. Split detail into references/ files.

Read skillsbench's skill-creator before writing one from scratch.