Suite Creator

Suite Creator

Creates new Margin Eval test suites from scratch, including the directory structure, configuration files, prompts, Dockerfiles, and grading scripts.

Suite Structure

<suite-name>/
├── suite.toml
└── cases/
    └── <case-name>/
        ├── case.toml
        ├── prompt.md
        ├── env/
        │   └── Dockerfile
        ├── tests/
        │   ├── test.sh
        │   └── <supporting test files>
        └── oracle/              # optional
            └── solve.sh

suite.toml

kind = "test_suite"
name = "<suite-name>"
description = "<what this suite evaluates>"
cases = [
  "<case-1>",
  "<case-2>",
]

The cases array lists directory names under cases/, in the order they should run.

case.toml

kind = "test_case"
name = "<case-name>"
description = "<one-line description>"
test_cwd = "/"
test_timeout_seconds = 1800

[metadata]
difficulty = "easy"
category = "programming"
tags = ["tag1", "tag2"]

Key rules:

• name must match its directory name exactly
• kind is always "test_case"
• test_timeout_seconds is an integer (seconds)
• test_cwd is the working directory where test.sh runs inside the container

Image handling — exactly one of:

• image = "registry/repo@sha256:<64hex>" for a pre-built, digest-pinned image
• Omit image and place a Dockerfile at env/Dockerfile to build at compile time

prompt.md

The task description sent to the agent as its initial prompt. Must not be empty.

tests/test.sh

The grading script — this is the evaluator. It must terminate with exactly one of:

• 0 = pass
• 1 = fail
• 2 = infra

Do not write reward.txt. The verifier process exit code is the authoritative result. The script must be executable. All files in tests/ are packaged together and staged at {test_cwd}/tests/ in the container.

oracle/solve.sh (optional)

Reference solution. Not executed during normal eval runs — useful for dry runs and validation.

env/Dockerfile

Container environment. The entire env/ directory is the build context, so supporting files (setup scripts, seed data, config) can live alongside the Dockerfile.

Workflow

1. Define the suite scope

Clarify with the user:

• What capability or behavior is being evaluated?
• How many cases? What range of difficulty?
• What does the agent's environment look like? (language, frameworks, tools available)

2. Scaffold the directory structure

Create the suite directory, cases/ subdirectory, and a skeleton for each case. It helps to create all case directories first, then fill them in one at a time.

3. Write the Dockerfile

Start with the Dockerfile because it defines the environment everything else runs in. The Dockerfile should produce a container that:

• Has all dependencies the agent might need pre-installed
• Sets a WORKDIR (this becomes test_cwd in case.toml)
• Contains any seed data, starter code, or project scaffolding the task requires
• Does not contain the solution or hints toward it

Keep images minimal — install only what the task requires. Use specific version tags, not latest.

FROM python:3.12-slim

WORKDIR /app

# Pre-install project dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt

# Seed the workspace with starter code
COPY src/ ./src/

4. Write the prompt

The prompt is the only input the agent receives. Write it as if briefing a developer who has just been dropped into the container. It should include:

• What to do — the task objective, stated clearly
• Where things are — relevant file paths, project structure
• Constraints — specific requirements, output format, things to avoid
• Success criteria — what "done" looks like, in concrete terms

Avoid leaking test implementation details. The agent should not know how it will be graded — only what the correct behavior is.

A good prompt is specific enough that a competent developer could complete the task without asking clarifying questions, but does not prescribe a particular implementation approach.

5. Write the grading script

tests/test.sh determines pass/fail. The grading approach depends on what's being tested:

Use a simple explicit verdict API in every new harness:

#!/bin/bash
set -euo pipefail

pass()  { printf 'VERDICT: PASS\n'; exit 0; }
fail()  { printf 'VERDICT: FAIL\n'; exit 1; }
infra() { printf 'VERDICT: INFRA\n' >&2; exit 2; }

Policy:

• return pass or fail only when the harness reached a trustworthy verdict about the candidate
• return infra when the harness cannot reach a trustworthy verdict for reasons not attributable to the candidate
• missing candidate artifact, candidate compile/import/runtime failure, candidate timeout after candidate logic starts, and wrong output are all fail
• verifier/bootstrap/parser/config failures are infra

File/output verification — check that the agent produced the right files with the right content:

#!/bin/bash
set -euo pipefail

pass()  { exit 0; }
fail()  { exit 1; }
infra() { exit 2; }

if [ ! -f /app/output.json ]; then
  fail
fi

set +e
pytest tests/test_outputs.py -rA
exit_code=$?
set -e

case "$exit_code" in
  0) pass ;;
  1) fail ;;
  *) infra ;;
esac

Test suite pass-through — run the project's own test suite against the agent's changes:

#!/bin/bash
set -euo pipefail

pass()  { exit 0; }
fail()  { exit 1; }
infra() { exit 2; }

cd /app
set +e
npm test
exit_code=$?
set -e

case "$exit_code" in
  0) pass ;;
  1) fail ;;
  *) infra ;;
esac

Custom validation — for tasks where correctness is more nuanced:

#!/bin/bash
set -euo pipefail
python tests/validate.py

Guidelines for grading scripts:

• Install test dependencies inside the script (they run in a separate context from the agent)
• Test observable outcomes, not implementation details — the agent should be free to solve the problem however it sees fit
• Make sure the script is deterministic — same agent output should always produce the same verdict
• Keep terminal verdict paths explicit and easy to audit
• Prefer verifier-level changes over Dockerfile changes when clarifying verdict semantics
• If a parser or verifier cannot interpret the result, default to infra unless you can directly attribute the failure to the candidate