Suite Converter

Suite Converter

Converts downloaded eval datasets into valid Margin Eval test suites.

Purpose

Use this skill when converting an external eval dataset into the Margin Eval suite format.

The job has three parts:

• map the source format into Margin's filesystem contract
• preserve the source verifier's intended behavior without breaking Margin's execution model
• validate the conversion on a small sample with margin run before scaling up

Target Margin Contract

<suite-name>/
├── suite.toml
└── cases/
    └── <case-name>/
        ├── case.toml
        ├── prompt.md
        ├── env/
        │   └── Dockerfile
        ├── tests/
        │   ├── test.sh
        │   └── <other test files>
        └── oracle/              # optional
            └── solve.sh

suite.toml

kind = "test_suite"
name = "<suite-name>"
description = "<description>"
cases = [
  "<case-1>",
  "<case-2>",
]

The cases array lists directory names under cases/, in the order they should run.

case.toml

kind = "test_case"
name = "<case-name>"
description = "<one-line description>"
agent_cwd = "/"
test_cwd = "/"
test_timeout_seconds = 1800

[metadata]
difficulty = "easy"
category = "programming"
tags = ["tag1", "tag2"]

[metadata] can be preserved for human context and future tooling, but the current Margin compiler/runtime primarily cares about the execution fields such as kind, name, description, image, agent_cwd, test_cwd, and test_timeout_seconds.

Key rules:

• name must match its directory name exactly
• kind is always "test_case"
• test_timeout_seconds is an integer (seconds)
• agent_cwd is the directory where the agent is expected to start and do its work inside the container
• test_cwd is the working directory where test.sh runs inside the container

Image handling, exactly one of:

• image = "registry/repo@sha256:<64hex>" for a pre-built, digest-pinned image
• Omit image and place a Dockerfile at env/Dockerfile to build at compile time

prompt.md

The full task description sent to the agent as its initial prompt. Must not be empty. Copy the source's instruction/prompt file as-is — don't summarize or reformat it.

tests/test.sh

The grading script. This is the evaluator. There is no separate grader abstraction.

env/Dockerfile

Container environment. Can include supporting files alongside the Dockerfile. The entire env/ directory is the build context.

oracle/solve.sh

Optional reference solution. Not executed during normal eval runs. In the current Margin implementation, oracle/ is informational only and is not part of the active compile/runtime path.

Conversion Invariants

These rules are cross-source and should hold for every conversion.

Verifier rules

• Must be executable (chmod +x)
• Exit 0 = pass, 1 = fail, 2 = infra
• Do not write reward.txt. The verifier process exit code is the authoritative result.
• All files in tests/ are packaged together and staged at {test_cwd}/tests/ in the container
• If the source suite uses absolute test-asset paths such as /tests/..., rewrite them to Margin-compatible paths such as tests/... or "${PWD}/tests/...".
• If the source verifier derives test-asset paths from environment variables, normalize or override those variables to Margin-compatible values. Do not rely only on fallback expressions when the source environment may already set an incompatible path such as /tests.
• Ensure the verifier creates any directories it expects before writing output artifacts.
• Use exactly one authoritative test location. If the verifier copies tests from tests/ into the workspace, its test command must avoid rediscovering the mounted tests/ tree. If it runs tests directly from tests/, do not also copy them into the workspace.
• Never leave a wrapper that always exits 0. Every verifier must terminate through explicit pass, fail, or infra paths.
• If the source suite expects status or report artifacts in addition to the exit code, preserve that behavior, but do not treat any single artifact format as universal.

Verdict policy

Use this attribution rule across conversions:

• pass: the harness reached a trustworthy verdict and the candidate satisfied the task
• fail: the harness reached a trustworthy verdict and the candidate did not satisfy the task
• infra: the harness could not reach a trustworthy verdict for reasons not attributable to the candidate

Classify these as fail:

• missing candidate-owned artifact
• candidate compile/import/build/runtime failure
• candidate timeout after candidate logic begins
• wrong output, failed assertions, malformed candidate-generated output
• candidate-caused repo state that prevents intended test execution

Classify these as infra:

• verifier/bootstrap/parser dependency failure
• suite-owned config or hidden test asset is missing or malformed
• verifier cannot interpret logs or required verifier artifacts are missing
• harness-side timeout before candidate evaluation meaningfully begins

If a parser or verifier layer cannot produce a trustworthy verdict, default to infra unless the harness already has direct evidence of a candidate-caused failure.

Case identity rules

• case.toml.name must match the case directory name exactly
• If source-name sanitization causes collisions, append a stable suffix such as part of the source task ID or UUID

Working directory rules

• agent_cwd and test_cwd are separate concepts:
  • agent_cwd is where the agent starts
  • test_cwd is where the verifier runs
• Set agent_cwd to the directory where the source task expects the agent to operate on the codebase or files
• Set test_cwd to the directory where the verifier should execute
• Do not assume they are the same. Only use the same value for both when the source task clearly indicates that the agent and verifier operate from the same directory

Validation rules

• Do not assume the conversion is correct until a sample run succeeds without harness-level issues
• Distinguish real task failures from conversion failures
• If the sample exposes harness issues, fix them and rerun the sample before scaling up
• Validate all three terminal states when practical:
  • a known-good sample yields 0
  • a known-bad or unsolved sample yields 1
  • an induced verifier/setup failure yields 2

Decision Rules

Inferring working directories

Parse the Dockerfile for WORKDIR directives. Use the last WORKDIR value found as the default working directory inside the container.

Use that information carefully:

• infer test_cwd from the verifier and Dockerfile execution context
• infer agent_cwd from where the source task expects the agent to work on the repo or files
• do not default agent_cwd to test_cwd unless the source task clearly uses the same directory for both

If no better signal exists:

• default test_cwd to the last WORKDIR, or "/" if none exists
• choose agent_cwd from the most plausible agent workspace for the task, rather than assuming it matches test_cwd

Clarifying Docker behavior

If the source suite provides more than one viable environment path and the user has not explicitly said which to use, ask before converting.

Common ambiguous cases:

• The source provides both a Dockerfile-like environment and a prebuilt image reference
• The source provides environment files that could be rebuilt, but the user may prefer to keep the original image
• The user may want you to rebuild the image, push it to a registry, and reference the new image in case.toml

When Docker behavior is ambiguous, ask which of these they want:

• Preserve the original image reference in image
• Convert using env/Dockerfile
• Rebuild and publish a new image, then use that in image

Do not modify Dockerfiles just to express pass/fail/infra policy unless the user explicitly asks for environment changes. Prefer verifier-level changes for verdict semantics.