Atmos Tests

Execution and Failure Rules

• Direct children run in declaration order. test does not accept max_concurrency; put it on parallel or matrix.
• needs is supported on children of parallel/matrix, not direct children of test. Use it whenever a check consumes another concurrent check's result. A failed dependency skips its dependent checks.
• Parallel and matrix groups can be siblings under test, but cannot contain another parallel/matrix group. Nested test groups are unsupported.
• fail.mode: wait_all is the default: independent tests continue, then unhandled failures fail the group. Unset when permits continuation after a failure; explicit when: success would instead gate a check on success.
• fail.mode: fail_fast cancels remaining work; fail.max_failures configures the existing failure threshold. best_effort records failures but lets the group succeed. Use these deliberately, especially for deployment gates.
• Preserve explicit child continue and nested group failure policies. For example, continue: always tolerates a leaf failure, but that case still appears red in the report. Tolerated failures are not passing assertions.
• retry retries a leaf as one case; the final attempt determines its outcome. Use explicit expectations and bounded attempts for readiness checks.
• Use supported noninteractive registered handlers. Interactive prompts, terminal handoff, process replacement (exec, exit), background/async work, recording or emulator sessions, and wait/wait-all/cancel steps cannot run inside a test. Start services and prepare fixtures outside the group.

Output and Results

Set output: failures (the default) or output: all on the test group. These are scalar test-specific choices, not raw, none, log, or viewport. Both buffer per leaf; all also reveals successful logs at completion. Do not redirect leaf output to /dev/null: it removes the evidence needed when a test fails.

The terminal report uses a hierarchical tree and progress bar. CI/non-TTY output is static. Counts cover expanded leaves once, excluding group nodes; retries do not add cases. The summary includes passed, failed, skipped, canceled, and elapsed time. Group result metadata exposes total, passed, failed, skipped, and canceled.

Use scoped environment variables and step result values rather than shared files or process-global state to pass data. Matrix cases must not overwrite the same fixture paths. Keep secrets in existing Atmos auth/secret mechanisms; masking still applies to captured output, but do not deliberately print credentials.

Place optional success messages after the test group with when: success and content such as Tests passed. Message-only steps inside the group would count as cases without validating anything.

Canonical References

Consult the project docs for exact fields before extending a pattern:

• website/docs/workflows/workflows/workflow/steps/type/test.mdx
• The neighboring http.mdx, script.mdx, shell.mdx, require.mdx, atmos.mdx, container.mdx, parallel.mdx, and matrix.mdx files.
• website/docs/workflows/workflows/workflow/steps/continue.mdx and retry.mdx.
• examples/tests/atmos.yaml for runnable local examples.

If behavior is unclear, inspect pkg/schema/test_step.go and the registered handlers under pkg/runner/step/ rather than inventing test-only assertion fields.