Field Test

Field Test

Hands-on, adversarial test pass of $ARGUMENTS (the feature/command named when this skill was invoked, e.g. atmos vendor pull).

If no target was given, default to the change introduced on the current branch rather than asking. Resolve the actual pull-request base branch when one is available (gh pr view --json baseRefName -q .baseRefName for the current branch), falling back to the repository's default branch (gh repo view --json defaultBranchRef -q .defaultBranchRef.name) when no PR exists yet. Do NOT use the upstream tracking branch (@{u}) as the base — for a normal feature branch that tracks origin/<same-branch-name>, diffing against its own upstream produces an empty or near-empty diff, not the PR's actual changes, once the branch has been pushed.

A branch NAME from gh (e.g. develop) is not guaranteed to be a usable git ref in THIS checkout — shallow clones, detached HEADs, and worktrees with a narrow fetch refspec can have the name without the commits. Before running any diff, verify the base actually resolves here (git rev-parse --verify --quiet <candidate>^{commit}), trying in order: origin/<resolved-name>, <resolved-name> — take the first that verifies. Do not fall back to origin/main/main when resolved-name came from an actual PR base other than the default branch (e.g. a PR targeting develop) — diffing against main instead of the PR's real base compares against the wrong history and can silently miss real changes or include unrelated ones. origin/main/main are legitimate fallback candidates only in the no-PR case (gh pr view returned nothing, so resolved-name already IS the repository's default branch) or when gh itself isn't available at all. If a known, non-default PR base doesn't resolve as either origin/<resolved-name> or <resolved-name>, stop and ask the user which base to diff against — do not run `git diff

<base>...` against an unverified ref and let it fail with a confusing git error, and do not
silently substitute a different branch for a known PR base. Once a real base is confirmed, inspect
the FULL set of
changes relative to that base — content, not just a file-list summary: `git diff <base>...HEAD`
(the full patch, not `--stat`, since `--stat` only shows file names and line counts, not what
those lines actually do) for committed history; `git diff HEAD` for any staged/unstaged changes
to tracked files not yet committed; and `git ls-files --others --exclude-standard` to enumerate
untracked files — `git status --porcelain` lists their paths too but never their content, and
neither `git diff HEAD` nor a `--stat` summary includes untracked files at all, so a brand-new
implementation file can otherwise go completely unread. Read the actual content of every
untracked file this turns up, the same as any diff hunk. Derive the test target from all of
that — the CLI command(s), flag(s), config option(s), or subsystem the changed files implement —
and state explicitly what you inferred and why before proceeding to Phase 1. Only fall back to
asking the user if no candidate base ref resolves at all, there's truly nothing changed (clean
worktree, base equals HEAD, no untracked files), or the changes span multiple unrelated features
with no coherent single target (ask which one to focus on, don't silently pick one).

Goal: catch "vibe-coded slop" — behavior that looks fine in code review but breaks or misleads a real user — not to re-run what automated tests already cover. Anticipate plausible user misunderstandings, not just obvious bugs.

This pass is investigation only. Do not fix anything you find — see Phase 5. Report and stop; the user decides what to fix. Any follow-up fix work — including a plan to fix the findings, not just the implementation — must close with the fix-log skill, not this one.

Plan Mode

If plan mode is active when this skill is invoked (a <system-reminder> says so), Phase 3 (Build fixtures) and Phase 4 (Execute) can't run inline — writing fixture files and running real, sometimes state-mutating commands are exactly what plan mode exists to gate.

• Run Phase 1 (Research) and Phase 2 (Generate hypotheses) as normal — both are already read-only and fit plan mode's constraints without modification.
• Instead of proceeding into Phase 3/4, write the plan file: the target under test, the prioritized hypothesis list from Phase 2, the fixtures Phase 3 would build (note which extend/copy an existing fixture, per that phase's guidance), and which planned Phase 4 commands are read-only vs. state-mutating — the mutating ones are specifically what need sign-off.
• Call ExitPlanMode to request approval of that plan. Don't ask for approval any other way.
• Once approved, resume at Phase 3 using the approved plan as the fixture/execution blueprint.

Phase 1 — Research before touching anything

The goal is a map of "documented or plausible usage" minus "already tested" = what needs manual verification. This phase is broad, read-only research — delegate it to Agent subagent_type: "Explore" (1-3 agents in parallel, one per bullet below) rather than doing it all serially inline.

When defaulting to the current branch (no explicit target given), scope every bullet below to the target inferred from the branch diff — don't research the whole surrounding subsystem when the branch only touched one corner of it. If the branch's changed files span more than one command or package, treat each as a separate target to cover in Phase 2-4, prioritized by how much of the diff each accounts for.

• Implementation — the actual code, not just its docs or the skill describing it. Inspect every changed production package identified by the diff — new business logic belongs in narrow pkg/ packages per this repo's conventions, but internal/exec/ is still where a large amount of existing logic lives during its ongoing migration, so a branch touching files there must still be read, not skipped. Check both cmd/<command>/ (thin call site) and whichever pkg//internal/exec/ package(s) it delegates to for the real logic and error paths. When defaulting from a branch diff, read the full diff content itself first (not just the post-change files, and not just a --stat summary) — the diff shows what changed from, which is where a regression or half-finished edge case would show up. Include untracked files (git ls-files --others --exclude-standard) in this reading pass too — they never appear in any diff at all.
• Docs and skills — every relevant page under website/docs/cli/commands/, the matching .claude/skills/atmos-* skill(s) for the subsystem, and any README describing the feature. Note anything phrased with confidence you haven't independently confirmed against the code — docs and skills describe intended behavior, not necessarily current behavior. A capability can be documented on one page and missing from another that also covers it (a reference page, the skill's own summary, a related subsystem doc that mentions it in passing) — check every doc surface for the feature, not just the primary one. Also re-read each doc for internal self-contradiction: a rule stated in one paragraph can be silently overridden or contradicted by an example or a later paragraph in the same file.
• Existing automated tests — unit tests colocated with the code, tests/test-cases/ fixtures, tests/testdata/ golden snapshots. For each, note exactly what it does and doesn't exercise (mocked vs. real execution, which flags/paths/backends are hit).
• Every flag, config option, and documented action/mode — grep for them and list them. You will need to touch every one in Phase 4. Cross-reference mechanically: grep the command's flag registration calls in its Go source and diff that list against the flags documented in the corresponding .mdx's <dl> — every registered flag needs a matching <dt>, and every documented flag needs to actually be registered.

Phase 2 — Generate hypotheses, don't just wander

Before running anything, write down concrete things to try, prioritized by what a real user would plausibly do:

• Every flag combination that seems natural but might not be validated (two flags that should be mutually exclusive; two config fields whose combination is never cross-checked).
• Every flag's parsed value actually reaching the code path that would use it — not just that the flag exists, parses, and the command exits 0. A flag can be fully registered and documented yet silently dropped before the logic that should consume it (e.g. the command builds a fresh/empty config struct instead of the one built from parsed flags). Trace the value from flag definition to point of use for every flag, don't just confirm the command accepts it.
• Every place the docs/skill claim something you haven't verified against actual code.
• Any "safe-looking" command (plan/preview/--dry-run/list/describe) that might secretly mutate state or trigger side effects, if built the same way as a mutating command — Atmos has many of these pairs (e.g. terraform plan vs apply, vendor diff vs pull), so this is a high-yield category here specifically.
• Any action/mode described in docs but not exercised by ANY test or example in the repo — those are the highest-yield targets; if nothing has ever run it for real, assume it's broken until you prove otherwise.
• Copy-paste/misconfiguration scenarios — what happens if a user copies a working stack/component block and changes one field but forgets a related one?
• Error messages — accurate, do they name the actual flags/values involved, do they suggest a fix (per this repo's error-builder/hint conventions)?
• Idempotency/rerun-safety — run the same operation twice; does the second run behave correctly?
• Determinism — run the same read-only command several times with no state change between runs — is the output identical every time?
• Any command that does real work across multiple items (batch installs/updates, concurrent workers, multi-resource loops) — does it show live progress, or does it silently buffer everything and dump it all at once when the whole batch finishes? A command that takes 10+ seconds with zero output is indistinguishable from a hang to a real user. Also check whether its status lines actually use ui.Success/ui.Error/ui.Warning/ui.Info (icon + theme color) per CLAUDE.md's I/O and UI Usage section, rather than a hand-rolled glyph ("✓ %s", "✗ %s") printed through the plain ui.Writef/ui.Write — the two are easy to conflate since both compile and both "print a checkmark," but only the semantic function is themed/colored. This class of bug is easy to miss when every command in this pass has been run through the Bash tool — captured output can look fine in the transcript even when the real behavior (silent hang, unstyled text) would be obvious to a human watching a real terminal. This needs two separate checks, not one piped command doing double duty — piping through cat -v makes the command non-TTY, which exercises the non-live fallback renderer instead of the real live-progress path, and --force-color does not restore TTY behavior:
  • Live progress: run the command in a real pseudo-TTY, unpiped (e.g. script -q /dev/null build/atmos toolchain update --force-tty --force-color), and watch it — does a spinner/progress bar actually redraw in place, or does it silently buffer and dump everything at once?
  • ANSI styling: separately, pipe a --force-color run through cat -v (or grep for the raw \x1b[ / ^[[ escape sequence) to confirm color codes are actually present around each status line, not just plain text with a Unicode glyph. Piping is fine here since this check only cares about styling, not live-rendering behavior.
• Any "N -> M" / diff-style report line — construct a case where N and M are the same value in different string forms (e.g. v1.2.3 vs 1.2.3, the one equivalence normalizeVersion actually handles by stripping a leading v — don't test casing or trailing-metadata variants unless the specific normalizer under test explicitly documents supporting them). A raw string-equality comparison will misreport a no-op as a change, which also tends to corrupt whatever summary/count line tallies outcomes — check the tally against the individual lines above it, don't just trust it.