Integration Test

integration-test — traceway-cli

A repeatable protocol for verifying the CLI end-to-end against a live Traceway server, focused on the things the Go smoke suite can't easily cover.

Trigger

Invoke ONLY when the user explicitly asks. Never run as a side effect of edits, commits, or builds.

Relationship to the Go smoke suite

The just smoke-test target (test/smoke/*_test.go, build tag smoke) is the primary regression check. Against a live instance it already covers:

• JSON shape of every list endpoint and profiles list.
• Output-format coverage (json/table/yaml) for projects, profiles, exceptions, endpoints, logs.
• Client-side enum validation (--search-type, --order-by, --sort-direction, --aggregation, --tag).
• Time-range parsing edges (--since 7D, missing --to, --since + absolute mix, far-future windows).
• metrics query missing --name, bogus name → empty series, malformed tag.
• exceptions show <zeros> → exit 5 not_found.
• --profile no-such-profile → exit 4; --project 0…0 → non-zero, no panic, no connection_failed.

Do not re-implement these here. If they regress, that's a Go-test bug, not a skill failure.

What this skill adds beyond just smoke-test:

1. Real-data detail endpoints — exceptions show <captured-hash>, populated metrics query --name <real> with every aggregation + group-by.
2. Adaptive metric-name discovery — walk a candidate list until one populates.
3. TTY-vs-pipe default — table rendering to a real terminal.
4. Coverage matrix report — a human-readable artifact, on demand.
5. Safety doctrine — the forbidden-verb blocklist and confirmMutation env hygiene, applied to every probe.

Hard constraints

Read-only. No exceptions. Even if a subcommand looks safe by name, check --help for mutating flags before running.

Forbidden verbs and flags

Skip any subcommand whose name or --help mentions:

• archive, unarchive, resolve, unresolve, mute, ack, acknowledge
• create, delete, update, set, put, post, add, remove, rm
• assign, claim, close, reopen
• login, logout, token, rotate, regenerate
• --archive, --resolve, --delete, --write, --mutate, --apply, --commit

If a new subcommand is ambiguous (sync, refresh, replay, export), do not run it — list it under "skipped — manual review". If --dry-run exists, still skip write-shaped subcommands.

Mutation safeguards

The CLI gates mutations via confirmMutation (cmd/traceway/querycommon.go). The harness MUST:

• Never pass --yes.
• unset TRACEWAY_ASSUME_YES at the top of the script.
• Run with stdin from /dev/null.

So that if a forbidden verb slips through, the gate refuses with exit 2 usage_error instead of hanging on a prompt.

Pre-flight

Run in order. Stop if any fails.

1. Build: nix develop --command go build -o ./bin/traceway ./cmd/traceway.
2. Config exists (don't print — JWT inside): test -f "${XDG_CONFIG_HOME:-$HOME/.config}/traceway/config.json".
3. Reachability + capture TW_PROJECT_ID:

   ./bin/traceway projects list --output json | jq -e 'type=="array" and length>=1' >/dev/null
   TW_PROJECT_ID=$(./bin/traceway projects list --output json | jq -r '.[0].id')

projects list --output json returns a bare array, not a {data, pagination} envelope.

Detail-endpoint probes

exceptions show <captured-hash>

1. Capture a real hash:

   HASH=$(./bin/traceway exceptions list --since 720h --page-size 1 --output json | jq -r '.data[0].exceptionHash // empty')

   If empty, retry against other projects via --project <id>. If still empty, skip with reason no exception found across all projects.
2. With a real hash: three output formats + --help. JSON shape: {group: {...}, occurrences: [...], pagination: {...}} — assert .group and .occurrences.
3. Capture .occurrences[0].traceId if present for the logs probe below.

metrics query --name <real-metric> (adaptive)

Probe these names in order until one returns a populated series:

system.cpu.utilization
system.network.io
system.network.errors
system.network.dropped
http.server.duration
traceway.requests

If none populates, skip the live block with reason no live metric name found.

For the first metric that populates:

• Three output formats + --help.
• All aggregations: avg, sum, count, min, max, p50, p95, p99.
• --interval-minutes 15.
• --group-by direction for network metrics (splits __all__ into receive/transmit).

JSON shape: {results: [{name, unit, series: {<tag-key>: [{timestamp, value}, ...]}}]} — series is a map keyed by group tag, default key __all__.

logs query --trace-id <captured>

If a real trace id was captured above, run logs query --trace-id $TRACE --since 720h. Assert exit 0 and {data, pagination} shape.