Standards

CLI-Anything-Web Standards (Phase 4: Review + Publish + Verify)

Quality gate for cli-web-* CLIs. This skill owns the complete Phase 4: independent implementation review, structural quality checklist, publishing, and end-user smoke testing. Nothing ships until this phase passes.

Copy this checklist and check off items as you complete them:

Phase 4 Progress:
- [ ] Prerequisites: tests 100% pass, TEST.md Parts 1+2, <APP>.md present
- [ ] Step 1: 3 review agents dispatched, Critical findings = 0
- [ ] Step 2: validate-checklist Tier 1 exits 0, then full run reviewed
- [ ] Step 3: pip install -e . verified (binary on PATH, --help, doctor)
- [ ] Step 4: end-user smoke test — auth, READ, WRITE all pass
- [ ] Step 5: per-CLI skill generated (passes test_skill_quality.py)
- [ ] Step 6: registry entry added, devkit gates green, repo docs updated
- [ ] phase-state marked complete

Prerequisites (Hard Gate)

Do NOT start unless:

• All tests pass (100% pass rate from Phase 3)
• TEST.md has both Part 1 (plan) and Part 2 (results)
• All core modules are implemented and functional
• <APP>.md (API map) exists and documents all endpoints

If tests are not passing, invoke the testing skill first. If this gate or the phase state is in a failed/inconsistent state, follow skills/shared/RECOVERY.md §phase-state Check Failures.

Optional pre-review coverage scan: before dispatching the review agents, you MAY run the gap-analyzer skill (${CLAUDE_PLUGIN_ROOT}/skills/gap-analyzer/SKILL.md, pass APP_PATH=<app>/agent-harness) to diff captured endpoints (<APP>.md + traffic-capture/traffic-analysis.json) against implemented commands. It is optional here because the traffic-fidelity-reviewer agent covers endpoint coverage during Step 1; gap-analyzer is the mandatory first step of /refine, where no reviewer pass exists. Run it here when coverage looks doubtful and you want the structured report before the agents start.

Site Profile Exceptions

Not all checks apply to every CLI. When evaluating, consider the site profile:

• No-auth sites (public APIs): Skip auth-related checks (auth.py required, auth commands, auth smoke test). Mark as N/A.
• Read-only sites (no write operations): Skip write operation smoke test. Verify reads return real data instead.
• API-key auth sites: auth login takes a key argument, not playwright-cli. auth refresh is not applicable — use auth logout instead.

Mark inapplicable checks as "N/A — [reason]" rather than creating dead-code stubs.

Step 1: Implementation Review (3 Parallel Agents)

Before checking structure or publishing, verify the code actually does the right thing. Tests prove it runs; this step proves it's correct.

Dispatch 3 plugin agents in the same message using the Agent tool:

• traffic-fidelity-reviewer — API coverage (reads <APP>.md + client.py + commands/)
• harness-compliance-reviewer — Code conventions incl. JSON envelope STRUCTURE (reads CONVENTIONS.md + all source)
• output-ux-reviewer — User experience (runs --help, checks REPL, validates JSON)

Pass each agent: APP_PATH={app}/agent-harness, APP_NAME={app}, and site profile (auth_type, is_read_only). The agents are defined in the plugin's agents/ directory.

Each agent scores findings on a 0-100 confidence scale. When all 3 return:

1. Filter out findings with confidence < 75 (noise)
2. Categorize remaining findings:
   • Critical (90-100): Bugs, missing endpoints, data loss, auth broken
   • Important (75-89): Wrong fields, incomplete parsing, missing options
   • Minor (75, edge cases): Help text gaps, cosmetic issues
3. Present the review report
4. Fix all Critical issues before proceeding — re-run only the affected agent to verify the fix
5. Fix Important issues (not strictly blocking but strongly recommended)

Gate: Do not proceed to Step 2 until Critical count = 0.

Step 2: Structural Quality Checklist (tiered)

The checklist is tiered (see references/quality-checklist.md "Tiers"): Tier 1 (critical) failures block publish; Tier 2 (comprehensive) failures are warnings that should still be fixed.

2a. Tier 1 fail-fast first. Run only the critical checks and fix every FAIL before doing anything else — there is no point reviewing a CLI whose structure, packaging, or --json envelope is broken:

python ${CLAUDE_PLUGIN_ROOT}/scripts/validate-checklist.py \
  <app>/agent-harness --app-name <app> --auth-type <auth-type> --tier1-only

Non-zero exit = Tier 1 failures. Fix and re-run until it exits 0.

2b. Full run. Then run the complete checklist (both tiers):

python ${CLAUDE_PLUGIN_ROOT}/scripts/validate-checklist.py \
  <app>/agent-harness --app-name <app> --auth-type <auth-type>

The summary shows per-tier counts. Exit is non-zero only on Tier 1 failures (add --strict to make Tier 2 failures blocking too). Fix Tier 2 FAILs before publishing unless explicitly deferred with a reason.

The validator automates the mechanical checks; the remaining judgment-based items (documentation quality, error message guidance, fixture realism) are reviewed manually per references/quality-checklist.md.

Step 3: Install and Verify

setup.py was generated by scaffold-cli.py in Phase 2 (namespace packages, entry point, profile-correct dependencies) — verify it, don't rewrite it.

cd <app>/agent-harness
pip install -e .
which cli-web-<app>          # entry point on PATH
cli-web-<app> --help         # command tree renders
cli-web-<app> doctor         # environment self-diagnosis (install, auth, deps)

If doctor reports fail-level findings, fix them before the smoke test — they are exactly the problems an end user would hit first.

Step 4: End-User Smoke Test (MANDATORY)

Run the automated smoke test first for quick validation:

python ${CLAUDE_PLUGIN_ROOT}/scripts/smoke-test.py cli-web-<app> --auth-type <auth-type>

This checks CLI binary resolution, --help, --version, auth status, and --json output for protocol leaks. Then proceed with manual verification below.

This is the most critical verification step. The agent MUST simulate what a real end user would do after pip install cli-web-<app>. If this fails, the pipeline is NOT complete -- go back and fix the issue.

If no-auth site: Skip items 1-2 (auth). Go directly to item 3 (READ).

If read-only site: Skip item 4 (WRITE). Verify reads return real data.

1. Authenticate as an end user would:

cli-web-<app> auth login

This uses Python sync_playwright() -- opens a browser, user logs in, cookies saved. This is what end users will run. If this fails, the CLI is broken for end users.

2. Verify auth status shows LIVE VALIDATION OK:

cli-web-<app> auth status

Must show: cookies present, tokens valid. If it shows "expired", "redirect", or any auth failure -- STOP. Fix auth before proceeding.

3. Run a READ operation and verify real data:

cli-web-<app> --json <first-resource> list

This must return real data from the live API -- NOT an error, NOT empty, NOT "auth not configured". Verify the JSON response contains expected fields.

4. Run a WRITE operation and verify it actually worked: This is the step the agent most commonly skips. Reading data is easy -- the real test is whether the CLI can CREATE, UPDATE, or GENERATE something.

# For CRUD apps (Monday, Notion, Jira):
cli-web-<app> --json <resource> create --name "smoke-test-$(date +%s)"
cli-web-<app> --json <resource> list   # verify the created item appears
cli-web-<app> --json <resource> delete --id <id-from-create>

# For generation apps (Suno, Midjourney, NotebookLM audio):
cli-web-<app> --json <resource> generate --prompt "test" --wait
# Verify: JSON response contains a real ID, status=complete, not an error
# If the command has --output, verify the file was downloaded and size > 0

# For search/query apps:
cli-web-<app> --json search "test query"
# Verify: results array is non-empty

If ANY write/generate command fails, the pipeline is NOT complete. Reading a list of existing items only proves auth works -- it does NOT prove the CLI can actually do useful work. The whole point is to CREATE things, not just read them.

5. Only after items 1-4 ALL pass, continue to Step 5.

Smoke Test Checklist

• auth login works (Python playwright, API key, or N/A for no-auth)
• auth status shows valid (or N/A for no-auth)
• At least one READ returns real data
• At least one WRITE/CREATE/GENERATE succeeds (or N/A for read-only)
• The CLI works standalone -- no debug Chrome, no port 9222, no MCP
• Output sanity: no raw protocol data leaks in --json output (see below)