--- name: standards version: 0.5.0 description: > Runs Phase 4 review/publish/verify for a cli-web-* CLI: implementation review by 3 parallel agents, the tiered quality checklist (Tier 1 critical fail-fast, then comprehensive), pip install + smoke test, and per-CLI skill generation. Use when a CLI's tests pass and it is ready to be validated and published. when_to_use: > Trigger phrases: "validate CLI", "publish CLI", "review CLI", "smoke test", "quality check", "start Phase 4", "quality checklist", "generate Claude skill", "verify implementation quality", or after the testing skill completes. Not for capture, implementation, or test writing. --- # CLI-Anything-Web Standards (Phase 4: Review + Publish + Verify) Quality gate for cli-web-* CLIs. This skill owns the complete Phase 4: independent implementation review, structural quality checklist, publishing, and end-user smoke testing. Nothing ships until this phase passes. Copy this checklist and check off items as you complete them: ``` Phase 4 Progress: - [ ] Prerequisites: tests 100% pass, TEST.md Parts 1+2, .md present - [ ] Step 1: 3 review agents dispatched, Critical findings = 0 - [ ] Step 2: validate-checklist Tier 1 exits 0, then full run reviewed - [ ] Step 3: pip install -e . verified (binary on PATH, --help, doctor) - [ ] Step 4: end-user smoke test — auth, READ, WRITE all pass - [ ] Step 5: per-CLI skill generated (passes test_skill_quality.py) - [ ] Step 6: registry entry added, devkit gates green, repo docs updated - [ ] phase-state marked complete ``` --- ## Prerequisites (Hard Gate) Do NOT start unless: - [ ] All tests pass (100% pass rate from Phase 3) - [ ] TEST.md has both Part 1 (plan) and Part 2 (results) - [ ] All core modules are implemented and functional - [ ] `.md` (API map) exists and documents all endpoints If tests are not passing, invoke the `testing` skill first. If this gate or the phase state is in a failed/inconsistent state, follow `skills/shared/RECOVERY.md` §phase-state Check Failures. **Optional pre-review coverage scan:** before dispatching the review agents, you MAY run the `gap-analyzer` skill (`${CLAUDE_PLUGIN_ROOT}/skills/gap-analyzer/SKILL.md`, pass `APP_PATH=/agent-harness`) to diff captured endpoints (`.md` + `traffic-capture/traffic-analysis.json`) against implemented commands. It is *optional* here because the traffic-fidelity-reviewer agent covers endpoint coverage during Step 1; gap-analyzer is the *mandatory first step* of `/refine`, where no reviewer pass exists. Run it here when coverage looks doubtful and you want the structured report before the agents start. ### Site Profile Exceptions Not all checks apply to every CLI. When evaluating, consider the site profile: - **No-auth sites** (public APIs): Skip auth-related checks (auth.py required, auth commands, auth smoke test). Mark as N/A. - **Read-only sites** (no write operations): Skip write operation smoke test. Verify reads return real data instead. - **API-key auth sites**: `auth login` takes a key argument, not playwright-cli. `auth refresh` is not applicable — use `auth logout` instead. Mark inapplicable checks as "N/A — [reason]" rather than creating dead-code stubs. --- ## Step 1: Implementation Review (3 Parallel Agents) Before checking structure or publishing, verify the code *actually does the right thing*. Tests prove it runs; this step proves it's correct. Dispatch 3 plugin agents in the **same message** using the Agent tool: - `traffic-fidelity-reviewer` — API coverage (reads .md + client.py + commands/) - `harness-compliance-reviewer` — Code conventions incl. JSON envelope STRUCTURE (reads CONVENTIONS.md + all source) - `output-ux-reviewer` — User experience (runs --help, checks REPL, validates JSON) Pass each agent: APP_PATH=`{app}/agent-harness`, APP_NAME=`{app}`, and site profile (auth_type, is_read_only). The agents are defined in the plugin's `agents/` directory. | Agent | Focus | What it reads | What it catches | |-------|-------|---------------|-----------------| | Traffic Fidelity | API coverage | `.md` + `client.py` + `commands/` | Missing endpoints, wrong params, broken response parsing, dead client methods, stale API map | | HARNESS Compliance | Code quality + JSON envelope structure | CONVENTIONS.md + checklist + all source | click.ClickException bypass, missing to_dict(), retry_after lost, auth retry missing, stderr UTF-8 | | Output & UX | User experience | `--help` output, `--json` output, REPL | Protocol leaks, stale REPL help, dead command files, broken entry points | Each agent scores findings on a 0-100 confidence scale. When all 3 return: 1. **Filter out findings with confidence < 75** (noise) 2. Categorize remaining findings: - **Critical** (90-100): Bugs, missing endpoints, data loss, auth broken - **Important** (75-89): Wrong fields, incomplete parsing, missing options - **Minor** (75, edge cases): Help text gaps, cosmetic issues 3. Present the review report 4. **Fix all Critical issues** before proceeding — re-run only the affected agent to verify the fix 5. Fix Important issues (not strictly blocking but strongly recommended) **Gate: Do not proceed to Step 2 until Critical count = 0.** --- ## Step 2: Structural Quality Checklist (tiered) The checklist is tiered (see `references/quality-checklist.md` "Tiers"): **Tier 1 (critical)** failures block publish; **Tier 2 (comprehensive)** failures are warnings that should still be fixed. **2a. Tier 1 fail-fast first.** Run only the critical checks and fix every FAIL before doing anything else — there is no point reviewing a CLI whose structure, packaging, or `--json` envelope is broken: ```bash python ${CLAUDE_PLUGIN_ROOT}/scripts/validate-checklist.py \ /agent-harness --app-name --auth-type --tier1-only ``` Non-zero exit = Tier 1 failures. Fix and re-run until it exits 0. **2b. Full run.** Then run the complete checklist (both tiers): ```bash python ${CLAUDE_PLUGIN_ROOT}/scripts/validate-checklist.py \ /agent-harness --app-name --auth-type ``` The summary shows per-tier counts. Exit is non-zero only on Tier 1 failures (add `--strict` to make Tier 2 failures blocking too). Fix Tier 2 FAILs before publishing unless explicitly deferred with a reason. The validator automates the mechanical checks; the remaining judgment-based items (documentation quality, error message guidance, fixture realism) are reviewed manually per `references/quality-checklist.md`. --- ## Step 3: Install and Verify `setup.py` was generated by scaffold-cli.py in Phase 2 (namespace packages, entry point, profile-correct dependencies) — verify it, don't rewrite it. ```bash cd /agent-harness pip install -e . which cli-web- # entry point on PATH cli-web- --help # command tree renders cli-web- doctor # environment self-diagnosis (install, auth, deps) ``` If `doctor` reports fail-level findings, fix them before the smoke test — they are exactly the problems an end user would hit first. --- ## Step 4: End-User Smoke Test (MANDATORY) Run the automated smoke test first for quick validation: ```bash python ${CLAUDE_PLUGIN_ROOT}/scripts/smoke-test.py cli-web- --auth-type ``` This checks CLI binary resolution, --help, --version, auth status, and --json output for protocol leaks. Then proceed with manual verification below. This is the most critical verification step. The agent MUST simulate what a real end user would do after `pip install cli-web-`. If this fails, the pipeline is NOT complete -- go back and fix the issue. **If no-auth site:** Skip items 1-2 (auth). Go directly to item 3 (READ). **If read-only site:** Skip item 4 (WRITE). Verify reads return real data. **1. Authenticate as an end user would:** ```bash cli-web- auth login ``` This uses Python sync_playwright() -- opens a browser, user logs in, cookies saved. This is what end users will run. If this fails, the CLI is broken for end users. **2. Verify auth status shows LIVE VALIDATION OK:** ```bash cli-web- auth status ``` Must show: cookies present, tokens valid. If it shows "expired", "redirect", or any auth failure -- STOP. Fix auth before proceeding. **3. Run a READ operation and verify real data:** ```bash cli-web- --json list ``` This must return real data from the live API -- NOT an error, NOT empty, NOT "auth not configured". Verify the JSON response contains expected fields. **4. Run a WRITE operation and verify it actually worked:** This is the step the agent most commonly skips. Reading data is easy -- the real test is whether the CLI can CREATE, UPDATE, or GENERATE something. ```bash # For CRUD apps (Monday, Notion, Jira): cli-web- --json create --name "smoke-test-$(date +%s)" cli-web- --json list # verify the created item appears cli-web- --json delete --id # For generation apps (Suno, Midjourney, NotebookLM audio): cli-web- --json generate --prompt "test" --wait # Verify: JSON response contains a real ID, status=complete, not an error # If the command has --output, verify the file was downloaded and size > 0 # For search/query apps: cli-web- --json search "test query" # Verify: results array is non-empty ``` **If ANY write/generate command fails, the pipeline is NOT complete.** Reading a list of existing items only proves auth works -- it does NOT prove the CLI can actually do useful work. The whole point is to CREATE things, not just read them. **5. Only after items 1-4 ALL pass, continue to Step 5.** ### Smoke Test Checklist - [ ] `auth login` works (Python playwright, API key, or N/A for no-auth) - [ ] `auth status` shows valid (or N/A for no-auth) - [ ] At least one READ returns real data - [ ] **At least one WRITE/CREATE/GENERATE succeeds** (or N/A for read-only) - [ ] The CLI works standalone -- no debug Chrome, no port 9222, no MCP - [ ] **Output sanity: no raw protocol data leaks in `--json` output** (see below) ### Output Sanity Run every command with `--json` and check for raw protocol leaks (`wrb.fr`, `af.httprm`, empty `[]`, null required fields). Full red-flags table: `skills/shared/CONVENTIONS.md` §Protocol-Leak Smoke Check. **#1 gap to watch for:** Agent runs `list` (GET with auth — easy), declares done, but never tests create/generate (POST with CSRF, encoding). Always test at least one write. --- ## Step 5: Generate the Per-CLI Skill See "Generate Claude Skill" below. The skill must pass the quality gate before Step 6. ## Step 6: Register the CLI (registry-driven — most repo docs are generated) `registry.json` is the source of truth: the CI test matrix, the README fleet table, and the install block are all **generated from it**. Add the registry entry, then run the generators — never edit those outputs by hand: ```bash # 1. Add the entry to registry.json (schema below), then: cli-web-devkit registry validate # entry <-> fleet cross-check cli-web-devkit docs # regenerates README regions + registry site data cli-web-devkit about --apply # syncs the GitHub "About" CLI count (needs gh admin) cli-web-devkit resync --app # vendored files in sync + manifest updated cli-web-devkit drift # must report 0 drifted/missing # 2. Offline fleet contract for the new CLI (help/version/REPL/MCP/doctor): python -m pytest tests/contract -q -k ``` Remaining hand-edited files (independent — update in parallel if you like): - `CHANGELOG.md` — entry under [Unreleased] -> Added - `CLAUDE.md` — row in the Generated CLIs table - `README.md` hero badge counts (outside the generated markers) - `cli_web//README.md` — fill in the scaffolded skeleton **Start from the scaffolded skeletons.** `scaffold-cli.py` (v2) already rendered `README.md` and the per-CLI `SKILL.md` skeletons from `templates/README.md.tpl` and `templates/SKILL.md.tpl` during Phase 2 — fill in the remaining placeholders with actual CLI data from ` --help` and `.md` rather than writing from scratch. ### Generate Claude Skill **Goal:** Create a project-local Claude skill so that Claude can use this CLI automatically in future conversations — no manual lookup required. **IMPORTANT:** The skill must exist in TWO locations: 1. `.claude/skills/-cli/SKILL.md` — for Claude Code discovery (project-level) 2. `/agent-harness/cli_web//skills/SKILL.md` — portable with `pip install` (included via `package_data` in setup.py) Create the skill once, then copy it to both locations. ### Step 1: Find the .claude directory Create `/.claude/skills/-cli/SKILL.md`: 1. **Read `references/skill-authoring.md` first** — it defines the frontmatter rules, description format, body limits, and the standard section structure. The skeleton rendered from `templates/SKILL.md.tpl` during Phase 2 already follows it; fill in the FILL_IN markers. 2. Run `cli-web- --help` and each group's `--help` — every command you document must be verified against the real surface (a stale example is worse than no example). 3. Validate before publishing: the skill must pass `python -m pytest ${CLAUDE_PLUGIN_ROOT}/scripts/tests/test_skill_quality.py` (frontmatter fields, description ≤1024 chars third-person, body ≤500 lines, reference links resolve). --- ## Update Repository README Add presentation metadata to `registry.json`, then run `cli-web-devkit docs` to regenerate the README table and registry site. Add a hand-written quick-start example in the README's "Try Them" section only when it adds distinct value. ### Update registry.json and CLAUDE.md Add the new CLI to `registry.json` at the repo root: ```json { "name": "cli-web-", "website": "", "protocol": "", "auth": "", "directory": "/agent-harness", "namespace": "cli_web.", "commands": ["", "", ...], "install": "pip install cli-web-", "description": "", "site_icon": "", "site_category": "", "site_tags": [""] } ``` Also add to the Generated CLIs table in `CLAUDE.md`. --- ## Pipeline Complete The pipeline is NOT done until ALL of these are checked: ### Smoke Tests - [ ] Auth works (login + status, or N/A for no-auth) - [ ] At least one READ returns real data - [ ] At least one WRITE succeeds (or N/A for read-only) ### Skills (TWO copies) - [ ] `.claude/skills/-cli/SKILL.md` exists (Claude Code discovery) - [ ] `cli_web//skills/SKILL.md` exists (portable with pip install) - [ ] Based on the scaffolded skeleton from `templates/SKILL.md.tpl` ### Package - [ ] `setup.py` has `package_data={"": ["skills/*.md", "*.md"]}` - [ ] `__main__.py` exists for `python -m cli_web.` support ### Documentation - [ ] `cli_web//README.md` exists (filled in from the `templates/README.md.tpl` skeleton) - [ ] `.md` API map exists - [ ] `tests/TEST.md` has Part 1 (plan) + Part 2 (results) ### Repo-Level Updates - [ ] `registry.json` — entry with name, website, protocol, auth, commands, install, description, skill path (+ `canary` read-only invocations for no-auth CLIs). The CI matrix, README table, and registry site derive from this entry. - [ ] `cli-web-devkit registry validate` + `docs` + `drift` all green - [ ] `python -m pytest tests/contract -k ` passes (offline contract) - [ ] `CHANGELOG.md` — entry added under [Unreleased] → Added - [ ] `CLAUDE.md` — new row in Generated CLIs table - [ ] `README.md` hero badge counts updated (outside the generated markers) - [ ] Branch protection: if required status checks are pinned by job name, add the new matrix job names — the matrix itself updates automatically from registry.json (never edit tests.yml) All key rules (naming, auth, --json, REPL, rate limits) are defined in `skills/shared/CONVENTIONS.md` — HARNESS.md and CLAUDE.md only index them. When every box above is checked, mark the pipeline finished: ```bash python ${CLAUDE_PLUGIN_ROOT}/scripts/phase-state.py complete --phase standards ``` --- ## Integration | Relationship | Skill | |-------------|-------| | **Preceded by** | `testing` (Phase 3) | | **Followed by** | None — this is the final phase | | **References** | HARNESS.md (Generated CLI Structure), `skills/shared/CONVENTIONS.md` (all rules), `skills/shared/RECOVERY.md` (gate failures) | --- ## Related - **`testing`** skill -- Phase 3 test planning/writing/documentation - **`methodology`** skill -- Phase 2 analyze/design/implement - **`capture`** skill -- Phase 1 traffic recording - **`/cli-anything-web:validate`** -- Command to run the full tiered checklist validation - **`gap-analyzer`** skill -- Optional coverage scan (mandatory first step of `/refine`)