Methodology

Implementation Rules

All rules below are DEFINED in skills/shared/CONVENTIONS.md — this section tells you when to apply them during implementation.

• exceptions.py -- implement first. Required hierarchy and error-code mapping: CONVENTIONS.md §Exception Hierarchy. Complete code: references/exception-hierarchy-example.py.

• client.py -- HTTP client with exception mapping and auth retry:

  • HTTP library choice:
    • httpx (default) — for most sites (REST, GraphQL, batchexecute)
    • curl_cffi — for Cloudflare-protected sites. Uses Chrome TLS fingerprint impersonation to bypass bot detection without cookies or auth:

      from curl_cffi import requests as curl_requests
      resp = curl_requests.get(url, impersonate="chrome")

      Use curl_cffi when Phase 1 detects Cloudflare (cf-ray header, challenge page). Add curl_cffi, beautifulsoup4 to setup.py instead of httpx.
  • Centralized auth header/cookie injection
  • Automatic JSON parsing with response body verification
  • Status code → exception mapping: 401/403→AuthError, 404→NotFoundError, 429→RateLimitError, 5xx→ServerError (CONVENTIONS.md §Exception Hierarchy)
  • Auth retry (3-attempt auto-refresh): current cookies → reload auth.json → headless refresh, never more. The full table is CONVENTIONS.md §Auth Rules; the templates generate it by default.
  • Exponential backoff for rate limits (CONVENTIONS.md §Exponential Backoff & Polling; code in references/polling-backoff-example.py)
  • For apps with 3+ resource types: split into namespaced sub-clients (client.notebooks.list(), client.sources.add())
  • See references/client-architecture-example.py for the full pattern

• auth.py -- handles token storage, refresh, expiry. Implementation depends on auth type:

  For no-auth sites: DO NOT create auth.py, session.py, or auth command groups. These files are dead code for public APIs and confuse users. The CLI should have NO auth-related files or commands. The only exception is if the site has optional auth (e.g., API key for write operations) — in that case, implement a minimal auth module.

  For browser-delegated auth (Google, Microsoft, etc.): Python sync_playwright() login flow with cookie domain priority for international users (CONVENTIONS.md §Auth Rules).

  Storage, env var, cookie priority, and dual-format handling are defined in CONVENTIONS.md §Auth Rules; implementation code for each pattern is in references/auth-strategies.md (read section-addressed).

• Anti-bot resilient client construction (when detected in Phase 2):

  • Extract session tokens via CDP first (cookies), then HTTP GET + HTML parsing (CSRF, session IDs)
  • Never hardcode build labels (bl), session IDs (f.sid), or CSRF tokens -- extract dynamically at runtime
  • Replicate same-origin headers captured during Phase 1 traffic (e.g., x-same-domain: 1 for Google apps)
  • Implement auto-retry on 401/403: re-fetch homepage -> re-extract tokens -> retry once
  • See references/google-batchexecute.md for the complete Google pattern

• RPC codec subpackage (for non-REST protocols like batchexecute): When the API uses a non-REST protocol, add core/rpc/ with:

  • types.py -- method ID enum, URL constants
  • encoder.py -- request encoding (protocol-specific format)
  • decoder.py -- response decoding (strip prefix, parse chunks, extract results) The client.py still exists but delegates encoding/decoding to rpc/.

• Progress feedback -- Use rich>=13.0 spinners for operations >2s (suppress in --json mode). See references/rich-output-example.py.

• JSON error output -- --json mode errors are JSON too, not plain text (CONVENTIONS.md §JSON Envelope). Implement via utils/output.py json_error().

• All commands use handle_errors(json_mode) context manager — centralizes error handling, exit codes (1=user, 2=system, 130=interrupt), and JSON errors. See references/helpers-module-example.py.

• Generation commands support --wait, --retry N, --output path — CONVENTIONS.md §Exponential Backoff & Polling; code in references/polling-backoff-example.py.

• Windows UTF-8 fix — at the top of <app>_cli.py, reconfigure BOTH stdout AND stderr to UTF-8 before any import that prints (CONVENTIONS.md §Windows UTF-8 Fix has the exact snippet).

• HTML table parsers MUST extract ALL visible columns — not just name/price, because missing fields in --json output make the CLI useless for filtering and analysis. If the site shows version, club, nation, stats, skills, weak foot — parse all of them. Empty fields in --json output = incomplete parser.

• Entry point: cli-web-<app> via setup.py console_scripts (CONVENTIONS.md §Naming Conventions)

• Namespace: cli_web.*

• utils/repl_skin.py, utils/doctor.py, and utils/mcp_server.py are all vendored by scaffold-cli.py (canonical source: cli-web-core/cli_web_core/, synced via cli-web-devkit resync) — never hand-edit the per-CLI copies. The entry point registers the fleet-standard doctor and mcp-serve commands from the vendored adapters (register_doctor_command(cli, ...), register_mcp_command(cli, ...)); both derive from the Click tree, so no per-command wiring is needed.

• utils/helpers.py -- shared CLI helpers (generate for every CLI):

  • resolve_partial_id(partial, items) — prefix-match UUIDs for get/rename/delete
  • handle_errors(json_mode) — context manager replacing try/except in all commands
  • require_notebook(notebook_arg) — gets notebook ID from arg or persistent context
  • sanitize_filename(name) — safe filenames from artifact titles
  • poll_until_complete(check_fn) — exponential backoff polling
  • get_context_value(key) / set_context_value(key, value) — persistent context.json See references/helpers-module-example.py for the complete module.

Not all helpers apply to every CLI. Include only what the CLI uses: handle_errors and print_json are always needed. resolve_partial_id only for UUID-based apps. require_notebook/context helpers only for apps with persistent context. poll_until_complete only for generation/async operations.

REPL Implementation Rules (Critical)

The four REPL rules are defined with code examples in CONVENTIONS.md §REPL Rules — apply them as you wire up <app>_cli.py:

1. Parse REPL lines with shlex.split(line), never line.split().
2. Propagate --json by PREPENDING it to the args list passed to cli.main(args=..., standalone_mode=False) — never **ctx.params.
3. Help-sync: every commit that adds a command/option updates _print_repl_help() in the same commit.
4. Required single values are @click.argument positionals, not @click.option(..., required=True).

These bugs appear in almost every generated REPL — read the §REPL Rules section before writing the entry point, not after the REPL breaks.

Parallel Implementation (dispatch independent modules as subagents)

When the CLI has 3+ command groups (e.g., notebooks, sources, chat, artifacts), dispatch parallel subagents -- one per command module. Each agent gets:

• The <APP>.md API spec for its resource
• The client.py and auth.py interfaces it depends on
• Clear scope: "Implement commands/notebooks.py with list, get, create, delete"

Parallelization opportunities:

Implementation order (with maximum parallelism):

Phase A (sequential): Write core foundation
  exceptions.py → client.py → auth.py (if needed) → models.py

Phase B (parallel): Dispatch ALL independent work simultaneously
  ┌─ Agent 1: commands/notebooks.py
  ├─ Agent 2: commands/sources.py
  ├─ Agent 3: commands/chat.py
  ├─ Agent 4: commands/artifacts.py
  ├─ Agent 5: rpc/encoder.py + rpc/decoder.py (if non-REST)
  └─ Agent 6 (background): test_core.py (unit tests for core modules)
  All run concurrently — each only depends on Phase A modules

Phase C (sequential): Wire everything together
  utils/helpers.py → <app>_cli.py → __main__.py → setup.py
  (repl_skin.py, doctor.py, mcp_server.py were already vendored by
   scaffold-cli.py in Step B.0; the entry point registers doctor + mcp-serve)

Key parallelism rules:

• Dispatch independent command modules as parallel subagents (one per commands/*.py file)
• Start unit test writing as a background agent during command implementation
• Entry point (<app>_cli.py, setup.py) must come last (depends on all commands)

Mandatory Smoke Check (Before Testing Phase)

Before invoking testing, install (pip install -e .) and verify:

1. cli-web-<app> --help loads
2. cli-web-<app> auth status --json shows valid (if auth-required)
3. cli-web-<app> <resource> list --json returns real data
4. One WRITE command works (if applicable)

Red flags — fix before testing: the full table is CONVENTIONS.md §Protocol-Leak Smoke Check (wrb.fr/af.httprm leaks, empty []/null, parser index mismatches). One methodology-specific case: a null WRITE response may mean the operation is client-side — see references/google-batchexecute.md "Client-Side Operations".

Update phase state:

python ${CLAUDE_PLUGIN_ROOT}/scripts/phase-state.py complete <app> \
  --phase methodology --output <app>/agent-harness/

Next Step

When implementation is complete and the smoke check passes, invoke the testing skill to plan and write tests.

Do NOT skip testing -- every CLI must have comprehensive tests before publishing.

Companion Skills

Integration

Reference Files

• references/traffic-patterns.md -- Common API patterns (REST, GraphQL, RPC)
• references/auth-strategies.md -- Auth implementation strategies
• references/google-batchexecute.md -- Google batchexecute RPC protocol spec
• references/ssr-patterns.md -- SSR framework patterns and data extraction strategies
• references/exception-hierarchy-example.py -- Complete exception hierarchy with HTTP status mapping
• references/client-architecture-example.py -- Namespaced sub-client pattern with auth retry
• references/polling-backoff-example.py -- Exponential backoff polling and rate-limit retry
• references/rich-output-example.py -- Rich progress bars, JSON error responses, table formatting