Openclaw QA Testing

OpenClaw QA Testing

Use this skill for qa-lab / qa-channel work. Repo-local QA only.

Read first

• docs/concepts/qa-e2e-automation.md
• docs/help/testing.md
• docs/channels/qa-channel.md
• qa/README.md
• qa/scenarios/index.yaml
• extensions/qa-lab/src/suite.ts
• extensions/qa-lab/src/character-eval.ts

Model policy

• Normal live suite runs rely on QA Lab source- and auth-aware defaults.
• Do not pass --model, --alt-model, or --fast by default. Omitted --fast does not mean fast is disabled; fast behavior is source-owned.
• For scenario-specific runs, the complete execution.summary is authoritative and overrides generic default guidance, including when it requires other flags. Add explicit provider/model pins only when execution.config.requiredProvider or requiredModel requires them.

Default workflow

1. Read the scenario pack and current suite implementation.
2. Decide lane:
   • mock/dev: mock-openai
   • real validation: live-frontier
3. For a normal live suite, use:

pnpm openclaw qa suite \
  --provider-mode live-frontier \
  --output-dir .artifacts/qa-e2e/run-all-live-frontier-<tag>

4. Watch outputs:
   • summary: .artifacts/qa-e2e/run-all-live-frontier-<tag>/qa-suite-summary.json
   • report: .artifacts/qa-e2e/run-all-live-frontier-<tag>/qa-suite-report.md
5. If the user wants to watch the live UI, find the current openclaw-qa listen port and report http://127.0.0.1:<port>.
6. If a scenario fails, fix the product or harness root cause, then rerun the full lane.

OTEL smoke

For local QA-lab OpenTelemetry validation, use:

pnpm qa:otel:smoke

This starts a local OTLP/HTTP trace receiver, runs the otel-trace-smoke scenario through qa-channel, decodes the emitted protobuf spans, and verifies the exported trace names and privacy contract. It does not require Opik, Langfuse, or external collector credentials.

QA credentials and 1Password

• Use op only inside tmux for QA secret lookup in this repo.
• Quick auth check inside tmux:

op account list

• Direct Telegram npm live test secrets currently live in 1Password item:
  • vault: OpenClaw
  • item: Telegram E2E
• That item is the first place to look for:
  • OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKEN
  • OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN
  • OPENCLAW_QA_PROVIDER_MODE
  • OPENCLAW_NPM_TELEGRAM_PACKAGE_SPEC
• Convex QA secrets currently live in 1Password items:
  • vault: OpenClaw
  • item: OPENCLAW_QA_CONVEX_SITE_URL
  • item: OPENCLAW_QA_CONVEX_SECRET_MAINTAINER
  • item: OPENCLAW_QA_CONVEX_SECRET_CI
• Additional related notes/login items seen during QA credential work:
  • vault: Private
  • items: OPENCLAW QA, Convex, Telegram
• If a required value is missing from those notes:
  • do not guess
  • ask the maintainer/operator for the current value or the current 1Password item name
  • for Telegram direct runs, OPENCLAW_QA_TELEGRAM_GROUP_ID may be stored separately from Telegram E2E
  • for Convex runs, the leased Telegram credential should provide the Telegram group id and bot tokens together; do not require a separate OPENCLAW_QA_TELEGRAM_GROUP_ID
  • for Convex runs, prefer OpenClaw/OPENCLAW_QA_CONVEX_SITE_URL; if that is stale or unclear, ask for the active pool URL before running
• Prefer direct Telegram envs for the npm Telegram Docker lane when available:

OPENCLAW_QA_TELEGRAM_GROUP_ID="..." \
OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKEN="..." \
OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN="..." \
OPENCLAW_QA_PROVIDER_MODE="mock-openai" \
OPENCLAW_NPM_TELEGRAM_PACKAGE_SPEC="openclaw@beta" \
pnpm test:docker:npm-telegram-live

• Prefer Convex mode when the goal is stable shared QA infra:
  • round-robin credential leasing
  • thinner wrapper for channel-specific setup
  • CLI/admin flows around the pooled credentials
• Live npm Telegram Docker lane note:
  • scripts/e2e/npm-telegram-live-runner.ts reads OPENCLAW_NPM_TELEGRAM_PROVIDER_MODE
  • do not assume OPENCLAW_QA_PROVIDER_MODE is consumed by that wrapper
  • if a 1Password note only gives OPENCLAW_QA_PROVIDER_MODE, map it explicitly to OPENCLAW_NPM_TELEGRAM_PROVIDER_MODE before running the Docker lane
• Verified live shape:
  • Convex mode can pass the real Docker lane without direct Telegram env vars
  • leased Telegram payload includes the group id coupled to the driver/SUT tokens
  • a real run of pnpm test:docker:npm-telegram-live passed with:
    • OPENCLAW_QA_CREDENTIAL_SOURCE=convex
    • OPENCLAW_QA_CREDENTIAL_ROLE=maintainer
    • OPENCLAW_QA_CONVEX_SITE_URL
    • OPENCLAW_QA_CONVEX_SECRET_MAINTAINER
    • OPENCLAW_NPM_TELEGRAM_PROVIDER_MODE=mock-openai
• If direct Telegram env is missing locally and op signin blocks, prefer dispatching the manual GitHub lane because the qa-live-shared environment already has Convex CI credentials:

gh workflow run "NPM Telegram Beta E2E" --repo openclaw/openclaw --ref main \
  -f package_spec=openclaw@YYYY.M.D-beta.N \
  -f package_label=openclaw@YYYY.M.D-beta.N \
  -f provider_mode=mock-openai

• Poll the exact run id from the dispatch URL. gh run view --json artifacts is not supported; list artifacts with:

gh api repos/openclaw/openclaw/actions/runs/<run-id>/artifacts

WhatsApp live credentials

Use this when setting up or replacing Convex kind=whatsapp credentials.

• Treat WhatsApp QA credentials as operator-owned live accounts, not generated fixtures.
• Use two dedicated WhatsApp-capable test numbers: one driver account and one SUT account. Do not use personal numbers or personal OpenClaw WhatsApp accounts in the shared pool.
• Register and link each account manually with WhatsApp or WhatsApp Business, storing Web auth only in isolated local auth dirs outside the repo.
• For group coverage, create a dedicated test group that includes both QA accounts and store its JID as groupJid; otherwise the group mention-gating scenario should be skipped by default and fail when explicitly requested.
• Package the two Baileys auth dirs into base64 .tgz payload fields and add a new active Convex credential row. Prefer adding a fresh row and disabling stale/broken rows over overwriting credentials in place.
• Expected payload fields: driverPhoneE164, sutPhoneE164, driverAuthArchiveBase64, sutAuthArchiveBase64, and optional groupJid.
• Keep credential material out of the repo, logs, PRs, and screenshots. Redact phone numbers unless the operator explicitly asks for local debugging.
• Validate with pnpm openclaw qa whatsapp --credential-source convex --credential-role maintainer --provider-mode mock-openai and preserve artifact paths plus redacted pass/fail summaries.
• If WhatsApp expires or invalidates a linked Web session, relink locally, package fresh auth archives, add a new Convex row, then disable the stale row.