Qabot

qabot - QA Testing Skill

You are running QA tests and reporting results to the qabot dashboard. qabot is a platform for QA reports - operators (humans with Claude, or clawdbot for automated runs) authenticate via a shared API key and push test results.

Environment

Requires agent-browser >= 0.20.0 (native Rust daemon, no more Node.js/Playwright).

Secrets are stored in ~/.secrets (sourced by ~/.zshrc). Required env vars:

• QABOT_API_KEY - shared API key for write access to qabot
• QABOT_OPERATOR - your operator name (e.g. "gomes", "clawdbot") - labels who ran what
• NATIVE_WALLET_PASSWORD - native wallet password for agent-browser wallet unlock
• QABOT_KEYSTORE_PATH - absolute path to a keystore JSON file used for keystore-based wallet import (e.g. ~/Desktop/thorswap-keystore-*.txt)

# Verify env is set - all four MUST be present
echo "QABOT_API_KEY: ${QABOT_API_KEY:+set}"
echo "QABOT_OPERATOR: ${QABOT_OPERATOR:+set}"
echo "NATIVE_WALLET_PASSWORD: ${NATIVE_WALLET_PASSWORD:+set}"
echo "QABOT_KEYSTORE_PATH: ${QABOT_KEYSTORE_PATH:+set}"

If any of these are missing, tell the user to add them to ~/.secrets. The API key is shared among trusted operators.

Ports

• Web dev server (ShapeShift app): localhost:3000 (or $PORTLESS_URL if using Portless, e.g. http://<branch>.web.localhost:1355)
• qabot API/dashboard: localhost:8080 (dev) or deployed URL

Portless note: If the dev server is running via Portless, PORTLESS_URL is set automatically (e.g. http://feat-x.web.localhost:1355). The qabot profile stores wallet state per-origin, so each Portless origin is separate from localhost:3000 - import the wallet once per new origin.

Modes

Interactive Mode (no fixture provided)

When the user says something like "qa test the trade page" or "run qabot" without a specific fixture:

1. Ask what to test - use AskUserQuestion to clarify what functionality to test
2. Help craft a fixture - build a YAML fixture collaboratively with the user
3. Save the fixture - write to e2e/fixtures/<name>.yaml
4. Execute the fixture - run the tests via agent-browser
5. Report results - push to qabot

Fixture Mode (fixture provided or automated)

When a specific fixture file is provided, or running as part of an automated flow (clawdbot release, CI trigger):

1. Load the fixture - read the YAML file from e2e/fixtures/
2. Resolve dependencies - if the fixture has depends_on, load those fixtures first (recursively)
3. Execute all steps - run dependency fixtures first, then the main fixture, all in one browser session
4. Capture results - screenshots + pass/fail per step (step indices are continuous across all fixtures)
5. Report to qabot - create run, upload screenshots, push results, post PR comment if applicable

Fixture Format

name: Test Name
description: What this tests
route: /trade
depends_on:
  - wallet-health.yaml       # runs this fixture first, wallet unlock etc.
steps:
  - name: Step name
    instruction: Natural language instruction for agent-browser
    expected: What should be true after this step
    screenshot: true  # optional, take screenshot after step

Composability

Fixtures can declare depends_on - a list of other fixture filenames that must run first.

• Dependencies are resolved recursively and deduplicated (each fixture runs at most once)
• All fixtures run sequentially in one browser session - the page state carries over
• Step indices are continuous across all fixtures (dep fixture steps come first)
• All steps from all fixtures go into a single qabot run
• The fixtureFile for the run is the top-level fixture name

Example: eth-to-fox-swap.yaml depends on wallet-health.yaml. When you run eth-to-fox-swap:

1. wallet-health runs first (7 steps: dismiss onboarding, unlock wallet, verify page)
2. eth-to-fox-swap steps run next (5 steps: select assets, enter amount, verify quote)
3. Total: 12 steps in one run, indices 0-11

Onboarding Dialog

On first visit to any origin (yeet.shapeshift.com, release.shapeshift.com, etc.), ShapeShift shows an onboarding splash dialog ("Self-Custody", "You own your keys") with "Skip" and "Next" buttons. The wallet-health fixture handles dismissing this. Always run wallet-health as a dependency for other fixtures.

agent-browser Session

IMPORTANT: Always use the qabot profile. The native wallet is stored in this profile's IndexedDB per-origin.

Use a shell-scoped command alias at session start to reduce command noise:

AB='agent-browser --session qabot --profile ~/.agent-browser/profiles/qabot'

Then use $AB for all commands in that shell session:

$AB open <url>
$AB snapshot
$AB click "Connect Wallet"
$AB screenshot /tmp/step-0.png

When you need a headed run, append --headed only for that command:

$AB --headed open <url>

agent-browser --session qabot --profile ~/.agent-browser/profiles/qabot open <url>

The profile at ~/.agent-browser/profiles/qabot stores the native wallet (IndexedDB, localStorage, cookies) per origin. Import the wallet once per origin, then reuse.

First-time setup per origin (keystore import - works headless):

agent-browser --session qabot --profile ~/.agent-browser/profiles/qabot open <url>

Only use --headed if the user explicitly wants to import via seed phrase (which requires visual interaction). Keystore import is fully automatable headless.

Keystore import flow (when no native wallet exists for the origin):

The keystore file path comes from $QABOT_KEYSTORE_PATH. If unset, ask the operator to export a keystore from their wallet and point that env var at it.

1. Click "Connect Wallet" button (use JS eval if click times out): eval "document.querySelectorAll('button').forEach(b => { if(b.textContent.includes('Connect Wallet')) b.click() })"
2. Click "Import existing"
3. Click "Keystore" (the "Import Keystore File" option)
4. Upload the keystore file to the hidden file input: upload "input[type=file]" "$QABOT_KEYSTORE_PATH"
5. Fill the keystore password: fill "input[placeholder*=Password]" "$NATIVE_WALLET_PASSWORD"
6. Click "Import Keystore"
7. On the "Create a New Password" screen:
   • Fill nickname: fill "input[placeholder*=nickname]" "test"
   • Fill password: fill "input[placeholder*='Enter Password']" "$NATIVE_WALLET_PASSWORD"
   • Fill confirm: fill "input[placeholder*='Confirm Password']" "$NATIVE_WALLET_PASSWORD"
   • Click "Next"
8. Skip onboarding carousel if shown
9. Wallet "test" should appear in top-right. Subsequent runs reuse the persisted profile.

Origins where the wallet has been imported:

• http://localhost:3000 (local dev, legacy)
• http://<branch>.web.localhost:1355 (local dev via Portless - origin varies per branch, e.g. develop.web.localhost:1355)
• https://release.shapeshift.com (release staging)

Wallet Unlock

The native wallet requires a password on each session start. The wallet-health fixture handles this. If running without wallet-health, handle it manually:

1. Check for onboarding dialog first - click "Skip" if present
2. The "Enter Your Password" dialog may or may not appear automatically depending on the page:
   • On some pages, the wallet modal opens automatically with the unlock prompt
   • On other pages (e.g. /trade), you'll see "Connect Wallet" in the top-right nav instead
   • If you see "Connect Wallet", click it, then wait 5 seconds for native wallets to load in the modal
   • If the "test" wallet appears in the wallet list on the left side of the modal, click it to get the password prompt
   • Note: the nav button may still say "Connect Wallet" even while the wallet is loading/connecting - this is normal
3. Click the wallet name button (e.g. "test", "teest") if wallet selection is shown
4. Focus the password input via JS eval (click --ref often times out on external origins): eval "document.querySelector('input[type=password], input[placeholder*=Password]')?.focus()"
5. Type the password - use keyboard type (preferred) or press char-by-char (NOT fill - React controlled inputs need keypress events):

   # PREFERRED: keyboard type command
   agent-browser --session qabot keyboard type "$NATIVE_WALLET_PASSWORD"
   # LEGACY: press char-by-char

6. Click "Next" via JS eval: eval "$(cat /tmp/click-next.js)"
7. Wait 8+ seconds for external origins to fully hydrate

PR Review Reliability Checklist (localhost)

When using qabot for PR review validation on localhost:

1. Follow PR Testing steps verbatim before adding extra assertions.
2. Do a manual-first pass with agent-browser in the same live session:
   • reach exact page/state
   • confirm account/wallet assumptions
   • validate selectors and click-path before reporting
3. Keep wallet setup as preflight only (never as reported qabot steps), but ensure required preconditions are visible before step 1 (e.g. Send button).
4. Only after manual flow is stable, create/report qabot run steps.
5. If automation friction is selector-related, add/ask for precise data-testid at the failing UI control.

Tips

JS Eval & Smart Quotes (CRITICAL)

• Smart quotes kill JS eval: Claude's output produces unicode smart quotes (" " ' ') which cause SyntaxError: Invalid or unexpected token in agent-browser eval.
• PREFERRED: Use --stdin or --base64 to avoid smart quote issues entirely (no temp files needed):

  # PREFERRED: pipe JS via stdin (no temp files needed)
  echo 'document.querySelectorAll("button").forEach(b => { if(b.textContent.includes("Skip")) b.click() })' | agent-browser --session qabot eval --stdin
  
  # ALTERNATIVE: base64 encode to avoid all escaping issues
  agent-browser --session qabot eval --base64 $(echo -n 'your JS code' | base64)
  
  # LEGACY (still works): write to temp file
  printf 'code here' > /tmp/click.js
  agent-browser --session qabot eval "$(cat /tmp/click.js)"

• Pre-write common click helpers at session start (can also be done with --stdin): Before executing any steps, create these reusable JS files in /tmp/. This avoids smart quote issues and speeds up execution:

  # Write all click helpers upfront
  printf 'var btns=document.querySelectorAll("button"); for(var i=0;i<btns.length;i++){if(btns[i].textContent.trim()==="Close"){btns[i].click();}}' > /tmp/click-close.js
  printf 'var btns=document.querySelectorAll("button"); for(var i=0;i<btns.length;i++){if(btns[i].textContent.trim()==="Maybe Later"){btns[i].click();}}' > /tmp/click-later.js
  printf 'var btns=document.querySelectorAll("button"); for(var i=0;i<btns.length;i++){if(btns[i].textContent.includes("Switch Assets")){btns[i].click();break;}}' > /tmp/click-switch.js
  printf 'var btns=document.querySelectorAll("button"); for(var i=0;i<btns.length;i++){if(btns[i].textContent.includes("Sign")&&btns[i].textContent.includes("Swap")){btns[i].click();break;}}' > /tmp/click-sign.js
  printf 'var btns=document.querySelectorAll("button"); for(var i=0;i<btns.length;i++){if(btns[i].textContent.trim()==="Confirm and Trade"){btns[i].click();break;}}' > /tmp/click-confirm.js
  printf 'var btns=document.querySelectorAll("button"); for(var i=0;i<btns.length;i++){if(btns[i].textContent.trim()==="I understand"){btns[i].click();break;}}' > /tmp/click-understand.js
  printf 'var btns=document.querySelectorAll("button"); for(var i=0;i<btns.length;i++){if(btns[i].textContent.trim()==="Preview Trade"){btns[i].click();break;}}' > /tmp/click-preview.js
  printf 'var btns=document.querySelectorAll("button"); for(var i=0;i<btns.length;i++){if(btns[i].textContent.trim()==="Got It"){btns[i].click();break;}}' > /tmp/click-gotit.js
  printf 'var btns=document.querySelectorAll("button"); for(var i=0;i<btns.length;i++){if(btns[i].textContent.trim()==="Skip"){btns[i].click();break;}}' > /tmp/click-skip.js
  printf 'var btns=document.querySelectorAll("button"); for(var i=0;i<btns.length;i++){if(btns[i].textContent.trim()==="Next"){btns[i].click();break;}}' > /tmp/click-next.js

  Then use them: agent-browser --session qabot eval "$(cat /tmp/click-close.js)"

Snapshots

# PREFERRED: interactive elements only (clean, flat list)
agent-browser --session qabot snapshot -i

# Full accessibility tree (verbose, for debugging layout)
agent-browser --session qabot snapshot

# Scope to specific element subtree
agent-browser --session qabot snapshot --selector "main"

# Include cursor-interactive elements (onclick, pointer)
agent-browser --session qabot snapshot -C

Wait Commands (prefer over sleep)

Use wait commands instead of arbitrary sleep calls where possible:

# Network idle detection (page fully loaded):
agent-browser --session qabot wait --load networkidle

# Wait for specific text to appear:
agent-browser --session qabot wait --text "Preview Trade"

# Wait for text to disappear (loading states):
agent-browser --session qabot wait --fn "!document.body.innerText.includes('Loading...')"

# Wait for element to appear:
agent-browser --session qabot wait "[data-testid=trade-form]"

Note: still use sleep 8-10 for wallet unlock since app hydration isn't detectable via wait.

Network Request Capture

Built-in network request capture (no more manual fetch interceptors):

# Capture network requests
agent-browser --session qabot network requests
agent-browser --session qabot network requests --filter "chaindefuser"

# Clear captured requests
agent-browser --session qabot network requests --clear

# Route/mock requests
agent-browser --session qabot network route "**/api/quote" --body '{"error":"test"}'
agent-browser --session qabot network unroute

Clipboard

Clipboard read/write/copy/paste is available via agent-browser --session qabot clipboard <read|write|copy|paste>.

Clicking on External Origins

• Clicking on external origins: click --ref and click --text frequently time out on release.shapeshift.com (elements blocked by overlays or slow hydration). Always prefer JS eval for clicking on external origins.
• .click() vs dispatchEvent: Some buttons on external origins don't respond to .click() (e.g. asset picker avatars). Use dispatchEvent(new MouseEvent("click",{bubbles:true,cancelable:true})) as a more reliable fallback. Example for asset avatar buttons:

  var btn=document.querySelector("button[class*=avatar]"); if(btn) btn.dispatchEvent(new MouseEvent("click",{bubbles:true,cancelable:true}));

• Cookie/tracking banner: On external origins, a cookie banner ("Our dApp uses anonymized click tracking...") appears with a "Got It" button. Dismiss this immediately after page load - it can block interactions. The wallet-health fixture should handle this.

Asset Picker

• Asset picker multi-chain: Assets like FOX exist on multiple chains. Clicking the asset button once expands to show chain variants - click the specific chain variant (e.g. "Ethereum (FOX)") from the expanded list. Primary assets like SOL, BTC, RUNE don't need expansion.
• Asset picker interaction: Open sell asset picker by JS-clicking the sell asset avatar button. Search by focusing the search input via JS eval then pressing characters. Select results by JS-clicking the matching button.
• Switch Assets is unreliable after swaps: After completing a swap, the "Switch Assets" button may not reverse assets. Always verify via snapshot after clicking. If it didn't work, fall back to manually selecting assets via the pickers.

Screenshots

• Always use absolute paths for screenshots: Use /tmp/step-N-name.png, NOT relative paths. Relative paths cause resolution issues between agent-browser's cwd and the shell's cwd. curl -F file@... will fail with exit code 26 if the path is wrong.

  agent-browser --session qabot screenshot "/tmp/step-0-dismiss-onboarding.png"

• Screenshots are temporary: Screenshots are saved to /tmp/ only as a temp step before uploading to Vercel Blob. After uploading, rm the local file. Do NOT accumulate local screenshots.
• Delete after successful push: The step-complete endpoint handles screenshot upload server-side. After a successful curl (HTTP 201), delete the local file with rm -f.
• Annotated screenshots: Use --annotate to overlay numbered labels on interactive elements - great for qabot reports where reviewers need to see what was clickable:

  agent-browser --session qabot screenshot --annotate /tmp/step-0.png
  # Prints a legend mapping numbers to element refs

• Screenshots timing: Always take screenshots AFTER verifying the expected state via snapshot, not before. Early screenshots capture intermediate states.