Woo AI Smoke

You are running the /woo-ai-smoke pipeline end-to-end. Follow the SKILL.md
at .claude/skills/woo-ai-smoke/SKILL.md as your reference for the Swift
template, parse protocol, hard invariants, rubric, JSONL format, and
reporting format. Everything below is your SCOPED task.

Inputs:
- mode: rest           # only "rest" is wired up; MCP support is deferred
- suite: SUITE          # "default" (24 scenarios × N samples) or ad-hoc "t1; t2"
- samples: N            # 1 for ad-hoc, 3 for default
- baseline: BASELINE    # path to baseline JSONL to compare against
- run_label: LABEL      # short tag for the stored run file, e.g. "post_prompt_revision"
- head_sha: SHA         # from `git rev-parse --short HEAD`
- branch: BRANCH        # from `git branch --show-current`

Pipeline (execute in this order, no skipping). Arm a `trap` cleanup at the start so a build crash never leaves the temp Swift file or log behind:

```bash
trap 'rm -f Modules/Tests/WooAIAssistantTests/SmokeRespondContractTests.swift /tmp/woo-ai-smoke.log /tmp/woo-ai-smoke-store.env' EXIT

Keep the subagent dispatch in a single Task tool call. Never split the
pipeline into multiple subagent turns — the parse state has to stay
inside the subagent's context.

## How it works

1. **Load scenarios** — default suite (24 scenarios) from `baseline.json`, or ad-hoc via `scenario "turn1; turn2"`.
2. **Verify credentials** in `~/.woo-ai-smoke/store.env` (see "Credentials" below). Swift reads the dotenv directly each run.
3. **Preflight fixtures** for the selected scenarios. Verify/create smoke-owned products, orders, and customers through the WooCommerce REST API before running the model.
4. **Write `Modules/Tests/WooAIAssistantTests/SmokeRespondContractTests.swift`** using the template below. Scenarios get expanded into the `@Test(arguments:)` parametrised suite.
5. **Run the smoke via `xcodebuild`**, capture stdout.
6. **Parse each `[smoke|...]` line** into a turn record — prompt, tool names, tool arg snippets, tool results, assistant text, card kinds.
7. **Claude judges each turn** against the scenario's `rubric_notes` and the global rubric (details below). Fill in scores per dim.
8. **Apply hard invariants** (deterministic pass/fail).
9. **Write run** to `.claude/skills/woo-ai-smoke/runs/<ISO-timestamp>_<sha>.jsonl`.
10. **Compare to baseline** — flag REGRESSION when hard invariants fail or rubric mean drops below `rubric_pass_threshold`.
11. **Report** a markdown table + summary counts.
12. **Delete** the temp Swift file, `/tmp/woo-ai-smoke.log`, and the `/tmp/woo-ai-smoke-store.env` mirror (via the `trap` armed at the start of the run).

## Prerequisites

- Xcode + iOS simulator (the project's `bootstrap` skill covers this).
- A WooCommerce demo store with an admin **application password** (for the REST tool calls) and an authenticated iOS app session whose WPCOM OAuth bearer can be captured (for the woo-mobile-ai LLM calls).
- Required CLI tools (all macOS-default): `xcodebuild`, `xcrun simctl`, `open`.
- Store credentials in **`~/.woo-ai-smoke/store.env`** with `WOO_SITE_URL`, `WOO_SITE_ID`, `WOO_USERNAME`, `WOO_APP_PASSWORD`, and `WOO_DOTCOM_ACCESS_TOKEN`. On first run the skill scaffolds the file with placeholders and opens it for editing — see Credentials below.

The skill never commits credentials. Swift reads `~/.woo-ai-smoke/store.env` directly so nothing leaks to `/tmp`.

## Credentials

The engineer maintains `~/.woo-ai-smoke/store.env` (the source of truth, dotenv format). The skill stages a `/tmp/woo-ai-smoke-store.env` mirror at run-start because the iOS simulator process sandboxes `~` to its own container and can't read the host's home directly; the `trap` cleanup deletes the `/tmp` mirror at run-end. Swift reads from `/tmp/woo-ai-smoke-store.env`.

The harness sends LLM traffic through the wpcom `woo-mobile-ai` backend wrapper using a captured iOS-app WPCOM OAuth bearer (`WOO_DOTCOM_ACCESS_TOKEN`). For pre-merge testing the engineer can route locally via mitmproxy, `/etc/hosts`, or a temporary hardcoded URLSession in the harness (not committed); the committed code only ships production-URL routing because nginx on the wpcom sandbox vhost rejects requests whose `Host` header isn't `public-api.wordpress.com`. REST tool calls still hit the merchant store directly with the application password.

**First-run flow**: if `~/.woo-ai-smoke/store.env` doesn't exist, scaffold it with placeholders, open it for the engineer to fill in, then stop. The engineer saves the file and re-runs the skill.

```bash
ENV_FILE="$HOME/.woo-ai-smoke/store.env"
STAGED_ENV="/tmp/woo-ai-smoke-store.env"

# First run: scaffold the file with placeholders, open it for editing, stop.
if [ ! -f "$ENV_FILE" ]; then
  mkdir -p "$(dirname "$ENV_FILE")"
  cat > "$ENV_FILE" <<'TEMPLATE'
# Woo AI smoke credentials - fill these in, save, then re-run the smoke skill.
# WOO_SITE_ID is the WordPress.com blog id of the demo store. Find it in
# wp-admin/options-general.php?page=jetpack or via the Jetpack AI JWT mint.
WOO_SITE_URL=https://your-demo-store.example.com
WOO_SITE_ID=123456
WOO_USERNAME=your-admin-username
WOO_APP_PASSWORD=xxxx xxxx xxxx xxxx xxxx xxxx
# WPCOM OAuth bearer captured from an authenticated iOS app session. Required
# for the woo-mobile-ai LLM path. Grab it by inspecting any /me request the
# app issues.
WOO_DOTCOM_ACCESS_TOKEN=
TEMPLATE
  chmod 600 "$ENV_FILE"
  open "$ENV_FILE"
  echo "Created $ENV_FILE with placeholders. Fill it in, save, then re-run the skill." >&2
  exit 0
fi

# Stage a /tmp mirror the simulator process can read; trap deletes it at run-end.
cp "$ENV_FILE" "$STAGED_ENV"
chmod 600 "$STAGED_ENV"

"fixtures": {
  "products": [
    {
      "sku": "woo-ai-smoke-winter-jacket",
      "name": "Woo AI Smoke Winter Jacket",
      "type": "simple",
      "status": "publish",
      "regular_price": "89.00",
      "manage_stock": true,
      "stock_quantity": 7,
      "stock_status": "instock"
    }
  ]
}

from pathlib import Path

def read_store_env(path=Path.home() / ".woo-ai-smoke/store.env"):
    values = {}
    for raw in path.read_text().splitlines():
        line = raw.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        key, value = line.split("=", 1)
        values[key.strip()] = value.strip().strip('"').strip("'")
    return values

curl -fsS -u "$WOO_USERNAME:$WOO_APP_PASSWORD" \
  "$WOO_SITE_URL/wp-json/wc/v3/products?sku=woo-ai-smoke-winter-jacket"

import Foundation
import Testing
@testable import WooAIAssistant

struct SmokeRun {
    struct Scenario {
        let id: String
        let category: String
        let turns: [Turn]
    }
    struct Turn {
        let prompt: String
        let autoDeclineWrites: Bool
    }

    static let samplesPerScenario = SAMPLES_PLACEHOLDER  // 1 for ad-hoc, 3 for default

    static let scenarios: [Scenario] = [
        // FILLED IN by the skill from baseline.json or the ad-hoc args.
        // Each scenario expanded to `samplesPerScenario` arguments to
        // @Test so Swift Testing parallel-runs them.
    ]

    // Expand each scenario N times so swift-testing runs N independent
    // samples per scenario in parallel.
    static let expanded: [(Scenario, Int)] = scenarios.flatMap { s in
        (1...samplesPerScenario).map { (s, $0) }
    }

    @Test(arguments: expanded)
    func runScenario(_ arg: (scenario: Scenario, sample: Int)) async throws {
        guard let creds = WooAssistantHeadless.credentialsFromStoreEnv() else { return }
        let harness = WooAssistantHeadless(credentials: creds)
        for (index, turn) in arg.scenario.turns.enumerated() {
            let turnNum = index + 1
            // `resolveConfirmation` returns a `ConfirmationDecision` for each
            // pending confirmation. `.decline` blocks the write. If
            // `autoDeclineWrites` is true we must return `.decline`. Getting
            // this inverted means the demo store actually mutates
            // (destructive writes get approved).
            let resolver: WooAssistantHeadless.ConfirmationResolver = { _ in
                turn.autoDeclineWrites ? .decline : .approve
            }
            let result: WooAssistantHeadless.ConversationTurnResult
            do {
                result = try await harness.send(turn.prompt, resolveConfirmation: resolver)
            } catch {
                print("[smoke|#\(arg.scenario.id)|\(arg.scenario.category)|s\(arg.sample)|t\(turnNum)] THREW: \(error.localizedDescription)")
                return
            }
            Self.dump(scenario: arg.scenario, sample: arg.sample,
                      turn: turnNum, prompt: turn.prompt, result: result)
        }
    }

    static func dump(scenario: Scenario, sample: Int, turn: Int, prompt: String, result: WooAssistantHeadless.ConversationTurnResult) {
        let tools = result.toolCalls.map(\.name)
        let toolArgs = result.toolCalls.map { "\($0.name)(\($0.argumentsJSON.prefix(120)))" }
        let cards = Array(Set(result.cards.map(\.kind))).sorted().joined(separator: ",")
        let confirmations = result.confirmations.map { "\($0.toolName)[\($0.classification)]=\($0.decision)" }
        let fail = result.failureMessage ?? ""
        let textEscaped = result.assistantText
            .replacingOccurrences(of: "\n", with: "\\n")
            .replacingOccurrences(of: "\"", with: "\\\"")
        print("[smoke|#\(scenario.id)|\(scenario.category)|s\(sample)|t\(turn)] prompt=\"\(prompt)\" n=\(tools.count) tools=\(tools) toolArgs=\(toolArgs) cards=[\(cards)] confirmations=\(confirmations) fail=\"\(fail)\" text=\"\(textEscaped)\"")
    }
}

xcodebuild -workspace WooCommerce.xcworkspace \
  -scheme WooAIAssistant \
  -destination 'platform=iOS Simulator,name=iPhone 17' \
  test -only-testing:"WooAIAssistantTests/SmokeRun" 2>&1 \
  | tee /tmp/woo-ai-smoke.log \
  | grep -E "\[smoke\||passed after|failed after|Test run with|error:"