Bat Story Eval

BAT Story Evaluation

You are the evaluator. Run the steps in order; each one uses the output of the step before it.

Parse Arguments

From $ARGUMENTS, extract:

• --baseline: REQUIRED. Git tag/branch of the released version (e.g., v6.6.1).
• --agents: Agent list (default: gemini). Comma-separated.
• --stories: Force specific pre-built story IDs (e.g., s01,s02). Overrides triage selection.
• --all-stories: Skip triage, run ALL pre-built stories.
• --keep-container: Keep HA containers alive after run for manual inspection.
• --model: Model for Claude agent (e.g., haiku, sonnet).

If $ARGUMENTS is --help or missing --baseline, show usage and stop:

/bat-story-eval --baseline v6.6.1
/bat-story-eval --baseline v6.6.1 --agents gemini,claude
/bat-story-eval --baseline v6.6.1 --stories s01,s02
/bat-story-eval --baseline v6.6.1 --all-stories --agents claude --model haiku

Step 0: Triage (Diff Analysis + Custom Story Design)

0a. Compute Diff

cd "$(dirname "$(git rev-parse --path-format=absolute --git-common-dir)")/worktree/uat-stories"
git diff <baseline>..HEAD -- src/ha_mcp/ --stat
git diff <baseline>..HEAD -- src/ha_mcp/ --name-only

Classify changed files:

• Tool modules (tools/tools_*.py): specific tool implementations changed
• Core code (client/, server.py, errors.py, tools/util_helpers.py): affects all tools
• Utilities (utils/, resources/): may affect all tools
• No src/ changes: only tests/docs/config — select 2 smoke-test stories

0b. Select Pre-built Stories

Skip if --stories or --all-stories was passed.

1. Read the diff from 0a
2. Read all story YAMLs in tests/uat/stories/catalog/s*.yaml (title, description, prompt, setup)
3. For each story, reason about whether the diff could affect its outcome:
   • What tools/code paths would this story exercise?
   • Do any of those overlap with what changed?
4. Rules:
   • Story likely exercises changed code -> selected
   • Core code changed (client/, server.py, errors.py) -> all stories selected
   • No src/ changes -> 2 representative stories as smoke test
5. Report which stories were selected and why (one sentence per story)

0c. Design Custom Stories (at least 1)

Read the diff carefully. Your job is to catch regressions. For each changed code path NOT covered by selected pre-built stories, ask: "could this break something a user would notice?" If yes, design a custom story to test that hypothesis.

Guidelines: Always create at least 1 custom story. Each must test a distinct regression hypothesis — don't create stories that overlap. Stop when you've covered the risky gaps.

Write each as /tmp/custom_c<NN>.yaml using the standard story format:

id: c01
title: "Short description of what is being tested"
category: custom
weight: 5
description: >
  Rationale: [what changed in the diff and why this scenario tests it]

setup:
  - tool: ha_config_set_helper
    args:
      helper_type: "input_boolean"
      name: "Test Entity Name"

prompt: >
  [Natural language request a real user would make that exercises the changed code]

teardown: []

verify:
  questions:
    - "Did the agent achieve the expected outcome?"
    - "Did it use the expected tools?"

expected:
  tools_should_use:
    - ha_search
  description: >
    [What a correct agent should do]

Design principles:

• Focus on code paths that changed in the diff
• Plausible user scenarios, not synthetic edge cases
• Setup creates realistic HA state via FastMCP in-memory steps
• Prompts are what a real user would type
• At least 1. Each tests a distinct regression hypothesis. Stop when gaps are covered.

Step 1: Run Baseline Version

For EACH agent, run all stories against the baseline version. One container per agent, reused across all stories.

1a. Start container with first story

cd "$(dirname "$(git rev-parse --path-format=absolute --git-common-dir)")/worktree/uat-stories"
uv run python tests/uat/stories/run_story.py \
  tests/uat/stories/catalog/<first_story>.yaml \
  --agents <agent> --keep-container \
  --branch <baseline> \
  --results-file local/uat-results.jsonl

CAPTURE from stderr: HA URL (e.g., http://localhost:32771), token, session file path.

1b. Verify, then run remaining pre-built stories

After each story, verify via ha_query.py using the story's verify.questions:

uv run python tests/uat/stories/scripts/ha_query.py \
  --ha-url http://localhost:PORT --ha-token TOKEN \
  --agent <agent> \
  "Does an automation with alias 'Sunset Porch Light' exist?"

Record each answer as confirmed / denied / unclear. A non-zero exit from ha_query.py means the query itself failed (the output carries an [exit N] marker; [exit 124] is a timeout) — that is not one of the three outcomes; re-run it, and if it keeps failing record the story as unverified (Step 5) rather than scoring it. See references/evaluation-protocol.md.

Run remaining pre-built stories on the same container:

uv run python tests/uat/stories/run_story.py \
  tests/uat/stories/catalog/<next_story>.yaml \
  --agents <agent> --ha-url http://localhost:PORT --ha-token TOKEN \
  --branch <baseline> \
  --results-file local/uat-results.jsonl

Verify each immediately after running.

1c. Run custom stories on same container

uv run python tests/uat/stories/run_story.py \
  /tmp/custom_c01.yaml \
  --agents <agent> --ha-url http://localhost:PORT --ha-token TOKEN \
  --branch <baseline> \
  --results-file local/uat-results.jsonl

Verify each via ha_query.py using the custom story's verify.questions.

1d. Stop container

Stop only the container kept in step 1a. PORT is the host port in its Container kept alive: http://localhost:PORT line:

docker stop $(docker ps -q --filter "publish=PORT")

Step 2: Run Target Version

Repeat Step 1 for the target (local code). Same stories, same order, fresh container.

The only difference: omit --branch so run_story.py uses local code.

uv run python tests/uat/stories/run_story.py \
  tests/uat/stories/catalog/<first_story>.yaml \
  --agents <agent> --keep-container \
  --results-file local/uat-results.jsonl

Same container reuse for remaining stories (--ha-url). Same verification after each.

Step 3: White-Box Analysis

For each story on each version, read the session file captured during the run.

Gemini sessions (JSON):

python3 -c "
import json, sys
data = json.load(open(sys.argv[1]))
for msg in data.get('messages', []):
    for tc in msg.get('toolCalls', []):
        print(f\"  {tc['name']} ({tc.get('status', '?')})\")
" /path/to/session.json

Claude sessions (JSONL):

python3 -c "
import json, sys
for line in open(sys.argv[1]):
    entry = json.loads(line)
    if entry.get('type') == 'assistant':
        for b in entry.get('message', {}).get('content', []):
            if b.get('type') == 'tool_use':
                print(f\"  {b['name']}\")
" /path/to/session.jsonl

Compare against expected.tools_should_use:

• All expected tools used? (High weight)
• Tool failures with recovery? (Medium weight)
• Total tool call count (Low weight, note it)

Step 4: Score & Compare

Scoring Matrix