Bat Adhoc

What it does

When your agent uses it

• Detecting regressions
• Verifying tool changes end-to-end with Claude/Gemini CLIs

Example prompts

• “/bat-adhoc”

Requirements

• Python 3
• Pre-approved tools (allowed-tools): Bash, Read, Write

Workflow steps

6 steps, taken from the first numbered list in SKILL.md.

1. 1Analyze the change: Read the diff, identify which tools are affected
2. 2Design scenario: Generate a scenario JSON with setup/test/teardown prompts
3. 3Run the script: Pipe the scenario to uv run python tests/uat/run_uat.py
4. 4Evaluate summary: Check all_passed per agent. If true, you're done.
5. 5Dig deeper on failure: Read results_file for full output, stderr, raw JSON
6. 6Regression check: If test fails, re-run with --branch master to compare

What it can do on your machine

Read from SKILL.md and the folder at commit b274a93. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves these tools, so the agent can use them without asking each time:

  • Bash
  • Read
  • Write

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Shell commands in SKILL.md call:

  • uv

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md. Its commands use uv, which can reach the network depending on how they are called.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Bat Adhoc loads about 1.4k tokens when it runs. Until then it costs about 54 tokens; SKILL.md has 506 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~54

When it runs · the whole SKILL.md, loaded when a task matches

~1.4k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

• NotePre-approves every shell command (allowed-tools: Bash)SKILL.md

  allowed-tools: Bash, Read, Write

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from homeassistant-ai/ha-mcp at commit b274a93, republished under its MIT licence (© homeassistant-ai). 506 words, ~1,382 tokens.

name

bat-adhoc

description

Run bot acceptance tests to validate MCP tools work correctly from a real AI agent's perspective. Use when testing PRs, detecting regressions, or verifying tool changes end-to-end with Claude/Gemini CLIs.

allowed-tools

Bash, Read, Write

disable-model-invocation

true

argument-hint

scenario-description or --help

{
  "results_file": "/tmp/bat_results_abc123.json",
  "agents": {
    "gemini": {
      "all_passed": true,
      "test": {
        "completed": true,
        "duration_ms": 8100,
        "exit_code": 0,
        "num_turns": 5,
        "tool_stats": { "totalCalls": 4, "totalSuccess": 4, "totalFail": 0 }
      },
      "aggregate": {
        "total_duration_ms": 15300,
        "total_turns": 12,
        "total_tool_calls": 9,
        "total_tool_success": 9,
        "total_tool_fail": 0
      }
    }
  }
}

cat <<'EOF' | uv run python tests/uat/run_uat.py --agents gemini
{
  "setup_prompt": "Create a test automation called 'bat_error_test' with a time trigger at 23:59 and action to turn on light.bed_light.",
  "test_prompt": "Try to get automation 'automation.nonexistent_xyz'. Report if the tool signaled an error or returned a normal response. Then get automation 'automation.bat_error_test' and report its structure.",
  "teardown_prompt": "Delete automation 'bat_error_test' if it exists."
}
EOF

# Baseline: master
uv run python tests/uat/run_uat.py --scenario-file <scenario.json> --branch master --agents gemini

# Target: local code
uv run python tests/uat/run_uat.py --scenario-file <scenario.json> --agents gemini

/bat-adhoc test automation create with sunrise trigger then modify to sunset

© homeassistant-ai, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in .claude/skills/bat-adhoc of homeassistant-ai/ha-mcp.

Open the folder on GitHubat commit b274a93