Prove Feature

Prove a feature works (or doesn't)

Build a throwaway project and exercise a prove_it feature through the real dispatcher pipeline. The output is a human-readable transcript the user can read to confirm the system works end-to-end.

What "prove" means — read this first

Proving a feature means watching the feature do its actual job, not just watching the dispatcher accept a config and return a decision.

If the feature is a reviewer that detects dead code, you must:

1. Create a project that contains dead code → run the reviewer → see it catch the dead code
2. Create a project that has no dead code → run the reviewer → see it pass clean

If the feature is a task that validates API design, you must:

1. Write an API file with real design violations → see the task reject it
2. Write a clean API file → see the task approve it

If the feature is a when-condition gate, you must:

1. Run with the condition unmet → see the task get skipped
2. Run with the condition met → see the task actually execute and produce its real output

The pattern is always the same: construct a realistic situation where the feature's logic is exercised, then observe it both succeed and fail. A test that only checks "did the dispatcher return allow/deny" without verifying the feature itself inspected the right thing and made the right call is not a proof.

The critical question

Before writing any code, answer this: "What real-world situation does this feature exist to handle, and how will I simulate that situation in a throwaway project?"

If you can't answer that, you don't understand the feature well enough to prove it yet. Stop and think harder.

Anti-patterns — do NOT do these

Plumbing-only tests. Testing that the dispatcher routes to a script and returns the script's exit code is testing prove_it's plumbing, not the feature. The feature is what the script does. You must create input that makes the script's logic actually fire.

Trivial scripts. Writing #!/bin/bash\nexit 0 and #!/bin/bash\nexit 1 proves the dispatcher handles exit codes. It proves nothing about the feature. The script must contain the feature's real logic (or a faithful stand-in), and the test input must be realistic enough to exercise it.

Success-only tests. If you only prove the feature passes, you haven't proved it works — you may have proved it does nothing. A reviewer that always says "looks good" is broken. Always prove the feature can fail/reject/deny before proving it can pass/approve/allow. Failure-first is how you know the feature has teeth.

Config-exists tests. Writing a config, loading it, and checking it parsed correctly is a config test, not a feature test. The feature is what happens after the config is loaded.

Arguments

<feature description> — a short description of what to prove. Examples:

• "the dead-code reviewer catches unused functions"
• "the API design validator rejects endpoints without error schemas"
• "script appeal prevents doom loops"
• "multi-when arrays are logical OR"
• "backchannel writes bypass PreToolUse enforcement"
• "linesChanged threshold triggers at the right count"

Method

Step 1: Design the scenario BEFORE writing code

This is the most important step. Think through:

1. What does the feature actually do? Not what config key enables it — what real-world thing does it detect, enforce, or transform?

2. What does a realistic failing input look like? Build a project state that the feature should catch. Be specific: if it's a code reviewer, write actual bad code. If it's a design validator, write an actual bad design. If it's a file-size gate, create an actual large file.

3. What does a realistic passing input look like? Build the "clean" version of the same scenario. Same project structure, but with the problem fixed.

4. What side effects should you observe? Beyond pass/fail decisions, what should the feature produce? Error messages with specific details? Log entries? Modified files? The proof should verify these too — they're how you know the feature understood the input, not just guessed.

Write down your scenario plan as comments in the script before implementing it.

Step 2: Write a self-contained Node.js test script

Create a single file at /tmp/prove_it_feature_<name>/prove.js that does everything: creates the temp project, writes config, builds the realistic scenario, runs dispatches, and prints results. This is not a node:test test — it's a standalone script that prints a human-readable transcript.

Use this skeleton:

#!/usr/bin/env node
'use strict'

const { spawnSync } = require('child_process')
const fs = require('fs')
const path = require('path')
const os = require('os')

// ── Configuration ──

// Choose the dispatcher source:
//   LOCAL SHIM: tests the working tree (for pre-release validation)
//   RELEASE:    tests /opt/homebrew/bin/prove_it (for proving shipped features)
const REPO_ROOT = '<prove_it repo root>'
const USE_LOCAL = <true|false>
const PROVE_IT = USE_LOCAL
  ? path.join(REPO_ROOT, 'test', 'bin', 'prove_it')
  : '/opt/homebrew/bin/prove_it'

// Isolated environment — never touches real user config
const PROJECT_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'prove_feature_'))
const FAKE_HOME = fs.mkdtempSync(path.join(os.tmpdir(), 'prove_home_'))
const PROVE_IT_DIR = path.join(FAKE_HOME, '.prove_it')
const SESSION_ID = `prove-${Date.now()}`

// ── Helpers ──

function gitIn (dir, ...args) {
  spawnSync('git', args, { cwd: dir, encoding: 'utf8' })
}

function writeFile (relPath, content) {
  const full = path.join(PROJECT_DIR, relPath)
  fs.mkdirSync(path.dirname(full), { recursive: true })
  fs.writeFileSync(full, content)
}

function makeExecutable (relPath) {
  fs.chmodSync(path.join(PROJECT_DIR, relPath), 0o755)
}

function writeConfig (config) {
  const cfgPath = path.join(PROJECT_DIR, '.claude', 'prove_it', 'config.json')
  fs.mkdirSync(path.dirname(cfgPath), { recursive: true })
  fs.writeFileSync(cfgPath, JSON.stringify(config, null, 2))
}

function invoke (hookSpec, input, extraEnv = {}) {
  const env = {
    PATH: USE_LOCAL ? `${REPO_ROOT}/test/bin:${process.env.PATH}` : process.env.PATH,
    HOME: FAKE_HOME,
    CLAUDE_PROJECT_DIR: PROJECT_DIR,
    PROVE_IT_DIR,
    ...extraEnv
  }
  const result = spawnSync(PROVE_IT, ['hook', hookSpec], {
    input: JSON.stringify(input),
    encoding: 'utf8',
    env,
    cwd: PROJECT_DIR,
    timeout: 30000
  })
  let output = null
  try { output = JSON.parse(result.stdout) } catch {}
  return {
    exitCode: result.status,
    stdout: result.stdout || '',
    stderr: result.stderr || '',
    output
  }
}

function invokePreToolUse (toolName, toolInput, extraEnv = {}) {
  return invoke('claude:PreToolUse', {
    session_id: SESSION_ID,
    tool_name: toolName,
    tool_input: toolInput,
    cwd: PROJECT_DIR
  }, extraEnv)
}

function invokeStop (extraEnv = {}) {
  return invoke('claude:Stop', {
    session_id: SESSION_ID,
    hook_event_name: 'Stop',
    cwd: PROJECT_DIR
  }, extraEnv)
}

function decision (result) {
  return result.output?.hookSpecificOutput?.permissionDecision
    || result.output?.decision
    || '(silent)'
}

function reason (result) {
  return result.output?.hookSpecificOutput?.permissionDecisionReason
    || result.output?.hookSpecificOutput?.message
    || result.output?.reason
    || ''
}

// ── Logging ──

const PASS = '\x1b[32mPASS\x1b[0m'
const FAIL = '\x1b[31mFAIL\x1b[0m'
const INFO = '\x1b[36mINFO\x1b[0m'
const WARN = '\x1b[33mWARN\x1b[0m'
let totalPass = 0
let totalFail = 0

function header (text) {
  console.log(`\n\x1b[1m── ${text} ──\x1b[0m\n`)
}

function info (text) {
  console.log(`  ${INFO}  ${text}`)
}

function check (label, condition, detail) {
  if (condition) {
    console.log(`  ${PASS}  ${label}`)
    totalPass++
  } else {
    console.log(`  ${FAIL}  ${label}`)
    if (detail) console.log(`         ${detail}`)
    totalFail++
  }
}

function printRaw (label, result) {
  const d = decision(result)
  const r = reason(result).split('\n')[0].slice(0, 120)
  console.log(`         ${label}: decision=${d}  reason=${r}`)
}

// ── Setup ──

gitIn(PROJECT_DIR, 'init')
gitIn(PROJECT_DIR, 'config', 'user.email', 'prove@test')
gitIn(PROJECT_DIR, 'config', 'user.name', 'Prove')
writeFile('README.md', '# Test project\n')
gitIn(PROJECT_DIR, 'add', '.')
gitIn(PROJECT_DIR, 'commit', '-m', 'init')

header('Environment')
info(`prove_it:   ${PROVE_IT}`)
info(`source:     ${USE_LOCAL ? 'local shim (working tree)' : 'release (Homebrew)'}`)
info(`project:    ${PROJECT_DIR}`)
info(`session:    ${SESSION_ID}`)
info(`PROVE_IT_DIR: ${PROVE_IT_DIR}`)

// ... scenario implementation goes here ...

// ── Session transcript ──

function printSessionLog () {
  const logFile = path.join(PROVE_IT_DIR, 'sessions', `${SESSION_ID}.jsonl`)
  header('Session Transcript')
  if (!fs.existsSync(logFile)) {
    console.log('  (no session log)')
    return
  }
  const lines = fs.readFileSync(logFile, 'utf8').trim().split('\n')
  console.log('  TIME      STATUS   TASK                 REASON')
  console.log('  ────────  ───────  ───────────────────  ──────────────────────────────')
  for (const line of lines) {
    try {
      const e = JSON.parse(line)
      const t = new Date(e.at).toISOString().slice(11, 19)
      const s = (e.status || '').padEnd(7)
      const n = (e.reviewer || '').padEnd(19)
      const r = (e.reason || '').split('\n')[0].slice(0, 60)
      console.log(`  ${t}  ${s}  ${n}  ${r}`)
    } catch {}
  }
}

function printSessionState () {
  const stateFile = path.join(PROVE_IT_DIR, 'sessions', `${SESSION_ID}.json`)
  if (!fs.existsSync(stateFile)) return
  header('Session State')
  console.log(JSON.stringify(JSON.parse(fs.readFileSync(stateFile, 'utf8')), null, 2)
    .split('\n').map(l => '  ' + l).join('\n'))
}

// ── Summary ──

function summary () {
  printSessionLog()
  printSessionState()
  header('Summary')
  if (totalFail === 0) {
    console.log(`  \x1b[32m✓ All ${totalPass} checks passed\x1b[0m\n`)
  } else {
    console.log(`  \x1b[31m✗ ${totalFail} of ${totalPass + totalFail} checks failed\x1b[0m\n`)
  }
}

// ── Cleanup ──

function cleanup () {
  fs.rmSync(PROJECT_DIR, { recursive: true, force: true })
  fs.rmSync(FAKE_HOME, { recursive: true, force: true })
}

Step 3: Implement the scenarios — failure first, then success

Fill in the // ... scenario implementation goes here ... section. Every scenario must follow this structure:

1. Build the realistic project state. Write actual source files, configs, scripts — whatever the feature needs to inspect. These must be realistic enough that the feature's logic is meaningfully exercised. A dead-code detector needs real code with real dead functions. A design validator needs a real API spec with real violations.

2. Prove the feature catches the problem (failure/deny case first). Run the dispatcher against the bad input. Assert not just the decision, but that the reason or output shows the feature understood what was wrong. A deny with a generic reason like "script exited 1" is not proof — the reason should reference the specific problem (e.g., "unused function oldHandler detected in src/routes.js").

3. Fix the problem in the project, then prove the feature approves. Modify the project to resolve the issue (remove the dead code, add the missing schema, etc.), then re-run. Assert that the feature now passes. This confirms the feature is actually sensitive to the input, not just randomly failing.

4. Check side effects. If the feature should produce logs, annotations, modified files, or specific error messages, verify those exist and contain the right content.

Follow these patterns for common feature types: