Grill Jev

You are generating a Jev question battery to interrogate an artifact.
The artifact is provided as state at key "artifact".

Generate between 20 and 50 questions. Scale the count to the artifact's
complexity — a 3-step bug fix needs fewer questions than a multi-service
migration plan.

For each question, choose the most appropriate Jev primitive:
- Noul: yes/no probability. Use when you want to know whether something
  is true or absent. Always add structured criteria (true/false with
  what + examples) when the boundary is non-obvious.
- Choice: pick one from a known set. Use for risk levels, categories,
  reversibility classifications.
- Score: position on a spectrum. Use for completeness, timeline
  realism, detection speed.

Use array compare instructions when two or more fields in the state
need to be compared side by side.

Focus questions on the actual content of the artifact. A plan with no
database steps needs no database migration questions. A plan with a
single deploy step needs no multi-service blast radius question.

Output a JSON dict of question_id -> question definition.

# File artifact
python3 scripts/grill-jev.py --file task_plan.md --mode plan

# Inline text
python3 scripts/grill-jev.py --text "$(cat task_plan.md)" --mode plan

# The executing LLM writes /tmp/grill-questions.json, then Jev evaluates it.
python3 scripts/grill-jev.py --file design.md --questions-file /tmp/grill-questions.json

GRILL-JEV FINDINGS — mode: plan — 31 questions (generated)
============================================================
HIGH SIGNAL (requires attention):
  [completeness/has_success_criteria] noul=0.89 TRUE — no measurable success criteria
    → add explicit success criteria: observable outcomes, not "it works"
  [risk/migration_live_safety] noul=0.84 TRUE — migration runs against live traffic
    → add maintenance window or use online migration tool

CATEGORY SUMMARY:
  completeness  2 findings   risk  1 finding

OVERALL READINESS: 1.4/3 — partially complete; address findings before executing

# Noul with structured criteria
"has_success_criteria": {
    "type": "noul",
    "instructions": {"question": "Does the plan define measurable success criteria?",
                     "inspect": "artifact"},
    "criteria": {
        "true":  {"what": "Specific, observable outcomes are named",
                  "examples": ["all tests pass", "p95 latency < 200ms"]},
        "false": {"what": "Success is vague or absent",
                  "examples": ["it works", "done"]}
    }
}

# Choice with what/not_for/examples
"overall_risk": {
    "type": "choice",
    "instructions": {"question": "What is the overall risk level?",
                     "focus": "Weigh irreversibility, dependency count, blast radius."},
    "criteria": {
        "low":  {"what": "Reversible, few dependencies, limited blast radius",
                 "not_for": "Any step that cannot be undone",
                 "examples": ["adding an optional config flag"]},
        "medium": {"what": "Some irreversibility or cross-system dependencies",
                   "examples": ["schema migration with rollback plan"]},
        "high": {"what": "Irreversible steps, wide blast radius",
                 "examples": ["deleting a table", "replacing auth system"]}
    }
}

# Score with summary/signals levels
"completeness": {
    "type": "score",
    "instructions": {"question": "How complete is this plan?",
                     "note": "Judge whether a competent engineer could execute it without guessing."},
    "criteria": [
        {"summary": "Critically incomplete",
         "signals": ["missing phases", "undefined terms", "no rollback"]},
        {"summary": "Partially complete",
         "signals": ["main path clear", "some steps vague"]},
        {"summary": "Mostly complete",
         "signals": ["all phases named", "minor details missing"]},
        {"summary": "Complete",
         "signals": ["all steps actionable", "success criteria defined"]}
    ]
}

# Noul with array compare instructions
"assumption_vs_reality": {
    "type": "noul",
    "instructions": {
        "question": "Do the plan's assumptions conflict with the known system context?",
        "compare": ["artifact", "context"],
        "focus": "Look for things the plan takes for granted that could be false."
    }
}

import subprocess

result = subprocess.run(
    ["python3", "scripts/grill-jev.py", "--file", plan_path, "--mode", "plan", "--questions-file", questions_path],
    capture_output=True, text=True
)
print(result.stdout)
# Non-zero exit when HIGH SIGNAL findings exceed threshold