Deciding With Confidence

Deciding with confidence

scripts/decide.py takes an OpenAI Decisions API request (input plus questions of type predicate, choice or score) and returns its response shape: a probability per option, the chosen value or score, and confidence. Claude Haiku 5.5 answers. Its logits are not exposed, so the distribution is estimated: each of k parallel samples states a probability per option with the options in a different order, the samples are averaged, and the average is temperature-scaled from a labelled eval. Disagreement between samples lowers confidence and shows up in diagnostics.agreement.

Expect about 85% of what a purpose-built decision model gets: on the bundled 104-item eval, accuracy 0.846 against Jev's 0.894, with a calibration error (ECE 0.041) that matches it. The whole accuracy gap was on eight-way intent questions; yes/no and rubric questions tied. Disagreement between samples is the strongest warning sign. references/method.md has the numbers.

When NOT to use this skill

Abandon a run when diagnostics.agreement is below 1 on most of a batch: the questions or option descriptions are ambiguous, and more samples will not fix them. Rewrite the options (distinct, observable criteria) before sampling more.

Procedure

1. Write the request as JSON in the OpenAI shape. Give every question a unique name. Options need descriptions that say when each applies; add an other choice when the set is not exhaustive. Score levels go lowest first.

   {"input": "I was charged twice for my order.",
    "questions": [
      {"type": "choice", "name": "department", "instructions": "Which department should handle this?",
       "choices": [{"value": "billing", "description": "Payments, invoices, refunds."},
                   {"value": "shipping", "description": "Delivery and tracking."},
                   {"value": "other", "description": "Anything else."}]},
      {"type": "predicate", "name": "angry", "instructions": "Is the customer expressing anger?"},
      {"type": "score", "name": "severity", "instructions": "How severe is this?",
       "levels": [{"label": "Cosmetic", "description": "No lost functionality."},
                  {"label": "Workaround", "description": "Fails, another way works."},
                  {"label": "Blocked", "description": "Fails with no workaround."}]}]}

2. Pick the transport. --transport auto (the default) takes the first that works:

   Transport	Needs	Wall time per request
   api	pip install anthropic and ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN	~0.7 s expected (the API time measured inside claude -p)
   cli	the claude CLI, logged in	~4 s, measured
   bedrock	pip install anthropic, AWS_REGION, AWS credentials; explicit only	as api

   With none of them, use the subagent path in step 3b.

3. Run it.

   a. With a transport:

   S=/mnt/skills/user/deciding-with-confidence/scripts   # or wherever the skill lives
   python3 $S/decide.py run request.json            # k=3 by default
   echo '{...}' | python3 $S/decide.py run -

   From Python: sys.path.insert(0, S); import decide; decide.decide(req) returns the response dict, or None when no sample was usable.

   b. In a Claude Code session, the plugin's decider subagent is the cheaper path: it is pinned to claude-haiku-5-5 (the model the calibration was fitted on) and carries its instructions inline, so it answers without a tool call. Install the deciding-with-confidence plugin, or copy agents/decider.md to .claude/agents/; agent definitions load at session start. Then:

   python3 $S/decide.py emit request.json -k 3 --agent deciding-with-confidence:decider > batch.json

   Use the name the Agent tool lists: deciding-with-confidence:decider from the plugin, decider from .claude/agents/. Send each samples[i].prompt verbatim as an Agent call with that subagent_type, all in one message. Collect the reply texts into a JSON list in replies.json, then python3 $S/decide.py aggregate batch.json replies.json. Pool before acting: a single reply carries no agreement signal and no calibration.

   Without the agent installed, omit --agent: the system prompt then rides inside each prompt for a stock general-purpose subagent with model: haiku. That works and costs about 58K tokens a sample against 9K.

4. Read the answer. Predicates give probability (P true). Choices give choice, probabilities and confidence. Scores give score (the probability-weighted level index, so 1.4 sits between levels 1 and 2), probabilities and confidence. Every answer carries diagnostics: agreement, spread, the pre-calibration raw pool and per_sample.

5. Act on it with thresholds, never on the top answer alone.

   • agreement < 1 (the samples' top answers differ): escalate. On the bundled eval, unanimous answers were 92% right and split ones 44%.
   • Top probability under 0.7: escalate. Together with the split rule that sent 23 of 104 items onward and left 6 wrong, assuming the escalation target got every escalated item right. The 0.7 was chosen on the same data, so treat it as a starting point.
   • Set the final thresholds from labelled examples of your own traffic, weighing the cost of a false positive against a false negative.

6. Calibrate on your own labels when you have 50 or more per question type:

   python3 $S/decide.py eval my_labelled.jsonl --out results.jsonl   # {id, input, questions, labels}
   python3 $S/decide.py calibrate results.jsonl --out my-cal.json
   export DECIDER_CALIBRATION=$PWD/my-cal.json

   The bundled assets/calibration.json fits only choice (T = 0.62, fitted through the cli transport). Predicate and score keep T = 1 until a type has 50 labels, because smaller sets swung between folds.