Distill Feedback

distill-feedback — close the learn-from-corrections loop

The Stop hook session-feedback-capture.py queues finished sessions into ~/.claude/feedback/queue.jsonl. This skill processes that queue: it finds the user turns that were durable corrections of the agent's work and turns them into rules — so the same correction never has to be given twice.

Why semantic review: standing preferences depend on context, not a trigger-word count. The formerly cited private effectiveness-test/RESULTS.md was absent when checked on 2026-09-06; its F1 claims are withdrawn from this skill until the dataset, labels, model, held-out split and raw predictions can be inspected. The rubric below is a review method, not a demonstrated accuracy guarantee. Do not substitute another paper's scores for our own.

Research and authority: ACE studies evolving context; TRACE studies compiling corrections into runtime checks. Neither proves this local extractor's accuracy. The approval boundary comes from our applicable user instructions and autonomy-risk-tiers.md, not an inferred paper mandate. New standing rules are proposals; an already authorized correction to an existing rule can be implemented within that exact authority. Codex memory changes also require the separate explicit user request and supported memory-update channel.

Procedure

1. Extract the queue (deterministic)

python ~/.claude/skills/distill-feedback/scripts/extract_feedback_queue.py --limit 8

Returns JSON: {pending, sessions:[{session_id, cwd, ts, user_turns:[...]}]}. --limit bounds the LLM pass (billing: distillation is opt-in, not every-session). If pending is 0, stop — nothing to do.

2. Detect durable corrections (LLM-semantic, prefer a fresh sub-agent)

For independence (Generator-Evaluator), spawn a sub-agent with the rubric below and the extracted user_turns. Ask it to return, per genuine correction: {quote, durable_rule, applicability_condition, confidence, session_id}. Pass only the turns — not your own reasoning.

RUBRIC — a user turn is a DURABLE CORRECTION if the user pushes back on / redirects the agent's behavior in a way that implies a STANDING preference or a mistake to avoid in future:

• explicit pushback / redirection ("no, do X instead", "wrong file again")
• reminder of a prior agreement ("we agreed you'd ask first", "мы же договаривались сначала бэкап")
• standing-preference marker ("from now on / always / never / by default / в следующий раз / впредь")
• frustration at a REPEATED mistake ("опять", "again", "you keep")
• polite redirection phrased as a question ("could you not overwrite latest.pth each time?")
• revert with a reason ("верни как было, твоя версия хуже")
• praise THEN correction — judge the whole turn ("great it runs, but always pin versions" = YES)

NOT a durable correction: new feature/task request · diagnostic question ("why did the build fail?", "почему-то падает") · factual/info statement even with "should be / by default / never" ("deploy should be done in 5 min", "по умолчанию там 8080") · agreement ("actually that makes sense, go ahead") · reassurance ("don't worry about the tests") · praise-only · off-topic chatter.

3. Dedup + draft atomic rules

For each detected correction: write it as ONE atomic rule with an applicability condition. Dedup against existing rules/memory (grep ~/.claude/rules/ and the project memory) — if it is already a rule, skip or propose an EDIT, not a new ADD. Cluster duplicates across sessions into one rule.

4. Resolve authority for the exact change

Show the user a compact table: each proposed rule + its applicability condition + source quote + target file + action (ADD new / EDIT existing / SUPERSEDE old / SPLIT). If that exact change is not already authorized, ask for approval and retain the proposal. Do not repeatedly request permission already given for the same in-scope correction. New always-on rules, SUPERSEDE and DELETE require the applicable explicit authority.