Hns Lsel Curator

hns-lsel-curator — LSEL CLUSTER + drain engine

Namespace: hns-lsel-* is user-owned dogfood (CLAUDE.local.md §24). This skill is NOT mirrored into internal/template/templates/ — it lives only in this repo. Graduation to moai-lsel-* + 16-language distribution is a separate SPEC (out of scope per spec.md §G).

M1 scope: drain + cluster + stage candidates. NO APPROVE, NO APPLY (M3). M2 scope: drain + cluster + PROPOSE shadow (no APPROVE, no APPLY). The PROPOSE stage emits shadow proposals + self-critiques; APPROVE/APPLY land in M3 via the fresh hns-lsel-applier path. M2 does NOT write to memory/ — the first feedback_*.md topic file is an M3+ deliverable after APPROVE.

What this skill does

The MoAI-ADK repo accumulates failure-event stubs in .moai/lessons-inbox.jsonl — a moving target; the measured composition and its dated baseline live in the scope anchor document (.moai/docs/learning-channel-scope.md), never in prose here. The constitution names the orchestrator as the drain actor, but until this skill there was zero mechanical drain code — the drain existed only as a doctrine paragraph (moai-constitution.md:147). This skill closes that gap in user-owned surfaces, without touching the frozen Go applier (internal/harness/applier.go:22 — its write-flag stays false; REQ-LSEL-003: bypass, never unfreeze).

Channel scope (bounded claim): the inbox records failure-event stubs only — the two wired families tool_failure:<tool>:<sig> and test_fail:<pkg>: (the test-fail family is wired via the evidence writer's test-fail path; the writer itself is untouched here). It does not capture defect families that produce neither a tool failure nor a test failure — those travel through the human-mediated loop (lane discovery → lead judgment → auto-memory feedback_*.md + MEMORY.md record), which is their learning channel. Numbers live only in the anchor document.

The drain is split into a mechanical core (drain.sh, deterministic, testable) and a model-mediated layer (this SKILL.md + your judgment, invoked for M2+ importance refinement and proposal drafting).

The mechanical core — drain.sh

drain.sh is a portable bash + jq script that lives next to this SKILL.md. It performs the deterministic half of the drain:

drain.sh --inbox <path-to-lessons-inbox.jsonl> --state-dir <path-to-lsel-state>

Pipeline (REQ-LSEL-009 + AC-LSEL-009 / AC-LSEL-010):

1. Companion offset — read <state-dir>/drain-offset.json (seed {"offset":0} if absent). The inbox is append-only and is NEVER mutated; the offset marks consumed stubs (SPEC-HARNESS-RATCHET-REWIRE-001 D3 companion-offset pattern).
2. Slice — read stubs from the offset onwards (tail -n +<offset+1>).
3. Drain-side severity filter (AC-LSEL-010) — discard noise BEFORE clustering:
   • tool_failure:Bash:UnknownFailure — the opaque ~65% timeout/sandbox bucket (the dominant noise share; report §2).
   • tool_failure:Bash:SandboxViolation — environment constraint, not a code defect.
   • any *:TimeoutError (Bash + MCP timeouts). The filter is drain-side because internal/hook/failure_observer.go (the inbox writer) is OUTSIDE the six loop-writable surfaces (plan.md §F.1 [DECISION RESOLVED]), so the loop cannot edit the writer — it filters on read instead.
4. Cluster by event_key with frequency count, first/last seen, and up to 3 sample summaries.
5. Singleton gate — discard clusters with frequency < 2 (single-occurrence noise per the constitution Lessons Protocol drain paragraph).
6. Importance — score each survivor with a Generative-Agents-style 1-10 gate: importance = min(10, frequency) (frequency as proxy; the model augments this in M2+ with a severity hint and retrieval-weighted judgment).
7. Emit candidates to <state-dir>/clusters.json; advance the companion offset.

clusters.json schema

{
  "drained_at": "2026-08-04T08:41:00Z",
  "offset_before": 0,
  "offset_after": 624,
  "total_read": 624,
  "noise_discarded": 533,
  "singletons_discarded": 4,
  "candidates": [
    {
      "event_key": "tool_failure:Agent:UnknownFailure",
      "frequency": 41,
      "first_seen": "...",
      "last_seen": "...",
      "sample_summaries": ["...", "...", "..."],
      "source": "tool:Agent",
      "importance": 10
    }
  ]
}

Empty-delta no-op

If the inbox has not grown past the offset, drain.sh writes an empty-candidate clusters.json and leaves the offset unchanged. Not a failure (acceptance.md §E edge case).

The model-mediated layer (you, when invoked)

drain.sh produces the deterministic candidate set. When this skill is invoked for a real curation pass (M2+), your job on top of the mechanical output is:

• Read clusters.json and rank candidates by importance then frequency.
• Augment importance with a severity hint the mechanical core cannot see: a recurring Bash:ExitError cluster points at a real command-shape defect (high signal); a recurring Agent:ContextCancelled cluster may be session-teardown noise (lower signal). Record the rationale in the candidate's prose when you draft the M2 proposal — do NOT rewrite clusters.json (it is the mechanical artifact; your augmentation lives in the proposal).
• Do NOT write to memory/ in M1. Candidates stage in clusters.json only. The first feedback_*.md topic file is produced by the M2 PROPOSE stage after retrieval-before-propose and self-critique (REQ-LSEL-010).

What this skill does NOT do (M1 boundaries)

• No APPROVE / APPLY — the parallel user-owned applier (hns-lsel-applier) is M3.
• No edits to frozen doctrine — .claude/rules/moai/**, AGENTS.md, internal/template/templates/**, retained agents, moai-* skills, and the frozen Go applier / curator_dispatch.go are all byte-for-byte untouched (REQ-LSEL-001 / §B.3).
• No new .moai/config/sections/ file — loop state lives under .moai/state/lsel/ (a new section file would be wiped on moai update; plan.md §B.4 / AP-LSEL-005).
• No orchestrator-only synchronous user-question channel — this is a subagent-owned mechanism skill; it never invokes the orchestrator's user gate. On a missing input, return a structured blocker report; the orchestrator runs the user gate (askuser-protocol.md).

Durable operations — the session-start trigger (session_drain.sh)

The drain's original "schedule" was a session-scoped /loop recipe that died with its owning session (2026-08-04) and executed nothing even while alive — the inbox stalled for 3 weeks with nobody notified (SPEC-LSEL-DRAIN-STALL-001 §B). The trigger is now mechanical:

ALL drains route through session_drain.sh (next to this SKILL.md), never drain.sh directly. The wrapper adds what the frozen core deliberately lacks: an exclusive drain lock (contention = safe no-op), an unconditional archive of any existing clusters.json to clusters-history/ BEFORE any overwrite (a direct drain.sh call bypasses archiving and can silently discard staged candidates — drain.sh overwrites clusters.json on both the drain path and the no-op path), a one-line status, and fail-open (any internal error degrades to a stderr notice and exit 0 — the hook never blocks session start).

session_drain.sh [--inbox <path>] [--state-dir <dir>]   # defaults: live paths

PROPOSE reads the archived copies in .moai/state/lsel/clusters-history/ (newest first), NOT the live clusters.json — under per-session-start drains the live file is ephemeral: the next no-op session start overwrites it with candidates: []. The wrapper's archive is what survives.

Local wiring is a maintainer-machine deliverable (applied in M2, NOT carried by the PR — a tracked .claude/settings.json entry would be wiped by every moai update, so tracked wiring is affirmatively wrong): add BOTH session_drain.sh and backlog_check.sh to .claude/settings.local.json .hooks.SessionStart with an explicit "timeout": 30 each (the measured full-backlog drain is <1s for a 1.1MB inbox; 30s matches the live SessionStart hook precedent). Wiring verification: jq '.hooks.SessionStart' .claude/settings.local.json.

Removed dead anchors: the former CLAUDE.local.md section-28 anchor references in backlog_check.sh (header comment + reminder body — both occurrences) were removed in SPEC-LSEL-DRAIN-STALL-001 M1; the operating instructions now live in this section, mirrored into the restored CLAUDE.local.md LSEL section (M2 local deliverable).

Verification (run before declaring a drain complete)

# 1. The drain mechanics + the wrapper (fixture-based characterization tests —
#    AC-LSEL-009/010 and AC-LDS-001..006 + the mutant probe):
.claude/skills/hns-lsel-curator/drain_test.sh
.claude/skills/hns-lsel-curator/session_drain_test.sh

# 2. A real drain of the live backlog — VIA THE WRAPPER (all drains are
#    wrapper-mediated; re-measure first, the inbox is a moving target). Capture
#    BEFORE the drain, then judge the ARCHIVED copy: a later no-op session-start
#    drain overwrites the live clusters.json with candidates: [], but the wrapper
#    archives unconditionally before any overwrite, so the bulk-drain result
#    survives in clusters-history/.
OFFSET_BEFORE=$(jq -r .offset .moai/state/lsel/drain-offset.json)
LIVE_COUNT=$(wc -l < .moai/lessons-inbox.jsonl | tr -d ' ')
.claude/skills/hns-lsel-curator/session_drain.sh --inbox .moai/lessons-inbox.jsonl --state-dir .moai/state/lsel
ARCHIVE=$(ls -t .moai/state/lsel/clusters-history/clusters-*.json | head -1)
jq --argjson n "$LIVE_COUNT" --argjson b "$OFFSET_BEFORE" \
   '(.offset_after == $n) and ((.candidates // []) | length >= 1) and (.total_read == ($n - $b))' "$ARCHIVE"
# must print: true  (offset==live AND candidates>=1 AND self-consistent — the
# AC-LDS-010 predicate; the session_drain_test.sh mutant probe proves it rejects
# an offset-only-advance fake)

# 3. M1 invariant — zero memory/ writes from the drain:
find memory -newer <drain-start-timestamp> -name 'feedback_*' 2>/dev/null | wc -l   # must be 0

Characterization test

drain_test.sh (next to this SKILL.md) is the TDD RED→GREEN harness. It builds a synthetic inbox with known noise + signal stubs, runs drain.sh, and asserts the drain semantics: noise excluded pre-cluster, signal clustered with correct frequencies, singletons discarded, offset advanced, candidates emitted, zero memory/ writes, idempotent re-drain. Run it after any edit to drain.sh. session_drain_test.sh (same directory) is the wrapper harness — five paths (drain / lock contention / archive-before-overwrite / no-op / fail-open) plus the mutant probe. Run it after any edit to session_drain.sh.

Cross-references

• SPEC: .moai/specs/SPEC-LSEL-LOCAL-EVOLUTION-001/{spec,plan,acceptance,progress}.md
• Design report (SSOT): .moai/reports/moai-local-self-evolution-design-20260804.html §6 stage 2 (CLUSTER), §10 P1, §11 mustFix B#1/B#3.
• Frozen applier (reference only): internal/harness/applier.go:22 (the write-flag, kept false), internal/harness/curator_dispatch.go.
• Constitution drain paragraph (the "0 Go code" stub this skill replaces): .claude/rules/moai/core/moai-constitution.md:147.
• Namespace guard: internal/template/split_namespace_test.go, internal/template/internal_content_leak_test.go (extended in M2 — AC-LSEL-006).

PROPOSE stage (M2 — shadow proposals)

The PROPOSE stage consumes the candidate clusters archived in .moai/state/lsel/clusters-history/ (newest archive — the live clusters.json is ephemeral under per-session-start drains: the next no-op drain overwrites it with candidates: []; SPEC-LSEL-DRAIN-STALL-001 REQ-LDS-010) and emits shadow proposals — one per candidate worth acting on — at .moai/state/lsel/proposals/<proposal-id>/. M2 proposals are SHADOW only: no APPROVE, no APPLY. APPROVE/APPLY land in M3 via the fresh hns-lsel-applier path (NOT via the dead moai-harness-learner Tier-4 flow — see "Tier-4 finding" below).

Retrieval-before-propose (Reflexion)

BEFORE drafting a proposal, retrieve relevant feedback_*.md topic files from ~/.claude/projects/<hash>/memory/. The retrieval grounds the proposal in prior lessons (Reflexion-style) and is evidenced in the proposal's retrieval_evidence block. A proposal without retrieval evidence is malformed and MUST NOT be emitted.

Proposal payload schema (AC-LSEL-011)

Each proposal lives at .moai/state/lsel/proposals/<id>/ and contains exactly:

proposal.md YAML frontmatter carries the full schema (8 required keys):

---
proposal_id: lsel-001
target_surface: <one of the 6 evolvable surfaces, spec.md §B.3>
rationale: |
  <what + why>
WHY-not-just-WHAT: |
  <the reasoning, not just the change — catches "what" proposals that skip the "why">
prediction: <a FALSIFIABLE expected effect — the verify_command must be able to falsify it>
verify_command: <a runnable command that, if green, confirms the prediction>
blast_radius: <which surfaces the diff touches; used by the CSA forced-gate match>
memory_type: semantic|procedural|episodic   # CoALA taxonomy
retrieval_evidence:
  - <path to a feedback_*.md retrieved before drafting>
status: blocked   # blocked | ready — blocked if self-critique has an UNRESOLVED objection
---