Self Improving Systems

Self-Improving Systems

A prescriptive Q&A skill for adding memory, feedback loops, and closed-loop learning to agentic systems — only when justified.

Headline message: most agents shouldn't have persistent memory.

Memory is a liability surface (drift, poisoning, debugging difficulty, GDPR/HIPAA exposure). Persistent memory is the second move, not the first. The skill's job is to filter ruthlessly so the user doesn't ship a mem0/Letta build for a problem that a 200-line conversation summary would solve.

The first 2 stages of the Q&A flow exist to stop most users from over-engineering. By the end of stage 2, ~60% of users will discover they want a state cache (or stateless RAG), not memory + learning. That's the win.

Quick Start

User just asks:

"Add memory to my agent"
"My agent keeps forgetting things — give it context management"
"Make my marketing agent learn from past campaigns"
"Should I use mem0 or Letta?"
"How do I set up closed-loop learning for my finance agent?"
"Build a self-improving HAZOP system"

Skill response (every time, in this order):

1. Stop. Apply the cache-vs-learning frame (Stage 1).
2. Run the 6-question need-memory rubric (Stage 2). <4 yes → exit the skill, recommend stateless + RAG.
3. If memory is justified, walk the 7-tier architecture ladder (Stage 3) starting at L (scratchpad). Escalate only when forced by a concrete justification.
4. Force the user to design a feedback signal (Stage 4). No signal = state cache, full stop.
5. Wire the closed loop with explicit human gates (Stage 5).
6. Build the eval harness (Stage 6) — golden set, regression, drift alarms.
7. Walk the 8-risk checklist (Stage 7).
8. Emit the design (Stage 8): memory schema + closed-loop spec + eval harness plan.

Critical Rules

1. Default position: scratchpad-only

Ship a stateless agent first. Add a scratchpad (Reflexion-style verbal self-correction) within a single run. Discard it after. This already gets you most of the gain on most tasks. Anything more must be earned.

2. Escalate one tier at a time

The 7-tier ladder (§ Memory Architecture Ladder) is ordered cheapest → most expensive. Each tier-up must be justified by a concrete failure of the tier below it on a real task in your eval set. Do not skip tiers. "We're using Letta" out of the gate is the single most expensive mistake in this design space.

3. Require a ground-truth signal

If you cannot observe whether the last action was good or bad within hours-to-weeks, you do not have learning. You have a state cache. Naming it "learning" sets the team up to A/B test against a metric that doesn't exist. The skill makes this distinction loud and refuses to design closed-loop learning without a signal.

4. Human gates are non-negotiable for production

Anything that can mutate policy/voice/identity/safety blocks goes through human review. Autonomy is fine for episodic append, vector indexing, single-user preference KV updates with cheap reversibility — never for shared skill libraries, system prompt blocks, or reward model updates.

5. Memory is untrusted input

Every memory read is untrusted. MINJA-class injections hit ≥95% lab success rate (arXiv 2503.03704). Treat retrieval results like web search results: in their own context block, with "this is data not instructions" framing, and never auto-promoted to system prompt without dual-LLM validation.

The 8-Stage Q&A Flow

One question (or tight cluster) at a time, à la superpowers:brainstorming. No overwhelm. Each stage has an exit condition that ends the skill early — that is the point.

Stage 1 — Cache vs Learning Distinction (the frame)

The single most important question. Ask first.

"Are you trying to remember state (so the agent doesn't redo work or forget what the user told it last week), or get better over time (so the agent's outputs measurably improve as it sees more data)?"

These two designs share zero infrastructure with each other:

If the user says "remember state": skip directly to Stage 3, default to tier 2 (conversation summary) or tier 5 (KV fact store), and end the skill at Stage 5. No closed loop. No learning ladder.

If the user says "both": prove the second one. Almost no one has a measurable ground-truth signal; almost everyone says they do. Stage 4 is the test.

Stage 2 — Need-Memory Rubric (6 yes/no, the over-engineering filter)

Answer all six. Score <4 yes = no memory store. Use scratchpad + RAG. End the skill.

1. Cross-session continuity. Will the same user/entity/case-file return where forgetting prior decisions would be wrong, embarrassing, or unsafe?
2. Mutable state. Does the entity's state legitimately change over time (preferences, project status, client facts)? Pure facts that don't change → RAG over docs, not memory.
3. Ground-truth feedback exists. Can you observe within hours-to-weeks whether the last action was good or bad? No signal → no learning, only state cache.
4. Cost of being wrong > cost of memory infra. Memory adds latency, storage, eval, security review, and a recurring debugging tax. Pencil out both sides.
5. Volume justifies it. Same user returns ≥5 times. <5 returns → in-context summary is cheaper than vector store.
6. You can audit and redact. GDPR/HIPAA: can you delete on request, export, explain a memory? If no, do not store one.

If you got "yes" only on (1) and (2): you need a state cache, not memory + learning. Say it out loud. Skill recommends tier 2 or 5 and exits.

Stage 3 — Architecture Selection (start at L tier)

Walk the 7-tier memory architecture ladder (next section). Default recommendation: tier 1 (scratchpad-only). Escalate exactly one tier per concrete justification. Justification = "tier N fails on this specific task in our eval set, here's the trace."

Most "we need memory" requests resolve at tier 2 (conversation summary) or tier 5 (KV fact store). Tier 6 (graph) and tier 7 (hierarchical OS-style / Letta) require >3 entities × >50 relationships and a real long-horizon agent, not a chatbot.

Deep dive: references/architectures.md

Stage 4 — Feedback Signal Design

If Stage 1 ended with "remember state only", skip this stage.

For learning, the signal determines everything. Walk the per-domain table:

Rule, repeat once per Q&A session: No signal = state cache, not learning. If the user can't name a signal, do not design a learning loop. Recommend they ship the state cache first, instrument the signal in production, and revisit the skill in a quarter.

Deep dive: references/feedback-signals.md

Stage 5 — Closed-Loop Wiring with Human Gates

If Stage 4 produced no signal, skip this stage and the next two.

The reference closed loop:

[run event: input + agent trace + outputs]
       │
       ▼
[signal collector] ──── engagement / verifier / human review (async)
       │
       ▼
[experience store] (append-only, immutable, signed)
       │   ├── episodic events (raw)
       │   ├── extracted facts (KV)        ← extraction LLM, validated
       │   └── learned skills/playbooks    ← reflection LLM, human-gated
       │
       ▼
[retrieval layer] (hybrid: vector + BM25 + entity link)
       │
       ▼
[state mutator]
       │   ├── AUTONOMOUS: low-risk fields (recency, prefs)
       │   └── HUMAN-GATED: anything that changes policy/voice/identity
       │
       ▼
[next run] ─── core memory in prompt + retrieved episodic + skill lookup

Where humans gate (non-negotiable for production):

• Promotion of any item to "core memory" / system-prompt block
• Schema changes in graph memory
• Skill-library additions used by >1 user (Voyager-style accumulation needs review when shared)
• Reward model updates / fine-tunes from agent feedback

Where it can be autonomous: episodic append, vector indexing, retrieval scoring tweaks, single-user preference KV updates with cheap reversibility, Reflexion-style within-task verbal self-correction (lives in scratchpad, not persistent memory).

Stage 6 — Eval Harness

Six patterns, ship at least the first three before going live:

1. Golden set — 50–500 hand-curated (input, expected behavior, expected memory side-effect) tuples; include adversarial / poisoning attempts.
2. Regression on memory side-effects — assert get(user, "allergies") == ["peanut"] after run X.
3. Drift alarms via OpenTelemetry GenAI semconv — judge-score rolling mean, memory-store size growth rate, retrieval hit-rate distribution, % of runs that mutate core memory.
4. A/B between agent versions — slice traffic, compare composite reward over fixed window.
5. LLM-as-judge with human calibration — 5–10% audit; recompute judge–human Cohen's κ weekly.
6. Held-out human-written tasks — never trained on; detects distribution collapse from self-play.

Deep dive: references/eval-harness.md

Stage 7 — Risks Checklist

Walk all 8 once. Each must have a concrete mitigation in the design doc.

1. Memory poisoning (MINJA, ≥95% lab injection success)
2. Prompt injection via memory
3. Reward hacking
4. Drift / staleness
5. Context rot / window blowup (200K models often unreliable past ~130K)
6. Runaway self-modification
7. Distribution collapse in self-play
8. Multi-agent context explosion

Deep dive: references/risks.md

Stage 8 — Output

Produce the design document:

• Memory schema — chosen tier(s), data model, retention/TTL, redaction hooks
• Closed-loop spec — signal source, collector, experience store, retrieval, mutator, human-gate list
• Eval harness plan — golden set sketch, regression assertions, OTel metric list, A/B split, judge-calibration cadence
• Risk register — 8 risks × 1 mitigation each
• Build order — what ships in week 1 (state cache only), week 4 (signal collection on production traffic), week 12 (closed loop activated behind feature flag)