Skillify

Skillify — The Meta Skill

Relationship to /cross-modal-review: That skill is the manual mid-flow "second opinion" gate (one model reviews work product before commit). This skill's Phase 3 below uses gbrain eval cross-modal instead — three different-provider frontier models score-and-iterate on a documented dimension list before tests cement behavior. Use /cross-modal-review for ad-hoc second opinions; use Phase 3 here when skillifying a feature.

Contract

A feature is "properly skilled" when all 15 checklist items (0 + 1–14; 3b rides with item 3) pass. gbrain skillify check audits the mechanical items (1–11); items 0, 3b, 12, 13, and 14 are procedural gates the agent verifies directly. Item 3 (cross-modal eval) is informational in the audit — it does not gate gbrain skillify check, but a missing or stale receipt is surfaced so the user knows where the gate stands.

Idempotency guarantee: skillify can run on the same skill any number of times. Each run:

1. Detects existing artifacts (SKILL.md, tests, evals, code, resolver entries)
2. Identifies what's new: bug report, user feedback, new input, or quality gap
3. Improves existing files rather than rewriting from scratch
4. Preserves what works, fixes what's broken
5. Re-runs the checklist and only touches items that fail

No-regression law: any edit to a skill must score ≥ the previous iteration's cross-modal eval on the same task and dimensions. Forward only, never back (Phase 3.5).

The Checklist

Other skills and workflows delegate to this checklist — reference items by number (e.g. "run skillify items 4–6") against skills/skillify/SKILL.md. The numbering is stable; additive changes only.

□ 0.  Eval contract      — skill declares goal + skill-specific dimensions + hard-fails (Phase 2.5)
□ 1.  SKILL.md           — skill file with frontmatter + contract + phases
□ 2.  Code               — deterministic script if applicable
□ 3.  Cross-modal eval   — 3 frontier models from 3 providers critique output vs the contract; informational in the audit
□ 3b. No-regression gate — new eval scores ≥ the previous iteration (forward only; Phase 3.5)
□ 4.  Unit tests         — cover every branch of deterministic logic
□ 5.  Integration tests  — exercise live endpoints
□ 6.  LLM evals          — quality/correctness cases for LLM-involving steps
□ 7.  Resolver trigger   — entry in skills/RESOLVER.md with real user trigger phrases
□ 8.  Resolver eval      — test that triggers route to this skill
□ 9.  Check-resolvable   — DRY + MECE audit, no orphans
□ 10. E2E test           — smoke test: trigger → side effect
□ 11. Brain filing       — if it writes pages, entry in brain/RESOLVER.md
□ 12. Scheduled-run observability — if the skill backs a cron/recurring job, runs route through minions so they are logged and inspectable (Phase 6)
□ 13. Scheduled-task re-run — if the skill backs a cron, an edit re-runs a representative task + evals it (Phase 3.5)
□ 14. Plugin membership   — record the skill in openclaw.plugin.json OR skills/plugin-exclusions.json (the membership test requires exactly one)

Phase 0: Determine Mode (New vs Improve)

Before anything, determine the mode.

Mode A: New Skill (no SKILL.md exists)

Check:

• Will this be invoked 2+ times? (One-off work ≠ skill)
• Is there >20 lines of logic? (Trivial helpers don't need full infrastructure)
• Does it have a clear trigger phrase a user would actually say?

If ANY answer is no, it's a script, not a skill — stop here. Do not scaffold, write a SKILL.md, run evals, or write tests for it. Tell the user why and move on.

Scope check (upper bound): one skill = one capability = one coherent trigger family. If the target spans multiple distinct intents users would invoke separately ("run the build" / "roll back the deploy" / "notify the team" are three intents, not one), do NOT build one skill covering them all. Stop, propose splitting into separate skillify targets, and ask the user which one to skillify first.

DRY/MECE pre-check: before scaffolding, grep skills/RESOLVER.md and the skill manifest for overlapping trigger phrases and near-duplicate descriptions. If a proposed trigger collides with an existing skill, prefer merging — add a trigger, mode, or phase to the existing skill, bump its minor version, and run this checklist on the merged result — over creating a near-duplicate. If you create a separate skill despite overlap, record the one-sentence distinction in the new SKILL.md. gbrain check-resolvable (Phase 5) is the backstop that catches what the pre-check misses.

Mode B: Improve Existing Skill (SKILL.md exists)

This is the idempotent path. Triggered when running skillify on a skill that already has a SKILL.md, when a bug was found, when the user gave feedback or new input, or when a quality pass is requested.

Improvement protocol:

1. Read ALL existing artifacts: skills/<slug>/SKILL.md, the skill's script (whatever path SKILL.md references), routing-eval.jsonl, and test/<slug>.test.ts.
2. Identify the delta — what changed?
   • Bug fix: the user reported a specific failure. Add a test for the exact bug, fix the code, add a HARD RULE to SKILL.md to prevent recurrence.
   • New input: the user provided new context/requirements. Extend SKILL.md, update code, add tests for the new behavior.
   • Quality pass: no specific bug, just running the checklist again. Audit, cross-modal eval, fill gaps.
3. Apply changes surgically — edit existing files, don't rewrite them.
4. Re-run ALL existing tests to ensure no regressions.
5. Update the version in frontmatter (patch for bug fixes, minor for new capability).
6. Run the No-Regression Gate (Phase 3.5) before shipping.

Bug fix template (add to the skill's SKILL.md):

## Bug: [Date] — [Short description]
- **What happened:** [Concrete failure]
- **Root cause:** [Why]
- **Fix:** [What changed]
- **Hard rule added:** [New constraint to prevent recurrence]
- **Tests added:** [List of new test cases]

Phase 1: Audit

Feature: [name]
Code: [path]
Missing items: [check each of the 15]

Phase 2: Write SKILL.md + Code (items 1-2)

SKILL.md frontmatter template (copy-paste):

---
name: my-skill
version: 1.0.0
description: |
  One paragraph. What it does, when to use it.
triggers:
  - "trigger phrase users actually say"
  - "another real trigger"
tools:
  - exec
  - read
  - write
mutating: false  # true if it writes to brain/disk
---

Body must include: Contract (what it guarantees), Phases (step-by-step), Output Format (what it produces).

Extract deterministic code into scripts/*.ts.

Phase 2.5: Eval Contract (item 0) — DEFINE THE GOAL + RUBRIC BEFORE EVALUATING

A cross-modal eval is only as good as what you ask it to judge. Generic dimensions catch slop but miss whether the output achieves THIS skill's specific purpose. Every skill carries an explicit eval contract: what the skill is FOR, and HOW to tell if a given output is excellent vs. a failure.

Every skill declares its eval contract in frontmatter

eval_contract:
  goal: |
    One or two sentences: what this skill is supposed to achieve, for whom, and what
    "excellent" looks like in the real world. Be concrete and skill-specific.
  dimensions:
    - "DIMENSION_NAME — the specific question this dimension answers for THIS skill"
    # 3-6 dimensions, tuned to the skill. Example for a per-subject news briefing skill:
    # - "FACTUAL_INTEGRITY — is every claim true and verifiable?"
    # - "SUBJECT_ANCHOR — is each item tied to something concretely THIS subject's?"
    # - "WHY_IT_MATTERS — is the relevance real and specific, not a templated lane label?"
    # - "CLICKWORTHY — would the reader actually want to open these links?"
  hard_fails:
    - "A failure mode that auto-zeroes the eval regardless of other scores."
    # e.g. "Any fabricated fact = automatic 1."

How to obtain the eval contract (in priority order)

1. Read it from the skill. If eval_contract already exists in frontmatter (or a clear Goal/Contract section in the body), use it. Idempotent runs reuse the same contract so scores compare apples-to-apples across iterations — this is what makes the no-regression gate meaningful.
2. Infer it from context. No declared contract? Derive the goal + dimensions from the SKILL.md body (Contract/Output Format), the scheduled task prompt that invokes it, recent user feedback (especially complaints — a complaint that a fact was wrong pins FACTUAL_INTEGRITY as a hard-fail dimension), and the representative output's audience.
3. Ask the user when it's unclear. If the goal or the bar for "excellent" is genuinely ambiguous and you can't infer it confidently, ASK — one tight question, then proceed. Do not guess a rubric for a high-stakes skill (anything whose output posts to a live channel or is read by others). A wrong rubric produces a confident-but-meaningless score.
4. Write it back. Once settled, persist the contract into the skill's eval_contract frontmatter so it's reused and versioned. Editing the contract is itself a skill edit — re-eval to confirm the new rubric still scores the current output forward.

Wire the contract into the eval

Pass the contract's goal as --task and its dimensions as --dimensions to gbrain eval cross-modal. The eval then measures the skill against ITS stated purpose, not a generic checklist. State hard-fail conditions inside the task text so judges zero the score when they fire ("If any claim is fabricated, the output fails regardless of other qualities").

Phase 3: Cross-Modal Eval (item 3) — THE QUALITY GATE

Why this comes before tests

Tests lock in behavior. If the behavior is mediocre, tests lock in mediocrity. Cross-modal eval proves the quality bar FIRST, then tests cement it.

Step 1: Pick a representative input

Choose the input that exercises the skill's hardest documented use case. If unsure: use the primary trigger example from SKILL.md, or the most complex real-world input from the last 7 days of memory files.

Step 2: Run the skill, capture output

Run the skill on the representative input. The OUTPUT FILE is what gets evaluated.

Step 3: Run the eval gate

gbrain eval cross-modal \
  --task "The eval contract's goal, including its hard-fail conditions" \
  --dimensions "dim_one,dim_two,dim_three" \
  --output skills/<slug>/SKILL.md

The command runs 3 frontier models from 3 different providers in parallel, scores the OUTPUT against the TASK on the given dimensions (default: 5 standard dimensions when --dimensions is omitted), and writes a receipt under ~/.gbrain/.gbrain/eval-receipts/<slug>-<sha8>.json (the sha-8 binds the receipt to the current SKILL.md content — re-running after edits writes a new receipt, so every iteration leaves its own baseline).

Default models (override per slot via --slot-a-model, --slot-b-model, --slot-c-model):

These MUST be frontier models from DIFFERENT providers. Using a single provider's family or budget models defeats the purpose — different families have less correlated blind spots. Model names drift; pin the live aliases at run time. The frontier-only, different-providers rule is the durable part.

Pass criteria (BOTH must be true):

1. Every dimension's mean across successful models ≥ 7.
2. No single model scored any dimension < 5 (the floor).

Inconclusive: fewer than 2 of 3 models returned parseable scores. Receipt is still written (forensics) but the gate is not authoritative. Exit code 2; CI wrappers should treat this as "did not run cleanly", not "failed quality gate".

Step 4: Cycle until you pass (≤3 cycles)

CYCLE 1:
  Eval → scores + top 10 improvements
  IF pass: → done, write tests
  ELSE:
    Apply top 10 improvements to the actual file
    Log: which improvements applied, what changed

CYCLE 2:
  Re-eval the FIXED output (same 3 models, same dimensions)
  Compare: before/after scores per dimension (track delta)
  IF pass: → done, write tests
  ELSE: apply remaining improvements + new ones

CYCLE 3 (final):
  Re-eval
  IF pass: → ship
  ELSE: → ship with KNOWN_GAPS section listing:
    - Which dimensions are still below 7
    - Which improvements couldn't be resolved
    - Why (e.g., "would require architectural change")

Cycles + cost guardrails

• Default --cycles 3 in TTY, --cycles 1 in non-TTY (limits scripted bulk spend in CI loops).
• The command prints an estimated max-cost-per-cycle from a small pricing constant before each run. Real cost varies with prompt size; treat the estimate as a ceiling for default --max-tokens 4000.
• A --budget-usd N hard cap is a v0.27.x follow-up TODO.

Provider configuration

Models resolve through the gbrain AI gateway. Configure once with:

gbrain providers test    # see what's configured
gbrain config            # set keys

Or set env vars: OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_GENERATIVE_AI_API_KEY, TOGETHER_API_KEY, etc. The gateway reads from ~/.gbrain/config.json plus process.env.

Cost expectations

3 cycles × 3 models = 9 frontier calls max per run. Expect $1–3 per full run on default --max-tokens 4000 with frontier-class models. Receipts include the per-call model identifiers so you can audit retroactively.

Skip cross-modal eval when:

• Output is < 200 tokens (trivial — not worth 9 API calls).
• The skill is a thin wrapper around a single API call (one cycle is enough).

Phase 3.5: No-Regression Gate + Scheduled-Task Re-Run (items 3b, 13) — THE FORWARD-ONLY LAW

A scheduled skill can regress silently and ship a bad output to a live channel before anyone re-checks its quality. The law: we only go forward. An edit that scores worse than the last iteration is a REGRESSION and must NOT ship — revert or re-fix.

3b. No-Regression Gate (every skill edit, MANDATORY)

Absolute pass (every dimension mean ≥ 7, no single score < 5) is necessary but NOT sufficient. The edited output must also beat the previous iteration's eval on the same task and dimensions.

1. Find the most recent prior receipt. Receipts are bound to the SKILL.md content sha, so every iteration leaves its own file:

   ls -t ~/.gbrain/.gbrain/eval-receipts/<slug>-*.json | head -5

2. Read the baseline numbers from the prior receipt:

   jq '{overall: .aggregate.overall, dims: (.aggregate.dimensions | map_values(.mean))}' <prior-receipt>

3. Run the new eval on the edited output with the SAME --task and --dimensions as the prior run — apples to apples. Reusing the eval contract (Phase 2.5) makes this automatic.
4. Compare:
   • FORWARD (ship): new overall ≥ prior overall AND no dimension mean dropped by more than 0.5.
   • REGRESSION (block): new overall < prior, OR any dimension regressed by more than 0.5. Do NOT ship. Apply the top improvements and re-eval, or revert the edit. Log the regression.
5. If there is NO prior receipt (first-ever eval), the absolute pass (≥ 7) is the bar and this run becomes the baseline.
6. Record the before/after comparison (prior overall → new overall, verdict) in the commit or PR message so the forward-only history is auditable.

Exception: a pure deterministic bug fix whose behavior is fully locked by new tests may skip the re-eval — but the Phase 7 re-verify still runs. Any edit that changes prose, prompts, or output shape re-evals.