Dhub Skill Creator

Decision Hub Skill Creator

Create modular skill packages (SKILL.md + optional resources) that turn Claude into a specialist. This skill guides the full lifecycle: define the domain, design the architecture, build and validate the skill, and package it for distribution.

For skills intended for Decision Hub, the workflow naturally extends into defining runtime environments and writing evaluation criteria — not as a separate mode, but as a natural consequence of what the skill needs.

Anatomy of an Effective Skill

What a Skill Contains

Progressive disclosure — each layer loads only when needed:

1. Metadata (always in context): name + description in frontmatter. Determines when the skill activates.
2. SKILL.md body (when triggered): The agent system prompt. Core procedures, workflow, constraints.
3. Bundled resources (as needed):
   • scripts/ — deterministic code the agent executes via Bash
   • references/ — domain knowledge the agent reads on demand
   • assets/ — templates, sample data, output formats
   • agents/ — subagent system prompts for delegation

Patterns That Make Skills Effective

What to Avoid

• Generic TODO templates the agent fills with boilerplate
• Excessive placeholder files that create clutter to delete
• Vague descriptions like "A helpful skill" — triggers for wrong contexts
• Instructions written for humans instead of agents
• Duplicating content between SKILL.md and references
• Overly long SKILL.md — move detail to references/

Skill Creation Workflow

Four phases. Not rigid steps — they overlap and the depth of each depends on the skill's complexity.

Phase 1 — Define the Domain

Understand what the skill does before building anything. Ask focused questions:

• What specific tasks does this skill handle? Not "data analysis" but "causal inference for A/B tests, lift analysis, treatment effect estimation."
• What triggers should activate this skill? Maps directly to the description field. Think about what a user would say.
• What does the agent need to know that it doesn't already know? This is the test for whether content belongs in the skill. If Claude already knows it, don't include it.

Ask 2-3 focused questions. Never more than 5. Gather enough to make design decisions, then move on.

Phase 2 — Design the Architecture

Choose the structural pattern and identify resources. Read references/skill_patterns.md for detailed patterns.

Choose a structural pattern:

• Workflow-based — multi-step processes with phases and review gates
• Task-based — focused input/output with processing rules
• Agent-delegation — multiple subagents, each handling one concern
• Reference-based — augmenting with domain knowledge the agent lacks

Identify bundled resources:

• What scripts need to exist in scripts/?
• What reference material goes in references/?
• Does the skill need subagents in agents/?
• Are there template files for assets/?

Determine if the skill needs runtime or evals blocks. These are about the skill's nature — whether it has executable code or should be automatically testable — not about where the skill will be published.

• Runtime block: Ask "Does this skill include executable code or dependencies?" If yes, ask what language/packages it needs and any required API keys, then define the runtime block in frontmatter. See references/format_spec.md for field details.
• Evals block: Ask "Should this skill be automatically testable?" If yes, ask "What does 'correct' look like? Describe a scenario where it should pass and one where it should fail." Then guide eval case authoring in Phase 3b.

Phase 3a — Build the Skill

1. Scaffold. Run scripts/init_skill.py to create the directory:

   python internal-skills/dhub-skill-creator/scripts/init_skill.py <name> --path <dir> [--with-runtime] [--with-evals] [--description "..."]

2. Write the SKILL.md body. Follow the writing guidelines below. The body is the agent system prompt — procedural knowledge the agent cannot infer on its own.

3. Build resources. Create scripts, references, assets, agents as designed. For runtime skills, ensure the entrypoint exists and dependencies are declared.

4. Validate early. Run scripts/validate_skill.py during development, not just at the end.

Phase 3b — Author Evaluation Cases

When the skill has an evals block, help the user construct eval cases through a structured interview.

Step 1 — Identify what to test. Each eval case tests one specific behavior.

• "What's the most important thing this skill must get right?"
• "What's the most common way it could fail?"

Step 2 — Write the eval prompt. A realistic user message — what a real person would say to trigger this skill. Keep it focused. Include test data in evals/data/ if needed.

Step 3 — Compose judge criteria. The judge_criteria field is free-text interpreted by an LLM judge. Build it from structured blocks — pick whichever are relevant:

Required Behaviors — things the agent MUST do:

## Required Behaviors
- Checks data distribution before selecting a statistical test
- Reports confidence intervals, not just p-values

Forbidden Behaviors — things that cause automatic failure:

## Forbidden Behaviors
- Applies parametric tests without verifying normality
- Hallucinates data that wasn't in the input file

Expected Output Contains — specific patterns or concepts:

## Expected Output Contains
- A test statistic and p-value
- An interpretation in plain language

Calibration Examples — good/bad snippets so the judge knows what "right" looks like:

## Examples
Good: "Shapiro-Wilk test (p=0.003) rejects normality, using Mann-Whitney U..."
Bad: "Running a t-test gives p=0.04, so the treatment works."

Threshold — how to combine criteria into pass/fail:

## Scoring
PASS if all Required Behaviors present AND no Forbidden Behaviors appear.

Interview the user to populate these blocks:

• "Describe what a correct output looks like" → Required Behaviors + Expected Output
• "What would a wrong output look like?" → Forbidden Behaviors + bad Example
• "Can you show a snippet of ideal output?" → good Example

For simple cases, a single sentence works: "PASS if the agent creates a valid CSV file with headers matching the schema. FAIL otherwise."

Step 4 — Assemble the eval YAML. Create evals/<case-name>.yaml with name, description, prompt, and judge_criteria fields. See references/format_spec.md for the complete spec and references/skill_patterns.md for the eval criteria authoring guide.