Hermes Self Evaluation

Hermes Self-Evaluation

Use this skill when the user asks to audit, review, or optimize Hermes's own performance — analyzing session data, skills, configuration, costs, and usage patterns to identify improvements, automation opportunities, and system optimizations.

Don't use for: skill content quality grading (use skill-auditor instead), one-off task analysis, or session artifact indexing after a work session (use session-artifact-indexing).

Overview

The core workflow: gather live evidence about Hermes's state and history → validate each finding against the subsystem's actual semantics and control surface → either produce a direct evidence-backed review or compose a structured analyst prompt for an independent model → verify recommendations before implementation.

External models are useful anomaly detectors and critics, but they are not automatically reliable root-cause analysts. Separate the observed symptom, supported interpretation, confirmed producer/root cause, and proposed change. Before acting on any self-check finding,

When to Use

Triggers:

• "How can we improve Hermes?"
• "Analyze my sessions and tell me what to optimize"
• "I want to have another model evaluate X"
• "Where do sessions/skills live so I can analyze them?"
• "Do an audit of the system"
• "What's the token cost breakdown?"

Workflow

Fast Path: Evaluate a Single Runaway Session

When the user names a specific session and says it was “working” too long, hit a tool-call guardrail, ignored “stop,” or needs the problem evaluated, do not build a broad system-audit prompt first. Diagnose the named session directly.

Use for the exact SQL/Python checks. Minimum evidence to collect:

1. Session metadata from ~/.hermes/state.db: source, title, model, start/end times, message count, tool-call count, token totals, end reason.
2. Role counts and top tool counts.
3. User-message timeline and non-tool assistant replies, especially around compaction/restore and the latest steering instruction.
4. Repeated assistant tool_call IDs. Exact repeated call_ids are a strong sign of stale tool-call replay after context compaction or gateway restore.
5. Log markers for the session ID: max_iterations_reached, Preflight compression, Pre-API compression, gateway shutdown, Operation interrupted, tool-call guardrail, idempotent_no_progress, and transport retry loops.
6. If relevant, check whether any live process from the runaway task is still active before saying it is safe to abandon.

Reporting rule: separate root cause from secondary symptoms. For example, browser/CUA failures may explain retries, but repeated historical tool-call IDs point to restore/replay contamination. If the user says “stop and evaluate,” stop the operational task immediately and evaluate; do not continue trying to finish the stale task.

Step 1: Map the Session Store

The canonical session database is ~/.hermes/state.db. Query its current size and schema live; never hardcode historical counts or gigabytes.

Key tables:

-- sessions: id, source (cli/cron/telegram/tui/api_server/subagent/discord/bluebubbles/speech-bridge),
-- model, input_tokens, output_tokens, reasoning_tokens, cache_read_tokens,
-- cache_write_tokens, message_count, title, started_at, ended_at,
-- estimated_cost_usd, handoff_state, git_branch
CREATE TABLE sessions (...)

-- messages: session_id, role (user/assistant/tool), content (full text),
-- tool_calls, token_count, timestamp, reasoning, finish_reason
CREATE TABLE messages (...)
-- Has FTS5 + trigram full-text search indexes on messages.content

Lifecycle semantics: sessions.ended_at IS NULL means an open DB row. Total retained rows are not active sessions. Gateway routing files map resumable platform conversations; they do not prove a process is currently executing. Review detached sources by age and treat long-lived messaging rows separately. Use the queries above and the interpretation rules in this section.

Other session data:

• ~/.hermes/session-log/*.md — daily Markdown activity logs when the session-log plugin is enabled
• ~/.hermes/sessions/ — gateway routing/session artifacts and optional transcript snapshots; inspect current contents rather than assuming a format or count

Step 2: Map the Skill Library

• Default profile: ~/.hermes/skills/
• Profile-specific overrides: ~/.hermes/profiles/<profile>/skills/
• Each skill is a directory with SKILL.md plus optional references/, templates/, and scripts/ directories.

List skills and inspect the live category tree. Never hardcode skill counts or total size: installs, curator actions, and profile changes make those values stale quickly.

Step 3: Gather Usage Statistics

Run aggregation queries against state.db to get the profile picture:

-- Sessions by source
SELECT source, COUNT(*) as sessions, SUM(input_tokens + output_tokens) as total_tokens,
 SUM(message_count) as total_msgs, ROUND(SUM(estimated_cost_usd), 2) as cost
FROM sessions GROUP BY source ORDER BY total_tokens DESC;

-- Token cost by model
SELECT model, SUM(input_tokens), SUM(output_tokens),
 SUM(input_tokens + output_tokens) as total
FROM sessions WHERE model IS NOT NULL GROUP BY model ORDER BY total DESC;

-- Most active hours / patterns
SELECT strftime('%H', datetime(started_at, 'unixepoch')) as hour, COUNT(*) as sessions
FROM sessions GROUP BY hour ORDER BY sessions DESC;

-- Longest sessions (high message count)
SELECT id, source, message_count, input_tokens, output_tokens, started_at
FROM sessions ORDER BY message_count DESC LIMIT 20;

-- Repeated session titles (recurring task types)
SELECT title, COUNT(*) as freq FROM sessions
WHERE title IS NOT NULL AND title != '' GROUP BY title ORDER BY freq DESC LIMIT 30;

-- Cron session energy (most expensive cron jobs)
SELECT s.id, s.title, s.message_count, s.input_tokens, s.output_tokens
FROM sessions s WHERE s.source = 'cron'
ORDER BY s.input_tokens + s.output_tokens DESC LIMIT 20;

Also collect:

• Total session count and total tokens across all sources
• Current model/provider setup (check config.yaml and .env)
• Cron job list (cronjob action='list')
• Multi-profile architecture notes (default → domain-specific profiles delegation pattern)

Step 4: Compose the Analyst Prompt

Use the analyst-prompt template structure below as the starting point.

The prompt must be self-contained — the receiving model has no knowledge of this conversation. Include:

1. Data locations — exact paths the model would use to query or reference
2. Usage profile — session counts, token costs, message volumes per source
3. Skill library structure — categories, key skills, per-profile overrides
4. Business context — the user's businesses and workflows
5. Architecture summary — profiles, model setup, multi-machine layout
6. Analysis dimensions — what you want the model to evaluate (automation, skill quality, costs, architecture, UX, system health)
7. Output format — priority-ranked findings with evidence and expected impact
8. Starter queries — SQL the model can use to deep-dive

Step 5: Write the Prompt File

Save the composed prompt to a known location so the user can:

• Feed it to another model (Claude, GPT, local Heretic)
• Review and modify it before analyzing
• Re-run the same evaluation later with updated data

Standard location: Notes/hermes-[audit|evaluation]-prompt-<date>.md