Holon Runtime Ops

Holon Runtime Operations Skill

Summary

Use this skill to inspect Holon runtime health, diagnose agent and lifecycle failures, maintain incremental error checkpoints, produce explicitly authorized patrol reports, and prepare sanitized upstream bug reports.

This skill extends the platform-neutral ops workflow. It does not grant credentials, remediation authority, scheduled execution, direct database access, or GitHub publication authority.

Safe daemon restart contract

A restart is a destructive lifecycle operation because it terminates the current daemon cgroup and can interrupt active turns. Never run systemctl --user restart holon.service, holon daemon restart, or an equivalent synchronous stop/start from a turn executing inside that service.

Before the restart, create one durable operation id and persist its phase as planned, then atomically advance it to scheduled together with the owning turn's destructive-operation fence. Execute the restart from outside the daemon cgroup, preferably with a transient unit:

systemd-run --user --unit=holon-restart-<operation-id> --collect \
  --on-active=2s systemctl --user restart holon.service

The scheduling command must return before the current turn settles. After startup, perform only readiness/version/status verification and advance the operation to verified. If the operation id already exists, treat it as already scheduled and never issue a second stop/start. Do not use nohup, shell backgrounding, or a fixed sleep as a cgroup escape.

When To Use

• diagnosing Holon daemon, agent, WorkItem, task, wait, timer, event-ingress, provider, tool, workspace, or delivery behavior
• preparing an authorized Holon restart, upgrade, rollback, or recovery
• counting and deduplicating new Holon runtime errors
• running an explicitly configured report-only patrol
• preparing or publishing a sanitized holon-run/holon issue under the applicable bug-reporting policy

Do Not Use

• for general server or service administration unrelated to Holon
• to read task objectives, messages, prompts, transcripts, memory, or model request bodies for an activity report
• to turn a diagnosis or patrol authorization into repair authority
• to access the runtime database by default or depend on a private schema
• to publish security, privacy, credential, personal-data, or corruption findings automatically

Authority Gates

Treat these as separate authorizations:

1. read-only runtime diagnosis
2. maintenance or remediation
3. scheduled patrol
4. anomaly notification
5. direct runtime-database read
6. direct runtime-database write
7. GitHub duplicate search
8. issue draft creation
9. issue publication

Record the scope, source, expiry, and revocation conditions of standing authorizations. If a gate is absent or ambiguous, stop at the last authorized step and report what is needed next.

Source Priority

Collect evidence in this order:

1. native runtime tools for their declared responsibility
2. declared machine-readable holon CLI commands
3. authoritative service or deployment logs
4. deployment configuration and version metadata
5. separately authorized read-only runtime-database queries as a final deep-diagnostic fallback

Use holon commands to discover the current CLI contract and holon context to inspect caller provenance. Do not manually construct caller-context environment variables and do not use recursive holon run or holon prompt as a control-plane substitute.

Direct database writes are not a diagnostic technique. They are H3 recovery actions requiring exact approval, a snapshot, a bounded mutation, rollback, and verification.

Live Runtime Database Diagnosis

Keep the runtime database readable under its existing deployment permissions, but do not treat direct access to a live runtime.sqlite as the default diagnostic path. Prefer native runtime tools and APIs.

On Linux, inspect runtime_db_protection in runtime status or readiness:

• starting is limited to startup and migration before the guard is published
• protected means the process-lifetime OFD sidecar guard is active
• quarantined means sidecar integrity failed and runtime DB access is permanently fail-stopped for the process
• unsupported means the platform does not provide the Linux OFD guarantee

DB-dependent requests in quarantine return 503 Service Unavailable with error code runtime_db_quarantined and Retry-After: 1. Do not loop on that response or assume the runtime will reconstruct the sidecars automatically.

If direct structural inspection is necessary:

• require the direct runtime-database read authority gate
• inspect a verified offline backup unless separate maintenance authority has been granted to stop the Holon service
• if maintenance authority has not been granted and no verified backup exists, request authorization instead of stopping the service
• an unmanaged SQLite open/query/close must not remove live sidecars while protection is active, but it remains a last-resort diagnostic path and unmanaged writes are unsupported
• never delete, rename, replace, restore, or copy over live -wal or -shm sidecars

If Holon reports quarantined, a deleted-open WAL/SHM file, or an inode divergence:

1. do not delete files or immediately restart over the evidence
2. record the protection evidence, database path, sidecar path, process ID, FD, device, and inode
3. preserve the main database and both sidecars before recovery
4. perform recovery or restart only under explicit maintenance authority, using the preserved files and an offline verification step

These are operational safety rules, not access-control requirements. Do not change directory ownership or permissions unless the operator separately authorizes that deployment change.

Installation Inventory

Keep one record per Holon installation:

work/inventory/installations/<installation-id>/info.yaml

Recommended fields:

schema: holon.ops.installation.v1
id: local-dev
environment: development
owner: operator
deployment_mode: local-binary
version:
  reported: 0.1.0
  commit: "<git-sha-or-null>"
runtime:
  endpoint_ref: local-control-plane
  service_ref: null
authoritative_sources:
  - kind: runtime
    ref: native-tools
  - kind: deployment
    ref: "<safe-reference>"
data_boundaries:
  activity_reports: runtime-metadata-only
  database_access: disabled
notes: null

Do not place credentials, callback URLs, secret values, private keys, or raw payloads in inventory.

Standard Diagnosis

1. Identify — installation, environment, version, deployment mode, impact, reporting window, and authority.
2. Snapshot — current agent/runtime state and deployment state using the highest-priority available sources.
3. Bound — choose the smallest relevant agents, WorkItems, tasks, errors, components, and time range.
4. Collect — lifecycle metadata, statuses, timestamps, exit classes, redacted errors, and authoritative log references.
5. Correlate — order evidence by time and explicit state transitions.
6. Classify — expected behavior, configuration, dependency, deployment, suspected Holon defect, security/privacy, or unknown.
7. Report — facts, inference, confidence, impact, workaround, missing evidence, and recommended next action.
8. Change only if authorized — use the ops preflight, execution, rollback, operation-record, and verification workflow.

Do not treat queued, yielded, blocked, waiting, and current WorkItems as interchangeable. Use the runtime's declared lifecycle views rather than reconstructing scheduler state from incidental logs.

Incremental Error Analysis

Maintain one checkpoint per installation and source:

work/checkpoints/<installation-id>-<source-id>.json

Example:

{
  "schema": "holon.ops.checkpoint.v1",
  "source": "runtime-events",
  "cursor_kind": "timestamp_and_id",
  "cursor": {
    "timestamp": "2026-09-06T00:00:00Z",
    "id": "event-redacted"
  },
  "last_successful_run": "2026-09-06T00:10:00Z"
}

Rules:

• use a deterministic boundary such as (timestamp, stable_id) when possible
• query with overlap, then deduplicate, so equal timestamps and late records are not lost
• fingerprint from sanitized component, error class, normalized message shape, relevant state transition, and version; never hash a secret-bearing raw body and publish the hash as if it were safe
• distinguish occurrence count, affected agents or operations, first seen, last seen, and new-versus-known status
• state explicitly when sources are missing, truncated, reset, or inconsistent
• write the new checkpoint only after collection, analysis, report persistence, and required delivery all succeed
• on failure, retain the previous checkpoint and record the failed run