Agentic System Audit

Agentic System Audit

Audit the whole agentic contract, not isolated files.

Find where code, prompts, tools, state, results, docs, tests, configuration, packaging, or runtime capabilities disagree or leave the next safe action ambiguous.

Prefer deletion, consolidation, one canonical owner, and the smallest durable correction.

This is a read-only audit. Do not modify the repository.

Operating model

Delegate the audit. The orchestrator coordinates and returns the result; it should not perform a competing repository review.

Use:

• reviewer: lead auditor, evidence reconciliation, synthesis;
• scout: focused local evidence;
• researcher: external/API verification when material conclusions depend on it;
• implementer: never during the audit;
• generalist: fallback only when the required role is unavailable and its effective policy is read-only.

Start with:

{ "action": "list" }

Use effective capabilities from the current context. Do not assume definition-level delegation, tools, skills, or extensions survive overrides or depth.

Preferred topology

orchestrator
└── reviewer
    ├── scout A: runtime/contracts/reachability
    ├── scout B: instructions/capabilities
    ├── scout C: presentation/docs/tests/package glue
    └── researcher: only when the research trigger is met

Use one lead reviewer. Do not spawn a second generic reviewer by default.

Leaf fallback

If the lead reviewer cannot delegate:

1. run the three scout lenses directly in parallel;
2. run researcher only when triggered;
3. pass exact scout/researcher resultPath values to one reviewer through files;
4. have that reviewer verify, reconcile, prioritize, and synthesize.

The orchestrator must not redo the scouts' analysis.

Lead reviewer contract

The reviewer may not receive this skill directly, so its assignment must carry the rules required for a valid audit.

Give the reviewer this objective:

Audit the repository as one agentic system. Remain strictly read-only.

Delegate three focused scouts:

• A: runtime producers, validation, state, identity, lifecycle, and production reachability;
• B: instruction ownership and effective capabilities;
• C: model/UI projections, docs, tests, and repository/package glue.

Use researcher when a candidate material finding or its impact depends on external API/runtime truth.

Before reporting any P0-P2 finding:

• identify the semantic source and relevant producers;
• prove the disputed state/value is reachable on a production path when behavior is at issue;
• trace it through every consumer boundary relevant to the claim;
• resolve material upstream/runtime assumptions for the repository's supported versions using primary evidence when available;
• verify that a claimed regression check would fail for the old behavior.

Do not report representable-but-unreachable states as defects. Do not treat differently named fields as equivalent without tracing semantics. Do not leave an answerable material uncertainty as follow-up.

If required evidence genuinely cannot be established, report a bounded Unverified risk, not a definitive finding.

Compare root/parent/child, definition/session/fork, model/UI, success/failure/recovery, configured/effective capability, and static/dynamic guidance.

Independently verify material conclusions and return one de-duplicated report using the required output contract with the smallest durable fix for each finding.

Pass supplied canonical diffs, snapshots, specs, or source artifacts to scouts instead of making them rediscover the same material.

Material finding closure

Scouts return candidate evidence. The reviewer closes candidates before severity is assigned.

Trace every boundary used by the claim. For result/state findings, prefer:

semantic source
→ producer
→ validation/normalization
→ runtime/durable state
→ structured/public projection
→ model consumer
→ human/API consumer
→ docs/tests

Not every step applies to every finding.

Provenance

Find where the disputed fact originates and how it changes.

Do not infer semantic equivalence from similar field names. Trace producers and scope first.

Reachability

Do not confuse representability with reachability.

A type, schema, fixture, or doc proves a value can be described. A behavioral finding requires a production path that can actually produce the material state/value.

If the harmful state cannot occur on the claimed path, dismiss or narrow the candidate.

Consumer contract

Establish what the actual consumer receives.

For model-visible claims, determine whether the model gets content, details, both, or another transformation.

For human claims, distinguish compact TUI, expanded TUI, plain/RPC/JSON, and logs/files.

Never assume renderer-visible metadata is model-visible.

External contract

If a material conclusion depends on upstream behavior, resolve it for the repository's supported version range.

Prefer:

supported-version primary source/API/type contract
→ supported-version official docs
→ installed dependency source
→ targeted runtime smoke only if still unresolved

Do not substitute latest behavior for the supported range. Do not require a smoke test when source plus local integration already establishes the behavior.

Test discrimination

When claiming a test gap or proposing a regression:

Would this test fail for the old bug and pass for the corrected behavior?

For lifecycle/recovery bugs, pair the harmful case with its nearest legitimate opposite when that distinction matters, such as a stale working parent versus a parent correctly waiting on unresolved child work.

Do not count assertion volume as coverage.

Closure outcomes

Every material candidate becomes exactly one of:

• Verified finding: evidence establishes defect and consequence.
• Dismissed/narrowed: intentional, unreachable, semantically different, or immaterial.
• Unverified risk: evidence genuinely cannot be established with available read-only capabilities.

An Unverified risk must state known evidence, missing evidence, version/runtime scope, why it matters, and one exact resolver.

Do not use it for an answerable question that was not investigated.

Scout A: runtime contracts and reachability

Own:

Can this state/value actually happen, where is it produced, and what runtime contract governs it?

Prove this liveness invariant for unresolved directly-owned assignments:

Every unresolved directly-owned assignment must either remain capable of making progress without its owner, or have a reliable reconciliation path that brings the exact owner back when action is required.

Trace:

schema/input
→ validation
→ effective configuration
→ execution
→ runtime/durable state
→ public result
→ next legal action

Check:

• schema vs runtime acceptance/rejection;
• defaults, normalization, mutually exclusive fields;
• validation before mutation;
• sibling-path validation consistency;
• logical label, definition, Pi session, owner, request, ask, run, workspace, tab, pane, Herdr identity;
• stale/replacement identity binding;
• public state vs actual action eligibility;
• working, blocked, settling, unknown, steerability, staleness, pending ask/result, child gating, cleanup convergence;
• unresolved-work liveness across owner idle/busy state, failed attention delivery, reminder recurrence, episode termination, and restart;
• parent/child distinctions where a working parent may itself be stale while a legitimately waiting parent projects blocked;
• fresh delegation, continuation, fork, replacement, completion, parent-child completion, ask/reply, close, rollback, restart/recovery;
• source identity, request correlation, result paths, truncation, cleanup warnings, primary/cleanup errors, rollback/retry state.

Return exact production paths, reachability evidence, and candidate mismatches. Do not decide architecture from local evidence alone.

Scout B: instruction and capability stack

Own:

Who owns this rule, and is it truthful for the effective execution context?

Map:

system/base prompt
tool description
controller scope
shared agent prompt
role body
body @file
skills
dynamic success/error guidance
human docs

For each normative rule, find its intended single owner.

Flag:

• zero owner: instruction gap;
• multiple authoritative owners: DUPLICATE_AUTHORITY;
• conflicting owners: CONTRADICTION;
• wrong owner: AUTHORITY_LEAK.

Compare:

definition
→ override composition
→ effective tools/skills/extensions/context
→ launch args
→ actual role/depth
→ model instructions
→ presentation

Find:

• GHOST_CAPABILITY;
• HIDDEN_CAPABILITY;
• root/parent/leaf capability mismatches;
• metadata advertising removed capabilities;
• infrastructure accidentally removable by normal policy;
• user-configurable policy treated as mandatory infrastructure;
• impossible or underspecified instructions;
• global policy embedded in role prompts;
• API mechanics redundantly taught by prompts/skills.

Return exact instruction text, owner, effective runtime evidence, and candidate conflicts.

Scout C: presentation, docs, tests, and repository glue

Own:

Where does runtime truth go, what does each consumer actually see, and do docs/tests describe that contract?

Model-visible presentation

Inspect list, assignment/steer/reply/close acknowledgements, completion delivery, errors, and truncation.

Ask:

Using only what the model actually receives, can it identify the source, understand the state, and choose the next legal action?

Flag missing correlation, hidden material details, ambiguous terminology, duplicated IDs, static policy repeated on every result, or missing dynamic next-action evidence.

For opacity candidates, trace projection semantics, not just field names.

Human presentation

Compare status widget, definitions view, completion renderer, warnings, compact/expanded views, and any plain/API surface.

Find hidden failures, duplicated data, display identity reused as machine identity, or UI claims stronger than runtime evidence.

Documentation

Find shipped-but-undocumented behavior, documented-but-unshipped behavior, duplicate canonical owners, stale terminology, rejected examples, and implementation details presented as public contracts.

Tests

Find public contracts without discriminating regressions, tests that pass for correct and broken behavior, mocks that cannot detect claimed integration failures, missing sibling invariants, and stale fixtures.

Repository/package glue

Check package files, discovery, README links, AGENTS/maintainer guidance, validation scripts, dead compatibility/configuration, and removed concepts that still ship or remain referenced.

Return projection chains, consumer visibility, docs/tests evidence, and candidate mismatches.

Research trigger

Research is conditional. Once triggered, verification is required when capability exists.

Trigger researcher when:

a candidate likely to affect a P0-P2 finding, verdict, or remediation
depends on external API/runtime behavior

Research input should contain only:

exact disputed assumption
supported version range
preferred primary source/repository
required direct answer

Require primary/official evidence, exact version relevance, and only material findings.

If web capability is unavailable:

1. inspect installed/local upstream source;
2. inspect pinned/supported dependency source where reachable;
3. use other authoritative read-only evidence;
4. otherwise return Unverified risk.

Do not add dependencies or require optional web tooling to run the audit.

Cross-surface contract matrix

Cover each applicable chain.

Input

schema ↔ description ↔ validation ↔ errors ↔ docs ↔ tests

Capability

definition ↔ override ↔ effective config ↔ launch ↔ role/depth ↔ tools/prompts ↔ presentation

Instruction

controller ↔ shared prompt ↔ role body ↔ skills ↔ dynamic results

Identity

label ↔ definition ↔ session ↔ request ↔ owner ↔ Herdr identity ↔ result/error/UI

State

runtime evidence ↔ public state ↔ steerability ↔ eligibility ↔ displayed state ↔ next action

Completion

completion ↔ durable result ↔ result file ↔ owner message ↔ model content ↔ human rendering ↔ cleanup

Configuration

syntax ↔ parsing ↔ merge ↔ validation ↔ effective metadata ↔ runtime ↔ persistence/UI ↔ docs

Recovery

failure ↔ retained evidence ↔ autonomous progress/attention owner ↔ wake/delivery ↔ recurrence/termination ↔ structured error ↔ model/human guidance ↔ retry boundary

Distribution

repository ↔ package manifest ↔ installed files ↔ discovery ↔ maintainer guidance

Mandatory boundary comparisons

Check where applicable:

Controller depth

root | direct agent/parent | nested child/leaf | unmanaged session

Agent generation

fresh | exact-session continuation | fork

Visibility

model content | structured details | compact TUI | expanded TUI | plain/RPC/JSON | logs/files

Outcome

success | blocked | failure | rollback failure | close | cleanup pending | overflow | restart recovery

Configuration

bundled | partial override | standalone global | empty fields | false | [] | invalid