CSO Security Audit

Phase 12: Evidence rubric and independent challenge

This CSO evidence rubric governs CSO instead of shared review confidence instructions. Keep three separate judgments:

• Severity: impact and realistic attacker prerequisites in this application. A pattern or CVSS number alone does not determine severity.
• Confidence: how strongly available evidence supports that precise claim; explain unknowns and counterevidence. Do not turn a number into proof.
• Evidence: candidate hypothesis, supported static evidence, or a helper-recorded reproduction. Code tracing can support a finding; it cannot establish that an application booted or a repair passed tests.

Daily reports contain supported findings: a concrete attacker-controlled entrypoint, a path across an intended security boundary, demonstrated impact, and a challenge of relevant protective controls. Comprehensive reports retain unresolved candidates separately as labeled hypotheses, never mixed into supported totals. Disproved candidates are retained as disposition/coverage evidence, not vulnerabilities.

Do not apply blanket exclusions for development dependencies, availability/resource attacks, historical secrets, user-role prompt injection, or gstack-owned skills. Analyze attacker control and impact. Likewise, UUIDs do not provide authorization; user-controlled URL paths can still cross a sensitive boundary; environment variables may originate from untrusted workflows; and safe defaults can be bypassed by framework escape hatches. Missing hardening alone needs a concrete failure scenario before becoming a finding.

For each candidate, use an already-authorized independent reviewer when available. Give it the relevant locations, invariant, and rubric without the producer's conclusion; have it inspect callers, middleware, configuration, validation, legitimate behavior, and mitigations. Use at most three; await them. Do not request broader tool access solely to obtain an independent reviewer. Otherwise perform a separate skeptical pass labeled sequential challenge; independent agent unavailable. Record dissent and assumptions. Agreement and scanner warnings do not prove runtime behavior.

Search for root-cause variants after supporting a finding, honoring scope. Prioritize by impact, dependency reachability/exposure, known exploitation, and likely user benefit. Unknown reachability remains unknown, not “unreachable.”

Comprehensive verification. Read the schema and call runtime-plan before preparing the harness or patch. Continue only when the helper returns a matching qualified runtime catalog profile; otherwise record the exact execution prerequisite and continue static assessment. Copy its startup, full-test commands, and immutable inputs exactly. Execute only through verify, which records:

1. Original application boots; a legitimate control passes; the regression fails at the intended security assertion. Startup failure, a generic nonzero exit, or a printed claim of vulnerability is inconclusive.
2. On a pristine second copy, the identical security assertion and legitimate behavior pass, and the canonical full test suite reports success. Project-test completion remains self_reported: target code shares that process and can forge reporter output or terminate the runner. Command, count, exit, and output hashes record diagnostics but cannot upgrade that assurance.
3. Harness, fixtures, verifier dependencies, toolchain, runtime, and sandbox policy are unchanged. Application source, configuration, and dependencies may change only as declared, independently reviewed patch effects; record before/after configuration and dependency closures separately.
4. A skeptical review checks root-cause repair and feature preservation. Run record-review; put its artifact ID in review.artifactId before verify. Current review identity is self_attested. Changed assertions, removed behavior, boundary-replacing mocks, or disappearing warnings cannot upgrade assurance.

Keep finding evidence, reproduction outcome, patch validation, test-completion assurance, review assurance, and current-source closure separate. Unwitnessed passing observations produce a redacted, provenance-bound repair_candidate and proposed repair; they cannot issue or replay a RepairBundle or close current source. The helper-owned witness can authenticate the separate external boot, legitimate-control, and security assertions and emit runtime_tested, but it cannot authenticate completion of a target-controlled project-test process. Therefore every currently issued bundle records testCompletionAssurance: self_reported and must never be presented as tested. The tested state remains reserved until a target-independent completion witness exists. A claimed string is insufficient. Missing witness, qualified catalog profiles/dependencies, local Docker, or containment is an exact prerequisite, never fabricated proof.

Phase 13: Report and repair artifacts

Finish every audit through the helper, including empty, cancelled, blocked, or interrupted audits when possible. Every report begins with complete, partial, or not assessed, followed by scope and material gaps. Completeness is independent of finding count. For an empty supported set, say “No supported findings in the assessed scope.”, or not assessed if nothing was. Lead incomplete reports with the helper's Status/Ran/Reason/Next lines. Never infer a clean result from setup failure or absent scanner output.

Present a compact SECURITY FINDINGS table with stable finding ID, severity, confidence/rationale, evidence state, location, and impact. Each finding needs an attacker scenario, supporting references, counterevidence considered, and a concrete repair recommendation. Include coverage, transformations, scanner versions/outcomes/freshness, runtime prerequisites, timing, and proposed repair-candidate paths. Include runtime_tested bundle paths only when an authenticated assertion witness actually produced one. Beside every bundle, show assertion, test-completion, and review assurance exactly as recorded; never collapse those labels or imply that self-reported project tests are authenticated. Comprehensive hypotheses belong in a separate labeled appendix.

Use RunReportV3, FindingV3, CoverageRecord, VerificationManifest, and RepairBundle from lib/cso/contracts.ts through the installed schema command. Helper validation/persistence is mandatory. If saving fails, report PERSISTENCE_FAILED and a safe summary; never claim the report was saved. If redaction fails, withhold the payload entirely; do not show raw excerpts to explain the failure.

The private state namespace is security/cso/<repo>/<run> under the existing state root, outside synchronization allowlists. Execution copies and services are removed immediately. Sanitized snapshots expire after seven days; redacted reports, repair candidates, and bundles after thirty days. Public archive caches are hash-verified, capped at 10 GiB, and evicted by LRU. Explicitly exported artifacts remain under user control. Repair candidates preserve their redacted request, patch, external observations, self-reported project-test outcome, provenance, and required input hashes. Authenticated bundles additionally preserve the inputs required for replay and their separate assurance labels. Currently issued runtime_tested bundles record authenticated external assertions and self_reported project-test completion.

Phase 14: Recovery and current-source rechecks

Give the run ID and the useful next operation. resume uses the retained snapshot and original policy; it never substitutes current HEAD or silently replenishes an exhausted budget. replay requires matching recorded source/runtime/dependency inputs. After snapshot expiry, missing matching supplied source returns MISSING_INPUT. A fresh audit is an explicit new run.

recheck snapshots current source and links the old finding. Establish closure only from new evidence covering the same root cause and security boundary. Resolution must identify fresh caller evidence and the original boundary by snapshot path/handle and line; the helper records its current hash or proves that boundary path is absent. A partial or incompatible audit, changed title, absent warning, or proposed bundle cannot resolve an old finding. Stable v3 identity uses root cause/location and advisory identities rather than generated titles. Import v2 reports read-only, retain the import ID, and use inspect-v2 to read the preserved redacted report. Legacy VERIFIED means legacy review evidence, never reproduced, runtime-tested, or tested.

For cancellation, deadline, or agent loss, let the helper's independent watchdog clean exact run-owned resources and preserve recoverable partial state. Never issue global Docker prune or remove resources identified only by a broad name. End with the actual completion status, supported results, exact prerequisites, and saved artifact paths returned by the helper.