Audit

Audit — Full Security Audit Methodology

10-phase security audit methodology for arbitrary source code repositories. Each phase defines what to analyze and what to produce. Orchestration (branching, state management, agent dispatch) is handled by the plugin commands.

When to Apply

• Full security audit of a repository
• Advisory regression check combined with static analysis
• Deep vulnerability research on a specific codebase
• Large or unusual architectures where default SAST modeling is likely incomplete
• "Run the security agents" / "audit for vulnerabilities" / "is this secure?"

Idempotency & Resumption

Before executing any phase, apply these rules to avoid redundant work:

• Skip completed phases: if the expected KB sections already exist and neither the codebase nor the threat-model inputs have fundamentally changed, reuse the existing content and mark the phase skipped.
• Re-audit as KB update: for re-audits, piolium/attack-surface/knowledge-base-report.md from the previous run is the starting knowledge base. Do not regenerate it from scratch. Load it, diff the codebase against the commit field of the last completed entry in piolium/audit-state.json, and update only the sections whose source inputs changed. Mark unchanged sections with [reused from <short-sha>]. Documentation-only changes require no re-audit; changes to auth, core business logic, or attack-surface components invalidate the Static Analysis Summary and Phase 10 Addendum (## Phase 10 Addendum, written by the Review Chambers) sections at minimum.
• Partial resumption (Phases 7-9): if piolium/findings-draft/ already contains draft files, resume from the existing drafts rather than starting fresh.
• State recording: piolium/audit-state.json is append-only. Before starting a new audit, append a new entry to the audits array with status: "in_progress". After each phase completes or fails, update that entry's phases map in-place. Never overwrite or remove earlier entries — they are the permanent audit history. See references/report-templates.md for the full schema.

Setup

Before executing Phase 1, perform the following environment checks:

1. Security folder dirtiness check: Run git status piolium/ to determine if piolium/ has uncommitted changes or untracked files.

2. Concurrent agent detection: Check whether another agent is actively writing to the source code tree. Indicators include uncommitted modifications outside piolium/ in the working tree (e.g., staged edits, new untracked source files) or the presence of .claude/ task state that suggests a live session.

3. Worktree isolation: If BOTH conditions hold — piolium/ is dirty AND another agent is actively modifying source code — create an isolated git worktree pinned to the current HEAD so the audit analyses a stable snapshot and writes to a conflict-free path:

   AUDIT_WT="../audit-$(basename "$PWD")-$(date +%s)"
   git worktree add --detach "$AUDIT_WT" HEAD

   Run the entire audit from inside $AUDIT_WT. All piolium/ output is written to the worktree copy. At the end of Phase 15, merge the retained artifacts back and tear down the worktree:

   cp -r "$AUDIT_WT/piolium/attack-surface/"        piolium/
   cp -r "$AUDIT_WT/piolium/final-audit-report.md"  piolium/
   cp -r "$AUDIT_WT/piolium/findings/"              piolium/
   git worktree remove --force "$AUDIT_WT"

   If only one condition holds (security is dirty but no concurrent agent, or a concurrent agent exists but security is clean), proceed without a worktree — no isolation needed.

10-Phase Workflow

flowchart TD
    Start["Audit Request"] --> Setup["Setup + reuse check"]
    Setup --> P1["1. Intelligence Gathering"]
    P1 --> P2["2. Patch Bypass Analysis (per-patch parallel)"]
    P2 --> P3["3. Knowledge Base"]
    P3 --> P4["4. Static Analysis (incl. inline Enrichment)"]
    P3 --> P6["6. Spec Gap Analysis"]
    P4 --> P7["7. Deep Bug Hunting"]
    P6 --> P7
    P7 --> P8["8. P11-LITE: FP Check + Cold Verification (CRITICAL/HIGH only)"]
    P8 --> P9["9. Variant Analysis"]
    P9 --> P10["10. Exploitation & Final Reporting"]

Phase 1 — Intelligence Gathering

Use the advisory-hunter workflow to collect:

• advisories, CVEs, GHSAs, and patch commits
• coarse architecture inventory: components, transports, execution contexts, trust boundaries
• a general component inventory ("SBOM"): every software component the target directly relies on across all categories — runtimes, packages, frameworks, datastores, external services, container/OS layer, build/CI tooling, shelled-out binaries, and vendored code — written to piolium/attack-surface/sbom.json
• security-relevant dependencies: the security_relevant subset of that inventory, with runtime context noted for each one. Use the supply-chain-risk-auditor skill to systematically assess the flagged components' risks.

Treat dependency findings as hypotheses until the audit proves the affected runtime path is reachable.

Write all findings to the ## Advisory Intelligence and ## Component Inventory sections of piolium/attack-surface/knowledge-base-report.md, and the full inventory to piolium/attack-surface/sbom.json.

Phase 2 — Patch Bypass Analysis

For each advisory patch:

• fetch the full diff and surrounding callers
• test bypass hypotheses: alternate entry points, config-gated checks, default-state gaps, compatibility branches, parser differentials, missing normalization
• check whether a sibling or related path remains vulnerable even if the patched path is sound
• cluster duplicate advisories by the same upstream commit or PR so one fix is not re-audited as multiple distinct bugs

Write all findings to the ## Bypass Analysis section of piolium/attack-surface/knowledge-base-report.md.

Phase 3 — Knowledge Base

Build the project model from source:

• classify project type: web app, API, CLI, desktop, library, plugin, protocol, worker, CI action
• map attacker-controlled inputs, trust boundaries, and security-critical decisions
• build compact DFD/CFD slices only for the highest-risk flows
• record implemented specs and RFCs
• Use the security-threat-model skill to formally document the threat model.
• Run Domain Attack Research (Modes A, B, and C) to build a domain-specific attack playbook: use last30days, wooyun-legacy, MCP tools, and web search. See references/domain-attack-playbooks.md for the research action sequence and per-domain templates.

Before completing Phase 3, add a ## Phase 4 CodeQL Extraction Targets section to the KB draft. For each high-risk DFD slice, record the expected CodeQL source type (RemoteFlowSource, LocalUserInput, EnvironmentVariable) and the expected sink kind (sql-execution, command-execution, file-access, http-request, code-execution, deserialization). This is the data contract that tells Phase 4's structural extractor which per-slice call graph queries to run. Leave the section blank if no DFD slices were identified — structural extraction will run generic enumeration only.

Produce piolium/attack-surface/knowledge-base-report.md with all Phase 3 sections populated. This is the single knowledge base file for the entire audit. The threat model and attack surface content live inside it as sections — no separate files.

The Phase 3 threat model is mandatory input for all later phases.

Phase 4 — Static Analysis

CRITICAL ENFORCEMENT: You MUST physically execute the SAST tools. Do not hallucinate results or skip execution. You must ensure codeql successfully runs and that semgrep uses the Pro engine (--pro) exclusively. Standard Semgrep is only permitted as a fallback when Pro is unavailable due to an authentication or licensing failure; document the fallback and its reason in the report.

Sub-step 4.1 — Structural Extraction (runs first, before security scan)

Before running any security suite, build the CodeQL database and store it at piolium/codeql-artifacts/db/ (not a transient path). Do not delete it after this sub-step.

Follow the ## Structural Extraction Workflow in references/architecture-aware-sast.md to produce:

• piolium/codeql-artifacts/entry-points.json
• piolium/codeql-artifacts/sinks.json
• piolium/codeql-artifacts/call-graph-slices.json
• piolium/codeql-artifacts/flow-paths-raw.sarif (git-ignored, retained until Phase 12)
• piolium/codeql-artifacts/flow-paths-all-severities.md
• Machine-generated DFD and CFD Mermaid diagrams embedded in piolium/attack-surface/knowledge-base-report.md

Populate the ## CodeQL Structural Analysis section of piolium/attack-surface/knowledge-base-report.md after extraction completes.

Sub-step 4.2 — Security Scan

Baseline requirements:

• Delegate to the codeql skill to run built-in security suites against the database built in 4.1.
• Delegate to the semgrep skill with --pro enforced for all passes (baseline, language, framework, and custom). Fall back to standard Semgrep only if the Pro engine fails with an authentication or licensing error; document the fallback reason in the ## Static Analysis Summary section of piolium/attack-surface/knowledge-base-report.md.
• Explicitly output the list of CodeQL queries and Semgrep rules that you actually ran.
• For Java applications, run SpotBugs with the FindSecBugs plugin as a required baseline pass; treat this as additive to CodeQL and Semgrep.
• Run GitHub Actions review with agentic-actions-auditor when .github/workflows/ exists; write results to the ## GitHub Actions Audit section of piolium/attack-surface/knowledge-base-report.md.

Custom Architecture Generalization (Dynamic Rules):

• Do not solely rely on generic or pre-baked rules. You MUST dynamically generate custom CodeQL queries and Semgrep rules specifically tailored to the ad-hoc architecture, framework, and threat model identified in Phase 3 (e.g., custom MCP protocols, specific custom RPC boundaries).
• Store all dynamically generated custom rules in piolium/codeql-queries/ and piolium/semgrep-rules/.
• Document exactly what custom rules were created, why they match the Phase 3 architecture, and their execution results in the ## Static Analysis Summary section of piolium/attack-surface/knowledge-base-report.md.

Operational rules:

• Keep SAST concurrency low enough to avoid exhausting CPU/RAM.
• Merge SARIF outputs with sarif-parsing if needed.

Sub-step 4.3 — Inline Enrichment (Security Relevance Filter)

After SAST runs complete and before deep bug hunting, classify each candidate finding as one of:

• likely security
• likely correctness/robustness
• likely environment/tooling/admin-only

For every candidate, answer:

1. What attacker controls the input?
2. Which runtime executes the vulnerable path?
3. What trust boundary is crossed?
4. Is the effect cross-user, cross-tenant, cross-privilege, or only same-user?
5. Is the vulnerable dependency/code path actually used in that runtime?
6. Query piolium/codeql-artifacts/call-graph-slices.json for the finding's source-to-sink slice. If reachable: true, that strengthens the finding. If reachable: false and both source and sink appear in the enumeration files, that is evidence to downgrade — document the discrepancy. For findings without a pre-computed slice, run an on-demand query against piolium/codeql-artifacts/db/.

Downgrade or exclude by default when the issue is only:

• build-time, source-controlled, CI-only, test-only, or dev-only
• browser-only usage of a server-side CVE, or server-only usage of a browser-side CVE
• same-user state/cache/UI correctness without a broader data boundary break
• admin safety, migration robustness, retry/deadlock hardening, data-loss prevention, or workflow correctness
• local tooling behavior where the attacker already has equivalent code execution
• assessable as Low severity after answering the questions above — drop immediately; do not carry forward to Phase 10

Write enrichment verdicts to the ## SAST Enrichment section of piolium/attack-surface/knowledge-base-report.md. In the ## CodeQL Structural Analysis section, note any entry points from entry-points.json not present in the Phase 3 DFD slices, and any sinks from sinks.json mapping to unmodeled high-risk flows.

Sub-step 4.4 — Cleanup

Delete Semgrep cache, semgrep-res/, and codeql-res/. Do not delete piolium/codeql-artifacts/db/ — it is retained for Phases 7 and 9. Full database deletion happens at the end of Phase 12.

Phase 9 — Spec Gap Analysis

If the repo implements specs or RFCs:

1. Read the ## Domain Attack Research section of piolium/attack-surface/knowledge-base-report.md first — it contains pre-computed domain attack patterns from Phase 3 that directly inform which spec gaps to prioritize.
2. Fetch the relevant documents using built-in web search or fetch tools (do not restrict yourself to MCP tools).
3. Research the RFC for historical attacks, known edge cases, and common implementation failures. Cross-reference against the domain attack playbook from Phase 3.
4. Use spec-to-code-compliance.
5. Focus on parsing, normalization, sanitization, canonicalization, and state-machine compliance.
6. Identify gaps between the RFC spec and the codebase implementation clearly.
7. Keep only medium-to-critical findings with a credible exploit path.

Write all findings to the ## Spec Gap Analysis section of piolium/attack-surface/knowledge-base-report.md. If no specs or RFCs were identified in Phase 3, mark the section "None identified" and skip.

Phase 10 — Review Chamber Deep Bug Hunting

Phase 10 uses a Review Chamber multi-agent debate system. Instead of a single deep-auditor agent, four specialized roles collaborate through structured argumentation to produce findings with higher creativity and lower false-positive rates.

Chamber Formation

After Phase 4 (SAST + inline enrichment) and Phase 9 (spec gap) complete:

1. Read ## High-Risk DFD Slices and ## High-Risk CFD Slices from piolium/attack-surface/knowledge-base-report.md
2. Group slices by shared trust boundary or component affinity into threat clusters
3. Each cluster becomes one Review Chamber (typical audit: 3-8 chambers)
4. Priority: authentication/authorization first, then data ingestion, then API surface

Create piolium/chamber-workspace/ and piolium/attack-pattern-registry.json. The orchestrator seeds the registry as {"patterns": []} before this phase — append to it, and do not delete it even when no chamber confirms a new pattern. Phase 12 reads it.

Four Debate Roles

Each chamber spawns four agents that communicate through an append-only debate transcript at piolium/chamber-workspace/<chamber-id>/debate.md:

• Attack Ideator (Red Team Creative): generates 3-7 attack hypotheses per cluster by cycling through 8 creative modes — vulnerability chaining, business logic abuse, race conditions/TOCTOU, second-order/stored attacks, trust boundary confusion, parser/protocol differentials, state machine attacks, and supply chain interaction. See references/creative-attack-modes.md. Does NOT trace code or issue verdicts.

• Code Tracer (Technical Analyst): takes each hypothesis and traces it through actual code. Uses Method 2.6 from references/deep-analysis.md (call-graph slices, entry-points.json, sinks.json, flow-paths-all-severities.md, on-demand QL queries). Produces reachability verdicts (REACHABLE / UNREACHABLE / PARTIAL) with file:line evidence chains. Does NOT generate hypotheses or issue final verdicts.

• Devil's Advocate (Challenger): challenges EVERY finding. Searches 5 protection layers (language, framework, middleware, application, documentation). Checks all 8 Claude-Specific FP patterns from references/triage-and-prereqs.md. Must argue against even obvious vulnerabilities — inability to construct credible defense is itself strong evidence. Does NOT generate hypotheses or issue verdicts.

• Chamber Synthesizer (Coordinator + Judge): orchestrates debate rounds, reads all arguments, resolves disputes, assigns calibrated severity per references/triage-and-prereqs.md, and writes finding drafts. Only role that writes to piolium/findings-draft/. Manages the attack pattern registry. May request up to 2 follow-up investigation rounds per hypothesis.

Optional 5th role — Variant Scout: monitors debate for confirmed patterns and concurrently searches for structural variants in sibling components, front-loading Phase 12 work.

Debate Protocol

Each chamber proceeds through structured rounds:

Round 1 (Ideation):  Ideator generates 3-7 hypotheses
Round 2 (Tracing):   Tracer traces each hypothesis through code
Round 3 (Challenge): Advocate writes defense brief per hypothesis
Round 4 (Synthesis): Synthesizer evaluates arguments, issues verdicts
Round 5-6 (Optional): Focused re-investigation on unresolved hypotheses

Convergence criteria — debate ends for a hypothesis when:

• Tracer: UNREACHABLE + Advocate confirms no alternate path → DROP
• Tracer: REACHABLE + Advocate cannot find blocking protection (2 attempts) → VALID
• Tracer: REACHABLE + Advocate finds blocking protection → FALSE POSITIVE
• 3 rounds without resolution → Synthesizer judgment call or INCONCLUSIVE
• Low severity after calibration → DROP (low severity)

Limits: max 7 hypotheses per batch, max 3 rounds per hypothesis, max 3 concurrent chambers.

See references/chamber-protocol.md for complete debate format, transcript template, and convergence rules.

Pre-Finding Quality Gate

Before writing any finding draft, the Synthesizer applies this 5-point check:

1. Attacker control verified by Tracer (not just inferred)?
2. Framework protection searched by Advocate (all 5 layers)?
3. Trust boundary crossing confirmed (not same-origin)?
4. Exploitation requires normal attacker position (not admin)?
5. Vulnerable code ships to production (not test/example)?

If any check fails, drop the finding. If ambiguous, add Pre-FP-Flag: check-N-ambiguous to the draft for Phase 11 priority.

Cross-Chamber Intelligence

Chambers share a pattern registry at piolium/attack-pattern-registry.json. When a Synthesizer confirms a finding, it adds the root cause pattern with detection signatures (CodeQL, grep, Semgrep). Other chambers read the registry before new ideation rounds, enabling cross-domain pattern discovery.

Output

• Finding drafts: piolium/findings-draft/p7-<NNN>-<slug>.md (Medium+ only, Low dropped)
• Debate transcripts: piolium/chamber-workspace/<chamber-id>/debate.md (audit artifact)
• Variant candidates: piolium/chamber-workspace/<chamber-id>/variant-candidates/ (for Phase 12)
• Pattern registry: piolium/attack-pattern-registry.json (for Phases 8, 9)

KB Feedback Loop

After all chambers close, append a ## Phase 10 Addendum section to piolium/attack-surface/knowledge-base-report.md containing: newly discovered attack surfaces, revised trust boundary assumptions, and additional DFD/CFD paths found during chamber debates. Forward-append only — Phase 3 content preserved for auditability.

Specialized Skill Delegation

Chambers may delegate to specialized skills for scope NOT already covered by Phase 3 domain attack research:

• insecure-defaults — fail-open configurations, weak auth defaults
• sharp-edges — API design issues, dangerous configurations
• wooyun-legacy — web vulnerability techniques
• zeroize-audit — C/C++/Rust secret handling

Context: Read references/chamber-protocol.md, references/creative-attack-modes.md, references/deep-analysis.md, and references/triage-and-prereqs.md.