Collab Audit

/collab-audit — AI Collaboration Audit

Purpose

Analyzes conversation history, artifacts, and work patterns to generate behavioral and psychological insights. Based on direct behavioral observation during actual work rather than self-report surveys. More accurate than survey-based methods.

Dominant Variable

Whether the analysis infers reasons behind observed patterns, not just lists facts (facts alone ≠ success)

Trigger

• /collab-audit
• "AI 협업 진단"
• "협업 분석"
• "AI 협업 진단해줘"

Discard If

• Fewer than 2 observed sessions AND fewer than 100 messages — insufficient sample to extract patterns
• Simple code review request → use code-reviewer instead
• Only quantitative session metrics needed → use project-check instead

Key Assumptions

1. 2+ observed sessions OR 100+ messages (meeting either one is enough) — if both are broken: Discard If triggers.
2. Access to memory/session-handoff-LATEST.md — if broken: handoff pattern analysis unavailable, skip that dimension.

Mode Detection (execute first)

• Input contains /collab-audit compare or "compare" → Compare mode (separate section below)
• Otherwise → Audit mode (14-section analysis)

Input Validation (Step 0 — execute first)

Minimum conditions — one of:

• 2+ sessions
• 100+ messages
• Single-session high-density exception → refer to Invariant 4 criteria. If exception met, mark ⚠ Single-session analysis — pattern confidence limited then proceed.

Deterministic gate (preferred): if session-meta JSON files are available (one per session, with fields such as message_count, artifact_count, originator, first_message, cwd), run scripts/session_hygiene_scan.py --meta-dir <DIR> and read meets_minimum / single_session_exception straight from its JSON output — do not re-derive these booleans by eyeballing the transcripts. Fall back to manual counting only when no such metadata directory exists (e.g. a single live conversation with no file access).

If session-meta JSON does not exist yet, build it first from real Claude Code session JSONL (the gate cannot run on raw JSONL):

python scripts/extract_session_meta.py --out-dir <scratchpad>/collab-meta --days 30 --max 20
python scripts/session_hygiene_scan.py --meta-dir <scratchpad>/collab-meta

The extractor reads ~/.claude/projects by default (--projects-dir overrides) and stores counts only, never prompt text; its docstring is the canonical field definition.

If all unmet:

"Insufficient data — minimum 2 sessions or 100 messages required. Currently [N] messages, [M] artifacts observed."

Output and stop immediately. Reject even "prediction-based" requests.

Workflow

Step 0.5: Tone Detection (determine delivery intensity)

Determines delivery intensity for Section 11 blind spots only. Other sections are factual, so tone variance is minimal.

Read signals from conversation patterns. Long messages mix two causes — explaining feelings/context and pasting code or logs — so exclude long messages that are more than half code blocks (```) or log lines from the style signal. If the deterministic gate (Step 0.6) was not actually run, write "manual count — script not run" in the report; never present it as a script result.

• High ratio of short, direct messages / "facts only" / speed-first requests → Direct (maintain current default)
• High emotion expression frequency / preference for long explanations / feedback-receptive signals → Calibrated (same blind spot content, but provide context before delivery)

Mark result 1 line before Section 11: [Delivery intensity: Direct / Calibrated]

Mid-session re-assessment: If conversation tone shifts noticeably (emotion spike, request method change, defensive responses appear), re-assess right before Section 11 output. Initial assessment does not lock in the entire session.

Important: Changing delivery intensity does not change blind spot content (accuracy). Only adjusts temperature.

Step 0.6: Source Hygiene Filter (deterministic-first measurement)

When multiple observation sources exist (e.g. session JSONLs), determine — before analysis — whether each source is an organic user session or an automation byproduct. A qualitative caveat alone (the old Step 1 approach) is not enough; automated sessions can be mistaken for user behavior.

Deterministic gate: run scripts/session_hygiene_scan.py --meta-dir <DIR> against the directory of session-meta JSON files. The script classifies every session into one of three states — include (organic) / uncertain (cannot confirm either way) / exclude (confidently auto-derived) — and returns included_count, excluded_count, uncertain_count, meets_minimum, and single_session_exception in one JSON object — read those fields directly rather than re-judging exclusion by eye. --meta-dir reads *.json files non-recursively (files in subfolders are ignored). Required field types: message_count (int), artifact_count (int), deep_conversation_ratio (float), session_meta.source.thread_spawn (bool) — a wrong type classifies the session as unreadable. If unreadable > 0, report the count of unreadable sessions and mark the result PARTIAL.

Detection criteria the script applies:

• Confident exclude: session metadata contains auto-derivation markers such as subagent/thread_spawn/agent_nickname; OR cwd matches a naming convention specific to paired/multi-arm experiment harnesses (e.g. pair-run, arm-a/arm-b, ab-test); OR originator is an SDK/bot/exec-type process with no direct user-input signal (natural conversational opening message) present.
• Uncertain (needs review, not auto-folded either way): cwd contains only a generic automation-adjacent word (currently: pipeline) with no other automation marker — a real user project named e.g. data-pipeline-tool must not be silently misclassified as an automated harness just because the word appears; OR the session-meta object is present but empty (no fields at all) — an empty object is not evidence of an organic session and must not be auto-classified as one.
• Include (organic): none of the above, and the object carries actual fields.

Malformed metadata (session-meta root is not a JSON object, e.g. [], or a count field like message_count/artifact_count is a non-numeric type such as a string) is routed to unreadable, not silently coerced or crashed on — count fields must be actual numbers, not something that merely looks numeric.

Exclusion: sessions the script flags exclude are removed from the analysis population; report the exclusion count and reason in 1 line straight from the script output (e.g. "16 of 16 sessions excluded — all were thread_spawn subagent sessions"). Do not substitute a qualitative impression ("seems skewed toward one type") for the script's explicit denominator.

Uncertain handling: sessions flagged uncertain are neither included nor excluded automatically. Report the uncertain_count and list the reasons in 1 line, then ask the user to confirm (or apply their own knowledge of which sessions are real) before deciding whether to fold each one into the analysis population — do not silently default uncertain sessions to either bucket.

Skip condition: if the only observation source is the current conversation (no multi-session file access, no session-meta JSON available), the script cannot run — fall back to the qualitative criteria above and proceed to the next step.

Step 1: Data Collection

Collect all available observation sources, restricted to organic sessions surviving Step 0.6:

• Current session conversation history (message length, frequency, content)
• MEMORY.md, session-handoff files (if present, Read access)
• User-created artifacts (code, documents, config files — if present, Read)
• Tool usage patterns (which tools requested, how often)

Post-collection disclosure (mandatory): State analysis limitations — [1-2 skewed work types], [whether failures/abandonment observed] in 1-2 lines, then proceed to Step 2. If data skews toward specific domain (e.g., coding only, conversation only), flag it.

Step 2: Evidence Mapping

Extract evidence needed for each section first. Secure evidence before output.

• Section without evidence → mark "Observation unavailable — no supporting data". Do not omit.
• Sections 7-8 (Claude-specific): If no Claude usage patterns → "No Claude usage data — N/A"

Step 3: Framework Application

Fit collected evidence into each section's analysis framework. Link behavioral evidence to all framework labels (MBTI, DiSC, etc.) — mandatory. Outputting labels without evidence is analysis failure.

Step 4: Output in 14-Section Order

Do not change section order or arbitrarily omit sections. For data-empty sections, mark "Observation unavailable" then proceed to next.

Step 5: File Save + Gitignore Protection

1. Include file header:

   ---
   profile_version: 1.0 # bump when the section count or section names change; a report without this frontmatter is treated as `legacy` in compare mode
   sections: 14
   date: YYYY-MM-DD
   language: [ko|en]
   ---
   
   # MAGIC DOC: AI Collaboration Audit YYYY-MM-DD

2. Save to ~/.claude/collab-audits/YYYY-MM-DD.md.
   • Re-run same day: use -2.md suffix (no overwrite)
3. Check ~/.claude/.gitignore:
   • If missing → create .gitignore with single line collab-audits/.
   • If exists but missing collab-audits/ → add that line.
   • If already present → do not modify.
   • Verify actual tracked status — do not infer it from the .gitignore entry alone: a pattern present in .gitignore does not retroactively untrack a file that was already committed before the pattern existed. Run git -C ~/.claude rev-parse --is-inside-work-tree first; if that fails, ~/.claude is not a git repo and the entry is simply inert (state that, not "blocked"). If it is a repo, run git -C ~/.claude ls-files --error-unmatch collab-audits/ 2>&1 — a non-error match means one or more files under collab-audits/ are already tracked despite the ignore rule.
4. After save, display in conversation:
   • Verified untracked (not a git repo, or .gitignore present and ls-files finds no tracked match under collab-audits/):

     Saved: ~/.claude/collab-audits/YYYY-MM-DD.md
     ⚠ Personal audit result — git tracking blocked (~/.claude/.gitignore)

   • ls-files shows this file (or another file under collab-audits/) is already tracked:

     Saved: ~/.claude/collab-audits/YYYY-MM-DD.md
     🔴 Already tracked by git despite .gitignore — adding a pattern does not retroactively untrack committed files. Run: git -C ~/.claude rm --cached <path> to actually untrack it.

Output Structure (14 sections, fixed order)

1. Artifact Structure Analysis

Reverse-engineer values from creations (code, documents, systems).

• Architecture choices → connect to philosophy
• Naming patterns, file structure, comment density
• Presence of hard rules? If so, what principles?
• Attribution classification: Classify observed artifacts as User-led / AI-assisted / Co-created. If inseparable, mark Co-created and explicitly downgrade that section's confidence.
• No artifacts → "Observation unavailable — no artifact data"

2. Communication Pattern

• Message length distribution (short confirmation ratio vs long explanation ratio)
• When shortened, when lengthened (identify triggers)
• Emotion expression style (direct/indirect, intensity)
• Closure expression patterns ("done", "understood", etc.)

3. Question Typology

Classify questions on two axes:

• Confirmation type: information collection then immediate decision
• Tracking type: tracing causes/intentions
• Ratio of both types + context where each appears

4. Delegation & Trust Structure + Maturity Level

Maturity level assessment (choose one):

• L1 Paste type: uses results as-is, no verification
• L2 Review type: verifies then uses
• L3 Delegation type: gives conditions, delegates, verifies result
• L4 System type: pre-controls AI behavior via rules and guard rails

Delegation vs ownership:

• What is delegated (exploration, analysis, implementation)
• What is never delegated (judgment, prioritization, timing)

5. Failure & Blockage Response + Recovery Strategy

Recovery strategy classification (state observed types):

• Identical prompt repeat type
• Workaround path search type
• Problem redefinition type
• Abandon then manual handling type
• Explicit hold then restart type

How is blockage distinguished from failure? Is failure logged in the system?

Collaboration anti-pattern flags (observation-only — omit if not observed):

<!-- Anti-pattern flags for repeated inefficiencies -->

Flag only inefficiency habits observed 2+ times. Exclude one-offs. Unlike blind spots (Section 11), evidence is behavioral frequency count — N observations only, no speculation.

• Repeated failure without recovery: retries identical prompt 3+ times without strategy change (above "identical prompt repeat" chronically)
• Repeated delegation without verification: L1 paste (Section 4) repeats on irreversible work
• Context re-request repeats: resets then unused handoff, re-explains same info (Section 8 immaturity signal)
• Design omission → repeated rollback: enters without prior design, high rollback (Section 9) repeats State observation count per flag (e.g., "repeated failure without recovery — 3 observations"). Omit if no count.

6. Energy Distribution + Time Horizon (combined)

Energy landscape:

• Work types dwelled on longest
• Work types skipped quickly
• Token (conversation length) to output (files, code, decisions) ratio → verbose vs execution-focused

Time horizon structure:

• Immediate / short-term / conditional (e.g., "after hardware") / perpetually held classifications
• Per-session task volume (how many tasks digested per session)

7. Tool Usage Pattern (Claude-specific)

If no Claude usage data → mark "N/A" and proceed to next section.

• Read/Grep/Glob ratio vs Bash reliance — "direct type" vs "search type"
• Agent spawn frequency — "delegation type" vs "direct execution type"
• Top 3 frequently used tools, rarely used tools
• New tool adoption speed — immediate adoption vs wait-and-see

8. Context Management Maturity (Claude-specific)

If no Claude usage data → mark "N/A" and proceed to next section. Level assessment (choose one):

• L0: No CLAUDE.md, re-explain every session
• L1: CLAUDE.md present but static (no updates)
• L2: MEMORY.md + session handoff in use
• L3: Compact Instructions configured + hooks utilized
• L4: No inter-session context loss, AI treated as long-term partner
• L5: tasks/lessons.md in use — AI behavior correction loop exists. Meta-layer built to convert repeated mistakes into rules

Also document recovery patterns after context loss.

9. Rollback Frequency (Rollback Pattern)

Measure "undo", "revert", "remove that" frequency.

• High: signals lack of brainstorming/planning
• Low: mature pre-design OR no verification (distinguish direction)
• Post-rollback retry pattern — same direction retry vs direction shift

10. Psychological Framework Mapping

Link behavioral evidence to each framework and mark confidence (High/Medium/Low).

10-A. Reader AI User Type Classification (most important) Judge primary + secondary types:

• Designer type: System, rules, architecture first. AI as implementation tool
• Executor type: Fast results first. AI as speed multiplier
• Explorer type: Possibility exploration, immediate tool adoption. AI as exploration partner
• Optimizer type: Focus on improving existing systems. AI as tuning tool

10-B. MBTI Indicators (apply only when observation data actually supports it — if evidence is weak, mark that axis Observation unavailable/Low and it may be skipped) 4-axis direction + strength estimate per axis. Mark confidence.

10-C. DiSC Profile (apply only when observation data actually supports it — if evidence is weak, mark that axis Observation unavailable/Low and it may be skipped) D/i/S/C proportion estimate. Primary + secondary style.

10-D. Enneagram Hypothesis (apply only when observation data actually supports it — if evidence is weak, mark that axis Observation unavailable/Low and it may be skipped) Type + Wing hypothesis. Format "this behavior supports it" — minimum 2 evidence pieces.

10-E. Big Five Estimate (apply only when observation data actually supports it — if evidence is weak, mark that axis Observation unavailable/Low and it may be skipped) O/C/E/A/N each High/Medium/Low. One behavioral basis per dimension.