Nightly Eval Investigation

Step 4: Perform Qualitative Deep-Dives (Investigation Playbook)

For each task listed in the Flagged Tasks Table of nightly_investigation_report.md, you must perform a qualitative deep-dive to locate the root cause and recommend actionable fixes.

[!IMPORTANT] CRITICAL COMPLIANCE RULES:

1. Do Not Modify Source Files: This is a passive/diagnostic investigation. Do NOT touch, edit, or modify any files under guides/ (such as task prompts, guide markdown files, or grader TypeScript files). All diagnostic findings and recommendations must be written only inside the markdown report.
2. Complete All Flagged Tasks: You MUST qualitatively investigate and fully populate the summary for every single flagged task. Leaving placeholder text or skipping any flagged task is unacceptable.
3. Strict Heading Omission: Under Diagnostic Details, only include headings/bullet points for the specific flags that were triggered for the task. Omit any non-triggered flags completely from the markdown file. Do not write "N/A" or "No issues".
4. Long-Running, Thorough Investigation: The qualitative investigation phase is a long-running, intensive task. You MUST thoroughly examine each flagged guide/test one by one, inspecting the target prompt (task.md), reference guide (guide.md), and grading code (grader.ts) to build a high-quality, task-specific diagnostic summary and actionable recommendation. Do not rush, skip steps, or bundle tasks.

Follow this linear playbook for each flagged task:

1. Locate Source Files Find the task's directory under the local workspace: guides/[category]/[use-case]/

   • Prompt: tasks/task.md
   • Guide: guide.md
   • Grader: grader.ts and expectations.md

2. Review Extracted Failed Assertions The script investigate.ts automatically extracts and intersects the failed assertions across all three agents, writing them directly into the report.

   • Omission Rule: This section is only populated if the task triggered the LOW_GUIDED_PASS_RATE flag. For all other flags, it is completely omitted.
   • Only assertions that failed across all three agents are included.
   • Review these common failures to analyze their root causes.

3. Map Flags to Diagnostic Explanations Analyze the source files based on the specific flags that were triggered, and write a diagnostic summary explaining why the task was flagged under each active flag.

   [!IMPORTANT] OMISSION RULE: Only include bullets for the specific flags that were triggered for the task. Omit any flags that were not triggered (do not write "N/A" or "No issues").

   • HIGH_UNGUIDED_PASS_RATE: Explain why unguided runs passed. Inspect tasks/task.md to see if the prompt is overly prescriptive (explicitly mentioning CSS attributes or API details that act as giveaways). Inspect grader.ts to check for loose/vacuous assertions.
   • LOW_GUIDED_PASS_RATE: Explain why guided runs failed. Inspect guide.md for bugs, outdated modules, or incorrect syntax. Inspect grader.ts for calibration drift or rigid checks. Check for browser/emulation issues in the sandbox.
   • MISSING_EXPECTED_GUIDE: Explain why the expected guide was missed. Inspect the task prompt in tasks/task.md to see if it lacks search keywords, or the guide metadata/synonyms in guide.md.
   • TOO_MANY_GUIDES_CONSUMED: Explain why multiple guides (3 or more) were consumed. Check for overly broad prompts that cause query sprawl, or overlapping guide descriptions.

4. Draft Specific, Actionable Recommendations Determine which recommendations to include based on the diagnostics mapped in previous steps. All recommendations must be directly justified by the findings from your investigation of the prompts, guides, and graders. Vague recommendations like "Fix prompt" or "Fix guide" are not acceptable; you must propose exact, concrete changes.

   [!IMPORTANT] OMISSION RULE: Only include recommendation lines for components that actually require changes. If a component does not require updates (e.g., the grader is correct as-is), you MUST OMIT the corresponding recommendation line completely from the markdown file (do not write "Keep as is" or "No changes").

   • Prompt: Propose the specific new phrasing or keywords to add/remove.
   • Guide: Specify the description, metadata, or content updates needed.
   • Grader: Detail the exact logic changes or assertions to modify/loosen.

5. Synthesize the Markdown Report Write the summary under the task's section in out/nightly-investigation/nightly_investigation_report.md following the template below.

Step 5: Inform the User of the Publishing Script

Once the qualitative diagnostics and recommendations have been fully written and saved to the markdown report, present the final report to the user and inform them that they can run the automated publisher script to create the GitHub parent issue and the task-level engineering/devrel subissues.

[!WARNING] DO NOT EXECUTE THE PUBLISHER SCRIPT YOURSELF: The agent must never run the publisher script (publish_report.ts) or create/publish any GitHub issues itself. It must only inform the user of the command so they can verify the report first and execute it manually.

node .agents/skills/nightly-eval-investigation/scripts/publish_report.ts

Step 6: Present and Link

Present the final report to the user, providing a clickable link, and remind them of the publishing command:

• Markdown Report: nightly_investigation_report.md

Report Format Standards

All investigation reports MUST strictly follow these templates to ensure downstream compatibility:

Markdown Standard (out/nightly-investigation/nightly_investigation_report.md)

# Nightly Evaluation Investigation Report

**Generated:** YYYY-MM-DD
**Suites Investigated:**
- **claude_code**: `[folder_name_1]`
- **codex_cli**: `[folder_name_2]`
- **jetski_cli**: `[folder_name_3]`

---

## Summary of Flagged Tasks

- **Total distinct tasks analyzed:** [total_count]
- **Total flagged tasks:** [flagged_count]

### Flagged Tasks Table

| Task / Guide | Flags | Details |
| :--- | :--- | :--- |
| `[task_name]` | `[FLAG_1]`, `[FLAG_2]` | [Detail explanation of flags] |

## Flagged Tasks Details

### `[task_name]`

#### Flags:
- **`[FLAG_1]`**: [Detail explanation]

#### Run Details:

| Agent | Guided Pass Rate | Unguided Pass Rate | Guides Consumed |
| :--- | :---: | :---: | :--- |
| **claude_code** | [guided_rate]% | [unguided_rate]% | `[guide_1]`, `[guide_2]` |
| **codex_cli** | [guided_rate]% | [unguided_rate]% | `[guide_1]` |
| **jetski_cli** | [guided_rate]% | [unguided_rate]% | *None* |

#### Qualitative Diagnostic Summary:

- **Diagnostic Details**:
  - *Failed Assertions*:
    - "[Exact assertion error message string]" (Test ID: `[test_id]`)
  - **[FLAG_1]**: TODO
  - **[FLAG_2]**: TODO

- **Actionable Recommendations**:
  - [ ] **Prompt**: TODO
  - [ ] **Guide**: TODO
  - [ ] **Grader**: TODO

---