--- name: debugging-workflows description: Diagnose gh-aw failures using logs and audits; follow the shared strategy for patches and permitted active debug loops. --- # Workflow Diagnosis and Debugging Evidence Use this reference to diagnose workflows: download/analyze existing logs, audit runs, and trace failures. These reads are not an active debug loop. Follow the [shared local-first strategy](../../aw/debug-agentic-workflow.md) for all reproduction, fixes, uploads, and live tests. This page is an evidence and CLI reference, not a separate execution policy. Respect explicit no-dispatch contexts. Apply its live-outcome table, credential triage and untrusted-evidence rules. Without accessible existing logs, use source/fixtures; never dispatch for evidence. ## Table of Contents - [Quick Start](#quick-start) - [Downloading Workflow Logs](#downloading-workflow-logs) - [Auditing Specific Runs](#auditing-specific-runs) - [How Agentic Workflows Work](#how-agentic-workflows-work) - [Common Issues and Solutions](#common-issues-and-solutions) - [Advanced Debugging Techniques](#advanced-debugging-techniques) - [Reference Commands](#reference-commands) ## Quick Start ### Download Logs from Recent Runs ```bash # Download logs from the last 24 hours gh aw logs --start-date -1d -o .github/aw/logs/recent # Download logs for a specific workflow gh aw logs weekly-research --start-date -1d # Download logs with JSON output for programmatic analysis gh aw logs --json ``` ### Audit a Specific Run ```bash # Audit by run ID gh aw audit 1234567890 # Audit from a GitHub Actions URL gh aw audit https://github.com/owner/repo/actions/runs/1234567890 # Audit with JSON output gh aw audit 1234567890 --json ``` ## Downloading Workflow Logs The `gh aw logs` command downloads workflow run artifacts and logs from GitHub Actions for analysis. ### Basic Usage ```bash # Download logs for all workflows (last 10 runs) gh aw logs # Download logs for a specific workflow gh aw logs # Download with custom output directory gh aw logs -o .github/aw/logs/custom ``` ### Filter Options ```bash # Filter by date range gh aw logs --start-date 2024-01-01 --end-date 2024-01-31 gh aw logs --start-date -1w # Last week gh aw logs --start-date -1mo # Last month # Filter by AI engine gh aw logs --engine copilot gh aw logs --engine claude gh aw logs --engine codex # Filter by count gh aw logs -c 5 # Last 5 runs # Filter by branch/tag gh aw logs --ref main gh aw logs --ref feature-xyz # Filter by run ID range gh aw logs --after-run-id 1000 --before-run-id 2000 # Filter firewall-enabled runs gh aw logs --firewall # Only firewall-enabled gh aw logs --no-firewall # Only non-firewall ``` ### Output Options ```bash # Generate JSON summary gh aw logs --json # Parse agent logs and generate Markdown reports gh aw logs --parse # Generate Mermaid tool sequence graph gh aw logs --tool-graph # Set download timeout gh aw logs --timeout 300 # 5 minute timeout ``` ### Downloaded Artifacts When you run `gh aw logs`, the following artifacts are downloaded for each run: | File | Description | |------|-------------| | `aw_info.json` | Engine configuration and workflow metadata | | `safe_output.jsonl` | Agent's final output content (when non-empty) | | `agent_output/` | Agent logs directory | | `agent-stdio.log` | Agent standard output/error logs | | `aw.patch` | Git patch of changes made during execution | | `workflow-logs/` | GitHub Actions job logs (organized by job) | | `summary.json` | Complete metrics and run data for all runs | ### Example: Analyze Recent Failures ```bash # Download failed runs from last week gh aw logs --start-date -1w -o .github/aw/logs/debug # Check the summary for patterns cat .github/aw/logs/debug/summary.json | jq '.runs[] | select(.conclusion == "failure")' ``` ## Auditing Specific Runs The `gh aw audit` command investigates a single workflow run in detail, downloading artifacts, detecting errors, and generating a report. ### Basic Usage ```bash # Audit by numeric run ID gh aw audit 1234567890 # Audit from GitHub Actions URL gh aw audit https://github.com/owner/repo/actions/runs/1234567890 # Audit from job URL (extracts first failing step) gh aw audit https://github.com/owner/repo/actions/runs/1234567890/job/9876543210 # Audit from job URL with specific step gh aw audit https://github.com/owner/repo/actions/runs/1234567890/job/9876543210#step:7:1 ``` ### Output Options ```bash # JSON output for programmatic analysis gh aw audit 1234567890 --json # Custom output directory gh aw audit 1234567890 -o ./audit-reports # Parse agent logs and firewall logs gh aw audit 1234567890 --parse # Verbose output gh aw audit 1234567890 -v ``` ### Check Whether an Error Recurs ```bash # Compare existing runs, without dispatching new ones gh aw audit 1234567890 1234567891 1234567892 --group --json ``` Use grouped per-run finding codes/counts, then cached individual reports/logs for exact signatures. Plain multi-run diffs focus on metrics/firewall/tools; absent findings or skipped runs do not prove the error disappeared. Match the first failing boundary and normalized error/tool/status signature across comparable workflows, revisions, triggers/inputs and configurations. Count each matching run once, report matching/inspectable runs and IDs, and keep missing evidence unknown. Repeated HTTP 403 alone does not establish one root cause. ### Audit Report Contents The audit command provides: - **Error Detection**: Errors and warnings from workflow logs - **MCP Tool Usage**: Statistics on tool calls by the AI agent - **Missing Tools**: Tools the agent tried to use but weren't available - **Execution Metrics**: Duration, token usage, and cost information - **Safe Output Analysis**: What GitHub operations were attempted ### Example: Investigate a Failed Run ```bash # Get detailed audit report gh aw audit 1234567890 --json > audit.json # Extract key information cat audit.json | jq '{ status: .overview.status, conclusion: .overview.conclusion, errors: .errors, missing_tools: .missing_tools, tool_usage: .tool_usage }' ``` ## How Agentic Workflows Work Understanding the workflow architecture helps in debugging. ### Workflow Structure Agentic workflows use a **markdown + YAML frontmatter** format: ```markdown --- on: issues: types: [opened] permissions: contents: read timeout-minutes: 10 engine: copilot tools: github: mode: remote toolsets: [default] safe-outputs: staged: true create-issue: labels: [ai-generated] --- # Workflow Title Natural language instructions for the AI agent. Use GitHub context like ${{ github.event.issue.number }}. ``` ### Execution Flow ``` 1. Trigger Event (issue opened, PR created, schedule, etc.) ↓ 2. Activation Job - Validates permissions - Processes mcp-scripts - Sanitizes context ↓ 3. AI Agent Job - Loads MCP servers and tools - Executes AI agent with prompt - Agent makes tool calls - Agent produces output ↓ 4. Safe Outputs Job - Processes agent output - Creates GitHub resources (issues, PRs, etc.) - Applies labels, comments ↓ 5. Completion - Workflow summary generated - Artifacts uploaded ``` ### Key Components | Component | Purpose | Configuration | |-----------|---------|---------------| | **Engine** | AI model to use | `engine: copilot`, `claude`, `codex` | | **Tools** | APIs available to agent | `tools:` section with MCP servers | | **MCP Scripts** | Context passed to agent | `mcp-scripts:` with GitHub expressions | | **Safe-Outputs** | Resources agent can create | `safe-outputs:` with allowed operations | | **Permissions** | GitHub token permissions | `permissions:` block | | **Network** | Allowed network access | `network:` with domain/ecosystem lists | ### Compilation Process ```bash # Compile workflow to GitHub Actions YAML gh aw compile # Result: .github/workflows/.md → .github/workflows/.lock.yml ``` The `.lock.yml` file is the actual GitHub Actions workflow that runs. ## Common Issues and Solutions ### Missing Tool Errors **Symptoms**: - Error: "Tool 'github:read_issue' not found" - Agent cannot access GitHub APIs **Solution**: Add GitHub MCP server configuration: ```yaml tools: github: mode: remote toolsets: [default] ``` ### Permission Errors **Symptoms**: - HTTP 403 (Forbidden) errors - "Resource not accessible" errors **Solution**: First distinguish SAML/token-source denial from missing permissions using the shared credential triage. Grant required read permissions to the agent and configure writes through safe outputs. Keep debugging outputs staged; inspect individual job/token permissions rather than adding write permissions to the agent: ```yaml permissions: contents: read safe-outputs: staged: true create-issue: {} ``` ### Safe-Input Errors **Symptoms**: - "missing tool configuration for mcpscripts-gh" - Environment variable not available **Solution**: Configure mcp-scripts: ```yaml mcp-scripts: issue: script: | return { title: process.env.ISSUE_TITLE, body: process.env.ISSUE_BODY }; env: ISSUE_TITLE: ${{ github.event.issue.title }} ISSUE_BODY: ${{ github.event.issue.body }} ``` ### Safe-Output Errors **Symptoms**: - Agent tries to create resources but fails - "Safe output not enabled" errors **Solution**: Enable safe-outputs: ```yaml safe-outputs: staged: true # Preview safe outputs while debugging create-issue: labels: [ai-generated] ``` ### Cascading Safe-Output Message Failures (Process Safe Outputs step) **Symptoms**: - `Process Safe Outputs` reports multiple failed messages in one run - One failed `update_pull_request` message includes a 403 workflows-permission warning - Other failed messages (for example `add_comment`) include `Bad credentials` **What this means**: - Do not assume all safe-output failures share one root cause. - A 403 workflows-permission error on `update_pull_request` can be expected/non-fatal in some workflows. - A 401-style `Bad credentials` error on other messages is a separate authentication failure that needs its own fix. **Diagnostic steps**: ```bash # Summarize failed safe-output messages and types gh aw audit # Include additional artifacts when diagnosis needs more context gh aw audit --artifacts usage,github-api,mcp,agent # Escalate to full artifact collection for hard-to-classify failures gh aw audit --artifacts all # Inspect full failing job logs to classify each message failure gh run view --job= --log ``` - Triage each failed message by its own HTTP status code and tool/action name. - Check `permissions:` for missing scopes when 403 errors appear. - Compare the "failed message count" against the individual failed message lines to confirm whether there are multiple independent failures. - If several credential failures cluster together in time, investigate token freshness/expiry and token source for the run. ### Network Access Errors **Symptoms**: - Firewall denials - URLs appearing as "(redacted)" **Solution**: Configure network access: ```yaml network: allowed: - defaults - python # For PyPI - node # For npm - "api.example.com" # Custom domains ``` ### Timeout Errors **Symptoms**: - Workflow exceeds time limit - Agent loops or hangs **Solution**: Increase timeout or optimize prompt: ```yaml timeout-minutes: 30 # Increase from default ``` ## Advanced Debugging Techniques ### Polling In-Progress Runs Classify command exit separately from workflow outcome. A nonzero audit exit may mean artifacts are not ready: confirm the same run with `gh run view --json status,headSha,conclusion`, then poll within the approved interval/deadline. Audit/log permission denial blocks further live iteration; report evidence unavailable, not workflow failure. Follow the shared outcome table for dispatch timeouts and SHA mismatches; never redispatch for logs. ### Inspecting MCP Configuration Preflight declarations, startup effects and isolated test bindings using the shared strategy before commands that can start/connect servers. ```bash # Inspect MCP servers for a workflow gh aw mcp inspect # List all workflows with MCP servers gh aw mcp list ``` ### Checking Workflow Status ```bash # Show status of all agentic workflows gh aw status ``` ### Downloading Specific Artifacts ```bash # Download only the agent log artifact GH_REPO=owner/repo gh run download -n agent-stdio.log ``` ### Inspecting Job Logs ```bash # View specific job logs gh run view gh run view --job --log ``` ### Analyzing Firewall Logs ```bash # Parse firewall logs for network issues gh aw logs --parse # Check firewall-enabled runs gh aw logs --firewall ``` ### Diagnostic and Development Compilation ```bash # Strict/staged development compilation; Docker-based checks are optional gh aw compile --dry-run # Recommended when using a reviewed test environment gh aw compile --dry-run --environment gh-aw-debug # Additional source validation; does not run scanners gh aw validate # Run those scanners on the emitted dry-run lock file when possible gh aw compile --dry-run --zizmor --actionlint --poutine ``` Docker unavailability does not block the dry-run gate. Use native `shellcheck` for required run-step linting without Docker, and report unavailable optional checks as unverified. When Docker is available, recommend both commands: `validate` checks source without emitting files, while the scanner-enabled compile command runs zizmor, actionlint and poutine. A successful `validate` does not establish scanner coverage. ## Reference Commands ### Log Analysis Commands | Command | Description | |---------|-------------| | `gh aw logs` | Download logs for all workflows | | `gh aw logs ` | Download logs for specific workflow | | `gh aw logs --json` | Output as JSON | | `gh aw logs --start-date -1d` | Filter by date | | `gh aw logs --engine copilot` | Filter by engine | | `gh aw logs --parse` | Generate Markdown reports | ### Audit Commands | Command | Description | |---------|-------------| | `gh aw audit ` | Audit specific run | | `gh aw audit ` | Audit from GitHub URL | | `gh aw audit --json` | Output as JSON | | `gh aw audit --parse` | Parse logs to Markdown | | `gh aw audit ... --group --json` | Group existing-run findings for recurrence | ### MCP Commands | Command | Description | |---------|-------------| | `gh aw mcp list` | List workflows with MCP servers | | `gh aw mcp inspect ` | Inspect MCP configuration | ### Status Commands | Command | Description | |---------|-------------| | `gh aw status` | Show all workflow status | | `gh aw compile` | Compile all workflows | | `gh aw compile ` | Compile specific workflow | | `gh aw compile --dry-run` | Enforce shared development-testing checks | ### Active Debugging Commands (Permitted Contexts Only) | Command | Description | |---------|-------------| | `gh aw run --ref ` | Only after shared human-validation gates; explicit no-dispatch rules take precedence | | `gh run view --json status,headSha,conclusion` | Same-run monitoring within approved bounds | ## Additional Resources - [Workflow Health Monitoring Runbook](../../aw/runbooks/workflow-health.md) - Step-by-step investigation procedures - [Common Issues Reference](../../../docs/src/content/docs/troubleshooting/common-issues.md) - Frequently encountered issues - [Error Reference](../../../docs/src/content/docs/troubleshooting/errors.md) - Error codes and solutions - [GitHub MCP Server Documentation](../github-mcp-server/SKILL.md) - Tool configuration reference