Debug Production Issue

Debug Production Issue

Estimated time: 30 minutes to 8 hours (depending on issue complexity and observability quality)

This workflow chains four atomic skills into a structured diagnostic process for production issues. It covers the path from initial alert or report through systematic investigation to confirmed root cause and verified resolution. Unlike the full incident-response workflow (which includes postmortem and organizational coordination), this workflow focuses purely on the technical debugging process for a single engineer or small team.

Critical note: If the issue is a P1 outage affecting all users, use the handle-production-incident workflow instead. This workflow is for targeted debugging of specific issues that do not require full incident management coordination.

When to Use

• User is investigating a production bug, error spike, or performance degradation
• User has received an alert or user report about unexpected production behavior
• User needs a systematic approach to narrow down root cause from symptoms
• User wants to move beyond ad-hoc debugging to structured diagnostic process
• Do NOT use when: the issue is a full-scale outage requiring incident management (use handle-production-incident), the issue is in a development environment, or the user is doing planned performance optimization (use performance-profiling directly)
• Do NOT use for single-step tasks that one atomic skill can handle independently

Prerequisites

Before starting this workflow, ensure:

1. Issue is confirmed in production: The bug, error, or degradation is happening in the production environment, not just in staging or development
2. Observability tools are accessible: You have access to log aggregation, monitoring dashboards, and application performance monitoring for the affected service
3. Reproduction information exists: You have at least one of: an alert with timestamps, a user report with steps to reproduce, or a monitoring anomaly with a time range
4. Deployment history is available: You can check what was deployed recently (last 24-72 hours) to the affected service

Steps

Step 1: Triage and Classify the Issue (uses: incident-response)

Assess the issue scope, severity, and blast radius to determine the investigation approach. This step uses incident triage techniques to quickly classify the problem and decide whether to investigate or escalate.

• Input: Alert details, user reports, initial symptoms (error messages, degraded metrics, unexpected behavior), affected service identification
• Output: Issue classification with severity assessment, affected user count estimate, blast radius (which services and endpoints), initial hypothesis for root cause, and investigation plan (which logs and metrics to examine first)
• Key focus: Spend no more than 10 minutes on triage. Classify by user impact: how many users affected, what functionality is broken, is there a workaround. Check deployment history immediately since over 70 percent of production issues are caused by recent changes.

Step 2: Investigate Logs and Traces (uses: logging-patterns)

Dive into structured logs and distributed traces to identify the root cause or narrow the investigation. This step uses log correlation techniques to trace the issue from symptom to source component.

• Input: Issue classification from Step 1 (affected services, time window, initial hypothesis), access to log aggregation platform, request IDs or trace IDs from affected requests
• Output: Investigation findings with identified error patterns, affected code paths, timeline reconstruction (when the issue started, was there a triggering event), and root cause hypothesis (confirmed or narrowed to 2-3 candidates)
• Key focus: Start with the most recent deployment or configuration change. Filter logs by the issue time window and look for the first ERROR-level entry, as the earliest error is usually closest to the root cause. Use request IDs to trace a single failing request through all services. Distinguish symptoms from causes: a spike in 500 errors is a symptom, database connection pool exhaustion causing them is the cause.

Step 3: Profile Performance and Resource Usage (uses: performance-profiling)

If the issue involves latency, timeouts, or resource exhaustion, profile the affected service to identify the bottleneck. This step uses performance profiling techniques to pinpoint whether the issue is CPU-bound, memory-bound, I/O-bound, or caused by external dependency latency.

• Input: Investigation findings from Step 2 (suspected component, affected code paths), profiling tools for the technology stack, baseline performance metrics for comparison
• Output: Performance profile identifying the bottleneck: slow database queries (with query plans), memory leaks (allocation patterns), CPU hotspots (flame graph analysis), or external dependency latency (timeout patterns and retry storms)
• Key focus: Compare current performance against the pre-issue baseline. Quantify the degradation: "Query X takes 4.2 seconds now versus 120ms before the issue started." Look for resource exhaustion patterns: connection pool depletion, thread pool saturation, memory growth over time, or file descriptor limits.

Step 4: Verify Resolution with Monitoring (uses: monitoring-alerting)

After applying the fix (code change, configuration update, rollback, or scaling), use monitoring to verify the resolution. This step confirms that the fix actually resolves the issue and does not introduce new problems.

• Input: Applied fix (the change made to resolve the issue), monitoring dashboards, pre-issue baseline metrics, resolution success criteria defined during investigation
• Output: Verification report confirming metrics have returned to baseline, affected error rates have dropped to normal levels, and no new anomalies appeared as a side effect of the fix
• Key focus: Define specific resolution criteria before applying the fix: "Error rate on endpoint X drops below 0.1 percent for 10 consecutive minutes." Monitor for at least 15-30 minutes after the fix to catch delayed effects. Check adjacent services for unexpected impact from the fix.

Decision Points

• At Step 1: If the triage reveals the issue is a security breach or data loss, stop this workflow and escalate to the security-audit-codebase or handle-production-incident workflow. Security incidents require additional coordination beyond technical debugging.

• At Step 2: If logs are insufficient to identify the root cause (poor logging coverage, logs rotated, or no structured logging), skip to Step 3 (performance profiling) to gather data from a different angle. Add improved logging as a follow-up task.

• At Step 3: If the issue is not performance-related (functional bug, incorrect business logic, data corruption), skip this step and proceed directly to fix implementation and Step 4 verification.

• After Step 4: If the fix does not resolve the issue (metrics do not return to baseline), the root cause hypothesis was incorrect. Return to Step 2 with new information and investigate alternative hypotheses.

Failure Handling

• Step 1 cannot determine severity: If the impact is unclear (intermittent errors, unclear user count), assume higher severity and investigate urgently. Downgrade after Step 2 provides more data.

• Step 2 logs show no errors: If logs appear clean despite the reported issue, check for silent failures: swallowed exceptions, incorrect error handling that returns success, or client-side errors not reaching the server. Also verify you are looking at the correct service and time window.

• Step 3 profiling impacts production performance: If attaching a profiler causes additional latency, use sampling-based profiling (low overhead) rather than instrumentation-based profiling. Alternatively, reproduce the issue in a staging environment with production-like data and profile there.

• Step 4 fix introduces new issues: Roll back the fix immediately. The original issue is known and bounded; a new issue from the fix is unknown and potentially worse. Return to Step 2 to find a different fix approach.

• Issue is intermittent and cannot be reproduced: Add targeted logging and monitoring for the specific failure condition identified in Step 2. Set alerts to trigger immediately on recurrence. Prepare the investigation context so the next occurrence can be debugged quickly.