Rca Methodology

Problem: Payment service was down for 30 minutes.

Why 1: Why did the payment service go down?
-> The DB connection pool was exhausted.

Why 2: Why was the connection pool exhausted?
-> Slow queries held connections for extended periods.

Why 3: Why did slow queries occur?
-> A full table scan was performed without an index.

Why 4: Why was there no index?
-> The index addition was missed in the migration during the new feature deployment.

Why 5: Why was the missing index not detected?
-> Query performance verification was not included in the deployment pipeline.

Root Cause: Absence of query performance verification step in the deployment pipeline

                    +- People -------- Insufficient deployer training
                    |                  Code review skipped
                    |
                    +- Process -------- Deployment checklist not followed
                    |                   Performance testing not conducted
                    |
Payment Svc Down <--+- Technology ---- Missing index
                    |                   Connection pool size insufficient
                    |
                    +- Environment ---- Production data scale not reflected
                    |                   Staging environment mismatch
                    |
                    +- Monitoring ----- Slow query alert not configured
                                        DB metrics dashboard absent

                    Payment Service Outage (Top Event)
                          |
                    +-----| OR |-----+
                    |                |
              DB Failure        Application Failure
                |                    |
          +-----| OR |-----+  +---| AND |---+
          |                |  |             |
    Pool Exhaustion   Disk Full  Slow Queries  High Traffic
          |                        |
    +-----| AND |-----+     +-----| OR |-----+
    |                |     |                |
 Missing Index  Large Data  New Code Deploy  Schema Change

OR gate:  P(A OR B)  = 1 - (1-P(A)) x (1-P(B))
AND gate: P(A AND B) = P(A) x P(B)

Example: P(slow query)=0.3, P(high traffic)=0.2
    P(app failure) = 0.3 x 0.2 = 0.06 (6%)

Investigate changes before and after the incident:

| Change Time | Change Content | Changed By | Impact Scope | Incident Correlation |
|------------|---------------|-----------|-------------|---------------------|
| T-2h | Payment API v2.3 deploy | Dev team | Payment service | HIGH |
| T-1h | Redis config change | Ops team | Cache service | MEDIUM |
| T-30m | CDN cache purge | Marketing | Frontend | LOW |

Correlation criteria:
1. Temporal proximity: Change time <-> Incident time
2. Scope match: Change scope <-> Incident impact scope
3. Rollback effect: Whether rolling back the change resolves the incident