Spine Design

## Components

| Component        | Responsibility                          | Tech              | Scales by              |
|------------------|-----------------------------------------|-------------------|------------------------|
| API Server       | HTTP request handling, auth, validation | FastAPI / Express | Horizontal (stateless) |
| Background Jobs  | Async processing, retries, scheduling   | BullMQ / Sidekiq  | Horizontal             |
| Primary DB       | Persistent application state            | PostgreSQL (RDS)  | Vertical + read replicas |
| Cache            | Session data, hot reads, rate limits    | Redis (Elasticache) | Vertical / cluster   |
| Object Storage   | Files, images, exports                  | S3 / GCS          | Managed                |

## Data Flow: [User Action]

Happy path:
  Client → POST /resource → Auth middleware
         → Validate input → Write to DB → Enqueue background job
         → Return 201 with created resource

Failure paths:
  DB write fails    → 500, job not enqueued, client retries with idempotency key
  Job fails         → Retry with backoff (3x), dead letter queue after max attempts
  Downstream timeout → Circuit breaker opens, return 503 with Retry-After

## Failure Modes

| Component      | Failure mode              | Detection                    | User impact         | Mitigation                            |
|----------------|--------------------------|------------------------------|---------------------|---------------------------------------|
| API Server     | Process crash             | Health check fails           | 502 until restart   | Multiple instances + auto-restart     |
| PostgreSQL     | Primary goes down         | Connection error              | Writes fail         | Automatic failover to replica (RDS)   |
| Redis          | Cache miss / down         | Timeout or connection error   | Slower reads        | Fallback to DB, cache miss is fine    |
| External API   | Timeout / 5xx             | Timeout > threshold           | Feature degraded    | Circuit breaker, cached fallback      |
| Background Jobs | Worker down              | Job queue depth grows         | Async features delayed | Auto-restart, queue depth alert    |

## Scaling Roadmap

**Now (0–10k users):**
- Single API server instance
- Single Postgres primary, no replicas
- Redis single node
- Vertical scaling is fine; operational simplicity beats premature distribution

**10x (10k–100k users):**
- Add read replica for analytics and heavy read queries
- Move to multiple API server instances behind a load balancer
- Add CDN in front of static assets and cacheable API responses
- Background job workers scale horizontally — add more workers, not more queues

**100x (100k–1M+ users):**
- Evaluate connection pooling (PgBouncer) before horizontal sharding
- Identify which tables are write-hot; consider partitioning or archive strategy
- At this point microservices may make sense for one or two clearly bounded domains — not by default, only where independent scaling or deployment is demonstrably needed

## Decision Log

| Decision                     | Chosen           | Rejected             | Reason                                                            |
|------------------------------|------------------|----------------------|-------------------------------------------------------------------|
| Service topology             | Monolith         | Microservices        | Team is 4 engineers. No independent deployment requirement yet.   |
| Database                     | PostgreSQL       | MongoDB              | Data is relational. ACID guarantees matter for financial records. |
| Queue                        | BullMQ (Redis)   | Kafka                | <10k jobs/day. Kafka operational overhead not justified.          |
| API style                    | REST             | GraphQL              | One client (web app) with predictable access patterns.            |
| Search                       | Postgres FTS     | Elasticsearch        | Search is secondary feature. Can revisit at 50k records.         |

## System Design: [Name]

### Decision
[One paragraph: what you're building, what topology, what stack, why]

### What was ruled out
[Bullets: each rejected option + one-line reason]

### Components
[Table from Step 3]

### Data Flow: [Key action 1]
[Arrow diagram]

### Data Flow: [Key action 2]
[Arrow diagram]

### Failure Modes
[Table from Step 5]

### Scaling Roadmap
[Three horizons from Step 6]

### Decision Log
[Table from Step 7]