Load Balancer

Operates at: TCP/UDP level
Sees: Source IP, destination IP, ports
Cannot see: HTTP headers, URLs, cookies, request body

How it works:
  Client -> LB (TCP connection) -> Backend (new TCP connection)
  Routing decision based on: IP + Port only

# ... (condensed) ...
  - Non-HTTP protocols (SMTP, custom TCP)
  - Maximum performance requirements
  - SSL passthrough (client-to-backend encryption)

Operates at: HTTP/HTTPS level
Sees: Full HTTP request (headers, URL, cookies, body)

How it works:
  Client -> LB (HTTP request parsed) -> Backend (new request forwarded)
  Routing decision based on: URL path, host header, cookies, headers, etc.

Pros:
  # ... (condensed) ...
  - Microservices with path-based routing
  - SSL termination
  - A/B testing, canary deployments

Is it HTTP/HTTPS traffic?
  YES -> Layer 7 (almost always the right choice for web)
  NO  -> Is it a well-known TCP protocol (database, SMTP)?
    YES -> Layer 4
    NO  -> Is it a custom protocol?
      YES -> Layer 4
      NO  -> Layer 4

# ... (condensed) ...

Do you need maximum throughput with minimal latency?
  YES -> Layer 4

Distributes requests sequentially across backends.

Backend A -> Backend B -> Backend C -> Backend A -> ...

Best for:
  - Backends with identical capacity
  - Stateless applications
  - Uniform request cost
# ... (condensed) ...
      server 10.0.1.11:8080;
      server 10.0.1.12:8080;
  }

Like round robin, but backends receive traffic proportional to their weight.

Backend A (weight=5) gets 5x more traffic than Backend C (weight=1)

Best for:
  - Backends with different capacities (different hardware)
  - Gradual rollout (canary with low weight)
  - Migrating between different instance types
# ... (condensed) ...
      server 10.0.1.11:8080 weight=3;   # 30% traffic
      server 10.0.1.12:8080 weight=2;   # 20% traffic
  }

Sends new requests to the backend with fewest active connections.

Best for:
  - Requests with varying processing times
  - Long-lived connections (WebSocket, gRPC streams)
  - Backends with uneven load from other sources

Example (Nginx):
  # ... (condensed) ...
      server 10.0.1.11:8080;
      server 10.0.1.12:8080;
  }

Combines least connections with weights. Considers both active connections
and backend capacity.

Score = active_connections / weight
Route to backend with lowest score.

Example (HAProxy):
  backend app_servers
      balance leastconn
      server web1 10.0.1.10:8080 weight 5
      server web2 10.0.1.11:8080 weight 3

Routes requests from the same client IP to the same backend.
Uses a hash of the client IP to select backend.

Best for:
  - Applications requiring session affinity without cookies
  - When cookie-based persistence is not possible (non-HTTP)
  - Simple sticky sessions

# ... (condensed) ...
      server 10.0.1.11:8080;
      server 10.0.1.12:8080;
  }

Like IP hash, but adding/removing backends only affects a small fraction
of the key space (minimal disruption).

Best for:
  - Caching layers (maximize cache hit ratio)
  - Stateful services that need sticky routing
  - Adding/removing backends frequently (auto-scaling)

# ... (condensed) ...
      server 10.0.1.11:8080;
      server 10.0.1.12:8080;
  }

Active health checks:
  LB periodically sends probe requests to backends.
  If probe fails N times, backend is marked unhealthy.
  If probe succeeds M times, backend is marked healthy again.

Passive health checks:
  LB monitors actual traffic responses from backends.
  If a backend returns too many errors, it is marked unhealthy.
  No extra probe traffic, but slower to detect failure.

Best practice: Use BOTH active and passive health checks.

# Nginx (active health checks - requires Nginx Plus or OpenResty)
upstream backend {
    zone backend 64k;
    server 10.0.1.10:8080;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
}

# ... (condensed) ...
    server 10.0.1.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=30s;
}

# HAProxy health checks
backend app_servers
    option httpchk GET /health
    http-check expect status 200

    server web1 10.0.1.10:8080 check inter 5s fall 3 rise 2
    server web2 10.0.1.11:8080 check inter 5s fall 3 rise 2
    server web3 10.0.1.12:8080 check inter 5s fall 3 rise 2

# inter: check interval
# fall:  consecutive failures to mark unhealthy
# rise:  consecutive successes to mark healthy

LB sets a cookie on the first response, subsequent requests with that cookie
go to the same backend.

Pros: Precise, survives IP changes (mobile)
Cons: Requires cookie support (browsers), cookie overhead

# HAProxy cookie persistence
backend app_servers
    balance roundrobin
    cookie SERVERID insert indirect nocache httponly secure
    server web1 10.0.1.10:8080 cookie web1
    server web2 10.0.1.11:8080 cookie web2
    server web3 10.0.1.12:8080 cookie web3

# Nginx Plus sticky cookie
upstream backend {
    sticky cookie srv_id expires=1h domain=.example.com httponly secure path=/;
    server 10.0.1.10:8080;
    server 10.0.1.11:8080;
}

Client --[HTTPS]--> Load Balancer --[HTTP]--> Backend

Benefits:
  - Centralized certificate management
  - Reduced CPU on backends
  - Simplified backend configuration
  - Load balancer can inspect and route based on HTTP content

# ... (condensed) ...

  backend app_servers
      server web1 10.0.1.10:8080 check

Client --[HTTPS]--> Load Balancer --[HTTPS]--> Backend
(LB does NOT decrypt; routes at TCP level)

Benefits:
  - End-to-end encryption (LB cannot see content)
  - Compliance requirements (data never decrypted in transit)
  - Backend controls its own certificates

# ... (condensed) ...
  backend app_servers_tcp
      mode tcp
      server web1 10.0.1.10:443 check

Client --[HTTPS]--> Load Balancer --[HTTPS]--> Backend
(LB decrypts, inspects, re-encrypts to backend)

Benefits:
  - Content-based routing AND encryption to backend
  - Defense in depth

Drawback:
  - Double encryption overhead
  - More complex certificate management

Global Load Balancing routes users to the nearest regional cluster.

                       ┌──────────────┐
                       │   DNS-based  │
          User ------->│  Global LB   │
                       │  (Route 53,  │
                       │  Cloudflare) │
                       └──────┬───────┘
                              # ... (condensed) ...
  Latency:   Route based on measured latency to each region
  Weighted:  Route percentage of traffic to each region
  Failover:  Route to secondary if primary health check fails

AWS:
  - Global Accelerator: Anycast IPs, TCP/UDP, static IPs
  - CloudFront: HTTP/S CDN with origin failover
  - Route 53: DNS-based (geolocation, latency, weighted, failover)

GCP:
  - Global HTTP(S) Load Balancer: Single anycast IP, global
  - Global TCP/SSL Proxy: L4, anycast IP
  # ... (condensed) ...
Cloudflare:
  - Load Balancing: Global, DNS or proxy-based
  - Anycast network: Automatic geographic routing

1. Load increases -> Auto-scaler adds new instances
2. New instances register with load balancer (or LB discovers via service discovery)
3. Health check passes -> LB starts sending traffic
4. Slow start period -> Gradually increase traffic to new instance

5. Load decreases -> Auto-scaler wants to remove instances
6. LB stops sending NEW requests to instance being removed
7. Existing connections drain (connection draining period)
# ... (condensed) ...
  - Health check grace period: Time for new instances to start before checking
  - Slow start: Gradually ramp up traffic to new instances
  - Deregistration delay: Time to drain connections before removal

When removing a backend from the pool:
  1. Stop sending NEW connections to the backend
  2. Allow EXISTING connections to complete
  3. Wait up to drain_timeout seconds
  4. Force-close remaining connections after timeout

Configuration:
  AWS ALB: Deregistration delay (default 300s, recommend 30-120s)
  # ... (condensed) ...
  echo "set server app_servers/web1 state drain" | socat stdio [system-path]
  # Wait for connections to complete, then:
  echo "set server app_servers/web1 state maint" | socat stdio [system-path]

Gradually increase traffic to new backends over a warm-up period.
Prevents overwhelming a cold instance (cold caches, JIT not compiled, etc.)

AWS ALB: Slow start duration (30-900 seconds)
  - New target starts at 0 traffic
  - Linearly increases to full share over the duration

HAProxy:
  # ... (condensed) ...
  upstream backend {
      server 10.0.1.10:8080 slow_start=30s;
  }

global
    log stdout format raw local0
    maxconn 50000
    stats socket [system-path] mode 660 level admin

defaults
    mode http
    log global
    # ... (condensed) ...
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST

Traffic Metrics:
  - Request rate (requests/second)
  - Active connections (current)
  - New connections (per second)
  - Bandwidth (bytes in/out)

Health Metrics:
  - Healthy backend count
  # ... (condensed) ...
  - Connection errors
  - Timeout rate
  - Rejected connections (capacity)

CRITICAL:
  - All backends unhealthy (zero healthy targets)
  - Error rate > 10% for 5 minutes
  - Response time P99 > 10 seconds for 5 minutes

WARNING:
  - Healthy backend count < minimum threshold
  - Error rate > 1% for 10 minutes
  # ... (condensed) ...
  - Backend added/removed from pool
  - Traffic spike (> 2x normal)
  - Connection draining started

Core Configuration:
  [ ] Appropriate algorithm selected for workload
  [ ] Health checks configured (active and passive)
  [ ] Health check endpoint returns meaningful status
  [ ] Connection timeouts set appropriately
  [ ] Retries and redispatch configured
  [ ] Connection limits set to prevent overload

# ... (condensed) ...
  [ ] Access logs with request timing information
  [ ] SSL certificate expiration monitoring
  [ ] Capacity planning alerts (connection limits)