Monitoring Engineer

Types:
  Counter:   Monotonically increasing (requests_total, errors_total)
  Gauge:     Can go up or down (temperature, queue_depth, active_connections)
  Histogram: Distribution of values (request_duration_seconds)
  Summary:   Pre-calculated quantiles (less flexible than histograms)

Naming conventions (Prometheus):
  <namespace>_<name>_<unit>
  http_requests_total          (counter)
  http_request_duration_seconds (histogram)
  process_memory_bytes          (gauge)
  node_cpu_seconds_total        (counter)

Structured logging format (JSON):
{
  "timestamp": "2024-01-15T10:30:45.123Z",
  "level": "error",
  "message": "Failed to process order",
  "service": "order-processor",
  "trace_id": "abc123def456",
  "span_id": "789ghi012",
  "order_id": "ORD-12345",
  "error": "connection timeout",
  "duration_ms": 5032,
  # ... (condensed) ...
  INFO:    Normal operations (request received, job completed)
  WARN:    Unexpected but recoverable (retry, fallback used)
  ERROR:   Operation failed (needs investigation)
  FATAL:   Service cannot continue (process will exit)

Trace Structure:
  Trace (unique trace_id)
    └── Span A: API Gateway (parent)
        ├── Span B: Auth Service (child of A)
        ├── Span C: Order Service (child of A)
        │   ├── Span D: Database Query (child of C)
        │   └── Span E: Cache Lookup (child of C)
        └── Span F: Notification Service (child of A)

Each span contains:
  - trace_id, span_id, parent_span_id
  - operation name
  - start time, duration
  - status (OK, ERROR)
  - attributes (http.method, http.status_code, db.statement)
  - events (exceptions, log messages within the span)

SLI (Service Level Indicator):
  A metric that measures a specific aspect of the service.
  Example: "Proportion of successful HTTP requests" = successes / total

SLO (Service Level Objective):
  A target value for an SLI over a time window.
  Example: "99.9% of requests succeed over a 30-day rolling window"

SLA (Service Level Agreement):
  A contract with consequences for missing the SLO.
  Example: "If availability drops below 99.9%, customer gets 10% credit"

Error Budget:
  100% - SLO = Error Budget
  99.9% SLO = 0.1% error budget = ~43 minutes downtime per 30 days

HTTP API:
  Availability: successful_requests / total_requests
  Latency: requests_below_threshold / total_requests (e.g., P99 < 500ms)
  Error rate: 5xx_responses / total_responses

Data Pipeline:
  Freshness: time_since_last_successful_run < threshold
  Correctness: valid_records / total_records
  Throughput: records_processed_per_second >= target

Storage System:
  Durability: objects_intact / total_objects
  Availability: successful_reads / total_reads
  Latency: reads_below_threshold / total_reads

Target   | 30-day budget | Annual budget
---------|---------------|---------------
99%      | 7h 12m        | 3d 15h 36m
99.5%    | 3h 36m        | 1d 19h 48m
99.9%    | 43m 12s       | 8h 45m 36s
99.95%   | 21m 36s       | 4h 22m 48s
99.99%   | 4m 19s        | 52m 33s
99.999%  | 26s           | 5m 15s

P1 - Critical (page immediately):
  - Service is down for all users
  - Data loss or corruption occurring
  - Security breach detected
  - SLO burn rate exceeds 14.4x (2% budget consumed in 1 hour)
  Response: Acknowledge in 5 min, mitigate in 30 min

P2 - High (page during business hours):
  - Significant degradation for subset of users
  - SLO burn rate exceeds 6x (5% budget consumed in 6 hours)
  - Capacity approaching limits
  # ... (condensed) ...
  - Performance trends to watch
  - Upcoming certificate expiration
  - Resource utilization trends
  Response: Review weekly

# Prometheus alerting rules for SLO burn rate
groups:
  - name: slo-alerts
    rules:
      # Page: 2% of 30-day budget consumed in 1 hour
      - alert: HighErrorBudgetBurn_Page
        expr: |
          (
            sum(rate(http_requests_total{code=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          # ... (condensed) ...
        labels:
          severity: warning
        annotations:
          summary: "Elevated error budget burn rate (ticket)"

AVOID:
  x Alerts without runbooks (what should I do when this fires?)
  x Alerts that fire and auto-resolve repeatedly (flapping)
  x Alerting on causes instead of symptoms (CPU high vs latency high)
  x Static thresholds without context (CPU > 80% is not always bad)
  x Duplicate alerts for the same problem
  x Alerts that require no human action

PREFER:
  + Alert on user-facing symptoms (error rate, latency)
  + Multi-window burn rate alerts
  + Every alert has a linked runbook
  + Alerts have clear ownership (team, on-call rotation)
  + Regular alert review (prune noisy alerts quarterly)

Level 1: Executive / Service Overview
  - Overall SLO status (green/yellow/red)
  - Error budget remaining
  - Deployment timeline
  - Top-line business metrics

Level 2: Service Dashboard
  - Request rate (QPS)
  - Error rate (4xx, 5xx breakdown)
  - Latency (P50, P95, P99)
  - Saturation (CPU, memory, connections)
  # ... (condensed) ...
  - Cache hit/miss ratio
  - Queue depth and processing rate
  - Individual endpoint breakdown
  - Resource utilization per pod/instance

USE Method (for infrastructure resources):
  Utilization: % of resource being used (CPU usage, disk usage)
  Saturation: How overloaded is it (queue depth, swap usage)
  Errors: Error count (disk errors, network errors)

RED Method (for services/APIs):
  Rate: Requests per second
  Errors: Error rate (5xx/total)
  Duration: Latency distribution (P50, P95, P99)

Apply RED to every service, USE to every resource.

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s

rule_files:
  - "rules/*.yml"

alerting:
  alertmanagers:
    # ... (condensed) ...
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

groups:
  - name: service-slis
    interval: 30s
    rules:
      # Request rate
      - record: service:http_requests:rate5m
        expr: sum by (service) (rate(http_requests_total[5m]))

      # Error rate
      - record: service:http_errors:ratio5m
        expr: |
          # ... (condensed) ...

      # Availability (1 - error rate)
      - record: service:availability:ratio5m
        expr: 1 - service:http_errors:ratio5m

# Request rate per service
sum by (service) (rate(http_requests_total[5m]))

# Error percentage
100 * sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# P95 latency
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

# Top 5 endpoints by error rate
topk(5, sum by (path) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (path) (rate(http_requests_total[5m])))

# Memory usage percentage
100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

# CPU saturation (load average > CPU count)
node_load1 > on(instance) count by (instance) (node_cpu_seconds_total{mode="idle"})

// tracing.js - Initialize before any other imports
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { OTLPMetricExporter } = require('@opentelemetry/exporter-metrics-otlp-grpc');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { Resource } = require('@opentelemetry/resources');
const { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } = require('@opentelemetry/semantic-conventions');

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: 'api-server',
    # ... (condensed) ...
  ],
});

sdk.start();

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    # ... (condensed) ...
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [loki]

Rotation:
  - Weekly rotations (Mon 9am to Mon 9am)
  - Primary + Secondary on-call
  - Handoff meeting at rotation boundary
  - Maximum 1 week on-call per 4 weeks

Escalation Policy:
  1. Alert fires -> Primary on-call notified (PagerDuty/Opsgenie)
  2. No acknowledgment in 5 min -> Secondary notified
  3. No acknowledgment in 10 min -> Engineering manager notified
  4. P1 not mitigated in 30 min -> Incident commander engaged

Compensation:
  - On-call pay or comp time
  - Page during sleep = extra compensation
  - If on-call burden > 2 pages/shift, address root causes

## Alert: [Alert Name]

### What This Alert Means
[Brief explanation of what triggered and why it matters]

### Impact
[What users experience when this fires]

### Immediate Actions
1. Check [dashboard link] for current state
2. Run `kubectl get pods -n production` to check pod health
# ... (condensed) ...

### Escalation
- If not resolved in 30 min, page [team-lead]
- If data loss suspected, immediately page [engineering-director]

1. DETECT:    Alert fires or user reports issue
2. TRIAGE:    Assess severity, assign incident commander
3. MITIGATE:  Stop the bleeding (rollback, scale up, enable circuit breaker)
4. RESOLVE:   Root cause fix deployed and validated
5. FOLLOW-UP: Blameless post-mortem, action items tracked to completion

SEV1 - Critical:
  Complete outage, data loss, security breach.
  All hands. War room. Status page updated.
  Communicate every 15 minutes.

SEV2 - Major:
  Significant degradation for many users.
  Dedicated incident response. Status page updated.
  Communicate every 30 minutes.

SEV3 - Minor:
  # ... (condensed) ...

SEV4 - Low:
  Cosmetic or non-user-facing issue.
  Tracked as regular bug.

## Incident Post-Mortem: [Title]

**Date:** YYYY-MM-DD
**Duration:** X hours Y minutes
**Severity:** SEV-X
**Author:** [name]
**Reviewers:** [names]

### Summary
[2-3 sentences: what happened, how many users affected, how long]

# ... (condensed) ...
| Add circuit breaker for Z | @team | 2024-02-15 | TODO |

### Lessons Learned
[Key takeaways for the organization]

Metrics:
  [ ] RED metrics for every service (rate, errors, duration)
  [ ] USE metrics for all infrastructure (utilization, saturation, errors)
  [ ] Business metrics tracked (signups, orders, revenue)
  [ ] Recording rules for frequently-used queries
  [ ] Retention policy defined (15 days hot, 13 months cold)

Logs:
  [ ] Structured JSON logging everywhere
  [ ] Trace ID correlation in every log line
  [ ] Log levels used correctly (no ERROR for expected conditions)
  # ... (condensed) ...
  [ ] Escalation policy configured
  [ ] Runbooks up to date
  [ ] Post-mortem process established
  [ ] On-call handoff meetings scheduled