Observability

Observability

You are wiring the inside view of a service: when something breaks at 3am, an engineer must be able to answer "what happened, where, and why" from telemetry alone — without adding a console.log and redeploying into the fire. This skill emits a concrete artifact: SDK init code, instrumentation (spans/metrics/structured logs), a Collector config, and alert rules that the instrumentation makes possible. The outside-in half — is it up, who gets paged — is ../monitoring/SKILL.md.

The one rule

Every signal carries the same correlation identity: trace_id, service.name, deployment.environment. A log line you cannot pivot to its trace, or a spiking metric you cannot pivot to an exemplar span, doubles your mean-time-to-resolution — you are back to grepping. Three signals that don't share keys are three disconnected tools; three that do are one queryable system. Set the resource once at SDK init, inject trace_id/span_id into every log, and never emit a metric you can't tie back to a service and environment.

Start from the operator's questions

Before choosing signals, write two to four questions on-call must answer during the likely incident. For example: “Are payment retries recovering?”, “Which dependency and failure class drives exhaustion?”, “Can one payment be charged twice?”, “Which customer-visible operations need intervention now?” Then assign the cheapest signal: metrics say that/how much, traces show where/causal path, logs explain why for this event. If a proposed event or label answers none of the questions, do not emit it.

The three pillars — when each earns its place

Don't emit all three of everything. Each signal answers a different question at a different cost.

The "fourth pillar," continuous profiling (CPU/heap flame graphs over time), is now a first-class OTel signal — reach for it only when traces say "the time is inside this function" and you need to know which line.

Architecture

Instrument once, route anywhere. The app talks OTLP to a Collector; the Collector fans the firehose out to backends.

  ┌─────────────┐   OTLP/gRPC :4317        ┌───────────────────────────┐
  │  app + SDK  │   OTLP/HTTP :4318  ───▶   │       OTel Collector       │
  │ (resource:  │   /v1/traces             │  receivers → processors    │
  │ service.name│   /v1/metrics            │  → exporters (per signal)  │
  │  +env+ver)  │   /v1/logs               │  wired in service.pipelines│
  └─────────────┘                          └───────────────────────────┘
                                                 │        │        │
                                            logs │ traces │ metrics│
                                                 ▼        ▼        ▼
                                              Loki     Tempo   Mimir/Prom   (+ Grafana to view)
                                                  └── or a single vendor: Datadog / Honeycomb ──┘

OTLP is the wire format: gRPC on 4317 (TLS + gzip by default), HTTP on 4318 with per-signal paths /v1/traces, /v1/metrics, /v1/logs. Always export to a Collector, never straight to the vendor. Why: the Collector gives you one place to batch (fewer round-trips), retry (survive a backend blip), redact PII, and swap or add a backend without redeploying the app. App SDKs should be dumb pipes; policy lives in the Collector.

Instrument in the right order: auto first, manual second

Never hand-roll a span for something auto-instrumentation already covers (HTTP servers, DB clients, queues). You will miss edges and waste effort. Turn on zero-code instrumentation, confirm traces flow, then add manual spans only where your business logic lives.

# Node — zero-code, no app changes. SDK is 2.0+; the register hook is compatible.
npm i @opentelemetry/api @opentelemetry/auto-instrumentations-node
OTEL_SERVICE_NAME=checkout-api \
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod,service.version=1.4.2 \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
node --require '@opentelemetry/auto-instrumentations-node/register' app.js

# Python — zero-code via the launcher; it patches known libraries on import.
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
OTEL_SERVICE_NAME=checkout-api \
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod,service.version=1.4.2 \
opentelemetry-instrument python app.py

Then add a manual span only around a meaningful business operation — and give it attributes and a status, or it tells you nothing:

// Bad — a span with no attributes and no status. You learn that "something ran."
const span = tracer.startSpan('work');
await chargeCard(order);
span.end();

// Good — named for the business op, carries the inputs you'd filter by, records outcome.
const { trace, SpanStatusCode } = require('@opentelemetry/api');
const tracer = trace.getTracer('checkout');
await tracer.startActiveSpan('charge_card', async (span) => {
  span.setAttribute('order.id', order.id);          // searchable dimension
  span.setAttribute('payment.provider', 'stripe');
  try {
    await chargeCard(order);
    span.setStatus({ code: SpanStatusCode.OK });
  } catch (err) {
    span.recordException(err);                        // attaches the stack as an event
    span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
    throw err;
  } finally {
    span.end();                                       // a span you never end leaks forever
  }
});

Per-stack init (Node SDK 2.0 manual setup, Go SDK, context propagation across HTTP/queue hops, the GenAI span template) lives in references/instrumentation-recipes.md.

Resource and semantic conventions

Use the standard attribute names; never invent bespoke keys. The whole correlation story and every prebuilt backend dashboard assume http.*, db.*, gen_ai.*, service.*. A homegrown mycompany.endpoint attribute is invisible to every tool that expects http.route.

• Set on the resource, once: service.name (required — unset means telemetry lands as unknown_service), service.version, deployment.environment.
• Kubernetes attributes (k8s.*) reached release candidate (2026-03); DB conventions are on their 2nd RC. Prefer the standard names even while an area is still stabilizing.
• For LLM calls, GenAI conventions exited experimental for client spans (early 2026) — use gen_ai.* so the same span feeds your spend view:

// LLM call as a span: model + token attrs. These attributes feed ../cost-tracking/SKILL.md.
await tracer.startActiveSpan('chat gpt-4o', async (span) => {
  span.setAttribute('gen_ai.system', 'openai');
  span.setAttribute('gen_ai.request.model', 'gpt-4o');
  const res = await openai.chat.completions.create({ /* ... */ });
  span.setAttribute('gen_ai.usage.input_tokens', res.usage.prompt_tokens);
  span.setAttribute('gen_ai.usage.output_tokens', res.usage.completion_tokens);
  span.end();
});

This skill emits the signal (token attributes on a span). Turning those tokens into a dollar figure and a budget is ../cost-tracking/SKILL.md.

Structured, correlated logs

Logs are JSON, carry the trace context, and never carry PII. Free-text print lines cost you twice: you can't query them, and you can't jump from the log to its trace.

# Bad — unstructured, unsearchable, unlinkable to a trace.
print("charged user " + email + " amount " + str(amount))

# Good — structured, leveled, correlated, no PII (id not email).
import logging, json
from opentelemetry import trace

def log_charge(order_id, amount):
    ctx = trace.get_current_span().get_span_context()
    logging.info(json.dumps({
        "event": "charge.succeeded",
        "level": "info",
        "order_id": order_id,            # an opaque id, not the customer's email
        "amount_cents": amount,
        "trace_id": format(ctx.trace_id, "032x"),  # ← the pivot back to the trace
        "span_id": format(ctx.span_id, "016x"),
    }))

Level discipline: error = a human should look, warn = degraded but handled, info = business milestones, debug = off in prod. If everything is error, nothing is.