Prometheus Cardinality Troubleshooter

Histogram blowup

Tell: *_bucket metric at the top of seriesCountByMetricName. Multiplier ≈ 14×.

Fix:

1. First, reduce labels on the histogram at the source — every label removed saves 14× series. Trim path, method, or status_code in the instrumentation code (don't labeldrop them at scrape — that merges distinct histograms and corrupts the buckets). For series already in Grafana Cloud you can't change, aggregate them with Adaptive Metrics.
2. Then, reduce bucket count if appropriate (custom buckets vs. defaults).
3. For high-resolution latency tracking, consider native histograms (Prometheus 2.40+) — single sparse series replaces the bucket family.

kube-state-metrics label explosion

Tell: kube_pod_labels or kube_pod_annotations at the top, with label_* or annotation_* labels driving cardinality.

Fix: configure kube-state-metrics with --metric-labels-allowlist and --metric-annotations-allowlist. By default it emits all labels and annotations as series.

# kube-state-metrics flags
--metric-labels-allowlist=pods=[app,team,version]
--metric-annotations-allowlist=pods=[checksum/config]

Path / route blowup from a new endpoint

Tell: http_requests_total (or framework equivalent) grew 10×+ overnight. topk(20, count by (path) (http_requests_total)) shows hundreds of /users/123456-style values.

Fix: the real fix is to template the path in application code (/users/:id) — route the user to prometheus-label-strategy. For series already in Grafana Cloud, Adaptive Metrics can aggregate path away correctly — route to adaptive-metrics.

Do not "normalize" path with a relabel replacement rule — collapsing /users/123, /users/456, … into one /users/:id value at scrape merges distinct series and produces duplicate-sample errors and broken rate(). The merge has to happen at the source (templating) or post-ingest (Adaptive Metrics), never at scrape.

If you must stop a production fire right now and templating isn't deployable yet, the only safe scrape-time action is to drop the entire offending metric (you lose it completely until the code fix lands — a deliberate trade, not a silent corruption):

# Emergency: drop the whole metric until the source is templated
metric_relabel_configs:
  - source_labels: [__name__]
    regex: http_requests_total
    action: drop

Application emitting a debug metric in prod

Tell: A metric you don't recognize in the top 10. Grep the source — often a _details or _per_request debug metric the developer forgot to gate.

Fix: drop entirely at scrape:

metric_relabel_configs:
  - source_labels: [__name__]
    regex: my_app_request_details
    action: drop

Open a ticket against the team to remove it from the code.

App-emitted labels colliding with target labels

Tell: Series count for one job is several × what it should be. Looking at one series, you see both an app-emitted instance=... AND the target instance=... collided into something weird (Prometheus renames the conflicting one to exported_instance).

Fix: the right fix is in the application — stop emitting instance/node/host from code; they belong to the scrape target. Confirm honor_labels is false (the default) so the target labels win.

If you need a scrape-time stopgap, you may remove a label only where it exactly duplicates a target label — that's the one safe labeldrop, because the target label still provides uniqueness. Scope it tightly to the duplicated names and never include pod (or any other label that is the source of uniqueness):

# Stopgap ONLY for app-emitted duplicates of target labels.
# Drops the `exported_*` collisions — NOT pod, which makes K8s series unique.
metric_relabel_configs:
  - regex: exported_(instance|node|host)
    action: labeldrop

Then the target labels from relabel_configs apply cleanly. Prefer fixing the app.

Federation amplifying cardinality

Tell: A federated Prometheus or Mimir global view has way more series than expected. Each source has its own cluster / region label, multiplying.

Fix: this is usually expected — federation by design preserves source labels. If the series count is too high, federate only aggregated recording rules, not raw metrics:

- job_name: federate
  honor_labels: true
  metrics_path: /federate
  params:
    'match[]':
      - '{__name__=~".*:.*"}'  # Recording-rule naming convention only

Remediation Decision Tree

Cardinality fire confirmed
│
├── Need to stop the bleeding NOW (production OOM, ingest 429s)
│   └── Drop the ENTIRE offending metric via metric_relabel_configs (action: drop on __name__)
│       (also applies to Alloy/Agent — same syntax)
│       Do NOT labeldrop a distinguishing label — it breaks the data, see "The One Rule".
│       Then schedule the proper fix.
│
├── It's a Grafana Cloud Active Series bill issue, not a perf issue
│   ├── Cardinality is structural and you can't fix the app
│   │   └── Route to `adaptive-metrics` skill (post-ingest aggregation rules — the safe way)
│   └── You want metric-by-metric DPM breakdown
│       └── Route to `dpm-finder` skill
│
├── It's a fixable application bug (unbounded label, debug metric in prod)
│   ├── Short-term: drop the whole metric at scrape, OR aggregate via Adaptive Metrics
│   └── Long-term: fix in code; route to `prometheus-label-strategy` for design guidance
│
├── It's histogram cardinality
│   ├── Reduce labels on the underlying histogram AT THE SOURCE (14× win per label)
│   ├── Reduce bucket count if appropriate
│   └── Consider native histograms for high-resolution latency
│
└── It's churn (deploy-driven)
    ├── Stop EMITTING `version`/`git_sha`/`instance` from app code (use info-metric for version)
    ├── Keep `pod` — never drop it; if pod-level series are too costly, use Adaptive Metrics
    └── Verify K8s SD relabel rules aren't mapping in `uid` or other ephemeral fields

Emergency Drop Patterns (copy-paste ready)

These are the safe scrape-time emergency actions: dropping an entire unwanted metric. They do not merge distinct series, so they don't corrupt the data.

⚠️ There is intentionally no labeldrop of a distinguishing label and no value-normalizing relabel here. Both merge distinct series and break rate()/DPM (see The One Rule). To reduce cardinality without dropping the whole metric, fix the source or use Adaptive Metrics (route to adaptive-metrics). The only safe labeldrop is removing a label that exactly duplicates a target label (e.g. exported_instance) — see App-emitted labels colliding with target labels.

For Prometheus scrape_configs:

metric_relabel_configs:
  # Drop a specific bad metric entirely
  - source_labels: [__name__]
    regex: bad_metric_name
    action: drop

  # Drop a set of debug/temporary metrics by name prefix
  - source_labels: [__name__]
    regex: debug_.*
    action: drop

For Grafana Alloy (prometheus.relabel component):

prometheus.relabel "drop_bad_metric" {
  forward_to = [prometheus.remote_write.default.receiver]

  rule {
    source_labels = ["__name__"]
    regex = "bad_metric_name"
    action = "drop"
  }
}

Always test in staging first, and prefer fixing the source or using Adaptive Metrics over any scrape-time drop.

When to Hand Off

• "Now design a label strategy so this doesn't happen again" → prometheus-label-strategy
• "We need to keep these metrics but reduce cost" → adaptive-metrics
• "Which metric is the most expensive in DPM terms?" → dpm-finder
• "Write the PromQL to find this" → promql
• "Configure this in Alloy" → alloy
• "Why is my Loki slow?" → loki-label-analyzer (different system, same family of problems)

This skill's lane is diagnosis under pressure. Prevention, design, and post-ingest cost optimization live elsewhere.