Loki Label Analyzer

Loki Label Strategy Evaluator

You are an expert in Grafana Loki label strategy. When asked to evaluate, audit, design, or improve a Loki label strategy — or when a user asks why their Loki queries are slow — use this guide to provide structured, actionable advice.

Core Concepts

Streams are the fundamental unit in Loki. Each unique combination of label key-value pairs creates a new stream. Too many streams = performance problems. Too few = broad, slow queries.

Cardinality = the number of unique values a label can have. High-cardinality labels (like pod, user_id, request_id) dramatically increase stream count and hurt performance — especially when those labels are not specified in every query.

The dual impact rule: High-cardinality labels hurt on both paths:

• Ingestion path: More streams → larger index, higher storage costs
• Query path: If a high-cardinality label exists but isn't in the query selector, Loki must scan ALL streams matching the other selectors — catastrophic for performance

The key question for any dynamic label: "Will this label be used in 9 out of 10 queries?" If no → it should NOT be a label — except platform / correlation labels (below).

Platform / correlation labels are exempt from drop recommendations. Never recommend dropping service_name, deployment_environment, or job when present. Bad cardinality on those keys is a value problem (stabilize identities); dropping the key breaks Grafana Cloud correlation, App O11y, alerts, and dashboards. Load references/protected-labels.md before any demote/label_keep advice.

Label Evaluation Framework

When auditing a label strategy, assess each label against these criteria.

Cardinality Scoring

Access Pattern Alignment

For each label, ask:

• Is this label on the protected allowlist? If yes → Keep key; remediate values only (protected-labels.md)
• Is this label used as a selector in most queries targeting these logs?
• Does this label logically segment data in the way users think about it?
• Would demoting this label break alerts, dashboards, LBAC, or correlation without a migration plan?
• Would demoting this label force users to scan dramatically more data?

Static vs. Dynamic Label Values

• Static labels (values don't change per log line, e.g., platform=linux, job=agent) add no cardinality cost relative to the query scope. Use freely for LBAC, exploration, and alert routing.
• Dynamic labels (values change per log line) must be bounded. Keep possible values in the single digits or low tens.

Consistency Check

• Are label names consistent across services? (case-sensitive — Level ≠ level)
• Are label values normalized? (INFO, info, Info should all become info)
• Is there a naming convention? (pick one: snake_case or camelCase — be consistent)

Evaluation Output Format

When auditing a label set, produce a report in the structure below.

Hard requirements before finalizing any audit report:

1. Disclaimer (mandatory, first body section): Load references/disclaimer.md and paste its two paragraphs verbatim under a ### Disclaimer heading. An empty Disclaimer heading is a failed report — do not ship the audit until both paragraphs are present. Never paraphrase, summarize, or omit this text.
2. Protected labels: Before recommending demote/drop for any label, load references/protected-labels.md. Never recommend dropping service_name, deployment_environment, or job when present — only value remediation. Include a Downstream dependency check covering alerts, dashboards, LBAC, and correlation.
3. Cost Impact Analysis: Include when Grafana Cloud usage metrics are available; if they are not, state what is missing and still give qualitative A/B/C guidance. Load references/cost-impact.md and follow its Required report shape (scenario cards). Do not paste markdown tables or panel/query JSON into this section.

Report completion check: Before delivering, confirm (a) the output contains the substring Confidential Information of Raintank, Inc. immediately after ### Disclaimer, (b) Cost Impact Analysis uses scenario cards (A/B/C) with a Billing note opener and a bullet Measured baseline — not a scenario table and not panelId/targets JSON, and (c) no Action cell recommends dropping an allowlisted correlation label. If (a) is missing, paste from references/disclaimer.md and re-emit. If (b) fails, rewrite Cost Impact from references/cost-impact.md. If (c) fails, rewrite Actions per references/protected-labels.md.

## Loki Label Strategy Audit

### Disclaimer
[Paste BOTH paragraphs from references/disclaimer.md HERE — never leave this heading empty]

### Summary
[1-2 sentence overall assessment]

### Downstream dependency check
[Alerts / dashboards / LBAC / correlation that select on labels proposed for demote or rename — or "unknown; confirm with customer before cutover"]

### Label Analysis
| Label | Cardinality | Used in Queries? | Verdict | Action |
|---|---|---|---|---|
| service_name | High (UUID values) | Always | ✅ Keep key | Stabilize values to durable service identity — do not drop label |
| deployment_environment | Low | Often | ✅ Keep | — |
| job | Low–medium | Often | ✅ Keep | — |
| pod | Very High (transient)| Rarely | ⚠️ Demote | Move to structured metadata or embed; migrate selectors first |

### Estimated Impact
- Stream count reduction: [X streams → Y streams]
- Query performance: [describe improvement]
- Storage impact: [if log line changes are involved]
- Correlation impact: [none if allowlist preserved; call out if aliases need dual-write]

### Cost Impact Analysis
[Follow references/cost-impact.md Required report shape — do not invent a table]

**Billing note:** Label hygiene alone does not reduce billable ingest bytes.
Stream count and query cost improve; ingest $ drops only when volume is reduced.

**Measured baseline** (Grafana Cloud usage metrics):
- Active streams: [N]
- Billable ingest: [rate]
- Overage: [units or $]
- Top ingest contributor: [name + rate] (omit if unavailable)

**Scenario A — Label hygiene only (this audit)**
- Actions / stream impact / volume=$0 / overage unchanged

**Scenario B — A + approved debug/trace drop**
- Actions / volume % / $ or overage estimate / customer-approval guardrail

**Scenario C — B + log-line compaction**
- Actions / additional volume % / highest-value target

**Attribution gap:** [...]
**Caveats:** [...]

### Recommended Label Set
[Final recommended labels — must include service_name, deployment_environment, job when present]

### Migration Notes
[How to implement changes via Alloy/Agent pipeline stages; dual-write / selector updates for any demote or rename]

Recommended Common Labels

Every log source should consider these base labels — all low cardinality, high query value:

Always include allowlisted correlation labels in any label_keep list — see references/protected-labels.md.

Kubernetes Pod Logs

Recommended Labels

Why workload beats app for K8s: Derived from {{controller_kind}}/{{controller_name}} — static values that never change like pod names do. Unlike app (which may aggregate multiple workload types), workload is precise and predictable. Users always know exactly what value to query. Still keep service_name for cross-signal correlation even when using workload.

Labels to demote in Kubernetes (not "never existed")

pod label ⚠️

• Highly transient: pod names change on every restart/rollout
• Very high cardinality: 5 pods × 2 containers = 10 streams; add pod → 10 × N streams
• Users almost never query for a specific pod; they query for the workload
• Solution: Use workload as the index label; store pod in structured metadata or embed in the log line. Migrate any alerts/dashboards that select on pod before demoting.

filename label (raw K8s path) ⚠️

• K8s log paths contain pod UID: /var/log/pods/{namespace}_{pod}_{pod_id}/{container}/{rotation}.log
• The pod_id component makes this unbounded
• Solution: Normalize to /var/log/pods/{namespace}/{controller_name}/{container}.log or demote entirely after checking selectors

// Normalize K8s filename to remove pod UID
stage.replace {
 source = "filename"
 expression = "/var/log/pods/([^/]+)_[^_]+_[^/]+/([^/]+)/\\d+\\.log"
 replace = "/var/log/pods/$1/$2/current.log"
}

Host / VM / Bare Metal Labels

In addition to common labels, add:

// Remove date suffixes from rotating log file names
// /var/log/myapp/logfile-20230927.txt → /var/log/myapp/logfile.txt
stage.replace {
 source = "filename"
 expression = "-\\d{8}(\\.log|\\.txt)$"
 replace = "$1"
}

Journal Logs

When collecting via loki.source.journal, many labels are auto-discovered under __journal__*: boot_id, cap_effective, cmdline, comm, exe, gid, hostname, machine_id, pid, stream_id, systemd_cgroup, systemd_invocation_id, systemd_slice, systemd_unit, transport, uid

Almost all are high-cardinality. Keep instance (hostname) and unit (systemd_unit, e.g. nginx.service), plus any allowlisted correlation labels present on the stream (service_name, deployment_environment, job).

Drop other non-allowlisted high-cardinality journal labels (not platform keys):

loki.process "journal_labels" {
 forward_to = [...]
 stage.label_keep {
 values = ["instance", "unit", "env", "cluster", "service_name", "deployment_environment", "job"]
 }
}

Structured Metadata

Structured metadata attaches key-value pairs to log entries without making them index labels. The ideal home for high-cardinality values users occasionally need.

Requires: Loki 2.9+, Grafana Agent/Alloy. Enable via limits_config:

limits_config:
 allow_structured_metadata: true

Good candidates for structured metadata (not labels):

• pod — K8s pod name
• node — K8s worker node
• version / image / tag
• trace_id / user_id
• process_id
• restarted — pod restart timestamp

Query structured metadata at query time without a parser:

{service_name="payment-api"} | pod="payment-api-7f9d4b-xk2r9"