Health Alert Authoring

Author Netdata Health Alerts

Use this skill when changing or reviewing a health-alert definition. Alerts are production policy: a syntactically valid expression can still page incorrectly, manufacture a recovery, duplicate an incident owner, or make a disappearing entity look healthy.

The normal goal when translating an alert from another system is Netdata-adapted operational equivalence, not execution-engine emulation. Preserve the operator-visible incident as closely as Netdata can express it through NIDL instances, alert beats, database lookups, native gap behavior, and chart obsoletion. Treat differences from the source engine as explicit design facts to document and test, not as defects by default.

Read The Right Sources

Read the target alert, its collector/profile signal contract, and the existing incident owner. Search for duplicate names, contexts and equivalent generic alerts. Select the affected references:

For review, use the contract and completion sections as questions against the complete assigned change and existing design/test evidence. Read an active SOW when available; do not create one or demand a new authoring note merely to review. Missing evidence matters when it leaves an affected contract unproven. Implementation follows the root SOW gate.

Never query or reconfigure a live Agent merely to validate an alert unless the user has explicitly authorized that access.

Establish The Alert Contract Before Editing

Write down the following in the active SOW before changing a non-trivial alert:

• Incident and owner: What real incident does this alert represent? Which one source owns it? Do not create a product-named duplicate of an existing collection-failure, component-failure, or generic host alert just to change routing or severity.
• Signal contract: Identify every source value and its meanings, including zero, non-zero, tri-state values, absent dimensions, temporary collection failure, and known entity disappearance.
• Source intent and adaptation: Separate the source alert's operator intent from its engine syntax. Record which condition, scope, severity, persistence intent, identity, and recovery semantics Netdata preserves; record every deliberate difference. Use NETDATA-ADAPTED as the normal classification. Reserve EXACT for a proven coincidental match across timing, gaps, identity, recovery, and removal—not merely a similar expression.
• NIDL instance map: Record the monitored component, context, one instance type, RRDSET/chart ID identity, dimensions, stable identity labels, current metadata labels, and the collector's obsoletion condition. An alert attaches to one RRDSET/chart instance; a template applies that same rule independently to matching instances in one context.
• Identity: Choose an alarm only for a specific chart instance; choose a template for a context-wide rule. An identity label MUST identify the same monitored entity across its intended lifetime. A label whose source value can change for that entity is current metadata, not RRDSET identity: preserve the chart ID and update/promote the label. Confirm labels preserve the incident identity and do not create unbounded instances. Prove whether each promoted source label is live metadata, creation-time metadata, or source-frozen history. Do not describe a label as "current" merely because Netdata can update labels.
• Variables and aggregation: An unqualified variable resolves in the alert's local monitored component/instance before broader candidates. Use a fully qualified chart/dimension reference only after proving the other chart is the matching instance and labels select it unambiguously. Never synthesize an infrastructure-level alert by combining multiple RRDSET instances in alert configuration; require a source-owned aggregate RRDSET at the higher component level instead. The alert component: classification field does not replace this NIDL mapping.
• Lifecycle: Specify expected UNINITIALIZED, CLEAR, WARNING/CRITICAL, UNDEFINED, and REMOVED behavior. State whether an ordinary recovery emits a zero or whether the chart/dimension disappears instead.
• Timing: State the source update cadence, explicit alert cadence, lookup window, source persistence intent, selected Netdata adaptation, startup behavior, partial-gap behavior, stale-chart behavior, and notification policy. Do not call a lookup window an implementation of another engine's for: state machine.
• Validation: List the boundary, transition, missing-data, recovery, and duplicate-ownership tests that prove the contract.
• Policy change disclosure: for a change to a shipped alert's lookup, window, threshold or recovery, you MUST show the old and the new outcome on one short source trace that includes an active event followed by quiet samples, and state what clears the alert and whether it can clear while the underlying cumulative value stays nonzero. A lookup name and window do not disclose that trade-off; the user-owned decision is not informed without it.

Pause for a user decision when the change creates a public alert contract, changes default notification policy, changes the owner of an incident, needs shared health/query framework work, or cannot preserve the approved operator-visible incident closely enough with Netdata-native behavior. A difference from Prometheus execution alone is not such a blocker.

Combine Values Across Dimensions, Charts, And Alerts

Choose the smallest pattern that can express the condition truthfully. Do not introduce cross-chart or intermediate-alert plumbing when the required values already share the alert's chart.

Pattern 1 — dimensions on the alert's chart

Use unqualified dimension variables in calc, warn, or crit when every required value is a dimension on the chart named by on::

template: example_ratio
      on: app.state
     calc: $total - $used

Runtime facts:

• Matching is by dimension ID or name in the alert's own chart.
• The default dimension value is collector.last_stored_value: the latest database-stored value after interpolation, not a database-window aggregate and not necessarily the raw last collector sample.
• A missing dimension makes the expression fail with an unknown variable and the alert value become NaN.
• Prefer explicit non-finite guards when UNDEFINED is the required missing-input state; do not rely on arithmetic or comparisons to preserve NaN.

Use $dimension_raw only when the contract specifically needs the last collected value before interpolation/storage presentation, and $dimension_last_collected_t only when it needs that dimension's collection timestamp.

Do not treat equal dimension timestamps as a collection identifier: PLUGINSD SET calls timestamp dimensions independently. pluginsd_set calls rrddim_set_by_pointer (or its double variant), which stamps a timeval with microsecond precision; health's variable_lookup_in_chart exposes only its tv_sec field. One complete collection can cross a second boundary. Conversely, an omitted dimension retains its previous raw value, so freshness and arithmetic coherence alone can miss a partial collection. When using raw values, also test stored-input gaps: rrdset_done_interpolate sets the normal dimension value to NaN when an omitted dimension reaches a storage beat. Validate omission after a valid zero, as well as a never-present dimension. These checks provide native freshness and gap eligibility; they do not prove atomic transaction membership, particularly when collections occur faster than the storage cadence.

Pattern 2 — dimensions from another chart or context

Use a dotted chart/context reference when values live on different charts:

template: example_cross_chart
      on: app.usage
     calc: $this * 100 / ${app.capacity.total}

The dotted prefix is interpreted from right to left as chart/context plus the final dimension name. Runtime resolution:

1. An exact chart ID is checked first.
2. A chart name may match.
3. If the prefix is a context, every chart instance in that context contributes candidates for the final dimension.
4. Every candidate is scored against the alert's chart labels by counting equal key/value labels.
5. The candidate with the highest score wins. Equal scores are resolved by candidate traversal order, so avoid designs where a tie is possible.

Consequences:

• This is the natural way to compare related chart instances, but the matching labels must make the intended instance unique. A generic shared label such as only component=ceph is usually insufficient across many cluster instances.
• Like Pattern 1, the selected dimension value is the latest stored value, not a query over a time window.
• Fully qualified names containing punctuation must use ${...} braces.
• A reference that resolves to no candidate fails as an unknown variable and produces NaN.

Pattern 3 — an intermediate alert as a computed variable

Create a non-notifying helper alert when the required value itself needs a database lookup or multi-stage computation, then reference that alert by name from the consuming alert:

template: example_window
      on: app.work
   lookup: average -1h of requests
     calc: $this
      to: silent

template: example_consumes
      on: app.current
   lookup: average -1m of requests
     calc: $this / $example_window
     warn: $this > 1
      to: sysadmin

Runtime facts:

• Every running alert with the referenced name is a candidate.
• Candidates are selected by the same equal-label score used for cross-chart variables.
• The selected candidate contributes its current alert value (rc->value), after that helper's own lookup and calc.
• This is the only one of the three patterns that can combine database-window results such as “max of the last hour of X with the average of the last minute of Y”.
• Health evaluates all lookup/calculation phases before warning/critical phases, but helper snapshots are published as each alert completes. A consumer with a different cadence can therefore read the helper's previous published value on its first beat or after cadence drift. Align every: or disclose this timing difference.
• Prefer to: silent on helpers and document why they exist; a helper is instrumentation, not a second incident owner.

Choosing a pattern

• Same chart, latest values: Pattern 1.
• Different charts/contexts, latest values: Pattern 2.
• Any input needs a database window or a staged calculation: Pattern 3.
• Never use Pattern 3 merely to avoid a qualified dotted reference; its added timing and lifecycle coupling must earn its place.
• Never combine multiple RRDSET instances into one infrastructure-level alert on the alert side. If no source-owned chart provides the required aggregate, that is a collector/profile/framework gap, not an alert-expression workaround.

Model The Lifecycle Truthfully

Important rules:

• A collection failure is not disappearance. Collectors MUST preserve a gap when they cannot measure and MUST obsolete only an entity they know is gone. Do not add a false zero to make an alert recover.
• A chart can remain alert-eligible during a gap. The health loop skips an obsolete chart, but does not require a newly collected value for every scheduled evaluation. $last_collected_t and $update_every expose freshness when an alert is the deliberate stale-collection owner.
• A live chart does not guarantee every lookup remains runnable forever. For a relative database lookup, the health loop eventually skips evaluation when the newest stored point is too old for the requested window plus its bounded update-interval tolerance. Skipping evaluation preserves the prior alert state; it is different from evaluating an all-null result to UNDEFINED.
• Obsoletion ends the instance alert. Once the collector knows an entity is gone and obsoletes its RRDSET, the health rule no longer evaluates that instance. Do not attempt to emulate an infrastructure-level continuation in another alert; collect a source-owned aggregate instance if that is the required product signal.
• A null result does not fabricate healthy state. The health loop assigns NaN to an empty lookup result. A condition that itself evaluates to NaN is UNDEFINED.
• Comparisons can consume NaN. In the expression evaluator, NaN is false in boolean contexts and a comparison such as $this == 0 produces a finite false result. That can yield CLEAR, not UNDEFINED. If the contract requires UNDEFINED, make and test an expression path that preserves NaN; never assume a comparison does so.
• Some NULLs are not all NULLs. Query aggregation accepts numeric points and omits non-numeric ones. Do not assume a partial collection gap invalidates a window unless the current runtime and the approved product contract say it does. A general strict-coverage rule is shared-framework work, not an alert-file shortcut.
• Do not duplicate freshness ownership. Use $now - $last_collected_t for an explicit collection/staleness alert only when no existing generic collection-failure alert already owns the incident. A data-state alert normally owns the measured condition, not the collector outage.

For a data-state alert that MUST become UNDEFINED when its lookup is non-finite, use and test this condition shape:

warn: ($this == nan or $this == inf) ? (nan) : (<numeric predicate>)

Use crit: in the same way. This preserves the non-finite result only; it does not turn a partial-null window into a collection-failure alert or replace a known-obsolete chart's REMOVED lifecycle.

Adapt Timing To Netdata

Current State

Use calc when the source's current value is the full condition. Test start-up, missing input, normal recovery, and chart obsoletion separately. delay: controls notification hysteresis, not the evaluated alert state; it is not a substitute for a Prometheus for: duration.

Control Flapping With Three Combinable Layers

Stability involves signal smoothing, threshold hysteresis and notification delay. Address them in that order. State-dependent thresholds are hysteresis; delay: postpones notification execution while the evaluated alert state changes immediately. See src/health/REFERENCE.md#alert-line-delay and its conditional-operator hysteresis section.

Layer 1 — stabilize the queried value

Use lookup: to make $this a stable aggregate over an explicit observation window:

lookup: average -5m unaligned of latency

• average smooths noisy utilization, rate, latency, and utilization-like signals.
• min requires every observed numeric sample to remain active, appropriate for persisted binary fault states.
• max preserves worst-case excursions when the incident is defined by peaks.
• countif expresses percent-of-observed-time semantics.

This reduces noise but cannot prevent a stable aggregate from hovering near one threshold. A 5-minute average near 100 ms can still cross a 100 ms threshold repeatedly.