Elasticsearch Anomaly Detection Explainer

Anomaly Detection Score Explainer

Explain anomaly scores, model behavior, and why results look the way they do. Use the ML REST API for job config and the standard _search API against .ml-anomalies-* for results — no ES|QL, fully compatible with Elastic Serverless. For job lifecycle (create, start, stop), use the elasticsearch-anomaly-detection skill.

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the elastic CLI. If the elastic CLI is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping, GET /{index}/_settings/index.mode, POST /_query). The Operations table at the end of this document maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API directly.

<!-- end-partial: preamble -->

Prerequisite: ML anomaly detection requires a Platinum-equivalent license on self-managed clusters. Serverless projects include ML. The caller needs monitor_ml to read job config and anomaly results.

Serverless note: The _ml/.../results/* REST endpoints return HTTP 410 in Elastic Serverless. Always use POST /.ml-anomalies-*/_search for result queries instead — fully supported everywhere this skill runs.

Process

1. Decide whether to fetch data or interpret what the user supplied. If the user embeds an anomaly record (or job config) in the prompt, interpret it directly using the domain knowledge below — do not call APIs to re-fetch fields already present. If the job ID, time range, or record is missing, retrieve it from the cluster.

   The decision: proceed with judgment-only explanation when the record contains record_score, initial_record_score, actual, typical, and function; otherwise fetch the missing pieces before explaining.

2. Verify connectivity when calling the cluster. Call GET /. If the call fails, stop and surface the connection error — do not guess endpoints or credentials.

3. Resolve the job ID and load config. When the job ID is unknown, call GET /_ml/anomaly_detectors to list candidates. Call GET /_ml/anomaly_detectors/{job_id} for full analysis_config (bucket_span, detectors, custom_rules, use_null, model_plot_config) and GET /_ml/anomaly_detectors/{job_id}/_stats for state, model_size_stats.memory_status, and data counts.

   The decision: confirm detector function and direction match the user's question before interpreting scores. A low_count job legitimately fires on drops; a high_count job does not.

4. Retrieve anomaly records for the time range. Call POST /.ml-anomalies-*/_search with result_type: record, the job ID, a timestamp range, and optional record_score filter. Read initial_record_score, record_score, actual, typical, function, multi_bucket_impact, and anomaly_score_explanation.

   Always show both initial_record_score and record_score. The gap is the renormalization story.

5. Classify the score pattern before speculating on causes.

   • initial_record_score >> record_score — Renormalization. A later, more extreme anomaly rescale this record downward. This is expected, healthy model behavior — not a broken model or reason to distrust the detection. Use initial_record_score for alerting severity; show both scores and explain the gap explicitly.
   • initial_record_score == record_score — No renormalization has occurred since detection.
   • actual << typical with low_count, count, or low_mean — Absence / drop anomaly. A high score is legitimate — the job detected an outage, pipeline stall, or service failure. This is not a false positive. Recommend incident investigation, not score tuning.
   • actual >> typical with high_count or high_mean — Spike anomaly; confirm with single_bucket_impact.

   Only cite anomaly_score_explanation factors present in the record. If high_variance_penalty is false, do not blame variance. If a factor is absent, note that it was not returned — do not invent it.

6. Quantify renormalization across the job (optional). Re-query POST /.ml-anomalies-*/_search for records in the time range sorted by timestamp ascending. Compute score_drift = initial_record_score − record_score per record and filter to |score_drift| ≥ 20. Large negative drift (initial >> record) confirms renormalization after a more extreme anomaly appeared later.

7. Add context when the user asks "what caused this?" or "why so low/high?"

   • Model bounds — If model_plot_config.enabled is true, call POST /.ml-anomalies-*/_search with result_type: model_plot for the same job and time range. Compare actual to model_lower / model_upper.
   • Influencers — Call POST /.ml-anomalies-*/_search with result_type: influencer for the bucket time range; sort by influencer_score descending.
   • Categorization jobs — Call POST /.ml-anomalies-*/_search with result_type: category_definition to list learned log patterns (terms, regex, examples per category_id).

   For aggregations, cross-job queries, bucket-level results, or custom filters beyond score and time, see references/explainer-reference.md.

Common multi-step workflows

Critical principles

• Retrieve the record first (or use the one the user supplied). Never explain scores without initial_record_score, record_score, actual, typical, and function.
• Renormalization is healthy. When initial_record_score >> record_score, a more extreme anomaly appeared later and lowered this score — expected behavior, not a model failure.
• Direction matters. low_count fires when values drop; high_count fires on spikes. A high score on a traffic stop with low_count is correct detection, not a false positive.
• Explain factors before speculating. Read anomaly_score_explanation from the record. Only address factors that are present and relevant.
• Job config is essential. bucket_span, detector function, custom_rules, use_null, and memory status all affect scores. Inspect job config when a score is surprising.
• Model plot is the most visual explanation. When enabled, show model bounds to illustrate where the actual value falls relative to the expected range.
• For job health ("missing documents", "memory limit", "datafeed not running") use the elasticsearch-anomaly-detection skill.

Domain knowledge

Score types

anomaly_score_explanation factors

The anomaly_score_explanation field on each record breaks the score into components: