Kibana Anomaly Detection

Elastic ML Anomaly Detection

Expert process for ML anomaly detection: attribute incidents to entities, explain scores and model behavior, diagnose job lifecycle failures, and manage jobs. Read anomaly results from POST /.ml-anomalies-*/_search (Serverless-safe) and job/datafeed state from ML REST APIs. When the user embeds fixture evidence (influencer rows, job stats) in the prompt, apply the judgment below directly — do not re-fetch fields already supplied.

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the elastic CLI. If the elastic CLI is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping, GET /{index}/_settings/index.mode, POST /_query). The Operations table at the end of this document maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API directly.

<!-- end-partial: preamble -->

Mode selector

When a question spans modes: Investigate → Explain → Troubleshoot. Finish one mode before blending logic.

Serverless note: Legacy /_ml/anomaly_detectors/{job_id}/results/* endpoints return HTTP 410 in Serverless. Always query .ml-anomalies-* via POST /.ml-anomalies-*/_search with result_type filters.

Score quick reference

• record_score bands: >75 critical · 50–75 warning · 25–50 minor · <25 informational
• multi_bucket_impact ≥ 3 → sustained shift (not a transient spike)
• initial_record_score >> record_score → renormalization (model saw worse anomalies later)
• actual << typical with count/low_count/low_mean → absence/outage, not just a low value
• Low scores across many jobs > one high score — composite cross-job signal often beats single-detector severity

Full score definitions, renormalization mechanics, and anomaly_score_explanation components: references/score-reference.md.

Core concepts

Treat .ml-anomalies-* as layered result types via result_type in search queries:

Read scores this way:

• anomaly_score / record_score = current normalized values (move as the model sees new extremes).
• initial_anomaly_score / initial_record_score = immutable snapshots from detection time.
• influencer_score ranks entity responsibility within a bucket — the highest score is the primary suspect, not the bucket-level anomaly_score alone.
• Map entities via partition_field_value / by_field_value / over_field_value.
• Read multi_bucket_impact (-5 to +5) to separate single-bucket spikes from sustained trends.

---

Mode: Investigate — RCA

When: "what broke?", "which entity caused this?", cross-job correlation, blast radius, attack/cascade chains.

Process

1. Discover jobs. Call GET /_ml/anomaly_detectors when the job ID is unknown. Call GET /_ml/anomaly_detectors/{job_id} and GET /_ml/datafeeds/datafeed-{job_id} to learn source indices, entity fields (by_field_name, over_field_name, partition_field_name), and bucket_span. The decision: identify the related job group — jobs sharing a datafeed index or entity field monitor the same system from different angles.

2. Scope the incident window. Call POST /.ml-anomalies-*/_search with result_type: bucket, a time range, and optional minimum anomaly_score. The decision: fix the incident start/end and count how many jobs co-fire in that window. Low scores across many jobs simultaneously often indicate a systemic root cause.

3. Attribute to entities (critical for RCA). For the anomalous bucket timestamp, call POST /.ml-anomalies-*/_search with result_type: influencer, the job ID(s), and the bucket time range. Sort by influencer_score descending. The decision: name the entity with the highest influencer_score as the likely cause — it ranks how unusual each entity is in that bucket. Do not restate only the bucket anomaly_score without attributing responsibility. Recommend drilling into that entity's records next.

4. Cross-job confirmation. Re-query influencers (or bucket records) across related job IDs for the same entity values and time window. Entities anomalous in 2+ jobs are prime suspects (resource fault or systemic failure); single-job entities are often downstream victims. See references/protocols/investigation.md.

5. Drill into records. Call POST /.ml-anomalies-*/_search with result_type: record, exact job ID, entity filters (partition_field_value, by_field_value), and low minimum record_score (25 or lower). Read multi_bucket_impact ≥ 3 as sustained behavioral shift. Read actual vs typical for fault class (spike vs absence/outage).

6. Confirm with source evidence. Call POST /{index}/_search on the datafeed source index for the suspect entity and time window. Raw source documents are ground truth — never close an RCA without them.

7. Synthesize. Report: root cause entity · affected jobs · temporal progression · fault class · severity · recommended actions. Worked walkthrough: references/worked-example.md. Query templates: references/investigation-queries.md.

Rules

1. Rank by influencer_score, not anomaly_score, for "which entity?" — bucket score is aggregate; influencer score attributes cause.
2. Multi-job entities are prime suspects; single-job entities are usually victims.
3. Earliest anomaly timestamp wins — reconstruct chronology from record timestamps across jobs.
4. multi_bucket_impact ≥ 3 = sustained behavioral shift, weight higher than transient spikes.
5. Use low score thresholds (25 or lower) for influencer/record queries — high thresholds miss correlated entities.
6. Never close an RCA without source evidence from the datafeed index.

---

Mode: Explain — Score / model behavior

When: "why is my score 30/90?", "score dropped overnight", "what is renormalization?", "why wasn't this detected?".

Process

1. Decide fetch vs interpret. If the user supplies a record with record_score, initial_record_score, actual, and typical, interpret directly. Otherwise load config with GET /_ml/anomaly_detectors/{job_id} and records with POST /.ml-anomalies-*/_search (result_type: record).

2. Always show both initial_record_score and record_score. The gap is the renormalization story. Large positive drift (initial_record_score >> record_score) means a later, more extreme anomaly rescale this record downward — expected healthy behavior, not a broken model.

3. Classify the pattern before speculating.

   Pattern	Interpretation
   initial_record_score >> record_score	Renormalization — explain before suggesting config changes
   actual << typical with low_count/count/low_mean	Absence/outage anomaly — investigate the outage, not score tuning
   high_variance_penalty: true in anomaly_score_explanation	Noisy metric — wide bounds absorbed the spike
   incomplete_bucket_penalty: true	Ingest lag or sparse bucket — score legitimately reduced

   Only cite anomaly_score_explanation factors present in the record.

4. Quantify renormalization (optional). Re-query records sorted by timestamp; compute score_drift = initial_record_score - record_score and flag large drift.

5. Add visual context when needed. If model_plot_config.enabled, query result_type: model_plot and compare actual to model_lower/model_upper. For categorization jobs, query result_type: category_definition.

6. Check job health when scores look wrong persistently. Call GET /_ml/anomaly_detectors/{job_id}/_stats — model_size_stats.memory_status of hard_limit corrupts learning and can invalidate scores. Escalate to Troubleshoot mode.

anomaly_score_explanation components

Rules

1. Explain renormalization before diagnosing config — score drift is the most common "score dropped" cause.
2. actual << typical with count/low_count is an absence anomaly — distinguish outages from value spikes.
3. Weekly seasonality needs ≥3 weeks of training data — flag young jobs as the cause.
4. Detector function direction matters — see references/anomaly-detection-functions.md.

---

Mode: Troubleshoot — Job lifecycle

When: "missing documents", "datafeed stopped", hard_limit, "results look wrong", lifecycle changes.