Analytics Opensearch Expertise

OpenSearch Domain Health Assessment

Overview

This skill performs a comprehensive, read-only health assessment of Amazon OpenSearch Service domains using AWS APIs available through the customer's local AWS profile. It produces a structured report with findings and actionable recommendations.

Scope: CLI-agent compatible. No external packages, no IDE workspace, no human review gates.

Trigger Keywords

opensearch, opensearch health, domain health, cluster review, opensearch review, opensearch assessment, opensearch check, open search, opensearch performance, opensearch security, opensearch cost

Prerequisites

• AWS CLI profile configured with read-only access to the target account
• IAM permissions required:
  • es:DescribeDomain
  • es:DescribeDomainHealth
  • es:GetCompatibleVersions
  • es:DescribeReservedInstances
  • es:ListDomainNames
  • cloudwatch:GetMetricData

Execution Flow

Input

Required: Domain ARN (e.g., arn:aws:es:us-west-2:123456789012:domain/my-domain)

Parse the ARN to extract:

• Region: field 4 (e.g., us-west-2)
• Account ID: field 5 (e.g., 123456789012)
• Domain name: after domain/ in field 6

Fallback / Target Resolution (apply these rules exactly — never silently substitute a target):

1. Exact match: the ARN (or name + region) resolves via DescribeDomain → proceed with the assessment.
2. No domain specified: run ListDomainNames in the user's likely region(s) (cross-region scan if needed) and present the discovered domains as options for the user to choose. Do not pick one yourself unless exactly one domain exists in the entire account — state that you did so.
3. Specified but NOT found: search for candidates (ListDomainNames in the ARN's region, then other regions). Then STOP and confirm — do NOT assess:
   • Report clearly: "Domain {given} not found in {region}."
   • List the closest candidate(s) with their regions (e.g., name differs by a suffix, or same name in another region).
   • Ask the user to confirm which domain to assess. Never run the assessment against a domain the user did not name without their explicit confirmation — in accounts with similarly-named sibling domains (e.g., prod-es-content / prod-es-users), a silently substituted target produces a confident report about the wrong domain, which is worse than an error.
4. If the user provides only a domain name with no region, ask for the region (or apply rule 3's candidate search).

Steps

1. Parse domain ARN → region, account, domain name
2. Resolve the target (Target Resolution rules above) — confirm with the user if not an exact match
3. Describe domain configuration (DescribeDomain)
4. Describe domain health (DescribeDomainHealth)
5. Pull CloudWatch metrics (last 24h)
6. Check upgrade eligibility (GetCompatibleVersions; ListVersions fallback)
7. Check reserved instance coverage (DescribeReservedInstances)
8. Analyze findings against thresholds
9. Load the Remediation Reference — read_skill_resource(skill_id='analytics-opensearch-expertise',
   path='references/remediation-reference.md') — MANDATORY before writing recommendations
10. Generate report with recommendations

API Compatibility (DevOps Agent tooling)

The DevOps Agent AWS tool exposes the LEGACY Elasticsearch-compatible client. Call legacy operation names FIRST; fall back to the modern names only if the legacy call fails:

• IAM note: the execution role's managed policy grants es:Describe* and es:List* — Describe/List operations succeed under either naming; Get* actions (BOTH GetCompatible variants) are typically denied. This is exactly why the 1.4 ListVersions fallback exists.
• DescribeDomainHealth has NO legacy equivalent. If it fails, do NOT skip checks 2.2/2.3 — use CloudWatch instead: Shards.active (Maximum) = active shards, Shards.unassigned (Maximum) = unassigned, TotalShards ≈ active + unassigned; ClusterStatus.green/yellow/red for 1.1/3.7. Never report the checks as SKIPPED when this fallback is available.
• All CloudWatch metrics in this skill: namespace AWS/ES, dimensions DomainName and ClientId (the account ID).

Checks & Decision Logic

Category 1: Cluster Health & Configuration

1.1 Cluster Status

• API: DescribeDomainHealth
• Field: ClusterHealth (green | yellow | red)
• Logic:
  • GREEN → ✅ All primary and replica shards assigned
  • YELLOW → ⚠️ All primaries assigned, some replicas unassigned
  • RED → 🔴 Some primary shards unassigned — data loss risk

1.2 Node Count & AZ Balance

• API: DescribeDomain → ClusterConfig
• Fields: InstanceCount, ZoneAwarenessEnabled, AvailabilityZoneCount
• Logic:
  • If ZoneAwarenessEnabled == true:
    • InstanceCount % AvailabilityZoneCount != 0 → ⚠️ "Data node count ({count}) is not a multiple of AZ count ({az_count}). This causes uneven shard distribution across AZs, reducing fault tolerance."
    • Recommendation: "Adjust data node count to a multiple of {az_count} (e.g., {nearest_multiple})."
  • If ZoneAwarenessEnabled == false and InstanceCount > 1:
    • ⚠️ "Multi-AZ is not enabled despite having {count} data nodes. Single-AZ deployment risks full cluster unavailability during AZ failure."

1.3 Dedicated Master Configuration

• API: DescribeDomain → ClusterConfig
• Fields: DedicatedMasterEnabled, DedicatedMasterType, DedicatedMasterCount
• Logic:
  • DedicatedMasterEnabled == false AND InstanceCount >= 3 → ⚠️ "No dedicated master nodes. Clusters with 3+ data nodes should use dedicated masters for stability."
  • DedicatedMasterCount == 2 → ⚠️ "Even number of master nodes ({count}) risks split-brain. Use 3 dedicated master nodes."
  • DedicatedMasterCount < 3 → ⚠️ "Fewer than 3 master nodes reduces fault tolerance."

1.4 Engine Version & Upgrade Eligibility

• API: GetCompatibleVersions with DomainName
• Fallback (use when GetCompatibleVersions returns AccessDeniedException — do NOT skip): call ListVersions and derive upgrade candidates using these two rules:
  1. Direct upgrade candidates = versions in ListVersions with the SAME major version as the domain and a LATER minor version (e.g., a 2.11 domain → every later 2.x in the region).
  2. Next-major path = via the LATEST available minor of the current major (compute it from ListVersions; never hard-code a specific version number). Label fallback results as "derived upgrade candidates (ListVersions + documented upgrade-path rules)" and cite https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html for the reader. Only mark 1.4 SKIPPED if BOTH APIs are denied. Note placement: the fallback provenance note belongs INSIDE the 1.4 finding ONLY — do NOT surface it in the Executive Summary or any report-level note box, and do NOT name API errors (e.g., AccessDeniedException) anywhere in the report. Phrase it customer-friendly: "Upgrade candidates were derived from the available version listings and are not API-confirmed — verify eligible target versions in the console before planning an upgrade." Render candidates as a SET of direct single-hop targets (comma-separated) — NEVER as an arrow chain (NOT "2.13 → 2.15 → 2.19"): every listed same-major version is directly reachable in ONE hop. Any upgrade recommendation MUST propose a single hop to the latest same-major version (plus one more hop for a major jump). Do NOT invent sequential multi-hop plans or per-hop time estimates.
• Logic:
  • If compatible target versions exist → ℹ️ "Current version: {current}. Upgrades available: {targets}."
  • If current version is more than 2 releases behind the latest same-major version → ⚠️ "Consider upgrading for performance improvements and security patches."
  • Counting rule (apply exactly): "releases behind" = the COUNT of later same-major versions returned by ListVersions/GetCompatibleVersions (e.g., 2.11 with {2.13, 2.15, 2.17, 2.19} available = 4 releases behind) — NEVER the numeric delta between version numbers (2.19 − 2.11 is NOT "8 versions behind").

Category 2: Storage & Shard Strategy

2.1 EBS Configuration

• API: DescribeDomain → EBSOptions
• Fields: VolumeType, VolumeSize, Iops, Throughput
• Logic:
  • VolumeType == "gp2" → ⚠️ "Using gp2 volumes. gp3 offers better price/performance with configurable IOPS and throughput."
  • Report total cluster storage: VolumeSize × InstanceCount
• API (throttle detection): CloudWatch IopsThrottle, ThroughputThrottle, ReadIOPSMicroBursting, WriteIOPSMicroBursting, ReadThroughputMicroBursting, WriteThroughputMicroBursting (Maximum statistic, last 7 days)
• Logic (throttle detection):
  • IopsThrottle Max > 0 OR ThroughputThrottle Max > 0 in the window → ⚠️ "EBS IOPS/throughput throttling detected. Increase provisioned Iops/Throughput on gp3 via UpdateDomainConfig."
  • Throttle metrics 0 but any microbursting metric > 0 on multiple days → ℹ️ "Workload bursts above the provisioned EBS baseline; monitor — sustained growth will lead to throttling. Consider raising gp3 Iops/Throughput."
  • No datapoints (non-EBS domain, e.g. instance-store or OR remote-store instance families) → skip this sub-check silently; do NOT mark check 2.1 SKIPPED.

2.2 Shard Count & Density

• API: DescribeDomainHealth
• Fields: TotalShards, ActiveShards
• Compute: shards_per_node = TotalShards / InstanceCount
• Version-dependent limits (from DescribeDomain → EngineVersion):
  • OpenSearch ≤ 2.15: Hard limit = 1,000 shards/node
  • OpenSearch 2.17+: Hard limit = 1,000 shards per 16 GB of data node heap
• Heap estimation (MUST use the real instance type, not an assumed value): estimate per-node JVM heap from the data node instance type: heap_GiB ≈ min(32, instance_RAM_GiB / 2) (e.g., t3.medium.search 4 GiB RAM → ~2 GiB heap; r6g.xlarge.search 32 GiB RAM → ~16 GiB heap). Instance RAM reference (GiB): t3.small=2 · t3.medium=4 · c6g.large=4 · m6g.large=8 · c6g.xlarge=8 · r6g.large=16 · m6g.xlarge=16 · c6g.2xlarge=16 · r6g.xlarge=32 · m6g.2xlarge=32 · r6g.2xlarge=64 (heap caps at 32 GiB). For unlisted types use the EC2 equivalent's memory — never guess silently; name the assumed RAM in the report.
• Best-practice threshold: bp_limit = 25 × heap_GiB shards/node.
• Logic (apply these exact thresholds — do not substitute your own; the verdict tier is determined ONLY by which line matches, never by judgment):
  • shards_per_node > hard_limit → 🔴 "Exceeds hard shard limit for engine version {version}."
  • shards_per_node > 4 × bp_limit → 🔴 "Shard density ({shards_per_node}/node) is more than 4× the best-practice limit ({bp_limit}/node for ~{heap_GiB} GiB heap). Severe heap/GC and recovery risk." (This tier is 🔴, not ⚠️ — do not downgrade.)
  • shards_per_node > bp_limit → ⚠️ "Shard density ({shards_per_node}/node) exceeds best-practice threshold of 25 shards/GiB heap ({bp_limit}/node for ~{heap_GiB} GiB heap)."
  • shards_per_node > 0.5 × bp_limit → ℹ️ "Moderate shard density. Monitor JVM heap pressure."
  • shards_per_node <= 0.5 × bp_limit → ✅ PASS. (Verbatim: at or below half the best-practice limit the verdict is PASS, never ℹ️ — low density is not a finding.)

2.3 Active Shards vs. Total Shards

• API: DescribeDomainHealth
• Fields: ActiveShards, TotalShards
• Expected: ActiveShards == TotalShards (all shards placed and serving)
• Logic:
  • ActiveShards < TotalShards:
    • Delta = TotalShards - ActiveShards
    • ⚠️ "{delta} shards are not active. Non-active shards may be:"
    • "• Initializing — being allocated after a node restart or scaling event (transient, typically resolves in minutes)"
    • "• Relocating — being moved during a rebalance operation (transient)"
    • "• Unassigned — cannot be placed on any node (persistent problem). Common causes: insufficient disk space, allocation filtering rules, AZ awareness constraints with unbalanced node counts, or max shards per node limit reached."
    • Recommendation: "Check _cluster/allocation/explain for unassigned shard root cause."

2.4 Free Storage Space

• API: CloudWatch FreeStorageSpace (Minimum statistic, last 24h)
• CRITICAL — the Minimum statistic is PER-NODE (the lowest single node's free MB), NOT a cluster-wide total. Compute free % against ONE node's volume, never the cluster total: free_pct = FreeStorageSpace_min_MB / (VolumeSize_GiB × 1024) × 100 (Dividing the per-node Minimum by VolumeSize × InstanceCount understates free space by a factor of InstanceCount — this is wrong. Cross-check: the Sum statistic ÷ (VolumeSize × InstanceCount × 1024) should give approximately the same percentage.)
• Note: usable space per node is less than the raw volume size due to OS/service reserved overhead — an empty node typically shows ~72% free, not ~97%.
• Logic (apply these exact thresholds verbatim):
  • free_pct < 10% → 🔴 "Free storage below 10%. High watermark may trigger read-only mode."
  • free_pct < 20% → ⚠️ "Free storage below 20%. Approaching low watermark. Plan capacity increase."
  • free_pct < 30% → ℹ️ "Storage utilization above 70%. Monitor trend."
  • free_pct >= 30% → ✅ PASS.
  • If the metric returns NO datapoints (new domain, or configuration change in progress) → SKIPPED "FreeStorageSpace metric unavailable" — never silently omit the check.

Category 3: Performance Metrics

Note on percentiles: CloudWatch metrics for OpenSearch (SearchLatency, IndexingLatency) report aggregate values, not per-request percentiles. We use the Maximum statistic over 1-minute periods as a "worst-case" proxy. True p50/p99 latency requires OpenSearch slow-log analysis or UltraWarm query insights, which is outside the scope of API-only checks.

Time range: Last 24 hours, 1-minute period granularity.

3.1 JVM Memory Pressure

• Metric: JVMMemoryPressure, Statistic: Maximum — the verdict MUST be computed from the Maximum statistic. Never substitute Average (an idle cluster commonly shows Avg ~43% while Max sustains ~74% — these produce different verdicts). Report Average only as context.
• Logic:
  • Max > 92% → 🔴 "JVM heap pressure critically high ({value}%). Cluster is at risk of circuit breaker trips and OOM. Consider scaling up instance type or reducing shard count."
  • Max > 80% → ⚠️ "JVM heap pressure elevated ({value}%). Approaching GC thrashing threshold."
  • Max > 70% → ℹ️ "JVM heap pressure moderate. Monitor trend."

3.2 CPU Utilization

• Metric: CPUUtilization, Statistic: Average (sustained) and Maximum (spikes)
• Logic:
  • Avg > 80% sustained (>50% of datapoints) → ⚠️ "Sustained high CPU. Cluster may be undersized for current workload."
  • Max > 95% → ℹ️ "CPU spikes observed. Check for expensive queries or bulk indexing bursts."

3.3 Search Latency

• Metric: SearchLatency, Statistic: Maximum (worst-case proxy)
• Logic:
  • Max > 500ms → ℹ️ "Worst-case search latency {value}ms — elevated spikes observed. Investigate slow queries if user-facing impact is reported."
  • Report average as baseline context.
  • Informational only — search latency NEVER renders ⚠️ or 🔴. The 500ms marker is a heuristic: AWS does not define a canonical latency threshold; acceptable latency is workload-dependent, so frame findings against the customer's own baseline.

3.4 Indexing Latency

• Metric: IndexingLatency, Statistic: Maximum (worst-case proxy)
• Logic:
  • Max > 500ms → ℹ️ "Worst-case indexing latency {value}ms. Check merge pressure or refresh interval if ingestion throughput is impacted."
  • Report average as baseline context.
  • Informational only — indexing latency NEVER renders ⚠️ or 🔴. Heuristic marker; frame against the workload's own baseline.