Msk Operations

Amazon MSK Operations

Operate, troubleshoot, and assess Amazon MSK (Managed Streaming for Apache Kafka) Provisioned clusters — both Standard and Express broker types. This skill covers day-to-day operations (health assessments, monitoring setup) and ad-hoc incident response (performance degradation, consumer lag, storage full, unexpected broker reboots).

When to Use

Activate this skill when the user asks to:

• Review, audit, or assess an MSK cluster for best practices, health, or operational readiness.
• Troubleshoot an MSK cluster problem: high CPU, high produce/fetch latency, consumer lag, broker storage running out, TrafficShaping events, under-replicated partitions, or an unexpected broker restart.
• Set up MSK monitoring: choose a monitoring level, create recommended CloudWatch alarms and dashboards, understand the metrics available in the AWS/Kafka namespace.
• Plan an MSK maintenance event: rolling restart, Kafka version upgrade, security patching, broker instance type change.
• Advise on Kafka client (producer / consumer) configuration when the client is connecting to an MSK cluster.

Do not activate this skill for MSK Connect, MSK Serverless, or MSK Replicator — those are separate services with their own operational surfaces.

Broker Type Determination

Determine the broker type first — many checks differ between Standard and Express.

aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>

Check ClusterInfo.Provisioned.BrokerNodeGroupInfo.InstanceType:

• Starts with kafka. (e.g. kafka.m5.large, kafka.m7g.xlarge) → Standard broker.
• Starts with express. (e.g. express.m7g.large) → Express broker.

Key Standard vs Express differences

Standard brokers use customer-managed EBS volumes for storage. You choose instance types (kafka.m5.*, kafka.m7g.*), provision EBS, and manage storage scaling. Standard brokers have scheduled maintenance windows.

Express brokers provide fully managed, pay-as-you-go storage with no EBS provisioning. Instance types are prefixed with express.m7g.*. Express brokers offer up to 3× more throughput per broker than Standard, and have no maintenance windows. Express enforces a fixed replication factor of 3 and min.insync.replicas=2 — you cannot create topics with RF=1.

Critical Warnings

• This skill is read-only. Every command in this file and in references/ that mutates cluster state — update-broker-storage, create-configuration, update-cluster-configuration, update-monitoring, put-metric-alarm, reboot-broker, and any partition reassignment — is a recommendation for the operator to run after review. Present these as proposed remediations with expected impact and preconditions; do NOT execute them, and do NOT imply that the agent will run them.
• NEVER reboot brokers while UnderReplicatedPartitions > 0 (Standard only — Express brokers do not emit URP). This risks data loss and extended outages.
• NEVER recommend partition reassignment without first checking replication status. Reassignment during URP compounds the problem.
• linger.ms=0 is the #1 cause of "high CPU" on MSK. ALWAYS check client batch configuration before recommending broker scaling.
• EBS throughput ceilings are invisible in Kafka metrics — ALWAYS check EBS volume metrics (VolumeReadBytes, VolumeWriteBytes, VolumeQueueLength) when diagnosing Standard broker latency.
• Express brokers have NO customer-managed EBS — do NOT recommend EBS expansion or provisioned EBS throughput for Express clusters.
• Express brokers enforce fixed RF=3 and min.insync.replicas=2 — do NOT attempt to create topics with RF=1 on Express. If RF=1 is needed, use Standard brokers.

Quick Diagnostics

These five checks cover the most common MSK issues. Use them before loading a reference file.

1. CpuUser + CpuSystem > 60%: Check RequestHandlerAvgIdlePercent (PER_BROKER monitoring level). If < 30%, request threads are saturated. Check client batch.size and linger.ms before recommending scaling.

2. KafkaDataLogsDiskUsed > 85% (Standard only): Recommend to the operator that EBS be expanded via aws kafka update-broker-storage (do not execute). Identify high-growth topics via per-topic BytesInPerSec to size the increase. Express clusters use StorageUsed metric instead and storage is fully managed.

3. UnderReplicatedPartitions > 0 (Standard only): Check if a maintenance operation or broker restart is in progress. If URP is decreasing, wait for recovery. Do NOT restart brokers or reassign partitions during URP. Express brokers do not emit this metric — monitor ProduceThrottleTime, FetchThrottleTime, and consumer lag instead.

4. Consumer OffsetLag / MaxOffsetLag increasing: Determine if broker-side (high ProduceTotalTimeMsMean, CPU saturation) or client-side (slow processing, insufficient consumers). Per-partition lag from PER_TOPIC_PER_PARTITION monitoring level helps isolate hot partitions.

5. BytesInPerSec near throughput ceiling: For Standard, check EBS volume type and calculate: BytesInPerSec × ReplicationFactor vs volume throughput limit. For Express, check against the per-broker sustained performance limits in the MSK quotas.

Which Reference Do You Need?

Route to a reference file based on the customer intent. Read the reference in full before answering — do not paraphrase from memory.

For sizing questions (broker count, instance type choice, monthly cost), refer the user to the Amazon MSK best practices — right-size your cluster documentation. Do not size from memory.

Operational Review Workflow

Use this workflow when the user asks for a review, audit, health check, or assessment of an MSK cluster. The routing table above handles ad-hoc troubleshooting; this section produces a consistent, comprehensive report.

Follow the steps in order for each target cluster. Do not skip steps. If a step cannot be completed (e.g. a metric requires a higher monitoring level than the cluster has enabled), record the gap in the report rather than silently omitting the check.

Step 1 — Identify Target Clusters

Ask the user which MSK clusters to review. Accept any of:

• Specific cluster names or ARNs and regions
• "all clusters" in specific regions
• "all MSK clusters in all regions"

If no scope is given, default to all configured account regions. Enumerate clusters with aws kafka list-clusters-v2 per region.

Step 2 — Determine Broker Type Per Cluster

For each cluster:

aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>

Read ClusterInfo.Provisioned.BrokerNodeGroupInfo.InstanceType. Standard brokers (kafka.*) and Express brokers (express.*) require different checks in the later steps — some metrics only exist on one type.

Step 3 — Collect Cluster Configuration

For each cluster, gather:

aws kafka describe-cluster-v2 --cluster-arn <arn>
aws kafka list-nodes --cluster-arn <arn>
aws kafka get-bootstrap-brokers --cluster-arn <arn>
aws kafka list-cluster-operations-v2 --cluster-arn <arn>   # last 30 days
aws kafka describe-configuration-revision \
     --arn <configuration-arn> --revision <revision>       # if a custom config is applied

Capture:

• Cluster: state, Kafka version, number of broker nodes, AZ distribution (ZoneIds), storage mode (EBS / Tiered), current version.
• Broker: instance type, EBS volume size (Standard), provisioned throughput (if any).
• Encryption: EncryptionInTransit.ClientBroker (TLS / TLS_PLAINTEXT / PLAINTEXT), EncryptionInTransit.InCluster, EncryptionAtRest.DataVolumeKMSKeyId.
• Auth: ClientAuthentication.Sasl.Iam.Enabled, ClientAuthentication.Sasl.Scram.Enabled, ClientAuthentication.Tls.Enabled, ClientAuthentication.Unauthenticated.Enabled.
• Public access: BrokerNodeGroupInfo.ConnectivityInfo.PublicAccess.Type.
• Monitoring level: EnhancedMonitoring (DEFAULT / PER_BROKER / PER_TOPIC_PER_BROKER / PER_TOPIC_PER_PARTITION).
• Logging: LoggingInfo.BrokerLogs (CloudWatch / S3 / Firehose destinations and their Enabled flags).
• Open monitoring: OpenMonitoring.Prometheus.JmxExporter.EnabledInBroker, NodeExporter.EnabledInBroker.
• Recent operations: From list-cluster-operations-v2, note any SECURITY_PATCHING, BROKER_UPDATE, UPDATE_CLUSTER_CONFIGURATION, UPDATE_STORAGE, or UPDATE_MONITORING events in the review window.

Step 4 — Detect Monitoring Level and Gaps

The EnhancedMonitoring value from Step 3 determines which checks are available. At DEFAULT, most per-broker health metrics are still available (CPU, disk, network, partitions, connections, memory, TrafficShaping), but the following checks are not possible without upgrading:

• ReplicationBytesInPerSec / ReplicationBytesOutPerSec (inter-broker replication load)
• RequestHandlerAvgIdlePercent / NetworkProcessorAvgIdlePercent (thread pool saturation)
• BwInAllowanceExceeded / BwOutAllowanceExceeded (detailed bandwidth breaches)
• VolumeQueueLength (EBS I/O queue depth — Standard only)
• VolumeReadBytes / VolumeWriteBytes (EBS throughput utilization — Standard only)
• IAM connection metrics (IAMTooManyConnections)

Record the monitoring level and list any dimensions that will be scored partially or skipped. Recommend upgrading to PER_BROKER if any dimension is degraded by the current level.

Step 5 — Collect CloudWatch Metrics (7-Day Historical)

Namespace: AWS/Kafka. Dimensions: Cluster Name and Broker ID for per-broker metrics; Cluster Name and Consumer Group and Topic for consumer lag; Cluster Name only for cluster-wide metrics.

Use one cloudwatch.GetMetricData batch per cluster where possible. Period: 3600 (1 hour). StartTime: 7 days ago. EndTime: now.

5.1 Cluster-wide (both broker types)

5.2 Per-broker — Standard (kafka.*)

5.3 Per-broker — Express (express.*)

Express brokers do NOT emit: UnderReplicatedPartitions, UnderMinIsrPartitionCount, HeapMemoryAfterGC, TrafficShaping, Volume* metrics, KafkaDataLogsDiskUsed, ProduceMessageConversionsPerSec, FetchMessageConversionsPerSec.

5.4 Consumer lag (both broker types)

Per consumer group (identified from the customer or from the broker logs):

Step 6 — Alarm Coverage

Collect existing alarms:

aws cloudwatch describe-alarms --namespace AWS/Kafka

Compare against the 13-alarm recommended set (details in references/monitor-and-alarm.md) and produce a coverage table (present / missing / firing).

Step 7 — Analyze Against Best Practices

Assign a severity to every finding: CRITICAL / HIGH / MEDIUM / LOW / INFO (see Severity Definitions later in this file).

Evaluate across seven dimensions. Some checks are skipped for Express — noted inline.