New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Official agent skill

Gke AI Troubleshooting Tpu Performance Degradation

by google in google/skills

Diagnose GKE Cloud TPU training throughput drops and step-time regressions (15%+ TPU duty-cycle drop) using ML Diagnostics Workload Monitoring (gcloud alpha mldiagnostics monitored-events /…

OfficialApache-2.0Auto-check passedDevOps & Cloud

Install Gke AI Troubleshooting Tpu Performance Degradation

skills CLI
$ npx skills add google/skills --skill gke-ai-troubleshooting-tpu-performance-degradation -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install google/skills gke-ai-troubleshooting-tpu-performance-degradation --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/google/skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation .claude/skills/gke-ai-troubleshooting-tpu-performance-degradation && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
gke-ai-troubleshooting-tpu-performance-degradation
GitHub stars
21k
Token cost
~3.5k tokens
SKILL.md length
1,179 words
Files
5 (incl. references)
Skills in repo
150
Repo updated
First seen
Licence
Apache-2.0

At a glance

Diagnose GKE Cloud TPU training throughput drops and step-time regressions (15%+ TPU duty-cycle drop) using ML Diagnostics Workload Monitoring (gcloud alpha mldiagnostics monitored-events /…

  • Works in 5 steps: Collect context and set the… → Query ML runs and monitoredEvents [Low… → Correlate with 1-minute Cloud Monitoring… → …
  • TPU training throughput
  • SKILL.md covers Prerequisites, Analyzer routing overview, Diagnostic workflow and Guardrails
  • Calls gcloud and kubectl

What it does

Gke AI Troubleshooting Tpu Performance Degradation is an agent skill from google/skills, published by the product's own GitHub organization. Diagnose GKE Cloud TPU training throughput drops and step-time regressions (15%+ TPU duty-cycle drop) using ML Diagnostics Workload Monitoring (gcloud alpha mldiagnostics monitored-events / hypercomputecluster.googleapis.com/v1alpha) and 1-minute Cloud Monitoring system metrics (kubernetes.io/node/accelerator/). Distinguishes hardware and network fabric throttling from workload resource bottlenecks (HBM capacity, host memory, or host CPU saturation). Use when TPU training throughput or duty cycle drops without…

Its SKILL.md is about 3.5k tokens, which your agent loads only when the skill is triggered. The skill folder holds 5 other files, including reference files (for example `references/path-a-workload-bottlenecks.md`, `references/path-b-infrastructure-throttling.md` and `references/path-c-no-detected-analyzer.md`).

It sits in DevOps & Cloud, covering Container orchestration. It works with Google Kubernetes Engine, Google Cloud and Kubernetes. The repository describes itself as: Agent Skills for Google products and technologies. The licence is Apache-2.0.

When your agent uses it

  • TPU training throughput
  • Duty cycle drops without crashing pods
  • PERFORMANCEDEGRADATION monitored events fire
  • Triaging slow multi-slice training steps

Example prompts

  • “/gke-ai-troubleshooting-tpu-performance-degradation”

Workflow steps

5 steps, taken from the step headings in SKILL.md.

  1. Collect context and set the investigation window [Low Risk]
  2. Query ML runs and monitoredEvents [Low Risk]
  3. Correlate with 1-minute Cloud Monitoring system metrics [Low Risk]
  4. Map culprit instance IDs to GKE nodes and topology [Low Risk]
  5. Remediation by analyzer category

What it can do on your machine

Read from SKILL.md and the folder at commit 4b940dd. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Shell commands in SKILL.md call:

    • gcloud
    • kubectl

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Links to these hosts (documentation or services it may open):

    • docs.cloud.google.com
    • cloud.google.com

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Gke AI Troubleshooting Tpu Performance Degradation loads about 3.5k tokens when it runs, and up to ~5.1k if it reads all its reference files. Until then it costs about 226 tokens; SKILL.md has 1,179 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~226
When it runs · the whole SKILL.md, loaded when a task matches
~3.5k
With references · SKILL.md plus every file in references/, read only if the agent opens them
~5.1k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from google/skills at commit 4b940dd, republished under its Apache-2.0 licence (© google). 1,179 words, ~3,481 tokens.

Download SKILL.mdSave it as .claude/skills/gke-ai-troubleshooting-tpu-performance-degradation/SKILL.md (or your agent's skills folder). This skill also uses 4 other files; get the full folder from GitHub.
name
gke-ai-troubleshooting-tpu-performance-degradation
description
Diagnose GKE Cloud TPU training throughput drops and step-time regressions (15%+ TPU duty-cycle drop) using ML Diagnostics Workload Monitoring (`gcloud alpha mldiagnostics monitored-events` / `hypercomputecluster.googleapis.com/v1alpha`) and 1-minute Cloud Monitoring system metrics (`kubernetes.io/node/accelerator/*`). Distinguishes hardware and network fabric throttling from workload resource bottlenecks (HBM capacity, host memory, or host CPU saturation). Use when TPU training throughput or duty cycle drops without crashing pods, when `PERFORMANCE_DEGRADATION` monitored events fire, or when triaging slow multi-slice training steps. Don't use for complete multi-slice XLA execution stalls with `HANG_DETECTED` logs (use gke-ai-troubleshooting-tpu-mxla-hang) or pod eviction/interruption restarts (use gke-ai-troubleshooting-jobset-interruption).
metadata.version
1.0.0
metadata.category
AiAndMachineLearning

Troubleshoot GKE TPU performance degradation with ML Diagnostics Workload Monitoring

Diagnose and mitigate Cloud TPU training throughput drops and step-time regressions (15%+ drop in TPU duty cycle) on Google Kubernetes Engine (GKE) by correlating ML Diagnostics Workload Monitoring MonitoredEvent analyzer reports with 1-minute Cloud Monitoring system metrics and GKE node topology labels.

Prerequisites

  • Tools: Install the Google Cloud SDK (gcloud with alpha component for gcloud alpha mldiagnostics) and kubectl.
  • Cloud Billing & Project Configuration: Verify an active billing account is linked (gcloud billing projects describe {project_id}), authenticate (gcloud auth login), set the target project (gcloud config set project {project_id}), and ensure container.googleapis.com, monitoring.googleapis.com, and hypercomputecluster.googleapis.com are enabled.
  • Supported workloads and GKE versions: Google Cloud ML Diagnostics only supports JAX on TPUs (see ML Diagnostics platform). Workload Monitoring is enabled by default, supports the jobset and job GKE job types, and is compatible with GKE versions 1.36.0-gke.4681000 and later, as stated in Configure GKE for ML Diagnostics. If a workload uses another framework (such as PyTorch) or another custom resource type, gcloud alpha mldiagnostics won't list ML runs or monitored events for it. On-demand profiling (Step 4 Path A) additionally requires the cluster setup described in that document.
  • Required IAM Roles:
    • Cluster Director Editor (roles/hypercomputecluster.editor), the role listed in the "IAM permissions" section of ML Diagnostics platform, for the ML Diagnostics CLI and API calls in this skill (ML runs, monitored events, and on-demand profiler sessions)
    • Monitoring Viewer (roles/monitoring.viewer) for the PromQL queries in Step 2
    • Kubernetes Engine Viewer (roles/container.viewer) for the kubectl get nodes query in Step 3
    • For remediation ([High Risk] steps): Kubernetes Engine Cluster Admin (roles/container.clusterAdmin)
  • Reference Documentation:

Read-only rule: Run read-only diagnostic commands only. Never drain, delete, or re-create nodes, or run any other command that changes the cluster. Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

Analyzer routing overview

By default, Workload Monitoring treats a 15% drop in TPU duty cycle as a performance degradation, raises a MonitoredEvent (type: PERFORMANCE_DEGRADATION), and runs its analyzers. Consult the Workload Monitoring and analyzers section in the official Cloud TPU documentation for the complete analyzer definitions, detection criteria, and subsections, and route remediation by category:

  • Category A: workload resource bottlenecks (never cordon or replace nodes):
  • Category B: infrastructure and network fabric throttling (culprit nodes):
  • Category C: PERFORMANCE_DEGRADATION event fired, but no analyzer reports detectionState: "DETECTED":

Diagnostic workflow

Show full SKILL.md (504 more words)Show less
Step 0: Collect context and set the investigation window [Low Risk]

Collect the target parameters. By default, query a 60-minute window `[T - 30m, T

  • 30m]around{issue_time}`:
  • {project_id}: Google Cloud project ID
  • {location}: Google Cloud region where the ML run and GKE cluster reside (for example, us-central1)
  • {cluster_name}: GKE cluster name
  • {workload_name} / {ml_run_id}: JobSet / workload name or ML Diagnostics run ID
  • {issue_time}: Timestamp when throughput degradation was observed (T, ISO-8601 UTC)
  • {start_time}: T - 30m
  • {end_time}: T + 30m
Step 1: Query ML runs and monitoredEvents [Low Risk]
  1. List active or recent ML runs: Give the user gcloud alpha mldiagnostics machine-learning-run list with a link to List machine learning runs in the ML Diagnostics CLI reference, and locate {ml_run_id} matching {workload_name}.
  2. List performance degradation events: Give the user gcloud alpha mldiagnostics monitored-events list with a link to Monitored-events commands, or the hypercomputecluster.googleapis.com/v1alpha API with a link to Access Workload Monitoring information through the API, to check for PERFORMANCE_DEGRADATION events during [{start_time}, {end_time}].
  3. Describe the MonitoredEvent: Give the user gcloud alpha mldiagnostics monitored-events describe with a link to the same Monitored-events commands section, and inspect the analyzerReports array (analyzer, detectionState, details, and recommendedActions).
    • If a PERFORMANCE_DEGRADATION event fired, run Step 2 to corroborate with the 1-minute system metrics; if none of its analyzerReports entries has detectionState: "DETECTED", follow Path C: Event fired with no DETECTED analyzer.
    • If no PERFORMANCE_DEGRADATION event exists and duty cycle is steady in Step 2, rule out TPU performance degradation by following Path D: Healthy telemetry.
Step 2: Correlate with 1-minute Cloud Monitoring system metrics [Low Risk]

Consult the System Metrics section in the Workload Monitoring documentation for the 1-minute Cloud Monitoring metrics exported for TPU and host devices, and run read-only PromQL queries over [{start_time}, {end_time}] to corroborate the analyzer report:

promql
# 1. Node TPU duty cycle (look for the drop on the affected nodes)
kubernetes_io:node_accelerator_duty_cycle{
  monitored_resource="k8s_node",
  project_id="{project_id}",
  cluster_name="{cluster_name}"
}

# 2. HBM utilization ratio by node (around 0.90 is approaching the limit)
sum by (node_name) (
  kubernetes_io:node_accelerator_memory_used{
    monitored_resource="k8s_node",
    project_id="{project_id}",
    cluster_name="{cluster_name}"
  }
)
/
sum by (node_name) (
  kubernetes_io:node_accelerator_memory_total{
    monitored_resource="k8s_node",
    project_id="{project_id}",
    cluster_name="{cluster_name}"
  }
)

# 3. Host memory and CPU allocatable utilization
kubernetes_io:node_memory_allocatable_utilization{
  monitored_resource="k8s_node",
  project_id="{project_id}",
  cluster_name="{cluster_name}"
}

kubernetes_io:node_cpu_allocatable_utilization{
  monitored_resource="k8s_node",
  project_id="{project_id}",
  cluster_name="{cluster_name}"
}
Step 3: Map culprit instance IDs to GKE nodes and topology [Low Risk]

When an infrastructure analyzer reports culprit numeric Compute Engine instance IDs in details or recommendedActions, map those numeric instance IDs to GKE Node names and physical topology blocks using this read-only kubectl query inspecting container.googleapis.com/instance_id:

bash
kubectl get nodes -l cloud.google.com/gke-tpu-accelerator \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\tinstance_id="}{.metadata.annotations.container\.googleapis\.com/instance_id}{"\tblock="}{.metadata.labels.cloud\.google\.com/gce-topology-block}{"\tsubblock="}{.metadata.labels.cloud\.google\.com/gce-topology-subblock}{"\thost="}{.metadata.labels.cloud\.google\.com/gce-topology-host}{"\n"}{end}'
Step 4: Remediation by analyzer category

Load and follow only the reference file that matches the analyzer category from Step 1:

Guardrails

  1. Never change GKE-managed instance groups or VMs through Compute Engine: Don't run gcloud compute instance-groups managed commands, such as delete, on a node pool's managed instance group. Handle nodes through GKE, as described in Path B: Infrastructure or network fabric throttling.
  2. Never cordon nodes for workload resource saturation: If only the HBM capacity, host memory, or host CPU utilization analyzers detected an issue, don't cordon or replace nodes.

© google, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 4 other files (references) in skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation of google/skills.

  • SKILL.md
  • references/path-a-workload-bottlenecks.md
  • references/path-b-infrastructure-throttling.md
  • references/path-c-no-detected-analyzer.md
  • references/path-d-healthy-telemetry.md

Open the folder on GitHubat commit 4b940dd

Compare with similar skills

Gke AI Troubleshooting Tpu Performance Degradation next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Gke AI Troubleshooting Tpu Performance Degradation compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Gke AI Troubleshooting Tpu Performance Degradation this skillgoogle/skills21k—~3.5kAutomated safety check: PassApache-2.0
Devopsnicepkg/auto-company1952 repos~814Automated safety check: PassMIT
Kcli Cluster Deploymentkarmab/kcli653—~1.5kAutomated safety check: PassApache-2.0
GCP Gkesickn33/agentic-awesome-skills47k2 repos~2.5kAutomated safety check: PassMIT
Apex Azure Cloud Migratejonathan-vella/apex217—~1.2kAutomated safety check: PassMIT
Dt Obs GCPDynatrace/dynatrace-for-ai163—~2.5kAutomated safety check: PassApache-2.0

Similar skills

  • Devops

    nicepkg/auto-company

    Deploy to Cloudflare (Workers, R2, D1), Docker, GCP (Cloud Run, GKE), Kubernetes (kubectl, Helm).

    195 GitHub starsUsed in 2 repos~814 tokens
    DevOps & CloudAuto-check passed

  • Kcli Cluster Deployment

    karmab/kcli

    Guides deployment and management of Kubernetes clusters with kcli.

    653 GitHub stars~1.5k tokensUpdated 2 days ago
    DevOps & CloudAuto-check passed

  • GCP Gke

    sickn33/agentic-awesome-skills

    Deploy and manage Google Kubernetes Engine clusters. An agent skill from sickn33/agentic-awesome-skills.

    47k GitHub starsUsed in 2 repos~2.5k tokens
    DevOps & CloudAuto-check passed

  • Apex Azure Cloud Migrate

    jonathan-vella/apex

    WORKFLOW SKILL — Assess and migrate cross-cloud workloads to Azure: assessments and code conversion from AWS, GCP, Heroku, Kubernetes or Spring.

    217 GitHub stars~1.2k tokensUpdated yesterday
    DevOps & CloudAuto-check passed

  • Dt Obs GCP

    Dynatrace/dynatrace-for-ai

    GCP cloud resources including Compute Engine, GKE, Cloud Run, Pub/Sub, VPC networking, DNS, IAM, Secret Manager, and monitoring.

    163 GitHub stars~2.5k tokensUpdated 10 days ago
    DevOps & CloudAuto-check passed

  • Mirrord Operator

    metalbear-co/mirrord

    Help users install and configure the mirrord Operator for team/enterprise environments.

    5.4k GitHub starsUsed in 1 repo~4.6k tokens
    DevOps & CloudAuto-check passed

More from google/skills

All 150 skills in this repo

  • Cloud Trace Querying

    google/skills

    Official

    Query Cloud Trace spans, filter by latency thresholds or error status, correlate distributed traces with Cloud Logging, and diagnose latency bottlenecks across Google Cloud services.

    21k GitHub stars~1.7k tokensUpdated yesterday
    Auto-check passed

  • Google Cloud PAM Helper

    google/skills

    Official

    Manages Google Cloud Privileged Access Manager entitlements and grants: create and edit entitlements, request temporary access, and approve or deny pending grants.

    21k GitHub stars~3.2k tokensUpdated yesterday
    Auto-check passed

  • Agent Platform Alert Configuration

    google/skills

    Official

    Writes Terraform alerting policies for AI agents that emit OpenTelemetry metrics, covering reliability, cost, safety, security and quality signals on Google Cloud.

    21k GitHub stars~4.2k tokensUpdated yesterday
    Auto-check passed

  • Model Garden Deployment

    google/skills

    Official

    Deploys open models or custom weights from Model Garden to Agent Platform endpoints, checks deployment status and cleans up endpoints, confirming before any change.

    21k GitHub stars~5k tokensUpdated yesterday
    Auto-check passed

  • Agent Platform Skill Registry

    google/skills

    Official

    Searches, manages and scaffolds skills in the Gemini Enterprise Agent Platform Skill Registry using bundled Python scripts and Google Cloud credentials.

    21k GitHub stars~584 tokensUpdated yesterday
    Auto-check passed

  • GCP Application Design Center Deploy

    google/skills

    Official

    Designs GCP infrastructure as local Terraform, validates and scans it against best practices, then imports it to Application Design Center for deployment and troubleshooting.

    21k GitHub stars~4.4k tokensUpdated yesterday
    Auto-check passed

Works with

Categories

Questions about Gke AI Troubleshooting Tpu Performance Degradation

What does Gke AI Troubleshooting Tpu Performance Degradation do?

Diagnose GKE Cloud TPU training throughput drops and step-time regressions (15%+ TPU duty-cycle drop) using ML Diagnostics Workload Monitoring (gcloud alpha mldiagnostics monitored-events /…. Gke AI Troubleshooting Tpu Performance Degradation is an agent skill from google/skills, published by the product's own GitHub organization.io/node/accelerator/).

When should I use Gke AI Troubleshooting Tpu Performance Degradation?

Gke AI Troubleshooting Tpu Performance Degradation fits situations like: TPU training throughput; duty cycle drops without crashing pods; PERFORMANCEDEGRADATION monitored events fire; triaging slow multi-slice training steps.

How do I install Gke AI Troubleshooting Tpu Performance Degradation in Claude Code?

Run `npx skills add google/skills --skill gke-ai-troubleshooting-tpu-performance-degradation -a claude-code`. Or copy the skill folder (skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation in google/skills) into .claude/skills/gke-ai-troubleshooting-tpu-performance-degradation in your project. Claude Code loads it when a task matches its description.

How do I install Gke AI Troubleshooting Tpu Performance Degradation in Codex?

Run `npx skills add google/skills --skill gke-ai-troubleshooting-tpu-performance-degradation -a codex`. Or copy the skill folder (skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation in google/skills) into .agents/skills/gke-ai-troubleshooting-tpu-performance-degradation in your project. Codex loads it when a task matches its description.

Can I use Gke AI Troubleshooting Tpu Performance Degradation in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add google/skills --skill gke-ai-troubleshooting-tpu-performance-degradation -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/gke-ai-troubleshooting-tpu-performance-degradation, .gemini/skills/gke-ai-troubleshooting-tpu-performance-degradation, .github/skills/gke-ai-troubleshooting-tpu-performance-degradation and .opencode/skills/gke-ai-troubleshooting-tpu-performance-degradation in your project.

What does Gke AI Troubleshooting Tpu Performance Degradation need to run?

Going by SKILL.md and its folder, Gke AI Troubleshooting Tpu Performance Degradation needs the command-line tools its instructions call (gcloud and kubectl).

Does Gke AI Troubleshooting Tpu Performance Degradation access the network?

SKILL.md names 2 domains. As links in the text: docs.cloud.google.com and cloud.google.com. This is read from the text; nothing was executed.

Is Gke AI Troubleshooting Tpu Performance Degradation safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Gke AI Troubleshooting Tpu Performance Degradation use?

Gke AI Troubleshooting Tpu Performance Degradation is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Gke AI Troubleshooting Tpu Performance Degradation use?

About 3.5k tokens (SKILL.md is roughly 14k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 1.6k tokens, read only when the agent opens those files.

What are the alternatives to Gke AI Troubleshooting Tpu Performance Degradation?

Skills that share tags, products or a category with Gke AI Troubleshooting Tpu Performance Degradation: Devops (nicepkg/auto-company, 195 stars), Kcli Cluster Deployment (karmab/kcli, 653 stars), GCP Gke (sickn33/agentic-awesome-skills, 47k stars) and Apex Azure Cloud Migrate (jonathan-vella/apex, 217 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Gke AI Troubleshooting Tpu Performance Degradation?

google (a GitHub organization, an official publisher) maintains it in google/skills, which has 21,097 GitHub stars. The repository holds 150 skills in this directory. The repository was last updated on October 9, 2026.

Source: google/skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.