Agent skill

Kubernetes Troubleshooting with Inspektor Gadget

by inspektor-gadget in inspektor-gadget/inspektor-gadget

Traces what the kernel is doing for a misbehaving pod using Inspektor Gadget's eBPF tools, tagged with namespace, pod, container and node, without changing workloads.

Apache-2.0Auto-check passedDevOps & Cloud

Install Kubernetes Troubleshooting with Inspektor Gadget

skills CLI
$ npx skills add inspektor-gadget/inspektor-gadget --skill kubernetes-troubleshooting -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install inspektor-gadget/inspektor-gadget kubernetes-troubleshooting --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/inspektor-gadget/inspektor-gadget.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/kubernetes-troubleshooting .claude/skills/kubernetes-troubleshooting && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
kubernetes-troubleshooting
GitHub stars
2.9k
Token cost
~2.3k tokens
SKILL.md length
882 words
Files
11 (incl. references)
Skills in repo
2
Repo updated
First seen
Licence
Apache-2.0

At a glance

Traces what the kernel is doing for a misbehaving pod using Inspektor Gadget's eBPF tools, tagged with namespace, pod, container and node, without changing workloads.

  • Works in 4 steps: Route. Map the symptom to a domain… → Discover. Run :latest -h to read the… → Run bounded. Always scope and time-box → …
  • Pods failing with DNS errors, connection resets or refused connections
  • SKILL.md covers Prerequisite: confirm IG is…, The one rule: discover, don't…, The loop (repeat until root… and Symptom → first gadget…, plus 2 more sections
  • Calls kubectl and jq

What it does

When application logs cannot explain a failure, this skill drives `kubectl gadget run` to watch live kernel events such as syscalls, network packets, DNS lookups, capability checks and OOM kills. Each event is labeled with its Kubernetes namespace, pod, container and node, so a symptom like a silently failing file open or a reset connection can be traced to the workload behind it. Everything it does is read-only.

The agent first checks that the gadget plugin and the in-cluster agent are present and, if not, follows the install reference and asks the operator before deploying a privileged DaemonSet. It never relies on remembered flags: each gadget's own help output is read at run time. Symptoms are routed to a domain (networking, security, process lifecycle, storage and filesystem, or performance), then to a candidate gadget, run in a bounded and scoped way and repeated until the root cause shows up. Reference files cover each domain, a gadget catalog and common flags.

When your agent uses it

  • Pods failing with DNS errors, connection resets or refused connections
  • Finding out which pod or process opened a file, made a syscall or ran a command
  • Diagnosing CrashLoopBackOff, OOMKilled or permission and seccomp denials
  • Looking into slow disk or file I/O in a running workload

Example prompts

  • “The checkout pod gets connection resets to the database; trace the TCP drops with Inspektor Gadget.”
  • “Which pod is making the DNS queries that fail in the payments namespace?”
  • “My container hits no such file or directory at startup but logs nothing; trace its file opens.”

Requirements

  • The kubectl gadget plugin installed locally
  • An Inspektor Gadget DaemonSet in the cluster (namespace gadget)
  • Troubleshooting RBAC and nodes with BTF for CO-RE
  • Compatibility (from SKILL.md): Requires the `kubectl gadget` plugin locally and an in-cluster Inspektor Gadget DaemonSet (`kubectl gadget deploy`, namespace `gadget`), on a cluster where you hold troubleshooting RBAC and nodes with BTF for CO-RE. Read-only. If the plugin or DaemonSet is missing, see references/install.md — ask the operator before installing.

Workflow steps

4 steps, taken from the first numbered list in SKILL.md.

  1. Route. Map the symptom to a domain (networking / security /
  2. Discover. Run :latest -h to read the real flags and fields.
  3. Run bounded. Always scope and time-box
  4. Read the columns. Inspect the enriched fields (k8s.*, proc.*, error

What it can do on your machine

Read from SKILL.md and the folder at commit b6dce71. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Shell commands in SKILL.md call:

    • kubectl
    • jq

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md. Its commands use kubectl, which can reach the network depending on how they are called.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

  • Compatibility

    Requires the `kubectl gadget` plugin locally and an in-cluster Inspektor Gadget DaemonSet (`kubectl gadget deploy`, namespace `gadget`), on a cluster where you hold troubleshooting RBAC and nodes with BTF for CO-RE. Read-only. If the plugin or DaemonSet is missing, see references/install.md — ask the operator before installing.

    From compatibility in the SKILL.md frontmatter.

Context cost

Kubernetes Troubleshooting with Inspektor Gadget loads about 2.3k tokens when it runs, and up to ~15k if it reads all its reference files. Until then it costs about 187 tokens; SKILL.md has 882 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~187
When it runs · the whole SKILL.md, loaded when a task matches
~2.3k
With references · SKILL.md plus every file in references/, read only if the agent opens them
~15k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from inspektor-gadget/inspektor-gadget at commit b6dce71, republished under its Apache-2.0 licence (© inspektor-gadget). 882 words, ~2,258 tokens.

Download SKILL.mdSave it as .claude/skills/kubernetes-troubleshooting/SKILL.md (or your agent's skills folder). This skill also uses 10 other files; get the full folder from GitHub.
name
kubernetes-troubleshooting
description
Debug live Kubernetes workloads at the kernel level with Inspektor Gadget via `kubectl gadget run`. Use when a pod or Service is misbehaving and application logs are not enough: DNS resolution failures or slow lookups, TCP connection resets / retransmits / drops / refused connections, CrashLoopBackOff, "no such file or directory", permission / capability / seccomp / LSM denials, OOMKilled, unexpected process execs, slow disk or file I/O, or "which pod/process made this syscall / connection / DNS query". Traces real kernel events with eBPF, auto-enriched with pod / namespace / container / node. Read-only, no workload changes. Not for plain application logs, long-term metrics dashboards, or editing cluster state.
compatibility
Requires the `kubectl gadget` plugin locally and an in-cluster Inspektor Gadget DaemonSet (`kubectl gadget deploy`, namespace `gadget`), on a cluster where you hold troubleshooting RBAC and nodes with BTF for CO-RE. Read-only. If the plugin or DaemonSet is missing, see references/install.md — ask the operator before installing.

Kubernetes troubleshooting with Inspektor Gadget

Inspektor Gadget (IG) runs eBPF programs in the kernel to trace live events — syscalls, network packets, DNS, capability checks, OOM kills — and enriches every event with Kubernetes metadata (namespace, pod, container, node). You drive it with kubectl gadget run <gadget>:latest. Each gadget is an OCI image pulled on demand; there is no built-in list of gadgets to memorize and no list-gadgets command.

Use IG when logs/metrics can't answer "what is the kernel actually doing for this workload right now?" — e.g. a request times out but the app logs nothing, a file open fails silently, a connection is reset before the app sees it.

Prerequisite: confirm IG is available (check first, then route)

This skill drives kubectl gadget. Confirm it's usable before the loop:

bash
command -v kubectl-gadget >/dev/null 2>&1 || echo "kubectl gadget plugin MISSING"
kubectl gadget version    # "Server version: not available" = inspect the gadget namespace

If the plugin is missing or the server version is unavailable, open references/install.md and follow it — don't guess an install command. Installing the plugin and running kubectl gadget deploy need cluster rights and start a privileged DaemonSet, so ask the operator before installing.

The one rule: discover, don't guess

Never hardcode a gadget's flags or field names from memory. Gadget images evolve and new gadgets ship. Always enumerate the real interface at run time:

bash
kubectl gadget run <gadget>:latest -h          # flags + a "--fields" block listing every data source & field
kubectl gadget run <gadget>:latest -A --timeout 5 -o json \
  | jq -s '(.[0] // {}) | if type == "array" then (.[0] // {}) else . end | keys'

The gadget's own -h/--fields output is the source of truth, so this skill never drifts from the shipped gadgets. If a symptom below names a gadget, still confirm its flags/fields with -h before relying on them. See references/discovering-params-and-fields.md.

The loop (repeat until root cause)

  1. Route. Map the symptom to a domain (networking / security / process-lifecycle / storage-fs / performance), then to a candidate gadget using the table below and the matching references/domain-*.md.
  2. Discover. Run <gadget>:latest -h to read the real flags and fields.
  3. Run bounded. Always scope and time-box: kubectl gadget run <gadget>:latest -n <ns> --timeout <sec> -o json (-n/-p/-c to filter; always set --timeout for streaming gadgets; --max-entries for top/snapshot). See references/common-flags.md.
  4. Read the columns. Inspect the enriched fields (k8s.*, proc.*, error codes) to confirm or refute a hypothesis, then narrow scope and repeat.

Symptom → first gadget (confirm flags/fields with -h)

Symptom (what the user reports)DomainStart withThen / disambiguate
DNS fails, slow, or NXDOMAINnetworkingtrace_dnslatency in latency_ns; see domain-networking
Connection reset / refused / hangsnetworkingtrace_tcptrace_tcpretrans (retransmits), trace_tcpdrop (kernel drops)
Packet loss / high latency between podsnetworkingtrace_tcpdropprofile_tcprtt, top_tcp, tcpdump
TLS/cert/SNI routing issuenetworkingtrace_snitrace_ssl (plaintext at TLS boundary)
Port bind fails / "address already in use"networkingsnapshot_sockettrace_bind (the failing bind + errno)
CrashLoopBackOff / unexpected restartsprocess-lifecycletrace_exectrace_signal (filter --signal 9/15 — Go SIGURG / glibc SIGRTMIN async-preempt noise), trace_oomkill
Killed / OOMKilledprocess-lifecycletrace_oomkilltop_process, profile_cpu
Container died too fast to trace live (post-mortem)process-lifecycletraceloopreplays recent syscalls from the ring buffer; empty if it wasn't already recording
Watch an interactive shell / tty / pts / keystrokesprocess-lifecyclettysnoopno --tty/--pts selector — scope by --pid/--comm/pod
"no such file" / missing config / wrong pathstorage-fstrace_opensnapshot_file, top_file
Slow disk / file I/Ostorage-fstrace_fsslowerprofile_blockio, top_blockio
Unexpected mount/umount, or suspicious hardlink/symlink (escape)storage-fstrace_mounttrace_link (type = HARDLINK/SYMLINK cuts the noise)
fd leak / "too many open files" / inotify watch stormstorage-fsfdpassfsnotify; snapshot_file (unclosed fds in one proc)
Permission denied despite correct RBAC/FSsecuritytrace_capabilitiesaudit_seccomp; node audit logs for AppArmor/SELinux
Seccomp denialsecurityaudit_seccompthe code + syscall identify the seccomp action; advise_seccomp to author a profile
AppArmor/SELinux denialsecuritynode audit logstrace_lsm can correlate hook activity, but does not expose another LSM's verdict
Kernel module loaded / rootkit / unexpected insmodsecuritytrace_init_moduletrace_capabilities (CAP_SYS_MODULE)
Harden / author a seccomp or NetworkPolicy profilesecurityadvise_seccompadvise_networkpolicy (policy from observed traffic)
High CPU, or slow despite low CPU% (CFS throttling / cgroup CPU limit)performanceprofile_cputop_cpu_throttle (capped?), top_process
Memory growth / leak (userspace)performancetrace_malloc (libc malloc only — statically-linked Go runtime allocator invisible)trace_malloc --collect-ustack (the leak site)
App hung / mutex deadlockperformancedeadlock (pthread only — Go sync.Mutex invisible)profile_cpu
GPU / CUDA out-of-memoryperformancetop_cuda_memoryper-proc device vs pinned; profile_cuda = libcuda Driver-API alloc/free
Quantify eBPF/gadget CPU or memory overhead (self-profiling)metabpfstatsneeds an active window — use a longer --timeout
Show full SKILL.md (209 more words)Show less

This is a deliberately thin shortlist, not the catalog — it names the few gadgets that fit the most common symptoms so you don't scan the full bundled set, keeping this always-loaded router small. Symptom not listed here, or unsure which row fits? Open references/gadget-catalog.md — it groups all bundled gadgets by domain with the disambiguation reasoning. The authoritative live list is whatever kubectl gadget run <name>:latest -h resolves; always confirm a gadget's flags/fields there before relying on them.

References (load only the one you need)

  • references/gadget-catalog.md — all upstream gadgets grouped by domain, one line each.
  • references/discovering-params-and-fields.md — the discover-don't-guess mechanics.
  • references/common-flags.md — filters, output modes, timeouts, gotchas.
  • references/domain-networking.md — DNS / TCP / drops / retransmits / TLS-SNI / NetworkPolicy.
  • references/domain-security.md — capabilities / LSM / seccomp / module loading.
  • references/domain-process-lifecycle.md — exec / signals / OOM / snapshots / syscall recording.
  • references/domain-storage-fs.md — open / slow FS / block I/O / mounts / fd / fsnotify.
  • references/domain-performance.md — CPU / throttle / RTT / deadlock / malloc / GPU.
  • references/install.md — detect whether IG is usable; install the kubectl gadget plugin and deploy the DaemonSet if missing.

Safety

Observation is read-only — gadgets trace, they never modify workloads. They require a privileged in-cluster IG (DaemonSet) or kubectl gadget, so use them in clusters where you have troubleshooting rights. Always bound streaming gadgets with --timeout and cap top/snapshot with --max-entries so a trace can't flood your context or the API server.

© inspektor-gadget, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 10 other files (references) in skills/kubernetes-troubleshooting of inspektor-gadget/inspektor-gadget.

  • SKILL.md
  • references/common-flags.md
  • references/discovering-params-and-fields.md
  • references/domain-networking.md
  • references/domain-performance.md
  • references/domain-process-lifecycle.md
  • references/domain-security.md
  • references/domain-storage-fs.md
  • references/gadget-catalog.md
  • references/install.md
  • references/linux-companion.md

Open the folder on GitHubat commit b6dce71

Compare with similar skills

Kubernetes Troubleshooting with Inspektor Gadget next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Kubernetes Troubleshooting with Inspektor Gadget compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Kubernetes Troubleshooting with Inspektor Gadget this skillinspektor-gadget/inspektor-gadget2.9k—~2.3kAutomated safety check: PassApache-2.0
Debugging Techniquesancoleman/ai-design-components525—~3.3kAutomated safety check: PassMIT
Kubeshark Installerkubeshark/kubeshark12k—~3.6kAutomated safety check: NotesApache-2.0
Kubeshark KFL2 Filter Referencekubeshark/kubeshark12k—~3.6kAutomated safety check: PassApache-2.0
KubeSphere ServiceMesh Managerkubesphere/kubesphere17k—~2.4kAutomated safety check: PassCustom licence
Kubernetes Network Root Cause Analysiskubeshark/kubeshark12k—~5.3kAutomated safety check: PassApache-2.0

Similar skills

  • Debugging Techniques

    ancoleman/ai-design-components

    Debugging workflows for Python (pdb, debugpy), Go (delve), Rust (lldb), and Node.js, including container debugging (kubectl debug, ephemeral containers) and production-safe debugging techniques with…

    525 GitHub stars~3.3k tokensUpdated 10 mo ago
    DevelopmentAuto-check passed
  • Kubeshark Installer

    kubeshark/kubeshark

    Installs and configures Kubeshark on a Kubernetes cluster, choosing between the quick CLI path and a Helm install with custom values.

    12k GitHub stars~3.6k tokensUpdated yesterday
    DevOps & CloudAuto-check: notes
  • Syntax reference for KFL2, the CEL-based display filter language used to search Kubernetes network traffic captured by Kubeshark, loaded before any filter is written.

    12k GitHub stars~3.6k tokensUpdated yesterday
    DevOps & CloudAuto-check passed
  • KubeSphere ServiceMesh Manager

    kubesphere/kubesphere

    Installs, checks and troubleshoots the KubeSphere ServiceMesh extension (Istio, Kiali, Jaeger), including grayscale release, sidecar injection, topology and tracing issues.

    17k GitHub stars~2.4k tokensUpdated 2 mo ago
    DevOps & CloudAuto-check passed
  • Investigates past Kubernetes incidents from Kubeshark traffic snapshots: takes captures, dissects API calls, extracts PCAPs and compares traffic over time.

    12k GitHub stars~5.3k tokensUpdated yesterday
    DevOps & CloudAuto-check passed
  • Runs the lifecycle of FrontendExtension resources in a Kubernetes cluster: create, rebuild, package, publish, unpublish, delete and debug stuck states.

    17k GitHub stars~3.2k tokensUpdated 2 mo ago
    DevOps & CloudAuto-check passed

More from inspektor-gadget/inspektor-gadget

  • Linux Troubleshooting with Inspektor Gadget

    inspektor-gadget/inspektor-gadget

    Debugs a single Linux host or container runtime at the kernel level with the standalone ig binary and eBPF gadgets, read-only and without Kubernetes.

    2.9k GitHub stars~2.4k tokensUpdated yesterday
    Auto-check: notes

Works with

Categories

Questions about Kubernetes Troubleshooting with Inspektor Gadget

What does Kubernetes Troubleshooting with Inspektor Gadget do?

Traces what the kernel is doing for a misbehaving pod using Inspektor Gadget's eBPF tools, tagged with namespace, pod, container and node, without changing workloads. When application logs cannot explain a failure, this skill drives `kubectl gadget run` to watch live kernel events such as syscalls, network packets, DNS lookups, capability checks and OOM kills. Each event is labeled with its Kubernetes namespace, pod, container and node, so a symptom like a silently failing file open or a reset connection can be traced to the workload behind it.

When should I use Kubernetes Troubleshooting with Inspektor Gadget?

Kubernetes Troubleshooting with Inspektor Gadget fits situations like: pods failing with DNS errors, connection resets or refused connections; finding out which pod or process opened a file, made a syscall or ran a command; diagnosing CrashLoopBackOff, OOMKilled or permission and seccomp denials; looking into slow disk or file I/O in a running workload.

How do I install Kubernetes Troubleshooting with Inspektor Gadget in Claude Code?

Run `npx skills add inspektor-gadget/inspektor-gadget --skill kubernetes-troubleshooting -a claude-code`. Or copy the skill folder (skills/kubernetes-troubleshooting in inspektor-gadget/inspektor-gadget) into .claude/skills/kubernetes-troubleshooting in your project. Claude Code loads it when a task matches its description.

How do I install Kubernetes Troubleshooting with Inspektor Gadget in Codex?

Run `npx skills add inspektor-gadget/inspektor-gadget --skill kubernetes-troubleshooting -a codex`. Or copy the skill folder (skills/kubernetes-troubleshooting in inspektor-gadget/inspektor-gadget) into .agents/skills/kubernetes-troubleshooting in your project. Codex loads it when a task matches its description.

Can I use Kubernetes Troubleshooting with Inspektor Gadget in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add inspektor-gadget/inspektor-gadget --skill kubernetes-troubleshooting -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/kubernetes-troubleshooting, .gemini/skills/kubernetes-troubleshooting, .github/skills/kubernetes-troubleshooting and .opencode/skills/kubernetes-troubleshooting in your project.

What does Kubernetes Troubleshooting with Inspektor Gadget need to run?

Going by SKILL.md and its folder, Kubernetes Troubleshooting with Inspektor Gadget needs the command-line tools its instructions call (kubectl and jq). Our summary lists: The kubectl gadget plugin installed locally; An Inspektor Gadget DaemonSet in the cluster (namespace gadget); Troubleshooting RBAC and nodes with BTF for CO-RE. Compatibility (from SKILL.md): Requires the `kubectl gadget` plugin locally and an in-cluster Inspektor Gadget DaemonSet (`kubectl gadget deploy`, namespace `gadget`), on a cluster where you hold troubleshooting RBAC and nodes with BTF for CO-RE. Read-only. If the plugin or DaemonSet is missing, see references/install.md — ask the operator before installing..

Does Kubernetes Troubleshooting with Inspektor Gadget access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Kubernetes Troubleshooting with Inspektor Gadget safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Kubernetes Troubleshooting with Inspektor Gadget use?

Kubernetes Troubleshooting with Inspektor Gadget is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Kubernetes Troubleshooting with Inspektor Gadget use?

About 2.3k tokens (SKILL.md is roughly 9k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 12k tokens, read only when the agent opens those files.

What are the alternatives to Kubernetes Troubleshooting with Inspektor Gadget?

Skills that share tags, products or a category with Kubernetes Troubleshooting with Inspektor Gadget: Debugging Techniques (ancoleman/ai-design-components, 525 stars), Kubeshark Installer (kubeshark/kubeshark, 12k stars), Kubeshark KFL2 Filter Reference (kubeshark/kubeshark, 12k stars) and KubeSphere ServiceMesh Manager (kubesphere/kubesphere, 17k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Kubernetes Troubleshooting with Inspektor Gadget?

inspektor-gadget (a GitHub organization) maintains it in inspektor-gadget/inspektor-gadget, which has 2,936 GitHub stars. The repository holds 2 skills in this directory. The repository was last updated on October 9, 2026.

Source: inspektor-gadget/inspektor-gadget on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.