Debug E2E Pipeline

Debug E2E Pipeline

When To Use

Use this skill when the user wants to debug a failing Prow e2e pipeline for cloud-provider-azure. Trigger on:

• Prow build URLs (prow.k8s.io/view/..., gcsweb.k8s.io/..., storage.googleapis.com/kubernetes-ci-logs/...)
• References to job names like cloud-provider-azure-*-capz
• Requests to "debug the e2e failure", "check why CI failed", "look at the build log", or "analyze the pipeline"

Inputs

Evidence Discipline

Follow these rules when analyzing failures and presenting findings.

• Distinguish facts from inferences. A fact is something directly visible in a log line, timestamp, or artifact. An inference is a conclusion drawn from facts. Always label inferences explicitly — say "this suggests" or "I infer," never state an inference as though it were directly observed.

• Beware single-snapshot bias. A single kubectl get nodes output or one status dump captures one moment in time. Do not treat a point-in-time observation as permanent state. Look for corroborating evidence across the full timeline: were pods scheduled to the node later? Do CAPI Machine conditions show it healthy at a later time? Do controller logs reference it as Ready?

• Require corroboration for strong claims. Before asserting a root cause, require at least two independent pieces of evidence that support the claim and verify that no available evidence contradicts it. If you only have one data point, say so.

• State what's missing. If key evidence is unavailable, say so explicitly rather than filling gaps with assumptions. Missing evidence is not confirming evidence.

• Classify confidence in findings. When presenting root cause analysis, label each finding:

  • Confirmed: supported by multiple independent pieces of evidence with no counter-evidence.
  • Likely: supported by evidence but with gaps — state what additional evidence would confirm it.
  • Uncertain: plausible hypothesis but insufficient evidence — state what you would need to investigate to confirm or rule it out.

Workflow

The workflow has two phases: investigation (delegated to a sub-agent) and review (done by you). The sub-agent gets a fresh context window dedicated to log analysis; you review its structured findings and present to the user.

Phase 1 — Investigation (sub-agent)

Delegate the investigation to a sub-agent. For multiple jobs, spawn one sub-agent per job in parallel — each gets a dedicated context for its job. Pass each sub-agent the context listed below, then tell it to follow the investigation steps that follow.

Context to pass the sub-agent:

• The pipeline URL
• The absolute skill directory path (for running the fetch script)
• The output directory path (default: _artifacts/e2e-debug/{JOB}/{BUILD}/ in the repo root)
• Any user-provided context about what to investigate
• The Evidence Discipline rules from this document
• Any specific investigation questions you have based on the user's request

Replace <SKILL_DIR> with the absolute path to this skill directory.

Step 1 — Fetch and Parse Artifacts

python3 <SKILL_DIR>/scripts/fetch_artifacts.py "<PIPELINE_URL>"

This downloads key artifacts from GCS, parses JUnit and build-log, and outputs a concise summary JSON to stdout. The full manifest is written to manifest.json inside the output directory (default: _artifacts/e2e-debug/{JOB}/{BUILD}/).

The stdout summary contains the most important fields for quick triage:

• status, duration, prow_url
• test_counts: {total, passed, failed, skipped}
• failed_tests: [{name, duration, message}]
• build_log_summary.ginkgo_summary: Ginkgo result line
• build_log_summary.ginkgo_result: "Test Suite Failed" / "Test Suite Passed" / null
• build_log_summary.node_info: raw kubectl get nodes -o wide table

If ginkgo_result is null, tests never produced results. This does not necessarily mean cluster provisioning failed — check build_log_summary.node_info and the build log to determine where the job stalled. Common causes: cluster failed to provision, tests timed out during initialization (e.g. log watcher setup), or pods were stuck Pending for hours. Only conclude "provisioning failure" if node_info is absent or shows NotReady nodes.

If the download fails (404, network error), report the error and ask the user to verify the URL.

Step 2 — Understand the Job

Prefer reading output files directly using your file-reading tools over shell commands for post-processing.

Read the full manifest.json from disk. Start with job_summary:

• job_summary.command / job_summary.args: the entrypoint script the job runs
• job_summary.extra_refs: which repos are cloned (e.g. CAPZ, windows-testing)
• job_summary.env: key environment variables — in particular CLUSTER_TEMPLATE (determines cluster topology and credential provider support), KUBERNETES_VERSION, and AZURE_LOADBALANCER_SKU
• job_summary.presets: preset labels that configure the job
• job_summary.repos: repo versions under test
• job_summary.duration: how long the job ran

This tells you what the job does — which script it runs, which cluster template it uses, and how it's configured. The cluster template determines whether the cluster has OOT credential provider, dual-stack networking, machine pools vs machine deployments, etc. To find which template was used:

1. Check job_summary.env for CLUSTER_TEMPLATE
2. If not set, search build-log.txt for "Using cluster template:"
3. The rendered resources in available_artifacts (e.g. */resources/default/KubeadmControlPlane/*.yaml) show exactly what was applied to the cluster

Check build_log_summary.node_info to see the cluster topology — OS versions, node roles, and container runtime versions. This is critical for platform mismatch issues (e.g. Windows Server 2019 images on 2022 nodes).

Also note available_artifacts — a flat list of all artifact paths available in GCS for this build. Use this as a table of contents: fetch any of these on demand via fetch_webpage on {gcs_base_url}/{path}.

Step 3 — Review Test Results

Check test_counts and tests in the manifest:

• tests.failed: each entry has name, duration, and message
• tests.passed: list of test names that succeeded
• tests.skipped: list of test names that were skipped

If test_counts.failed is 0 but the job failed, the issue is in cluster setup or the test harness, not in any individual test.