Test Pyramid

Test Pyramid Analysis

Goal: anything that can be tested as a unit test should be a unit test; only what genuinely needs a real cluster should be E2E. The end state is a pyramid — many fast unit tests, few E2E tests. This skill produces an evidence-backed rebalancing report; it does NOT move tests itself unless the user asks afterward.

Phase 1 — Inventory

Enumerate both layers and count actual test functions (not just files), so the report can show the pyramid shape numerically:

• Unit tests: git ls-files '*_test.go' | grep -v '^test/e2e/' plus git ls-files 'test/e2e/framework/*_test.go' and Python unit tests across all client packages (discover with git ls-files 'clients/**' | grep '/test[s]*/unit/' — covers clients/python/, clients/integrations/deepagents/, and clients/integrations/mcp-server/). Count func Test... per package (grep -c '^func Test') and def test_ for Python.
• E2E / system tests: test/e2e/*_test.go (excluding test/e2e/framework/), plus test/e2e/extensions/ (density, python-runtime, rollout, and other cluster-physics E2E), test/e2e/clients/python/ (SDK E2E), dev/tools/test-migration.py (upgrade/rollback), and test/stress/ (load). Count func Test... (and def test_ for Python / migration tests), and for table-driven E2E, the sub-scenarios.
• Note per-layer runtime cost if discoverable (CI job durations from dev/ci/, TestGrid tab names) — the payoff argument for each migration is time and flake surface removed from presubmit.

Phase 2 — Characterize every E2E test

Read each E2E test body (fan out parallel subagents over batches of 3-5 files for speed; each returns structured notes). For every test, record:

1. What it arranges (objects applied, cluster preconditions).
2. What it asserts — split assertions into:
   • Cluster-physics assertions: pod actually scheduled/running, kubelet behavior, image pulls, real networking/routing (sandbox-router paths), LoadBalancer/Gateway, RBAC enforcement, webhook admission via real API server, CRD conversion via real storage, controller<->controller timing, upgrade/rollback state survival.
   • Logic assertions: field values on objects after a reconcile, label/annotation stamping, status conditions, owner references, name hashing, defaulting, spec conversion, error classification, requeue decisions — anything a reconciler computes deterministically from inputs.
3. The seam: which function/reconciler produces each logic assertion's value (e.g. isAdoptable, computeAndSetStatus, merge_flaky_by_test). If you cannot name the seam, you cannot claim a unit test can cover it.

Phase 3 — Classify

• E2E → unit candidate: every logic assertion whose seam is reachable with the repo's existing unit patterns — table-driven tests with controller-runtime's fake client and newScheme(t) (see extensions/controllers/sandboxclaim_controller_test.go for the house style), direct function calls, or Python unittest.mock. A whole E2E test is a removal candidate only if ALL its assertions are logic assertions; otherwise propose extracting the logic assertions to unit tests and thinning the E2E to its cluster-physics core.
• Keep as E2E: tests dominated by cluster-physics assertions. Do not propose unit-testing scheduling, pod readiness, real router traffic, webhook round-trips, or migration/upgrade behavior — a fake client proves nothing there.
• Unit → E2E promotion (rare, keep the pyramid in mind): unit tests that mock so much they only test the mock (assert the seam is meaningfully exercised), or critical user journeys (create claim → adopt → route traffic → shutdown) with no E2E smoke path at all. Prefer ONE thin journey test over per-feature E2E.
• Redundant coverage: the same seam asserted at both layers with the same inputs — keep the unit test, list the E2E assertion as prunable.