Write E2E

Produce a mergeable E2E test for a change: correct scope, right environment, assertions that survive real infrastructure, and CI wiring that runs it.

Ownership: the team owning test/new-e2e/tests/<area>/ is paged when the test fails, not @DataDog/agent-devx, which watches main and release branches and dispatches flakes back to that team. That is why the JOBOWNERS entry in Step 8 and the reliability gates in Step 9 are not optional.

References

The body carries the procedure; the references carry the lookup tables. Load a file when its trigger fires.

The repo is the source of truth; this skill is the procedure. Resolve an option by grepping its definition (grep -n '^func With' <provisioner-dir>/*.go) rather than recalling it.

Step 0 — Establish scope

Given an argument, treat it as the scope and skip to Step 1. Otherwise derive it:

git status --short
git diff --name-only $(git merge-base HEAD origin/main)

Classify each changed path:

Resolve the owning team and its <area> directory from .github/CODEOWNERS, then report the scope table before writing anything. When several behaviors are candidates and covering all of them would be a large change, ask which to cover. When nothing in the diff is user-visible, say so and stop — that is a correct answer, not a failure.

Step 1 — Gate the scope

Invoke e2e-audit with the behavior. Continue to Step 2 only if the verdict is that E2E is justified; on any other verdict, report it, name the package where the test belongs instead, and stop. Gating before Step 2 is deliberate — reading implementations for the literals a test would assert on is wasted if the behavior belongs in a unit test.

Some areas own their own template and lifecycle rules through a directory-scoped skill: ls test/new-e2e/tests/<area>/.claude/skills/*/SKILL.md. If one exists, invoke it by its name and stop; directory-scoped skills are only offered when the working directory matches, so read the file directly when it is not in the skill list. Sibling skills named elsewhere in this file live under .agents/skills/<name>/SKILL.md.

Step 2 — Extend an existing suite before creating one

Provisioning dominates E2E cost, and it is paid per suite. A Kubernetes suite blocks on cluster and agent readiness before its first assertion — tests/containers allows up to ten minutes for it, twenty when FIPS or cri-o makes the images slower to pull — and a new suite beside it pays that again for one assertion. A new test method inside an existing suite is close to free.

Read the implementation now for the exact observable it produces, because the assertion depends on the literal string rather than on the feature name: a metric name, log service, check name, or tag if the behavior ships a payload; a service name, file path, registry key, package name, or CLI output if it does not. Then look for existing coverage and a suite to join:

ls test/new-e2e/tests/<area>/
grep -rn "<metric-or-config-name>" test/new-e2e/tests/<area>/
grep -rln "e2e.BaseSuite\[environments.<Env>\]" test/new-e2e/tests/<area>/

Widen the metric grep to all of test/new-e2e/tests/ only when the area turns up nothing.

Read any AGENTS.md inside the target directory first (ls test/new-e2e/tests/<area>/AGENTS.md) — several areas carry local rules that override the defaults below.

Step 3 — Choose the environment and cloud

Ordinary Windows behavior does not need WindowsHost. Suites like tests/agent-subcommands/config_win_test.go run on plain environments.Host with ec2.WithOS(e2eos.WindowsServerDefault), which is also what the cross-OS split in references/templates.md produces — so a Windows test can extend an existing suite instead of provisioning a new one. Move up to WindowsHost only for the scenario components it carries.

Cloud: use the cloud the feature is specific to, and AWS otherwise. Windows included — the Windows CI templates depend on AWS-side MSI artifacts. Azure has a working Windows provisioner and reportedly boots Windows faster, but no test uses it and no CI job wires it, so choosing it means solving both problems yourself.

Kubernetes flavor: use kind. It provisions faster, costs less, and fails less than EKS. Reserve EKS for behavior that is specific to EKS, and expect an EKS job to run on main and nightly rather than on pull requests.

Every stock provisioner ships a fakeintake by default, and on AWS it is an ECS Fargate task you would otherwise pay for and never query. When the test asserts only on host state, drop it. Host provisioners take a ProvisionerNoFakeIntake constructor; every other provisioner spells it differently and a few cannot do it at all, so use the table in references/environments.md rather than guessing.

Reach for a custom provisioner last; test/new-e2e/codereview_guideline.md says to avoid them. Check the option surface of the stock provisioner first — most needs are already an option.

Step 4 — Place the files

Tests live in test/new-e2e/tests/<area>/. In an existing area, copy the package clause from a file already there — the usual form is the directory with its separators removed (agent-runtimes holds package agentruntimes), but not every area follows it. Read test/new-e2e/AGENTS.md §§ "Layout", "Each entry point needs its own suite type", and "Build tags" for the rest: which <area>, how a Linux/Windows pair splits across files, and why every entry point needs its own suite type — getting that last one wrong silently puts two parallel suites on one stack. references/templates.md § "Splitting one suite across operating systems" has the code for the split.

Fixtures: a YAML snippet of ten lines or fewer goes inline as a const; anything longer, and any script, goes in fixtures/ loaded with //go:embed (which needs _ "embed" in the import block).

Step 5 — Write the suite

// Unless explicitly stated otherwise all files in this repository are licensed
// under the Apache License Version 2.0.
// This product includes software developed at Datadog (https://www.datadoghq.com/).
// Copyright 2016-present Datadog, Inc.

// Package myarea contains e2e tests for <feature>.
package myarea

import (
	_ "embed"
	"testing"
	"time"

	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"

	"github.com/DataDog/datadog-agent/test/e2e-framework/components/datadog/agentparams"
	e2eos "github.com/DataDog/datadog-agent/test/e2e-framework/components/os"
	"github.com/DataDog/datadog-agent/test/e2e-framework/scenarios/aws/ec2"
	"github.com/DataDog/datadog-agent/test/e2e-framework/testing/e2e"
	"github.com/DataDog/datadog-agent/test/e2e-framework/testing/environments"
	awshost "github.com/DataDog/datadog-agent/test/e2e-framework/testing/provisioners/aws/host"
)

//go:embed fixtures/myfeature.yaml
var myFeatureConfig string

type myFeatureSuite struct {
	e2e.BaseSuite[environments.Host]
}

func TestMyFeature(t *testing.T) {
	t.Parallel()
	e2e.Run(t, &myFeatureSuite{},
		e2e.WithProvisioner(awshost.Provisioner(
			// AWS agent options nest inside WithRunOptions. The flat
			// awshost.WithAgentOptions(...) reads naturally and is what
			// Azure and GCP use, but on AWS it does not exist.
			awshost.WithRunOptions(
				ec2.WithEC2InstanceOptions(ec2.WithOS(e2eos.Ubuntu2204)),
				ec2.WithAgentOptions(
					agentparams.WithIntegration("myfeature.d", myFeatureConfig),
				),
			),
		)),
	)
}

func (s *myFeatureSuite) TestMetricReachesIntake() {
	s.EventuallyWithT(func(c *assert.CollectT) {
		metrics, err := s.Env().FakeIntake.Client().FilterMetrics("myfeature.points")
		require.NoError(c, err)
		assert.NotEmpty(c, metrics, "no myfeature.points received yet")
	}, 2*time.Minute, 10*time.Second)
}

The common agentparams options are WithAgentConfig, WithIntegration, WithLogs, WithFile, WithTags, and WithHostname; references/environments.md has the rest and the container equivalents.

Invariants, each with the reason it exists:

• Embed e2e.BaseSuite[Env] and start through e2e.Run — that is what builds s.Env() and registers teardown.
• Call t.Parallel() first in the entry point so suites overlap instead of serializing their provisioning.
• Keep sibling test methods independent. Testify does not guarantee ordering, so a method relying on a sibling's side effects fails unpredictably; put shared setup in the suite, and use s.T().Run(...) subtests when you genuinely need sequence.
• Leave the environment as you found it. Retries reuse the same infrastructure when they can, and fall back to a full reprovision when a leftover artifact breaks the second attempt.
• When overriding SetupSuite, BeforeTest, or AfterTest, call the embedded method first. Do not add defer s.CleanupOnSetupFailure() — BaseSuite already registers a t.Cleanup hook for it, and calling it from a defer consumes the panic so testify reports the recovered value with no stack, costing you the file and line of the failure. Older suites still carry the defer; leave them alone rather than copying them.
• Log what a future debugger needs on failure only. The infrastructure is destroyed after the run, and E2E_DEV_MODE=true is a local-only escape hatch. Write large dumps to s.SessionOutputDir() as an artifact rather than into the job log.