Testing And CI

Testing and CI

Browser choice

Use headless Chrome for browser E2E tests, screenshots, developer-tools network traces, and conformance visual evidence. Avoid Firefox unless the user explicitly requests it for a browser-specific investigation. Prefer a pinned Chrome for Testing distribution for reproducible external workflows; keep its binary, driver, profiles, caches and artifacts under this repository's tmp/. Preserve real browser interaction and TLS validation. Do not set acceptInsecureCerts, ignore-certificate flags, disable web security, rewrite Origin headers, or fabricate successful callbacks to make a conformance run pass.

Overview

Use this skill for caddy-security test selection, fixture maintenance, and CI reproduction. Prefer the narrowest go test command while editing, then use Makefile targets when the user asks for the repository workflow, report artifacts, or CI-like validation.

The Go module is rooted at the repository top level. Most tests live in the root security package and cover Caddyfile parsing, Caddyfile adaptation, and runtime authcrunch config resolution.

Follow the repository scope. Create and run tests in this module only; never run sibling suites to validate work here or fix a failing test by editing the sibling. TEST_DIR and QUICK_TEST_DIR must select this module's packages, and COVERAGE_DIR must not point into another checkout. Existing local replacements are dependency inputs only. Keep compatibility fixtures and reports here and report required upstream test or implementation changes as separate work.

Required Coverage for Code Changes

When writing or changing code, ensure both unit tests and E2E tests exist and exercise the changed behavior. Add or amend tests where coverage is missing; existing tests count when they demonstrably verify that behavior. Run the relevant unit and E2E tests before considering the code change complete.

Unit tests should check focused behavior and meaningful failure cases. E2E tests should exercise the affected user-visible flow through the assembled system. For authentication, authorization, and lifecycle changes, use actual Caddy provisioning, routes, and HTTP requests as appropriate. Bound network, process, and worker completion; clean up test-owned resources. Parser/adapt tests and sibling go-authcrunch tests do not replace this repository's E2E coverage. Report missing coverage or blocked validation explicitly.

Use lifecycleAddress for disposable Caddy listener addresses. It checks both TCP and UDP availability because Caddy's default HTTPS server also starts a QUIC listener; a TCP-only ephemeral-port check can select an occupied UDP port. The occupied-QUIC-port regression also verifies release of the rejected TCP reservation. Reservations are released before Caddy takes ownership.

Every Caddyfile directive change also requires new or amended adaptation test cases in testdata/caddyfile_adapt/, even when the resulting JSON shape is unchanged. Register new cases in TestCaddyfileAdaptAuthenticationToJSON in caddyfile_adapt_test.go so the fixture is exercised. This is additional to unit and E2E coverage; use the fixture mechanics below.

Documentation/skill-only edits use metadata, link, and source checks from skill-authoring; they do not require new runtime tests.

Syntax and Example Audits

Follow Syntax maintenance when parser grammar or a dependency changes. Inventory standalone Caddyfiles, Go syntax comments, skill examples, and inline positive/negative tests. Adapt runnable examples with a binary built from the selected dependencies. Wrap fragments in their actual global/portal/policy/site scope; do not run syntax catalogues or intentionally invalid examples as complete configurations.

Adaptation verifies only the validation reached by that parser. Raw crypto, messaging, registration, ACL, and other deferred settings also need focused resolution/validation checks when their examples change. Do not provision a real deployment, contact an external provider, or change machine trust merely to audit syntax. Preserve intentional failure fixtures and legacy alias tests. Report external-module requirements and runtime checks separately from adapt success. Comment-only edits do not change parser behavior; use source review, formatting, and relevant existing tests instead of adding tests that mirror prose.

Command Selection

Use direct Go tests for quick feedback:

go test ./...
go test -run TestParseCaddyfileAuthorization ./...
go test -run TestCaddyfileAdaptAuthenticationToJSON ./...
go test -run TestResolveRuntimeAppConfig ./...

Use make test for the repository report lifecycle. go.mod pins github.com/greenpau/tested v1.1.0, invoked as go tool tested; it owns -json, -coverprofile, child-process status, and coherent reports. Do not reintroduce go test | tee, log-grep success detection, richgo, tparse, or go-test-report.

make test
make test TEST='TestParseCaddyfileAuthorization' TEST_DIR='.'
make qtest TEST='TestParseCaddyfileAuthorization'
make run-reports
make test-automation
make ci-check

Lifecycle runs use -mod=readonly -race -count=1 -p 1 -parallel 2 -timeout 60m -v. The macOS/Linux resource guard bounds the whole process tree, including compilers, browser/CLI children and report rendering. Read test resource controls when changing defaults or investigating an interrupted run. The default wall limit is 4,200 seconds, allowing compilation/report time around the 60-minute package limit. Keep live tested output enabled; the guard also prints progress every ten seconds. TEST is a regex (default .), TEST_DIR accepts package patterns (default ./...), and TEST_TIMEOUT overrides the quoted per-package limit. MINIMUM_COVERAGE defaults to 1 percent as a nonzero-profile check, matching go-authcrunch; it is not a substantial coverage target.

Go's timeout covers the entire package, including all sequential E2E parent tests. The Caddy journeys also have their own shorter child-process deadlines. If CI times out, inspect the captured test events and active test duration to distinguish an exhausted package budget from a stuck individual journey. Keep the job budget larger than the package budget so setup, builds and report upload can finish; the current workflow allows 75 minutes around the 60-minute Go package limit.

Reports land in .coverage. make qtest defaults to the root package (.) with reports in .coverage/quick; override QUICK_TEST_DIR and TEST for another scope. Use COVERAGE_DIR for separate evidence bundles; guarded runs in the same checkout cannot overlap, even with different output directories. Let tested refresh its managed files without deleting other bundles or investigation notes. make run-reports regenerates presentations from recorded evidence and preserves failures; make coverage aliases it without rerunning tests.

Use make build when validation needs bin/authcrunch or bin/caddy-authenticator; it builds both and prints their versions. Formatting is separate: make fmtcfg formats fixtures under testdata/caddyfile_adapt and assets/config. Builds/tests do not rewrite licenses, version files, module manifests, or Caddyfiles. make dep downloads/verifies pinned dependencies and resolves tested; it may need network access but does not install global tools.

Test Surfaces

Read test surfaces and qualification evidence when selecting existing unit/E2E tests, extending a feature suite, or changing coverage collection. It maps conditional authentication, CodeQL, subprocesses, logging, persistent state, operator examples, OIDC, refresh, native/CLI clients, local identity, authorization paths, lifecycle, and Caddyfile fixtures to their actual tests and limits.

For interactions among login, refresh, OIDC, upstream OAuth, authorization, edge metadata and reload, read composition qualification. The feature owner's acceptance contract determines what must be exercised; a list of parser fixtures alone never proves a user flow works.

Subprocess coverage

Root-package E2E helpers run the same instrumented test executable. The subprocess coverage contract owns collection before parent completion, isolated child directories, counter publication, killed-process limits, and the regression workflow. Read it when adding a subprocess or changing report collection.

Adding Coverage

When adding or changing a Caddyfile directive, add focused parser coverage in the nearest caddyfile_*_test.go file. Include both the successful config shape and a malformed input when the parser has a meaningful error path.

For a Caddyfile directive change, add or amend adaptation cases in testdata/caddyfile_adapt/: <prefix>.Caddyfile, <prefix>.json, and optionally <prefix>.env, and update the test case registration as needed. If runtime defaults, replacements, credentials, UI, OAuth, registration, or cookie behavior changes after adaptation, also add or update <prefix>_resolved.json and include the prefix in TestResolveRuntimeAppConfig.

After fixture edits, run the focused test first, then a broader command:

go test -run TestCaddyfileAdaptAuthenticationToJSON ./...
go test -run TestResolveRuntimeAppConfig ./...
go test ./...

If *_tmp_input.json or *_tmp_output.json files remain after a failing runtime resolution test, inspect them, fix the fixture or implementation, and remove the generated temp files before finishing unless the user explicitly wants debug artifacts kept.