Error Handling And E2E

Errors & e2e testing in Kopiur

Two linked disciplines. Strong typed errors make invalid states unrepresentable and tell the operator exactly how to recover; e2e tests prove the whole pipeline reaches the user-visible success condition. Both exist because this is data-protection software — a silently-swallowed error or an untested path can lose backups.

Read alongside [[kopiur-design]] (the type-safety thesis) and CLAUDE.md.

Part 1 — Strongly-typed, actionable errors

The pattern (mirror crates/controller/src/error.rs)

1. One thiserror enum per crate/domain, #[derive(Debug, thiserror::Error)]. One variant per distinct failure mode. Use #[from] to wrap upstream errors (kube::Error, KopiaError, serde_json::Error) so ? just works.
2. Exhaustive classification, not catch-alls. If the error drives behavior (requeue timing, fatal-vs-degrade), express that as a method with an exhaustive match and no _ => arm — a new variant must fail to compile until it is classified. See Error::class() -> ErrorClass (Transient/Structural → backoff). A _ arm here is a latent data-loss bug.
3. pub type Result<T, E = Error> alias per crate; reconcile/fallible fns return it.
4. Test the classification AND the message. Unit-test that each variant maps to the right class/behavior (error.rs tests) and — for any message a human acts on — assert on to_string() so the actionable text can't silently rot.

The message rule: what failed, why, how to fix

Every #[error("…")] an operator might read states what failed, the likely why, and the concrete fix (expected value, env var, command). This is the type-safety thesis extended to UX: homelabbers run this; a cryptic error wastes time and can mask a data-protection gap.

#[error(
    "OTEL_EXPORTER_OTLP_ENDPOINT='{value}' is not a valid URL. Use scheme+host+port, \
     e.g. http://otel-collector:4317 (OTLP/gRPC) or :4318 (OTLP/HTTP); unset it to disable OTLP"
)]
InvalidOtlpEndpoint { value: String, #[source] source: url::ParseError },

Prefer {named} fields over positional {0} when the message interpolates user input — it reads better and survives reordering. Keep the underlying cause via #[source]/#[from] so the chain is inspectable; don't bury it inside a string.

Degrade, don't crash, for non-critical subsystems

Reconciliation and data movement are critical; observability, OTLP export, and similar side-channels are not. A misconfigured non-critical subsystem must log the actionable error and continue, never abort the operator:

• init_telemetry() returns Result; on error the binary logs it at error and falls back (fmt-only logs + the always-on Prometheus pull). An optional KOPIUR_OTEL_STRICT=true makes it fail-fast for those who want it.
• Critical paths still fail loudly: a bad CRD spec is Structural (long requeue, surfaced on .status + an Event), a kopia/API outage is Transient (short requeue). Both go through error_policy_for, which records controller_reconcile_errors_total{kind,class} and requeues by class.

Surfacing errors to users

Reconcile errors should reach the user, not just the log: set a Condition / phase on .status and emit a kube Event via the Recorder (Context already carries one). The metric + log + status + Event together make a failure visible in kubectl get, dashboards, and alerts.

Part 2 — e2e testing (crates/e2e)

When to write an e2e test

Both bug fixes and new features ship with e2e coverage ([[kopiur-design]] — "Every change ships with tests"). Per [[regression-test-every-bugfix]], a fix ships with a test at the cheapest tier that exercises the broken path; a new feature additionally gets a full-pipeline e2e scenario whenever it has any runtime-observable behavior. Reach for e2e (not just a unit test) when the behavior only appears against a live operator: a reconcile that must reach its terminal phase, a runtime dependency, an RBAC/SA gap, an option that survives serialization but acts at runtime, a new backend/volume source the mover must actually mount, or metrics that must be emitted and scrapable.

No e2e is too large to include — this tool is data-protection software and must be thoroughly tested. If a feature needs new cluster infrastructure to exercise (an NFS server, an SFTP server, a fresh namespace, a second backend), provision it in the World rather than skipping the test:

• Add a Need variant + an ensure_* method (idempotent Fixture apply), plus a builders::* Deployment/Service for any in-cluster server. Mirror the existing servers (ensure_sftp/ensure_webdav/ensure_nfs).
• Pin every server image (never :latest) and verify it actually serves the mover on a real kind cluster before trusting a green run — a server the client can't talk to fails slowly and confusingly, not fast (see [[sftp-openssh10-go-ssh-hang]]). Note in the test/PR if it's pinned-but-not- yet-cluster-verified.
• A multi-minute run, an extra built image, or a privileged helper pod are all acceptable costs. The bar is "is the feature exercised end to end," not "is the test cheap."

The only features that legitimately skip e2e are those with no runtime-observable behavior (a pure type change/refactor) — and then say so explicitly.