Tests Query Corpus

Query Contract Corpus — developer contract

tests/query-corpus/ is an end-to-end correctness suite for the query engine of a stock netdata daemon. Fixtures enter through the real streaming protocol and queries use the normal HTTP API. Structured responses are strictly decoded before semantic checks; expectations are fixture-derived, source-derived, or explicitly labeled stability/parity pins under the classes below. tests/query-corpus/README.md describes L0-L11 plus cross-cutting API, options, and weights surfaces.

The suite is a self-contained Go module (github.com/netdata/netdata/tests/query-corpus). Always run go commands from inside tests/query-corpus/.

Choose the task

For review, assess the complete assigned change without creating authoring artifacts or starting a daemon merely because the running workflow appears below. Execution and Git steps apply only within existing task authorization; recording a merged fix does not itself authorize a rebase or push. Preserve the full-suite gate when corpus delivery and its push are authorized.

Correctness model — why this suite means something

The founding rule: semantic correctness expectations MUST be derived independently of engine output. Capturing engine output is allowed only for an explicitly labeled Class C stability pin; such a pin is not a correctness oracle. Every check in the corpus belongs to one of three classes, with different rules:

• Class A — first-principles oracles (the default). Fixtures are literal Go definitions: charts, dimensions, and explicit (timestamp, collected value, SN flags) points. Most deterministic fixtures anchor at fixture.T0; wall-clock-only cases use bounded envelopes. Expected values come from fixture arithmetic, conservation, additivity, metadata laws, group-by folds, weights math, and anomaly-bit counts. New checks MUST be Class A unless the transform genuinely cannot be derived from first principles.
• Class B — ports of engine algorithms. Some transforms are engine design decisions, not derivable math. fixture/sn.go, fixture/tier.go, fixture/timegroup.go, and fixture/viewpoints.go contain source-derived ports; they also contain explicitly labeled Class A or contract helpers. Class B code is written from the C source, never captured from engine output. Rules:
  • A port MUST cite owner/repo @ commit plus the exact repository-relative source path and line range it mirrors.
  • Every divergence found between a port and the engine MUST be resolved explicitly: either it is an engine bug (author a case for it) or an engine quirk adopted into the oracle with a recorded pending ruling in the manifest/SOW. Silently adjusting a port until the engine passes is the cardinal sin of this suite ("fit-to-engine") and is prohibited.
  • Where a port could hide drift, bound it independently (e.g. L2 pins the SN quantization error envelope against the ORIGINAL values).
  • A port MUST NOT be the oracle for a quantity that obeys an independent law. Where one exists — conservation, monotonicity, additivity — that law is derivable, so the check is Class A and the port has nothing to contribute but the engine's own opinion. ViewBuckets is a source-derived model of the default tier-0 point-selection/interpolation subset exercised by L9, not a port of the complete query executor. It is appropriate only for those covered level-at-an-instant shapes. ViewSumVolume is a Class A conservation oracle for volume and is deliberately independent of that selection model. For every Class B use, prefer an independent law when one exists.
• Fixtures MUST make the answer exact where the contract is exact. A conservation check reads a difference, so anything else that moves the total is noise the check will report as a defect. Above tier 0 the loudest such source is resolution: when a window edge or a plan seam cuts a stored record, the part inside can only be estimated from the record's width, and with varying data that estimate is wrong by a share of the record — which is what tiers cost, not a defect (see "Tiered rollups are not bugs").
  • Grid-align the window so no edge falls inside a record, and where a boundary is discovered rather than chosen (a rotated tier head), carry a flat dimension alongside the varying ones: a constant survives rollup exactly, so its share of a cut record is its exact truth and any difference left is arithmetic.
  • Worked example: CASE-026's seam contract first read 0.49s of data "missing" across a plan switch, entirely from the seam cutting one random-valued tier1 record. On the flat dimension the same three windows answer 7,200,000 to the digit at every zoom. Had the noisy version been committed, the corpus would have reported accepted rollup as an engine bug — the mirror image of fit-to-engine, and just as damaging.
• Class C — stability pins and parity checks. Formatter byte pins (primarily L7 and selected option tests) detect output-contract changes; parity checks prove internal coherence only. Neither establishes first-principles correctness. Rules:
  • A Class C pin MUST be paired with independent validity checks where they exist (e.g. "the payload parses as JSON", "values equal the fixture-derived numbers inside the pinned envelope").
  • Updating a pinned byte string requires a justified contract change (a PR that deliberately changes the output format), never "the test started failing".

Falsifiability discipline: first rule out an independently demonstrated fixture, harness or oracle defect. Correct such a defect from fixture math or the cited source contract, with evidence explaining the error; observing the engine result alone is never justification. Once the expectation is established and the engine disagrees, there are exactly two exits — the engine is wrong (the case stays as written and joins the broken list until it is fixed) or a recorded ruling says the behavior is intended (the case is rewritten to assert the ruled behavior, with the quirk documented). There is no third option where the oracle is quietly bent to match, and none where the disagreement is filed away as acceptable.

Architecture map

• stream/stream.go — the fixture child. Speaks plugins.d over the streaming socket: CHART/DIMENSION/CLABEL, live samples (BEGIN2/SET2/END2), v1 paced samples (BEGIN/SET/END), and replication (RBEGIN/RSET/REND). Protocol words quote-switch per word (qw()): plugins.d accepts both ' and " delimiters, so ids carrying an apostrophe ship double-quoted. SET2 sends the value explicitly because the # shorthand truncates fractional values on the parser side. Exact parser citations are listed under "Changing oracles, pins, and the harness."
• daemon/daemon.go — the harness. Resolves the binary and source tree as a pair, boots the stock binary with a scratch run dir, assigns a unique daemon identity, waits for that identity through the HTTP API, exposes query helpers and WaitRetention, and uses bounded shutdown.
• fixture/ — the fixture model plus source-derived Class B ports and explicitly labeled Class A or contract helpers.
• canon/ — the strict typed json2 decoder and point-column helpers. It rejects invalid schema indices, widths, labels, field types/ranges, and non-finite numeric metadata while decoding nullable value/hidden cells. Query-specific assertion helpers enforce whether those nulls are semantically valid and carry the required annotations.
• *_test.go — layer*.go ladder tests, cross-cutting surface tests, and per-bug caseNNN_test.go files.
• manifest.json + MANIFEST.md — the ledger data and its human-readable mirror (below). manifest.go embeds and validates the data. Keep all three in sync in the same commit.
• reference-python/ — local-only cross-check implementation. It is NOT tracked and MUST NOT be committed.

The manifest

Every contract case has an entry in manifest.json; its proves, cloud, and optional fixed_by fields are mirrored as a row in MANIFEST.md.

• One contract key MUST represent one independently actionable semantic invariant. Multiple fixtures or inputs MAY share a key when they all prove that same invariant. Independent claims, such as numeric value correctness and unit rendering, MUST use separate keys and separately registered test or subtest scopes. A shared green/red verdict is prohibited because an existing failure in one claim can hide a new regression in another while the broken contract roster remains unchanged.
• Register ordinary Go assertions with trackContract(t, name) before any operation that may fail or skip. Its cleanup records Error, Fatal, and Skip.
• Use assertContract(t, name, held) when the test computes an explicit contract verdict. Call registerContract(t, name) before shared work, so an earlier Fatal or Skip leaves the verdict visibly incomplete instead of letting a default-true accumulator report it green.
• When independent test scopes jointly prove one contract, declare their names in ManifestCase.Components and register each with trackContractComponent. One component passing never substitutes for another component that did not run.

A broken contract fails. Always. On master, on a feature branch, whether or not the break is already known.

• The manifest records NO expected outcome. There is no "known broken, and therefore fine" state, and adding one is prohibited: it makes a broken query engine report success, and this suite exists to name what is broken, not to keep a list of exceptions.
• An unfiltered root-package run ends with the deduplicated list of broken contracts and fails if any manifest contract or required component did not run. That complete list is the corpus's answer — the open-defect list produced by measurement rather than by hand.
• A filtered daemon-backed run reports how many contracts were fully evaluated and never claims the complete corpus holds. The deliberately named daemon-free harness/unit fast path prints only its Go test result and no query-contract verdict. -list also prints no contract verdict.
• go test ./... therefore exits non-zero while any contract is broken or the ledger is incomplete. That is the intended signal, not a problem to suppress. The corpus is not wired into CI.

A case name is <layer-or-CASE-id>/<slug>; Proves is one sentence a maintainer can read as the contract claim. A case whose bug is fixed keeps its test as the regression guard and records FixedBy: "#PR".

Running

• Build the daemon first: ninja -C build netdata from the repo root. By default the suite pairs ../../build/netdata with ../../src.
• For another checkout, set both paths: QUERY_CORPUS_NETDATA=/absolute/path/to/build/netdata QUERY_CORPUS_SRC=/absolute/path/to/src go test .... One-sided overrides are rejected. The pair is operator-declared provenance; the harness does not inspect binary build metadata.
• Full suite: cd tests/query-corpus && go test ./... -count=1. Expect several minutes; duration is hardware and case-selection dependent.
• One test: go test -count=1 -run 'TestName' .. Some tests consume a shared palette authored by an earlier layer; include that fixture-producing test in the filter or run the full suite when a test reports that its palette is unavailable. A prerequisite skip is not correctness evidence.
• Keep the daemon run dir for inspection: QUERY_CORPUS_KEEP=1 (it is always kept on failure; the path is printed as daemon run dir kept:).
• Capture the verdict honestly: go test ... ; echo "exit=$?" — piping through tail masks the exit code.
• Before every push of the corpus branch, run the full suite and compare the broken list to the previous run. It must not grow, and no case may break that was holding before. The list being non-empty is expected while the query engine still has open defects.