Opt Trace Mining

The seventeen gates

validate_store.py runs this repository's own tools/check_kernel_wiki.py against the staging root, so all nine of its gates apply — schema · ids · anonymization · raw-isolation · relations · index · self-contained · no-cross-reference · established-fact — and then eight that only this pipeline can run, because only it has the trace and the packets:

• profile — the record satisfies opt-trace-1.0, which additionally requires the trace provenance triple, measured_on, gain.kind, and established_fact on every anti-strategy, and closes evidence.raw so a dead path cannot be reintroduced.
• layout — the directory equals the record's own scope, derived exactly as wiki-gate derives it on insert. The store's ids gate checks the filename but not the path.
• verbatim — implementation.snippet must appear literally in the packet's sibling diff or kernel file. The gate and the distiller read the same file on purpose. Compared line by line, so a snippet assembled from two hunks passes.
• no-fabrication — every number in payload, worth.gain, evidence.summary and retrieval.signals.metrics must appear in the packet or in its code. Code-ish fields are exempt because they are verbatim source.
• provenance — evidence.raw must name this trace, and its git_commit must resolve to a commit in it. This is the whole of a record's auditability once the packets are deleted.
• ncu-attribution — a basis=profiler claim must cite a capture that measured the kernel under test. A capture taken without a --kernel-name filter is schema-valid and describes the wrong kernel, so it is actively misleading rather than merely empty; without this gate such a record looks well-evidenced.
• store-overlap — the id must still be free in the live store, because wiki-gate --commit insert refuses a duplicate and renumbering after a batch is the expensive part. An episode_key that already exists is reported, not failed: whether it is a rediscovery to confirm is the agent's judgement.
• journal-provenance — a granular long-horizon record must cite an existing journal or archived evidence file, every declared experiment id must occur in it, and its canonical promotion or revalidation commit must resolve. Evidence paths must be relative to the trace root; absolute paths, traversal, symlink escapes, files above 8 MiB, and aggregate evidence above 32 MiB are rejected before content is read.

Never weaken a gate to make records pass, and never let a distilling agent edit scripts/. When a gate looks wrong, prove it fires:

python3 validate_store.py --injection-tests

Each case mutates a copy of a real record — and, where the error lives there, its packet — and asserts the named gate complains. A gate without an injection test is how a store ends up falsely green.

One gate the predecessor had is deliberately gone: it checked that every record cited an existing markdown page. This repository has no markdown tree, so that gate could only be satisfied by writing a citation to a file that does not exist. Provenance replaced it.

Distillation

Spawn agents with references/distill-brief.md verbatim. Batch by record type so a failure has a small blast radius, and point every agent at the one record that already passes as the worked example.

Require each agent to run validate_store.py itself and iterate to green, and to report which fields the packet was too thin to fill and which gate blocked it. That report is the main signal for improving the pipeline; treat a batch that reports no difficulties with suspicion. When several agents write into one staging store concurrently, tell them explicitly to ignore gate failures naming records they do not own.

An anti-strategy segment whose evidence names neither a checkable condition nor a cause must not be written up at all. partition.py marks those with fact_precheck, but the flag is a hint, not a verdict: a regex must narrow and never judge, so it also flags genuine facts whose wording is unusual, and the agent resolves it from the packet's own evidence.

How a record reaches the store

skills/wiki-gate is the only writer into kernel_wiki/records/. Nothing this skill produces is served until it has been through the gate, whatever its own gates say:

1. gate.py --match --input <record.json> returns every same-scope candidate plus any exact episode_key match. It makes no decision.
2. The agent decides: no match → insert; a match pointing the same way → confirm (bumps the existing record's counters, idempotently); a match pointing the opposite way → conflict (queued for a human, exit 0).
3. gate.py --commit <action> executes it. insert re-runs the store's record-level gates, refuses an id that already exists, writes the record under records/<type>/<vendor>/<arch>/<dsl>/<operator_family>/ and appends the index entry.

A record rejected by the gate stays in staging. It is not deleted: once it gains the condition and the mechanism it lacked, or is independently rediscovered, it can go through again.

Porting to another trace archive

Everything is trace-agnostic except three places:

• config.py — TRACES (where your traces live and what hardware they ran on), TARGET_TABLE (hardware token → vendor/arch/product), and NON_TARGET_KERNELS.
• families.py — operator naming: raw directory name → record slug and workload family. This is the only file that decides where a record is filed, so it is self-contained and self-tested rather than shared: a change in another tree would silently refile records. python3 families.py checks the slug rules and that every family it can emit is still a value the schema allows.
• ingest.py — the only file that knows how a trace is shaped. A different layout means adapting read_commits / read_memory / read_profiles; everything downstream sees version rows and never a raw file.

Two things to check on a new archive before trusting the output: whether every canonical memory/v<N>.json addition can be paired with the intended kernel state (legacy subject-only traces use v<N>: as fallback), and what fraction of profiler captures measured the kernel under test — recon.py prints both.

Resources

• references/distill-brief.md — the agent brief. Pass it verbatim.
• assets/schema/opt-trace-1.0.schema.json — the corpus profile, generated by make_schema.py; run it with --show to read the patch list and the reason for each patch.
• skills/wiki-gate/references/established-fact-criteria.md — the normative admission bar for negative knowledge; partition.py imports the same regexes the store's gate uses, so triage and enforcement cannot disagree.
• Self-tests worth running after any edit: python3 families.py, python3 ladder.py, python3 anonymize.py, plus the repository's existing query and Wiki validation suites.