Validating Models Against Logs

Validating Computed Models Against the Log Corpus

Overview

src-tauri/logs.db (the dev copy — the AppData one is empty) is not just history: it is a labeled measurement corpus. Encounters store the raw event log, and the hook-read fields in it are ground-truth labels a model can be scored against offline, hit by hit: captioned SBA gains (SbaGainCause::Skill(action)), the per-hit damage_cap on every DamageEvent, buff lists, stun deltas. Core principle: exhaust the corpus before asking for a live round. The SBA weight table went to 68/68 verified (character, action) values with zero live play; treat a proposed capture session as a smell that the corpus hasn't been mined yet.

The loop

1. Implement the model as a pure function/table in the parser crate, loadable by examples (the sba_weights.rs / assets/*.json shape).
2. Write a committed diag example in src-tauri/examples/ that replays EVERY log and scores model vs ground truth per (character, id). Give it three modes: corpus-wide sweep, single-log detail dump (--dump-log — for identifying what an unknown id physically IS from its damage signature and company), and targeted-id measurement.
3. The sweep must emit its own worklist: every unexplained value, ranked by implicated magnitude, with a candidate solve where one exists (always printing n and max deviation).
4. Resolve the top worklist entries — game data first (the extracted table catalog beats guessing), measurement decides when the data is ambiguous or silent. Write provenance into the table (see below).
5. Re-run the sweep. Iterate until the worklist is empty. The sweep then becomes the regression gate you re-run after every game patch.

Key moves

• Absolute vs scale-ambiguous — decide first. Cap values are absolute: score them exact, no tolerance bands, ever. Gauge-like quantities are only measurable up to a per-fight constant K: find an authored anchor present in every log (LinkAttack = 5.0 for SBA) and divide it out. K cancellation is what makes offline verification possible at all; without an anchor, ratios between values are still exact evidence.
• Zero is a measurement, not a default. N hits with zero associated ground-truth signal proves the value is 0 — and the claim must state N ("317 summon hits across four characters, 0 captioned grants"). File it with that provenance, same standard as a nonzero value.
• Exact clusters beat noisy fits. Captioned local slots yield exact repeated values (Ferry 9995 measured as precisely {0.0439, 0.1385, 0.2}); indirect/remote fits are leads only. A fit with n≤2 or high deviation goes on the worklist, never in the table — the remote n=2 solves for action 80000 said 0.16–0.58; the corpus said exactly 0.
• Classify misses honestly, in named buckets (contaminated, unknown-id, no-evidence…), and drop contaminated samples WHOLE — contamination must never read as model drift. Fit with medians, report sums separately: sums are pulled by unobservable contamination (K_med < K_total turned out to be invisible damage-taken gauge, not drift).
• Errors must fail honest. Prefer the failure mode that under-claims (an explicit unnamed remainder) over one that invents values. Check the DIRECTION of residual error before accepting it as tolerable.
• Provenance lives in the table. Every value carries a confidence tier (verified/high/med/low), its source, n, and the log ids it was measured from — so the next game patch's re-derivation re-verifies mechanically (the build_final.py MEAS pattern) instead of re-arguing.