Elite Intel Diagnostic Run

elite-intel-diagnostic-run

Automated tester for companion routing. You play the role a human tester would: speak phrases to the app (by appending them to a file), wait like a person would for the assistant to answer, and check the app understood each phrase — all read back from a log file, because an LLM-driven app can't be unit-tested deterministically.

Crucially, you do not replay the test's phrases verbatim. The reference test's aliases are only a reference for the intent of each command; you compose your own natural utterances a real commander might say for that same intent, so the run tests whether routing generalizes beyond the exact training/alias phrases. Ground truth is the intent→action mapping taken from the test group; the words are yours.

Arguments and preconditions

• Language (required, no default) — one of EN, RU, UK, DE, ES, FR, IT, PT. It selects the reference test app/src/test/java/elite/intel/junit/prompt/NaturalSpeechIntegrationTest<LANG>.java and the cached glossary glossary/<LANG>.md.
• Count (optional, default 3) — generated utterances per command, e.g. /elite-intel-diagnostic-run RU 4.
• --refresh-glossary (optional) — rebuild the cached terminology from the web.
• --only <id1,id2,...> (optional) — restrict the run to those expected action ids (targeted mode); everything else is skipped. Turns drop from ~270 to len(ids) × count.
• --replay-fails [path] (optional) — regression mode: instead of generating fresh utterances, re-feed the utterances that FAILED in a prior run's saved plan (path defaults to the most recent runs/<LANG>-*.json). No generation, no web. This is the one case where verbatim re-feeding is correct — see Run modes.

Run modes

• Full generalization (default). All ~90 groups × count with FRESH utterances. This is the point of the skill: it measures whether routing generalizes, not memorization. Use it for a real audit of a language.
• Targeted (--only). Same fresh-generation logic, restricted to specific actions — for iterating on a few intents you just changed without paying for the whole run.
• Regression (--replay-fails). Re-feed the exact utterances a previous run flagged, to confirm a fix landed. Verbatim replay is deliberate here because the job is verifying a specific regression, not exploring new phrasings — so it does not contradict the "fresh, not memorized" rule that governs the other two modes. After a green replay, run a fresh targeted pass on the same ids to confirm the fix generalizes, not just that it satisfies those specific strings.

Validate the language first. If it is missing or not one of the eight codes, do nothing else — no files, no app launch, no web — print an error and stop, e.g. elite-intel-diagnostic-run: language argument is required — one of EN, RU, UK, DE, ES, FR, IT, PT (e.g. /elite-intel-diagnostic-run EN).

Run exactly ONE language per invocation. Turns ≈ (command groups, ~90) × count; at the default that is ~270 live companion LLM turns — the whole cost of the run. Never loop over several languages; if the user wants more, run them as separate, explicitly requested invocations. Raising count scales turns linearly — keep it modest.

Run without pre-flight questions. Given a valid language, do the whole procedure — build the glossary if missing, launch the app, feed, report. The provider/model is the user's in-app choice and irrelevant here (the skill only writes phrases and reads the log): do not detect it, factor it in, or ask.

Environment: OS-agnostic (Linux, macOS, or Windows). Do not assume a shell or hardcode platform-specific paths or commands; drive every file operation through the portable Python 3 one-liners in Harness I/O (cross-platform) below.

How the harness works

The app runs in diagnostics mode when the input file input.txt exists at startup — its presence is the gate. The app only ever READS it and never creates it, so you own its lifecycle: create it before launch, delete it after the run, so the mode does not stick to later launches.

Files (all under the diagnostics directory, resolved per-OS in Harness I/O (cross-platform)):

• input.txt — the gate; its existence enables the mode, and you append phrases to it during the run.
• language.txt — a language code read at startup for the boot language (a data file, NOT the gate).
• session.log — the app mirrors everything here.

The app truncates the log at its own startup, but that happens a while after you launch it, so you must ALSO clear the log yourself right before launching (step 5) — otherwise you read the previous run's log and its stale DIAG ready.

In diagnostics mode STT is off (input comes from the file, so the mic model isn't loaded — faster startup), but TTS is REAL: the companion voice stays audible. App turns are fast (~1 s each: the LLM answers, then a short spoken reply). If a run feels slow it is almost never the app — it is utterances generated lazily between feeds (step 3: generate everything UP FRONT).

Input line kinds

• A plain-text line = a spoken phrase → routed like microphone input.
• @visible <actionId> = the preferred way to set "where we are". Puts the game in the first context in which that action is visible to the router — using the app's real isVisibleForLLM, exactly as the routing test's applyStateFor does. Emit it before each utterance with that utterance's expected id; you never guess a context. Applies immediately, no turn. (Also tries a deployed fighter for fighter orders.)
• @status <context> = manual context override (main_ship, supercruise, docked, landed, srv, on_foot) — only for special cases; prefer @visible. Applies immediately, no turn.
• @fighter on / @fighter off = toggle the fighter-deployed gate manually (usually unnecessary — @visible covers it).
• @lang <CODE> = switch the app's command language at runtime. Do not use this to set the run language — it is too late for the semantic reducer, which freezes its language at construction. Set the run language via language.txt (step 5) instead. (Kept only for ad-hoc TTS-language tweaks.)
• A JSON line with an "event" field = inject a journal game event (advanced; rarely needed for routing).
• A line starting with # = comment, ignored.

Log markers you parse

Each line is <UTC-timestamp> <marker>:

• DIAG ready — all services up and the LLM endpoint confirmed reachable; the ONLY readiness signal — do not feed before it. (DIAG companion-started appears earlier, when the companion is merely wired — NOT readiness; DIAG llm-not-connected means the LLM probe failed and the app is retrying.)
• DIAG log opened — a fresh app instance re-opened the (cleared) log; expect it before this run's DIAG ready.
• DIAG input="<phrase>" — opens a turn for that phrase.
• DIAG dispatch tool=<actionId> — an action the companion dispatched this turn (the routing result you score).
• DIAG speaking=true|false — TTS boundaries.

DIAG turn-done — the terminal marker for a phrase's turn, written once the turn has started and gone quiet (after all its DIAG dispatch line(s) and the DIAG speaking=false that ends its spoken reply). Key turn-completion off this single anchor.

• DIAG boot-language=<CODE> — the app set the command language from language.txt at startup (before the companion/reducer were built). This is how the run language is applied. A DIAG boot-language error line means the code didn't parse (check for a stray BOM/newline).
• DIAG lang=<CODE> — a runtime @lang switch was applied (does not re-language the reducer).
• DIAG visible=<actionId> state=<ctx> — @visible set the context so <actionId> is visible. state=unknown-action = the id did not resolve (fix it); main_ship(fallback) = it was never visible in any probe state.
• DIAG status=<ctx> / DIAG fighter=<bool> — a manual directive was applied.
• LOG / DBG / AI / USER — mirrored SYSTEM LOG, useful for diagnosing a miss.

Pass criterion

A phrase passes if its expected action id appears in a DIAG dispatch tool= line in that phrase's turn (the lines between its DIAG input= and the next DIAG input= / DIAG status= / DIAG event=).

Harness I/O (cross-platform)

This skill is OS-agnostic — it runs on Linux, macOS, and Windows. Do NOT assume a shell or hardcode platform-specific paths or commands. Every mechanical file operation is a **Python 3 one-liner **: it behaves identically on every OS, writes UTF-8 without a BOM by default, and accepts Windows and POSIX paths alike (forward slashes work in all of them). Run them with whatever Python 3 launcher the host has — python3 on Linux/macOS, python or py -3 on Windows. The only per-OS difference is your own shell's quoting for the utterance argument, which you handle per call. These single-operation commands are NOT "building an orchestrator" (see Procedure); each is one file read or write, the exact analogue of the old per-line append/read calls.

Where the files live. The app resolves the diagnostics directory per-OS (mirroring AppPaths.getAppDataBase): Windows %LOCALAPPDATA%\elite-intel\diagnostics; Linux/macOS $XDG_DATA_HOME/elite-intel/diagnostics, falling back to ~/.local/share/elite-intel/diagnostics when XDG_DATA_HOME is unset. Resolve it ONCE at the start of the run (the one-liner also creates it) and reuse the printed absolute path, called <DIR> below:

python3 -c "import os,sys,pathlib; b=pathlib.Path(os.environ['LOCALAPPDATA']) if sys.platform=='win32' else pathlib.Path(os.environ.get('XDG_DATA_HOME') or pathlib.Path.home()/'.local/share'); d=b/'elite-intel'/'diagnostics'; d.mkdir(parents=True,exist_ok=True); print(d)"

Operations (substitute the real <DIR>, and for append the real line; the phrase is passed as an argv argument, never embedded in the Python source, so only your shell's quoting applies):

• Create the gate input.txt empty (arms diagnostics mode): python3 -c "import sys,pathlib; pathlib.Path(sys.argv[1]).write_text('',encoding='utf-8')" "<DIR>/input.txt"
• Write language.txt (boot language, code only): python3 -c "import sys,pathlib; pathlib.Path(sys.argv[1]).write_text(sys.argv[2],encoding='utf-8')" "<DIR>/language.txt" "EN"
• Clear session.log: python3 -c "import sys,pathlib; pathlib.Path(sys.argv[1]).write_text('',encoding='utf-8')" "<DIR>/session.log"
• Append ONE line — an utterance or an @directive — to input.txt: python3 -c "import sys; open(sys.argv[1],'a',encoding='utf-8').write(sys.argv[2]+'\n')" "<DIR>/input.txt" "<line>"
• Read the log tail INCREMENTALLY — pass the last byte offset (start at 0); the FIRST output line is the new offset to pass next time, everything after it is the appended log text: python3 -c "import sys; off=int(sys.argv[2]); f=open(sys.argv[1],'rb'); f.seek(off); d=f.read(); print(off+len(d)); sys.stdout.write(d.decode('utf-8','replace'))" "<DIR>/session.log" <OFFSET>
• Delete the gate at teardown: python3 -c "import sys,pathlib; pathlib.Path(sys.argv[1]).unlink(missing_ok=True)" "<DIR>/input.txt"

Launch the app with the Gradle wrapper from the repo root, in the background: ./gradlew :app:run on Linux/macOS, gradlew.bat :app:run on Windows.

Procedure

**You are the driver — do NOT build one. ** Do not write a script, harness, or any standalone program to orchestrate the run (a program cannot write natural-language utterances). You generate each utterance, then feed and observe it with plain one-off tool calls — append a line, read the new log lines, decide PASS/FAIL, repeat — using the portable one-liners in Harness I/O ( cross-platform). Those single-operation file commands are not an orchestrator; a loop-runner that generates and feeds turns on its own is. The only process you ever launch is the app itself (the Gradle wrapper). If you catch yourself authoring a loop-runner, stop and do the next step by hand.

0. Validate the language argument. Per Arguments and preconditions — on a bad/missing code, print the error and stop before touching any files, the app, or the web.

1. Resolve the expected action ids. The test references actions by Java constant (e.g. DeployLandingGearCommand.ID), but the log reports the runtime id string. Build the map once: grep public static final String ID = under app/src/main/java/elite/intel/ai/brain/actions/ to get ClassName → "id_string".

2. Parse the reference test NaturalSpeechIntegrationTest<LANG>.java. For each @ParameterizedTest method, read its assertRouted(input, <Class>.ID) for the expected class and its companion static Stream<String> <method>() for the reference aliases. Produce an ordered list of groups: { expectedId (from step 1), intentName, referenceAliases[] }. Skip startListening/ignoreMe unless asked — they are attention control, not routing.

   • --only <ids>: keep only groups whose expectedId is in the list; if an id doesn't match any group, name it and stop (likely a typo).
   • --replay-fails: skip generation entirely. Load the saved plan (runs/<LANG>-*.json, most recent unless a path was given), take the records with pass=false, and use those exact utterances as the plan — each already carries its expectedId. Jump to step 4.

3. Generate ALL utterances UP FRONT — before feeding anything. This is the point of the skill (see Terminology sourcing and Phrase generation), and one-pass generation is what keeps the run fast. For every group, infer the intent and required parameters from the reference aliases and the in-game vocabulary from the external references (NOT the app's own files), then write count fresh, natural utterances in the target language. Produce the WHOLE plan (all groups × count) at once and hold it in memory, THEN start feeding. Do not generate lazily between turns — pausing 30–120 s to think up the next line is the single biggest reason a run drags (app turns are ~1 s). These generated utterances — not the aliases — are what you feed; each utterance's expected action is its group's expectedId. (In --replay-fails mode there is no generation — the plan came from the saved file in step 2; skip straight to step 4.)

4. Set context with @visible, not by guessing. Before each group's utterances, emit @visible <expectedId>; the app puts the game in the first context where that action is visible (via its real isVisibleForLLM, exactly as the routing test does), so you never guess which @status a command needs. Confirm with DIAG visible=<id> state=<ctx>. state=unknown-action means the id is wrong (fix it); only fall back to a manual @status/@fighter if you deliberately want a non-default context.

5. Arm diagnostics mode and start the app — create the gate FIRST, then launch. Do this without asking. Use the operations in Harness I/O (cross-platform); resolve <DIR> once and reuse it.

   • Resolve and create the diagnostics directory (the resolver one-liner also does the mkdir).
   • **Create input.txt empty ** — this is the gate; its presence enables diagnostics mode. Do this BEFORE launching. (The Python one-liners write UTF-8 without a BOM, so a leading @directive or the language code parses cleanly; the app also strips a leading BOM defensively, but write clean.)
   • Write the boot language into language.txt (the code only, e.g. EN) — a separate data file, NOT @lang. The app reads it at startup and sets the command language before the companion is built. Required, because the semantic reducer freezes its language when constructed, so a later @lang never reaches it and it would miss obvious commands. Confirm with DIAG boot-language=<LANG>.
   • Clear session.log so you never mistake a previous run for this one.
   • Launch the app in the background with the Gradle wrapper (./gradlew :app:run, or gradlew.bat :app:run on Windows). The window can take ~1 min to appear; services autostart.
   • Readiness is decided only by a FRESH DIAG ready from THIS run. Because you cleared the log, it starts empty: wait for the new instance to re-open it (DIAG log opened), then a DIAG ready after it. Never treat a pre-existing DIAG ready as readiness, and do not narrate "still starting" from anything but the log. While waiting, do NOT pre-stage phrases — just wait. If DIAG ready never appears, report that the app didn't reach a ready state (check API credentials / that it's really this build).

6. Feed ONE line at a time — never in bulk. The language is already set (boot file, step 5), so no @lang is needed. Hold the whole plan in memory and emit it line by line — never pour it in with one write.

   • Per group, first append @visible <expectedId> (wait for DIAG visible=... — near-instant), then its utterances one per turn (step 7). Re-emit @visible <expectedId> for each group even if the previous differed — it is cheap and keeps context correct.
   • Append each line with the append one-liner from Harness I/O so non-ASCII is preserved. The bookkeeping comment # intent -> id (ref: ...) is for your own tracking/report — you need not write it to the file.

7. Per-utterance turn loop — feed one, wait for the whole turn, then the next. For each utterance:

   1. Append exactly that one line to input.txt.
   2. Read session.log INCREMENTALLY with the tail one-liner from Harness I/O (track the byte offset it returns; never re-slurp the whole file — that is the main way this skill wastes your tokens) until this turn is done: its DIAG input="<utterance>" appears, then its DIAG dispatch tool=<id> line(s), and finally the DIAG turn-done marker that ends the turn (fired after its dispatch line(s) and the DIAG speaking=false closing its spoken reply). Bound with a per-turn timeout (~90 s).
   3. Record PASS/FAIL (PASS iff expectedId is among that turn's DIAG dispatch tool= lines), then append the next line promptly — add no extra delay; the app already inserts a short conversational gap, so padding it makes the run drag.

   This mirrors real speech: one command, wait for the assistant to answer and stop talking, then the next. The app paces internally as a backstop, but you must still drive strictly one-at-a-time. Context is already correct via @visible, so a zero-dispatch miss is a real recall/visibility finding — do NOT cycle @status to "rescue" it. If DIAG visible= showed main_ship(fallback) or unknown-action, note that in the report; otherwise treat the miss as a genuine result for step 9.

   • Stream progress. A full run is 10–20 min of silence otherwise. After each group (or every ~10 turns), emit one terse line — group 34/90 · passed 78 · fail 6 — so the user sees it is alive and where the failures are clustering. Keep it to a single line per update, not a running transcript.

8. **Report in English ** (the project's working language for developer output — regardless of the language under test; only the generated utterances stay in the target language) a compact table: utterance | expected id | dispatched | PASS/FAIL, grouped by intent, plus totals and the failing utterances with what they routed to instead. Because the utterances are yours, before recording any FAIL sanity-check it against the Phrase generation rules: if it is genuinely ambiguous or drifted off-intent, that is a skill error — fix the utterance and re-run it, don't report it as an app failure. For a real FAIL where the reducer never offered the target, note it (the mirrored DBG/LOG lines around that turn show the reducer selection) so the user can tell a reducer/alias problem from an LLM-choice problem.

   • Classify each intent, not just each utterance (LLM routing is nondeterministic). With count ≥ 2, bucket the intent by how many of its utterances passed: PASS (all), FLAKY (some but not all), FAIL (none). A FLAKY intent is instability, not a clean bug — a single 0/1 or 2/3 result would over- or under-state it. Surface the three buckets with their ratios (e.g. deploy_landing_gear 2/3 FLAKY) so the signal is honest.
   • Persist the run to runs/<LANG>-<UTC-timestamp>.json next to this SKILL.md (same writable location as glossary/): { language, count, mode, timestamp, records: [{ intent, expectedId, utterance, dispatched, pass }] }. This is what --replay-fails reads, and it lets you diff two runs to see whether a fix actually moved the needle.

9. Improvement recommendations. Base concrete recommendations on systematic FAIL intents (0 passes) — those are real, actionable routing defects. List FLAKY intents separately as "unstable, not a clean bug": don't prescribe an llmDescription/alias edit off a single flaky turn (you'd be tuning to noise); if a FLAKY intent matters, suggest re-running it with a higher count via --only <id> to see whether it's really borderline. For each systematic FAIL, target the two owners of routing:

   • Function description — the action's llmDescription() (an intent directive, e.g. DeployLandingGearCommand.llmDescription()), which is what the companion LLM sees.
   • Training phrases — the action's aliases in ai_action_aliases_<lang>.properties (bundle i18n.ai_action_aliases, keyed by the same action id), which feed the reducer.

   Pick the recommendation from the failure mode (evidence: the reducer-selection line in the log):

   • Recall miss — the reducer never offered expectedId: the alias set under-covers this phrasing. Recommend concrete alias additions to the <expectedId> key in ai_action_aliases_<lang>.properties — full utterances carrying every required parameter, in real ED terminology, not duplicating an existing alias (respect the project's alias/probe hygiene rule).
   • Selection miss — the reducer offered expectedId but the LLM dispatched a different id X: the two descriptions don't separate cleanly. Recommend a concrete edit to <expectedId>.llmDescription() (and/or X's) that disambiguates the confusable pair — name the pair and the distinguishing cue, keep it a terse intent directive (no prose bloat).
   • No dispatch even after context retries — likely a visibility gate (isVisibleForLLM / wrong @status) or a deep recall gap; say which, and what to check.

   Group recommendations by action, dedupe, and ground every one in the logged evidence — never guess. Do not recommend editing the tests or the glossary; the fix belongs in the app's description or alias owner.

10. Tear down — delete the gate. When the run is done (or if you abort), always delete input.txt with the delete one-liner from Harness I/O. Its existence is the mode gate and the app never removes it, so leaving it would boot the next normal launch into diagnostics mode. Do this even on failure. (Do NOT delete it mid-run — its absence is only checked at startup, but keep it for the session.)