Compileiq Validate Result

What it does

When your agent uses it

• Validate result
• Extract best config
• Is my speedup real

Example prompts

• “s t-test plus Cohen”
• “validate result”
• “extract best config”
• “/compileiq-validate-result”

Requirements

• Python 3
• Pre-approved tools (allowed-tools): Bash, Read

Workflow steps

7 steps, taken from the step headings in SKILL.md.

1. 1Load the CSV
2. 2Extract candidates
3. 3Re-measure on fresh cache (the actual validation)
4. 4Statistical gate — the ship rule
5. 5Three false-positive patterns to actively check
6. 6Save the validated winner
7. 7Reproducibility log

What it can do on your machine

Read from SKILL.md and the folder at commit 743aca4. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves these tools, so the agent can use them without asking each time:

  • Bash
  • Read

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Ships 1 file in scripts/ (Python), which the agent can run.

  Shell commands in SKILL.md call:

  • python

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Compileiq Validate Result loads about 2.2k tokens when it runs. Until then it costs about 161 tokens; SKILL.md has 578 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~161

When it runs · the whole SKILL.md, loaded when a task matches

~2.2k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

• NotePre-approves every shell command (allowed-tools: Bash)SKILL.md

  allowed-tools: Bash, Read

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from NVIDIA/CompileIQ at commit 743aca4, republished under its Apache-2.0 licence (© NVIDIA). 578 words, ~2,167 tokens.

name

compileiq-validate-result

description

Use AFTER a Search has completed and BEFORE claiming any speedup or shipping an ACF. Loads the dump_results CSV, extracts top-K candidates (single-objective) or the Pareto front (multi-objective), re-measures each against the no-ACF baseline with 100+ trials on fresh caches, runs Welch's t-test plus Cohen's d, rejects three classic false-positive patterns (lucky-min / higher-variance / multiple-comparisons-of-N), and saves the validated winner as best.acf. Triggers on "validate result", "extract best config", "Welch's t-test", "is my speedup real", "save best ACF", "pareto front", "claim speedup", "ship config".

allowed-tools

Bash, Read

when_to_use

- tuner.start() returned and there's a results CSV. - User wants to ship an ACF to production. - Reported speedup feels too good to be true. Don't use when…

license

Apache-2.0

metadata.version

1.0.0

metadata.author

NVIDIA CompileIQ

metadata.domain

compiler-optimization

paths

**/*.csv, **/*.acf, **/*.py

from compileiq.results import SearchResult

results = SearchResult.from_csv("results.csv", problem_type="min", clear_duplicates=True)
df = results.get_results()
print(f"{len(df)} evaluations across {df['generation'].max()+1} generations")

best = results.get_best_result()
# dict: {metadata, generation, score_1, params, [norm_score_1]}
score = best.get("score_1", best.get("score"))   # legacy defensiveness
acf_hex = best["params"]                          # hex string

import pandas as pd
df_valid = df[pd.to_numeric(df["score_1"], errors="coerce") < 1e10]
top_k = df_valid.nsmallest(5, "score_1")          # nlargest for MAX problems

front = results.pareto_front()   # raises if num_objectives == 1
for candidate in front:
    print(candidate["score_1"], candidate["score_2"], candidate["params"])

import numpy as np
from scipy import stats

def validate_speedup(baseline_ms: np.ndarray, optimized_ms: np.ndarray) -> dict:
    t, p = stats.ttest_ind(baseline_ms, optimized_ms, equal_var=False)   # Welch's
    b_mean, b_std = baseline_ms.mean(),  baseline_ms.std(ddof=1)
    o_mean, o_std = optimized_ms.mean(), optimized_ms.std(ddof=1)
    pooled = np.sqrt((b_std**2 + o_std**2) / 2)
    d = (b_mean - o_mean) / pooled if pooled > 0 else 0.0
    return {
        "speedup_mean":  b_mean / o_mean,
        "speedup_median": np.median(baseline_ms) / np.median(optimized_ms),
        "p_value":       float(p),
        "cohens_d":      float(d),
        "significant":   bool(p < 0.05 and o_mean < b_mean and d > 0.2),
        "baseline":  {"mean": b_mean, "std": b_std,
                      "p5": np.percentile(baseline_ms,  5),
                      "p95": np.percentile(baseline_ms, 95)},
        "optimized": {"mean": o_mean, "std": o_std,
                      "p5": np.percentile(optimized_ms,  5),
                      "p95": np.percentile(optimized_ms, 95)},
    }

from compileiq.utils.helpers import save_compiler_config
save_compiler_config("best.acf", best["params"])

# Mixed search spaces: persist the user_space knobs separately
if "user_space" in best:
    import json
    Path("best.user_space.json").write_text(json.dumps(best["user_space"], indent=2))

python scripts/welch_validate.py \
    --acf best.acf \
    --baseline-cmd "python bench.py --routine matmul" \
    --opt-cmd "PTXAS_OPTIONS='--apply-controls=best.acf' python bench.py --routine matmul" \
    --trials 100 --warmup 50 \
    --score-regex 'mean: ([0-9.]+)' \
    --manifest booster-packs-YYYY.MM.DD \
    --framework "flashinfer <version>" \
    --input-shape "routine=matmul, M=..., N=..., K=..." \
    --output validation-log.csv

python scripts/welch_validate.py --self-test

© NVIDIA, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 1 other file (scripts) in agent-skills/compileiq-validate-result of NVIDIA/CompileIQ.

• SKILL.md
• scripts/welch_validate.py

Open the folder on GitHubat commit 743aca4