compileiq-validate-result

compileiq-validate-result

The score CompileIQ reports during a search uses N=5-15 trials per evaluation and a shared cache. That's appropriate for the search loop but wildly insufficient for shipping. This skill is the gate before any ACF goes to production.

When

• tuner.start() has returned and there's a dump_results= CSV on disk.
• User wants to claim a speedup or ship an ACF.
• A reported speedup feels too clean — validate it.

Steps

1. Load the CSV

from compileiq.results import SearchResult
 
results = SearchResult.from_csv("results.csv", problem_type="min", clear_duplicates

2. Extract candidates

Single-objective:

score_1 (with underscore-one) is the canonical key — it matches the multi-objective convention score_N. Older code sometimes uses plain score; the fallback above handles both shapes.

For top-K:

Multi-objective:

Mixed user+compiler search space: results carry separate keys — best["user_space"] for the user-side knobs, best["params"] for the ACF hex. Save both.

3. Re-measure on fresh cache (the actual validation)

Both the baseline (no ACF) and each top-K candidate are re-measured at validation N. The optimization-time measurement is too noisy to ship from.

4. Statistical gate — the ship rule

Ship rule: p_value < 0.05 AND cohens_d > 0.2 (preferably > 0.5) AND optimized.mean < baseline.mean. Anything weaker, do not claim a speedup.

5. Three false-positive patterns to actively check

6. Save the validated winner

7. Reproducibility log

Append one row per candidate decision to validation-log.csv. Fields, per docs/flashinfer_booster.md:135-148:

• timestamp (UTC ISO 8601)
• ACF filename + sha256
• manifest / release version
• benchmark command
• GPU model + driver version
• CTK version (nvcc release)
• ptxas, nvcc paths + versions
• framework version or commit (Triton / Helion / FlashInfer / cuTeDSL)
• input shape
• baseline mean ± std
• candidate mean ± std
• p-value
• Cohen's d
• decision: KEPT or REJECTED:<reason>

The scripts/welch_validate.py helper records the timing/statistical fields, ACF hash, benchmark commands, GPU/toolchain metadata, and common environment variables automatically. Pass --manifest, --framework, and --input-shape for workload-specific fields the helper cannot infer.

CLI helper

Prints KEPT or REJECTED:<reason> and appends a row to the log. Also importable: from welch_validate import validate_speedup.

Self-test

Synthesizes two identical normal distributions, asserts the statistical gate returns significant=False. Then differs them, asserts significant=True. Catches misconfigured scipy/numpy before a real validation.

Gotchas

• pareto_front() raises if num_objectives == 1. Guard with if results.num_scores > 1: or use try/except.
• score_1 vs score. Current API is score_1. Some older results exporters used plain score. The defensive read pattern best.get("score_1", best.get("score")) handles both.
• Don't validate on the same cache the search used. With CIQ_KEEP_CACHE=1 active during search, validation must explicitly wipe ~/.cache/compileiq or use a fresh TRITON_CACHE_DIR and HELION_SKIP_CACHE=1. Otherwise the optimization-time numbers re-appear and you're not validating anything.
• Validation N is independent of optimization N. Even if the search used N=5 per evaluation, validation needs N ≥ 100. Don't try to be clever and reuse search-time samples.

Next

• If the validated speedup ships: commit best.acf and validation-log.csv.
• If validation fails: compileiq-debug for diagnosis.
• For more thorough exploration: re-run compileiq-run-search with bigger pool_size/generations.