compileiq-debug

compileiq-debug

A symptom-indexed cheat sheet. Find the row that matches what the user is seeing, follow the first action, then dig into the matching detail section.

Symptom table

Details

Socket timeout / hang on first eval

Not BLAS. Do not send users to symlink libblas.so.

Current shipped binaries link only libm/libc/libstdc++/libgcc_s and do not require BLAS/LAPACK.

Real causes today, in order of frequency:

1. CIQ_SOCKET_TIMEOUT too low. Default is 20 seconds, which is fine for small search spaces but fails on big ones. Raise to 120 first; raise to 300+ for very large spaces.
2. Release-backed search-space fetch is slow or blocked. First call to PtxasSearchSpace().retrieve() downloads from github.com. On a corporate firewall this can stall. Pre-stage the mirror:

   gh release download search-spaces-latest -R NVIDIA/CompileIQ -D /shared/mirror
   export CIQ_SEARCH_SPACES_DIR=/shared/mirror
3. forkserver is unsupported on the host. Set CIQ_PROCESS_MODE=spawn. IsoMultiProcessWorker already uses fork by default.

Every eval returns INVALID_SCORE

Sanity-check the shape before assuming the worst:

sample = tuner.sample(1)[0]
score = objective(sample)
print(type(score), score)

Common shape mismatches:

• num_objectives=1 but objective returns a tuple (latency,). Drop the trailing comma.
• num_objectives=2 but objective returns a scalar.
• Correctness gate is rejecting everything because the reference call itself is wrong.
• task_timeout is shorter than a clean compile takes; raise it.

Every eval returns the SAME score

Almost always a framework cache serving a stale binary. Run the O0/O3 canary from the Debug pack to confirm — see compileiq-booster-pack for the exact test. If O0 doesn't regress vs baseline, the ACF is not reaching PTXAS. Fix:

TypeError around fromhex

Legacy pattern from the pre-2026 skill set:

# OLD — DO NOT USE
def objective(config_blob):
    with open(tmp_path, "wb") as f:
        f.write(bytes.fromhex(config_blob))
    ...

Replace with:

from compileiq.utils.helpers import save_compiler_config
 
def objective(config: str):
    save_compiler_config(tmp_path, config)
    ...

save_compiler_config does the bytes.fromhex internally. See compileiq-author-objective for the full pattern.

"Not in expected format"

The objective returned a shape CompileIQ's core doesn't expect. Rules:

• num_objectives=1: objective must return a single scalar (int | float). Not a 1-tuple, not a list.
• num_objectives>=2: objective must return a tuple or list of that length.

result = objective(sample)
assert (
    (search_config.num_objectives == 1 and isinstance(result, (int, float)))
    or (search_config.num_objectives  > 1 and len(result) == search_config.num_objectives)
), f"shape mismatch: {result!r} vs num_objectives={search_config.num_objectives}"

Convergence stalled

Three causes, in order:

1. Pool too small for the space. pool_size = max(2 * num_objectives + 1, 32) is the auto-derived floor — for spaces with >1k design points, raise to 64-128.
2. Mutation rate too low. Default mutate_rate=0.25. Raise to 0.3-0.5 if the search is converging on the first generation.
3. The kernel is genuinely near-optimal. Verify by running Search.sample(20) and timing each sample by hand — if the spread is <5%, the search space is shallow.

Increasing invalid rate

Probably a mutation arm spreading a structurally-bad config across the population. Re-run with CIQ_KEEP_CACHE=1 so the failing configs are preserved at ~/.cache/compileiq/, then replay them by hand to identify the common factor.

High CV% on validation

If cv = std/mean > 10%, validation can't tell the signal from the noise.

Fixes, in order:

1. Lock GPU and memory clocks (see compileiq-run-search for the nvidia-smi --lock-*-clocks snippet).
2. Pin CUDA_VISIBLE_DEVICES=<gpu> so the validation has the GPU to itself.
3. Watch nvidia-smi dmon -i <gpu> -s pucvm for thermal throttling events.
4. Raise warmup count (50 → 100) and trial count (100 → 200).
5. Switch from cudaEvent to NVBench (entropy-based stopping criterion, cold-cache between samples).

CIQ_KEEP_CACHE for post-mortem

When a search misbehaves, re-run with:

export CIQ_KEEP_CACHE=1

The cache at ~/.cache/compileiq/ is preserved after the run. You can:

• Inspect each generation's serialized configs.
• Replay a specific config without paying for a whole new search.
• Compare two runs by diffing their cache directories.

Diagnose from the dump_results CSV

Quick pandas snippet:

import pandas as pd
df = pd.read_csv("results.csv")
df["score_numeric"] = pd.to_numeric(df["score_1"], errors="coerce")
 
gen_summary = df.groupby("generation").agg(
    n=("score_numeric", "size"),
    invalid=("score_numeric", lambda s: s.isna().sum() + (s > 1e10).sum()),
    best=("score_numeric", "min"),
)
print(gen_summary)

If invalid doesn't decrease across generations, your search is structurally broken — try the O0/O3 canary in compileiq-author-objective.

For an automated version: python scripts/diagnose_csv.py results.csv.

NCU section (replaces old /compileiq-profile)

Profile only after a validated ACF candidate exists. Don't profile every config — it's slow.

Two-shot pattern

# Baseline (no ACF)
ncu --set full -o baseline -f --kernel-name my_kernel python bench.py
 
# ACF-applied — match the injection your objective uses
# Raw PTXAS:
ncu --set full -o opt -f --kernel-name my_kernel \
    bash -c 'PTXAS_OPTIONS="--apply-controls=best.acf" python bench.py'
# NVCC build:
nvcc -Xptxas --apply-controls=best.acf bench.cu -o bench && \
    ncu --set full -o opt -f --kernel-name my_kernel ./bench
 
# Diff
ncu --import baseline.ncu-rep --import opt.ncu-rep --csv --page raw > diff.csv

Five metrics that explain most CompileIQ wins

If the ACF moved any of these meaningfully, that's the mechanism. If none of them moved but the win is real, look at lower-level metrics (warp stalls, issue slot utilization) — those are harder to interpret but often the answer.

Register-spill check (no NCU needed)

ptxas -v -arch=sm_100 --apply-controls best.acf kernel.ptx 2>&1 \
    | grep -E "registers|spill|stack"

Reports Used N registers, X bytes stack frame, Y bytes spill stores, Z bytes spill loads. If Y + Z goes up vs baseline, the ACF traded register pressure for memory traffic — sometimes a real win, sometimes not. Investigate before shipping.

Self-test

python scripts/diagnose_csv.py --self-test

Synthesizes a small results.csv covering each pathology (clean convergence, rising invalid rate, stalled best-score) and asserts the heuristic classifications match.

Explicitly dropped from this skill (vs old /compileiq-debug + /compileiq-profile)

• BLAS symlink section — obsolete; current binaries don't link BLAS (verified by ldd). Carrying it forward sends users on a wild goose chase.
• Verbose COMMON_PTXAS_ERRORS dict mapping individual ptxas error strings to fixes — users get INVALID_SCORE instead, no need to recognize specific messages.
• Duplicate CUDA 13.3+ check — lives in compileiq-bootstrap.
• validate_objective_function introspection helper — replaced by Search.sample(1) + the Debug-pack O0/O3 canary.

Next

• For the canary itself: compileiq-booster-pack or compileiq-author-objective.
• For environment issues: compileiq-bootstrap.
• After fixing: re-run compileiq-run-search.*