perf-optimization-casebook

Performance Optimization Casebook

A curated library of classic and previously-successful TensorRT-LLM optimizations, written as decision precedents. Use it to answer "for this bottleneck / config / model / hardware, what has worked before, and how do I adapt it?" — instead of inventing an approach from scratch.

This skill is reference material, not a coordinator. It does not run profiling, edit code, or route to specialists by itself. It is consulted by the coordinators (perf-analysis after a bottleneck is classified, perf-optimization before prioritizing/routing) and by the perf-sweep system when choosing what to try next. Implementation always goes through the relevant specialist/skill named in each case.

When to Consult

• A bottleneck has been classified (compute / memory / launch / communication / sync) and you need candidate optimizations for that class.
• You are about to prioritize or route an optimization and want to check for prior art, known pitfalls, and the right specialist.
• You are choosing the next config to try in a serving sweep.
• You just landed a verified win and want to record it for future reuse.

How to Use a Case — the loop

1. Classify. Know the current bottleneck class and the workload context (model arch, hardware/SM, ISL/OSL, concurrency, current config). If you do not have this, get it from perf-analysis / profiling first.
2. Match. Search wide with free text, then rank — never let the query vocabulary narrow what is reachable:

   • Recall (grep the whole case files, free text): grep the case bodies with your own natural query terms — model name, knob, mechanism word, deployment feature — over the full files, not just structured fields:

     grep -ril "lora" references/                 # any term, whole file
     grep -ril "chunked prefill" references/

     Every concept a case covers lives in its prose (Applies when / Generalizes to / mechanism), so a free-text grep over the whole file is recall-complete on its own. Frontmatter is additive structure layered on top — it never removes a word grep could already find. Do not pre- translate the query into canonical vocabulary before searching; that is what would make grep miss LoRA / beam search / a reasoning model's thinking phase and other concepts that have no canonical signal. Browse the family index too if you want a curated table, but grep alone does not miss.

   • Rank (optional accelerator): among the hits, optionally use the frontmatter fields (canonical signals/pattern grep — see "Frontmatter fields" below) to decide which case file(s) to open first. This only reorders the free-text hits — it never defines or bounds them.

   • Then open only the winning case file(s). Match on signals, not on the optimization name.

3. Adapt — do not copy. A case is a precedent, not a guaranteed config. Adjust knob values to the current model/hardware; read the nearest checked-in config and the model's deployment guide before setting numbers.
4. Verify. Apply through the named specialist/skill, then measure. Numbers come from real measurement only (see Principles). A precedent's expected direction of effect does not excuse skipping measurement.
5. Record. If the change is a verified win (or a clear, instructive failure), add or update a case so the next decision is better informed (see "Adding a New Case").

Frontmatter fields (optional precision — not a search step)

Cases carry YAML frontmatter (vocabulary: data/tags.yaml; spec: references/case-template.md). It is additive structure for ranking and judging the free-text hits from Match step 2 — never a second search you must run, and never a filter that bounds what grep already found. Use it as needed: grep a canonical signals/pattern id (data/patterns.yaml) to rank hits, and read a candidate's eligibility: / interactions: before recommending (respect incompatible-with; follow depends-on to the producer case first). A synonym won't hit these fields (they store canonical terms only) — that is fine, because the free-text grep already found the case; never emit no-match from an empty field grep.

Legacy (not-yet-migrated) cases have no frontmatter — for those, fall back to the family index tables. When adding or editing a migrated case, keep the contract by hand: frontmatter values must be canonical terms from data/tags.yaml (never synonyms), and case↔pattern pointers must stay bidirectionally consistent (patterns: in the case ↔ instances: in data/patterns.yaml).

Consultation trace (machine-parseable — always emit)

Every consultation MUST end with exactly ONE trace line in your visible reply text, as soon as the Match step resolves:

CASEBOOK: matched=<family>/<case-slug> confidence=<high|medium|low> adapted="<one-line adaptation>"

or, when no precedent fits:

CASEBOOK: no-match query="<bottleneck class / signals you searched for>"

<case-slug> is the case file name without .md. Emit the line even when the consultation is a dead end: it is the join key for automated audit trails (which precedents were consulted vs cited vs hallucinated), and the no-match rate is the casebook's coverage metric for maintainers. One line per consulted question; never summarize multiple matches into one line. A no-match line asserts your free-text recall grep over the whole case files (Match step 2) turned up nothing relevant — it must never be emitted merely because a canonical-field grep returned empty.

Case Families — routing table

Open the family index relevant to your classified situation, then drill into a single case file from its case table (each family is a directory of one-file-per-case behind an index.md of patterns + a case-picker table).

Many runtime/kernel/communication cases were distilled from the TRT-LLM DeepSeek-R1/V3 serving stack (MLA attention, fine-grained MoE, MTP, attention-DP + expert-parallel), where these levers cluster — but each is recorded as a transferable pattern (see Generalizes to); match on signals, not on the DeepSeek origin. Each carries a Commits line so you can git show the source.

Quick index — bottleneck signal → where to look

Routing shortcut once the bottleneck is classified. Match the signal, open the family index, then match a case's Applies when / Generalizes to in its case table and open that case file.

Case Schema

Every case records these fields (see references/case-template.md for the canonical form):

• Commits (optional) — primary <hash> subject (#PR) plus folded-in related: follow-ups, when the case is distilled from specific commits, so a reader can git show the source. Omit when there is no upstream commit.
• Applies when — bottleneck class + signals (model arch, hardware/SM, workload shape, current config) that make the case relevant.
• Counter-signals — when NOT to apply: auto-disable regimes, degenerate shapes, negligible-share workloads, unmet requirements. Check this before recommending a matched case.
• Mechanism — why it helps, in one or two lines.
• Generalizes to — the transferable pattern this case instantiates and the adjacent situations it carries to (other model/arch, neighbouring op-chains, different knob values), plus what to change when adapting. This is what an agent matches on when it does not hit the exact instance — the casebook's main job. Match on this pattern, not the case title.
• Apply via — the exact config knob and/or the specialist/skill to delegate implementation to. Never a hand-written runnable config invented here.
• Expected effect — qualitative direction and the metric that should move. Quantitative deltas only when measured (cite the run) — see Principles.
• Accuracy risk — lossless, lossy, or mixed (a case bundling moves of differing risk — it must state which part is lossy). Lossy cases require an on-disk accuracy record plus a rollback criterion before being promoted as a "best" config (mirrors the perf-sweep correctness gate).
• Verify — what to measure to confirm the win and (for lossy) parity.
• Rollback — how to revert and the regression trigger that mandates it.
• Prior art — pointers to the owning skill, TRT-LLM repo docs, or PRs.

Principles

1. Adapt, don't copy. Cases are precedents. Knob values, hardware constraints, and model arch differ — re-derive numbers from the nearest checked-in config and the model guide. Treat any value carried over from a case as unverified until measured here.
2. Never fabricate numbers. Quantitative before/after figures must come from real profiling/benchmark output, with the source cited. A case states the direction of effect and the metric to watch; it does not assert a speedup that was not measured. Do not present casebook expectations as achieved results.
3. Respect the accuracy gate. Optimizations split into:
   • Lossless (output-equivalent): scheduling/batching/graph knobs, and backend selection (attn_backend, moe_config.backend — same math, different implementation). Safe to evaluate on throughput/latency alone.
   • Lossy (output-affecting): KV-cache quant (kv_cache_dtype fp8/int8/ nvfp4), weight/GEMM quant (quant_algo), low-precision MoE combine, low-precision EP dispatch/combine, relaxed spec-decode acceptance — and PDL (TRTLLM_ENABLE_PDL), which changes inter-kernel ordering/visibility and has produced real NaN/accuracy regressions per kernel (verify per kernel). Spec/MTP is output-equivalent by design but can drift on acceptance/draft bugs, so verify on first enable. Lossy wins need an accuracy record + rollback criterion, never promoted as "best" without it. This split matches perf-sweep-workflow core/gate.py — keep them aligned.
4. One change at a time, measure each. Combined changes hide regressions and confound attribution. Roll back on >5% slowdown or any accuracy regression outside tolerance.
5. Interface facts are as-of their pinned commit. Knob names, config bounds, file paths, and function names in a case describe the code at the case's commits: — they are historical anchors, not present-tense truths. Before acting on them (especially machine-evaluable eligibility: entries), re-verify against YOUR TRT-LLM checkout; cases point to where the live value lives (e.g. a file::symbol guard). A mismatch means the code moved since the pin — prefer the live source. Mechanism and pattern statements ("why it helps") are not version-bound; only interface facts are.

Adding a New Case

After a verified win (or an instructive failure):

1. Pick the family directory (references/<family>/); if none fits, propose a new family dir + index.md and add a row to the routing table above.
2. Create references/<family>/<slug>.md from the schema in references/case-template.md — H1 title + the breadcrumb line + every field bullet — then add one row to that family index's case table (Case → file, Applies-when, Generalizes-to, Risk), and, if it instantiates a listed pattern, link it from that pattern's _(Instance: …)_. Put real measured deltas with the source run; otherwise write "measured Δ to be recorded from ".
3. Keep it a decision precedent — applicability, mechanism, knob/specialist, risk, verification, rollback. Fill Generalizes to with the pattern the case instantiates and 2–4 adjacent situations it carries to: this is what lets an agent reuse the case when its situation is a variation, not the exact instance (match on the pattern, not the title). Do not re-explain subsystem architecture (how MoE/attention/KV cache work); link to the TRT-LLM repo developer guides for that (see docs/design/agent_toolkit_organization.md §2.3).
4. Cross-link the owning specialist/skill so implementation stays delegated.

Ingestion discipline (agent-scanned corpus)

This corpus is scanned and written by agents. Selection — "is this a transferable optimization precedent?" — stays with the writing agent's judgment; the rules below target known agent failure modes (fabricated detail, plausible over-generalization, cross-run divergence), not judgment.

• Trace, don't recall. Every number in a case (bounds, caps, shapes, dims) must be traceable to a specific line in a cited commit's diff or the pinned source; every model/arch applicability claim must be checked against the modeling source (e.g. the routing-method enum), never inferred from model-family resemblance. If you didn't look it up, don't write it.
• Fold vs. new. A follow-up PR that iterates the same decision (same pattern, same knob/kernel surface) folds into the existing case as a related: commit — a PR series refining one optimization is ONE case. A new case requires new applicability signals or a new pattern.
• Stub threshold. If the four core slots (applicability signals, mechanism, transferable pattern, apply-via) can be at least roughly filled, record the case with maturity: stub so it enters the match surface. If not even roughly — skip it entirely; do not park half-guesses in the corpus.
• Verify before landing. A case is added or upgraded only after an independent fact-check pass against its cited commits (quantitative claims, knob names, file paths, model-applicability claims). The writing agent's recall is not evidence.
• Scan cursor. Sweeps of TRT-LLM main record the last commit scanned here and resume from it. Deduplicate against the corpus by grepping the hash across case commits:/Commits: fields before distilling. Current cursor: not yet recorded — the pre-pilot corpus was swept ad hoc; set this at the next sweep.

Relationship to Other Skills

• perf-analysis classifies the bottleneck → this casebook supplies candidate optimizations for that class.
• perf-optimization prioritizes and routes → consult this casebook for prior art and the right specialist before delegating.
• perf-sweep-workflow chooses configs to try → cases inform hypotheses; the deterministic core still validates and the challenger still gates.
• Implementation is always delegated to the specialist/skill named in the case (kernel-, perf-torch-cuda-graphs, perf-torch-sync-free, perf-host-optimization, trtllm-serve-config-guide, etc.).