OverviewHistoryStatsSecurity
npx skills add ...
Documentation
SKILL.md
npx skills add nvidia/tensorrt --skill trt-onnx-quickstart
Build and verify a TensorRT engine from a Hugging Face model ID or ONNX file, with numerical parity checked against ONNX Runtime. Use when the user imports a non-LLM model to TensorRT, needs a verified engine from ONNX, hits trtexec "unsupported operator", must verify the engine matches ONNX numerically, debugs a polygraphy parity failure (large max abs diff at FP16), or configures multi-input dynamic shapes. Triggers: convert ONNX to TensorRT, Hugging Face to TensorRT, trtexec onnx, trtexec unsupported operator, optimum-cli export, polygraphy parity check, polygraphy run --trt --onnxrt, parity check failed, max abs diff, verify engine matches ONNX, --minShapes, dynamic shapes trtexec, multi-input shape profile, FP16 engine, INT64 warning. Adjacent skills: `trt-torch-quickstart` (PyTorch frontend), `trt-cpp-runtime-quickstart` (C++ engine load). LLM token generation belongs in TensorRT-LLM, not here.
npx skills add nvidia/tensorrt --skill trt-onnx-quickstart
Take a developer from "I have a Hugging Face model ID" or "an ONNX file" to "a TensorRT engine whose outputs match the source model within tolerance." Follows Path 1 (ONNX → TensorRT) of the Import Workflows Guide, specialized for the most common starting point: a Hugging Face Hub model.
| Scenario | Use this skill? |
|---|---|
Has a Hugging Face model ID (google-bert/bert-base-uncased) and wants TRT-accelerated inference | Yes |
Has an .onnx file and wants a .plan engine with verified parity | Yes |
Ran trtexec --onnx=... and hit a warning/error they don't recognize | Yes |
| Wants LLM token generation (Llama, Mistral, Qwen text generation) | No — route to TensorRT-LLM |
| Has a PyTorch model and wants to stay in PyTorch | No — use trt-torch-quickstart |
Already has a .plan engine and wants to run inference from C++ | No — use trt-cpp-runtime-quickstart |
| Is migrating a weakly-typed network to strongly-typed | No — use trt-strong-typing-migration |
| Has a full diffusion pipeline (SD, FLUX) | Partial — must be component-split first; this skill imports one component at a time |
Verify each before continuing. Failures here surface as confusing errors later, so fix them before running the import.
NVIDIA GPU + driver matching TensorRT 11.x. TRT 11 requires CUDA 13.x. Confirm:
Python ≥ 3.10. TRT 11 dropped 3.9 and earlier.
TensorRT 11 installed — including the trtexec CLI, this skill's primary build tool. The pip wheel does not ship trtexec. Pick one:
nvcr.io/nvidia/tensorrt:<tag> — bundles trtexec, Python bindings, libraries, and most common dependencies. Run with --gpus all..tar.gz or .deb from TensorRT downloads: installs trtexec to a system path.The pip path (pip install --extra-index-url https://pypi.nvidia.com tensorrt-cu13) provides Python bindings but not trtexec, so it is insufficient on its own. For pip-only, route the developer to the alternative Python-builder flow in import_workflows.md Path 1 Option B; this skill assumes trtexec is available.
Verification:
Polygraphy and optimum-onnx installed.
onnx_graphsurgeon is required by polygraphy surgeon sanitize (Step 3) — Polygraphy does not bundle it. onnxruntime is required by optimum-cli export (Step 2, to fix dynamic axes) and Polygraphy's --onnxrt parity check (Step 5); stock NGC containers do not always ship it.
Note: optimum-onnx is the active maintained package — the older optimum no longer includes ONNX integration. Do not use optimum-nvidia (last release 2025-01, unmaintained).
Consult the Supported Models matrix. If listed, the dtype column gives a known-good starting precision. If not, it is still expected to work — file an issue if it doesn't.
If the model is a diffusion pipeline, stop here — TRT cannot ingest the whole pipeline object. Split it into components (text encoder, UNet/DiT, VAE) and import each separately. See supported_models.md for validated splits, and consider whether trt-export-rewrite is a better fit.
This skill's worked example uses google-bert/bert-base-uncased from the encoder-NLP section.
The optimum-cli export onnx command's --task flag selects which head of the model to export. The wrong task yields a valid graph but the wrong output — and Step 5 parity validation won't catch it, because reference and engine are both wrong.
Pick the task from this table based on what the developer wants to do with the model:
| Developer's intent | --task | Typical model families |
|---|---|---|
| Get hidden-state embeddings or pooled output | feature-extraction | BERT, RoBERTa, sentence-transformers, CLIP |
| Classify images into N classes | image-classification | ResNet, ViT, MobileNet, EfficientNet |
| Classify text into N labels | text-classification | BERT for sentiment, DistilBERT, etc. |
| Detect / segment objects | object-detection, image-segmentation | DETR, Mask R-CNN |
| Transcribe speech | automatic-speech-recognition | Whisper, Wav2Vec2 |
| Generate captions / VQA | image-to-text, visual-question-answering | BLIP, LLaVA |
| Translation / summarization | text2text-generation | T5, BART |
For an unknown model, run optimum-cli export onnx --help or check the Optimum supported tasks table. If unsure, surface the choice to the developer rather than guessing.
If the model ID starts with sentence-transformers/ (e.g. all-MiniLM-L6-v2, bge-base-en-v1.5), install sentence-transformers before exporting:
Without it, optimum-cli falls back to plain transformers and emits raw token-level last_hidden_state instead of the pooled sentence embedding the model is designed to produce. The exporter prints library name was inferred as sentence_transformers, which is not installed. Falling back to transformers. — treat that warning as a hard error and install before retrying.
The exporter often prints a "max diff between reference and ONNX exported model is not within the set tolerance 1e-05" line (with max diff around 1e-5 to 1e-4). This is its own self-check at a strict tolerance — the export still succeeded. Step 5's Polygraphy parity check uses --atol 1e-2 --rtol 1e-2 --check-error-stat mean, the correct tolerance for FP16 inference. Don't surface this warning as a failure.
LLaMA, FLUX, and some Mistral and Qwen variants are gated behind HF agreements and require huggingface-cli login before download. If optimum-cli exits with a 401, run huggingface-cli login with a token from huggingface.co/settings/tokens and retry.
The most common cause is a HF model using patterns torch.export / torch.onnx.export cannot trace — complex-number arithmetic, data-dependent control flow, non-tensor forward arguments, output dataclasses. Do not patch the upstream library on disk. The correct workflow is agentic monkey-patching at runtime: route the developer to the trt-export-rewrite skill, which drives it with five concrete patterns from the Qwen-Image case study.
ONNX exporters often produce graphs with constant-folding opportunities, spurious dynamic axes, or shape-inference gaps. Sanitizing cleans these up, but it is a quality pass, not a correctness gate — TRT parses many un-sanitized exports fine. Both stages below are therefore optional and allowed to fail: each falls through to its input, and Step 3 always leaves a model.clean.onnx for the rest of the workflow (worst case, a copy of the exported model). Run:
Every step from here uses input names and ranks that depend on the model. Read them out of the sanitized ONNX before building. Do not assume input_ids / attention_mask — those are BERT-family-specific.
Example output for bert-base-uncased (verified against optimum-onnx 0.1.0, opset 18):
Inputs and outputs vary per model and per --task. Inspect, don't assume.
Build the shape strings from this output. Every input with a dynamic (named) axis needs a --minShapes / --optShapes / --maxShapes entry; fixed axes are reproduced literally. Use representative values — --opt is what the engine optimizes for; --min and --max define the supported range.
If any input reports an unknown-rank axis (printed as ?), shape inference failed; reroute through trt-export-rewrite to fix the export.
Start with FP16 — the safe default for encoder and vision models on modern GPUs. (Use BF16 on Blackwell/Hopper if the developer reports persistent FP16 accuracy issues. Sentence-embedding models sometimes prefer FP32 over lossy FP16 — surface the option.)
TRT 11 removed the weak-typing builder flags (--fp16, --int8, --bf16, BuilderFlag::kFP16, etc. — see the migration guide and the sibling trt-strong-typing-migration SKILL). In TRT 11 you bake precision into the model before building, and trtexec honors the dtypes it finds via ModelOpt AutoCast.
Detect the version first:
Worked example for BERT (three inputs, all dynamic on batch and sequence axes), TRT 11 path:
Worked example for ResNet-50 (one input, dynamic on batch axis only), TRT 11 path:
For purely static-shape models, omit all three shape flags.
--memPoolSize=workspace:N. If still failing, try --tacticSources=-CUBLAS_LT to disable expensive tactic sources.trtexec names the op. Route to trt-unsupported-op for triage (decompose / plugin / switch frontend / file bug).polygraphy surgeon sanitize --fold-constants.--minShapes / --optShapes / --maxShapes flags. Compare against Step 3.5's output and add the missing entry.This is the step most developers skip — and the one that most often surfaces a silent correctness bug.
Construct the parity command from the same input names used in Step 4. Beyond input shapes, three knobs matter; get them right or the result is misleading:
--val-range for any input with constrained semantics. Polygraphy fills with [0, 1) floats by default. For BERT-family input_ids you must constrain to vocabulary range, set attention_mask to all-ones, and token_type_ids to all-zeros — otherwise the model sees garbage and FP16 noise amplifies through the attention softmax.--check-error-stat mean for transformer outputs. FP16 commonly has a few outlier positions with large abs diff; the mean across all positions is the meaningful metric for downstream use (embedding similarity, classification). Default max is too pessimistic for FP16 transformers and produces false failures.--seed for repeatability across runs and CI environments.Worked example for BERT — run polygraphy against the same ONNX file you fed to trtexec (the AutoCast-converted .fp16.onnx on TRT 11; the .clean.onnx on TRT 10):
Expected output: Pass Rate: 100.0% with mean_absdiff ≤ 1e-3 per output. Pass Rate: 0.0% with max_absdiff near 1e-2 but mean_absdiff well below tolerance is the outlier pattern above — use --check-error-stat mean.
Worked example for ResNet-50 (same FP16 recipe; TRT-11 path shown — for TRT 10 add --fp16 and read from model.clean.onnx):
Polygraphy runs the model through ONNX Runtime (reference) and TensorRT (engine) on the seeded inputs and compares outputs per the chosen error stat.
--strongly-typed with an explicit FP32 cast on the offending subgraph (Polygraphy names the layer). For LLM-style models, prefer BF16 over FP16 on Hopper/Blackwell.polygraphy inspect model for shape-inference issues. If the model uses patterns covered by trt-export-rewrite, reroute.--minShapes / --maxShapes and rebuild.This confirms the saved .plan deserializes and runs (Step 5's parity used its own engine, not this file) and reports throughput and per-iteration latency. For production, see the developer guide on IExecutionContext and stream-aware execution; the .plan is portable to any compatible GPU + TRT runtime.
The skill is complete when all of the following are true:
bert.fp16.plan exists and trtexec --loadEngine=... runs without error.polygraphy run --trt --onnxrt --atol 1e-2 --rtol 1e-2 --check-error-stat mean reports PASSED across all configured input shapes.If any are not true, hand control back with the specific failing diagnostic — do not declare success.
samples/sampleOnnxMNIST/ for the C++ equivalent of the build flow