tensorrt-llm

TensorRT-LLM

NVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs.

When to use TensorRT-LLM

Use TensorRT-LLM when:

• Deploying on NVIDIA GPUs (A100, H100, GB200)
• Need maximum throughput (24,000+ tokens/sec on Llama 3)
• Require low latency for real-time applications
• Working with quantized models (FP8, INT4, FP4)
• Scaling across multiple GPUs or nodes

Use vLLM instead when:

• Need simpler setup and Python-first API
• Want PagedAttention without TensorRT compilation
• Working with AMD GPUs or non-NVIDIA hardware

Use llama.cpp instead when:

• Deploying on CPU or Apple Silicon
• Need edge deployment without NVIDIA GPUs
• Want simpler GGUF quantization format

Quick start

Installation

# Docker (recommended)
docker pull nvidia/tensorrt_llm:latest
 
# pip install
pip install tensorrt_llm==1.2.0rc3
 
# Requires CUDA 13.0.0, TensorRT 10.13.2, Python 3.10-3.12

Basic inference

Serving with trtllm-serve

Key features

Performance optimizations

• In-flight batching: Dynamic batching during generation
• Paged KV cache: Efficient memory management
• Flash Attention: Optimized attention kernels
• Quantization: FP8, INT4, FP4 for 2-4× faster inference
• CUDA graphs: Reduced kernel launch overhead

Parallelism

• Tensor parallelism (TP): Split model across GPUs
• Pipeline parallelism (PP): Layer-wise distribution
• Expert parallelism: For Mixture-of-Experts models
• Multi-node: Scale beyond single machine

Advanced features

• Speculative decoding: Faster generation with draft models
• LoRA serving: Efficient multi-adapter deployment
• Disaggregated serving: Separate prefill and generation

Common patterns

Quantized model (FP8)

Multi-GPU deployment

Batch inference

Performance benchmarks

Meta Llama 3-8B (H100 GPU):

• Throughput: 24,000 tokens/sec
• Latency: ~10ms per token
• vs PyTorch: 100× faster

Llama 3-70B (8× A100 80GB):

• FP8 quantization: 2× faster than FP16
• Memory: 50% reduction with FP8

Supported models

• LLaMA family: Llama 2, Llama 3, CodeLlama
• GPT family: GPT-2, GPT-J, GPT-NeoX
• Qwen: Qwen, Qwen2, QwQ
• DeepSeek: DeepSeek-V2, DeepSeek-V3
• Mixtral: Mixtral-8x7B, Mixtral-8x22B
• Vision: LLaVA, Phi-3-vision
• 100+ models on HuggingFace

References

• Optimization Guide - Quantization, batching, KV cache tuning
• Multi-GPU Setup - Tensor/pipeline parallelism, multi-node
• Serving Guide - Production deployment, monitoring, autoscaling

Resources

• Docs: https://nvidia.github.io/TensorRT-LLM/
• GitHub: https://github.com/NVIDIA/TensorRT-LLM
• Models: https://huggingface.co/models?library=tensorrt_llm