TensorRT-LLM: High-Throughput LLM Inference on NVIDIA GPUs

TensorRT-LLM is NVIDIA's open-source library for optimizing large language model inference on their GPUs. You would reach for it when you need maximum throughput and low latency on A100, H100, or GB200 hardware, especially with quantized models like FP8 or INT4. It compiles models into optimized engines, trading setup complexity for raw performance.

What it does

TensorRT-LLM takes a trained LLM and compiles it into a TensorRT engine that runs efficiently on NVIDIA GPUs. The compilation step applies kernel fusion, quantization, and memory optimizations that can yield 100x speedups over naive PyTorch inference. Once compiled, the engine supports in-flight batching, paged KV cache, and multi-GPU parallelism out of the box.

Before you start

• Hardware: You need an NVIDIA GPU from the A100, H100, or GB200 family. Older GPUs may work but are not the target.
• Software: CUDA 13.2.1, TensorRT 10.x, and Python 3.10-3.12 are required for the pip install path.
• Docker: The recommended installation uses NVIDIA's NGC container registry (nvcr.io), not Docker Hub. You must have Docker and the NVIDIA Container Toolkit installed.
• Model access: You need HuggingFace credentials or a local model path for the models you want to compile.
• Disk space: Compilation creates engine files that can be tens of gigabytes. Ensure you have enough free space.

Quick start

Installation

# Docker (recommended) — images are on NGC (nvcr.io), not Docker Hub.
# Replace x.y.z with the desired version (e.g. 1.2.1). Browse tags on NGC:
# https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release/tags
docker pull nvcr.io/nvidia/tensorrt-llm/release:x.y.z

# pip install (current stable GA)
pip install tensorrt_llm

# Requires CUDA 13.2.1, TensorRT 10.x, Python 3.10-3.12

Basic inference

from tensorrt_llm import LLM, SamplingParams

# Initialize model
llm = LLM(model="meta-llama/Meta-Llama-3-8B")

# Configure sampling
sampling_params = SamplingParams(
    max_tokens=100,
    temperature=0.7,
    top_p=0.9
)

# Generate
prompts = ["Explain quantum computing"]
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(output.text)

Serving with trtllm-serve

# Start server (automatic model download and compilation)
trtllm-serve meta-llama/Meta-Llama-3-8B \
    --tp_size 4 \              # Tensor parallelism (4 GPUs)
    --max_batch_size 256 \
    --max_num_tokens 4096

# Client request
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-8B",
    "messages": [{"role": "user", "content": "Hello!"}],
    "temperature": 0.7,
    "max_tokens": 100
  }'

Key features

Performance optimizations

• In-flight batching: Dynamic batching during generation
• Paged KV cache: Efficient memory management
• Flash Attention: Optimized attention kernels
• Quantization: FP8, INT4, FP4 for 2-4× faster inference
• CUDA graphs: Reduced kernel launch overhead

Parallelism

• Tensor parallelism (TP): Split model across GPUs
• Pipeline parallelism (PP): Layer-wise distribution
• Expert parallelism: For Mixture-of-Experts models
• Multi-node: Scale beyond single machine

Advanced features

• Speculative decoding: Faster generation with draft models
• LoRA serving: Efficient multi-adapter deployment
• Disaggregated serving: Separate prefill and generation

Common patterns

Quantized model (FP8)

from tensorrt_llm import LLM

# Load FP8 quantized model (2× faster, 50% memory)
llm = LLM(
    model="meta-llama/Meta-Llama-3-70B",
    dtype="fp8",
    max_num_tokens=8192
)

# Inference same as before
outputs = llm.generate(["Summarize this article..."])

Multi-GPU deployment

# Tensor parallelism across 8 GPUs
llm = LLM(
    model="meta-llama/Meta-Llama-3-405B",
    tensor_parallel_size=8,
    dtype="fp8"
)

Batch inference

# Process 100 prompts efficiently
prompts = [f"Question {i}: ..." for i in range(100)]

outputs = llm.generate(
    prompts,
    sampling_params=SamplingParams(max_tokens=200)
)

# Automatic in-flight batching for maximum throughput

Performance benchmarks

Meta Llama 3-8B (H100 GPU):

• Throughput: 24,000 tokens/sec
• Latency: ~10ms per token
• vs PyTorch: 100× faster

Llama 3-70B (8× A100 80GB):

• FP8 quantization: 2× faster than FP16
• Memory: 50% reduction with FP8

Supported models

• LLaMA family: Llama 2, Llama 3, CodeLlama
• GPT family: GPT-2, GPT-J, GPT-NeoX
• Qwen: Qwen, Qwen2, QwQ
• DeepSeek: DeepSeek-V2, DeepSeek-V3
• Mixtral: Mixtral-8x7B, Mixtral-8x22B
• Vision: LLaVA, Phi-3-vision
• 100+ models on HuggingFace

When not to use it

Use vLLM instead when you need a simpler setup and a Python-first API, want PagedAttention without TensorRT compilation, or are working with AMD GPUs or non-NVIDIA hardware. Use llama.cpp instead when deploying on CPU or Apple Silicon, need edge deployment without NVIDIA GPUs, or want the simpler GGUF quantization format.

Limits and gotchas

• Compilation is not instant. The first run of trtllm-serve or LLM() will download the model and compile it, which can take minutes to hours depending on model size and GPU count.
• The pip install path requires exact CUDA and TensorRT versions. Docker avoids this but adds its own overhead.
• Not all HuggingFace models are supported. Check the supported models list and the library tag on HuggingFace.
• Quantization (FP8, INT4) requires hardware support. Older GPUs may not support these formats.

Related references

• Optimization Guide - Quantization, batching, KV cache tuning
• Multi-GPU Setup - Tensor/pipeline parallelism, multi-node
• Serving Guide - Production deployment, monitoring, autoscaling

Resources

• Docs: https://nvidia.github.io/TensorRT-LLM/
• GitHub: https://github.com/NVIDIA/TensorRT-LLM
• Models: https://huggingface.co/models?library=tensorrt_llm