TensorRT-LLM: High-Throughput LLM Inference on NVIDIA GPUs

TensorRT-LLM is NVIDIA's open-source library for compiling and running large language models at high throughput on NVIDIA GPUs. It is the skill you reach for when a model has to serve real traffic on A100, H100, or GB200 hardware and every token per second matters. If you are prototyping on a laptop or running on AMD silicon, this is not the tool you want.

What it does

The skill wraps the TensorRT-LLM toolchain so Hermes Agent can drive model compilation, inference, and serving from a single place. You get a Python API for loading models and generating text, plus a server binary (trtllm-serve) that exposes an OpenAI-compatible chat completions endpoint. Under the hood the library applies in-flight batching, a paged KV cache, Flash Attention kernels, CUDA graphs, and quantization (FP8, INT4, FP4) to push throughput well past a plain PyTorch loop. The source reports 24,000 tokens/sec on Llama 3-8B with an H100, roughly 100x faster than PyTorch, and about 10ms per token of latency.

Before you start

This is an optional skill, installed on demand from optional-skills/mlops/tensorrt-llm. It is version 1.0.1, authored by Orchestra Research under the MIT license, and it targets Linux only.

The source lists three hard prerequisites: CUDA 13.2.1, TensorRT 10.x, and Python 3.10 through 3.12. In practice that means you need a recent NVIDIA driver and a GPU from the supported families (A100, H100, GB200 are the ones named). The Docker route is recommended because the images ship on NGC at nvcr.io, not Docker Hub, so you will not find them with a plain docker search. You need an NGC account to browse tags.

Installation

Two paths are documented. The Docker image is the recommended one; the pip package is the current stable GA release.

# Docker (recommended) — images are on NGC (nvcr.io), not Docker Hub.
# Replace x.y.z with the desired version (e.g. 1.2.1). Browse tags on NGC:
# https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release/tags
docker pull nvcr.io/nvidia/tensorrt-llm/release:x.y.z

# pip install (current stable GA)
pip install tensorrt_llm

# Requires CUDA 13.2.1, TensorRT 10.x, Python 3.10-3.12

Pick the image tag that matches the TensorRT-LLM release you want; the x.y.z placeholder is deliberate. If you go the pip route, confirm your CUDA and TensorRT versions line up with the requirements above before you start debugging import errors.

Basic inference

The Python API mirrors the shape of most inference libraries: construct an LLM, define sampling parameters, call generate.

from tensorrt_llm import LLM, SamplingParams

# Initialize model
llm = LLM(model="meta-llama/Meta-Llama-3-8B")

# Configure sampling
sampling_params = SamplingParams(
    max_tokens=100,
    temperature=0.7,
    top_p=0.9
)

# Generate
prompts = ["Explain quantum computing"]
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(output.text)

The first call does the heavy lifting: it loads the model and compiles the TensorRT engines. That compilation is why the first run takes noticeably longer than subsequent ones. Once the engine exists, generate returns output objects whose .text attribute holds the completion.

Serving with trtllm-serve

For production traffic you want the server rather than an in-process loop. trtllm-serve handles model download and compilation automatically, and the flags below set tensor parallelism across four GPUs plus batch and token ceilings.

# Start server (automatic model download and compilation)
# Tensor parallelism across 4 GPUs
trtllm-serve meta-llama/Meta-Llama-3-8B \
    --tp_size 4 \
    --max_batch_size 256 \
    --max_num_tokens 4096

# Client request
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-8B",
    "messages": [{"role": "user", "content": "Hello!"}],
    "temperature": 0.7,
    "max_tokens": 100
  }'

The server listens on port 8000 and speaks the OpenAI chat completions schema, so existing clients that target /v1/chat/completions work without changes. --tp_size 4 splits the model across four GPUs; --max_batch_size and --max_num_tokens cap how much work the scheduler will admit at once. Tune those two against your GPU memory, not against a guess.

Key features

The performance section covers in-flight batching (dynamic batching during generation), a paged KV cache, Flash Attention kernels, quantization at FP8, INT4, and FP4 for 2-4x faster inference, and CUDA graphs to cut kernel launch overhead.

Parallelism comes in four flavours: tensor parallelism (TP) splits the model across GPUs, pipeline parallelism (PP) distributes layers, expert parallelism targets Mixture-of-Experts models, and multi-node scaling goes beyond a single machine.

Advanced features include speculative decoding with draft models, LoRA serving for multiple adapters, and disaggregated serving that separates prefill from generation.

Common patterns

Quantized model (FP8)

FP8 is the sweet spot for most deployments: the source reports 2x speedup and 50% memory reduction versus FP16.

from tensorrt_llm import LLM

# Load FP8 quantized model (2× faster, 50% memory)
llm = LLM(
    model="meta-llama/Meta-Llama-3-70B",
    dtype="fp8",
    max_num_tokens=8192
)

# Inference same as before
outputs = llm.generate(["Summarize this article..."])

Multi-GPU deployment

For models too large for one card, set tensor_parallel_size to the number of GPUs you want to spread across.

# Tensor parallelism across 8 GPUs
llm = LLM(
    model="meta-llama/Meta-Llama-3-405B",
    tensor_parallel_size=8,
    dtype="fp8"
)

Batch inference

Passing a list of prompts to generate is the simplest way to exploit in-flight batching. The scheduler handles the rest.

# Process 100 prompts efficiently
prompts = [f"Question {i}: ..." for i in range(100)]

outputs = llm.generate(
    prompts,
    sampling_params=SamplingParams(max_tokens=200)
)

# Automatic in-flight batching for maximum throughput

Performance benchmarks

The source gives two reference points. On an H100 with Llama 3-8B: 24,000 tokens/sec, about 10ms per token, roughly 100x faster than PyTorch. On 8x A100 80GB with Llama 3-70B: FP8 quantization is 2x faster than FP16 and halves memory use. Treat these as targets to validate against your own workload, not guarantees.

Supported models

The library covers the LLaMA family (Llama 2, Llama 3, CodeLlama), the GPT family (GPT-2, GPT-J, GPT-NeoX), Qwen and Qwen2 and QwQ, DeepSeek-V2 and DeepSeek-V3, Mixtral-8x7B and Mixtral-8x22B, and vision models LLaVA and Phi-3-vision. The source claims 100+ models on HuggingFace.

When not to use it

The source is explicit about alternatives. Choose vLLM if you want a simpler setup with a Python-first API, PagedAttention without TensorRT compilation, or you are running on AMD or other non-NVIDIA hardware. Choose llama.cpp if you are deploying on CPU or Apple Silicon, need edge deployment without NVIDIA GPUs, or prefer the simpler GGUF quantization format. TensorRT-LLM earns its complexity only when you are committed to NVIDIA GPUs and need the throughput.

Limits and gotchas

The platform restriction is Linux only. The version matrix is tight: CUDA 13.2.1, TensorRT 10.x, Python 3.10-3.12. Compilation is part of the workflow, so expect a slow first run and plan for engine caching in your deployment pipeline. The Docker images live on NGC, not Docker Hub, which trips people up on first pull. And the benchmark numbers are tied to specific hardware (H100, 8x A100 80GB); your mileage will vary.

What pairs with it

The skill ships three reference documents: an Optimization Guide covering quantization, batching, and KV cache tuning; a Multi-GPU Setup guide for tensor and pipeline parallelism plus multi-node; and a Serving Guide for production deployment, monitoring, and autoscaling. Upstream resources are the NVIDIA docs at nvidia.github.io/TensorRT-LLM, the GitHub repository, and the HuggingFace model filter for tensorrt_llm.