When a GPU handles both the prefill and decode phases of large language model inference, long prompts can stall token generation for every concurrent request. AWS has introduced a solution called Disaggregated Prefill and Decode (DPD) on Amazon SageMaker HyperPod that removes this interference by running each phase on separate GPU pools. These pools are connected through Elastic Fabric Adapter (EFA) with Remote Direct Memory Access (RDMA).
Large language model inference involves two distinct phases. The prefill phase is compute-bound, processing the entire input prompt in parallel to generate the initial key-value cache. The decode phase is memory-bound, generating one token at a time and requiring substantial memory bandwidth to access model weights and the growing KV cache. By separating these into specialized engines, users can assign different parallel strategies to each phase. This separation enables independent tuning of time to first token (TTFT) and inter-token latency (ITL), more reliable control of tail latency compared to chunked prefill tuning, and prevents long-context prefills from blocking ongoing decode requests.
When to Use Disaggregated Inference
Disaggregating prefill and decode provides the strongest benefits for long-context, high-concurrency streaming workloads such as chat assistants, agentic pipelines, document-analysis endpoints, and Retrieval Augmented Generation (RAG) with large retrieved contexts. In these scenarios, a single long prompt on a colocated GPU can stall in-flight decoding for every other request, causing per-token latency spikes that DPD eliminates by design.
Users should consider DPD when their workloads have input prompts that regularly exceed 4,096 tokens, multiple concurrent users or requests, streaming responses where consistent token delivery matters, and mixed traffic with both long and short prompts. A colocated deployment remains the simpler choice for batch or offline workloads optimizing for TTFT, low-concurrency deployments, or short-prompt-only traffic. Below a configurable routing threshold, the fixed cost of transferring KV cache over EFA RDMA outweighs the benefit of isolating decode, so the DPD router sends those requests straight to a decoder.
DPD requires at minimum one prefill node and one decode node with RDMA-capable EFA networking. Supported instance types include the P5 and P6 instance families on AWS, which offer NVLink and EFA with RDMA read and write capabilities.
Architecture Overview
The HyperPod DPD implementation builds on the vLLM Production Stack router, with LMCache providing the KV cache transfer layer over NIXL and EFA. The deployment consists of three main components plus a transport stack.
The intelligent router tokenizes each prompt and applies a configurable token threshold to decide whether the request takes the disaggregated path or runs end-to-end on a decoder. Long-context prompts go through a prefiller then a decoder, while short prompts bypass the prefiller to avoid unnecessary cross-GPU KV transfer. The router supports multiple routing strategies including prefixaware, kvaware, session, and roundrobin.
The prefiller pod is a vLLM worker with LMCache as its KV connector. It computes KV cache for long prompts and pushes it to the chosen decoder via LMCache's PD sender backend, overlapping compute and transfer to keep GPUs saturated. LMCache also provides each prefiller with an L1 CPU cache that can serve recurring prefixes from CPU memory without GPU recomputation.
The decoder pod is a vLLM worker with LMCache as its receiver. It reserves GPU memory for incoming KV transfers, runs full CUDA graphs for the decode kernel, and begins generation as soon as the transfer completes. Because it never executes prefill, decode latency stays stable under concurrency.
KV cache transfer uses a four-layer stack, LMCache PD, NIXL, libfabric, and EFA, that HyperPod composes end-to-end. The transfer cost is negligible relative to prefill compute: on ml.p5.48xlarge instances with 3,200 Gbps of EFA, an 8,000-token transfer for Llama 3.3 70B takes single-digit milliseconds.
Deployment Process
Stay updated
Get the day's AI and automation news in your inbox. No spam, unsubscribe anytime.
Deploying a DPD model endpoint on a HyperPod cluster requires the HyperPod Inference Operator version 3.2 or later. The operator handles the underlying orchestration, including provisioning the router, wiring the prefill and decode pods together, and integrating with HyperPod observability.
The deployment involves preparing model checkpoints in Amazon S3, configuring an InferenceEndpointConfig manifest with DPD-specific fields, and applying the manifest to the cluster. DPD-relevant fields include the pdSpec declaration of prefill/decode topology, independent replica counts, resources, routing threshold, and role-specific arguments.
Worker images must include vLLM, LMCache, NVIDIA NIXL, and the EFA libfabric provider. AWS supports two image options: the open source LMCache image lmcache/vllm-openai v0.4.3, and the SageMaker Deep Learning Container vllm:server-hyperpod-cuda-v1.1.
Scaling Guidance
DPD currently supports a single decoder replica with multiple prefiller replicas. Users start with a 1:1 prefill-to-decode ratio for balanced workloads like chat and code generation, and scale to 2:1 or 3:1 when workloads are prefill-heavy, such as summarization or RAG with long retrieved contexts. With multiple prefillers, users can set routing strategies to maximize cache locality.
If per-token output latency climbs and output throughput plateaus despite prefiller availability, the single decoder is saturated. In that case, users should increase the PD buffer size, reduce max model length, or reduce concurrency until multi-decoder support becomes available.
Benchmarking Performance
Benchmarks used genai-bench with fixed-length synthetic prompts at concurrency levels 8, 16, and 32. The DPD configuration used one prefiller and one decoder across two nodes (16 GPUs) on ml.p5.48xlarge instances with H100 80GB GPUs. The baseline was a single node (8 GPUs) with the same model and settings.
Results showed that DPD delivers consistent gains on per-token output latency, end-to-end latency, and throughput as concurrency grows. Per-token latency stayed flat under load, with improvements ranging from 22% at low concurrency to 66% at high concurrency on H100, and 28% to 48% on H200. Output throughput improved up to 35% on H100 and up to 64% on H200 at higher concurrency. End-to-end latency at P50 improved 14-32% on H100 and 29-41% on H200.
DPD introduces a modest increase in time to first token due to KV cache transfer over EFA RDMA. For streaming workloads where consistent per-token delivery matters more than initial response, this tradeoff is favorable. The conditional routing threshold ensures short requests bypass disaggregation entirely.
Clean Up and Conclusion
Users can delete InferenceEndpointConfig resources to remove prefill pods, decode pods, routers, and associated services. Optional cleanup includes removing model checkpoints from S3 and scaling down or removing HyperPod instance groups.
Disaggregated Prefill and Decode on Amazon SageMaker HyperPod allows users to run prefill and decode on separate GPU pools with KV cache transfers over EFA using GPU-Direct RDMA. The separation removes interference between the compute-bound prefill and memory-bandwidth-bound decode phases, producing more predictable latency under mixed traffic and enabling independent scaling. Users activate DPD by adding a few fields to the same InferenceEndpointConfig resource used for non-disaggregated endpoints.