Netflix Details In-House LLM Serving Platform with Triton and vLLM

Netflix has revealed the production lessons learned from integrating large language model inference into its internal serving platform. The company detailed the challenges of supporting different model sizes, hardware requirements, and rapidly evolving inference engines.

The account covers the architectural choices and operational work involved in running real-time and batch workloads across CPUs and GPUs. Topics include model packaging, deployment, constrained decoding, and version compatibility.

Platform Architecture

The platform builds on Netflix's existing JVM-based serving layer. This layer continues to handle routing, feature retrieval, candidate generation, post-processing, and logging. Smaller models can run in-process on CPUs. Larger requests are delegated to MSS, where Triton takes over model loading, batching, GPU scheduling, and multi-framework serving. This approach allows the surrounding production workflow to remain consistent even when inference moves between local and remote hardware.

Inference Engine Selection

Within the GPU path, Netflix selected vLLM for its operational fit and extensibility. The company retained Triton's model-management and scheduling responsibilities. Triton controls the serving environment around the model, while vLLM performs inference and provides extension mechanisms for custom behavior. Netflix reported that mismatched Triton and vLLM versions can prevent deployments from loading. The company tests and pins compatible releases together.

Custom Model Integration

Custom models introduced another integration challenge. Hugging Face compatibility in vLLM was insufficient for some Netflix models. The company used vLLM extension points for custom architectures and decoding behavior.

Netflix also compared two Triton packaging approaches: Triton's Python backend and its vLLM backend. The company reported that the vLLM-backend approach allows models and frontends to evolve more independently than the Python-backend option. The choice affects how tightly a model is coupled to its serving environment rather than which engine performs inference.

API and Feature Gaps

The common serving interface did not eliminate differences between the underlying engines. Although Triton exposed an OpenAI-compatible API alongside KServe HTTP and gRPC frontends, Netflix still encountered gaps in how some features were handled across those integrations.

Constrained decoding was one example. It allows Netflix to force model responses into formats such as valid JSON by filtering the tokens the model may generate at each step. Because those rules depend on everything generated so far, the decoder must maintain state throughout the request. When vLLM pauses and later resumes a request to manage GPU resources, that state can fall out of sync with the token history. Netflix added logic to detect the change and rebuild it before generation continues.

Deployment and Versioning

Compatibility also affected deployment. Netflix pins tested Triton and vLLM versions together to prevent backend-loading failures. Red-Black and Versioned deployment strategies handle changes at the model level. Versioned deployments keep old and new revisions available separately, allowing consumers to migrate after adapting to incompatible input or output schemas.

Industry Context

Uber has described a related approach at the application boundary. Its generative AI gateway presents an OpenAI-compatible interface across externally hosted and internally managed models. It centralizes concerns including authentication, caching, observability, and routing. The implementation differs from Netflix's serving platform, but both separate application integrations from the models, runtimes, and hosting environments behind them.

Broader Implications

Netflix's experience shows how a common serving interface can sit above several distinct layers. This type of architecture reflects a broader effort to give application teams a stable integration surface while model providers and serving runtimes continue to change. Netflix's account also shows that the abstraction does not remove the underlying work. Packaging, compatibility controls, constrained decoding, and deployment isolation still require engineering at each layer.

Related on Neura Market:

• AI Tools Directory
• Automation Marketplace
• Platform Directory