vLLM

vLLM is a fast and easy-to-use open-source library for LLM inference and serving. Originally developed in the Sky Computing Lab at UC Berkeley, it has become one of the most active open-source AI projects with over 2,000 contributors. vLLM delivers state-of-the-art serving throughput through efficient management of attention key-value memory using PagedAttention, continuous batching of incoming requests, chunked prefill, prefix caching, and fast model execution with CUDA/HIP graphs. It supports a wide range of quantization methods including FP8, INT8, INT4, GPTQ/AWQ, GGUF, and more. vLLM offers optimized attention kernels (FlashAttention, FlashInfer, etc.) and MoE kernels, speculative decoding (n-gram, suffix, EAGLE), and disaggregated prefill/decode/encode. It integrates seamlessly with Hugging Face models, provides an OpenAI-compatible API server along with Anthropic Messages API and gRPC support, and supports tensor, pipeline, data, expert, and context parallelism for distributed inference. vLLM runs on NVIDIA, AMD, Intel GPUs, x86/ARM/PowerPC CPUs, and additional hardware plugins such as Google TPUs and Intel Gaudi.