Liger-Kernel

Free

Efficient Triton Kernels for LLM Training.

FreeFree tier

Type

Open Source

Company

Visit Website GitHub (38 stars)

About Liger-Kernel

Liger-Kernel is an open-source collection of Triton kernels developed by LinkedIn to optimize large language model (LLM) training. It delivers a 20% increase in multi-GPU training throughput and 60% reduction in memory usage, enabling longer context lengths, larger batch sizes, and massive vocabularies. The library provides Hugging Face compatible implementations of core operations (RMSNorm, RoPE, SwiGLU, CrossEntropy, FusedLinearCrossEntropy) and works seamlessly with Flash Attention, PyTorch FSDP, and Microsoft DeepSpeed. Additionally, Liger-Kernel includes optimized post-training kernels (DPO, ORPO, SimPO, CPO, KTO, JSD, and more) that achieve up to 80% memory savings for alignment and distillation tasks. It supports AMD GPUs and can be integrated with a single line of code.

Key Features

Collection of Triton kernels specifically designed for LLM training

Increases multi-GPU training throughput by 20%

Reduces memory usage by 60%

Hugging Face compatible implementations of RMSNorm, RoPE, SwiGLU, CrossEntropy, and FusedLinearCrossEntropy

Integrates out-of-the-box with Flash Attention, PyTorch FSDP, and Microsoft DeepSpeed

Optimized post-training kernels for DPO, ORPO, SimPO, CPO, KTO, JSD, and more, delivering up to 80% memory savings

Supports AMD GPUs (since v0.4.0)

One-line code integration for immediate benefits

Open-source with active community contributions and a dedicated Discord channel

Pros & Cons

Pros

Significant performance improvements: 20% throughput gain and 60% memory reduction
Easy integration with existing Hugging Face workflows and popular distributed training frameworks
One-line installation and usage for immediate speedups
Open source and community-driven, with regular updates and contributions
Support for AMD GPUs broadens hardware compatibility
Optimized post-training losses provide additional memory savings for alignment tasks

Best For

Optimizing LLM training throughput and memory efficiencyFine-tuning and alignment of large language models using techniques like DPO and ORPOExtending context length and increasing batch size in trainingHandling massive vocabularies in language modelsReducing memory footprint during post-training alignment and distillation