Megatron LM

Megatron-LM is an open-source, GPU-optimized library from NVIDIA for training large-scale transformer models. The repository contains two components: Megatron-LM (a reference example with pre-configured training scripts for quick experimentation) and Megatron Core (a composable library of GPU-optimized building blocks for custom training frameworks). It supports advanced parallelism strategies including tensor, pipeline, data, expert, and context parallelism (TP, PP, DP, EP, CP), and offers mixed precision training with FP16, BF16, FP8, and FP4. Megatron Bridge provides bidirectional checkpoint conversion between Hugging Face and Megatron formats. The project also includes support for emerging optimizers (e.g., Muon), Mixture-of-Experts models, and hybrid architectures like Falcon-H1. It is intended for research teams, ML engineers, and framework developers who need to train and fine-tune large language models efficiently on NVIDIA GPUs.