Neura Market
Megatron-LM logo

Megatron-LM

Free

Ongoing research training transformer models at scale.

FreeFree tier
Type
Open Source
Company
NVIDIA
Visit Website GitHub (38 stars)

About Megatron-LM

Megatron-LM is an open-source, GPU-optimized library from NVIDIA for training large transformer models at scale. It includes two components: Megatron-LM (a reference example with pre-configured training scripts for rapid experimentation) and Megatron Core (a composable library providing building blocks for custom training frameworks). It supports advanced parallelism strategies (tensor, pipeline, data, expert, and context parallelism), mixed precision training (FP16, BF16, FP8, FP4), and a variety of model architectures. The Megatron Bridge enables bidirectional checkpoint conversion between Hugging Face and Megatron. The library is designed for research teams, ML engineers, and framework developers building and scaling transformer models.

Key Features

Advanced parallelism strategies: Tensor Parallelism (TP), Pipeline Parallelism (PP), Data Parallelism (DP), Expert Parallelism (EP), and Context Parallelism (CP)
Mixed precision support: FP16, BF16, FP8, and FP4
Composable building blocks via Megatron Core for custom training frameworks
Megatron Bridge for bidirectional Hugging Face ↔ Megatron checkpoint conversion
Reference example with pre-configured scripts for quick experimentation
Includes support for Mamba, hybrid models, and reinforcement learning training
Production-ready recipes for pretraining and inference

Pros & Cons

Pros
  • GPU-optimized with state-of-the-art parallelism for scaling to thousands of GPUs
  • Comprehensive support for multiple parallelism strategies in a single library
  • Actively maintained by NVIDIA with regular updates and community contributions
  • Open source with permissive license and extensive documentation
  • Includes reference training scripts and examples for fast onboarding
Cons
  • Building from source can require significant memory; may need to limit parallel compilation jobs
  • Steep learning curve for configuring advanced parallelism strategies
  • Primarily focused on GPU training; not designed for CPU inference or deployment

Best For

Research teams training large transformer models at scaleLearning and experimentation with distributed training techniquesBuilding custom training pipelines for large language modelsConverting and fine-tuning Hugging Face models with MegatronDeveloping mixture-of-experts and hybrid architectures

FAQ

What is Megatron-LM?
Megatron-LM is an open-source, GPU-optimized library from NVIDIA for training large transformer models at scale. It includes both a reference example (Megatron-LM) and a composable library (Megatron Core) with advanced parallelism and mixed precision support.
How do I install Megatron-LM?
You can install from PyPI using 'uv pip install megatron-core' or clone the repository and install from source with 'uv pip install -e .'. Building from source can use a lot of memory; set MAX_JOBS to limit parallel compilation.
What parallelism strategies does Megatron-LM support?
Megatron-LM supports Tensor Parallelism (TP), Pipeline Parallelism (PP), Data Parallelism (DP), Expert Parallelism (EP), and Context Parallelism (CP).
Does Megatron-LM support mixed precision training?
Yes, it supports FP16, BF16, FP8, and FP4 mixed precision training.
What is the Megatron Bridge?
The Megatron Bridge provides bidirectional checkpoint conversion between Hugging Face and Megatron formats, including production-ready recipes.