Neura Market
Megatron-DeepSpeed logo

Megatron-DeepSpeed

Free

DeepSpeed version of NVIDIA's Megatron-LM that adds additional support for several features such as MoE model training, Curriculum Learning, 3D Parallelism, and others.

FreeFree tier
Type
Open Source
Company
Microsoft
Visit Website GitHub (38 stars)

About Megatron-DeepSpeed

Megatron-DeepSpeed is a fork of NVIDIA's Megatron-LM that integrates DeepSpeed optimization technologies for training large transformer language models at scale. It extends Megatron-LM with additional support for Mixture of Experts (MoE) model training, Curriculum Learning, and 3D Parallelism (tensor, sequence, and pipeline parallelism). The repository provides efficient, model-parallel, and multi-node pre-training of transformer-based models including GPT, BERT, T5, and vision transformers. It includes example scripts and recipes for running on Azure and AzureML, making it easier to deploy large-scale training workloads. The project is actively maintained by the DeepSpeed team at Microsoft and synced with upstream Megatron-LM.

Key Features

Integrates DeepSpeed optimization for efficient large-scale training
Supports Mixture of Experts (MoE) model training
Curriculum Learning for improved training dynamics
3D Parallelism: tensor, sequence, and pipeline parallelism
Multi-node pre-training of GPT, BERT, T5, and vision transformers
Example scripts and recipes for Azure and AzureML deployment
Synced with NVIDIA/Megatron-LM (over 1,000 commits merged)

Pros & Cons

Pros
  • Open source and free to use
  • Leverages DeepSpeed techniques for memory and speed improvements
  • Supports a wide range of model architectures (GPT, BERT, T5, vision)
  • Includes ready-to-use recipes for Azure and AzureML
  • Actively maintained by Microsoft's DeepSpeed team
  • Extends Megatron-LM with advanced parallelism and MoE support
Cons
  • Requires substantial computational resources (multi-GPU/multi-node clusters)
  • Complex setup and configuration, not beginner-friendly
  • Potential bugs due to large merge with upstream (acknowledged by maintainers)
  • Documentation mainly provided through example scripts rather than comprehensive guides

Best For

Large-scale transformer language model training (GPT, BERT, T5)Mixture of Experts (MoE) research and trainingCurriculum learning experiments for language modelsTraining vision transformers and multi-modal modelsDeploying massive model training on Azure cloud infrastructure

FAQ

What is Megatron-DeepSpeed?
Megatron-DeepSpeed is a fork of NVIDIA's Megatron-LM that adds DeepSpeed support for features like MoE training, Curriculum Learning, and 3D Parallelism.
What models can be trained with Megatron-DeepSpeed?
It supports training GPT, BERT, T5, and vision transformer models at scale.
Is Megatron-DeepSpeed free to use?
Yes, it is an open-source project available on GitHub under a permissive license.
Where can I find example scripts?
Example scripts are located in the examples_deepspeed/ folder, including Azure and AzureML recipes.