torchtitan
FreeA native PyTorch Library for large model training.
About torchtitan
torchtitan is a PyTorch native platform designed for rapid experimentation and large-scale training of generative AI models. It provides a minimal clean-room implementation of PyTorch native scaling techniques, offering flexible extension points for custom needs. The platform showcases the latest distributed training features, with built-in support for pretraining Llama 3.1 LLMs of various sizes. Key capabilities include multi-dimensional composable parallelisms (FSDP2, Tensor Parallel, Pipeline Parallel, Context Parallel), activation checkpointing, distributed checkpointing (including async), interoperable checkpoints that can be loaded into torchtune for fine-tuning, torch.compile support, Float8 and MXFP8 training, and Supervised Fine-Tuning (SFT). The codebase is designed to be easy to understand, use, and extend, with minimal model code changes required for parallelism.
Key Features
Pros & Cons
- Clean, minimal codebase designed for easy understanding and extension
- Minimal model code changes required when applying multi-dimensional parallelism
- Showcases PyTorch's latest distributed training features
- Interoperable checkpoints enable seamless transition to fine-tuning with torchtune
- Active development with regular updates and community contributions
- Open source with permissive license (MIT)
- Under extensive development; latest features may require recent PyTorch nightly builds
- Currently focused on Llama model architecture; adding new models requires manual effort
- Documentation and tutorials are still evolving