BMTrain
FreeEfficient Training for Big Models.
FreeFree tier
About BMTrain
BMTrain is an efficient large model training toolkit developed by OpenBMB for pre-training and fine-tuning models with tens of billions of parameters. It enables distributed training while keeping the code as simple as stand-alone training, supporting ZeRO optimization (including ZeRO-2) and tensor parallelism (since v1.0.0). Users replace PyTorch modules with bmtrain.DistributedModule, bmtrain.DistributedParameter, and wrap transformer blocks with bmtrain.Block to leverage memory savings and communication efficiency. BMTrain is open source, installable via pip, and integrates with other OpenBMB tools like OpenPrompt and OpenDelta.
Key Features
Distributed training with code simplicity comparable to single-machine training
ZeRO optimization (ZeRO-1, ZeRO-2) for memory efficiency
Tensor parallelism support (since v1.0.0)
Simple API: replace torch.nn.Module with bmtrain.DistributedModule and torch.nn.Parameter with bmtrain.DistributedParameter
Transformer block wrapping with bmtrain.Block for automatic optimization
Easy installation via pip (pip install bmtrain)
Integration with OpenPrompt and OpenDelta for prompt tuning and delta tuning
Pros & Cons
Pros
- Enables training of tens-of-billions parameter models on limited hardware
- Code remains similar to standard PyTorch training scripts
- Actively maintained with regular updates and clear changelog
- Open-source and free to use
- Supports both ZeRO and tensor parallelism for flexibility
Cons
- Requires modifying existing PyTorch model code (replace modules and parameters)
- Not compatible with PyTorch's native distributed module; must use BMTrain's own initialization
- Installation may take minutes due to C/CUDA compilation
- Documentation and examples primarily in English and Chinese (GitHub)
Best For
Pre-training large language models with billions of parametersFine-tuning large models on downstream tasksTraining models in distributed environments (multi-GPU, multi-node)Research on model scaling and efficient training techniques
FAQ
How do I install BMTrain?
The recommended method is via pip: `pip install bmtrain`. You can also install from source by downloading the package and running `pip install .`. Compilation may take a few to ten minutes.
What are the system requirements for BMTrain?
BMTrain requires a CUDA-enabled environment and PyTorch. It compiles C/CUDA source code during installation, so a compatible compiler toolkit is needed.
How does BMTrain differ from PyTorch's DistributedDataParallel?
BMTrain provides higher-level abstractions for large model training, including ZeRO optimization and tensor parallelism, while keeping the training script as simple as stand-alone. It replaces PyTorch's distributed module with its own initialization and communication.