nanotron
FreeMinimalistic large language model 3D-parallelism training.
FreeFree tier
Inputs: text
About nanotron
Nanotron is a minimalistic library for pretraining transformer models, developed by Hugging Face. It provides a simple and flexible API to pretrain custom LLMs, leveraging 3D-parallelism (tensor, pipeline, and data parallelism) for scalability from a single node to multi-node clusters. The library is optimized for speed and efficiency, with support for fused kernels (Flash Attention via Triton) and integration with the Hugging Face ecosystem. It includes an Ultrascale Playbook guide for efficient scaling and offers example configurations for training tiny LLaMA models.
Key Features
Minimalistic design for easy pretraining of transformer models
3D-parallelism: tensor, pipeline, and data parallelism
Simple and flexible API for custom datasets and configurations
Optimized for speed with fused kernels (Flash Attention via Triton)
Multi-node training support via Slurm
Ultrascale Playbook guide for efficient scaling
Integration with Hugging Face Hub and Weights & Biases
Example config for training a tiny LLaMA model in ~10 minutes on 8xH100s
Pros & Cons
Pros
- Open source and free to use (MIT license)
- Simple API reduces implementation complexity
- Scalable from single GPU to multi-node clusters
- 3D-parallelism enables training very large models
- Backed by Hugging Face with active maintenance
- Comprehensive documentation and Ultrascale Playbook guide
- Fast installation via uv and built-in example scripts
Cons
- Requires PyTorch and GPU infrastructure setup (no cloud-hosted service)
- Relatively new project with smaller community compared to alternatives (e.g., Megatron-LM)
- Focused on pretraining only; fine-tuning and inference not covered
- Dependency on Triton and Flash Attention may limit hardware compatibility
Best For
Pretraining large language models from scratchResearch on model scaling and parallelism strategiesCustom transformer training on domain-specific corporaAcademic experiments in LLM architecture and optimization
FAQ
What is Nanotron?
Nanotron is a minimalistic library for pretraining transformer models, developed by Hugging Face. It focuses on simplicity, performance, and scalability using 3D-parallelism.
How do I install Nanotron?
Install via Python virtual environment with uv, then install PyTorch, core dependencies, and optional fused kernels (e.g., Flash Attention via Triton). Full installation instructions are in the README.
Does Nanotron support multi-node training?
Yes, it supports multi-node training using Slurm. The repository includes a Slurm launcher script and a dedicated multi-node training guide.
Can I use Nanotron for fine-tuning or inference?
Nanotron is designed for pretraining only. It includes a generation script for testing checkpoints, but for fine-tuning and inference you may need other tools.