Meta Lingua
Freea lean, efficient, and easy-to-hack codebase to research LLMs.
FreeFree tier
About Meta Lingua
Meta Lingua is a minimal and fast LLM training and inference library designed for research, developed by Meta (Facebook AI Research). It provides easy-to-modify PyTorch components to experiment with new architectures, losses, data pipelines, and more. The library supports end-to-end training, inference, and evaluation, along with tools to analyze speed and stability. It includes scripts for data preparation (fineweb_edu, dclm_baseline_1.0) and tokenizer download (e.g., llama3), and supports launching jobs via SLURM or torchrun. Training results are provided for Transformer, minGRU, and minLSTM models on 60B DCLM tokens.
Key Features
Minimal and fast LLM training and inference library
Easy-to-modify PyTorch components for experimenting with new architectures, losses, and data
End-to-end training, inference, and evaluation pipeline
Tools for analyzing speed and stability
SLURM job scheduling support via stool tool
Data preparation scripts for fineweb_edu and DCLM datasets
Tokenizer download support (e.g., llama3)
Benchmarking results for Transformer, minGRU, and minLSTM models
Pros & Cons
Pros
- Open source with a permissive license (MIT)
- Lean and efficient codebase optimized for research
- Easy to hack and modify for custom experiments
- Provides benchmarking results to validate implementations
- Supports distributed training via SLURM and torchrun
Cons
- Currently under development and may lack full documentation
- Primarily designed for researchers with access to SLURM clusters and multiple GPUs
- Alternative architectures (minGRU, minLSTM) underperform compared to Transformer baseline in benchmarks
Best For
Research on new LLM architectures and training techniquesTraining 1B parameter LLMs on large-scale datasetsComparing model performance on downstream tasks (ARC, BoolQ, HellaSwag, etc.)Prototyping and testing novel loss functions or data pipelines
FAQ
What models does Meta Lingua support?
Meta Lingua includes implementations of Transformer, minGRU, and minLSTM architectures, as shown in the training results table.
How do I set up Meta Lingua?
Clone the repository, run bash setup/create_env.sh or sbatch setup/create_env.sh to create the environment, then activate it with conda activate lingua_date.
How do I prepare data for training?
Use the script python setup/download_prepare_hf_data.py <dataset> <memory> --data_dir ./data --seed 42 --nchunks <NCHUNKS> to download and prepare datasets like fineweb_edu.
How do I launch a training job?
You can launch using python -m lingua.stool script=apps.main.train config=apps/main/configs/debug.yaml for SLURM, or torchrun --nproc-per-node 8 -m apps.main.train config=apps/main/configs/debug.yaml for local distributed training.