Mesh Tensorflow
FreeMesh TensorFlow: Model Parallelism Made Easier.
FreeFree tier
About Mesh Tensorflow
Mesh TensorFlow (mtf) is a language for distributed deep learning that enables model parallelism across hardware processors. It formalizes distribution strategies using an n-dimensional mesh of processors, where tensors are distributed based on user-defined layout rules. This allows for complex parallelism such as splitting the batch over rows of processors and hidden units over columns. Mesh TensorFlow is implemented as a layer over TensorFlow and is particularly useful for training large language models (e.g. 5-billion-parameter models) or models with large activations (e.g. 3D image models) that do not fit on a single device.
Key Features
Language for distributed deep learning
Model parallelism formally specified via layout rules
Supports arbitrary n-dimensional processor meshes
Collective communication for parallel computation
Compatible with TensorFlow installation
Designed for large models exceeding single device memory
Pros & Cons
Pros
- Formalizes distribution strategies for clarity and correctness
- Layouts do not affect results, only performance
- Can handle models that do not fit on one device (parameters or activations)
- Works as a layer over TensorFlow, leveraging existing ecosystem
Cons
- Repository is archived and read-only as of January 2025
- Not necessary for simple data-parallel training
- Requires understanding of mesh layouts and tensor dimensions
- Depends on TensorFlow, which may not be ideal for all users
Best For
Training large language models with billions of parametersTraining models with large activations (e.g., 3D image models)Lower-latency parallel inference at batch size 1
FAQ
What is Mesh TensorFlow?
Mesh TensorFlow is a language for distributed deep learning that allows users to specify distribution strategies for tensor computations across a mesh of processors.
When should I use Mesh TensorFlow?
When your model parameters or activations do not fit on a single device, or for lower-latency parallel inference at batch size 1.
How do I install Mesh TensorFlow?
Install via pip install mesh-tensorflow for the stable version, or pip install -e git+https://github.com/tensorflow/mesh.git#egg=mesh-tensorflow for the development version. Note that installing mesh-tensorflow does not automatically install TensorFlow; you must install TensorFlow separately.