A new platform called Petals is offering a novel way to run large language models at home using a peer-to-peer approach similar to BitTorrent. The system allows users to generate text with models such as Llama 3.1 (up to 405 billion parameters), Mixtral (8x22B), Falcon (40B+), and BLOOM (176B) using only a consumer-grade GPU or Google Colab.
How Petals Works
Instead of requiring users to load an entire large language model onto their own hardware, Petals distributes the workload across a network of participants. Each user loads only a portion of the model and then joins a network of other people serving the remaining parts. This collaborative approach makes it possible to run models that would otherwise be too large for individual consumer hardware.
Performance and Capabilities
According to the project's website, single-batch inference runs at up to 6 tokens per second for Llama 2 (70B) and up to 4 tokens per second for Falcon (180B). These speeds are sufficient for chatbots and interactive applications, the developers say.
Beyond Standard APIs
Petals offers more than what typical large language model APIs provide. Users can employ any fine-tuning and sampling methods they choose, execute custom paths through the model, or inspect its hidden states. The platform provides the comforts of a standard API while maintaining the flexibility of PyTorch and Hugging Face Transformers.
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
Getting Started
The project is available to try now in Google Colab, with documentation hosted on GitHub. Users can also contribute their own GPU to the network to help serve model parts for others. The development team encourages following progress through Discord or email updates, which they say will be sent only once every few months and contain no spam.
Background and Recognition
Petals is part of the BigScience research workshop, a collaborative effort focused on large language model research. The project has been featured on several notable platforms, though the specific outlets are not listed on the main page.
The network status display on the project's site currently shows a loading state with an error message indicating it cannot load the network status. The page also lists top contributors, though that section is also in a loading state.