Petals
FreeBitTorrent style platform for running AI models in a distributed way.
FreeFree tier
Inputs: textOutputs: text
About Petals
Petals is a decentralized, BitTorrent-style platform that enables running large language models (LLMs) at home. It distributes model layers across a peer-to-peer network, allowing inference and fine-tuning of models such as Llama 3.1 (up to 405B parameters), Mixtral (8x22B), Falcon (40B+), and BLOOM (176B) without requiring high-end hardware. Users can connect to a public swarm or set up a private swarm for sensitive data. The system is community-run, open source, and offers up to 10x faster performance than offloading to CPU or disk. It integrates seamlessly with Hugging Face Transformers and supports hosting model layers on consumer GPUs.
Key Features
Distributed inference and fine-tuning of large LLMs (up to 405B parameters)
Up to 10x faster than offloading to CPU or disk
Peer-to-peer network connecting user GPUs
Supports models like Llama 3.1, Mixtral, Falcon, and BLOOM
Privacy options: public swarm or private swarm for sensitive data
Seamless integration with Hugging Face Transformers
Community-run, open source under bigscience-workshop
Pros & Cons
Pros
- Open source and free to use
- Faster than offloading to CPU or disk
- Lowers barrier to accessing large models
- Privacy controls via private swarms
- Community-driven, benefits from shared resources
Cons
- Inference speed dependent on network peer availability and latency
- Public swarm may expose data to other participants (mitigated by private swarm)
- Not as fast as running models on a dedicated high-end GPU locally
- Requires technical setup to host model layers
Best For
Running large language models on consumer-grade hardwareFine-tuning LLMs for custom tasks without expensive GPUsCollaborative model serving across a network of usersPrivate inference within trusted groups
FAQ
How does Petals work?
Petals distributes the layers of a large language model across a peer-to-peer network. When you run inference or fine-tuning, your requests are routed to peers hosting the required layers, allowing you to run models much larger than your local hardware could handle.
Is my data private when using the public swarm?
In the public swarm, your data is processed by other participants. The project notes that for sensitive data, you can set up a private swarm among people you trust.
What models are supported?
Petals supports models such as Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+), BLOOM (176B), and other models available on the Hugging Face Model Hub.
How can I contribute to the network?
You can contribute by hosting model layers on your GPU. The project provides instructions for Linux, Windows (WSL), Docker, and macOS (Apple M1/M2).
Is Petals free to use?
Yes, Petals is an open-source community-run system and is free to use. You can join the public swarm or create a private swarm.