exllama
FreeA more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.
FreeFree tier
Inputs: textOutputs: text
About exllama
ExLlama is a standalone Python/C++/CUDA implementation of Llama designed for efficient inference with 4-bit GPTQ quantized weights. It aims to be faster and more memory-efficient than the Hugging Face Transformers implementation, targeting modern NVIDIA GPUs (30-series and later). The project includes a runtime-compiled CUDA extension, support for ROCm via HIP, and provides example scripts for chatbot, batch generation, LoRA, CFG, and a web UI. Installation is simple via git clone and pip, with the extension compiled automatically on first run.
Key Features
Standalone Python/C++/CUDA implementation of Llama
Optimized for 4-bit GPTQ quantized weights
Fast and memory-efficient inference on modern NVIDIA GPUs
CUDA extension compiled at runtime, no separate install needed
Includes example scripts: chatbot, batch, CFG, Flask, websocket, LoRA
Web UI with flask and waitress
Perplexity evaluation and benchmark scripts
Theoretical ROCm support via HIP
LoRA fine-tuning support
Pros & Cons
Pros
- Memory-efficient compared to Hugging Face Transformers
- Fast inference on modern GPUs (RTX 30-series and later)
- Standalone package, easy to install from GitHub
- CUDA extension compiles automatically on first run
- Well-documented examples for various use cases
- Active project with periodic updates
Cons
- Poor performance on older GPUs (Pascal or older) due to FP16 support
- Still a work in progress (disclaimer in README)
- ROCm support is theoretical – no AMD GPU testing or optimization
- Windows setup requires manual installation of MSVC and CUDA Toolkit
- Limited to Llama architecture, not a general LLM framework
Best For
Running quantized Llama models locally for chatbot applicationsText generation and inference benchmarkingBatch inference for multiple promptsConditional generation with CFG (Classifier-Free Guidance)Web-based chatbot interfaceLoRA fine-tuning and inference
FAQ
What is ExLlama?
ExLlama is a standalone Python/C++/CUDA implementation of Llama designed for efficient inference with 4-bit GPTQ quantized weights. It is more memory-efficient than the Hugging Face Transformers implementation.
What hardware is supported?
ExLlama targets modern NVIDIA GPUs (30-series and later). 30-series and later are well supported. Pascal and older GPUs have poor FP16 support. ROCm via HIP is theoretically supported but not tested.
How do I install ExLlama?
Clone the repository, install dependencies with pip, and run a script. The CUDA extension will compile automatically on the first run.
What are the dependencies?
Python 3.9+, torch (tested on 2.0.1 and 2.1.0), safetensors, sentencepiece, and ninja. For the web UI, flask and waitress are needed.
Does ExLlama support LoRA?
Yes, ExLlama includes an example script for LoRA (example_lora.py).