Leanquant: Accurate and scalable large language model quantization with loss-error-aware grid

Analysis

Why This Paper Matters

Large language models (LLMs) have revolutionized AI but their massive memory and compute requirements hinder widespread deployment. Quantization is a key technique to reduce model size and speed up inference, but existing methods often suffer from accuracy loss, especially at very low bit-widths. Leanquant addresses this critical bottleneck by introducing a loss-error-aware approach that directly optimizes quantization to preserve model performance. This is particularly important as LLMs scale to hundreds of billions of parameters, where even small accuracy drops can compound.

The paper's focus on loss-aware quantization represents a paradigm shift from traditional heuristic or uniform quantization methods. By aligning the quantization process with the model's actual loss landscape, Leanquant ensures that the most sensitive parameters are preserved with higher fidelity. This makes it highly relevant for practitioners deploying LLMs in production, where both accuracy and efficiency are paramount.

Technical Contributions

• Loss-Error-Aware Grid Optimization: The core innovation is a grid-based quantization scheme where the grid points are learned to minimize the increase in the model's loss function, rather than just minimizing mean squared error of weights.
• Second-Order Approximation: Uses a Taylor expansion of the loss to efficiently estimate the impact of quantization without full retraining, making the method scalable.
• Mixed-Precision Support: The framework can automatically assign different bit-widths to different layers based on their sensitivity, further improving the accuracy-efficiency trade-off.
• Practical Implementation: Provides a straightforward algorithm that can be applied to pre-trained LLMs with minimal calibration data, avoiding the need for expensive fine-tuning.

Results

Leanquant was evaluated on several popular LLM families including LLaMA (7B, 13B, 33B, 65B) and OPT (6.7B, 13B, 30B). At 4-bit quantization, it achieves perplexity within 0.5 points of the full-precision model on WikiText-2, outperforming prior methods like GPTQ and LLM.int8() by 1-2 perplexity points. At 3-bit, the gap widens to 1-2 points, and at 2-bit, Leanquant maintains reasonable performance where other methods collapse. On downstream tasks (e.g., LAMBADA, HellaSwag), accuracy drops are less than 1% at 4-bit and under 3% at 3-bit. Memory reduction is up to 16x for 2-bit quantization, enabling deployment of 65B models on a single GPU.

Significance

Leanquant has significant implications for the AI field by making LLMs more accessible. It reduces the hardware barrier for running state-of-the-art models, enabling deployment on consumer GPUs and edge devices. This could accelerate applications in real-time language processing, on-device assistants, and privacy-preserving inference. The loss-aware methodology also opens new research directions for quantization-aware training and adaptive compression. As LLMs continue to grow, techniques like Leanquant will be essential for sustainable and scalable AI deployment.