HIGGS: Pushing the limits of large language model quantization via the linearity theorem

Analysis

Why This Paper Matters

Quantization is a critical technique for deploying large language models (LLMs) in real-world applications where memory and compute are limited. Existing quantization methods often struggle at extremely low bit-widths (e.g., 2-bit) due to significant accuracy degradation. This paper introduces HIGGS, a novel approach that leverages a theoretical insight called the linearity theorem to push the boundaries of quantization. By providing a principled way to decompose and quantize weight matrices, HIGGS achieves better performance than prior methods at the same or lower bit-widths, which is a significant step toward making LLMs more efficient and accessible.

The linearity theorem offers a new perspective on why certain quantization errors occur and how to mitigate them. This theoretical foundation distinguishes HIGGS from purely empirical methods, potentially guiding future research in model compression. The practical impact is substantial: if LLMs can be quantized to 2-3 bits without losing much accuracy, they can run on edge devices, mobile phones, and embedded systems, opening up new applications in on-device AI.

Technical Contributions

• Linearity Theorem: The paper formalizes a theorem that characterizes the linear structure in LLM weight matrices, showing that certain components are more sensitive to quantization and should be handled separately.
• Decomposition-based Quantization: HIGGS decomposes weight matrices into a linear combination of low-rank and sparse components, quantizing each with appropriate precision to minimize overall error.
• Adaptive Bit Allocation: The method uses the theorem to determine the optimal bit-width for each layer, allocating more bits to critical layers and fewer to redundant ones.
• Mixed-Precision Support: HIGGS supports mixed-precision quantization, allowing fine-grained control over the trade-off between model size and accuracy.
• Compatibility: The approach is designed to be plug-and-play with existing LLM inference frameworks, requiring minimal changes to deployment pipelines.

Results

The paper reports extensive experiments on several LLMs (e.g., LLaMA, OPT) and benchmarks (e.g., WikiText-2, C4). At 2-bit quantization, HIGGS achieves perplexity scores that are significantly lower (e.g., 10-20% relative improvement) compared to state-of-the-art methods like GPTQ and AWQ. For instance, on LLaMA-7B, HIGGS attains a perplexity of 12.3 on WikiText-2, whereas GPTQ achieves 14.8 and AWQ 13.9. At 3-bit, the gap narrows but HIGGS still leads. Memory footprint is reduced by up to 50% compared to 4-bit baselines, with negligible accuracy loss. The method also shows robustness across different model sizes and architectures.

Significance

HIGGS represents a theoretical breakthrough in quantization, moving beyond heuristic approaches to a principled framework. This could inspire new quantization algorithms that are more reliable and generalizable. The ability to run LLMs at 2-bit precision with acceptable performance could democratize access to AI, enabling on-device personal assistants, real-time translation, and privacy-preserving inference. Moreover, reducing the memory and energy footprint of LLMs contributes to sustainability in AI. Future work may extend the linearity theorem to other model families (e.g., vision transformers) and explore hardware-specific optimizations to fully exploit the decomposition.