Intactkv: Improving large language model quantization by keeping pivot tokens intact

Analysis

Why This Paper Matters

Large language models (LLMs) have become indispensable but their massive size makes deployment costly. Quantization is a key technique to reduce model size and speed up inference, but aggressive quantization often degrades performance. This paper addresses that trade-off by recognizing that not all tokens are equally important. By keeping pivot tokens intact, the method preserves the most critical information while still achieving compression. This is particularly relevant for practitioners who need to run LLMs on resource-constrained hardware.

The approach is motivated by the observation that certain tokens—such as those in attention heads with high variance—are disproportionately influential on model outputs. Protecting these tokens from quantization errors yields significant gains. The paper's state-of-the-art results suggest that selective preservation is a promising direction for future quantization research.

Technical Contributions

• Pivot token identification: The method defines pivot tokens based on their impact on model output, likely using gradient or attention-based metrics.
• Selective quantization: Only non-pivot tokens are quantized aggressively, while pivot tokens are kept at higher precision or left intact.
• Compatibility: The technique is designed to be combined with existing quantization methods (e.g., GPTQ, AWQ), making it easy to adopt.
• Simple-to-combine design: The authors emphasize that their method can be layered on top of other quantization pipelines without major modifications.

Results

The paper reports new state-of-the-art results for LLM quantization, though specific metrics (e.g., perplexity on WikiText-2, accuracy on downstream tasks) are not detailed in the abstract. The improvements are demonstrated across multiple tasks, indicating generalizability. The method likely achieves lower perplexity and higher task accuracy compared to prior quantization approaches at similar compression ratios.

Significance

This work advances the practical deployment of LLMs by making quantization more intelligent. Instead of treating all tokens equally, it leverages the inherent structure of language and attention to allocate precision where it matters most. This could lead to more efficient inference on edge devices, reduced memory footprint in cloud deployments, and lower energy consumption. The simple-to-combine nature also means it can be quickly adopted by the community, potentially becoming a standard component in quantization toolkits.