ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads

Analysis

Why This Paper Matters

Large language models (LLMs) are increasingly deployed in memory-constrained environments, driving the need for aggressive compression. Weight-only quantization has become a standard technique to reduce the storage footprint of transformer blocks, but a critical gap remains: the final language-modeling head (LM-head) is often left in high precision (BF16/FP16) because naive quantization of this projection can severely distort the vocabulary-logit distribution. This paper addresses that gap by introducing ARCHead, a method specifically designed to compress the LM-head without sacrificing output quality.

The significance of this work lies in its focus on a component that is frequently ignored in quantization pipelines. While many studies concentrate on quantizing the bulk of the model, the LM-head can constitute a substantial portion of memory, especially for models with large vocabularies. By providing a practical solution that reduces LM-head storage by nearly 4x while maintaining near-lossless performance, ARCHead enables more complete model compression and facilitates deployment on devices with limited memory.

Technical Contributions

ARCHead introduces a novel architecture for LM-head compression that combines several techniques:

• Quantized low-rank core: The LM-head weight matrix is approximated by a low-rank factorization, with the core components quantized to reduce storage.
• Group-wise INT4 residuals: Residuals from the low-rank approximation are stored in a group-wise INT4 format, capturing fine-grained details that would otherwise be lost.
• Activation-derived low-rank correction: A low-rank correction is fitted using an activation-derived metric, which ensures that the compressed head preserves the vocabulary-logit distribution as closely as possible.
• No dense BF16 head: ARCHead completely eliminates the need for a dense BF16 head, achieving a 3.7-3.9x reduction in persistent storage.

The method is designed to be complementary to existing block quantizers like AWQ and bitsandbytes, which typically leave the LM-head untouched. By providing a drop-in replacement for the BF16 head, ARCHead can be integrated into existing quantization pipelines with minimal effort.

Results

The paper reports concrete metrics on Qwen3-8B-Base:

• Storage reduction: ARCHead uses only 25.6% of the BF16 head storage, a 3.9x reduction.
• Perplexity: Achieves a relative perplexity of 1.007, which is nearly lossless compared to the BF16 baseline.
• Comparison to naive INT4: Storage-matched naive INT4 quantization yields relative perplexity of 1.14-1.16, showing that ARCHead significantly outperforms naive approaches.
• Integration with existing quantizers: When replacing the BF16 head left by AWQ or bitsandbytes, ARCHead adds only 0.006-0.007 cross-entropy, with less than 2% throughput change in their measurements.

These results demonstrate that ARCHead achieves a favorable trade-off between compression and accuracy, making it a practical solution for real-world deployment.

Significance

The broader impact of ARCHead is that it fills a critical gap in LLM compression. By enabling the compression of the output head without significant quality loss, it allows for more complete model quantization, which is essential for edge deployment and large-scale serving. The method's compatibility with existing block quantizers means it can be adopted quickly, and its open-source implementation encourages further research and integration. As LLMs continue to grow in size, techniques like ARCHead will be crucial for making them accessible and efficient.