Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting

Analysis

Why This Paper Matters

Large Language Models (LLMs) have become central to AI applications, but their massive size makes deployment expensive and slow. Post-training quantization (PTQ) is a key technique to compress these models without retraining, yet it often suffers from accuracy degradation due to mismatched weight distributions. Ostquant addresses this by introducing a lightweight preprocessing step that reshapes weight distributions to be more quantization-friendly. This is significant because it offers a simple, plug-and-play improvement that can be applied to existing PTQ pipelines, potentially making LLMs more accessible for edge devices and real-time applications.

The paper tackles a fundamental issue in quantization: the non-uniformity and correlation of weights in LLMs. By using orthogonal transformations to decorrelate weights and scaling to balance ranges, Ostquant aligns the data distribution with the assumptions of uniform quantizers. This approach is theoretically grounded and empirically effective, bridging the gap between signal processing and model compression.

Technical Contributions

• Orthogonal Transformation: Applies a learned orthogonal matrix to weight matrices, reducing inter-channel correlation and making the distribution more isotropic. This improves the effectiveness of uniform quantization.
• Scaling Transformation: Adjusts the scale of each channel to equalize dynamic ranges, preventing outlier channels from dominating the quantization error.
• End-to-End Optimization: The transformations are optimized jointly with the quantization process to minimize the final task loss, ensuring that the preprocessing directly benefits the compressed model.
• Lightweight Integration: The method adds minimal overhead (a few extra matrix multiplications) and can be combined with any existing PTQ technique.

Results

Ostquant was evaluated on several LLMs (e.g., LLaMA, OPT) across multiple bit-widths (4-bit, 8-bit). Key results include:

• Perplexity improvements of 0.5–1.5 points on WikiText-2 compared to baseline PTQ methods like GPTQ and AWQ.
• Accuracy gains of 1–3% on downstream tasks (e.g., ARC, HellaSwag) at 4-bit quantization.
• The method achieves near-lossless performance at 8-bit and significantly reduces the gap to full-precision models at 4-bit.
• Ablation studies confirm that both orthogonal and scaling components contribute to the gains, with the combination yielding the best results.

Significance

Ostquant advances the state of the art in LLM quantization by providing a principled way to adapt weight distributions for compression. Its impact extends beyond PTQ: the idea of preprocessing weights with learned transformations could inspire similar techniques for other compression methods (e.g., pruning, knowledge distillation). By making LLMs more efficient, Ostquant helps democratize access to powerful AI models, enabling deployment on consumer hardware and reducing energy consumption in data centers. The method's simplicity and compatibility with existing tools make it a practical contribution that can be quickly adopted by practitioners.