Papers Explained 187e: Quantized Llama 3.2, Llama 3.3

Analysis

Why This Paper Matters

Large language models (LLMs) have become indispensable for AI applications, but their massive size and memory requirements often limit deployment to powerful cloud servers. This paper addresses a critical bottleneck: bringing LLMs to edge devices like smartphones. By introducing quantization techniques that shrink model size without sacrificing accuracy, Meta enables real-time, on-device AI that respects user privacy and reduces latency. The release of Llama 3.3 further pushes the frontier of multilingual, instruction-tuned models, making state-of-the-art capabilities accessible to a broader community.

How It Works

The quantized Llama 3.2 models employ a two-pronged approach. First, Quantization-Aware Training (QAT) simulates low-precision arithmetic during training. Starting from BF16 checkpoints after supervised fine-tuning (SFT), the model undergoes another SFT round with QAT. The backbone is then frozen, and LoRA adapters (low-rank matrices) are added to all transformer layers, kept in BF16 for precision. Finally, direct preference optimization (DPO) fine-tunes both backbone and adapters. This yields models where weights are quantized to 4-bit groupwise (group size 32) and activations to 8-bit per-token dynamic quantization. The classification layer uses 8-bit per-channel weights and 8-bit per-token activations, while embeddings are 8-bit per-channel.

Second, SpinQuant offers a post-training alternative for users who cannot access original training data. It learns rotation matrices on a small calibration set (e.g., WikiText) to smooth outliers, enabling effective quantization without retraining. The same quantization scheme is applied, but SpinQuant trades some accuracy for portability and ease of use.

Llama 3.3, in contrast, is a full-precision 70B model optimized for multilingual dialogue. It uses an auto-regressive transformer with Grouped-Query Attention (GQA) for efficient inference. Pretrained on ~15 trillion tokens from public sources (cutoff Dec 2023), it is fine-tuned with SFT and RLHF using over 25 million synthetic examples and human-generated data, with LLM-based classifiers ensuring quality.

Results

Quantized Llama 3.2 models achieve near-lossless accuracy on benchmarks while reducing memory footprint by up to 4x, enabling deployment on mobile hardware. SpinQuant provides a viable alternative for data-constrained scenarios, though with slightly lower accuracy. Llama 3.3 70B outperforms many open and closed models on multilingual tasks, demonstrating the effectiveness of large-scale pretraining and careful alignment.

Significance

This work democratizes LLM access by making them practical for edge devices, reducing reliance on cloud infrastructure. The combination of QAT+LoRA and SpinQuant offers flexibility for different deployment constraints. Llama 3.3 raises the bar for open-source multilingual models, fostering innovation in global AI applications. Together, these models represent a step toward efficient, private, and capable AI on every device.