LLaMA

Analysis

Why This Paper Matters

The LLaMA paper from Meta marks a pivotal shift in the landscape of large language models (LLMs). At a time when the trend was toward ever-larger models trained on proprietary or undisclosed datasets, LLaMA showed that a family of models ranging from 7 billion to 65 billion parameters, trained exclusively on publicly available data, could match or even surpass the performance of much larger models like GPT-3 (175B) and PaLM (540B). This democratizes access to state-of-the-art LLM research, as the training data is open and the models themselves are released for non-commercial use. The key insight is that training on more data, rather than simply scaling parameters, is a more efficient path to high performance.

How It Works

LLaMA's success hinges on a meticulously curated training dataset and a streamlined transformer architecture. The training data, totaling about 1.4 trillion tokens, is sourced from seven public domains:

• English CommonCrawl (67%): Five dumps from 2017-2020 are processed with the CCNet pipeline, which includes line-level deduplication, language identification via a fastText classifier, and quality filtering using an n-gram language model. A linear model further filters pages based on Wikipedia reference likelihood.
• C4 (15%): The Colossal Clean Crawled Corpus adds diversity, with heuristic-based quality filtering (e.g., punctuation, word/sentence counts).
• GitHub (4.5%): Public repositories under permissive licenses (Apache, BSD, MIT) are filtered by line length and alphanumeric proportion, with boilerplate removed via regex, then deduplicated at the file level.
• Wikipedia (4.5%): Dumps from June-August 2022 covering 20 Latin/Cyrillic script languages are cleaned of hyperlinks, comments, and formatting.
• Gutenberg and Books3 (4.5%): Public domain books and the Books3 subset of ThePile are deduplicated at the book level (removing >90% overlap).
• ArXiv (2.5%): LaTeX files are processed to remove pre-first-section content, bibliography, comments, and inline-expand user macros.
• Stack Exchange (2%): The 28 largest sites' Q&A data is kept, HTML stripped, answers sorted by score.

Tokenization uses byte-pair encoding (BPE), splitting all numbers into individual digits and falling back to bytes for unknown UTF-8 characters. Most tokens are used once, except Wikipedia and books (about two epochs).

The architecture is a transformer with three key improvements:

• Pre-normalization (from GPT-3): RMSNorm is applied to the input of each sub-layer, improving training stability.
• SwiGLU activation (from PaLM): Replaces ReLU, with a dimension of 2/3 * 4d (instead of 4d in PaLM).
• Rotary Positional Embeddings (RoPE, from GPTNeo): Used at every layer instead of absolute positional embeddings.

Results

LLaMA is evaluated on 20 benchmarks across zero-shot and few-shot settings. Key findings:

• Common Sense Reasoning: LLaMA-65B outperforms Chinchilla-70B on all benchmarks except BoolQ, and surpasses PaLM-540B everywhere except BoolQ and WinoGrande. LLaMA-13B beats GPT-3 on most benchmarks despite being much smaller.

• Closed-book QA: LLaMA-65B achieves state-of-the-art zero-shot and few-shot exact match on NaturalQuestions and TriviaQA. LLaMA-13B is competitive with GPT-3 and Chinchilla despite being 5-10x smaller.

• Reading Comprehension: LLaMA-65B is competitive with PaLM-540B; LLaMA-13B outperforms GPT-3 by a few percentage points.

• Mathematical Reasoning: LLaMA-65B outperforms Minerva-62B on GSM8k without any fine-tuning on mathematical data, using majority voting (maj1@k).

• Code Generation: LLaMA-13B beats LaMDA-137B on HumanEval and MBPP; LLaMA-65B outperforms PaLM-62B even with longer training.

• MMLU: LLaMA-65B lags behind Chinchilla-70B and PaLM-540B, likely due to less book data (177GB vs up to 2TB).

• Instruction Finetuning: Even without fine-tuning, LLaMA-65B follows basic instructions. After instruction tuning (LLaMA-I), it achieves 68.9% on MMLU, outperforming other moderate-sized instruction-tuned models but still below GPT code-davinci-002 (77.4%).

Significance

LLaMA's primary contribution is demonstrating that open, efficient models can rival proprietary giants. By focusing on data quality and diversity rather than sheer scale, the paper provides a practical recipe for building high-performance LLMs with limited resources. The architectural choices (pre-norm, SwiGLU, RoPE) have become standard in subsequent models. The release of LLaMA (though initially restricted) spurred a wave of open-source LLM development, including fine-tuned variants like Alpaca and Vicuna. The main limitation is the performance gap on knowledge-intensive tasks like MMLU, likely due to less exposure to book corpora. Nonetheless, LLaMA set a new benchmark for efficiency and transparency in foundation models.