LLaMA 2 Long

Analysis

Why This Paper Matters

Large language models have traditionally struggled with long contexts due to quadratic attention costs and positional encoding limitations. LLaMA 2 Long addresses this by extending the popular LLaMA 2 model to handle up to 32,768 tokens, a significant leap from the original 4,096 token limit. This is crucial for real-world applications like legal document analysis, scientific paper summarization, and multi-turn conversations where context spans thousands of tokens. The paper's approach is notable for its minimal architectural changes and reliance on synthetic data, making it accessible for practitioners.

How It Works

The authors start with LLaMA 2 checkpoints and perform continual pretraining with longer sequences. They avoid sparse attention because for the 70B model (hidden dimension 8192), the attention cost only becomes a bottleneck beyond 49,152 tokens. Instead, they use Flash Attention to handle the increased sequence length efficiently.

A key modification is to the RoPE (Rotary Position Embedding) positional encoding. By decreasing the base frequency hyperparameter, they reduce the decay of attention weights for distant tokens, allowing the model to better utilize long-range dependencies. This is a minimal but effective change.

For instruction tuning, the authors face the challenge of scarce long-context human demonstrations. They leverage the existing LLaMA 2 Chat RLHF dataset and augment it with synthetic self-instructed long data. The process involves selecting random chunks from long documents in the pretraining corpus, prompting LLaMA 2 Chat to generate QA pairs, and then using a self-critique step to verify answers. Both short and long answers are collected, and for long instruction data, padding tokens are added to the right to avoid truncation.

Results

On standard short-context benchmarks, LLaMA 2 Long performs on par or better than LLaMA 2, with significant improvements in coding, math, and knowledge-intensive tasks like MMLU. It even outperforms GPT-3.5 on MMLU and GSM8k, attributed to additional compute and knowledge from long data.

On long-context tasks, the models achieve superior performance, with the 7B version matching only Together-7B 32k, which uses supervised fine-tuning. The effective context utilization analysis shows that language modeling loss follows a power-law plus constant scaling with context length, with larger models benefiting more (higher β).

Instruction tuning results on ZeroSCROLLS show the 70B chat model outperforming GPT-3.5-turbo-16k on 7 of 10 tasks, despite using no human-annotated long-context data. Human evaluation also shows competitive performance against proprietary models like MPT-30B-chat, GPT-4, GPT-3.5-turbo-16k, and Claude-2.

Significance

LLaMA 2 Long provides a practical blueprint for extending existing LLMs to long contexts without costly full retraining. The key insights—minimal RoPE modification, data quality over length, and synthetic instruction data—are actionable for practitioners. The work also underscores that long-context capabilities can be achieved with relatively simple techniques, democratizing access to models that can handle extensive documents and conversations. Future work may explore further scaling, more diverse synthetic data, and integration with retrieval-augmented generation.