PaLM 2

Analysis

Why This Paper Matters

PaLM 2 represents a significant step forward in large language model development, particularly for multilingual and code generation tasks. By training on a more diverse dataset with a mixture of objectives, it achieves stronger performance than its predecessor PaLM across a wide range of benchmarks, often rivaling or surpassing GPT-4. This paper is crucial for AI practitioners seeking to build more capable, efficient, and globally useful language models.

How It Works

PaLM 2 is the successor to PaLM, designed to be more compute-efficient and to leverage a richer training mixture. The training data includes a higher percentage of non-English text, as well as parallel data (source-target pairs) covering hundreds of languages, often with English on one side. This multilingual focus is a key differentiator.

The model is trained with a combination of pre-training objectives, each targeting different aspects of language understanding. This multi-objective approach helps PaLM 2 grasp syntax, semantics, and reasoning more effectively than single-objective models.

Scaling law experiments were conducted to determine the optimal allocation of compute between model size and data. The results indicate that as the FLOP budget increases, both the amount of training data (D) and model size (N) should grow proportionally for best performance.

Results

PaLM 2 demonstrates substantial improvements over PaLM in nearly every evaluated category:

• Language Proficiency Exams: PaLM 2 passes professional-level exams in multiple languages, outperforming PaLM across the board.

• English QA and Classification: Large gains are seen on most tasks, especially on adversarial NLI (ANLI) and reading comprehension (RACE).

• Multilingual QA: PaLM 2 consistently outperforms PaLM on TyDi QA, with larger models excelling in the no-context setting. Improvements are particularly notable for low-resource languages like Telugu, Swahili, and Indonesian, as well as non-Latin scripts.

• Reasoning: PaLM 2 surpasses PaLM on all reasoning datasets and is competitive with GPT-4. On the multilingual XCOPA dataset, it sets a new state of the art for under-represented languages.

• Coding: The small, specialized PaLM 2-S* model outperforms the much larger PaLM-540B-Coder on code generation benchmarks like ARCADE, despite being more efficient.

• Translation: PaLM 2 beats both PaLM and Google Translate on WMT21 and FRMT benchmarks, as measured by BLEURT and MQM scores.

• Memorization: On average, PaLM 2 memorizes less training data than PaLM. However, it shows higher memorization of frequently repeated n-grams, and memorization risk is slightly elevated for tail languages.

Significance

PaLM 2's success underscores the value of diverse, multilingual training data and multi-objective learning. For practitioners, it demonstrates that smaller, task-specific models (like PaLM 2-S*) can achieve high performance with lower latency and cost. The memorization analysis provides actionable insights for building more privacy-preserving LLMs. Overall, PaLM 2 sets a new standard for multilingual and code-capable language models, pushing the frontier of what is achievable with efficient training.