Nemotron-4 15B

Analysis

Why This Paper Matters

In the race to build ever-larger language models, Nvidia's Nemotron-4 15B offers a refreshing counterpoint: a relatively compact 15-billion-parameter model that punches well above its weight. Trained on 8 trillion tokens, this multilingual model demonstrates that thoughtful data curation and training strategies can yield performance that rivals or exceeds models several times its size. For AI practitioners, this paper is a practical guide on how to maximize model capability without ballooning compute costs, especially for multilingual and code tasks.

How It Works

Nemotron-4 15B adopts a standard decoder-only Transformer architecture with causal attention masks. Key design choices include Rotary Position Embeddings (RoPE), squared ReLU activations in MLP layers, and Grouped Query Attention to reduce memory and speed up inference. The model has 3.2 billion embedding parameters and 12.5 billion non-embedding parameters, with no bias terms and untied input-output embeddings.

The training data is a carefully blended mix: 70% English natural language, 15% multilingual natural language, and 15% source code. The English corpus includes web documents, news, scientific papers, and books. The multilingual and code components cover 53 natural languages and 43 programming languages respectively.

A SentencePiece BPE tokenizer with 256,000 tokens is trained on a random subset of the data, with non-English data upsampled to improve low-resource language coverage. The tokenizer preserves whitespace, splits digits, and uses byte-level fallback for unknown characters.

A notable innovation is the two-phase training schedule. Similar to Gemini, the authors switch data distribution and learning rate decay near the end of training. The first phase upweights higher-quality sources, while the second introduces a small number of benchmark-style alignment examples and further upweights areas where the model underperforms.

Results

Nemotron-4 15B is evaluated across multiple benchmarks. On commonsense reasoning (SIQA, ARC, PIQA, Winogrande, Hellaswag), it achieves the strongest average performance among compared baselines.

On aggregated benchmarks, it scores best on BBH, significantly outperforming LLaMA-2 70B, and achieves a competitive MMLU score.

In math and code, it matches Gemma 7B on math but lags behind Baichuan-2 and QWEN. On code, it is on par with QWEN 14B but slightly behind Gemma 7B. It outperforms Mistral 7B and LLaMA-2 13B/34B on both.

The model truly shines in multilingual tasks. For classification (XCOPA), it achieves the best performance, with a 12% improvement in the four-shot setting.

For generation (MGSM, TyDiQA), it significantly outperforms the next best model, PaLM 62B-cont, and achieves nearly 30% improvement over the closest competitor.

In machine translation (FLORES-101), Nemotron-4 15B outperforms LLaMA-2 13B by 90.2% and Baichuan-2 13B by 44.1%, and excels not only in Chinese-to-English but also in direct translation to other languages.

Significance

Nemotron-4 15B is a testament to the power of data-centric AI. By carefully balancing English, multilingual, and code data, and by employing a two-phase training schedule, Nvidia has created a model that competes with much larger counterparts. For practitioners, this paper offers actionable insights: invest in data quality and distribution, use a large tokenizer with byte-level fallback, and consider a final training phase with shifted data weights. The model's multilingual prowess is particularly noteworthy, suggesting that smaller models can be highly effective for global applications. While it has room for improvement in math and code, its overall efficiency and performance make it a strong candidate for deployment in resource-constrained environments.