mBART

Analysis

Why This Paper Matters

mBART is a landmark paper that brought the success of denoising auto-encoding pre-training (BART) to the multilingual domain. Prior to mBART, most multilingual pre-training focused on encoder-only models (e.g., mBERT) or encoder-decoder models trained on parallel data. mBART showed that a simple extension of BART to multiple languages, using only monolingual data, could yield dramatic improvements in machine translation quality. This was particularly impactful for low-resource languages where parallel data is scarce.

The paper demonstrated that the denoising objective forces the model to learn both language-specific and cross-lingual representations, effectively creating a shared latent space across languages. This insight opened the door for many follow-up works that combined mBART with back-translation, fine-tuning on parallel data, or other pre-training objectives.

Technical Contributions

• Multilingual BART objective: Applies the BART denoising auto-encoding objective to a concatenation of monolingual corpora from 25 languages, with a shared vocabulary and a single encoder-decoder model.
• Language-specific tokens: Uses special language ID tokens to indicate the target language during fine-tuning, enabling zero-shot translation.
• Large-scale pre-training: Pre-trains on 25 languages with a total of 500GB of text, using a 12-layer encoder and 12-layer decoder architecture.
• Fine-tuning strategy: Fine-tunes the pre-trained model on parallel data for machine translation, showing that the pre-trained model provides a strong initialization that accelerates convergence and improves final performance.

Results

• On WMT16 English-Romanian, mBART achieved 38.0 BLEU, outperforming previous state-of-the-art by 1.5 BLEU.
• On WMT16 English-German, mBART achieved 30.5 BLEU, a 1.0 BLEU improvement over strong baselines.
• For low-resource language pairs like English-Nepali, mBART improved BLEU by over 5 points compared to training from scratch.
• Zero-shot translation (e.g., Romanian-German) also showed substantial gains, indicating that the model learned a truly multilingual representation.

Significance

mBART had a profound impact on the field of multilingual NLP. It demonstrated that monolingual data alone, when combined with a denoising objective, can produce a powerful multilingual sequence-to-sequence model. This shifted the focus from parallel data to monolingual data for cross-lingual tasks. The approach was quickly adopted by major translation systems and inspired variants like mBART-50 (extending to 50 languages) and XLM-E (combining with masked language modeling). mBART remains a foundational building block for many state-of-the-art multilingual generation systems.