Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness

Analysis

Why This Paper Matters

Translation benchmarks have long been a cornerstone of progress in multilingual NLP, yet conventional single-version test sets suffer from two intertwined problems: contamination and inadequate localisation assessment. When a benchmark is publicly released, models may inadvertently or deliberately train on it, inflating scores without reflecting genuine translation ability. At the same time, a single reference translation often fails to capture the legitimate variation across locales, registers, and cultural contexts, making it difficult to distinguish a model that truly localises from one that merely memorised a canonical answer.

Cultivar addresses both issues by introducing a contrastive, locale-oriented translation benchmark. Instead of one static test set, it provides multiple controlled variants that differ in locale-specific aspects. This design turns evaluation into a diagnostic tool: if a model performs uniformly well across variants, it likely generalises; if it performs well only on the canonical version, contamination or overfitting is suspected. Similarly, by contrasting locale-appropriate translations with near-miss alternatives, the benchmark can quantify how robustly a model adapts to localisation requirements rather than defaulting to a dominant locale.

For practitioners, this matters because deployment decisions increasingly depend on translation quality across diverse markets. A benchmark that can separate true localisation robustness from contamination-driven scores offers a more trustworthy signal for model selection, fine-tuning, and safety auditing.

Technical Contributions

• Contrastive benchmark design: Cultivar pairs or groups translation instances that differ along locale-oriented dimensions, enabling controlled comparisons rather than absolute scoring alone.
• Contamination diagnosis: The multi-version structure allows researchers to detect whether a model's performance is inflated by exposure to a specific canonical test version.
• Localisation robustness quantification: By measuring performance gaps across locale variants, the benchmark provides a direct estimate of how well a model adapts to localisation demands.
• Analyses beyond single-version tests: The design supports diagnostic analyses that are impossible when only one reference translation exists, such as variant sensitivity and consistency profiling.

Results

The abstract does not report concrete metrics, baselines, or numerical comparisons. Its claims are methodological: Cultivar enables contamination diagnosis and localisation robustness quantification. Therefore, no specific accuracy, BLEU, COMET, or other scores can be cited from the provided text. The paper's contribution is the benchmark and the analytical capabilities it unlocks, not a set of empirical leaderboard results in the abstract.

Significance

If adopted, Cultivar could shift translation evaluation from static leaderboard chasing toward diagnostic assessment. It gives researchers a principled way to audit contamination and gives practitioners a clearer view of whether a model will perform well in a specific locale. More broadly, it highlights that robustness and localisation are measurable properties, not afterthoughts, and encourages the community to build evaluation suites that reflect the pluralism of real-world language use.