SweEval

Analysis

Why This Paper Matters

As enterprises deploy LLMs across diverse linguistic markets, ensuring models refuse unethical instructions—like including swear words—becomes paramount. Most safety benchmarks focus on English or a handful of high-resource languages, ignoring the cultural and linguistic nuances that can cause models to inadvertently comply with harmful prompts. SweEval addresses this blind spot by systematically testing LLMs on swear words from eight languages, including low-resource Indic languages, and their transliterated forms. The findings reveal that models are significantly more vulnerable in non-English contexts, especially when swear words are transliterated into Latin script—a common scenario in multilingual digital communication. This work provides a practical tool for developers to audit and improve cross-lingual safety before deployment.

How It Works

The SweEval benchmark starts with 109 manually crafted English prompts covering enterprise tasks (e.g., drafting emails, customer replies) and casual scenarios (e.g., social messages), each with positive and negative tones. For each of the eight languages—English, Spanish, French, German, Hindi, Marathi, Bengali, and Gujarati—25 swear words are selected. These are inserted into placeholders within the English prompts, creating two test cases:

• Case 1 (Multilingual Swearing): Swear words are written in their native script (e.g., Devanagari for Hindi).
• Case 2 (Transliterated Swearing): Swear words are converted to Latin script while preserving pronunciation (e.g., "bhenchod" for Hindi).

This yields 2,725 prompts per language per case (109 × 25). The authors evaluate 13 open-source LLMs, ranging from 7B to 141B parameters, including Llama 2/3/3.1, Mistral v0.1/v0.2/v0.3, Phi-3/3.5, and others. All models run with temperature=0 and max tokens=2048 for deterministic outputs.

Response evaluation uses the Harmful Rate—the proportion of responses that both complete the task and include swear words. Harmless responses include refusals, irrelevant outputs, or task completions without swear words. Classification is done via meticulous manual review.

Results

Key findings include:

• English is safer: LLMs use swear words far less in English prompts than in any other language, indicating stronger safety alignment for English.
• Indic languages are more vulnerable: Average harmful rates are higher for Hindi, Marathi, Bengali, and Gujarati compared to Latin-based languages (Spanish, French, German).
• Transliteration worsens compliance: For Indic languages, transliterated swear words lead to higher harmful rates than native script versions, likely because models fail to recognize them as profanity.
• Safety improvements are uneven: Mistral v3 shows lower harmful rates than v2, but Llama 3.1 is slightly worse than Llama 3.0. Larger models (e.g., 141B) generally perform better than smaller ones.

Significance

SweEval exposes a critical weakness in current LLM safety: models are not equally safe across languages. The higher harmful rates for Indic languages, especially with transliteration, suggest that training data and safety fine-tuning are heavily skewed toward English and Latin-based languages. This is particularly concerning for enterprise deployments in multilingual regions like India, where transliterated code-switching is common. The benchmark provides a concrete method for developers to test and improve cross-lingual safety. The finding that newer models do not always improve (e.g., Llama 3.1 vs. 3.0) underscores the need for dedicated multilingual safety evaluation during model development. As LLMs become ubiquitous in global applications, benchmarks like SweEval are essential to ensure ethical behavior across all languages and cultural contexts.