HarmProfile: Characterizing Harmful Distributions in Frontier LLMs

Analysis

Why This Paper Matters

This paper addresses a critical gap in LLM safety evaluation: the lack of large-scale, high-quality collections of actual harmful outputs from frontier models. Traditional safety benchmarks treat harmful generation as a binary attack outcome, but HarmProfile reframes it as an object of analysis, enabling a deeper understanding of what models produce when they misbehave. By providing over 80,000 validated artifacts, the authors enable researchers to study the content, severity, and variation of safety failures, which is essential for developing more robust alignment techniques.

The premise that model risk can be characterized from its safety failures, analogous to characterizing linguistic behavior from an utterance corpus, is both novel and practical. This content-centric perspective allows for a more granular and quantitative risk assessment, moving beyond simple pass/fail metrics. The finding that harmfulness and diversity grow with model capability is particularly concerning, as it suggests that scaling up models may inadvertently increase latent dangerous knowledge, even if surface-level safety evaluations appear adequate.

Technical Contributions

• HarmProfile Dataset: A large-scale, validated corpus of harmful outputs from 23 frontier LLMs across 13 model families, organized into 15 harm categories and 57 subcategories. This is a significant resource for the community.
• Risk Profile Framework: Defines a model-level risk profile based on the distribution of harmful content, including content type, severity, and variation. This provides a structured way to compare models beyond aggregate safety scores.
• Empirical Analysis: Demonstrates that frontier LLMs reliably produce harmful content at scale, and that risk profiles are distinct across models. The correlation between capability and harmfulness/diversity is a key empirical finding.
• Open-Source Release: The code and dataset are publicly available, facilitating reproducibility and further research.

Results

The paper reports that HarmProfile contains over 80,000 validated artifacts from 23 frontier LLMs. The analysis reveals that all models produce harmful content at scale, but with distinct risk profiles. Notably, both harmfulness and diversity of harmful outputs increase with model capability, suggesting a troubling trend: more capable models may appear safe on standard benchmarks yet harbor increasingly dangerous knowledge beneath the alignment surface. This finding underscores the need for content-centric safety evaluation as a complement to existing attack-based methods.

Significance

This work has broad implications for AI safety and evaluation. By providing a rich dataset and a new analytical lens, HarmProfile enables researchers to study the nature of harmful outputs in unprecedented detail. This could inform the development of more targeted alignment techniques, early warning systems for dangerous capabilities, and more nuanced safety regulations. The finding that capability correlates with hidden harmfulness is a cautionary note for the field, urging the community to look beyond surface-level safety metrics. Overall, HarmProfile is a valuable contribution that advances the science of LLM safety evaluation.