Robustbench: a standardized adversarial robustness benchmark

Analysis

Why This Paper Matters

Adversarial robustness research has been plagued by inconsistent evaluation protocols, leading to inflated claims and difficulty in comparing defenses. RobustBench addresses this by providing a standardized benchmark that uses a reliable attack (AutoAttack) and a unified threat model. This is crucial for the field because it allows researchers to objectively measure progress and identify which defenses actually work.

The paper's significance lies in its role as a community resource. By offering a public leaderboard and a collection of pre-trained models, it lowers the barrier to entry for new researchers and provides a common ground for comparing methods. This has a profound impact on the credibility of robustness research, as it filters out ineffective defenses and highlights the true state of the art.

Technical Contributions

• AutoAttack Integration: The benchmark employs AutoAttack, an ensemble of four attacks (APGD-CE, APGD-DLR, FAB, and Square Attack), which is parameter-free and has been shown to be highly reliable in evaluating robustness.
• Standardized Threat Model: It defines a fixed perturbation budget (L-infinity, epsilon=8/255) and a consistent evaluation protocol, ensuring that results are comparable across different models.
• Public Leaderboard and Model Zoo: The benchmark provides a leaderboard with robust accuracy scores and hosts pre-trained models, enabling easy reproduction and further analysis.
• Robustness Verification: By using AutoAttack, the benchmark reduces the risk of overestimating robustness due to weak attacks, providing a more accurate measure of true robustness.

Results

The paper reports robust accuracy for various models under AutoAttack. Notably, it shows that many defenses that claimed high robustness under weaker attacks fail under AutoAttack, with robust accuracy dropping significantly. For example, some models that reported >90% accuracy under PGD attacks only achieve <50% under AutoAttack. In contrast, adversarially trained models (e.g., those using TRADES or adversarial training with early stopping) achieve higher robust accuracy, often in the 50-60% range on CIFAR-10. The leaderboard provides a clear ranking, demonstrating that the benchmark effectively differentiates between robust and non-robust models.

Significance

RobustBench has become a de facto standard in adversarial robustness research. It has been widely adopted by the community, with many subsequent papers using it to evaluate their methods. The benchmark's emphasis on reliable evaluation has led to a more honest assessment of defenses, reducing the number of false claims and encouraging the development of genuinely robust models. Moreover, the public availability of models and code facilitates reproducibility and accelerates research. The benchmark also highlights the gap between human perception and model robustness, underscoring the need for further research in this area. Overall, RobustBench has significantly improved the scientific rigor of adversarial robustness research and serves as a model for other domains requiring standardized evaluation.