Stress Testing Large Reasoning Models

Analysis

Why This Paper Matters

Current evaluation of Large Reasoning Models (LRMs) relies heavily on single-question benchmarks like MATH500, which suffer from data contamination and ceiling effects (e.g., DeepSeek-R1 achieves 97.0%). This paper argues that such paradigms fail to capture the multi-context pressure typical of real-world deployment, where models must juggle multiple problems, prioritize context, and resist interference. By introducing REST, the authors address a critical gap: the need for stress testing that mimics realistic cognitive demands.

The significance lies in its potential to reshape how we benchmark reasoning models. REST not only exposes performance degradation in SOTA models but also provides stronger discriminative power, distinguishing models that appear equivalent under traditional benchmarks. This is crucial for guiding model development and selection, as it reveals hidden weaknesses that single-question tests miss.

Technical Contributions

• REST Framework: Simultaneously presents multiple problems to LRMs, moving beyond sequential single-question testing.
• New Evaluation Dimensions: Assesses contextual priority allocation, cross-problem interference resistance, and dynamic cognitive load management—capabilities not covered by existing benchmarks.
• Identification of Overthinking Trap: Shows that excessive reasoning (overthinking) under multi-context pressure leads to performance degradation, a novel insight.
• Training Technique Analysis: Demonstrates that models trained with 'long2short' techniques retain more single-problem accuracy under stress, suggesting a path to more robust models.
• Cost-Efficiency: Reduces reliance on continuous human annotation by providing a reusable stress-testing paradigm.

Results

The paper reports that even state-of-the-art models like DeepSeek-R1 exhibit substantial performance degradation under REST. Crucially, REST reveals pronounced performance differences among models that show similar, near-ceiling performance on single-question evaluations. For instance, while DeepSeek-R1 achieves 97.0% on MATH500, its performance under stress testing drops significantly, highlighting the limitations of current benchmarks. The overthinking trap is identified as a critical factor, and models trained with long2short techniques outperform standard-trained counterparts under REST, preserving more accuracy.

Significance

REST establishes a new evaluation paradigm that better reflects real-world reasoning demands, where models must handle multiple contexts simultaneously. This could influence training objectives, encouraging methods like long2short to mitigate overthinking and improve robustness. The framework's cost-efficiency and future-proof nature make it a valuable tool for the AI community, potentially becoming a standard stress test for LRMs. By exposing hidden performance gaps, REST aids in selecting models for deployment and drives progress toward more reliable reasoning systems.