Accuracy and Order Sensitivity Diverge Under Label-Free Strategies

Analysis

Why This Paper Matters

Multiple-choice benchmarks are ubiquitous in LLM evaluation, but they conflate knowledge with sensitivity to option order. This paper tackles a critical question: can we design label-free strategies that eliminate positional bias and thus yield more accurate measures of model knowledge? The findings are surprising and important: removing positional influence does not reliably improve accuracy, and in fact, cyclic permutation often helps. This challenges the prevailing assumption that positional bias is a major source of error in MCQ evaluations.

The paper's decomposition of the pipeline—separating the effect of withholding options from the matching step—provides a clear diagnosis. The bottleneck is not the matching algorithm but the act of withholding options, which likely forces the model to generate answers without the benefit of option-specific cues. This insight is valuable for researchers designing evaluation protocols, as it suggests that simply hiding option labels is not a panacea.

Technical Contributions

• Two label-free strategies: (1) Generation-then-matching: the model generates an answer without seeing options, then a matcher (LLM) aligns it to the correct option. (2) Isolated scoring: each option is scored independently, which is positionally unbiased by construction.
• Complete decomposition: The authors break down the pipeline to isolate the effect of withholding options versus the matching step, allowing them to attribute performance differences precisely.
• Evaluation of debiasing metrics: They test both an aggregate recall imbalance measure and a per-question order sensitivity measure in two-stage prompting, showing that neither reliably indicates debiasing.
• Comparison with cyclic permutation: They compare against cyclic permutation of options, which often improves accuracy, providing a contrast to label-free methods.

Results

The paper reports that neither label-free strategy reliably improves accuracy over the baseline. The only configuration that consistently matches the baseline is one that shows the model all options paired with an LLM matcher. Eliminating positional influence entirely does not yield reliable accuracy gains, while cyclic permutation often improves them. The decomposition shows that the bottleneck is withholding options, not the matching step. Additionally, both aggregate recall imbalance and per-question order sensitivity fail to show reliable debiasing in two-stage prompting.

Significance

This paper has significant implications for LLM evaluation. It cautions against assuming that removing positional bias will automatically improve benchmark validity. The finding that cyclic permutation often helps suggests that option order can sometimes provide useful cues, and that label-free strategies may inadvertently remove those cues. The decomposition methodology offers a template for future studies to isolate sources of bias in evaluation pipelines. Ultimately, this work pushes the field toward more nuanced evaluation designs that account for the complex interplay between knowledge, bias, and prompting strategies.