Learning When to Trust via Selective Context Preference Optimization

Analysis

Why This Paper Matters

This paper addresses a critical yet underappreciated failure mode in language models: their susceptibility to misleading external signals. While prior work focuses on making models robust to irrelevant or adversarial context, the authors argue that resistance alone is insufficient—a model that ignores all context is useless when context is genuinely helpful. By framing the problem as selective trust, they highlight the need for models to discern when to trust context and when to ignore it.

The introduction of MIST, a human-annotated benchmark with four matched conditions, provides a controlled setting to measure this behavior. The SC2W metric is particularly valuable because it directly quantifies the flip from correct to wrong due to misleading signals, offering a clear target for improvement. The finding that this susceptibility is universal across popular open-sourced models underscores the severity of the issue.

Technical Contributions

• MIST Benchmark: A human-annotated dataset where each reasoning item is rendered under four conditions: clean (no extra context), misleading (incorrect context), correct-context (supporting context), and irrelevant-context (neutral context). This design enables controlled comparisons.
• SC2W Metric: A paired metric that counts how often a misleading signal flips a clean-correct answer to wrong. This is a more direct measure of vulnerability than overall accuracy.
• SCOPE Method: A training approach that mines clean-correct/misleading-wrong failure pairs and applies DPO over matched preference pairs. Crucially, the pairs are balanced equally across all four conditions, preventing the model from simply ignoring all context.
• Balanced DPO Objective: By including clean, correct, and irrelevant context pairs, SCOPE encourages selective trust rather than blanket resistance.

Results

The paper reports that SCOPE substantially reduces SC2W on popular open-sourced models. Importantly, this improvement does not come at the cost of accuracy when the added context is clean, correct, or irrelevant. This suggests that the model learns to differentiate between trustworthy and misleading signals, rather than becoming overly cautious.

While specific numerical results are not provided in the abstract, the qualitative claim of "substantial reduction" indicates a significant improvement. The universal susceptibility finding across models highlights the broad applicability of the approach.

Significance

This work has the potential to reshape how the AI community evaluates and trains language models. By advocating for selective trust as a core capability, it moves beyond simple robustness and encourages a more nuanced understanding of context utilization. The MIST benchmark and SC2W metric provide concrete tools for future research, and SCOPE offers a practical training method that can be integrated into existing pipelines.

The broader impact extends to real-world applications where models must decide whether to trust retrieved information, user-provided context, or external knowledge. This is especially relevant for retrieval-augmented generation, fact-checking, and decision-support systems. The paper's argument that models should be judged on selective trust, not resistance alone, is a timely and important call to action for the field.