Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking

Analysis

Why This Paper Matters

Current retrieval evaluation systems, such as nDCG, score documents independently and aggregate them, ignoring critical inter-document interactions like redundancy, conflict, and complementarity. As LLMs and AI agents become primary consumers of search results, the quality of the entire document set—not just individual relevance—determines the upper bound of downstream generation. This paper addresses a fundamental gap: there is no systematic way to evaluate what makes one document set better than another. By proposing a complete evaluate-diagnose-optimize framework, the authors provide both a diagnostic tool and a practical solution for improving setwise quality.

The significance is amplified by the empirical finding that even the best reranker achieves only 45% coverage on the proposed benchmark, and cross-document coordination dimensions are universally weak. This suggests that current reranking methods are fundamentally insufficient for setwise optimization. The paper's training-free approach, Rubric4Setwise, directly addresses this by converting rubric-based criteria into selection signals, achieving superior downstream generation with fewer documents and search rounds. This is particularly relevant for real-world applications where computational cost and latency matter.

Technical Contributions

• SetwiseEvalKit: A three-level, nine-dimension evaluation benchmark covering both short-form and long-form scenarios, with approximately 28K high-quality evaluation rubrics. This is the first comprehensive benchmark designed specifically for document set quality, capturing dimensions like redundancy, conflict, and complementarity.
• Systematic evaluation of 12 rerankers: The study reveals that no single method maintains top performance across both short-form and long-form settings, and cross-document coordination is universally weak. This provides a clear diagnosis of current limitations.
• Rubric4Setwise: A training-free method that converts rubric-based evaluation criteria into document set selection signals. It does not require additional training or fine-tuning, making it easily deployable. The method achieves the best downstream generation performance with fewer documents and search rounds, and is the only method that maintains SOTA across both scenarios.

Results

• Even the best reranker achieves no more than 45% coverage on SetwiseEvalKit, indicating significant room for improvement.
• Cross-document coordination dimensions (e.g., redundancy, conflict) are universally weak across all 12 rerankers.
• Rubric4Setwise achieves the best downstream generation performance with fewer documents and fewer search rounds compared to all baselines.
• It is the only method that maintains state-of-the-art results across both short-form and long-form scenarios, validating the effectiveness of closing the loop from evaluation to optimization.

Significance

This work has broad implications for the AI field, particularly for information retrieval systems serving LLMs and AI agents. By shifting the evaluation paradigm from independent relevance scoring to setwise quality, it enables more nuanced and effective retrieval. The training-free nature of Rubric4Setwise means it can be immediately applied to existing systems without additional computational overhead. The benchmark and diagnostic insights provide a foundation for future research on setwise optimization, potentially influencing how search engines and retrieval-augmented generation systems are designed and evaluated.