Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation

Analysis

Why This Paper Matters

As AI systems increasingly rely on heterogeneous models and inference-time strategies, routing queries to the most suitable specialist is critical for balancing quality and cost. However, estimating each specialist's expected return is itself costly—cheap estimators are noisy, while accurate ones (e.g., fine-tuned models with retrieval or reasoning traces) are expensive. This paper addresses a fundamental gap by formalizing this tradeoff as Pandora's Box, a classical optimal search problem with costly inspection. By providing closed-form policies, it offers a theoretically grounded and practical solution for when to invest in more accurate value estimation.

The significance extends beyond mere efficiency. The paper introduces both centralized (Pandora's Router) and decentralized (Pandora's Bidder) frameworks, acknowledging that real-world systems may have specialists that act independently. This dual perspective is crucial for scalable deployment, where centralized control may be infeasible. The findings on strategic behavior in decentralized settings highlight important implications for multi-agent AI systems, where incentives and information asymmetry can affect overall efficiency.

Technical Contributions

• Formalization as Pandora's Box: The routing problem is mapped to optimal search with costly inspection, enabling the use of classical decision-theoretic tools.
• Closed-form value-of-information expressions: Under a Gaussian signal model, the authors derive explicit conditions for when refining a value estimate is worth its cost, making the policy computationally efficient.
• Pandora's Router: A centralized policy that decides, for each specialist and input, whether to query the expensive estimator based on the expected value of information.
• Pandora's Bidder: A decentralized extension where specialists independently decide whether to invest in self-assessment before accepting a price, modeling strategic behavior.
• Experimental validation: The policies are tested across three domains—multi-LLM benchmarks, retrieval-augmented specialists, and variable inference-time reasoning—demonstrating broad applicability.

Results

The paper reports that Pandora's Router matches the routing quality of exhaustive estimation (i.e., querying the expensive estimator for all specialists) while significantly reducing the number of expensive estimator calls. This implies substantial cost savings without sacrificing output quality. In the decentralized setting, value-of-information reasoning improves allocative efficiency when competing estimates are accurate, but when estimates are noisy, it can increase the utility of a strategic specialist at the expense of others. This highlights a potential pitfall in open multi-agent systems where estimates may be unreliable.

Significance

This work provides a principled framework for cost-aware model routing, which is increasingly important as AI systems become more heterogeneous and inference costs rise. The closed-form policies are practical and can be integrated into existing routing systems. Moreover, the decentralized analysis opens new research directions on incentive alignment and robustness in multi-agent AI. The findings on strategic behavior could inform the design of mechanisms that prevent exploitation when estimates are noisy. Overall, this paper bridges decision theory and practical AI system design, offering both theoretical insights and actionable algorithms.