The Embedder's Dilemma: LLMs Are Better, but at What Cost?

Analysis

Why This Paper Matters

This paper addresses a critical, practical question for AI practitioners: should you replace your text-embedding pipeline with a large language model? With the rapid adoption of LLMs, many teams assume that bigger and more capable models are always better, but the cost implications are often overlooked. The authors provide a rigorous, controlled comparison that quantifies the trade-offs, showing that while LLMs and embedding models achieve similar aggregate quality, the cost difference is staggering—up to 1,431x. This is a wake-up call for practitioners who may be over-engineering their pipelines with expensive LLMs when simpler embedding models suffice.

The study's scope is impressive: 10 LLMs across six families, 26 embedding models, and 37 tasks covering a wide range of NLP benchmarks. By including both open and proprietary models, the authors ensure that the findings are relevant to a broad audience. The paper also introduces the concept of 'reasoning tokens' as a major cost driver, which is a novel and actionable insight. This work is likely to influence how teams design their retrieval and classification systems, encouraging a more cost-aware approach.

Technical Contributions

• Comprehensive Benchmarking: The authors created a unified benchmark across 37 tasks, including classification, STS, clustering, pair classification, and retrieval, enabling a fair comparison between LLMs and embedding models.
• Cost-Aware Analysis: They introduce a cost metric (USD per benchmark pass) and measure token processing speed on the same GPU, providing a holistic view of efficiency beyond raw quality.
• Reasoning Token Ablation: They systematically vary the reasoning budget (number of reasoning tokens) for LLMs and show that reducing it can preserve or even improve retrieval quality for most models, while significantly cutting costs.
• Pareto Frontier Identification: By plotting quality against cost, they identify the set of models that offer the best trade-offs, which is a practical tool for model selection.
• Public Resources: The release of code, datasets, and results at https://github.com/embeddings-benchmark/embedders-dilemma enables reproducibility and further research.

Results

The headline result is that the best LLM (Gemini 3.1 Pro) achieves 77.6 average score, while the best embedding model achieves 77.2—a difference of only 0.4 points. However, the cost disparity is enormous: the LLM costs USD 154 per benchmark pass, while the embedding model costs USD 0.11, a 1,431x difference. Open LLMs are also significantly slower, processing tokens 2.5 to 736x more slowly than embedding models on the same GPU.

Task-specific analysis reveals that LLMs excel at reasoning-heavy retrieval, while embedding models dominate classification. On clustering, STS, and pair classification, the two paradigms are on par. The reasoning token ablation shows that reasoning tokens account for 28-81% of LLM inference cost, and reducing them can improve retrieval quality for most models, suggesting that many LLMs are over-reasoning. The Pareto frontier includes the leading embedding models and only one LLM (Gemini 3.1 Pro), indicating that for most use cases, embedding models offer the best cost-quality trade-off.

Significance

This paper provides a clear, evidence-based answer to a pressing question in the AI community. It challenges the assumption that LLMs are always the best choice and demonstrates that a division of labor—using embedding models for similarity, classification, and clustering, and reserving LLMs for reasoning-intensive retrieval—is both cost-effective and quality-preserving. The findings have immediate practical implications for system design, potentially saving organizations significant computational and financial resources.

Moreover, the paper introduces a methodology for cost-aware benchmarking that could become a standard practice in the field. By making their code and data public, the authors enable others to extend this analysis to new models and tasks. As LLMs continue to evolve, this work will serve as a valuable reference for making informed decisions about model deployment, ensuring that performance gains are weighed against their true cost.