In-context learning with retrieved demonstrations for language models: A survey

Analysis

Why This Paper Matters

In-context learning (ICL) has emerged as a powerful paradigm for adapting large language models (LLMs) to new tasks without fine-tuning. The selection of demonstrations—the input-output examples provided in the prompt—is critical to ICL performance. This survey addresses the growing body of work on retrieving demonstrations from external corpora to improve ICL, a topic that has seen rapid advancement but lacks a unified overview. By systematically categorizing retrieval methods and synthesizing empirical findings, this paper fills a crucial gap for researchers and practitioners who need to navigate the landscape of demonstration retrieval.

The survey's significance lies in its comprehensive scope, covering not only the methods but also the benchmarks and evaluation protocols used in the field. As LLMs are increasingly deployed in real-world applications, understanding how to optimally select demonstrations can lead to more efficient and accurate models. This paper provides a structured foundation for future research, helping to standardize evaluation and identify promising directions.

Technical Contributions

• Taxonomy of Retrieval Methods: The survey categorizes demonstration retrieval approaches into several families, including unsupervised similarity-based retrieval, supervised learning-based retrieval, and hybrid methods. This taxonomy helps clarify the design space and trade-offs.
• Analysis of Retrieval Signals: It examines various signals used for retrieval, such as embedding similarity, task-specific features, and model feedback, and discusses their relative merits.
• Benchmark and Evaluation Review: The paper compiles a list of commonly used benchmarks and evaluation metrics, highlighting inconsistencies and suggesting best practices for fair comparison.
• Identification of Open Challenges: It outlines unresolved issues, such as the scalability of retrieval to large corpora, the robustness of retrieval to distribution shift, and the interpretability of retrieved demonstrations.

Results

While the survey does not present new experimental results, it synthesizes findings from numerous studies. A consistent theme is that retrieved demonstrations outperform random selection, often by significant margins. For example, in many classification and generation tasks, retrieval-based ICL can improve accuracy by several percentage points. However, the gains are not uniform: they depend on the task type, the quality of the retrieval model, and the size of the demonstration pool. The survey also notes that some retrieval methods, while effective, may introduce computational overhead that could be prohibitive in real-time applications.

Significance

The broader impact of this survey is twofold. First, it provides a clear roadmap for researchers working on demonstration retrieval, potentially accelerating progress by consolidating knowledge and highlighting gaps. Second, it informs practitioners about best practices for deploying ICL with retrieved demonstrations, which can lead to more reliable and efficient LLM applications. As the field matures, the survey's call for standardized benchmarks and more robust retrieval methods will likely shape future research agendas, ultimately contributing to more capable and adaptable language models.