Revisiting demonstration selection strategies in in-context learning

Analysis

Why This Paper Matters

In-context learning (ICL) has become a cornerstone of modern LLM usage, allowing models to adapt to new tasks with just a few examples. However, the choice of which examples (demonstrations) to include in the prompt can dramatically affect performance. Despite its importance, the field has lacked a systematic comparison of selection strategies, with many practitioners relying on random selection or simple heuristics. This paper addresses that gap by providing a rigorous, unified evaluation of existing strategies and introducing a new method that outperforms them.

The significance of this work lies in its practical implications. As LLMs are increasingly deployed in real-world applications, optimizing prompt composition without additional training is a cost-effective way to boost performance. The paper's findings offer clear guidance on when to prioritize similarity versus diversity, helping practitioners make informed decisions. Moreover, the proposed method is simple to implement and does not require any model fine-tuning, making it accessible to a wide audience.

Technical Contributions

The paper makes several key technical contributions:

• Unified evaluation framework: The authors standardize the evaluation of demonstration selection strategies across multiple tasks, model sizes, and numbers of demonstrations, enabling fair comparisons.
• Hybrid selection method: They propose a novel approach that scores candidate demonstrations based on a weighted combination of semantic similarity to the test input and diversity relative to already selected examples. This balances the benefits of both criteria.
• Comprehensive analysis: The paper provides an in-depth analysis of why different strategies work, showing that similarity-based selection excels for tasks with clear input-output mappings (e.g., sentiment classification), while diversity becomes more important for tasks with broader output spaces (e.g., open-ended generation).
• Practical guidelines: The authors distill their findings into actionable recommendations, such as using similarity-based selection for classification tasks and diversity-aware selection for generation tasks.

Results

The empirical results demonstrate the effectiveness of the proposed method. Across a suite of classification and generation benchmarks, the hybrid approach consistently outperforms random selection and existing baselines. For instance, on the SST-2 sentiment analysis task, the method achieves a 3.2% accuracy improvement over random selection with 8 demonstrations. On the TREC question classification task, the gain is 5.1%. The method also shows robustness across different model sizes (e.g., GPT-2, GPT-3, and LLaMA) and numbers of demonstrations (4, 8, and 16).

Notably, the paper reveals that the performance gap between selection strategies narrows as the number of demonstrations increases, but the proposed method still maintains an edge. Additionally, the authors find that similarity-based selection is particularly effective for tasks with high input-output correlation, while diversity helps for tasks with more varied outputs.

Significance

The broader impact of this work is twofold. First, it provides a solid empirical foundation for understanding demonstration selection, moving the field from ad-hoc practices to principled guidelines. Second, the proposed method offers a simple yet effective tool that can be immediately integrated into existing ICL pipelines, potentially improving performance across a wide range of applications without any additional training cost.

Future work could extend this analysis to multilingual settings, larger models, and more complex tasks such as multi-step reasoning. The paper also opens the door to more sophisticated selection methods that could adaptively choose demonstrations based on the test input's characteristics. Overall, this research contributes to making LLMs more reliable and efficient in few-shot scenarios, which is crucial for their deployment in real-world systems.