Multi-expert Prompting with LLMs

Analysis

Why This Paper Matters

Large language models (LLMs) are increasingly used for knowledge-intensive tasks, but their responses often suffer from factual inaccuracies and biases. Prompting techniques have emerged as a low-cost way to steer LLM behavior, yet existing methods like ExpertPrompting rely on a single expert persona, which may not capture diverse perspectives. Multi-expert Prompting addresses this by simulating multiple experts and aggregating their insights, drawing on a well-established decision-making framework. This is significant because it offers a systematic, explainable approach to improve truthfulness and factuality, which are critical for real-world deployment.

The paper's grounding in the Nominal Group Technique (NGT) is particularly noteworthy. NGT is a structured method for group decision-making that encourages equal participation and reduces conformity pressure. By adapting NGT into seven subtasks, the authors provide a principled way to elicit and combine multiple expert viewpoints within a single LLM call. This not only enhances response quality but also makes the reasoning process more transparent, addressing the 'black box' concern of LLMs.

Technical Contributions

• Multi-expert Prompting Framework: Extends ExpertPrompting by generating multiple expert responses, aggregating them, and selecting the best, all in one chain of thought.
• Seven Subtasks from NGT: The method decomposes the process into seven carefully designed subtasks, including expert generation, response aggregation, and selection, ensuring a structured and reproducible approach.
• Single Chain of Thought: Unlike methods that require multiple LLM calls, this approach performs all steps in a single chain, making it efficient and cost-effective.
• Explainability: The aggregation and selection steps provide a rationale for the final answer, enhancing interpretability.
• Adaptability: The method is model-agnostic and does not require manual prompt construction, making it easy to apply across diverse scenarios.

Results

The paper reports that Multi-expert Prompting significantly outperforms ExpertPrompting and comparable baselines across multiple metrics. Specifically, it improves truthfulness, factuality, informativeness, and usefulness, while reducing toxicity and hurtfulness. The most striking result is an 8.69% improvement over the best baseline in truthfulness when using ChatGPT, achieving state-of-the-art performance. These gains are consistent across different models and tasks, indicating robustness.

The evaluation likely includes benchmarks such as TruthfulQA and other factuality tests, though the abstract does not specify. The reduction in toxicity and hurtfulness is also a notable contribution, as safety is a growing concern in LLM deployment.

Significance

This work has broad implications for prompt engineering and LLM alignment. By providing a structured, explainable method to improve response quality, it offers a practical tool for practitioners who need reliable and trustworthy LLM outputs. The use of NGT demonstrates how established decision-making frameworks can be adapted to AI, opening avenues for further research into other group dynamics techniques. Moreover, the efficiency of a single chain of thought makes it feasible for real-time applications. As LLMs become more integrated into decision support systems, methods like Multi-expert Prompting could become standard practice, enhancing both performance and user trust.