Multi-LLM Text Summarization

Analysis

Why This Paper Matters

Large language models (LLMs) have revolutionized text summarization, but they often struggle with long documents that exceed their context windows. This paper tackles that challenge head-on by introducing a multi-LLM framework that not only handles long texts but also significantly improves summary quality. The key insight is that multiple LLMs, each with their own biases and strengths, can collaborate to produce more robust and accurate summaries than any single model alone. This is particularly relevant for practitioners who need reliable summarization for lengthy reports, research papers, or legal documents.

How It Works

The framework addresses the context window limitation by first chunking the source document into manageable pieces, then summarizing each chunk independently. This two-stage process ensures that even very long texts can be processed without truncation.

At each round, the process involves two steps: generation and evaluation. In the generation step, k different LLMs each produce a summary of the same input text. This creates a diverse pool of candidate summaries, capturing different perspectives and reducing the risk of any single model's weaknesses dominating.

The evaluation step differs between the two approaches. In the centralized approach, a single LLM (the central agent) evaluates all candidate summaries and selects the best one, also providing a confidence score. If the confidence is below a threshold, the process iterates: in subsequent rounds, each LLM receives the original text along with summaries from the previous round, and generates an improved summary. This iterative refinement continues until the central LLM is confident enough or a maximum number of rounds is reached.

In the decentralized approach, all LLMs participate in evaluation. Each model votes on which summary it considers best, and convergence is achieved when a majority agrees. If no majority emerges after the first round, the process enters additional rounds with updated prompts that include previous summaries. If consensus is still not reached after the maximum rounds, a tie-breaker model makes the final selection.

Results

The paper evaluates the framework on two datasets: ArXiv (scientific papers) and GovReport (government reports). The multi-LLM approach significantly outperforms single-LLM baselines, with the centralized method achieving an average 73% improvement and the decentralized method 70% improvement. Notably, using just two LLMs and a single round of generation and evaluation yields substantial gains, making the approach highly cost-effective. Adding more LLMs or additional rounds does not lead to further improvements, suggesting that the framework quickly reaches an optimal performance plateau.

Significance

This work offers a practical blueprint for improving summarization quality without requiring massive computational resources. The finding that two LLMs and a single round are sufficient for significant gains is particularly valuable for practitioners with limited budgets. The framework is also flexible: it can be implemented with any set of LLMs, and the centralized and decentralized variants allow trade-offs between simplicity and robustness. By demonstrating that collaboration among LLMs can yield better results than any single model, this paper opens up new avenues for multi-agent systems in NLP tasks beyond summarization.