LLMLingua2

Analysis

Why This Paper Matters

Prompt compression is a critical challenge for deploying large language models (LLMs) efficiently, as long prompts increase computational cost and latency. Existing methods often require task-specific fine-tuning or rely on heuristics that fail to generalize. LLMLingua2 addresses this gap by proposing a task-agnostic compression approach that can be applied to any prompt without retraining. This is particularly significant for real-world applications where prompts vary widely and retraining for each task is impractical.

The use of data distillation to generate training data is a clever way to leverage the knowledge of a larger model (the teacher) to train a smaller, efficient compressor. This technique helps the model learn which tokens are redundant across many tasks, leading to better generalization. By framing compression as token classification, the method is both simple and effective, allowing for fine-grained control over which parts of the prompt to keep.

Technical Contributions

• Task-agnostic compression: The model is trained once and can compress prompts for any downstream task without additional fine-tuning.
• Data distillation pipeline: A teacher LLM is used to generate compressed versions of prompts, which serve as training data for the token classifier.
• Transformer encoder for token classification: A lightweight Transformer encoder predicts a binary label (keep or discard) for each token, enabling efficient compression.
• Generalizability: The method is evaluated on multiple benchmarks (e.g., question answering, summarization, reasoning) and shows consistent performance across tasks.

Results

The paper demonstrates that LLMLingua2 achieves compression ratios of 2x to 5x with minimal degradation in task performance. For example, on the GSM8K math reasoning dataset, it retains over 90% of the original accuracy while compressing prompts by 3x. Compared to prior task-agnostic methods like Selective Context, LLMLingua2 shows better retention of key information, especially in long prompts. The method also outperforms task-specific compressors on unseen tasks, highlighting its generalizability.

Significance

LLMLingua2 has the potential to make LLM inference more cost-effective and faster, especially for applications with long context windows like document analysis or multi-turn conversations. By removing the need for task-specific tuning, it lowers the barrier for deploying compression in production systems. This work also opens avenues for further research into learned compression for LLMs, such as integrating compression directly into the model architecture or extending it to multimodal inputs.