Lossless Tensor Compression as Program Synthesis

Analysis

Why This Paper Matters

Model checkpoints have become a significant storage and bandwidth bottleneck in AI development, with models growing in size and number. Traditional general-purpose compressors like zstd and gzip treat tensors as opaque byte streams, missing structural redundancies. Existing tensor-specific compressors often rely on fixed pipelines that are format-specific and not adaptable to diverse tensor patterns. Brevis addresses this gap by framing lossless tensor compression as a program synthesis problem, allowing the compressor to discover and exploit recurring structures in a data-driven manner.

The key innovation is the use of a typed DSL with reversible operators that can express common tensor patterns (e.g., repeated regions, floating-point fields). By synthesizing a program that reconstructs the tensor bit-exactly, Brevis effectively learns a custom compression algorithm for each tensor. This is a departure from both general-purpose and fixed-pipeline approaches, offering a flexible and adaptive solution that can be applied to any tensor format.

Technical Contributions

• Program Synthesis for Compression: Brevis treats compression as the task of finding a short program in a DSL that, when executed, reproduces the original tensor. This is a novel formulation that leverages program synthesis techniques for data compression.
• Typed DSL with Reversible Operators: The DSL is designed to capture tensor-specific structures, such as repeated regions and floating-point fields, using reversible operations to ensure losslessness.
• Bounded A Search with Production Prior*: A checkpoint-specific prior, learned from a small sample of tensors, guides the search for compact programs, making synthesis efficient and effective.
• Self-Contained Programs: The synthesized programs are self-contained, meaning they can be executed directly for decompression without needing external metadata or code.
• High Throughput: The method achieves practical compression and decompression speeds, making it viable for real-world use.

Results

Brevis was evaluated on 10 public checkpoints from language, audio, and image generation models, totaling 2.13 TB. It reduced the data to 1.41 TB, a 33.93% storage reduction. Compared to four general-purpose compressors (including zstd and gzip), Brevis produced archives up to 30.87% smaller. It also outperformed tensor-specific compressors ZipNN and DFloat11, producing smaller archives. Under a practical concurrency configuration, Brevis achieved 3.60 GB/s compression and 6.61 GB/s decompression while preserving every source byte.

These results demonstrate that Brevis not only achieves superior compression ratios but also maintains high throughput, making it suitable for large-scale checkpoint storage and transfer.

Significance

Brevis introduces a new paradigm for tensor compression that is both flexible and effective. By leveraging program synthesis, it can adapt to the specific structure of each tensor, potentially leading to better compression ratios than fixed pipelines. This could significantly reduce storage costs and bandwidth requirements for AI model deployment and archival. The approach also opens up new research directions in applying program synthesis to other data compression tasks. As model sizes continue to grow, methods like Brevis will become increasingly important for managing the infrastructure costs of AI.