Neura Market
Cyan4973/FiniteStateEntropy logo

Cyan4973/FiniteStateEntropy

Free

New generation entropy codecs : Finite State Entropy and Huff0

FreeFree tier
Type
Open Source
Visit Website GitHub (38 stars)

About Cyan4973/FiniteStateEntropy

FiniteStateEntropy (FSE) and Huff0 are two high-speed entropy coding libraries written in C. Huff0 is a Huffman codec optimized for modern CPUs, leveraging out-of-order (OoO) operations on multiple ALUs to achieve extremely fast compression (up to 600 MB/s) and decompression (up to 1350 MB/s). FSE is a novel entropy encoder based on Asymmetric Numeral Systems (ANS) theory by Jarek Duda, offering compression accuracy close to the Shannon limit (similar to arithmetic coding) at significantly higher speeds (up to 325 MB/s compression, 440 MB/s decompression). The library is released under the BSD-2-Clause license and includes benchmark programs, sample probability distributions, and a clear branch policy (master for stable releases, dev for contributions). It is designed for integration into applications requiring efficient, low-level entropy coding.

Key Features

Huff0 Huffman codec optimized for modern CPUs with out-of-order operations on multiple ALUs
FSE entropy coder based on ANS theory achieving compression accuracy near the Shannon limit
Extremely high compression and decompression speeds (Huff0 up to 600 MB/s compression, 1350 MB/s decompression; FSE up to 325 MB/s compression, 440 MB/s decompression)
Precise compression comparable to arithmetic coding but at much higher speeds
Open source library under BSD-2-Clause license
Written in C (98.5%) for performance and portability
Includes benchmark programs and sample probability files (e.g., proba80, proba14, proba02)
Clear branch policy: master for stable releases, dev for contributions
Supports block-based compression (example: 32 KB blocks)

Pros & Cons

Pros
  • Extremely high compression and decompression speeds, especially Huff0
  • FSE achieves compression accuracy close to the Shannon limit, outperforming Huffman on squeezed distributions
  • Designed for modern CPU architectures with OoO execution capabilities
  • Open source and freely available under a permissive BSD-2-Clause license
  • Actively maintained with a clear contribution branch policy
  • Portable C implementation with minimal dependencies
  • Comprehensive benchmark data provided for comparison with zlibh
Cons
  • Huff0 cannot compress below 1 bit per symbol, making it less efficient on highly entropic data (e.g., near-uniform distributions)
  • FSE compression and decompression speeds are lower than Huff0 (e.g., 325 MB/s vs 600 MB/s compression)
  • Library focuses solely on entropy coding; users must integrate it into a full compression framework
  • Requires C programming knowledge to use; no high-level language bindings or wrappers mentioned
  • Limited documentation beyond the README; no API reference directly provided

Best For

Data compression in software applications where high throughput is criticalEmbedded systems requiring fast entropy coding with low overheadReplacing zlib's Huffman encoder to gain speed without sacrificing compression ratioResearch and education in entropy coding, ANS theory, and Huffman codingStorage and transmission of data where compression speed is more important than ratio

FAQ

What is FiniteStateEntropy?
FiniteStateEntropy (FSE) is an entropy encoder based on ANS theory by Jarek Duda, providing compression accuracy near the Shannon limit at speeds much higher than arithmetic coding.
What are the main components of this library?
The library includes two high-speed entropy coders: Huff0 (a Huffman codec designed for modern CPUs) and FSE (an ANS-based entropy encoder).
How does FSE compare to arithmetic coding?
FSE achieves compression precision similar to arithmetic coding but at significantly higher compression and decompression speeds.
What license does this library use?
It is released under the BSD-2-Clause license, allowing free use, modification, and distribution.
On which platforms can it run?
It is written in C and has been tested on Linux Mint 64-bit with GCC 4.8.4; the code is designed to be portable to other platforms.