armon/bloomd
FreeC network daemon for bloom filters
FreeFree tier
About armon/bloomd
Bloomd is a high-performance C server that exposes bloom filters and operations over them to networked clients using a simple, human-readable ASCII protocol similar to memcached. It implements scalable bloom filters, supports asynchronous persistence to disk, and automatically faults cold filters out of memory to conserve resources. Built with a scalable non-blocking core, it handles many concurrent connections and is designed to be fast and easy to administer.
Key Features
Scalable non-blocking core for many concurrent clients
Implements scalable bloom filters with dynamic filter sizes
Asynchronous flushes to disk for persistence
Supports non-disk-backed bloom filters for high I/O
Automatically faults cold filters out of memory to save resources
Simple ASCII protocol similar to memcached
Human-readable configuration via INI file
Client libraries available for Python, Ruby, Erlang, Go, Perl, Node.js, PHP, Java, Lua/OpenResty
Pros & Cons
Pros
- High performance with scalable non-blocking I/O
- Simple ASCII protocol easy to integrate
- Supports both disk-backed and in-memory filters
- Automatic memory management by faulting cold filters
- Active ecosystem with multiple language clients
- Dead simple to start and administer
Cons
- Inherent false positive rate in bloom filter operations
- Limited to bloom filter operations; not a general-purpose database
- Requires tuning of false positive rate and memory trade-offs
- Installation requires building from source and dependencies (e.g., SCons, libcheck for tests)
- No built-in authentication or encryption in protocol
Best For
Large-scale approximate set membership checkingDeduplication in high-throughput data pipelinesCaching layer for bloom filter operationsNetworked bloom filter service for distributed systemsEfficient representation of large sets with tunable false positive rate
FAQ
What is Bloomd?
Bloomd is a high-performance C network daemon that exposes bloom filters and operations over them to networked clients. It uses a simple ASCII protocol similar to memcached.
How do I install Bloomd?
Clone the repository, install SCons if needed, and run `scons` to build. Optionally install libcheck dependencies to build test code. Pre-built RPMs are available for CentOS/RHEL via Vortex RPM.
What is a bloom filter?
A bloom filter is a probabilistic data structure used for set membership testing. It may produce false positives (claiming an element is in the set when it was never added) but never false negatives. The false positive rate can be tuned, but reducing it increases memory usage.
How does Bloomd handle persistence?
Bloomd supports asynchronous flushes to disk for persistence. It can also run without disk backing for high I/O scenarios. Cold filters are automatically faulted out of memory to save resources.
What clients are available?
Clients are available for Python, Ruby, Erlang, Go, Perl, Node.js, PHP, Java, and Lua/OpenResty. Links are provided in the README.