NouamaneTazi/bloomz.cpp
FreeC++ implementation for BLOOM
FreeFree tier
Inputs: textOutputs: text
About NouamaneTazi/bloomz.cpp
A pure C/C++ implementation for inference of HuggingFace's BLOOM-like language models, built on top of the llama.cpp project. It supports any model loadable via BloomForCausalLM.from_pretrained() and provides tools to convert HuggingFace weights to the GGML format, optionally quantize them to 4-bit for reduced memory usage, and run fast CPU inference using a simple command-line interface.
Key Features
Pure C/C++ implementation for BLOOM model inference
Supports all BLOOM models loadable via BloomForCausalLM.from_pretrained()
Weight conversion from HuggingFace format to GGML format
Optional 4-bit quantization to reduce model size and memory usage
CPU-based inference with adjustable number of threads and token generation parameters
Built on top of the well-optimized llama.cpp project
Pros & Cons
Pros
- Efficient pure C/C++ implementation with no Python runtime needed for inference
- Leverages the battle-tested llama.cpp codebase
- Supports 4-bit quantization for reduced memory footprint (~4.8 GB for 7B model)
- Easy conversion from HuggingFace pretrained models
- Open-source and free to use
Cons
- Limited to BLOOM-like architectures (not general transformer models)
- Requires manual conversion and quantization steps before inference
- No GPU acceleration support; runs only on CPU
- Memory usage remains high even with quantization (5.3 GB for 7B model)
- Documentation is minimal and focused on basic usage
Best For
Running BLOOM and BLOOMZ models locally on CPU without GPU dependenciesDeploying quantized BLOOM models for resource-constrained environmentsResearch and experimentation with BLOOM-like architectures in C++Building lightweight inference pipelines for multilingual text generation
FAQ
What models does bloomz.cpp support?
It supports any HuggingFace model that can be loaded with BloomForCausalLM.from_pretrained(), including BLOOM and BLOOMZ variants.
How do I convert a model to the required format?
You can use the provided Python script `convert-hf-to-ggml.py` to download and convert HuggingFace weights to GGML FP16 format. Optionally, you can quantize further using the `quantize` tool.
Is there a quick way to convert weights without running scripts?
Yes, there is a HuggingFace Space tool that converts and quantizes weights for you and uploads them to a repository of your choice.
Can I run the tool on Windows?
The project is built with Makefile and targets Unix-like systems (Linux, macOS). Windows support would require manual compilation or WSL.