marella/ctransformers
FreePython bindings for the Transformer models implemented in C/C++ using GGML library.
FreeFree tier
Inputs: textOutputs: text
About marella/ctransformers
Python bindings for Transformer models implemented in C/C++ using the GGML library. Provides a unified interface for loading and running various open-source LLMs (GPT-2, GPT-J, GPT-NeoX, Falcon, LLaMA, MPT, StarCoder, Dolly V2, Replit) on CPU or GPU. Supports streaming output, loading models directly from Hugging Face Hub, integration with 🤗 Transformers and LangChain, and GPU acceleration via CUDA, ROCm, and Metal.
Key Features
Unified interface for all supported models via AutoModelForCausalLM
Supports GPT-2, GPT-J, GPT-NeoX, Falcon, LLaMA, MPT, StarCoder, Dolly V2, Replit
Streaming text generation with stream=True
Load models directly from Hugging Face Hub
Integration with 🤗 Transformers pipeline and tokenizers
Integration with LangChain
GPU acceleration: CUDA, ROCm, Metal
Experimental GPTQ support for LLaMA via ExLlama
Pros & Cons
Pros
- Lightweight Python bindings with C/C++ backend for performance
- Broad model support across multiple architectures
- Easy installation with pip and optional GPU backends
- Streaming output for real-time text generation
- Works with popular libraries like Transformers and LangChain
Cons
- Experimental features (GPTQ, HuggingFace integration) may be unstable
- Limited to GGML/GGUF quantized model format
- Does not support model training or fine-tuning
- Some GPU backends require manual compilation flags
Best For
Local inference of large language models on CPU or GPUPrototyping and running open-source transformer models in PythonBuilding text generation applications with streaming outputIntegrating LLMs into LangChain workflowsRunning quantized GGML/GGUF models from Hugging Face Hub
FAQ
How do I install ctransformers?
Install via pip: pip install ctransformers. Optional GPU backends: pip install ctransformers[cuda] for CUDA, or set environment variable CT_HIPBLAS=1 or CT_METAL=1 before pip install for ROCm or Metal.
How can I use GPU acceleration?
Set the gpu_layers parameter when loading the model, e.g., AutoModelForCausalLM.from_pretrained('TheBloke/Llama-2-7B-GGML', gpu_layers=50). Install the appropriate backend (CUDA, ROCm, Metal) first.
How do I stream the generated text?
Pass stream=True to the model call: for text in llm('AI is going to', stream=True): print(text, end='', flush=True).
Can I load models from Hugging Face Hub?
Yes, use AutoModelForCausalLM.from_pretrained('username/model-name') directly. If the repo has multiple model files, specify with model_file parameter.