cactus-compute/cactus

Free

Low-latency AI engine for mobile devices & wearables

FreeFree tier

Inputs: text, imageOutputs: text

Type

Open Source

Company

Cactus Compute

Visit Website GitHub (38 stars)

About cactus-compute/cactus

Cactus is an open-source hybrid edge-cloud AI engine designed for mobile devices, wearables, and other resource-constrained hardware. It provides OpenAI-compatible APIs for text, speech, and vision tasks, enabling low-latency inference on-device. The engine is composed of several core components: Cactus Engine for runtime execution, Cactus Graph for zero-copy computation graphs, Cactus Kernels for optimized CPU/GPU kernels (supporting Apple, Samsung, Pixel devices), Cactus Quants for custom rotation-based quantization, and a Cactus Transpiler that converts PyTorch models into Cactus format. It supports features like auto-RAG, tool calling, streaming, and audio transcription. Cactus can run models such as Gemma-4-E2B and demonstrates fast inference speeds (e.g., 45ms time-to-first-token) with low memory usage.

Key Features

Hybrid edge-cloud AI engine for mobile devices and wearables

OpenAI-compatible APIs for text, speech, and vision

Zero-copy computation graph for efficient execution

Custom rotation-based quantization technique (Cactus Quants)

Optimized CPU/GPU kernels for Apple, Samsung, Pixel devices

Transpiler to convert PyTorch models to Cactus format

Auto-RAG support with text or directory of text files

Tool calling and streaming capabilities

Fast inference with low time-to-first-token and memory usage

Supports transcription and vision tasks

Pros & Cons

Pros

Low-latency inference suitable for real-time applications
Open source and free to use
Optimized for a variety of popular mobile hardware platforms
Comprehensive component architecture for flexibility
Auto-RAG and tool calling enable advanced conversational AI

Cons

Hardware support limited to Apple, Samsung, and Pixel devices
Requires specific setup and compilation for each platform
Limited ecosystem compared to larger frameworks like TensorFlow Lite
Documentation may be sparse for advanced usage

Best For

Running large language models on mobile devicesOn-device AI inference for wearablesEdge computing applications in IoTReal-time speech transcription and synthesisVision tasks on smartphone camerasBuilding private AI assistants on personal devices

FAQ

What is Cactus?

Cactus is an open-source hybrid edge-cloud AI engine designed for mobile devices and wearables. It provides OpenAI-compatible APIs and supports text, speech, and vision tasks with low-latency inference.

What hardware does Cactus support?

Cactus provides optimized CPU/GPU kernels for Apple devices, Samsung devices, and Pixel devices. Additional platforms may be added over time.

Can I run my own PyTorch model on Cactus?

Yes, Cactus includes a transpiler (Cactus Transpiler) that converts custom PyTorch models into Cactus format so they can be run on the engine.

Is Cactus free to use?

Yes, Cactus is open source and free to use.