Cactus
Low-latency AI engine for mobile devices & wearables
Last verified:
What is Cactus?
Cactus is a hybrid on-device AI inference engine designed for smartphones, laptops, wearables, and edge hardware. It enables developers to deploy language models (LLMs), vision models (VLMs), speech transcription, and embeddings locally on devices with automatic cloud fallback for complex tasks. The toolkit supports cross-platform development with APIs for React Native, Flutter, Kotlin, Swift, C/C++, and Dart, allowing developers to write once and deploy across iOS, Android, macOS, and wearables.
Key features include the Cactus Engine (an energy-efficient inference engine with NPU acceleration and INT4 quantization), Cactus Graph (a zero-copy computation graph for RAM efficiency), and Cactus Kernels (low-level ARM SIMD kernels optimized for Apple, Snapdragon, Google Exynos, and MediaTek processors). The Hybrid Cloud Router automatically measures model confidence in real-time and routes simple queries on-device while handing off complex or noisy data to cloud APIs. Cactus also supports Voice Activity Detection (Silero VAD), tool calling, auto RAG, and streaming transcription.
Cactus is built for mobile application developers, AI engineers, and companies building edge AI solutions for consumer and enterprise mobile platforms. It is particularly valuable for apps requiring low-latency (<120ms on-device), offline functionality, and strong privacy guarantees (HIPAA-friendly, GDPR-compliant, zero data retention). Use cases include mobile voice assistants, desktop notetakers, wearable intelligence, personalized RAG pipelines, and AI for privacy-sensitive industries like healthcare.
Cactus pricing
Pricing model: Freemium
Cactus will always have a free tier. The free tier includes basic on-device inference capabilities. Hybrid inference, custom models, and additional hardware acceleration are paid features. Over 80% of production transcription and LLM inference can be handled on-device, providing 5x cost savings over pure cloud solutions. Cloud API key (CACTUS_CLOUD_API_KEY) is required for hybrid cloud handoff features. Contact Cactus to enable cloud handoff for Language Model and Batch Transcription.
Cactus pros
- Sub-120ms latency on-device for real-time transcription
- Automatically routes audio between on-device and cloud based on quality
- Open source with fully auditable codebase
- Cross-platform support: iOS, Android, macOS, wearables from single SDK
- Zero-copy memory mapping for minimal RAM usage
- NPU acceleration for battery-efficient inference
- Supports any LLM or VLM available on HuggingFace
- INT4 quantization for efficient model execution
- Works offline with no internet required for on-device tasks
- HIPAA-friendly and GDPR-compliant with zero data retention
- 5x cost savings compared to pure cloud inference
- Over 80% of production transcription handled on-device
- React Native, Flutter, Kotlin, Swift, C/C++ APIs available
- Automatic cloud handoff when model confidence drops
- Homebrew installation on macOS for easy setup
- Built by team from Y Combinator, Oxford, Google, AWS, MIT
- 4.2k+ GitHub stars with active community
- Voice Activity Detection with Silero VAD model included
- Supports tool calling and MCP for practical app integration
Cactus cons
- Hybrid inference, custom models, and hardware acceleration are paid features
- Proprietary .cact format replacing GGUF may limit model compatibility
- Cloud handoff for Language Model and Batch Transcription requires contacting support
- Performance varies significantly across device tiers (mid-range phones slower)
- Limited model selection compared to full cloud API offerings
- Smaller on-device models may struggle with very complex tasks
- Requires setting CACTUS_CLOUD_API_KEY environment variable for hybrid features
- Some edge devices may have limited NPU support
Frequently asked questions about Cactus
Is Cactus free?
Cactus will always have a free tier. The free tier includes basic on-device inference capabilities. Hybrid inference, custom models, and additional hardware acceleration are paid features.
What model format does Cactus use?
With the v1 release, Cactus moves from GGUF to a proprietary .cact format, which is optimized specifically for battery-efficient inference and minimal RAM usage via zero-copy memory mapping. A list of supported models is available on their documentation.
Which models are supported?
Cactus supports any LLM or VLM available on HuggingFace directly, including Qwen, Gemma, Llama, DeepSeek, Phi, Mistral, SmolLM, SmolVLM, InternVLM, Jan Nano, and LiquidAI/LFM2 models. You can submit a request for model support or contribute by porting a model yourself.
How does Hybrid AI routing work?
Cactus measures model confidence in real-time during inference. When confidence drops below a threshold or when the query exceeds device capabilities, Cactus automatically hands off to a cloud model. Simple queries like clear audio or standard completions go to on-device NPU/CPU, while complex queries like noisy audio, long context, or ambiguous prompts go to Cactus Cloud.
What platforms does Cactus support?
Cactus is cross-platform and supports iOS, Android, macOS, and wearables from a single SDK. It provides APIs for React Native, Flutter, Kotlin, Swift, C/C++, and Dart, allowing developers to write once and deploy anywhere.
Does Cactus work offline?
Yes, Cactus works offline for on-device inference. Audio data never leaves the user's phone when running on-device only, making it HIPAA-friendly and GDPR-compliant with zero data retention. Cloud fallback is optional and only used for complex tasks when internet is available.
How do I install Cactus?
On macOS, you can install Cactus CLI with Homebrew: brew install cactus-compute/cactus/cactus. For other platforms, you can clone the GitHub repository (git clone [email protected]/cactus-compute/cactus), run source ./setup, then use cactus build and cactus run to execute models.
What is the latency performance?
Cactus offers sub-120ms latency on-device for real-time transcription. Benchmarks show iPhone 17 Pro achieves 300/33 tps with 108MB RAM for LFM2.5-1.2B, Galaxy S25 Ultra achieves 226/36 tps with 1.2GB RAM, and Mac M4 Pro achieves 582/77 tps with 76MB RAM for INT8 quantized models.
How do I enable cloud handoff?
Set the CACTUS_CLOUD_API_KEY environment variable and Cactus handles handoff automatically. For Live Transcription, handoff is fully automatic out of the box. For Language Model and Batch Transcription, you need to contact Cactus to enable cloud handoff. The CompletionResult includes needsCloudHandoff and confidence fields to check whether to route to cloud API.
Is Cactus open source?
Yes, Cactus is fully open source and community-driven. The entire codebase is auditable and available on GitHub at github.com/cactus-compute/cactus. You can inspect every line that runs on users' devices, view the codebase structure on their Visualize Repository, and contribute to the project. The project has 4.2k+ GitHub stars.