Bodhiapp
Run Open Source/Open Weight LLMs locally with OpenAI compatible APIs
Last verified:
What is Bodhiapp?
Bodhiapp Bodhi App is a cross-platform unified AI gateway that lets you run open-source large language models locally on your hardware while also integrating cloud API providers like OpenAI, Anthropic, and Groq in a single interface. Built on llama.cpp and integrated with the HuggingFace ecosystem, it provides a built-in Chat UI with markdown support, real-time streaming, model downloads, and OpenAI-compatible APIs for seamless integration with existing tools.
Bodhiapp pricing
Pricing model: Freemium
Bodhi App is completely free and open-source with no paid tiers or subscription fees. The app runs locally on your hardware without requiring API credits or costly subscriptions to providers like OpenAI. All features including the built-in Chat UI, model downloads, API access, enterprise authentication, role-based access control, and developer SDKs are included free. Users only pay for their own hardware and any cloud API providers they choose to integrate (OpenAI, Anthropic, Groq, Together AI) separately if they opt to use cloud models alongside local inference.
Bodhiapp pros
- Run LLMs completely locally with full data privacy
- Open-source and free to use without subscriptions
- Hybrid AI architecture combining local and cloud APIs
- OpenAI-compatible APIs for drop-in replacement
- Built-in Chat UI with markdown and code block support
- One-click model downloads from HuggingFace
- Real-time streaming with Server-Sent Events
- MCP tool integration for autonomous task completion
- Enterprise-grade OAuth2+JWT authentication with PKCE
- 4-level role-based access control (User, PowerUser, Manager, Admin)
- Interactive Swagger UI API documentation with live testing
- TypeScript SDK and React hooks for developer integration
- Model aliases to save and switch configurations instantly
- 8-12x GPU acceleration with CUDA, ROCm, Vulkan support
- Multi-platform desktop apps via Tauri (Windows, macOS, Linux)
- 7 Docker variants for different hardware architectures
- Background async model downloads with auto-resumption
- Real-time performance metrics showing tokens per second
- Thinking models support showing LLM chain-of-thought
- User management dashboard with access request workflows
Bodhiapp cons
- Currently only macOS M-series fully available, other platforms coming soon
- Limited to GGUF format models from HuggingFace
- Requires local hardware with sufficient RAM for model sizes
- OAuth2 authentication mandatory (no simple local-only mode for production)
- First user becomes admin automatically (security concern for teams)
- GPU acceleration depends on specific hardware support
- Ollama compatibility deprecated and limited to migration only
- No mobile app yet despite roadmap mentions
- Learning curve for advanced features like MCP setup
- Self-hosted setup requires technical knowledge for Docker deployment
Frequently asked questions about Bodhiapp
What is Bodhi App?
Bodhi App is a local LLM inference application built on top of the Huggingface and llama.cpp ecosystems. It features a built-in Chat UI, model downloads, API access, and dynamic configuration management. It serves as a unified AI gateway combining local LLM inference, cloud API proxying, and MCP tool integration with enterprise access control.
What platforms does Bodhi App support?
Currently, Bodhi App is available on macOS for M-series devices as a direct .dmg download. Desktop apps for Windows, macOS Intel, and Linux are available via Tauri. Additional platforms are coming soon. Docker variants are available for CPU (AMD64/ARM64), CUDA, ROCm, Vulkan, MUSA, Intel, and CANN.
How do I download a model?
Go to the Download Models section, provide the Huggingface repository name and filename, and submit your download request. Downloads are processed asynchronously, and you can monitor their status on the Downloads page. GGUF model files are downloaded into your local HuggingFace home folder and can be reused for inference.
How do I create an API token?
Access the Token Management section and click 'Generate Token.' The token is displayed only once—copy it immediately. API tokens use scope-based permissions with SHA-256 hashing and database-backed security for secure, programmatic access.
What is the difference between authenticated and non-authenticated mode?
Authenticated Mode provides enhanced security and role-based access control with OAuth2 PKCE. Non-Authenticated Mode offers quick, open access for local testing and exploration but has limited features and is not recommended for production or team use.
How can I access the API documentation?
Bodhi App provides interactive API documentation via the Swagger UI. You can access it from the API Documentation menu within the app or directly at http://<your-bodhi-instance>/swagger-ui. The documentation is auto-generated with Utoipa and continuously updated as new features are added.
What is Model Alias?
A model alias defines the default inference and server parameters for a model. You can create and save inference configurations by specifying preferred request parameters and llama-server settings such as temperature and context size. Switch between configurations seamlessly without any restarts. Alias configurations are stored separately in $BODHI_HOME/aliases.
What is the difference between session-based and API token authentication?
Session-based authentication uses browser login and cookies for interactive use. API token authentication uses tokens generated within the app for secure, programmatic access from external applications and scripts.
How do I update Bodhi App?
For macOS users: Download the latest .dmg file from the releases page, install the new version which automatically replaces the existing installation, and your settings and configurations will be preserved. Bodhi App is regularly updated with new features and improvements.
What should I do if I encounter a model loading error?
Review the logs at $BODHI_HOME/logs for specific error messages, try to resolve based on the logs, or restart the application. For additional guidance, consult the Troubleshooting page or reach out via Discord or GitHub Issues.