Omlx
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
Last verified:
What is Omlx?
oMLX is a native macOS LLM inference server built on Apple's MLX framework, specifically designed for Apple Silicon (M1/M2/M3/M4) chips. It provides local AI inference with smart caching that eliminates long wait times for coding agents like Claude Code, OpenClaw, and Cursor, reducing response time from 90 seconds to under 5 seconds from the second turn onward.
The core innovation is paged SSD KV caching, which persists every cache block to SSD in safetensors format. This two-tier architecture keeps hot blocks in RAM and cold blocks on SSD with LRU policy, allowing previously seen prefixes to be restored across requests and server restarts instead of being recomputed. It also features continuous batching that handles concurrent requests through mlx-lm's BatchGenerator, achieving up to 4.14× generation speedup at 8× concurrency.
oMLX is built for developers and AI coding agents who need reliable local LLM deployment. It includes a native macOS menu bar app (not Electron) for starting, stopping, and monitoring the server, plus a web dashboard for model management, chat, and real-time metrics. The server supports multi-model serving with LLM, VLM, embedding, and reranker models loaded simultaneously, and provides OpenAI + Anthropic drop-in API compatibility.
Omlx pricing
Pricing model: Freemium
Free and open-source under Apache 2.0 license. No paid tiers or subscriptions. Download the DMG file from GitHub releases or install from source via Homebrew. All features including paged SSD KV caching, continuous batching, multi-model serving, and API compatibility are included at no cost.
Omlx pros
- Paged SSD KV caching persists cache blocks to disk
- Continuous batching up to 4.14× speedup at 8× concurrency
- Native macOS menu bar app, not Electron
- TTFT under 5 seconds from 2nd turn on long contexts
- OpenAI and Anthropic drop-in API compatibility
- Works with Claude Code, OpenClaw, and Cursor out of the box
- Multi-model serving: LLM, VLM, embedding, reranker simultaneously
- LRU eviction when memory runs low
- Built-in HuggingFace model downloader in admin dashboard
- Reuses existing LM Studio model directory, no re-download needed
- Supports all major tool calling formats: JSON, Qwen, Gemma, GLM, MiniMax
- MCP tool integration with tool result trimming
- Signed and notarized with in-app auto-update
- Web dashboard generates exact config command for each tool
- Apache 2.0 open-source license
- Reasoning models get automatic <thought> tag handling
- Vision-Language Models supported since v0.2.0
Omlx cons
- Apple Silicon only (M1 or later), not Intel Macs
- macOS 15+ required, not compatible with older macOS
- 16GB RAM minimum, 64GB+ recommended for larger models
- Not available on Windows or Linux
- High memory models like GLM-5-4bit need 392GB+ peak memory
- No VRAM reporting in the interface
- Model format limited to MLX (safetensors)
- Standard port 8000 may conflict with other services
Frequently asked questions about Omlx
How does oMLX's SSD caching differ from Ollama and LM Studio?
Ollama and LM Studio cache KV state only in memory, so when context shifts mid-session the entire cache gets invalidated and recomputed from scratch. oMLX persists every KV cache block to SSD, so previously cached portions are always recoverable. This drops TTFT from 30-90 seconds to under 5 seconds on long contexts.
What are the system requirements for oMLX?
Apple Silicon (M1 or later) with macOS 15+. 16GB RAM is the minimum, but 64GB+ is recommended for comfortable use with larger models. The sweet spot for daily coding work is an M-series Pro/Max with 64GB+.
Does oMLX work with Claude Code, OpenClaw, and Cursor?
Yes. oMLX provides both OpenAI-compatible (/v1/chat/completions) and Anthropic-compatible (/v1/messages) API endpoints. It works as a drop-in backend for all three. The web dashboard has a one-click config generator — select your model, copy the command, paste into terminal.
Do I need to re-download models if I already use LM Studio?
No. oMLX reuses your existing LM Studio model directory — just point it at your models folder. You can also browse and download models directly from the built-in HuggingFace downloader in the admin dashboard.
What models are supported by oMLX?
Any MLX-format model from HuggingFace. This includes Qwen, LLaMA, Mistral, Gemma, DeepSeek, MiniMax, GLM, and more. Reasoning models (DeepSeek, MiniMax, Qwen) get automatic <thought> tag handling. Vision-Language Models are supported since v0.2.0 with the same paged SSD caching.
How do I install oMLX?
Download the DMG from GitHub releases and drag it to your Applications folder, or install from source via Homebrew using 'brew tap jundot/omlx https://github.com/jundot/omlx' followed by 'brew install omlx'. No command line or configuration files required for the DMG install.
What is continuous batching and how much faster is it?
Continuous batching handles concurrent requests through mlx-lm's BatchGenerator instead of queuing behind a single request. At 8× concurrency, oMLX achieves up to 4.14× generation speedup depending on the model. For Qwen3-Coder-Next-8bit, 8× batch reaches 243.3 tok/s vs 58.7 tok/s for single request.
Does oMLX support tool calling?
Yes. It supports all major tool calling formats: JSON, Qwen, Gemma, GLM, and MiniMax. It also includes MCP tool integration and tool result trimming for oversized outputs. Tool calling is configurable per model through the admin dashboard.
What is the two-tier KV cache architecture?
Hot blocks stay in RAM while cold blocks go to SSD with LRU policy. Cache blocks are persisted to disk in safetensors format. Previously seen prefixes are restored across requests and server restarts — never recomputed. This enables infinite SSD KV cache with no eviction.
Is oMLX open-source and what license does it use?
Yes, oMLX is fully open-source under the Apache 2.0 license. The GitHub repository is at github.com/jundot/omlx with over 14,500 stars. The project is a hobby project that welcomes contributions and is free for all users.