Node Llama Cpp
Run AI models locally with Node.js bindings for llama.cpp and generation-level JSON schema enforcement.
Last verified:
What is Node Llama Cpp?
Node Llama Cpp is a Node.js library for running large language models locally via llama.cpp bindings, with no Python install required. It provides a simple API for downloading models and running them optimized for your hardware (Metal, CUDA, Vulkan), with zero-config defaults in Node.js, Bun, and Electron. Key features include JSON schema enforcement on model output at the generation level, function calling, text completion and embedding generation, CLI tools for chatting and inspecting hardware, and pre-built binaries for macOS, Linux, and Windows — supporting GGUF model files.
Node Llama Cpp pricing
Pricing model: Freemium
Free and open source. Installed via npm install node-llama-cpp with no paid tiers, subscriptions, or API costs. Pre-built binaries are provided at no cost. The library runs entirely locally on your machine with no external service fees.
Node Llama Cpp pros
- Runs LLMs locally without any Python installation required
- Zero-config default setup works out of the box
- Works in Node.js, Bun, and Electron environments
- Pre-built binaries for macOS, Linux, and Windows
- Automatic hardware detection and optimization (Metal, CUDA, Vulkan)
- Simple npm install installation like any other npm package
- JSON schema enforcement on model output at generation level
- Function calling support even for models without official support
- Embedding generation for similarity search tasks
- CLI with chat, pull, inspect, and completion commands
- No API keys or environment variables needed
- Stateful inference without re-evaluating entire conversation history
- Supports GGUF model files from HuggingFace
- Fallback to building from source with cmake if binaries unavailable
- Speculative decoding with token prediction draft model support
Node Llama Cpp cons
- Only supports ES modules (cannot use require)
- Small models needed for CPU-only machines limit capabilities
- Models larger than available VRAM offload to RAM reducing speed
- JSON schema only supports a small subset of JSON schema spec
- Building from source requires C++ build tools and cmake
- Electron apps cannot build from source when packaged in Asar archive
- Non-instruct models may not work well for chat interactions
- GPU drivers and Vulkan SDK may need manual installation on Linux
- Model quality depends on choosing correct quantization level
Frequently asked questions about Node Llama Cpp
What is node-llama-cpp?
node-llama-cpp is a Node.js library that runs large language models locally on your machine using llama.cpp bindings. It allows you to use LLMs in Node.js and TypeScript without any Python installation, with a simple API for downloading models, running them optimized for your hardware, and integrating them into your projects.
Do I need Python to use node-llama-cpp?
No, node-llama-cpp is specifically designed to run LLMs without any Python at all. It ships with pre-built binaries for macOS, Linux, and Windows, and falls back to building from source with cmake if needed, requiring no Python or node-gyp.
How do I install node-llama-cpp?
Install it like any other npm package by running 'npm install node-llama-cpp' in your project directory. To scaffold a new project with everything set up, run 'npm create node-llama-cpp@latest'. You can also try it without installing using 'npx -y node-llama-cpp chat'.
What model format does node-llama-cpp support?
node-llama-cpp works with GGUF (Georgi Gerganov's Unified Format) model files. You can get GGUF models from HuggingFace from reputable community members like mradermacher or bartowski, or from model providers themselves.
Does node-llama-cpp support GPU acceleration?
Yes, it automatically detects and uses available GPU compute layers including Metal (Apple Silicon Macs), CUDA (NVIDIA GPUs), and Vulkan. It balances default settings for best performance without manual configuration. Use 'npx --no node-llama-cpp inspect gpu' to check your hardware.
How do I enforce JSON output from a model?
Use llama.createGrammarForJsonSchema() to create a grammar from your JSON schema. Pass this grammar to session.prompt() with the grammar option. The model will be forced to generate output conforming to your schema at the text generation level, and you can parse the result with grammar.parse().
Can node-llama-cpp work in Electron apps?
Yes, you can run models in Electron apps without additional setup on the user's machine. However, building from source defaults to 'never' in Electron since users may not have build tools. Build node-llama-cpp with your desired cmake options before building your Electron app, and ship it as an unpacked module.
What VRAM do I need for different model sizes?
Approximate VRAM requirements: 1B models need 1GB, 3B models need 3.5GB, 8B models need 6GB, 70B models need 55GB, and 405B models need 300GB. Ideally fit the entire model in VRAM for maximum GPU performance. Use 'npx --no node-llama-cpp inspect estimate <model-url>' for accurate estimates.
How do I enable function calling with models?
Use defineChatSessionFunction to define functions with description, params schema, and async handler. Pass the functions object to session.prompt() with the functions option. Some models like Functionary and Llama 3 Instruct have official support, while others use a generic fallback mechanism.
What quantization quality should I choose for models?
Q4_K_M offers the best balance between compression and quality, with Q5_K_M as a close second. Q8_0 is highest quality with compression but slower and uses more memory. f16 is uncompressed highest quality but not recommended for inference. Test quality with the chat command to see if it's sufficient for your needs.