Local Llm
Show HN: Best setup local LLM found for a 5090 (llama.cpp fork + turboquant)
Last verified:
What is Local Llm?
This tool is a technical guide and implementation report demonstrating how to run the Qwen 3.6 35B MoE (Mixture of Experts) large language model at an extended 450,000 token context window on a single 32GB VRAM GPU (NVIDIA RTX 5090) using llama.cpp. The setup runs natively on Windows and provides a complete configuration guide including model quantization choices, memory calibration, KV cache compression, and RoPE scaling techniques.
Key features include support for multimodal image inputs (PNG and JPEG only), Q6_K quantization preserving 99% of native BF16 perplexity, 3-bit TurboQuant KV cache compression reducing memory footprint by 80%, YaRN scaling to extend context beyond the native 262k tokens, Flash Attention activation, and a Go-based Windows Service manager for on-demand VRAM lifecycle control. The tool exposes a local HTTP API on port 9000 compatible with OpenAI-style interfaces.
This tool is designed for developers, researchers, and AI engineers who need to run powerful local LLMs with extreme context windows for tasks like analyzing large codebases, processing extensive logs, or generating complex scripts in a single shot. It is particularly valuable for users who require data privacy (no cloud APIs), want to avoid API costs, and have access to high-end consumer GPU hardware with 32GB VRAM.
Local Llm pricing
Pricing model: Freemium
Free - This is an open technical guide and self-hosted implementation. No paid plans exist. Users only need their own hardware (NVIDIA RTX 5090 with 32GB VRAM), a Hugging Face token to download the GGUF model from jimbothigpen/Qwen3.6-35B-A3B-GGUF, and to compile/use llama.cpp with the TurboQuant fork. All software components (llama.cpp, Go Gin Gonic manager, Python scripts) are open source.
Local Llm pros
- Runs 35B MoE model with 450k context on single 32GB GPU
- Q6_K quantization preserves over 99% of native BF16 perplexity
- Only ~3B active parameters per token maintains fast execution speeds
- 3-bit TurboQuant KV cache reduces memory footprint by 80%
- YaRN scaling extends context beyond native 262k token limit
- Supports multimodal image inputs (PNG and JPEG)
- Flash Attention reduces memory overhead and accelerates computation
- Fully offline - no data sent to external cloud APIs
- OpenAI-compatible API endpoint at port 9000
- Go-based service manager enables on-demand VRAM release
- Successfully generates complex code in single shot (e.g., Three.js apps)
- Windows-native scripts with Batch and PowerShell support
- Can digest large codebases or logs for analysis
- Compatible with OpenCode client for IDE integration
- Demonstrates cutting-edge local inference optimization progress
Local Llm cons
- Operates at absolute physical limits of 32GB VRAM with no margin
- WebP image format not supported - returns 400 Bad Request error
- YaRN scaling beyond 262k causes significant retrieval accuracy degradation
- Perplexity increases progressively near 450k context limit
- Logical reasoning capacity compromised at extreme context ranges
- 3-bit KV cache quantization introduces minor perplexity degradation
- Requires NVIDIA RTX 5090 or equivalent 32GB VRAM GPU
- Community fork (TheTom/llama-cpp-turboquant) not yet in upstream llama.cpp
Frequently asked questions about Local Llm
What GPU is required to run this setup?
A single NVIDIA RTX 5090 with 32GB VRAM is required. The model weights (Q6_K quantization) take 28.5GB, static overhead uses ~0.8GB, leaving only ~2.7GB for the KV cache. The guide is tested on Windows but Linux deployments work with minor path adjustments.
Why was Q6_K quantization chosen over Q4_K_M or Q5_K_S?
Q6_K was selected to preserve logical accuracy and syntax structure for complex code generation, maintaining over 99% of the native BF16 perplexity. Lower quantizations would reduce VRAM footprint further but sacrifice the precision needed for code generation tasks.
How does the tool achieve 450k context when native limit is 262k?
Two techniques are applied: (1) TurboQuant 3-bit KV cache compression (--cache-type-k turbo3, --cache-type-v turbo3) reduces memory by 80%, and (2) YaRN RoPE scaling (--rope-scaling yarn --yarn-orig-ctx 262144 --rope-scale 1.72) extends the rotary position embedding frequencies. However, this compromises retrieval accuracy beyond 262k tokens.
What image formats are supported for multimodal inputs?
Only PNG and JPEG formats are supported. The image decoder uses the stb_image library which does not support WebP. Attempting to upload WebP files returns a 400 Bad Request error with message 'Failed to load image or audio file'.
What is the purpose of the Go manager service?
Since the model occupies nearly 100% of GPU memory, the Go (Gin Gonic) manager runs as a Windows Service or Linux systemd daemon to provide on-demand lifecycle control. It starts/stops the server to release 100% of VRAM instantly when not needed, and polls port 9000 every 3 seconds to confirm server status.
What are the performance trade-offs of YaRN scaling beyond native context?
YaRN scaling introduces significant degradation in retrieval accuracy (needle-in-a-haystack tasks) and general reasoning capacity when operating far beyond the native 262k token boundary. Perplexity increases progressively as context utilization approaches 450k tokens, making the model less reliable for raw logical deduction. The 450k limit is best for exploratory search and broad summarization, not critical workloads.
What llama.cpp flags are essential for this configuration?
Key flags include: --no-mmap (forces contiguous VRAM load, prevents OS paging), --flash-attn on (reduces memory overhead), -ngl 99 (offsloads all 40 layers plus output to GPU), -c 450000 (context size), --rope-scaling yarn --yarn-orig-ctx 262144 --rope-scale 1.72, --cache-type-k turbo3 --cache-type-v turbo3, -b 512, -ub 512, --port 9000, --host 0.0.0.0.
Where do I download the model and vision projector?
Both files are downloaded from Hugging Face Hub using the repo ID jimbothigpen/Qwen3.6-35B-A3B-GGUF. The files are Qwen3.6-35B-A3B-Q6_K.gguf (28.5GB) for the model and Qwen3.6-35B-A3B-mmproj-F16.gguf (899MB) for the vision projector. A Hugging Face token is required for access.
Can this setup run on Linux instead of Windows?
Yes, the core execution parameters and performance calibrations are fully applicable to Linux deployments with minor path adjustments. The Windows-native Batch and PowerShell scripts would need to be converted to Bash, and the Go manager would run as a systemd daemon instead of a Windows Service. Backslashes in the server invocation command should be replaced with carets for Windows.