Test LLMs Side-by-Side
A local LLM client for parallel testing and evaluation
Last verified:
What is Test LLMs Side-by-Side?
Test LLMs Side-by-Side is a local-first desktop client designed to test, grade, and benchmark prompts across major LLMs side-by-side. Instead of guessing how a model will perform, users can prove it against their datasets with parallel testing capabilities. The tool provides a dedicated prompt workbench for the messy reality of prompt engineering, offering foundational tools to test, iterate, and benchmark without magic buttons.
Key features include parallel model testing where users send a single prompt template to GPT-4, Claude 3, and Gemini simultaneously, instantly comparing raw JSON outputs, latency metrics, and exact token consumption. The local-first privacy approach stores API keys and prompt history in a local SQLite database with nothing touching their servers. Automatic prompt checkpointing saves every iteration, allowing users to fork prompts to test new variables and track exact changes. Benchmark and evaluate capabilities let users inject test data into prompt templates to establish baselines and benchmark new LLMs against historical data before production trust.
AptSelect is designed for prompt engineers, AI developers, and teams shipping AI features who need certainty. It supports Windows, Mac, and Linux platforms. Users bring their own keys which are encrypted via OS keyring and sent strictly to providers. The tool includes request-level debugging to inspect raw API responses, latency stats, and exact token usage for every request, plus version control for prompts with clean history of iterations and easy reverting to past versions.
Test LLMs Side-by-Side pricing
Pricing model: Freemium
Public beta is currently completely free. The planned pricing is a one-time $29 perpetual license. No subscription model - users pay once and keep the license permanently. During beta, all features are available without payment.
Test LLMs Side-by-Side pros
- Parallel testing across GPT-4, Claude 3, and Gemini simultaneously
- Local-first privacy with API keys stored in local SQLite database
- Nothing touches AptSelect servers - keys sent only to providers
- Automatic prompt checkpointing saves every iteration
- Fork prompts to test new variables easily
- Track exact changes that improved output
- Benchmark new LLMs against historical baseline data
- Pass/fail grades to see which model handles edge cases
- Request-level debugging with raw API response inspection
- View exact token usage for every single request
- View latency metrics side-by-side
- Bring your own keys encrypted via OS keyring
- Cross-platform support for Windows, Mac, and Linux
- Version control for prompts with clean iteration history
- Easy revert to past prompt configurations
- Adjust temperature, top_p, and frequency penalties
- Toggle models on/off to keep workspace clean
- Connect OpenAI, Anthropic, Mistral, Gemini, and XAI providers
- No heavy framework overhead - vanilla JS + Web Components
- Context isolated security architecture
- Standard SQLite file for easy backup and version control
- Compare raw JSON outputs side-by-side
- No manual copy-pasting between browser tabs
- Batch-test datasets to prove model reliability
Test LLMs Side-by-Side cons
- Public beta currently free but perpetual license not yet available
- One-time $29 perpetual license still in planning
- Electron runtime may have higher memory usage than native apps
- Vanilla JS means no modern framework tooling benefits
- Local SQLite requires manual backup management
- No cloud sync for prompt history across devices
- Limited to desktop only - no web or mobile version
- Provider setup requires manual API key configuration
Frequently asked questions about Test LLMs Side-by-Side
What is AptSelect?
AptSelect is a local-first desktop client designed to test, grade, and benchmark prompts across major LLMs. It allows users to stop guessing how a model will perform and prove it against their datasets with parallel testing capabilities.
What platforms does AptSelect support?
AptSelect is available for Windows, Mac, and Linux. It was built using Electron for cross-platform support while keeping the stack simple with native web components.
How does local-first privacy work?
Your API keys and prompt history are stored in a local SQLite database on your disk. Nothing touches AptSelect servers. Keys are encrypted via your OS keyring and sent strictly to the LLM providers.
Can I test multiple LLMs simultaneously?
Yes, you can send a single prompt template to GPT-4, Claude 3, and Gemini simultaneously. The tool instantly compares raw JSON outputs, latency metrics, and exact token consumption side-by-side without managing multiple browser tabs.
How do I benchmark new LLMs?
Inject test data into your prompt templates to establish a baseline. When a new LLM drops, run your prompt against a full test dataset across multiple models at once and benchmark it against your historical data before trusting it in production.
What providers can I connect?
You can connect OpenAI, Anthropic, Mistral, Gemini, and XAI in seconds. Each provider requires you to bring your own API keys which are encrypted and stored locally.
How does prompt versioning work?
Every iteration is automatically saved to your local SQLite database. You can fork a prompt to test a new variable, track the exact changes that improved the output, and easily revert to past configurations or switch back to past versions.
What inference settings can I adjust?
You can adjust temperature, top_p, and frequency penalties to observe how different constraints impact your prompt results. All settings are managed from the central interface.
How do I grade model performance?
Run your prompt against a full test dataset across multiple models at once, review the batch outputs side-by-side, and assign pass/fail grades to see exactly which model handles your edge cases.
What is the pricing model?
The public beta is currently completely free. The planned pricing is a one-time $29 perpetual license with no subscription - you pay once and keep the license permanently.