Sql Benchmark
An Interactive Text to SQL Agent Benchmark
Last verified:
What is Sql Benchmark?
The Sql Benchmark site hosts an interactive, browser‑based benchmark for evaluating how well large language models generate SQL in an agentic workflow. It runs 25 English‑to‑SQL questions against a DuckDB‑backed dataset derived from the Microsoft AdventureWorks sample database, and models generate SQL that is executed, validated, and scored against known‑good results. The benchmark is explicitly agentic: models can repeatedly run SQL, inspect results or errors, and correct their queries within configured attempt and timeout limits, making it a realistic test of real‑world SQL‑agent behavior.
Users can run the same benchmark directly in the browser by pointing the tool at any OpenAI‑compatible API endpoint along with a model ID and optional API key; the underlying TypeScript runner executes exactly as it would at the command line. The site also provides detailed breakdowns of model performance across difficulty levels (Trivial, Easy, Medium, Hard), highlighting top scorers, token‑efficiency trade‑offs, and quantization effects. This makes it especially useful for developers and researchers who want to compare and select LLMs for local or hosted SQL‑agent use cases.
The tool is aimed at anyone building or experimenting with text‑to‑SQL agents, including self‑hosted data‑analyst tools, in‑browser analytics assistants, or model‑selection pipelines. It is targeted at both practitioners who want to quickly see which models handle agentic SQL generation well and at researchers who care about how factors like model size, quantization, and tool‑calling reliability affect real‑world SQL quality. By exposing the full benchmark data, model scores, and runtime characteristics, it helps users understand which models are best suited for speed, cost, accuracy, or self‑hosting requirements.
Sql Benchmark pricing
Pricing model: Freemium
The Sql Benchmark site itself is free to use; there is no paid tier for the benchmark UI or runner. However, using the benchmark with external LLM endpoints (such as commercial APIs or hosted models) incurs whatever pricing those providers charge per request or per token. The site discusses model costs in cents per run for various commercial models, but those costs are set by the respective vendors, not by the benchmark tool.
Sql Benchmark pros
- Fast 25‑question benchmark under 5 minutes in most cases
- Explicitly agentic with debug loop on SQL execution results
- Browser‑based, self‑hosted‑style execution via DuckDB‑WASM
- Evaluates functional correctness of results, not just SQL syntax
- Uses a realistic dataset derived from Microsoft AdventureWorks
- Supports OpenAI‑compatible endpoints including llama.cpp and OpenRouter
- Interactive page where you can run sample SQL queries in the browser
- Clear breakdown of model scores (Pass/Fail/Error) per question
- Categorized question difficulty (Trivial/Easy/Medium/Hard)
- Highlights token‑efficiency and speed differences between models
- Provides detailed trade‑off guidance for model selection
- Particularly strong for comparing small and self‑hosted models
- Shows impact of quantization and CPU offload on SQL performance
- Includes grammar‑mode experiments for small local models
- Helps identify which models consistently fail at tool‑calling for SQL agents
Sql Benchmark cons
- Benchmarks only 25 questions, so may not fully represent all SQL workloads
- Dataset is based on one sample database schema, limiting domain diversity
- Non‑local or API‑based models may incur separate costs beyond the tool
- Tool itself does not host models; users must supply their own endpoints
- Limited configurability of the agent loop from the UI side
- Some advanced features like grammar mode are experimental and not fully robust
- No built‑in support for custom schemas—schema is fixed to AdventureWorks‑derived tables
- Heavy quantizations can degrade performance on certain models
Frequently asked questions about Sql Benchmark
What is the Sql Benchmark site actually measuring?
The site measures how well an LLM can perform agentic natural‑language‑to‑SQL generation: given an English question and a schema, the model writes SQL, runs it in DuckDB, inspects results or errors, and corrects its SQL within configured limits. The final score is based on whether the result set matches the expected output across 25 predefined questions, not on the raw SQL text itself.
How does the agentic loop work in this benchmark?
For each question, the LLM receives the prompt and schema, then generates SQL that is run in DuckDB. Any errors or partial results are fed back to the model, which can then call a SQL‑run tool again or signal it is done. The loop continues until the model decides it is finished or until attempt/timeout limits are reached, after which the final SQL is validated against the known‑good result set.
What dataset and schema does the benchmark use?
The benchmark uses data derived from the Microsoft AdventureWorks Sample Database, structured into tables such as Product, Sales, and related entities. The schema is fixed and exposed in the browser via DuckDB‑WASM, so every model is evaluated against the same relational structure and sample data.
Can I run this benchmark against my own local model?
Yes. You can point the in‑browser runner at any OpenAI‑compatible API endpoint, including local servers such as llama.cpp or similar setups. Only the LLM API calls go to your endpoint; DuckDB‑WASM runs the SQL queries locally in the browser, so you can test your own models without modifying the underlying benchmark harness.
Are there different difficulty levels in the questions?
Yes. The 25 questions are split into Trivial, Easy, Medium, and Hard categories. Trivial questions involve single‑table selects with minimal columns and no aggregation, while Hard questions require complex joins, aggregations, and computed columns across multiple tables, making them much harder for models to get right.
How are results scored and what do the categories mean?
Each question produces one of three outcomes: Pass, Fail, or Error. Pass means the model’s SQL generates results that match the expected set (allowing for minor rounding differences). Fail means the SQL is valid but the result is not correct. Error covers cases where the SQL is invalid after multiple attempts, the model fails to use tools correctly, or the process times out or exceeds attempt limits.
Does the tool show which models are best for self‑hosting?
Yes. The site analyzes and highlights models that score highly while remaining feasible to run on consumer hardware, such as Qwen 3.5 27B. It also discusses quantization levels and CPU offload trade‑offs, helping users choose models that balance accuracy, speed, and resource constraints for local deployment.
What is grammar mode and why would I use it?
Grammar mode is an experimental option for small local models that cannot reliably handle tool‑calling. The benchmark uses a GGML‑style grammar to constrain the model’s output to valid SQL syntax and a small set of commands, so the model can return SQL directly or signal completion instead of using function calls. This can reduce harness‑level errors but is not yet robust across all small models.
How does the benchmark handle token usage and cost?
The benchmark reports total tokens used per model and runtime, enabling users to compare how efficiently different models solve the same tasks. The site notes that some models are slower but cheaper per token, while others are faster but more expensive, so users can factor token cost into their model choice even though the tool itself is free to use.
Can I contribute or extend this benchmark with custom questions or schemas?
The public benchmark page currently fixes the 25 questions and AdventureWorks‑derived schema, so you cannot change the schema or add new questions directly in the UI. However, the underlying runner is designed to be reused locally, so technically‑savvy users can adapt the codebase to run their own custom schemas or question sets, but that is not exposed as a standard feature on the website.