Giovannigatti
Show HN: I benchmarked LLM agents on fixing real-world security vulnerabilities
Last verified:
What is Giovannigatti?
CVE-Bench is a benchmark tool designed to evaluate how well LLM (Large Language Model) agents can fix real-world security vulnerabilities. The tool tests AI models on patching 20 real CVEs (Common Vulnerabilities and Exposures) across 18 Python projects including Pillow, GitPython, yt-dlp, and urllib3. Each agent runs in a sandboxed Docker container with access to the vulnerable project's source code and must fix the vulnerability within at most 20 turns.
The benchmark uses three distinct prompt conditions to test different aspects of security reasoning: Advisory (full security advisory with file/function names), Diagnose (exploit report without file path), and Locate (precise file/function name but only a hint of what's wrong). The primary metric is binary: all security tests must pass, as a 90%-patched vulnerability is still vulnerable. The tool also runs the project's existing regression suite to reject fixes that break previously supported behavior.
CVE-Bench is designed for security researchers, AI researchers, and practitioners deploying LLM agents for security patching. It helps understand how AI models fail when fixing vulnerabilities rather than just ranking models. The benchmark covers 15 CWE categories with CVSS scores from 2.1 to 9.8, with 16 of 20 CVEs disclosed after March 2026 to reduce contamination risk from training data.
Giovannigatti pricing
Pricing model: Freemium
CVE-Bench is free and open-source. The benchmark code, task files, and result data are all available at https://github.com/GiovanniGatti/cve-bench. The author spent nearly $100 on inference costs during development (mostly from Anthropic models), but Poolside offered free model access for their Laguna models during the evaluation period. No paid plans or tiers exist for the benchmark itself.
Giovannigatti pros
- Uses 20 real-world CVEs from actual security vulnerabilities
- Tests 5 frontier LLM models across 3 prompt conditions for 300 total runs
- Sandboxed Docker environment prevents agents from cheating or accessing external resources
- Security tests derived from maintainer's own fix provide ground truth validation
- Covers 15 different CWE vulnerability categories
- CVSS scores range from 2.1 to 9.8 for diverse severity testing
- Binary pass/fail metric avoids false confidence from partial patches
- Includes regression testing to catch fixes that break existing behavior
- Secondary signals capture cost and behavior (tokens, tool calls, reads before edit)
- 16 of 20 CVEs disclosed after March 2026 reduces training data contamination
- Open-source repository with benchmark, task files, and result data available
- Identifies four specific recurring failure patterns for actionable insights
- Locate condition tests genuine security reasoning without advisory hints
- Statistical significance testing using McNemar test with continuity correction
- Constrained toolset prevents agents from mining git history or scraping web
Giovannigatti cons
- Only 20 CVEs limits statistical power for within-family comparisons
- All Python projects only - no support for other languages
- Excludes monorepos and fixes touching compiled languages alongside Python
- 20-turn budget often causes models to exhaust before completing fixes
- No bash tooling limits flexibility and may handicap some models
- Agents cannot access security tests during fixing - no feedback signal
- Within-family model comparisons remain underpowered statistically
- Narrow task set skews toward compact, self-contained patches
- Missing complex vulnerability classes requiring protocol-level changes
- Contamination risk cannot be fully eliminated without training corpus access
Frequently asked questions about Giovannigatti
What is CVE-Bench and what does it measure?
CVE-Bench is a benchmark that evaluates how often LLM agents can fix real-world security vulnerabilities. It measures solve rates across 20 real CVEs in Python projects, testing five frontier models (gpt-5.5, gpt-5.4-mini, gpt-5.4-nano, laguna-m.1, laguna-xs.2) under three prompt conditions (advisory, diagnose, locate). The primary metric is binary: all security tests must pass, as partial patches leave vulnerabilities intact.
What are the three prompt conditions and how do they differ?
The three conditions test different aspects of security reasoning: (1) Advisory - model gets the full security advisory with file/function names and attack vector, testing how well models map described vulnerabilities to code; (2) Diagnose - model gets an exploit report (attacker can do X) but no file path, requiring it to search codebase and form hypotheses; (3) Locate - precise file and function name but only a hint of what's wrong, testing whether the model can recognize dangerous code on its own like a security auditor.
What is the most dangerous failure mode CVE-Bench discovered?
The most operationally dangerous failure mode is a patch that looks right, passes every visible regression test, and confidently claims the bug is fixed, but leaves the vulnerability intact. This occurs when the model fixes the wrong part of the vulnerability (e.g., removing the 'navigate/unspecified' XSRF exemption while leaving 'no-cors' exempt). No error signal reaches the security team because visible tests pass, making this a false confidence attack surface that can ship undetected.
Which model performed best and what were the solve rates?
gpt-5.5 performed best with 50% overall solve rate (30/60 tasks) and 60% under the most favorable advisory condition (12/20). gpt-5.4-nano achieved 48% (29/60), gpt-5.4-mini achieved 43% (26/60), laguna-m.1 achieved 32% (19/60), and laguna-xs.2 achieved 33% (20/60). All four cross-family pairwise comparisons reached statistical significance, but within-family gaps were not statistically significant.
Does spending more on expensive models improve results?
No. gpt-5.5 costs roughly 12× more per run than gpt-5.4-mini for statistically equivalent outcomes within the OpenAI family. The expensive models (gpt-5.5 and laguna-m.1) deliberate more with 19+ tool calls and abandon without editing in 16-20 runs out of 60, while cheaper models (mini and nano) act quickly with 13-14 tool calls. None of the extra deliberation translates into better outcomes, making cheaper OpenAI models the rational choice.
What are the four recurring failure patterns CVE-Bench identified?
(1) Wrong-search drift - model opens right file but pivots to wrong files due to incorrect early inference, exhausting budget without editing; (2) Budget exhaustion mid-implementation - correct diagnosis but incomplete fix, running out of turns between scaffolding and implementation; (3) Partial fix - real coherent edits to right code with visible tests passing, but hidden security tests cover vectors not implemented; (4) Correct file, wrong part of vulnerability - fixes wrong exemption/path while leaving the actual vulnerable path intact.
Which CVEs were unsolvable by any model?
Four CVEs were not solved by any model across all 15 runs (5 models × 3 prompt types): CVE-2026-26331 (yt-dlp netrc injection), CVE-2026-44431 (urllib3 proxy SSRF), CVE-2026-44432, and GHSA-r758-8hxw-4845. These are not benchmark defects - models made real edits in most cases but no edit was sufficient to pass security tests. All four had sparser editing than the rest of the task set, with several individual runs making no changes at all.
How does CVE-Bench prevent agents from cheating?
CVE-Bench uses a constrained toolset (list files, read files, search codebase, edit files, create/delete files, run pytest) within a sandboxed Docker container. The agent cannot read, write, or execute outside allowed folders. The author deliberately opted out of adding bash tooling because agents can mine git history, search for reference solutions on GitHub, and scrape the web - fighting this cheating is particularly hard and requires better steering, reward hack judges, and continuous sample reviews.
What is the contamination risk and how was it addressed?
Contamination is an open problem because CVEs become public only after the fix is merged, so the patch commit may predate CVE disclosure by months. 16 of 20 CVEs were publicly disclosed after March 2026 (after all evaluated models' training cutoffs), reducing but not eliminating exposure risk. What's less likely is that the full chain (advisory text, vulnerable code, and fix) appears together in training data in a form that would directly short-circuit the task. There's no principled way to verify this without access to training corpora.
How can I contribute to or use CVE-Bench?
The benchmark, task files, and result data are all open at https://github.com/GiovanniGatti/cve-bench. Contributions and task submissions are welcome. Each task has a setup script that initializes the vulnerable repository in a container and a test_security.py that fails on the vulnerable commit and passes on the fixed one. The author validated each security test against both commits using Claude Sonnet to generate them from the advisory, original code, and fix.