AI Agent Benchmark
Show HN: Black-box API bug detection across 7 AI systems
Last verified:
What is AI Agent Benchmark?
AI Agent Benchmark is an open benchmark designed to evaluate how effectively AI agents can identify functional bugs in live APIs using only a JSON request schema and a sample payload, with no access to source code, documentation, or additional context. Developed by KushoAI, this task benchmark specifically measures AI agent behavior in black-box API testing scenarios, evaluating end-to-end agent behavior including reasoning about API surfaces, designing targeted tests, and uncovering real bugs.
The benchmark consists of 20 carefully designed API scenarios spanning seven real-world application domains: e-commerce, payments, authentication, user management, scheduling, notifications, and search & filtering. Each scenario contains 3 to 8 planted bugs classified by complexity rather than severity—simple bugs (structural edge cases like missing fields), moderate bugs (field-level semantic constraints like invalid formats), and complex bugs (cross-field logic requiring understanding field relationships). The evaluation is fully automated, executing generated test cases against live reference implementations.
AI Agent Benchmark uses a weighted scoring model aligned with real-world priorities: Bug Detection (70% as the primary metric), Coverage (20% assessing API surface exploration), and Efficiency (10% evaluating signal-to-noise ratio). Score bands range from Weak (below 0.3) to Strong (above 0.7, comparable to a thorough human QA engineer). The complete dataset, scenarios, schemas, payloads, and evaluation harness are publicly available on Hugging Face under an MIT license.
This benchmark is designed for teams evaluating or building AI testing agents, including AI researchers, QA engineers, DevOps teams, software development teams, and technical project managers. It addresses a critical gap in AI-assisted API testing evaluation, as most existing benchmarks measure schema compliance or test volume rather than actual bug-finding capability under realistic black-box cond
AI Agent Benchmark pricing
Pricing model: Freemium
APIEval-20 is completely free to access and use. The full benchmark dataset including all 20 scenarios, request schemas, sample payloads, domain metadata, and evaluation harness is publicly available on Hugging Face under an MIT license. Users can browse scenarios, run evaluations through the hosted harness, and download the dataset at no cost. The benchmark has achieved 100+ downloads in its first week. KushoAI's broader API testing platform offers separate pricing: Lite plan is free for individuals (10 API endpoints, exhaustive functional test suites, unlimited AI-generated assertions); Starter plan is $49/month (50 API endpoints, export tests, audit logs); Pro plan is $249/month (250 API endpoints, real-time support); Enterprise plan starts at $999/month (unlimited endpoints and users, enterprise-grade security).
AI Agent Benchmark pros
- Open benchmark freely available on Hugging Face under MIT license
- Evaluates realistic black-box testing conditions with only schema and payload
- 20 diverse scenarios across 7 real-world application domains
- 97 known functional bugs across three difficulty tiers for rigorous testing
- Bug detection weighted at 70% emphasizing practical effectiveness
- Fully automated evaluation against live reference implementations
- Classifies bugs by reasoning complexity rather than severity
- KushoAI scores 76% on complex bugs, ranking first among all systems
- Lowest run-to-run variance critical for CI pipeline integration
- No source code or documentation access required for evaluation
- Score bands provide clear performance benchmarks (Weak to Strong)
- Strong performance (0.7+) comparable to thorough human QA engineers
- Versioned scenarios enable longitudinal comparison across releases
- Binary evaluation against live APIs ensures real bug detection
- First open benchmark specifically for AI API test generation
AI Agent Benchmark cons
- Functional testing only—security vulnerabilities explicitly out of scope
- Requires accurate JSON schema; poor specifications reduce effectiveness
- Initial training may be needed for teams unfamiliar with automated testing
- Limited support for non-standard API documentation formats
- No existing benchmark comparison data beyond KushoAI's own systems
- Complex bugs still challenging even for strongest coding agents (53% max)
- General-purpose LLMs perform poorly on complex tier (only 34%)
- Prompt chaining improves coverage but doesn't close complex bug gaps
Frequently asked questions about AI Agent Benchmark
What is APIEval-20 and what does it measure?
APIEval-20 is an open benchmark for evaluating how effectively AI agents can identify functional bugs in APIs using only a request schema and sample payload. It measures bug-finding capability under black-box conditions with no access to source code or documentation. The benchmark evaluates end-to-end agent behavior including reasoning about API surfaces, designing targeted tests, and uncovering real bugs across 20 scenarios.
How is the benchmark scored?
The final score combines three weighted factors: Bug Detection Score (70% primary metric measuring bugs found/total bugs), Coverage Score (20% measuring API surface exploration via field coverage), and Efficiency Score (10% penalizing unnecessarily large test suites). Final Score = 0.7 × Bug Detection Rate + 0.2 × Coverage Score + 0.1 × Efficiency Score. Score bands are: Weak (below 0.3), Developing (0.3-0.5), Proficient (0.5-0.7), and Strong (0.7-1.0).
What types of bugs does APIEval-20 test for?
Bugs are classified by complexity requiring different reasoning depths: Simple bugs require no semantic understanding (missing required fields, empty values, wrong data types). Moderate bugs require understanding field meaning (out-of-range values, format violations like malformed emails, invalid currency codes, enum boundary values). Complex bugs require understanding relationships between fields (mutually exclusive fields provided simultaneously, business rule violations, conditional field validity).
What inputs does the AI agent receive?
For each scenario, the agent receives exactly two inputs and nothing else: (1) JSON Schema containing the full request schema with field names, types, required/optional status, and documented constraints; (2) Sample Payload showing a concrete example of a valid request with realistic field values. The agent receives no response schema, implementation details, error messages, or changelog.
What domains are covered in the 20 scenarios?
The 20 scenarios span seven application domains chosen to reflect broad validation patterns, business logic complexity, and security sensitivity: E-commerce (order placement, coupon redemption, inventory adjustment), Payments (transaction creation, refund processing, currency conversion), Authentication (login, token refresh, password reset, session management), User Management (account creation, profile update, role assignment), Scheduling (appointment booking, availability queries, recurring events), Notifications (email dispatch, push configuration, preference management), and Search & Filtering (query construction, pagination, sort and rank).
How many bugs are in each scenario?
Each of the 20 scenarios contains between 3 and 8 planted bugs, for a total of 97 known functional bugs across the entire benchmark. Bugs are distributed across three complexity tiers: simple, moderate, and complex. The benchmark deliberately introduces constraints mirroring real-world conditions including incomplete context, evolving schemas, and hidden dependencies.
Where can I access the APIEval-20 dataset?
The complete APIEval-20 dataset is hosted on Hugging Face at huggingface.co/datasets/kusho-ai/api-eval-20. It includes all 20 scenarios, request schemas, sample payloads, domain metadata, and the evaluation harness. All scenarios are versioned with version suffixes (e.g., APIEval-20-v2) for longitudinal comparison. The dataset is available under an MIT license and users can browse scenarios and run evaluations through the hosted harness.
How does KushoAI's agent perform compared to other systems?
In the comparative benchmark evaluating seven AI systems across general-purpose LLMs, coding agents, and KushoAI's API testing agent, KushoAI ranked first across every complexity tier with 76% bug detection on complex bugs. The strongest coding-agent workflow detected 53% on the hardest tier, and the strongest general-purpose LLM detected only 34%. KushoAI also showed the lowest run-to-run variance, which is critical for teams integrating generated tests into CI pipelines.
What is the difference between APIEval-20 and other benchmarks?
APIEval-20 is the first benchmark measuring whether AI agents can find real API bugs from schema and payload alone under black-box conditions. Most existing benchmarks measure schema compliance, test volume, or syntactic correctness rather than actual bug-finding capability. APIEval-20 requires no implementation access or rich documentation, evaluates against live reference implementations, and uses binary evaluation where a bug is detected only when a test produces a response deviating from correct behavior corresponding to a planted defect.
What benchmarks are coming next after APIEval-20?
APIEval-20 is the first entry in a planned family of API testing benchmarks. Coming soon: APIEval-Security (dedicated benchmark for API security testing evaluating authentication weaknesses, authorization flaws, injection vulnerabilities, and OWASP API Security Top 10 categories), Agent Benchmark: Coding & Testing Agents (head-to-head comparison of Cursor, GitHub Copilot, Devin, and KushoAI on APIEval-20 scenarios), and APIEval-50 (larger set covering 50 APIs with expanded bug taxonomy including concurrency bugs, state-dependent failures, and multi-step workflow errors).