Agent-QA
Open-source AI end-to-end testing for web and mobile apps
Last verified:
What is Agent-QA?
agent-qa by Vostride is an open-source, agentic QA harness that lets teams write end-to-end tests in natural language and run them across web and mobile targets with execution memory and self-healing behavior. Tests are authored as human-readable steps (actions and assertions) and can be executed from a polished dashboard or a CLI, with detailed step-level telemetry and artifacts to triage failures. The platform builds and reuses execution memory across runs so future tests benefit from observed UI state and healed paths, reducing flaky failures and unnecessary rediscovery. It also exposes programmatic primitives (MCP, skills, and hooks) so engineering teams can integrate test runs into CI, seed fixtures, run sandboxed setup/teardown code, and control LLM providers and models for planning and reasoning.
Agent-QA pricing
Pricing model: Freemium
The website presents agent-qa as an open-source product (core runtime and tools available via GitHub and npm) and emphasizes local/CI-run usage without a bundled hosted pricing page; sandbox demos and docs show features available when running the project yourself. There is no public hosted tiered pricing table on the site—core components and examples indicate free, self-hosted usage of the open-source agent-qa runtime, while integrations with external LLM providers, hosted infra, or team-scale artifact storage would incur separate costs managed by the user or organization.
Agent-QA pros
- Natural-language test authoring for non-developers
- Execution memory that persists observations across runs
- Self-healing test execution to recover from UI drift
- Detailed step-level telemetry and run artifacts
- Polished dashboard plus a powerful CLI
- Action cache that reuses validated plans to speed runs
- Ability to run tests on web and mobile targets
- Sandboxed hooks (Node, Bun, Python, Bash) in Docker
- Version-controlled tests and configs for code review
- Programmatic API primitives (MCP, skills) for automation
- Bring-your-own-LLM support for provider flexibility
- Validated steps reuse to reduce token usage
- Support for structured outputs from hooks into runs
- Cache-driven execution speedups (measurable 5x example)
- Open-source core with npm packages for local integration
Agent-QA cons
- Requires understanding of YAML test structure for advanced flows
- Initial setup for sandboxed hooks and Docker adds complexity
- Memory model may require tuning to avoid stale context
- Self-healing can increase run time when exploring alternate paths
- Bringing own LLM requires managing provider credentials and costs
- Action cache benefits depend on similarity of subsequent runs
- Some advanced features require CLI or programmatic use (not only GUI)
- Observability and artifact storage can increase infrastructure costs
Frequently asked questions about Agent-QA
How do I write tests for agent-qa?
Write tests as YAML files using natural-language steps that list actions and assertions; the site shows examples where each step is a human-readable instruction (click, fill, select, assert) and can include identifiers like input labels and visible roles to guide agents.
What targets can agent-qa run against?
agent-qa runs across web and mobile targets; the documentation and demos demonstrate running tests against web applications (Chromium example) and mention cross-target suites so workflows can include multiple targets in one suite.
What is execution memory and how is it used?
Execution memory indexes observations from product, suite, and test runs and injects matching memory into future steps so agents can avoid rediscovering stable UI contracts and focus on new behavior; the site shows workspace navigation and command-palette memories as examples.
How does self-healing work?
When a sub-action fails, agent-qa re-observes the UI and attempts alternate paths within the same run (retries or different selectors) to recover from UI drift instead of failing immediately, and it curates healed steps into memory for future runs.
Can I run custom setup or teardown logic?
Yes: you can run sandboxed hooks written in Node, Bun, Python, or Bash inside isolated Docker containers to seed fixtures, call APIs, set environment variables, and emit structured outputs back into the active test run.
Do I have to use the provided LLMs?
No: agent-qa supports bring-your-own-LLM, letting you use OpenAI-, Anthropic-compatible endpoints, Gemini, local or open-source models, and other subscriptions so teams can pick models that meet their privacy, cost, and performance constraints.
How does caching speed up test runs?
The action cache reuses validated plans across similar subsequent runs so the planner skips redundant work; the site cites reductions in planner token usage and example speedups (e.g., 42s to 8s) when cached plans apply.
Can I integrate agent-qa with CI/CD?
Yes: agent-qa exposes CLI and programmatic primitives (MCP and skills) so tests, runs, artifacts, and triage workflows can be enqueued and inspected from CI pipelines and automation tooling.
How are tests reviewed and versioned?
Tests, hooks, memory, and suite logic are maintained as version-controlled code so teams can diff, review, and reuse test artifacts the same way they manage application code; docs show diff examples for test changes.
What observability is available for failed runs?
The dashboard and CLI provide step-level status, durations, artifacts, cache hits, and run attributes (runner, trigger), enabling teams to inspect failed steps, healed attempts, and memory contributions to triage regressions.