Midscene
AI-powered, vision-driven UI automation for every platform.
Last verified:
What is Midscene?
Midscene.js is an AI-powered, vision-driven UI automation SDK that allows developers to control web pages, mobile apps, and desktop applications using natural language. It serves as an AI operator for web, Android, iOS, HarmonyOS, and desktop platforms (Mac, Windows, Linux), enabling users to describe their goals and steps in plain language while Midscene plans and executes the UI interactions automatically.
The tool offers three core API categories: Interaction APIs (aiAct, aiTap, aiHover, aiInput, aiScroll, aiPinch, aiLongPress, aiDoubleClick, aiRightClick, aiKeyboardPress) for controlling the interface, Data Extraction APIs (aiQuery, aiAsk, aiBoolean, aiNumber, aiString) for pulling structured data in JSON format, and Utility APIs (aiAssert, aiLocate, aiWaitFor) for validation and waiting conditions. It supports both auto-planning mode where AI autonomously plans and executes flows, and workflow style where developers split complex logic into multiple steps for stability.
Midscene is designed for developers who need UI automation for testing, scraping, or task automation. It integrates with Puppeteer and Playwright for web automation, offers a Chrome Extension for zero-code experimentation, supports YAML scripts for non-coders, and provides MCP (Model Context Protocol) services for AI coding tools. The tool is particularly useful for automating beyond DOM/accessibility limitations, including canvas surfaces, and works with visual-language models like Qwen3-VL, Doubao-1.6-vision, gemini-3-pro, and UI-TARS for pure-vision-based element localization.
Midscene pricing
Pricing model: Freemium
Midscene.js is open-source and MIT licensed, completely free to use. There is no paid tier or subscription. The tool is decoupled from any cloud service and model provider, allowing users to choose either public or private deployment. Users only pay for the AI model API calls they make (e.g., OpenAI, Aliyun, Google Gemini, Volcengine). The Chrome Extension is also free from the Chrome Web Store. Self-hosting open-source models like UI-TARS or Qwen2.5-VL is free except for infrastructure costs.
Midscene pros
- AI-powered automation with natural language instructions
- Vision-driven approach works on web, mobile, desktop, and canvas
- Supports 7+ platforms: Android, iOS, HarmonyOS, Linux, macOS, Windows, Web
- Three core API types: interaction, data extraction, and utility
- Auto-planning mode lets AI autonomously plan and execute tasks
- Chrome Extension enables zero-code quick experience
- YAML script support for non-developers or simple smoke tests
- Pure-vision localization uses screenshots only, skipping DOM for actions
- Significantly reduces token consumption compared to DOM-based approaches
- Visualized replay report and built-in playground for easy debugging
- Caching feature reduces execution time from 51s to 28s in tests
- Supports multiple AI models: GPT-4o, Qwen3-VL, Gemini-2.5-Pro, UI-TARS
- Multi-model setup allows different models for planning and interaction
- MCP Server exposes atomic actions as tools for upper-layer AI agents
- Open source with MIT license, free to use and deploy
- Flexible integration: Puppeteer, Playwright, Bridge Mode, custom interfaces
- Deep Locate feature for precise element positioning when needed
- File upload support via fileChooserAccept parameter in aiTap
- Image prompting allows attaching reference images to prompts
- Custom model configuration via JavaScript options or environment variables
Midscene cons
- Cache is web-only for element localization, not available on mobile
- XPath-based caching has limitations with Canvas, cross-origin iframes, Shadow DOM
- Cache misses occur when DOM structure changes, requiring AI re-evaluation
- UI-TARS model requires special MIDSCENE_USE_VLM_UI_TARS config or JSON parsing errors
- Qwen-VL mode requires MIDSCENE_USE_QWEN_VL=1 env variable or uses more tokens
- Gemini-2.5-Pro requires MIDSCENE_USE_GEMINI=1 config to work properly
- Right-click and hover not available on Android platform
- Pinch gesture not supported on HarmonyOS due to uitest framework limitations
- Chrome extension cannot use Azure OpenAI Service mode
- Multi-model setup requires careful configuration of different model keys
- screenshotShrinkFactor above 3 may cause blurry images on mobile affecting accuracy
- Complex <iframe /> or <canvas /> content has limitations with GPT-4o
- Report with html-and-external-assets requires HTTP server, cannot open via file://
- Key combination not supported in aiKeyboardPress (single key only)
- Replanning cycles default to 20 (may need adjustment for complex tasks)
Frequently asked questions about Midscene
What is Midscene.js?
Midscene.js is an AI-powered, vision-driven UI automation SDK that lets AI serve as your browser and mobile app operator. Simply describe what you want to do in natural language, and it will help you operate web pages, validate content, extract data in JSON format, and perform complex UI automation tasks across web, mobile (Android/iOS/HarmonyOS), and desktop platforms.
How do I get started with Midscene?
For a zero-code quick experience, install the Midscene Chrome Extension from the Chrome Web Store. After installation, activate the extension and you'll see a 'Midscene' sidebar in your browser. Set up your AI model API key using environment variables (OPENAI_API_KEY, MIDSCENE_MODEL_NAME, etc.), and you can immediately start using Act, Query, Assert, and Tap features without writing any code.
What AI models does Midscene support?
Midscene supports multiple AI models including: GPT-4o (default multimodal LLM from OpenAI), Qwen3-VL and Qwen2.5-VL-72B-Instruct (Alibaba's vision-language models), Gemini-2.5-Pro (Google's multimodal model), and UI-TARS (open-source GUI agent model). It also supports custom models via OpenAI-compatible APIs. Different models have different strengths: VL models provide better visual grounding for element localization.
What is the difference between auto planning and workflow style?
Auto planning (aiAct/ai) lets AI autonomously plan and execute the entire flow to complete a task - you provide a natural language description and Midscene handles step decomposition. Workflow style split complex logic into multiple explicit steps using aiQuery, aiBoolean, aiTap, etc., which improves stability for complex tasks. Auto planning is more 'fashionous' but may be slower; workflow style gives developers more control.
How does caching work in Midscene?
Midscene caches AI planning steps and element localization data (XPath for web). With caching enabled, execution time can drop from 51 seconds to 28 seconds. Cache strategies include read-write (default), read-only (requires manual flushCache()), and write-only. Cache files are stored in ./midscene_run/cache/. Note that query results like aiBoolean, aiQuery, aiAssert are never cached. Cache is web-only for element localization.
Can I use Midscene without writing code?
Yes. Midscene offers a Chrome Extension for zero-code in-browser experience with Act, Query, Assert, and Tap features. It also supports YAML scripts where you can write automation flows in YAML format instead of JavaScript. The YAML format defines tasks with steps like ai, aiAssert, aiTap, making it accessible for team members who don't know the API.
What platforms does Midscene support?
Midscene supports Web (via Puppeteer, Playwright, or Bridge Mode), PC desktop applications on macOS, Windows, and Linux, and mobile devices including Android, iOS, and HarmonyOS. For web, you can integrate with Puppeteer/Playwright or use Bridge Mode to control desktop browsers. For mobile, use JavaScript SDK with adb (Android) or WebDriverAgent (iOS).
How do I extract data from a page?
Use the data extraction APIs: aiQuery() for structured data in JSON format (specify the expected format like 'string[], list of task names' or '{name: string, age: string}[]'), aiAsk() for open-ended questions, aiBoolean() for boolean values, aiNumber() for numeric values, and aiString() for string values. These APIs return data based on AI understanding of the UI, not just DOM extraction.
What is the pure-vision approach in Midscene?
Midscene is all-in on pure-vision for UI actions: element localization and interactions are based on screenshots only, not DOM. This works across web, mobile, desktop, and even canvas surfaces. It uses far fewer tokens by skipping DOM for actions, cutting cost and speeding up runs. For data extraction and page understanding, DOM can still be included when needed via domIncluded option.
How do I configure a custom AI model provider?
Use environment variables: OPENAI_API_KEY (required), OPENAI_BASE_URL (custom endpoint), MIDSCENE_MODEL_NAME (model name). For specific models: set MIDSCENE_USE_QWEN_VL=1 for Qwen, MIDSCENE_USE_GEMINI=1 for Gemini, MIDSCENE_USE_VLM_UI_TARS=1 for UI-TARS. You can also configure via JavaScript using modelConfig option in agent constructor or overrideAIConfig() function. Custom providers include Aliyun, OpenRouter, Volcengine, Azure OpenAI.