Spider
Low latency web data collector
Last verified:
What is Spider?
Spider is a fast web scraping and crawling API designed specifically for AI agents, RAG pipelines, and LLMs. It collects, transforms, and delivers web content at 100K+ pages per second with structured extraction, markdown output, and AI-ready formats. The API handles crawling, scraping, search, and real browser access through a single unified interface, making it easy for developers to get fresh web data at runtime.
Key features include multiple output formats (HTML, markdown, plain text, JSON, screenshots, PDF), smart rendering mode that automatically chooses between HTTP and Chrome based on page needs, built-in AI extraction for structured field pulling, and comprehensive anti-bot detection bypass with stealth headers, rotating residential proxies, and fingerprint rotation. Spider also offers a Search API that combines search engine queries with content extraction, screenshot capture, link graph extraction, and contact extraction pipelines.
Spider is built for AI engineers, data scientists, and developers building AGI systems, RAG pipelines, web data pipelines, market research tools, and content curation systems. The crawler is open-source under MIT license with 2K+ GitHub stars, allowing users to audit code, self-host, or use the managed cloud service without vendor lock-in. Official SDKs are available for Python, JavaScript, and Rust, plus integrations with LangChain, LlamaIndex, CrewAI, and other AI frameworks.
Spider pricing
Pricing model: Freemium
Pay-as-you-go pricing with free balance on signup (no card required). No monthly subscription and balance does not expire. Billing is based on bandwidth ($1/GB) plus compute ($0.001/min), with most pages costing a fraction of a cent. Credits are sold at $1 per 10,000 credits. Single purchases of $500 or more include automatic bonus up to 12% extra at $2K. No monthly minimum ($0). The Search API averages ~$0.003 per search. Usage can be tracked on the usage page and estimated with the pricing calculator at /compare.
Spider pros
- Crawls 100K+ pages per second with Rust-based async engine
- Open-source core with 2K+ GitHub stars, no vendor lock-in
- 85% pass rate on 80 anti-bot sites, highest in the field
- Single API for crawling, scraping, search, and real browser
- Smart mode auto-detects when JavaScript rendering is needed
- Clean markdown output strips ads, navigation, and boilerplate
- Built-in AI extraction pulls structured fields using LLM integration
- Rotating residential proxies and stealth headers by default
- Free balance on signup, no credit card required
- No monthly subscription, balance does not expire
- Failed requests billed at $0, only pay for successful responses
- 10,000 API requests per minute default rate limit
- Official SDKs for Python, JavaScript, and Rust
- Integrates with LangChain, LlamaIndex, CrewAI as document loader
- HTTP caching speeds up repeated crawls of same pages
- Streaming support for processing large crawls as they arrive
- Geo-targeted search with country_code parameter
- Full-page screenshots as base64 PNG and browser-rendered PDF
Spider cons
- Pay-as-you-go pricing may be expensive for high-volume scraping
- Billing based on bandwidth ($1/GB) plus compute ($0.001/min) can be complex
- No free tier beyond initial signup balance
- Rate-limited individual URLs per minute may slow target-heavy crawls
- Heavily protected sites require Browser Cloud which costs more
- No hard monthly minimum but no guaranteed fixed pricing either
- Self-hosting requires Rust knowledge and infrastructure management
- Search API costs ~$0.003 per search which adds up at scale
Frequently asked questions about Spider
What is Spider?
Spider is a fast web scraping and crawling API designed for AI agents, RAG pipelines, and LLMs. It returns clean structured data in markdown, HTML, JSON, or plain text formats. The API handles proxy rotation, JavaScript rendering, rate limiting, and anti-bot detection on your behalf.
How can I try Spider?
Sign up for a free balance to test with no credit card required. Alternatively, you can explore the open-source Spider engine on GitHub at https://github.com/spider-rs/spider to self-host the crawler.
Does my balance expire?
No. There is no monthly subscription. You can top up once and use the balance whenever you need it. The balance does not expire.
What are the rate limits?
Up to 10,000 core API requests per minute by default. If you need higher throughput, you can contact Spider for increased limits. Enterprise users can reach 500+ pages per second.
What output formats are supported?
HTML, raw, plain text, and several markdown formats are supported for content output. API responses also support JSON, JSONL, CSV, and XML. For AI workflows, markdown is recommended as it strips navigation, ads, and boilerplate.
Can Spider crawl all pages without a sitemap?
Yes. Spider crawls all necessary content without needing a sitemap. It follows links recursively from the seed URL. Spider rate-limits individual URLs per minute to avoid overloading the target server.
Does Spider respect robots.txt?
Yes. robots.txt compliance is on by default. However, you can disable it on a per-request basis when needed for your use case.
What happens if a crawl fails?
Failed requests are billed at $0. You only pay for responses that return data, so failed crawls do not cost anything.
What if I get blocked by anti-bot systems?
Spider includes an Unblocker with stealth, rotating proxies, and automatic retries. Heavily protected sites route to the Browser Cloud, which runs full browser sessions with anti-detection built in. Spider scored 85% pass rate on 80 anti-bot sites.
How does billing work exactly?
Each request is billed for bandwidth ($1/GB) plus compute ($0.001/min). Most pages cost a fraction of a cent. You can estimate your spend with the pricing calculator at /compare. Credits are purchased at $1 per 10,000 credits on a pay-as-you-go basis.