Scrapling
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
Last verified:
What is Scrapling?
Scrapling is an adaptive Web Scraping framework for Python that handles everything from a single HTTP request to full-scale concurrent crawls. Its parser learns from website changes and automatically relocates your elements when pages update, making scraping scripts resilient to design changes. The framework's fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box, and its spider system enables scalable concurrent, multi-session crawls with pause/resume functionality and automatic proxy rotation.
Key features include: a Scrapy-like Spider API with async callbacks, adaptive element tracking that survives website redesigns, three fetcher types (Fetcher for HTTP requests, DynamicFetcher for JavaScript-heavy sites, StealthyFetcher for anti-bot bypass), built-in proxy rotation with DNS leak prevention, pause/resume crawl checkpoints, streaming mode for real-time item delivery, an MCP server for AI-assisted scraping with Claude/Cursor, 92% test coverage with full type hints, and an interactive IPython shell for development.
Scrapling is designed for web scrapers of all levels - from beginners who want to scrape URLs from the terminal without writing code, to developers migrating from BeautifulSoup or Scrapy, to professionals building large-scale concurrent crawls. It's built by web scrapers for web scrapers and regular users, with something for everyone including complete async support, Docker images with all browsers pre-installed, and CLI tools for code-free scraping.
Scrapling pricing
Pricing model: Freemium
Scrapling is completely free and open source under BSD-3 License. The core library is free with no paid tiers. All features including Fetcher, DynamicFetcher, StealthyFetcher, spiders, adaptive scraping, and MCP server are free. Optional dependencies are installed via pip extras: scrapling[fetchers] for browser fetchers, scrapling[ai] for MCP server, scrapling[shell] for interactive shell and CLI extract command, scrapling[all] for everything. Docker images from DockerHub and GitHub registry are free. The donate page exists for supporting development through GitHub stars, Twitter follows, sponsoring, or buying coffee, but there are no mandatory paid plans.
Scrapling pros
- Adaptive element tracking automatically relocates elements after website redesigns
- Bypasses Cloudflare Turnstile/Interstitial anti-bot protection out of the box
- Scrapy-like Spider API familiar to Scrapy users with async parse callbacks
- Three specialized fetchers: Fetcher, DynamicFetcher, and StealthyFetcher for different use cases
- Pause and resume crawls with checkpoint-based persistence
- Built-in proxy rotator with cyclic or custom rotation strategies
- Streaming mode streams scraped items as they arrive with real-time stats
- MCP server integration for AI-assisted web scraping with Claude and Cursor
- 92% test coverage with full type hints for excellent IDE support
- Interactive IPython shell with Scrapling shortcuts for faster development
- Can scrape URLs from terminal without writing any code via CLI
- Built-in ad blocking covering ~3,500 known ad/tracker domains
- DNS-over-HTTPS support prevents DNS leaks when using proxies
- 10x faster JSON serialization than Python standard library
- Docker image with all browsers automatically built and pushed with each release
Scrapling cons
- Requires Python 3.10 or higher, not compatible with older Python versions
- Adaptive feature disabled by default, must be manually enabled
- Fetchers require separate installation with pip install scrapling[fetchers]
- Browser dependencies must be installed separately via scrapling install command
- MCP server requires pip install scrapling[ai] as optional dependency
- Shell features require separate pip install scrapling[shell]
- Memory usage higher than simple HTTP-only scrapers due to browser automation
- Development mode response cache meant for iterating on parse logic, not production use
Frequently asked questions about Scrapling
What is Scrapling and what does it do?
Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl. Its parser learns from website changes and automatically relocates elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. Its spider framework enables concurrent, multi-session crawls with pause/resume and automatic proxy rotation - all in a few lines of Python.
How does the adaptive scraping feature work?
The adaptive feature uses intelligent similarity algorithms to relocate elements after website changes. When you pass adaptive=True to CSS selector methods, Scrapling can find elements even if the website structure changes. You can also enable it globally with Fetcher.configure(adaptive=True) or StealthyFetcher.adaptive=True. The parser learns from website changes and automatically finds your elements when pages update.
What is the difference between Fetcher, DynamicFetcher, and StealthyFetcher?
Fetcher is for fast HTTP requests with browser fingerprint impersonation, TLS fingerprint spoofing, and HTTP/3 support - fastest speed but no JavaScript loading. DynamicFetcher handles dynamic websites with full browser automation using Playwright with Chromium/Chrome - supports JavaScript loading. StealthyFetcher adds advanced anti-bot bypass including Cloudflare Turnstile/Interstitial, CDP runtime leak prevention, WebRTC leak blocking, canvas noise generation, and headless detection patching - highest stealth level.
How do I use the pause and resume feature in spiders?
Pass the crawldir argument when starting your spider to enable checkpoint-based persistence. The Crawler Engine periodically saves pending requests and seen URL fingerprints to a pickle file. On graceful shutdown (Ctrl+C), a final checkpoint is saved. Restarting the spider with the same crawldir resumes from where you left off, skipping start_requests() and restoring the scheduler state. Checkpoints are automatically cleaned up upon successful completion.
What is the Scrapling MCP Server?
The Scrapling MCP Server brings Scrapling's web scraping capabilities directly to AI chatbots like Claude or AI agents that support MCP. It provides ten tools: get, bulk_get, fetch, bulk_fetch, stealthy_fetch, bulk_stealthy_fetch, screenshot, open_session, close_session, and list_sessions. It allows conversational scraping, bypasses Cloudflare Turnstile, and lets you select specific elements via CSS selectors before passing content to AI, saving tokens and time.
How do I install Scrapling with all features?
Install the base package with pip install scrapling which includes only the parser engine. For fetchers, run pip install scrapling[fetchers] then scrapling install to download browsers. For MCP server, run pip install scrapling[ai]. For shell features, run pip install scrapling[shell]. For everything, run pip install scrapling[all] then scrapling install. Alternatively, use Docker: docker pull pyd4vinci/scrapling for an image with all extras and browsers.
Does Scrapling support async operations?
Yes, Scrapling has complete async support across all fetchers and dedicated async session classes. Use AsyncFetcher for async HTTP requests, DynamicFetcher.async_fetch for async dynamic content fetching, and AsyncStealthySession for async stealth operations. The Spider API uses async def parse(self, response) callbacks. You can stream items asynchronously with async for item in spider.stream() for real-time processing.
How do I handle blocked requests in Scrapling?
The Crawler Engine automatically detects blocked responses and retries requests up to max_blocked_retries times. You can customize the blocking detection and retry logic through hooks. The is_blocked() and retry_blocked_request() hooks are built-in for this purpose. The Session Manager routes requests to the correct session based on the request's sid field, and you can enable proxy rotation with ProxyRotator for better unblocking.
What selection methods does Scrapling support?
Scrapling supports CSS selectors, XPath selectors, filter-based search, text search, and regex search. You can use page.css() for CSS selectors, page.xpath() for XPath, page.find() and page.find_all() for tag/attribute search, page.find_by_text() for exact text matching, page.find_by_regex() for regex text matching, and element.find_similar() to find similar elements. The API is familiar to Scrapy/Parsel users with the same pseudo-elements.
How does Scrapling compare to Scrapy?
Scrapling's spider system is Scrapy-inspired with similar concepts: Spider subclass with start_urls and parse(), Request/Response objects, scheduler with deduplication, and export methods. Key differences: Scrapling uses async start_requests() and async def parse(), has built-in deduplication in scheduler, Session Manager with multi-session support instead of middlewares, built-in is_blocked() detection vs custom middlewares, crawldir for pause/resume vs JOBDIR, and streaming mode via spider.stream() which Scrapy lacks. Scrapling also has adaptive scraping and anti-bot bypass out of the box.