D4Vinci /
Scrapling
Python web scraping framework, from one request to large crawls
- 89.2k stars
- 4.2k stars since 3 Oct
- Python
- Updated today
- Some experience
- Browser & computer use
Scrapers break when a site changes its layout or blocks automated visitors. Scrapling tries to cover both problems in one library. You can fetch a single page or run a crawl over many pages with the same tools.
It has three parts. Fetchers download pages, including a stealth option that uses a headless browser and, per the README, handles Cloudflare Turnstile. The parser selects elements with CSS or XPath. If you save elements with auto_save, you can pass adaptive=True later to find them again after the page structure changes. A Spider class lets you write a parse method and run concurrent crawls with pause/resume, proxy rotation, and a request pace that slows down when a site starts blocking.
The README also lists a command-line interface, an MCP server and an agent skill, so AI tools can use it for scraping. It doesn't compare itself with other tools by name. It suits people who would otherwise stitch together an HTTP client, a browser driver and a parser by hand.
What it does
Adaptive element matching
Save matched elements with auto_save=True. If the site's design changes later, css(..., adaptive=True) tries to find them again.
Several fetchers, one stealth
StealthyFetcher loads pages in a headless browser and, per the README, handles Cloudflare Turnstile. Other fetchers cover plain, async and dynamic page loading.
Spider framework for crawls
Subclass Spider, set start URLs and yield items from an async parse method. It supports concurrent multi-session crawls, pause/resume and real-time stats with streaming.
Proxy rotation and adaptive pacing
Crawls can rotate proxies automatically. Crawl speed adapts to how fast each site responds and backs off when it starts blocking.
CLI and MCP server
A command-line interface and an MCP server are documented, along with an agent skill for AI tools.
What you could build
- Scrape product listings from a shop and keep the scraper working after the site is redesigned.
- Crawl a whole site with a Spider and stream the items out as they arrive.
- Fetch pages behind Cloudflare Turnstile with a headless stealth browser.
- Rotate proxies across a long crawl and pause and resume it later.
- Give an AI agent web scraping ability through the MCP server.
Get started
The README excerpt shows no install command, so use the PyPI package name scrapling and the docs at scrapling.readthedocs.io. Then try the StealthyFetcher example to fetch a page and select elements with css().
Commands are copied from the project's README. Read it on GitHub for the full setup.
More in Browser & computer use
browser-use
Python library and CLI that lets an LLM drive a web browser to finish tasks
Agent-Reach
Setup layer that lets AI agents read the web, YouTube, GitHub, Reddit and more
playwright
One API to test and automate Chromium, Firefox and WebKit, for scripts and agents
crawl4ai
Python crawler that turns web pages into clean Markdown or JSON for LLMs
agent-browser
A command-line tool that lets AI agents drive a Chrome browser
UI-TARS-desktop
Open-source agent stack that drives browsers and desktops with vision models and MCP
playwright-mcp
Browser automation server that lets AI assistants drive web pages through Playwright
CloakBrowser
Patched Chromium that works as a drop-in Playwright replacement for bot-protected sites
page-agent
JavaScript library that lets people control a web page with plain-language commands
ego-lite
A macOS browser where AI agents work in their own Spaces while you keep browsing