AI repos for Web scraping
Open-source AI projects built around Web scraping or genuinely useful if you work with it, each with what it does for you. 13 repos, most relevant first.
-
firecrawl
Scrape, crawl and map sites into markdown or JSON without running your own proxies or headless browsers.
-
Scrapling
Fetch pages with stealth fetchers, re-find elements after site changes, and run multi-page spider crawls.
-
crawl4ai
Crawl pages in a real browser and get clean Markdown or structured data, with deep crawling.
-
camoufox
Use a fingerprint-masking Firefox build with Playwright and rotating proxies so scrapers look less like bots.
-
CloakBrowser
Scrape bot-protected sites with a patched Chromium that works as a drop-in Playwright or Puppeteer replacement.
-
browser-use
Let an LLM navigate pages and extract data from sites that need interaction.
-
Agent-Reach
Picks ordered scraping backends per platform and checks which channels work with a doctor command.
-
playwright
Automate Chromium, Firefox and WebKit to load dynamic pages and extract data.
-
last30days-skill
Searches Reddit, X, YouTube, HN and Polymarket in parallel and ranks by engagement.
-
OpenManus
Built-in browser automation lets the agent navigate and pull content from web pages.
-
agent-browser
Read pages, snapshot content and fill forms from the command line, including reading pages without launching a browser.
-
playwright-mcp
Read pages through accessibility snapshots with no vision model needed, so an LLM can extract content reliably.
-
tldr-radio
Fetches TLDR's public archive pages and turns them into scripts for audio.