Developers building RAG, agent or data pipelines who are comfortable with Python and async code; Docker helps if you want the server.

What can you build with crawl4ai?

Convert documentation sites into Markdown to load into a RAG index. Give an AI agent a tool that fetches and reads live web pages. Scrape product or quote listings into JSON using a CSS schema, with no LLM cost. Crawl a whole site breadth-first and save each page as Markdown. Run a private crawling API in Docker for your team's pipelines.

unclecode /

crawl4ai

Python crawler that turns web pages into clean Markdown or JSON for LLMs

86.1k stars
1.4k stars since 3 Oct
Python
Updated today
Some experience
Browser & computer use

Open on GitHub (opens in a new tab) Website (opens in a new tab)

Language models read plain text better than raw HTML full of menus, scripts and footers. Crawl4AI loads a page in a real browser (Playwright), runs its JavaScript, and returns Markdown that keeps headings, tables and code blocks. The author says it began as a free alternative to a paid web-to-Markdown service.

You can use it three ways: as a Python library, as your own Docker server with a REST API and MCP support, or through the hosted Crawl4AI Cloud. The library and server are free under Apache-2.0. The cloud is pay-as-you-go and adds web search, answers, and handling of bot-protected pages.

Beyond Markdown, it can pull structured data using CSS or XPath schemas (no LLM needed) or an LLM into a typed JSON schema. It can also follow links across a site, manage sessions and proxies, and crawl many URLs at once.

What it does

Clean and fit Markdown
Converts pages to Markdown with headings, lists, tables, code and numbered citations. Optional filters (pruning, BM25 for a query, or LLM-based) strip boilerplate.
Structured extraction
Uses CSS, XPath or regex schemas without an LLM, or any LLM provider into a typed JSON schema. A schema generator can write a reusable schema for you.
Deep and adaptive crawling
Supports BFS, DFS and best-first site crawls with crash recovery. An adaptive crawler stops once it has learned enough to answer your query.
Browser control
Handles persistent profiles, sessions, proxies, cookies, remote browsers over CDP, stealth mode, and Chromium, Firefox or WebKit.
Docker server and MCP
Runs a token-protected REST API with a dashboard and playground. It can connect agents such as Claude Code over MCP.

What you could build

Convert documentation sites into Markdown to load into a RAG index.
Give an AI agent a tool that fetches and reads live web pages.
Scrape product or quote listings into JSON using a CSS schema, with no LLM cost.
Crawl a whole site breadth-first and save each page as Markdown.
Run a private crawling API in Docker for your team's pipelines.

Get started

Install the package, run the setup command once to download the browser, then call AsyncWebCrawler on a URL and print result.markdown. The crwl command line tool offers the same from a terminal.

pip install -U crawl4ai

crawl4ai-setup

crwl https://news.ycombinator.com -o markdown

Commands are copied from the project's README. Read it on GitHub for the full setup.