# crawl4ai

> Python crawler that turns web pages into clean Markdown or JSON for LLMs

- Repository: https://github.com/unclecode/crawl4ai
- Category: Browser & computer use
- Stars: 86,061 · Forks: 8,944 · Language: Python · License: Apache-2.0
- Experience: Some experience
- Page: https://ir.adyog.com/repos/unclecode/crawl4ai/

Language models read plain text better than raw HTML full of menus, scripts and footers. Crawl4AI loads a page in a real browser (Playwright), runs its JavaScript, and returns Markdown that keeps headings, tables and code blocks. The author says it began as a free alternative to a paid web-to-Markdown service.

You can use it three ways: as a Python library, as your own Docker server with a REST API and MCP support, or through the hosted Crawl4AI Cloud. The library and server are free under Apache-2.0. The cloud is pay-as-you-go and adds web search, answers, and handling of bot-protected pages.

Beyond Markdown, it can pull structured data using CSS or XPath schemas (no LLM needed) or an LLM into a typed JSON schema. It can also follow links across a site, manage sessions and proxies, and crawl many URLs at once.

## What it does

- **Clean and fit Markdown**: Converts pages to Markdown with headings, lists, tables, code and numbered citations. Optional filters (pruning, BM25 for a query, or LLM-based) strip boilerplate.
- **Structured extraction**: Uses CSS, XPath or regex schemas without an LLM, or any LLM provider into a typed JSON schema. A schema generator can write a reusable schema for you.
- **Deep and adaptive crawling**: Supports BFS, DFS and best-first site crawls with crash recovery. An adaptive crawler stops once it has learned enough to answer your query.
- **Browser control**: Handles persistent profiles, sessions, proxies, cookies, remote browsers over CDP, stealth mode, and Chromium, Firefox or WebKit.
- **Docker server and MCP**: Runs a token-protected REST API with a dashboard and playground. It can connect agents such as Claude Code over MCP.

## What you could build

- Convert documentation sites into Markdown to load into a RAG index.
- Give an AI agent a tool that fetches and reads live web pages.
- Scrape product or quote listings into JSON using a CSS schema, with no LLM cost.
- Crawl a whole site breadth-first and save each page as Markdown.
- Run a private crawling API in Docker for your team's pipelines.

## Who it's for

Developers building RAG, agent or data pipelines who are comfortable with Python and async code; Docker helps if you want the server.

## Get started

Install the package, run the setup command once to download the browser, then call AsyncWebCrawler on a URL and print result.markdown. The crwl command line tool offers the same from a terminal.

```
pip install -U crawl4ai
```
```
crawl4ai-setup
```
```
crwl https://news.ycombinator.com -o markdown
```

Tags: web-scraping, crawler, markdown, rag, mcp