AI repos for data engineers

Open-source AI projects worth it for people who build data pipelines, ETL, warehouses and document processing, each with what it does for you. 10 repos, most relevant first.

  1. markitdown Convert PDFs, Office files, audio and HTML into Markdown for document processing pipelines. Memory, RAG & data191.7k3.8k
  2. Unlimited-OCR Convert scanned multi-page PDFs and document images into parsed text, locally or behind a vLLM endpoint. Models & local AI28.2k1.6k
  3. liteparse Convert PDFs and Office files to layout-aware Markdown or JSON with bounding boxes, locally and fast. Memory, RAG & data12.9k142
  4. firecrawl Crawl and extract structured data from sites without running proxies and headless browsers. Memory, RAG & data194.3k6.7k
  5. Scrapling Scrape sites with a parser that survives redesigns, plus a spider framework for large crawls. Browser & computer use89.2k4.2k
  6. crawl4ai Extract structured JSON from pages with CSS schemas and no LLM cost, self-hosted via Docker. Browser & computer use86.1k1.4k
  7. GLiNER Extract entities and redact PII from large document collections on CPU, with no task-specific training. Models & local AI4.1k
  8. WeKnora Ingests and syncs documents from multiple sources into searchable knowledge bases and wikis. Memory, RAG & data35.1k3.4k
  9. semantica Builds governed knowledge graphs from warehouse tables in Databricks, Snowflake or SAP. Memory, RAG & data14.2k587
  10. PixelRAG Index folders of PDFs with tables and diagrams where text extraction loses structure. Memory, RAG & data10.3k216

    ↑↓ to move, Enter to open, Esc to close