unclecode/crawl4ai
Python web crawler and scraper that converts pages into clean, structured Markdown for LLM and RAG pipelines. It extracts structured JSON through CSS, XPath, regex or LLM strategies and drives real browsers with persistent profiles, proxies, stealth mode and async crawling across Chromium, Firefox and WebKit. It runs as a library, CLI or token-protected Docker server exposing a REST API and MCP endpoint, supports deep and adaptive crawls, and is Apache-2.0 licensed.
Maintenance activity