# unclecode/crawl4ai

> Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN

Navigation: [Artificial Intelligence](category/artificial-intelligence.txt) · [Index](../llms.txt)

## Facts

- GitHub: https://github.com/unclecode/crawl4ai
- HTML: https://awesomeopensource.dev/project/unclecode-crawl4ai
- Website: https://crawl4ai.com/?ref=github-about
- License: Apache-2.0
- Language: Python
- stars: 87254
- Topics: ai, ai-agents, crawl4ai-cloud, crawler, data-extraction, llm, markdown, mcp, open-source, playwright, python, rag, scraper, web-crawler, web-crawling, web-scraping
- Activity score: 4.9/5 (Thriving)
- Last commit (any branch): 2026-11-10
- Last release: v0.9.5 2026-10-14

## Description

Python web crawler and scraper that converts pages into clean, structured Markdown for LLM and RAG pipelines. It extracts structured JSON through CSS, XPath, regex or LLM strategies and drives real browsers with persistent profiles, proxies, stealth mode and async crawling across Chromium, Firefox and WebKit. It runs as a library, CLI or token-protected Docker server exposing a REST API and MCP endpoint, supports deep and adaptive crawls, and is Apache-2.0 licensed.

## Maintenance activity

- Recency: 99% (weight 40%) — last commit: 1 days ago
- Commit volume: 100% (weight 25%)
- Releases: 95% (weight 25%) — last release: 28 days ago
- Bug fixes: 100% (weight 10%)
- Commits: 186 / 90d, 596 / 365d

## Similar projects

- [openclaw/openclaw](openclaw-openclaw.md): Open-source AI assistant that lives on your own hardware and in your chat apps · Activity score 5.0/5 (Thriving)
- [n8n-io/n8n](n8n-io-n8n.md): Fair-code workflow automation with a visual canvas, custom code and 1500+ integrations · Activity score 5.0/5 (Thriving)
- [langgenius/dify](langgenius-dify.md): Visual platform for building LLM apps with RAG pipelines, agents and model integrations · Activity score 5.0/5 (Thriving)
- [open-webui/open-webui](open-webui-open-webui.md): Self-hosted web UI for chatting with Ollama and OpenAI-compatible models offline · Activity score 5.0/5 (Thriving)
- [langchain-ai/langchain](langchain-ai-langchain.md): Python framework for building agents and LLM applications from composable components · Activity score 5.0/5 (Thriving)
- [paperclipai/paperclip](paperclipai-paperclip.md): Manage a team of AI agents with goals, budgets and task tracking · Activity score 5.0/5 (Thriving)
