# spatie/crawler

> Crawl websites concurrently from PHP, including JavaScript-rendered pages

Navigation: [Developer Tools](category/developer-tools.txt) · [Index](../llms.txt)

## Facts

- GitHub: https://github.com/spatie/crawler
- HTML: https://awesomeopensource.dev/project/spatie-crawler
- Website: https://freek.dev/308-building-a-crawler-in-php
- License: MIT
- Language: PHP
- stars: 2829
- Topics: concurrency, crawler, guzzle, php
- Activity score: 4.3/5 (Active)
- Last commit (any branch): 2026-11-05
- Last release: 9.4.2 2026-08-07

## Description

Spatie Crawler provides a PHP class that follows links on a website and reports each crawled URL and response. Guzzle promises fetch multiple URLs concurrently, while optional Browsershot integration runs Chrome and Puppeteer to render JavaScript sites. It supports depth limits, internal-only crawling and collected URL lists, and ships with a dedicated test suite.

## Maintenance activity

- Recency: 94% (weight 40%) — last commit: 8 days ago
- Commit volume: 55% (weight 25%)
- Releases: 84% (weight 25%) — last release: 98 days ago
- Bug fixes: 100% (weight 10%)
- Commits: 2 / 90d, 76 / 365d

## Screenshots

- [Screenshot of a Spatie Crawler terminal session listing crawled URLs and HTTP status codes](https://awesomeopensource.dev/uploads/869/5e74fbc1f3788365.png)

## Similar projects

- [stablyai/orca](stablyai-orca.md): Run several coding agents in parallel, each in its own Git worktree · Activity score 5.0/5 (Thriving)
- [tauri-apps/tauri](tauri-apps-tauri.md): Build small desktop and mobile apps with a web frontend and a Rust core · Activity score 5.0/5 (Thriving)
- [microsoft/terminal](microsoft-terminal.md): Tabs, panes and GPU rendering for PowerShell, cmd, WSL and SSH on Windows · Activity score 5.0/5 (Thriving)
- [ant-design/ant-design](ant-design-ant-design.md): A UI Design Language. · Activity score 5.0/5 (Thriving)
- [microsoft/playwright](microsoft-playwright.md): End-to-end testing and browser automation across Chromium, Firefox and WebKit · Activity score 5.0/5 (Thriving)
- [nvm-sh/nvm](nvm-sh-nvm.md): POSIX shell script for installing and switching between Node.js versions · Activity score 5.0/5 (Thriving)
