gocolly/colly
Colly provides a collector interface for writing crawlers and spiders in Go, with callbacks for HTML elements, requests and responses. It manages request delays, concurrency per domain, cookies, sessions, caching, robots.txt and non-UTF-8 encodings, and supports synchronous, asynchronous and parallel scraping. Extensions add features such as distributed crawling, and it handles more than a thousand requests per second on a single core.
Maintenance activity