(https://github.com/ArchiveTeam/grab-site)
The archivist’s web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Created by ArchiveTeam on Feb 05, 2015.
Posted
by
David Bisset
Hashtags: archiving, crawl, crawler, Python, spider, warc
