Topic

Site Crawler

A site crawler follows links through a website the way a search engine does and records what it finds at each URL. It is how you see a site's real structure, its broken paths and the pages nothing links to. What it reports depends on how it is configured, so understanding the settings matters as much as reading the output.

1 guide on Site Crawler.