spider 
Also known as a Web crawler, this class of software is used by search sites such as Google and Lycos to build the index of Web pages that their search engines use. A spider explores the Web by retrieving a document and following all the hyperlinks in it, covering vast amounts of the Internet by starting from just a few sites.