Finding:

1. close

   Web crawler

   A web crawler (also known as a web spider, web robot, or—especially in the FOAF community—web scutter) is a program or automated script that browses the World Wide Web in a methodical, automated manner.
2. close

   Focused crawler

   A focused crawler or topical crawler is a web crawler that attempts to download only web pages that are relevant to a pre-defined topic or set of topics.
3. close

   Crawler

   Web crawler, a computer program that gathers and categorizes information on the Internet
4. close

   Web search engine

   These pages are retrieved by a Web crawler (sometimes also known as a spider) — an automated Web browser which follows every link it sees.
5. close

   Spider trap

   dynamic pages like calendars that produce an infinite number of pages for a web crawler to follow.
6. close

   Heritrix

   Heritrix is the Internet Archive’s web crawler which was specially designed for web archiving.
7. close

   HTTrack

   HTTrack Developed by Xavier Roche Latest release 3.43 (28 September 2008) OS Cross-platform Type Web crawler License GNU General Public License Website http://www.httrack.com ... Free web crawlers
8. close

   Archive site

   By using a web crawler the service will not depend on an active community for their content, thereby building a larger database faster, which usually results in the community growing larger as well.
9. close

   Deep Web

   Ntoulas et al. (2005) created a hidden-Web crawler that automatically generated meaningful queries to issue against search forms.
10. close

    Web archiving

    Some web servers may return a different page for a web crawler than it would for a regular browser request.

Explore the following pages on Powerset:

• Web crawler,
• Focused crawler,
• Crawler,
• Web search engine,
• Spider trap,
• Heritrix,
• HTTrack,
• Archive site,
• Deep Web,
• Web archiving