Crawler is the component of a search engine that gathers listings by automatically "crawling" the Web - A search engine's "crawler" (also called a "spider" or "robot"), follows links to web pages. It makes copies of the web pages found and stores these in the search engine's index.
From : www.anduro.com/glossary.html