Crawling
Crawling is the process by which search engine bots, such as Googlebot, discover and download web pages by following links and reading sitemaps.
Before a page can rank, a search engine must find it. Crawlers start from known pages and sitemaps, follow links to new URLs and fetch their content. They then queue pages for rendering and indexing.
In more detail
Crawlers respect robots.txt rules, which can block parts of a site. They decide how often to revisit pages based on how important and how frequently updated they seem. Broken links, redirect chains, slow servers and endless URL variations from filters or calendars all waste crawl effort. Pages with no internal links pointing to them, called orphan pages, are easy for crawlers to miss.
Why it matters
If crawlers cannot reach a page, it does not exist as far as search is concerned. Clean internal linking, a current XML sitemap and a fast, stable server keep crawling efficient.
A news site publishes 50 articles a day. By listing new articles in a news sitemap and linking them from the home page, it gets most articles crawled within minutes of publishing.