A crawler, or web crawler, is a program that search engines use to crawl the web . Bots visit web pages to collect information and add to or update a search engine's index.
They are also known as bots, spiders, or web crawlers.

Crawling or Tracking

The process of collecting information, using a crawler, from billions of public web pages to update, add, and organize web pages into a search .

Crawling: the process by which search engines discover your web pages.

Tracking guidelines

Instructions for the crawler about what you want to crawl and index on your website.

Crawl Budget

"Crawl budget" is the total number of URLs that search engines can and want to crawl on a website during a specific period of time.

Crawl Error

URLs that a search engine bot cannot crawl. URLs that return a status code error, usually 404.