Google Crawling of the Website is the process by which Googlebot discovers new and updated pages to be added to the Google Search Engine index. Google use a huge set of computers to crawl billions of pages on the web. The program that does the Crawling is called Googlebot and also known as a robot or spider. Googlebot uses an algorithmic process: computer programs determine which sites to crawl, how often, and how many pages to fetch from each site.

Google’s crawl process begins with a list of web page URLs, generated from previous crawl processes, and augmented with Sitemap data provided by webmasters. As Googlebot visits each of these websites it detects links on each page and adds them to its list of pages to crawl. New sites, changes to existing sites, content updates on the web page, image updates and dead links are noted and used to update the Google index. Google doesn’t accept payment to crawl a site more frequently, and we keep the search side of our business separate from our revenue-generating Adwords service

Google Indexing: Googlebot processes each of the pages it crawls in order to compile a massive index of all the words it sees and their location on each page.

1. Google will process the information included in key content tags and attributes, such as Title tags and ALT attributes.
2. Googlebot can process many, but not all, content types. For example, Google cannot process the content of some rich media files or dynamic pages.